Clarification about "update" flow-events: when are they sent? and based on what logic?

[verzulli]

I'm trying to understand the gory detail of "flows" lifecycle (when they "start" [NEW]; when they "finish" [END|IDLE]; what events are triggered during their life [DETECTED|NOT-DETECTED|GUESSED|DETECTION-UPDATE]; etc.).

I'm basing my reasoning on the graph reported here.

In addition to events reported in the graph, I'm registering "UPDATE" events. As I see that they are sent "...based on various conditions...", I'm interested in better understanding their role during all the "flow" lifecycle.

The main problem such UPDATE events are posing to me, related to "FLOW duration" or, better, to the time-interval between the "NEW" and "END|IDLE" events

For TCP-based flows, there are no big problems, as "END" event should be fired when a FIN or RST packet is recognized by nDPI.

Problems are for UDP-based flows, where there is NO clear indication about the latest packet to be associated to such flow. Based on what I see, even if the UDP-protocol has terminated the L7-connection, from the nDPId point of view it stay "alive" at least since some timeout occur (due to not seeing new traffic related to such flow). While waiting for such timeout, several UPDATE events are fired.

Here is an example regarding a DNS query:

Here you can see:

• some overall flow-related data (top/left)
• details of the 7 events collected for such DNS-query (table at the bottom)

In the table you can see that "NEW" and "DETECTED" events are basically sent concurrently, at the very start of the flow (time in the 3rd, 4th and 5th columns are relative to "first_seen"). You can also see that those two events are generated based only on the SRC request (I guess, the very first UDP packet containing the DNS request). You can see this in the 35 byte payload of the single packet in the SRC->DST direction.

After a tiny amount of time, nDPI recognize the DNS query attributes and send the 3rd event (DETECTION-UPDATE). Here you can see that this data are captured from the DNS reply, as in the 3rd event we have 1 packet reply, with 114 byte payload.

I guess that at this time everything is completed (in terms of traffic analysis) as nothing is expected to happen later on.. And actually, event 4, 5 and 6 are UPDATE that basically add nothing the detection behaviour. Finally, at 7, an IDLE event is sent, probably 'cause some timeout have been triggered (...I guess, due to the absence of activity).

If all the above is correct, my problem is that:

• the "real" FLOW duration shoud be 53.991, actually the MAX value between src_last_pkt_time and dst_last_pkt_time as captured by the IDLE event;
• the "calculated" FLOW duration, between flow_first_seen and thread_ts_usec of the IDLE event, is 184.490.446, definitely way higher than the real value.

So, in the end, here is my question:

• which is the login behind "UPDATE" events? Can I fine-tune nDPId to better adapt them to my needs?
• as for the IDLE event, which is the "TIMEOUT" related, and how can it be tuned?

Thanks in advance!

[utoni]

Hey Damiano,

thanks for starting this conversation.
The question regarding the role of update events was (partially) asked before (FYI: #41).
I'll need to update README.md with a more precise and detailed description of the aforementioned event.

The current documentation states:

4. update: inform nDPIsrvd or other apps about a long-lasting flow, whose detection was finished a long time ago but is still active

Regarding your problems/questions:

the "real" FLOW duration shoud be 53.991, actually the MAX value between src_last_pkt_time and dst_last_pkt_time as captured by the IDLE event;

There is a difference between the "real" flow duration and the lifetime of a flow within nDPId.
The issue here is that UDP is a connection-less protocol (unlike TCP which is connection-oriented), meaning a graceful termination of a connection (i.e. TCP-FIN) does not exist.
That brings me to the idle time. nDPId needs a timeout handling of UDP flows, see udp-max-idle-time.
Otherwise, the flow would always consume memory without any benefit.

the "calculated" FLOW duration, between flow_first_seen and thread_ts_usec of the IDLE event, is 184.490.446, definitely way higher than the real value.

The value of 184.490.446us (~184s) is totally fine and within the expecated range of possible values.
The last packet captured and processed by nDPId came from the source and wanted to travel to the destination. nDPId saw this packet on wire right after 53.991us (~54ms) passed either in real-time or in a recorded PCAP file. The default UDP (which I guess is the timeout you've used) max idle time is 180.000.000us (180s). In addition there is also a flow scan interval which is the time that has to pass until nDPId will scan it's memory for flows that might have been already timed-out or need an "update" event. The default value for the flow-scan-interval is 10.000.000us (10s).
Since it seems that you're reading a PCAP file, packet time (stored as timestamp in the PCAP file) and real time are different. But as long as the flow timeout is between a minimum and maximum value, everything should be fine.
The minimum value of the UDP flow timeout should be the UDP idle time of 180.000.000us (180s) in your case.
The maximum value of the UDP flow timeout should be the UDP idle time plus flow scan interval: 190.000.000us (190s): 190 = 180 + 10 in your case.
So 184.490.446us (~184s) is between 180s and 190s which seems fine to me.

which is the login behind "UPDATE" events? Can I fine-tune nDPId to better adapt them to my needs?

The idea is that other applications i.e. distributor apps may get informed of long lasting flows. Sometimes I want to dynamically activate / deactivate distributor applications. Without the update event, they would never know if there was such a flow. The update events are influenced by flow-scan-interval and udp-max-idle-time in your case.
But there is also a value which is hard coded and can not be tuned for now. An update event is sent every time 1/4 of the idle time has passed. That also explains why you get three update events shown within your screenshot. The fourth is not sent, because the flow timed already out before nDPId could check for flows that need an update event. This behavior is not documented, but it should be. But I am also thinking about changing that behavior, which is not optimal right now.

as for the IDLE event, which is the "TIMEOUT" related, and how can it be tuned?

The idle events are sent when nDPId didn't see any data on the wire related to that flow for a certain amount of time.
The timeouts depend very much on the corresponding Layer4 protocol:
For ICMP: icmp-max-idle-time
For UDP: udp-max-idle-time
For TCP: tcp-max-idle-time
For everything else: generic-max-idle-time

I hope this helps to clarify things a bit. Please give me some feedback if it was or wasn't helpful, so I can adjust README.md documentation.