NF Unit Testing

[kevindweb]

Feature Request for Unit Tests

Is your feature request related to a problem? Please describe.
We currently don't have a way to test arbitrary NFs that are developed onto the platform. This means that if new bugs come into ONVM, CI doesn't know unless they affect Speed Tester or Basic Monitor.

Describe the solution you'd like
Unit testing would be a great way to fix this. Every new NF would need a testing structure that the developer creates before making a PR. Unit testing tools for C exist already that we should take advantage of if possible.

Describe alternatives you've considered
An alternative to the prior Unit Testing strategy, we could make a test version of the onvm manager. The thought here is how do you test an NF? NF's are not library code that provide explicit output, based on easy inputs like strings or integers. As a result, simulation of packets similar to speed tester would be required to test the Firewall NF for example.

Resources
This is a thread to get a head start on this for the summer. I will post here more material that I find online. The DPDK Test Suite (DTS), with documentation here, is probably a good place to start. As @Vivek-anand-jain mentioned, we should have a make test functionality for initiating all tests at once. I know unit testing was a huge advancement for Gabe's Composite project, so this would help advance the security and reliability of the onvm platform significantly.

06/29/2020 Meeting Notes

After a meeting with Tim, here's the roadmap for the ONVM Testing Suite

3 Main Tests

1. Generic "Valid NF" Tests to check that the NF can handle packet flow through the API
   • Can this NF handle 100 packets or 10 seconds consecutively without failing (ignoring speed or output)
   • TCP/UDP packet flow without crashing
   • Potentially IPV4/6? (will break some NFs currently)
2. NF-specific (developer-written) Unit Tests
3. Functional Tests from an Initiator to the NF under test

Specifically for number 3, how do we verify the test passed? This will be up to the NF developer. There are 3 ways to determine this. They will need to register this choice upon starting the test framework.

• Packet Data (analyzed from PCAP)
  • Example - AES Encrypt, the output PCAP should be each packet's body encrypted (easy to create and check)
• Trace Log - action taken for every specific packet
  • Example - Firewall, based on specific set of rules, we know whether the packets should be dropped
• Console Output - redirected stdout to a logfile
  • Example - Basic Monitor, no change in the packets or trace log, but stdout should be consistent based on packet flow

NFlib Accommodations

A testing flag (or macro) will be supplied for the manager to simplify the analysis of the unit tests.

• Every packet the Unit Test NF processes, whether that be dropping, sending to another NF, etc. will be logged
• The packet (if it wasn't marked "dropped") will be output using the libpcap library to an outfile

Analysis

A python program to analyze the result of the unit test and confirm everything worked correctly. This will also be in charge of the setup and teardown of the NFs. The unit tested NF will need to specify their "choice" in what data will determine their validity. For the PCAP file and/or trace log, the output can be MD5 hashed against what the expected result. If they're different, the NF failed in some way

First Steps

• Create an NF similar to Speed Test that can read in a PCAP file (possibly called functional_test NF)
• Take Firewall as an example to write a "tests.c" that can be run with make test firewall
• Python program that mimics the NF Chain Python config, spinning up functional_test and calling the firewall's test function.
  • Teardown and analysis on failure or after the PCAP file has been consumed
• Adjust the onvm_nflib code to check for test flag - check the unit tests choice of pcap, trace log, or stdout, and process their results accordingly - probably in a macro to avoid performance issues outside of test

[dennisafa]

This is a good idea. This would have to be coupled with something that automates packet sending to the NF on startup..maybe an instance of speed_tester with a standard pcap file? I think it might be difficult to do it with pktgen on each test since that'll require a 2 node setup for the unit tests.
I also wonder how abstract the NF unit tests should be. Are we just testing for segfaults, or specific NF outputs? It might require the NF unit test structures to be loosely defined -> inspected by us when we do PR reviews.

[bdevierno1]

I was thinking once we get @catherinemeadows pr merged we will create a json with a NF chain of most of our main NFs. We will need to figure out what will be the expected output of this NF chain.

[bdevierno1]

This is a good idea. This would have to be coupled with something that automates packet sending to the NF on startup..maybe an instance of speed_tester with a standard pcap file? I think it might be difficult to do it with pktgen on each test since that'll require a 2 node setup for the unit tests.
I also wonder how abstract the NF unit tests should be. Are we just testing for segfaults, or specific NF outputs? It might require the NF unit test structures to be loosely defined -> inspected by us when we do PR reviews.

I think this is true it might be a bit harder to test with CI as maintainers will have to look out for NF outputs as well as other things such as seg-faults. I think we could make a platform such that it is easy for developers and maintainers to run a broad range of test locally.

[kevindweb]

Here are a few good resources we should use to get started on this. This guy has a nice article talking about CMOCK. It might work for our case, but the main point is that we need to view NFs as infinite processes that need to be "bounded" in some way for the testing of their functionality. The article points to a series of C related testing ideas.

[kevindweb]

@sdnfv/gw-undergrads I updated the issue guide with the title "06/29/2020 Meeting Notes". Take a look to see if you want to participate in the Unit Testing framework. These notes are from a meeting with Tim this afternoon.