Testing the stack
Even though we all understand the importance of testing some of us might not like this part of development. Using the proper tools and developing tests from the beginning makes it easier and can relieve a lot of stress down the lane. This is exactly how the picoTCP project works.
Tested in many different ways and on multiple levels, ranging from unit testing all the way to RFC compliance, we make sure the quality of picoTCP is what you’d expect of a product you want to incorporate in your own solution.
Running unit tests is done with Check. It provides you with an easy-to-use interface to write your tests but doesn’t bother you with a bunch of details. This results into a unit testing set that is clean and very readable.
Unit testing is very often used in conjunction with mocking, the act of replacing a piece of code with something that acts similar but that allows you to choose the input and output values. Mocking is important to have full control over one single ‘unit’ (e.g. a function). Very often a unit relies on another function, a peripheral… Testing both the dependencies and the API of a unit at the same time allows you to cover all possible usages.
Dozens of frameworks for mocking exist, but it is quite easy to add a mocking feature yourself. The GCC linker provides us with the “weak” attribute for functions. Any weak function can be overridden by an equally named function. The linker will take care of it for you. If you have a look at pico_config.h you’ll see the MOCKABLE define is there for you to use.
Lastly, all of the unit tests are compiled together with the entire picoTCP source. It’s noteworthy that we don’t include the header files, but rather include the source files. This has multiple advantages: It verifies that you do not have any namespace collisions (whether they are functions or global variables) and it allows you to test static functions that would otherwise not be accessible. Running this executable will either return a pass or a fail very soon after a change has been committed.
The next stage uses smoke tests to ensure the basic functionality of a full picoTCP implementation. We check if multiple picoTCP instances can communicate with each other through Virtual Distributed Ethernet plugs in a Linux environment.
These first series of tests can quickly catch any malfunction by checking the return values of the API calls. However, this process doesn’t cover every little detail, which brings us to...
A picoTCP version containing all modules and features is tested by a test framework we created based on the Robot Framework. High level tests are executed to make sure the stack meets all the requirements in the RFCs we claim to implement. It’s important that during development functionality isn’t accidently changed or broken. By running these tests the team can sleep soundly at night without worrying to wake up to angry picoTCP users.
To measure the coverage, the framework has a tag for each test which refers to a specific section of its RFC. In the blink of an eye anyone can see if something is up.
All scenarios are tested with picoTCP running on a platform it is designed for: a microcontroller with limited resources.
The framework is running on a Linux machine which can access the API on the device that is being tested over serial port. For example, using remote procedure calls it can initialize a TCP server, send the server some packets over the network and check if the results are correct. Verification is done both on the replied packets and the API return codes. At the end of every scenario, a hard reset of the environment is done so every test can start with a clean sheet.
We use the TAHI and Robot frameworks to cover the IPv6 RFC requirements. TAHI has a slight different setup than Robot Framework. A network connection is created to talk with an application using the picoTCP stack running on an MCU. Based on the packets that are sent back, TAHI decides if the testcase has passed.
To ensure that picoTCP keeps running without leaking memory or crashing due to long term operation, a special application was developed to perform endurance tests. This runs on a microcontroller that communicates with a mocking script over the network. This script controls the picoTCP test application through a TCP socket while also serving as a remote side network to exchange data with.
This test cycles through each specific part of the stack’s functionality (UDP unicast, multicast, TCP server/client, DNS, DHCP,...) in an endless loop without rebooting.
The mocking script is also used to log memory statistics that are sent over the network as well as the state of the network itself and the traffic going through it.
If for some reason network traffic performance is not as good as expected or a response is taking too long, the test will be marked as failed and an investigation is started to find and fix the cause.
Every time a change is made to the code our Jenkins jobs compile picoTCP for a range of targets with different module compositions. When a build is successful, unit tests as well as smoke tests are done. When those results are positive, static code analysis is executed by TIOBE TICS which measures code quality metrics such as
Code coverage (Bullseye Coverage)
Abstract interpretation (CppCheck Parasoft)
Coding standards (TASS coding standards)
Next to the triggered compilation and validation, tests are scheduled to run every night as well. Detailed reports are delivered to the development team at the start of each day.