Quality assurance is one of the key features of picoTCP. Our QA team is constantly testing picoTCP by running several Jenkins jobs. This Continuous Integration (CI) process is very powerful in giving fast feedback to changes in the picoTCP code. However, constantly checking the status of the Jenkins jobs is cumbersome and unlikely to happen in the heat of developing. At the picoTCP workspace, we used the Gitgear Extreme Feedback lamp to monitor our picoTCP Jenkins jobs and give everybody the latest status of picoTCP. This way we instantly see if there are failing tests and can take rapid action to fix the issue at hand.
Our QA team had a great idea: what if picoTCP would monitor its own quality? The electronics for such a lamp are basic and we have the necessary technology: the idea of a board running picoTCP and a lamp was born.
The lamp, which can be green (everything is OK), yellow (there are issues) or red (critical error) reacts on the result of a Jenkins job. Furthermore, the colour can be solid or flashing, the latter indicating an ongoing build or test execution.
We chose a thin client and fat server model. The client is responsible for displaying the colour while all the heavy lifting (which jobs are connected to which lamps, transmitting the state of jobs, …) is done on the server side.
The connection between the lamp and the jobs at the Jenkins server is provided by a plugin written in Java. It keeps track of the various lamps connected to the Jenkins server.
Managing CI lamps is easy using the auto-discovery function of the plugin. Once discovered, a lamp can be given a name and jobs to monitor with a few simple clicks. The layout of the plugin is mostly based on the existing code by Gitgear. Though our CI lamp code differs significantly. The Gitgear lamp works as follows:
When a job execution is finished, the server will check if a lamp is monitoring its status and will send an update accordingly.
Our lamp/plugin setup was designed the other way around.
It will contact the server periodically to check if an update is available.
The reason we use polling rather than pushing is mainly due to the fact a firewall could block incoming connections. If we contact the CI server from inside the network we won’t have any connection issues due to firewall setups.
We extended the Remote Access API provided by Jenkins to have communication between lamp and server.
Jenkins can be fully controlled remotely in a REST-like way. For example, shutting down or restart Jenkins can be done by surfing to http://myjenkinsserver:8080/exit but you can also add or remove jobs using a HTTP post request and even trigger a new build. Plugins can extend this API in such a way that you can extract info from Jenkins.
We use this to provide the colour, state and name for the lamp. The lamp will issue a GET request to the URL http://myjenkinsserver:8080/plugin?macAddr=myMacAddr, after which the Jenkins server will respond with a JSON formatted message:
Name : "picoTCP IPv6 Tests",
Text : "A build error occured",
The MAC address is supplied as a GET parameter to uniquely identify the lamp. If a lamp is unknown to the Jenkins server/plugin, it is added it to the list. The return message will be parsed by the lamp firmware.
Notice that the above method assumes that the Jenkins server is publicly reachable.
The client consists of an STM32F4 running a picoTCP application with an extreme feedback lamp attached to it. The board will contact the Jenkins server and ask which colour to show as described above. We use the native libhttp module of picoTCP to issue the GET request to the server. For parsing the JSON replies, the jsm library is used which was written for embedded systems.
But first things first, how does the lamp know which Jenkins to contact? As mentioned earlier, if the lamp and the Jenkins server are situated in the same network, the Jenkins server can auto-discover the lamp by sending out a multicast-message to the lamp. The lamp will respond by issuing a GET request to the Jenkins server. This way the lamp will be registered and you’re done!
But it’s possible the Jenkins server and the lamp are not in the same network. That’s why we provided a configuration mode in the lamp.
When booting the lamp, you can select configuration or normal mode.
If you select configuration mode, the lamp will choose a static IP address and set up a local HTTP server
The local server will provide a configuration page to set the URL of the Jenkins server to contact
After a reboot the lamp will contact the configured Jenkins instance.
Our CI project would not be complete if we didn’t provide continuous integration for our CI lamp as well. The firmware and plugin both have unit and functional tests which are run on our Jenkins server. This means we can monitor our CI lamp code quality by attaching a CI lamp to our Jenkins jobs.
The plugin tests are written using JUnit while firmware unit tests use the check framework. For functional tests, we set up a local HTTP server written in Python and compile our lamp in sandbox mode. This enables us to run the CI lamp firmware on any Linux machine, so we can easily check if the right action is taken when receiving a colour using the standard stdin and stdout.
The lamp is fully operational but there are still plenty of things to add such as
an LCD which can display failing jobs or basic info about the lamp
attaching sound to the board to further indicate successes or failures
adding keep-alives to the Jenkins plugin to monitor the lamps