Best practices for CI for embedded platforms (ESP32/Arduino)

Question

Development processes for embedded software always lags behind that for web development. When looking at unit testing, automated builds and code analysis there are many limitations to popular tools.

From experience I have already dismissed the Arduino IDE to using platformIO with VS Code since, in regards to CI, it provides improved dependency installation and build configuration automation.

The question is what tools and processes already exist and which can be repurposed to be used within the Arduino framework to enable CI as with web development.

Meta: the next most relevant stack exchange group would be IoT, but not a single post mentions continuous integration

Answer 1

Since "deployment" to the IoT device, especially in bulk, will not be done using the CI system. Then the purpose of the CI system is mostly to make sure that it will work okay.

This means that you mostly want to run some tests using the CI. Automated testing is definitely possible, both for unit tests and for integration tests.

The most important thing for unit tests is usually very short runtime and a short feedback loop. For a large code base, if possible to compile parts of it using the native compiler of the host and run unit tests there - it is a great option. If you must, you can also cross-compile to the target platform and run the unit tests using some emulator to have the necessary speed. All of this can happen during a CI "build+test" job. Compiling for the target and running unit tests on the target will just be painfully slow if done properly, but can also be done - just make sure to only flash the smallest possible piece of code and have it report back via the Serial that it worked on the device.

For integration tests, if possible to install your code on an actual device and the have some chatter with the device using the serial interface - you can ensure that the code actually works on a target device. If for some reason the code fails or works incorrectly, maybe draws too much power and/or restarts then the integration tests will catch that. You can also have another test device performing measurements and reporting via the Serial interface to the CI job. Can use an Arduino, or another ESP to maybe measure voltage and/or amperage during the execution of your code parts and checking that against known good values. You don't expect the device to pull 400mA when in a deep-sleep mode, for example, this is easy to test with a "test harness".

All of these unit tests and integration tests can be triggered completely automatically on the CI server. Naturally, you will probably have this "server" running on-premises and not rely on a cloud solution that might or might not support your latest integration testing.

With hardware, smoke tests are more "real", since you can actually hook-up the device and see the magic smoke come out. Usually, you don't even need to run code before it burns since the problem is often in the hardware.

Since I mentioned testing, there is also integration testing which is done on the hardware level. If you have a good test harness that checks that the device works well by probing it, you can use the same test harness to do your QC cycle in the factory where they produce the devices later. There is quite a lot of information about making these testbeds. One example for this is here - https://www.sparkfun.com/tutorials/138.

Answer 2

From your comment

I was hoping for unit tests of target code, some analysis à la valgrind all the way to CD

I had an answer to the Arduino StackExchange question that was mentioned, which addresses some of this. But since writing that answer I've added some memory-checking features to arduino_ci [of which, full disclosure, I am the author].

Although it doesn't do static analysis, it can detect certain kinds of memory problems while running unit tests:

==2284==ERROR: AddressSanitizer: heap-use-after-free on address 0x603000000280 at pc 
0x0001032b987e bp 0x7ffeec94c0b0 sp 0x7ffeec94c0a8
READ of size 1 at 0x603000000280 thread T0
    #0 0x1032b987d in Adafruit_WS2801::show() Adafruit_WS2801.cpp:203
    #1 0x1032b84b1 in Strip::init(unsigned char, unsigned char) Strip.cpp:10
    #2 0x1032baaff in test_set_strip_values::task() strip.cpp:10
    #3 0x1032bd900 in Test::test() ArduinoUnitTests.h:165
    #4 0x1032bcfd4 in Test::run(Test::ReporterTAP*) ArduinoUnitTests.h:137
    #5 0x1032bcb79 in Test::run_and_report(int, char**) ArduinoUnitTests.h:155
    #6 0x1032bcab8 in main strip.cpp:43
    #7 0x7fff769d0114 in start (libdyld.dylib:x86_64+0x1114)

More context in the pull request that fixes this problem