2 add explanation on why unit testing on a target device is slow
source | link

Since "deployment" to the IoT device, especially in bulk, will not be done using the CI system. Then the purpose of the CI system is mostly to make sure that it will work okay.

This means that you mostly want to run some tests using the CI. Automated testing is definitely possible, both for unit tests and for integration tests.

The most important thing for unit tests is usually very short runtime and a short feedback loop. For a large code base, if possible to compile parts of it using the native compiler of the host and run unit tests there - it is a great option. If you must, you can also cross-compile to the target platform and run the unit tests using some emulator to have the necessary speed. All of this can happen during a CI "build+test" job. Compiling for the target and running unit tests on the target will just be painfully slow if done properly, but can also be done - just make sure to only flash the smallest possible piece of code and have it report back via the Serial that it worked on the device.

For integration tests, if possible to install your code on an actual device and the have some chatter with the device using the serial interface - you can ensure that the code actually works on a target device. If for some reason the code fails or works incorrectly, maybe draws too much power and/or restarts then the integration tests will catch that. You can also have another test device performing measurements and reporting via the Serial interface to the CI job. Can use an Arduino, or another ESP to maybe measure voltage and/or amperage during the execution of your code parts and checking that against known good values. You don't expect the device to pull 400mA when in a deep-sleep mode, for example, this is easy to test with a "test harness".

All of these unit tests and integration tests can be triggered completely automatically on the CI server. Naturally, you will probably have this "server" running on-premises and not rely on a cloud solution that might or might not support your latest integration testing.

With hardware, smoke tests are more "real", since you can actually hook-up the device and see the magic smoke come out. Usually, you don't even need to run code before it burns since the problem is often in the hardware.

Since I mentioned testing, there is also integration testing which is done on the hardware level. If you have a good test harness that checks that the device works well by probing it, you can use the same test harness to do your QC cycle in the factory where they produce the devices later. There is quite a lot of information about making these testbeds. One example for this is here - https://www.sparkfun.com/tutorials/138.

Since "deployment" to the IoT device, especially in bulk, will not be done using the CI system. Then the purpose of the CI system is mostly to make sure that it will work okay.

This means that you mostly want to run some tests using the CI. Automated testing is definitely possible, both for unit tests and for integration tests.

The most important thing for unit tests is usually very short runtime and a short feedback loop. For a large code base, if possible to compile parts of it using the native compiler of the host and run unit tests there - it is a great option. If you must, you can also cross-compile to the target platform and run the unit tests using some emulator to have the necessary speed. All of this can happen during a CI "build+test" job.

For integration tests, if possible to install your code on an actual device and the have some chatter with the device using the serial interface - you can ensure that the code actually works on a target device. If for some reason the code fails or works incorrectly, maybe draws too much power and/or restarts then the integration tests will catch that. You can also have another test device performing measurements and reporting via the Serial interface to the CI job. Can use an Arduino, or another ESP to maybe measure voltage and/or amperage during the execution of your code parts and checking that against known good values. You don't expect the device to pull 400mA when in a deep-sleep mode, for example, this is easy to test with a "test harness".

All of these unit tests and integration tests can be triggered completely automatically on the CI server. Naturally, you will probably have this "server" running on-premises and not rely on a cloud solution that might or might not support your latest integration testing.

With hardware, smoke tests are more "real", since you can actually hook-up the device and see the magic smoke come out. Usually, you don't even need to run code before it burns since the problem is often in the hardware.

Since I mentioned testing, there is also integration testing which is done on the hardware level. If you have a good test harness that checks that the device works well by probing it, you can use the same test harness to do your QC cycle in the factory where they produce the devices later. There is quite a lot of information about making these testbeds. One example for this is here - https://www.sparkfun.com/tutorials/138.

Since "deployment" to the IoT device, especially in bulk, will not be done using the CI system. Then the purpose of the CI system is mostly to make sure that it will work okay.

This means that you mostly want to run some tests using the CI. Automated testing is definitely possible, both for unit tests and for integration tests.

The most important thing for unit tests is usually very short runtime and a short feedback loop. For a large code base, if possible to compile parts of it using the native compiler of the host and run unit tests there - it is a great option. If you must, you can also cross-compile to the target platform and run the unit tests using some emulator to have the necessary speed. All of this can happen during a CI "build+test" job. Compiling for the target and running unit tests on the target will just be painfully slow if done properly, but can also be done - just make sure to only flash the smallest possible piece of code and have it report back via the Serial that it worked on the device.

For integration tests, if possible to install your code on an actual device and the have some chatter with the device using the serial interface - you can ensure that the code actually works on a target device. If for some reason the code fails or works incorrectly, maybe draws too much power and/or restarts then the integration tests will catch that. You can also have another test device performing measurements and reporting via the Serial interface to the CI job. Can use an Arduino, or another ESP to maybe measure voltage and/or amperage during the execution of your code parts and checking that against known good values. You don't expect the device to pull 400mA when in a deep-sleep mode, for example, this is easy to test with a "test harness".

All of these unit tests and integration tests can be triggered completely automatically on the CI server. Naturally, you will probably have this "server" running on-premises and not rely on a cloud solution that might or might not support your latest integration testing.

With hardware, smoke tests are more "real", since you can actually hook-up the device and see the magic smoke come out. Usually, you don't even need to run code before it burns since the problem is often in the hardware.

Since I mentioned testing, there is also integration testing which is done on the hardware level. If you have a good test harness that checks that the device works well by probing it, you can use the same test harness to do your QC cycle in the factory where they produce the devices later. There is quite a lot of information about making these testbeds. One example for this is here - https://www.sparkfun.com/tutorials/138.

1
source | link

Since "deployment" to the IoT device, especially in bulk, will not be done using the CI system. Then the purpose of the CI system is mostly to make sure that it will work okay.

This means that you mostly want to run some tests using the CI. Automated testing is definitely possible, both for unit tests and for integration tests.

The most important thing for unit tests is usually very short runtime and a short feedback loop. For a large code base, if possible to compile parts of it using the native compiler of the host and run unit tests there - it is a great option. If you must, you can also cross-compile to the target platform and run the unit tests using some emulator to have the necessary speed. All of this can happen during a CI "build+test" job.

For integration tests, if possible to install your code on an actual device and the have some chatter with the device using the serial interface - you can ensure that the code actually works on a target device. If for some reason the code fails or works incorrectly, maybe draws too much power and/or restarts then the integration tests will catch that. You can also have another test device performing measurements and reporting via the Serial interface to the CI job. Can use an Arduino, or another ESP to maybe measure voltage and/or amperage during the execution of your code parts and checking that against known good values. You don't expect the device to pull 400mA when in a deep-sleep mode, for example, this is easy to test with a "test harness".

All of these unit tests and integration tests can be triggered completely automatically on the CI server. Naturally, you will probably have this "server" running on-premises and not rely on a cloud solution that might or might not support your latest integration testing.

With hardware, smoke tests are more "real", since you can actually hook-up the device and see the magic smoke come out. Usually, you don't even need to run code before it burns since the problem is often in the hardware.

Since I mentioned testing, there is also integration testing which is done on the hardware level. If you have a good test harness that checks that the device works well by probing it, you can use the same test harness to do your QC cycle in the factory where they produce the devices later. There is quite a lot of information about making these testbeds. One example for this is here - https://www.sparkfun.com/tutorials/138.