Ensure software quality by early and automated testing

How to create high-quality software through early testing in the embedded system development of applications with 16/32-bit microcontrollers? Why is this an issue affecting many embedded software development projects?
 

Microcontrollers have an enormous spread and are essential components of embedded system projects. Just think of the control of electric drives. Today's motors still follow the electromechanical principles from their early days. The adaptation of these motors to the most diverse application profiles is now generally based on controls with microcontrollers. For algorithms of motor controls, program packages with more than half a million lines of code are not uncommon.

In addition, product families are typical for these markets, and their diversity is also reflected in correspondingly different and hence complex software environments. And finally, many developers are working on such projects, and not just at one location. To keep track of program changes and hold code integrity is a big challenge here.

Early testing

Experience shows in many embedded system projects software is not tested early enough and testing is not automated! Why?
 

• The target electronics are often not yet available. Development takes place on evaluation boards, for which there are typically no test systems.
• The usual hardware-in-the-loop (HIL) test systems are often too large and too expensive to allow continuous testing during development.
• Test systems in combination with simulation elements are technically complex and difficult to automate.
• Unit tests are poorly suited for testing hardware-related, concurrent or asynchronous functions (e.g. drivers and motor controllers).

Protos is looking back more than 20 years in the design and development of very many embedded software projects. The analysis of many customer projects made it obvious:

Testing early in the project, with a cost-effective test tool (hardware & software tool chain), is a method to significantly improve the software and product quality. Furthermore, early testing accelerates the development process and reduces costs.

Thus, the Protos miniHIL test system was launched in 2017 in a customer project and now it is a proven tool in many successful embedded system projects. miniHIL is a small (size of a DIN A4 letter page) Hardware-in-the-loop test system, that fits on a developer’s desk, is by far cheaper than a traditional HIL-system and is supported by a rich software tool chain. It is designed to address exactly the requirements and testing gap summarized above. Furthermore, it offers test automation by Hardware-in-the-loop testing with Continuous Integration.
 

How does it fit together?

Any small code change to an embedded system can have the most malicious effects. Moreover, if you find and fix the problems too late, the fix is very expensive. Therefore, it should be possible to test every change to an embedded system within minutes. This is the only way to ensure that you can detect and fix the problems quickly.
For pure software projects, complete automation of the tests via Continuous Integration helps. Here, every code change automatically triggers the build and test of the complete software. This allows a full regression test to discover bugs in existing and new code within minutes.

But how can this be achieved for system or integration testing of embedded systems? Hardware-in-the-loop tests or manual system tests required for this are often difficult to automate.
 

• Real hardware parts in the test setup prevent automation and repeatability (motors, batteries, sensors, …)
• Only the “happy path” can be tested, but not error conditions
• The test bench is often too expensive to be available for automation for every project at any time

Lightweight hardware-in-the-loop testing with the miniHIL allows you to fully automate testing via Continuous-Integration.

How does it work?

• Hardware interfaces are only simulated on signal level, and therefore fit well into test automation
• Test hardware is largely replaced by software simulation
• Faulty behavior can be easily simulated and tested via faulty signals
• The necessary hardware is significantly simpler, smaller, and cheaper than traditional HIL systems. A separate test setup for automation is available for each project at any time

Thus, any change can be tested automatically within minutes. Only through complete and fast test automation complex embedded software can be developed effectively in teams. In a test rack numerous projects - different hardware and software variants - may be tested in parallel locally and remotely. The continuous integration keeps track of program changes and holds code integrity in those complex development environments.

In a cooperation with our partner Hitex a webinar on this topic was recorded to provide a deeper look into this technology. You can watch the webinar here: