$80,000.  That was the amount I was recently quoted for a certified RTOS package.  The amount was per processor configuration, with an expected lead time of about eight months. The therapy device my client was developing had two similar, but not identical ARM processors, so their cost would have been $160,000.  The package would have included a customized report for each CPU configuration, support for representing the RTOS package to the FDA, and changes to that package if the FDA required them.

It was a big ask: why should they pay $160,000 to substitute the safety-certified version of an RTOS they were already using for free?  The eight-month lead time was also a concern, since that would have been pushing right up against the window when they were planning to be done.

This was a major decision, and the time constraints wouldn’t allow much time to think about it.

The client elected not to purchase the package, to stay with the free version of the RTOS, and tackle the full certification effort on their own.  The remainder of this article examines why I think they made a mistake – probably a pretty costly one.

There are many free and open-source RTOS on the market.  While not an RTOS, embedded Linux has also gained broad adoption for device software, and for many devices, its performance is sufficient.  In the realm of true real-time operating systems, the list includes FreeRTOS, Zephyr, NuttX, RTEMS, RIOT, µC/OS, and others.  For many of these, there is paid commercial support available for general use.  RTEMS is considered readily certifiable and has been used for safety-critical projects, especially within Aerospace.  For some others, there are commercial versions that have been intentionally tailored for certification in safety-critical systems.   SafeRTOS, an adaptation of FreeRTOS, is an example.  There are several other commercial RTOS that are suitable and often used in safety-critical systems, and some are excellent; but what I’m describing here are those that let you begin development on a free version of the OS.  It also may be easier to find developers who are familiar with free ones, since they are often the choice of hobbyists – many of whom are also professional developers. 

General-purpose RTOS are generally designed for utility and reliability, but not necessarily for assurance of safe operation in a safety-critical system. Their designs may omit checks, restrictions, and failure handling that a safety analysis would identify as necessary for the intended safety role.  Safety-critical systems have concerns about memory corruption, resource exhaustion, etc.  As a developer on one of these platforms, you may know exactly what your code is doing, but even if you’ve followed every precaution, you don’t necessarily know how the RTOS will accomplish what you ask of it.  You may have an intuition about it, but you’d have to analyze or trace it to be sure.  In a common application, you probably make thousands of calls to the RTOS.  Add to that the complexities of scheduling and interrupt handling and the specifics of these for various MCU platforms, and the safety argument you’re trying to make on behalf of your RTOS can become quite challenging.  This is even worse if your developers are not clear in advance about where they need to take precautions.

At this point, you may realize that there are still some certifiable safety-critical RTOS that can be had for a relatively low cost, and even some that are free.  Certifiable is not the same as certified.  Certifiable just means that there is a certain path to certifying them on your hardware. 

An RTOS designed or tailored for safety-critical work has undergone extensive analysis to ensure that relevant safety properties hold over its defined operating conditions, provided that the developer follows the vendor’s usage guidelines.  This makes the safety argument substantially less risky, but it doesn’t reduce the obligation for a thorough analysis and verification on the selected MCU configuration.

Finally, there is the safety-critical RTOS that is pre-certified by a vendor.  This is a certifiable safety-critical RTOS for which the vendor provides the analysis, documentation, and testing needed to support certification of the RTOS on a defined MCU and toolchain configuration. The device manufacturer must still show that its integration remains within the RTOS vendor’s documented conditions of use.

“Pre-certified” is a bit of a misnomer.  Given the way the process often works, unless the vendor has previously done this exact configuration for another customer, they may not actually finish the work until months after your purchase.  Because of this, maybe “certified-on-demand” would be more accurate than “pre-certified”, although it is still the device manufacturer’s responsibility to gain the certification.  Nonetheless, the vendor takes on much of the RTOS-specific verification work and evidence creation, allowing the device team to focus its efforts on integration and the device’s primary function.

A common misconception, sometimes held by even the most senior developers, is that thorough system testing makes separate verification of the RTOS unnecessary. In safety-critical work, that does not hold true. System testing can show that a device behaved correctly in the conditions the tests exercised. It cannot, by itself, establish that the operating system will behave safely across the timing, interrupt, memory, fault, and resource conditions that may occur during the life of the device. A rare scheduling interaction, exhausted resource, corrupted internal state, or hardware fault may not appear in ordinary system testing at all.

 

That is why the RTOS has to be treated as a component with properties of its own. The safety argument cannot simply assume that it will schedule, allocate, recover, and fail in acceptable ways because the application appeared to work in testing.

An important consideration in selecting your path is the safety assurance level required for the software and the system it supports. At the lower end of the relevant safety schemes, such as IEC 62304 Class A, IEC 61508 SIL 1, or DO-178C DAL D, the analysis required to certify even a general-purpose RTOS may be feasible, though it is still not a zero-effort endeavor. At the higher end, where software failure could contribute to death or serious injury, such as IEC 62304 Class C, IEC 61508 SIL 3, or DO-178C DAL A, a vendor-supported pre-certified RTOS will almost certainly be worth considering. Between those extremes is where there is room for debate.

In my earlier example, $80,000 for a pre-certified RTOS sounds quite expensive.  Consider that if you were to replicate the effort at a burdened rate of $150 that might be expected for a highly qualified team, that amount reflects slightly more than thirteen man-weeks of effort.  That’s thirteen weeks to analyze and document all of the relevant calls to the RTOS, and develop a test suite sufficient to show that, on the specific MCUs using the specific compiler version, and under all kinds of loading conditions that might occur for this system, the calls function as expected or fail in a way that can be identified and recovered from.  Then, for the latter, it needs to be ascertained that the application code calling these functions is actually watching out for the possible errors.  For a reasonably limited scope of calls to the RTOS, it seems possible to achieve this in 13 weeks, but far from certain.  If you succeed, you’ve only broken even with the commercial offering, and it cost you schedule time for one or two of your team members.  Now consider if you, like my client, have two different MCUs, with two different toolchains; so your effort and cost is essentially doubled.

When considering the value of a pre-certified RTOS, consider that the vendor has probably helped dozens, if not hundreds, of customers certify their systems for safety-critical use.  For the RTOS they support, they have performed these analyses, in depth, with every new release.  They have most likely maintained their analyses in configuration management, in the same way that developers maintain source code. With each iteration, the analysis can evolve, getting more complete, going deeper where needed.  They have developed document templates, fully reusable and extensible test suites, and all of the other resources needed to make this type of certification package reliable, repeatable, profitable for them, and cost-effective for their customers.

Continuing with the $80,000 example, here’s what you would be really getting for the money:

Despite the obvious benefits, there may be good business reasons for not selecting the pre-certified RTOS path. If the device has a relatively low safety-assurance level and uses only a small, well-bounded subset of RTOS functionality, the necessary analysis may be feasible for the development team to perform itself. The calculation changes further if the organization has retained relevant analysis, documentation templates, and test cases from a markedly similar prior effort. A vendor package may also be a poor fit when the product uses an unsupported MCU or toolchain configuration, or needs RTOS modifications beyond the vendor’s documented conditions of use. In those cases, building and maintaining the RTOS-specific evidence internally may be the more sensible decision. A limited production run may also favor internal development of the RTOS-specific evidence, but only where that approach still provides the required level of assurance at lower total program cost.

There are tough choices to be made when selecting an RTOS for a safety-critical application. The right comparison is not the license cost of the RTOS against the price of a certification package; it is the total cost, schedule risk, and assurance burden of producing a credible safety case for the configuration actually being built.