A New Trick Can Save the Day

Medical device manufacturers face a steep regulatory burden.  Even the least critical medical systems must receive FDA approval before they can be placed on the market.  This Fortune 500 medical device manufacturer was preparing to ship an important, but low-risk upgrade to a stalwart family of institutional medical equipment.  Then at the last minute, the program hit a wall – at least it looked that way.

The System

The updated equipment included 2 different models of the company’s medical sterilizers – FDA Class I devices, essentially sophisticated ovens with precise timing controls, now upgraded with Ethernet-based remote monitoring.

The Challenge

Things had been smooth and steady on the program.  Development had been completed, and there was nothing left to do but the final round of regression tests. Then it happened. The technical manager walked into the morning meeting looking like she hadn't slept. The night before, she'd had a difficult conversation with the test lead. He had demanded a pay raise, and when things didn’t go his way, he had walked off the job, taking with him the only knowledge of where the regression test suite lived on the network. The suite was gone.

The program was nearly finished.  The new features had been tested and verified over the past few weeks. Only regression testing remained — about two weeks of work — before a contractually committed ship date. The team faced a minimum four-week delay to rebuild a full suite of tests from scratch. That would break the contract. This was the manager’s first major project.  She was very upset and expressed genuine concern about losing her job.

SafeCode's consultant was embedded in the program in a QA and verification role. He quickly suggested a potential solution that was met with bewilderment and skepticism.

The Approach

He proposed a safety analysis — a structured impact analysis drawn from his aerospace experience. Since the device had previously been certified and the new features had been independently verified as correct, it was possible to formally demonstrate that the new code could not have affected the behavior of any previously certified subsystem. If the argument was rigorous and well-documented, it would satisfy the evidentiary requirement that regression testing was designed to meet.

The program manager looked at him as if he were speaking another language. "We're screwed anyway," she said. "Do whatever you want."

He got to work. The code was straightforward. After all, these were little more than glorified ovens with software timers. He traced every point where the new code touched the old, analyzed the interactions, and by late that day had completed the first analysis. By mid-morning the next day, he had a three-page document making a clear, formal argument for why none of the changes could have affected previously certified functionality. He gave it to the technical manager. She skimmed it, gave him an intrigued look, and said she'd have it checked. Half an hour later, she was back — excited.  “They liked it!”  The regulatory office was satisfied with the report. It was acceptable evidence. She asked the consultant if he would do the same for the other sterilizer model?

He smiled, nodded, and got to work. Due to similarities in the legacy code architectures and near identical implementations of the new functionality, the second analysis took only about ninety minutes, and he had the report for her that same afternoon.

The Outcome

Both analyses were accepted as evidence that original functionality had not been adversely affected by the upgrade. The program shipped on schedule, finishing two weeks ahead of where it would have landed without the near-disaster. No test suite was reconstructed. The assumption that full regression was mandatory had been driven by unfamiliarity with other verification methods permitted by FDA regulations. That distinction, it turned out, was worth six weeks of program schedule.  Broad cross-domain experience gave SafeCode’s consultant access to the skills he needed to hand the program a win, when it looked like a loss was certain.

Corollary Lesson

Use good configuration management practices, even for test code, and always make sure there are always at least two people, preferably more, who know where everything is.