An Architectural Solution to an Integration Problem

People can become accustomed to inconvenience and expense when it seems inevitable.  A large group of engineers attempting to integrate a complex group of applications in an equally complex device containing a large collection of sensors – why should that be easy? 

The System

SafeCode's consultant was engaged in the development of a Level-A aircraft navigation system at a major aerospace OEM.  This was a high-profile project, as the system was to be featured in a widely publicized new aircraft.

The development team consisted of about 150 systems and software engineers, divided into 9 teams.  When our consultant joined the team, development had been underway for about 2 years, and it still had a long way to go.

The Challenge

It quickly became apparent that every system build was costing the program several days. These builds were done once every 4 – 6 weeks.  Engineers had come to accept it as part of the rhythm of development — build, find the race conditions in the startup sequence, resolve them manually, proceed. It had been happening long enough that nobody questioned it anymore.

He was relatively new to the program, but the consultant recognized the pattern for what it was: not an inevitable cost of complex software, but a problem with an architectural solution. He proposed a straightforward way to address it.

The Approach

What he proposed was a new platform-level library: an event-driven startup sequencer. Instead of relying on timing assumptions that varied with each build, each application module would explicitly declare its startup dependencies at initialization. The sequencer would hold each module in a ready state until every declared dependency had confirmed it was running. No more assumptions. No more races.

The software technical lead liked the idea but wanted to assign the implementation to an engineer with more domain experience. The consultant pushed back, made his case, and was given the green light to build it himself. It took about six weeks. When it was done, he personally trained all of the software teams on how to integrate the new component into their respective subsystems.

The Outcome

The system never saw another build stoppage due to startup race conditions. The sequencer became a permanent element of the platform architecture and was carried forward into subsequent program increments, and was even adapted for use in future products. What the team had accepted as an unavoidable fact of life turned out to be a solvable problem. It just needed someone to stop accepting the premise and ask why it kept happening.  SafeCode’s consultant had the foresight and initiative to propose and develop a robust and reusable solution.