Datelines Legend
A-02/01/94 Target-PFOD C B-04/09/94 Actual-Roll out C-06/12/94 Actual-First Flight D-10/04/94 Target-ETOPS FF
For the Boeing Commercial Airplane Group (BCAG), the 777 airplane represented an unprecedented software challenge in terms of the size and complexity of the airplane's systems. The 2.5 million lines of newly developed software were approximately six times more than any previous Boeing commercial airplane development program. Including commercial-off-the-shelf (COTS) and optional software, the total size is more than 4 million lines of code. This software was implemented in 79 different systems produced by various suppliers throughout the world. Connecting these systems is a complex network of broadcast and pointtopoint data buses. Moreover, we committed to provide unprecedented product maturity and quality at first delivery.
This article discusses how the software challenges were successfully met, which resulted in the on-schedule delivery of the first 777 to United Airlines four and one-half years after the program's kickoff. As with all successful projects, first and most important, the skill, dedication, and efforts of all the people involved were the key to its success. However, many large software projects have skilled and dedicated workers, yet they are plagued with missing functionality and schedule slides. Rather than address all the aspects of a successful project, this article discusses techniques used to address two areas that typically create major challenges for large embedded software development efforts: requirements specification and management of the software development process.
Requirements Specification
The specification of airplane systems requirements to the suppliers of the systems is done through specification control drawings (SCD). A major component of that specification is the interface control document (ICD). To assure that the systems will combine to perform as intended, extensive effort was put into the development, control, and validation of an airplane-level ICD and detailed SCDs for each of the airplane systems.
Specification Control Drawings
SCDs were developed for each of the systems on the 777. They ranged in size from approximately 100 pages for the simplest systems to over 10,000 pages for the most complex. The SCD is the primary means for Boeing system designers to communicate the requirements to the supplier of the system. It formed the basis of an ongoing dialog between system designer and developer and between the designers of the various systems. Changes to the SCD were subject to rigorous change board review and approval. The reviews assured that all aspects of a change were addressed, and the effects of the change were reflected in all the affected systems. These reviews also controlled the amount of change, assuring that only necessary changes were made to the systems. Control of changes was increasingly important as development progressed and the effects of changes could increasingly jeopardize the schedule. This led the BCAG to apply increasingly stringent criteria on acceptability of changes. The importance of controlling change cannot be overemphasized.
By going into great detail in many areas of the specification, the SCD captured not only the understanding of how the individual systems must operate but also insured that the integrated functionality of the system would perform as intended. It also facilitated the system-level analyses that allowed validation of the airplane functionality described later.
ICD Database
As the airplane architecture and functionality of the various systems were evolving, the definitions of the interfaces of each system were placed in a large relational database. The database describes over 40,000 digital data items and 3,000 analog signals. Crucial to the successful use of this database was a tool developed to scan for discrepancies in data definition between the provider and user of the interface data.
As the functionality of the systems continued to evolve, a series of interface block points was established and resolution of discrepancies continued to be a major focus. The block points were established to satisfy the incremental development of overall systems functionality. The block points represented a coherent definition of system interfaces that supported a pre-defined set of systems functionality. The definition of these sets included specific capabilities in terms of airplane functions as well as the level of maturity of the system and its ability to tolerate failure conditions. The definition of the block points allowed integration of the systems to proceed in parallel with continued development of functionality within the systems. This not only achieved an accelerated system integration, but also identified integration problems at a time when they could be addressed as the systems development proceeded.
Reports from this database formed the basis of the interface specifications for the individual system ICDs. Rigorous configuration control was applied to each block point. Change boards were conducted for changes both at the major function level and at the airplane level to assure the changes were necessary and that all consequences of the changes were considered throughout the airplane.
In the first two months, analysis of the database uncovered over 50,000 discrepancies. An intensive, high priority effort reduced the discrepancies to 3,000 within two months. Use of the ICD database and associated tools allowed us to deal with the increased interface complexity of our system, yet have unparalleled success in integrating the systems in the laboratory and on the airplane.
System Functional Analysis
It is important to have well-defined and carefully controlled requirements, but it is equally important to minimize changes to those requirements as the systems are implemented. On a large, complex system such as the 777, it is common to find specification problems late in the program when the systems are brought together in the integration facility or even in flight test. To prevent these problems, a disciplined series of functional analyses was performed throughout the development of the system specifications. In all cases, both nominal and failure conditions were considered. This effort was led by a dedicated system integration organization that coordinated and organized the process. In all cases, there were regular management reviews of compliance and completion of the process.
The first phase of analysis was performed early in the airplane program during the development of the SCDs and ICD. Using regulatory requirements and highlevel functional definitions of the airplane, the functionality of each system was reviewed. The reviews covered both nominal and offnominal operating conditions. Allocation and definition of requirements were performed. The output of this process was reflected in the SCDs and ICD.
The next phase of analysis focused on the interfaces between the systems. This analysis was performed on selected systems, based on complexity, criticality, and customer visibility of the system. For each selected system, the inputs and outputs were reviewed. The goal was to verify that all users of data were using it correctly. This activity was performed after the first phase of analysis, but at a very early stage in the development of the systems by the suppliers.
The last phase of analysis was at the airplane functionality level. These analyses were performed as the development of the systems was well underway, but prior to final system integration. This activity validated correct performance of the airplane systems as a whole. For each significant airplanelevel function that required multiple systems, we verified that the function was properly performed and that failure conditions were properly handled.
These were all manual analyses performed early enough in the program to minimize impact on development of the individual systems and in time to significantly reduce the problems found as systems were delivered and integrated. The result of the three phases of analysis was that problems in the specification of the airplane systems were identified and corrected at the earliest possible point, thus mitigating the impact of corrections on the overall program schedule.
Managing the Software Development Process
Even with the best specifications, completion of large, complex embedded software systems on schedule has proved elusive. Systems are frequently declared as 90 percent complete only to find they are less than half done. Major delays in product deliveries are often only "discovered" when it is too late to recover from the situation.
With an aggressive flighttest program, the most ambitious in Boeing commercial airplane history, and 79 systems to integrate, it was essential that we had visibility on the real status of the software development on all our systems. With requirements complete for nearly all our systems and software development well underway, an airplanewide software metrics program was instituted. The metrics included development plans and progress and utilization of computational resources.
Development Metrics
The development metrics tracked progress against the plans for design, code, test procedure creation, and test completion. They included the predicted total software size (in lines of source code) and total number of tests. The metrics charts showed key milestones in the airplane program. These milestones represented interim points to measure our progress against the ultimate completion of the program. Associated with these milestones were success criteria based on completion of design, code, and test of the software products.
Figure 1 shows the total airplane roll-up of these metrics. The metrics also measured utilization of computational resources (throughput and memory); however, this discussion will focus on the design, code, and test metrics.
Figure 1: Total Metrics Roll Up for the 777.
Datelines Legend
A-02/01/94 Target-PFOD C B-04/09/94 Actual-Roll out
C-06/12/94 Actual-First Flight D-10/04/94
Target-ETOPS FF
The process of using these metrics was as follows:
· Each supplier was requested to prepare plans for their design, code, and test activities. These plans showed expected totals and the planned completion status for each of the biweekly reporting periods until the task is complete. Even at this early stage in the metrics process, we received our first benefits as we discovered that some suppliers' initial plans did not support the program milestones. This proved invaluable and in a few cases was the only major corrective action we needed to take to assure the supplier supported the program.
· Following the initial plan submittal, there were biweekly updates that showed the actual status of the development in terms of completed design, code, and tests. Any changes in the estimated total size of the effort were also provided and the plans modified to correctly reflect the new total. Plans could also be changed at any time; however, previously reported plans and actual status could not be adjusted. The metrics information was shared with the developers of the systems. This led to important discussions on how we were going to succeed at an early enough point in the program that we could actually do something about it.
· Indications of a healthy program are fairly obviousa plan that supports program milestones and status that consistently tracks the plan. Programs that needed special attention often were several weeks behind the plan line, had numerous replans, or had plans that required unprecedented productivity to be successful. We quickly established reasonable productivity figures that could easily test the feasibility of suppliers' plans based simply on head count, work to go, and time to go. The metrics provided an excellent vehicle for discussions about how the program was going to deal with their current status and get back on a schedule that supported the overall airplane program.
The above process was as important as the measures themselves in assuring our success. However, there were certain characteristics of the metrics program that were key to supporting this process and making it all work. They were as follows:
· Uniformity.
· Frequent updates.
· Clear definition.
· Objective measures.
· Replans, as needed, are allowed and even encouraged.
· Past plans and actuals are held constant.
The uniform nature of the metrics enabled comparison across systems and supported communication of objective status to all levels of program management. This was particularly important with the large number of organizations involved in the software development. We were also able to combine status information from several different systems provided by a single supplier. This provided unique opportunities to discuss how the supplier was supporting our overall program and to focus needed resources to solve schedule problems.
Considerable effort was made to clearly define measures. This led to a 21-page set of instructions to our suppliers on how to prepare metrics data. The data items measured were objective and easily observed. The combination of these meant there was little confusion about what the metrics meant and real conclusions could be drawn from the data. Moreover, without this we could not have achieved the desired uniformity.
Two aspects of the metrics plans were critical: replanning when needed was encouraged, and past data was never changed. The essence of a plan is it shows how to get from here to there. Once you have significantly deviated from a plan it no longer serves that purpose. Throughout the metrics process we used deviation from the plan as an indicator of problems. Since replanning was encouraged, the only reason to not be close to your plan is you don't have a plan. We found that projects that were several weeks behind their plan did indeed need help.
This approach to software project metrics repeatedly "saved our bacon." Starting with initial plans, they indicated where program milestones were not being supported. Continuous monitoring through testing identified schedule problems early in the development process. The metrics were invaluable in showing us where program risk points were soon enough to take corrective action.
What About Ada?
Prior to the start of the 777 program, BCAG determined that Ada would be the standard programming language for airborne systems. This decision was based on the intrinsic value of having a standard language and the fact that Ada was designed to facilitate sound software engineering. It was also hoped that we could leverage off the considerable efforts spent on Ada in the development of defense systems.
What we have learned so far about the use of Ada on the 777 is a mixed message. Ada was used on 60 percent of the systems and represented 70 percent of the lines of code developed on the 777. We found no correlation between the language used and the number of problems found on the system. We found instances where Ada was used effectively, and the developers felt it substantially reduced software integration problems. In other cases, development was hampered by problems with compilers and other support tools. Many suppliers chose to use a restricted subset of Ada, which led to fewer problems but lesser benefit.
In situations where the imposition of Ada would have created more risk than benefit, other languages such as C or Assembly were allowed. By taking a pragmatic position on this standard, we removed risk and helped the program succeed. It is likely that much of the benefit of Ada will be seen later in maintenance of the software, but that will be difficult to quantify. The richness and complexity of the language helped knowledgeable users with mature tools achieve modest productivity gains. However, the complexity of the language caused headaches for other users who had to work through compiler problems.
We continue to evaluate our development standards. I expect we will retain Ada as the standard language. A standard language allows the use of tools to aid in the development of software that would be difficult or impossible to implement in a multilanguage environment. As the systems get larger and more complex and the developers and tools they use mature, the value of Ada should increase.
Conclusions
Many aspects of the 777 program were done right to support successful completion of the software systems. Rigorous requirements development process and vigilant management of software development were two keys to that success. This required specific processes and tools as well as the commitment of the program to follow through on their execution. The results were requirements that were stable and complete at an earlier stage in the program and supplier software development processes that had sufficient oversight to allow corrective action and on time completion of the product.
Ronald J. Pehrson
Manager, Embedded Software Engineering
Boeing Commercial Airplane Group
Everett Division _ Mail Stop 03-JC
P.O. Box 3707
Seattle, WA 98126-2207
Voice: 206-294-1115
Fax: 206-294-6504
E-mail:
rjp8002@kgv1.profs.ca.boeing.com
About the Author
Ron Pehrson has over 20 years experience in software engineering with the Boeing Company. He has worked as a design engineer on such projects as B-1 and B-2 avionics, B-52 offensive avionics system, and B-1B avionics simulation. He was a software design supervisor on the B-1B weapon system trainer, advanced tactical fighter demonstration/validation, Boeing Commercial Airplane Group central software engineering, and the 777 propulsion and avionics software. He is the manager of embedded software engineering for the Boeing Commercial Airplane Group. Pehrson has both a Bachelor of Science in mathematics and a Master of Science in computer science from the University of Washington.