NASA uses multiple testing levels, independent validation, standards, safety communities, and tools to ensure safety. Darrel Raines gave a talk about software development and testing for the Space Shuttle and the Orion MPCV at NDC Tech Town. He explained how they learn from failures and near misses and continually improve their process.
Software development is much different today than it was at the beginning of the Space Shuttle era because of the tools that we have at our disposal, Darrel Raines explained in Learning from Embedded Software Development for the Space Shuttle and the Orion MPCV.
Various levels of testing help ensure safe software development, Raines said. Programs require between 4 and 7 levels of testing before the software is deemed safe for flight, and each one depends on a different person or group to perform. Programs have an independent verification and validation group, which performs whatever testing they deem necessary to ensure software safety. This diversity of people working on the tests helps bring in new sets of eyes and different perspectives than the original developer, Raines mentioned.
Raines mentioned that there is a safety community at NASA, and with their contractors. There is also a NASA Software Quality Assurance (SQA) team, and usually a partner contractor SQA team:
These teams are asked to watch over the software development process and point out actual and potential defects in the software. They also perform the task of ensuring overall software quality. They pay attention to safety critical functionality of the software.
With all of these checks in place, we can begin to feel that we have sufficient checks and balances in place to produce safe software, Raines said. But even so, we continually re-evaluate our safety stance and look to make improvements, he added.
Software development team at NASA choose tools that make testing easier without reducing the veracity of the tests, Raines explained:
We use everything from in-circuit emulators to stand-alone test environments. The goal is to use a diversity of tools to find all of the potential errors possible by letting the strengths of each tool come to bear on finding defects.
There are NASA standards for software development, and contracts have additional requirements for a particular vehicle, Raines said. NPR 7150.2 requires unit testing to measure multiple condition/decision coverage (MC/DC) for all tests. This helps ensure the quality of the software as long as the unit tests are focused on demonstrating all of the functionality, Raines said.
Raines mentioned that they cannot find all the defects in their code, but they have to try. “The more we look, the more we will find,” he argued. Therefore, they have to allow enough time during development for adequate testing.
Engineering is not as much of a science as it needs to be, Raines said. Software development is a relatively new discipline, he explained:
When I was in college, there was not even a degree for software engineering. Therefore, we have to keep questioning what we “know” and try to improve our knowledge of this field.
InfoQ interviewed Darrel Raines about measuring software testing and learning.
InfoQ: How do you measure the effectiveness of software testing?
Darrel Raines: One study based on 30 years of Space Shuttle software development showed that each level of testing finds the same percentage of the remaining defects as each other level. Therefore, we can extrapolate that the more testing we do, the more defects we will find. This has diminishing returns, but is an effective method of reducing the number of software errors.
InfoQ: How do you analyze disasters and near misses?
Raines: We do a very in-depth analysis of each case. When we lose a spacecraft, like we did with Challenger and Columbia, we will shut down a program until the root causes of those problems are explained and fixed. This is very time-consuming. But we want to get it right.
There is a heavy toll on the psyche of team members when we look at an accident like Challenger. We will continue to ask ourselves, “What could I have done personally to prevent that from occurring?” When we finally recover our ability to move on, we take with us the reminder that mistakes we make in software development may cause someone to lose their loved-one. It takes a special type of person to live with that possibility and still make progress in their work.
InfoQ: How do you learn from each case?
Raines: We record findings from disaster investigations. I train yearly on past failures and how to avoid them going forward. It is part of our safety heritage that we realize that we have to perform our mission, even though it is very difficult. This training and the NASA focus on safety give me the right tools to do my job well. I just have to perform.
I must continue to learn. I challenge myself each year to try something different, do something different, improve on what I do, and learn something more about my discipline. And as I learn things, I share my discoveries with those who work around me. That means I have to LISTEN to my colleagues as well. As much as I know now, there is more to learn.
