Ascension

Ascension

Life Lessons from the Space Shuttle Columbia

Tragedy for Engineers, Managers, and Leaders

by steven r. hirshorn

National Aeronautics and Space Administration

Office of Communications

NASA History Office

Washington, DC 20546

NASA SP-2025-4115

ISBN: 978-1-62683-087-5 (e-book)

ISBN: 978-1-62683-086-8 (paperback)

Library of Congress Control Number: 2025938198

This publication is available as a free download at

http://www.nasa.gov/ebooks.

Table of Contents

Ascension

Dedication

Foreword

Prologue

A Bad Day

Chapter 1

Chapter 2

Chapter 3

Chapter 4

Mishap Recovery

Chapter 5

Chapter 6

Chapter 7

Chapter 8

Investigation

Chapter 9

Chapter 10

Chapter 11

Chapter 12

Return to Flight

Chapter 13

Chapter 14

Chapter 15

Chapter 16

Chapter 17

Epilogue

Acknowledgments

About the Author

Acronym List

The NASA History Series

Back Cover

Dedication

It is commonly difficult for people in their early twenties to fully comprehend the impact of others in their lives. For me, at that age, life was full of potential and there were no limits on what I might experience, and similarly high limits on my desire to experience them. Life is rich and the world is my oyster, or so I thought. People are interesting, and occasionally helpful, but I’ve got this on my own.

Today, I realize I was wrong. I’ve taken a fascinating and enriching path through my career at NASA, from Mission Control to human presence on the Moon, from the agency’s science missions exploring the solar system to our aeronautics research endeavors, and from operations to development and engineering. Throughout that path I have been graced by the wisdom of various mentors and teachers, each filling in gaps in my own judgement and helping to guide me on this journey. One of those first mentors was my graduate advisor at the University of Texas at Austin, Dr. Wallace T. Fowler.

Today Dr. Fowler is the Paul D. and Betty Robertson Meek Centennial Professor Emeritus in Engineering at UT–Austin, but during my tenure as a graduate student in 1988–1989, Dr. Fowler was both less and more—less in that he was “only” a graduate professor in aerospace engineering, and more in that he was my advisor, my professor, my font of wisdom, and in ways I like to think about it now, my spirit guide.

I always greatly admired Dr. Fowler’s commitment to his students. His enthusiasm for aerospace engineering, spaceflight, and mission design was, in my mind, exceeded only by his dedication to the education of those attending the university. Dr. Fowler’s brilliance in design and engineering was matched by the joie de vivre he brought to the task of advising students. He was advisor, but more so he was colleague and even friend to me and to countless other classmates.

It wasn’t until many years later, now solidly ensconced into my NASA career, when I began to reflect on the influence he made on my life. Not just on the technical merits of being an engineer, but on those qualities of the job that transcend the technical. Dr. Fowler taught me the values of integrity and the importance of perseverance. From his influence did I learn a model of competence that envisioned for me what an engineer should be. And beyond my job, from him I learned important life lessons on friendship, camaraderie and fellowship. Collectively, these things have guided me throughout my career and continue to guide me today.

As I write these pages today, Dr. Fowler insists that I call him Wally. Somehow, that seems both befitting of our friendship and also grossly inadequate to the high esteem in which I hold him. But I also feel honored to be able to parlay in that familiarity, as if it is a sacred trust.

For the influence he has had on me, and on generations of engineering students who preceded and followed me at UT–Austin’s Department of Aerospace Engineering, I humbly dedicate this book to Dr. Wallace T. Fowler.

Foreword

I’m very glad Steven Hirshorn has written this book. As a space historian who teaches NASA engineers a counterintuitive, often neglected aspect of their agency’s history—the profound, make-or-break impact of human behavior on success and failure—I can tell you that one of the hardest parts of spaceflight programs, even tougher than much of the “rocket science,” is learning and remembering the lessons of the past. History vividly shows how shockingly easy it is to lose the awareness so painfully acquired in the wake of the last accident and go back to the same patterns of thinking that led very smart, well-meaning people away from success and down the slippery slope to catastrophic failure. Space Shuttle Columbia’s disintegration during reentry on February 1, 2003, happened almost exactly 17 years after Challenger exploded during launch, but in reality, NASA had begun to lose the lessons of Challenger long before then. Columbia is a sobering reminder that awareness has a shelf life.

What Hirshorn has given us with this book is something precious: A way to learn and remember some very important lessons. Ascension is a wonderfully detailed, highly personal, and remarkably candid account of his experiences before, during, and after the ill-fated STS-107 mission. At the time of the accident, he was technical assistant (TA) for Shuttle in the Systems Division of the Johnson Space Center’s Mission Operations Directorate, a role that built on his many years as a Shuttle flight controller and made him a direct participant in the events as they unfolded. Through his eyes, we see how shock and a terrible feeling of failure gave way to determination to find the causes of the accident and the two-year struggle to return the Shuttle to flight. He writes of the grueling conditions endured by hundreds of people from NASA and other agencies, aided by local residents, in searching for debris scattered over eastern Texas and western Louisiana. We witness the painstaking detective work of sifting through mountains of data for clues to what had happened to Columbia, and the dawning realization that a problem that had plagued the program for two decades—shedding of foam by the Shuttle’s external tank (ET), which had come to be regarded not as a safety threat but a “maintenance issue”—had finally caught up with them.

Only in hindsight did it become clear how many times warning signs had been missed or misinterpreted. On several flights foam strikes had caused significant damage to the Shuttle, but each time the astronauts had returned safely to Earth, reinforcing engineers’ perceptions that the problem was one they could live with, undesirable but acceptable. And one key misperception persisted even after the accident: Many of Hirshorn’s colleagues continued to believe that the reinforced carbon-carbon material used as thermal protection for the leading edges of Columbia’s wings would not have been seriously damaged when the Orbiter collided with a briefcase-size block of foam 82 seconds after liftoff. Only after they viewed the results of impact tests carried out at the actual 500-mile-per-hour speed of the collision did they know their assumption had been wrong—a jolting realization that Hirshorn captures in his narrative.

Like a trusted mentor, Hirshorn shares the insights he gained from these experiences as “life lessons for engineers, managers, and leaders” including one that goes right to the heart of the accident’s root causes: “When the hardware talks, listen to it.”

Ascension rings with Hirshorn’s conviction that out of failure and tragedy should come wisdom and his faith in the power of storytelling to impart awareness and to change for the better how we see our work and ourselves. Thanks to him, I have a new appreciation for what he and his colleagues went through during and after the accident, their inspiring passion and persistence, and their struggles, which did not always succeed, to make the Shuttle a better and more resilient space vehicle. And I am moved by the burden of regret that they still carry, even to this day. Hirshorn’s powers of observation, unblinking self-reflection, and willingness to tell hard truths come together on these pages to create a book whose value is not only for engineers who confront the daunting challenges of human spaceflight, but anyone engaged in difficult group endeavors.

Andrew Chaikin

Author of A Man on the Moon: The Voyages of the Apollo Astronauts

Prologue

After nourishment, shelter and companionship, stories are the thing we need most in the world.

Philip Pullman

A story has no beginning or end: arbitrarily one chooses that moment of experience from which to look back or from which to look ahead.

Graham Greene

Stories. Over the long arc of our species’ existence, stories have been used to convey information, to capture culture and history, and to portray the vibrancy and effervescence of human life. Stories recount the great events of civilization, the development of societies and empires and the destruction of war and conflict. Stories bring to life accounts of exploration, both of lands unseen and of experiences never imagined.

To many, beyond the simple enjoyment they may instill, stories are important to the human condition as mechanisms of learning. Before the advent of written communications (the invention of the alphabet, calligraphy, and other scripts) and the inking of these characters onto parchments and scrolls, elder generations used the format of storytelling to pass on experience, and wisdom, to their small-knit communities. These stories took the form of parables (which taught morals and ethics), descriptions of long journeys, quests, and great challenges (which taught resilience and perseverance), and used to convey knowledge (recipes, procedures, knowledge). Oral history was a way of transferring hard-won knowledge to future generations, who would pass it on to their own children, and the cycle would continue throughout the ages.

Having spent the majority of my National Aeronautics and Space Administration (NASA) career as an engineer, I know from experience that engineers are trained to relate information in terms of the tangible, the quantifiable, and the predictable under reproducible conditions. We engineers value hard factual data that allows little-to-no room for interpretation. We look for reams of numbers and volumes of analysis runs—millions of them sometimes—that can help us reduce uncertainty to the absolute limits of our tools and technologies. Or, more precisely, to levels that allow us to make decisions within acceptable risk. Tests, telemetry, analysis, computational modeling, active demonstrations, and other tangible methodologies are the ways in which we bring data to the table. Often that data is raw, unrefined, and voluminous, but that’s what we like and value, and that’s what we look for when decisions are made.

Stories, however, are another thing entirely. Stories do not need to be quantifiable. In fact, they can consciously be chosen to be the opposite of quantifiable because, unlike data, stories strive to evoke emotions—the thoughts and feelings of the associated protagonists and antagonists. While conveying these emotions, stories also stimulate emotions and feelings in the reader, transporting them to times and places far removed from their present circumstances and exposing the reader to experiences well beyond their own, even to realms they may not have ever imagined. And in those cases when stories are not so grandiose and stay closer to the mundanity of life, they still can resonate in ways that analyses and a million Monte Carlo runs1 cannot. Data may be undeniable, but stories reach us in ways that hard facts alone cannot.

And yet, engineers may be hesitant to utilize stories in the conduct of their work. Is this a bias? Maybe, but some of that hesitation may also be logical. While stories can convey aspects of the human condition that data and analyses cannot, they are many times, at best, apocryphal. They might get altered over time and through repeated telling’s, shifting in details and specifics. And opinion, perspective, and perception (or misperception) can be infused into them at the discretion (or through some unconscious bias) of the storyteller. Because of these, stories may not be reliable and certainly can’t be considered certifiable. Stories are, by definition, malleable things, often interpreted differently by those who hear them. An engineer could justifiably feel uncomfortable certifying a piece of hardware when the only evidence of the component’s performance is a story. That’s a reasonable argument; you cannot assess whether a component’s reliability is within three-sigma dispersions2 through a story; you need data, testing and analysis.

But the point here is that stories can play an important role in how we conduct our jobs. We may not be able to include stories in our certification packages or verification reports, but we can use them to help us learn and become better engineers. They are tools with which to better understand how hardware operates and why they sometimes fail, how others have approached problems and the solutions they derived, and how humans perceive reality and can get blinded to what objectively should be obvious. As with the ancient storytellers, stories can effectively pass on both knowledge and experience. Engineers can learn from stories and through them become better in the art and science of our business. Stories can be invaluable in creating and maintaining healthy organizations, promoting, and encouraging safe environments and helping us avoid the past mistakes and oversights that have led to tragedy. In this way, stories are sometimes even more important than data to the success of our NASA mission.

This book, then, is filled with stories.

And not just any stories, but a very personal accounting of the STS-107 Columbia Space Shuttle accident, investigation, and recovery. Many books have been published on the subject of Columbia that admirably recount the facts of the accident and the root causes—both technical and organizational—that led to the terrible events of February 1, 2003. That was a day when our nation lost not only its very first and oldest Space Shuttle Orbiter but also, more importantly, the lives of seven brave explorers. The purpose of this book is not to duplicate those historical expositions, nor to deeply dissect the organizational shortcomings that contributed to that very bad day. The comprehensive coverage of the Columbia Accident Investigation Board (CAIB) report3 and the NASA response as documented in the Diaz Report4 (both freely available on the internet) remain the official evaluations of those matters. Instead, this book is intended to offer something quite different.

A combination of both my NASA job and a bit of serendipity placed me at ground zero to these events. I was able to witness them from the inside as a participant rather than as an outside observer. I was one of those within NASA who experienced the tragedy as someone who grieved for the losses we incurred and also as someone who played a role in understanding what happened and why. And through these experiences, I was one of those who ended this journey striving to ensure that events such as these would never be repeated.

At the time of the accident, I was part of the relatively small and very connected family of Shuttle Mission Control flight controllers who were charged with the safety and success of NASA’s human spaceflight endeavors. Afterwards, I was part of the investigation team whose task was to wade through the last bits of telemetry transmitted by Columbia and to scour through the thousands of pieces of debris recovered from searches in the thicket-forests of eastern Texas, all in search for answers. I was a member of the two-year-long effort to get the Shuttle fleet flying again and was back in Mission Control when we finally launched STS-114 on Space Shuttle Discovery on July 26, 2005, returning NASA’s astronauts once again to low-Earth orbit.

Through these events, I experienced NASA at both one of its lowest points and one of its highest, gaining an understanding of how the agency can simultaneously exceed and fall short of its expectations. And what’s more, working at ground zero and living through these experiences and the emotions they invoked, I discovered something that I hope might be considered wisdom, a gift which I carry with me every day whether in the course of my job or in life in general. There are lessons from STS-107 and the events which followed which are directly applicable to the broader quest of living our lives. Communicating the ties between those two is the intended purpose of this book, more than the simple recitation of historical facts. These life lessons for engineers, managers, and leaders (or anyone) from the Space Shuttle Columbia tragedy and the stories from which they derive are my way of passing this wisdom on to future generations.

Context and a Few Terminologies Used Within this Book

During the events described in this book—from February 1, 2003, through July 26, 2005—I held the position of TA for Shuttle in the Mission Operations Directorate’s (MOD) Systems Division at NASA’s Lyndon B. Johnson Space Center (JSC), outside Houston, Texas. You might be asking what all that means? Thus follows a short organizational primer:

During my tenure, the Mission Operations Directorate (known as MOD and pronounced Emm–Ohh–Dee) was the second largest organization at NASA’s JSC (second only to the Engineering Directorate). Located halfway between Houston and Galveston, JSC has been the center of gravity of NASA’s human spaceflight operations and is the home to the eponymous Mission Control. Within JSC, MOD can trace its history all the way back to the creation of the center (then known as the Manned Spacecraft Center) in 1962. The forebearers of MOD include such historical luminaries as Christopher Kraft, Eugene Kranz, Glynn Lunney, some of the original flight directors of NASA’s earliest spaceflight programs such as Mercury, Gemini, and Apollo (in fact, when I first hired into NASA and MOD, Gene Kranz was still working there as the director of MOD with Glynn Lunney leading the organization’s contractor team). The primary function of MOD then was to plan human space missions and develop their timelines, procedures, and operations; train the crews of astronauts assigned to each mission and the cadre of flight controllers who would oversee, direct, and assist from Mission Control; and fly the missions from launch to wheel stop (in the case of the Space Shuttle) or splashdown (in the case of capsules parachuting into the ocean).5

Organizationally, when I hired, MOD consisted of a number of divisions, each populated by the people who planned, trained, or flew Shuttle missions. The trainers were grouped into their own division (uncreatively called the Training Division), but the other divisions were differentiated more by flight controller function. The Flight Dynamics Division contained groups who were experts in trajectory, orbital mechanics, and the precision of motion required to launch rockets and return spacecraft back to Earth. The Operations Division was populated by those dedicated to the conduct of payloads—deployable (and sometimes retrievable) spacecraft and experiment packages, laboratory modules, and to the science conducted within, all a plethora of activities that justified each Shuttle mission. There was also a division dedicated to the Mission Control Center (MCC) itself—the building, consoles, mainframe computers and distributed communication networks that allowed those of us on the ground to communicate with astronauts orbiting Earth, and them with us.

And then there was the Systems Division—my division. The Systems Division consisted of the teams of flight controllers who focused on the Space Shuttle’s hardware and software (or systems) accounting for over 50 percent of the people one would typically see on console in the flight control room. Our division of flight controllers oversaw the Shuttle Orbiter’s computer systems (console call-sign: DPS); communications systems (INCO); mechanical systems (MMACS); propulsion (PROP), guidance, navigation & control (GNC); electrical systems (EGIL) and environmental control and life support systems (EECOM). Collectively, those of us in the systems division were experts in every component in these spacecrafts, familiar with each moving part, each wire and electrical harness, every component and mechanism, in addition to all the checklist procedures, flight rules and operations that governed human spaceflight.

These were large responsibilities, particularly for a team of people mostly in their twenties and thirties, recent college graduates like me and employees still considered in the early- and mid-stages of their career. We trained rigorously, following regimens that enabled an almost intuitive knowledge of operations. During critical timeframes, such as launch and reentry, much of that dedicated training forged within us the ability to recommend crew actions without taking the time to think through the problem—time is a luxury dynamic flight phases didn’t allow. Instead, that rigorous training allowed flight controllers to respond almost instinctually to situations.

Figure 1. An early rendition of the Mission Operations Directorate (MOD) patch, designed by space artist Robert McCall. (NASA Image S88-29089)

Figure 2. The MOD patch during the author’s tenure, after the ISS was added. (NASA Image JSC2005-E-23169)

We were a resolute group of people, resilient to pressure and overwhelmingly dedicated to the pursuit of human exploration of space. That dedication formed the basis of a culture within the flight controller community. Being a flight controller wasn’t simply a job or an occupation; rather, it was a complete immersion into a unique community and closely knit family, one with its own language and phrases, its own history, and its own philosophies and mindset. One wasn’t just hired into the flight control ranks; it was more an immersion into a vibrant culture and community with high demands and rich rewards.

For my part, I worked at the EGIL console, responsible for the Orbiter’s Direct Current (DC) and Alternating Current (AC) electrical distribution systems, the fuel cells that generated power, and the onboard cryogenics which fed the fuel cells. Through the period of my active flight control (1990–2001) I supported 55 Shuttle missions, including 10 launches and reentries, on some of NASA’s most challenging and memorable Shuttle flights. These included the deployment of the Hubble Space Telescope, the Ulysses solar probe, and the Gamma Ray Observatory; countless missions dedicated to scientific exploration of Earth and the Sun and to understanding the biological effects of microgravity; operations with the Russian MIR space station; and, eventually, the early stages of construction of the International Space Station. Of those 55 missions, 13 were flown on Columbia.

In 2001, after working my last flight as an active flight controller on STS-97, the fourth ISS construction mission, I turned my attention to technical management and was selected as the TA (technical assistant) for the Systems Division, a promotion for me. The TA position had historically been responsible for integration duties across the division, such as generation of the Certificate of Flight Readiness (CoFR) package, created for every Shuttle flight and stipulating the division’s readiness to support the upcoming Shuttle mission. However, another major duty, one that exposed me to the vast intricate network of the Space Shuttle Program (SSP) and especially the myriad activities required for keeping the Shuttle fleet flying, was as the MOD rep to the Orbiter Project Office (OPO). The OPO was the office responsible for maintaining each Space Shuttle Orbiter (Columbia, Challenger, Discovery, Atlantis, and eventually Endeavour), consuming a budget of a billion dollars a year in that pursuit. In the Systems Division TA role, I was the part of the OPO team (albeit a matrixed member) and straddled the fence between operations and project management. The OPO “owned” the Shuttle hardware: its components, computers, pumps, power units, wiring, plumbing, reaction jets, windows, thermal tiles, even the ship’s aluminum structure, everything that when integrated makes up a Space Shuttle Orbiter. As the OPO owned the hardware, MOD operated it.

At the time Columbia launched on the STS-107 mission, those were my responsibilities.

Connection to Three Sigma Leadership (Or, The Way of the Chief Engineer)

This book is intended to be a companion, in ways, to a previous book I wrote, published by NASA in 2019, titled Three Sigma Leadership (Or, the Way of the Chief Engineer)—Leadership Skills for NASA’s Corps of Chief Engineers.6 Within that first book I tried to instill a set of guidance to both those who are and those who may aspire to be a chief engineer. In 24 chapters I covered an amalgam of technical topics such as serving as a technical authority, acting as a lead technical integrator, becoming a master of risk, and others. But contrary to other tomes on the technical requirements for such a position, the majority of that book focuses on the more “soft skills” that are critical to the success of a chief engineer but less routinely taught. For example, the book covers topics such as showing accountability, building a team, demonstrating emotional intelligence (the book’s first chapter), negotiating solutions, and more. Collectively, these are skills that a chief engineer will need to master.

Figure 3. The author working a shift at the EGIL console. Image taken some time in the late 1990s. (Photo provided by the author.)

In Ascension, I am taking a more nuanced approach to guidance for those who parlay in the engineering field. The stories within the book you are now reading all pertain to my experiences with the Columbia accident, recovery, investigation, and return of the remaining Space Shuttles to flight, but each story is augmented with a theme of that chapter characterized as a life lesson for engineers, managers, and leaders. Some are direct and obvious; others may be more subtle and not immediately apparent. But all, I believe, are applicable to our role as engineers, managers, and leaders.

1 The Monte Carlo method is a type of computational algorithm that uses repeated, random sampling to obtain numerical results for each “run.” Uncertain variables are given a range of possible values and the results of each run taken together provide insight into the likelihood of potential outcomes.

2 A statistical calculation that equates to being successful about 99.7% of the time, which in engineering terms is normally considered “good enough.”

3 Harold W. Gehman, Jr. et al., Columbia Accident Investigation Board Report Volume 1. (Washington, DC: Columbia Accident Investigation Board, August 2003). https://sma.nasa.gov/SignificantIncidents/assets/columbia-accident-investigation-board-report-volume-1.pdf (accessed Jan. 21, 2025).

4 Al Diaz and team. A Renewed Commitment to Excellence. (Washington, DC: NASA, Jan. 30, 2004). https://www.nasa.gov/wp-content/uploads/2015/01/55691main_Diaz_020204.pdf (accessed Jan. 21, 2025)

5 Today MOD is referred to as the Flight Operations Directorate (FOD), but when I hired on in 1990 it was MOD and when I transitioned to NASA HQ in 2011 it remained MOD, so I will utilize the term MOD throughout this book.

6 Steven R. Hirshorn, Three Sigma Leadership, Or, the Way of the Chief Engineer, (Washington, DC: NASA SP-2019-220374, 2019). https://www.nasa.gov/ebooks/three-sigma-leadership (accessed Jan. 22, 2025).

A Bad Day

Chapter 1

A Call from the Boss (Part 1)

Don’t Discard the Obvious; Sometimes it is the Right Answer

February 1, 2003, was a Saturday. It dawned clear and brightly sunny in League City, Texas, southeast of Houston and close to Johnson Space Center, where I lived. It was cool for Houston that morning, temperatures hovering in the low-50s, as the sun began its morning ritual of evaporating the dew from the grass and shrubs that fell overnight from the coastal moisture-laden atmosphere. I had just awoken, irritated that the bright white of the bedroom’s walls so effectively reflected the incoming sunlight. It felt like the plaster and paint were conspiring to focus the sun’s beams directly onto my sensate form. I pulled the sheets further up to cover my head and try to block out the unwelcome radiance. If I had bothered to look at the clock it would have read a bit after 9 a.m.—not particularly early by most opinions, but I wasn’t yet ready to get out of bed and start my day.

And then the phone rang.

Each day for the previous two weeks I was going into the MCC and supporting the Orbiter Project Office’s daily meetings in preparation for that day’s Mission Management Team (MMT). STS-107 was on orbit, the crew conducting science experiments in round-the-clock operations out of the SpaceHab double-module mounted in Columbia’s payload bay. While the flight control team dutifully watched over Columbia and her crew, the MMT would gather each afternoon to wade into mission strategic decision-making. The MMT consisted of senior Space Shuttle Program, NASA center (JSC, the Kennedy Space Center [KSC], and the Marshall Space Flight Center [MSFC]), and agency leadership responsible for maintaining management oversight of Shuttle flights both in the days before launch and while the crew was on orbit executing their mission. The flight control team directed operations on a day-to-day, hour-to-hour and even minute-to-minute basis, but the MMT reserved its judgement (and authority) for the more strategic issues that any mission might need to contend with, such as: whether to extend the mission’s duration or even reduce it; whether to proceed with critical mission objectives like rendezvousing with an orbiting satellite or conducting an impromptu spacewalk; or whether to attempt a reentry or bypass a landing opportunity because of poor predicted weather. The flight control team occupies their consoles 24/7, but the MMT only meets once per day, usually for a few hours in the early afternoon, and then breaks until the following day’s meeting.

Since the MMT made these critical decisions, a significant amount of preparation was required prior to the start of the meeting. The OPO, responsible for the vehicle that was orbiting Earth and providing a safe and habitable environment to the crew of astronauts, obviously played a large role in the MMT. Some hours before the MMT, the OPO would gather in a conference room in the control center’s Mission Evaluation Room (or MER) to review how the spacecraft’s hardware was performing, assess any component failures, and collect any recommendations to be brought to that day’s MMT.

The MER itself can trace its lineage back to the very spark of NASA’s human spaceflight programs. It was the room where “the engineers hung out”—that is, the subsystem managers and supporting staff who oversee and manage the various pieces and parts that collectively make up the Space Shuttle Orbiter. If a component (say, a computer, a valve, a pump or mechanism) failed during a mission, it was the subsystem manager who was responsible for understanding the failure, processing the component’s removal from the Orbiter after the mission, testing it if necessary to better understand the cause of the failure, overseeing its repair and managing that part’s return to inventory. These technical experts provided a critical “behind the scenes” role in the success of a Shuttle mission and played large roles in the continued performance of the Orbiter fleet over the lifetime of the program.

During Apollo in the 1960s and much of early Shuttle Program through the 1980s, the MER consisted of rows of government-issued tables, closely bunched together like a European cafe. Engineers would gather at these tables, sometimes shoulder to shoulder, to review and discuss the performance of their hardware. The room could often be filled with cigarette smoke, casting a blue haze within its cramped confines. In those days there were no computer monitors in the MER and perhaps one or two televisions, but there were copious quantities of paper—rolls of strip chart recorder paper, hardcopies and printouts of telemetry data, volumes of hardware certification data, stacks of test reports, and other supporting information.

Figure 4. Two views inside Mission Control’s original Mission Evaluation Room (MER), taken during the flight of Apollo 11. (NASA Images S69-52725 and S69-52729)

That all changed in 1996 when NASA christened the “new” control center built adjacent to the one used for Projects Gemini, Apollo, and early Shuttle flights.1 You may recall the flight control room in the “old” Mission Control: four rows of olive-shaded consoles in a tiered stadium configuration—the rearmost consoles elevated higher than those in front of it—all set beneath large rear-projected movie screens with images of the Orbiter’s position above Earth and other technical information. The new MCC, however, ushered in an age of eggshell-blue consoles, all on the same level, now updated with advanced computer technologies. It was more spacious and better ventilated with its higher ceilings, although to us flight controllers it was less intimate.

Along with this “new” MCC also came a new MER, upgraded from the original stuffy environs to a now expansive room filled with modern computer consoles, enough for almost every engineer, who could spend hours upon hours perusing all the stored telemetry that streamed almost continuously from the Orbiters. Gone were the crowded tables and chairs; now there was ample personal space for engineers to come in and review how their hardware was performing.

The MER conference room resided off to the side of the Mission Evaluation Room proper, accessible through one of two doors either from the MER or from an adjoining hallway. As conference rooms went this one was on the larger side, able to accommodate up to 70 people or more. A single colossal faux-wood table commandeered the center of the rectangular room, around which were placed a collection of blue cushioned rolling chairs. The perimeter of the room was occupied by rows of sedate, non-rolling chairs, also fashioned in blue fabric. Slung on the far wall of the room was a mammoth white screen, onto which was projected drafts of presentation charts, graphics, schematics of hardware showing all their internal guts, plots of downlinked and recorded telemetry, and other engineering data.

During the flight of STS-107, we would gather each day in this conference room, assembling in late morning to prepare for the MMT’s 2:00 p.m. start time. It was standard practice for the MER to maintain an In-Flight Anomaly (IFA) list, an accounting of each hardware component that had either outright failed in flight or its performance was consider off-nominal or “out of family” (meaning outside of past performance history). The IFAs were kept up to date to ensure the entire team could track and maintain cognizance of the state of the Orbiter, and also to better manage the hardware when the vehicle returned to Earth and assess the spacecraft’s readiness to fly its next mission. In general, each Shuttle mission would accumulate 10 or 20 such discrepancies that warranted the term “anomaly,” and in this regard STS-107 was no different, proceeding as a very typical mission. One of those anomalies was that a piece of the ET’s thermal protection foam came off during launch and struck the Orbiter’s left wing. This anomaly was discussed early in the mission during these OPO meetings and at the MMT, but not too significantly, as at the time it didn’t appear to warrant any more attention than the other IFAs.

So, I had come in to the MCC every day of the STS-107 mission, beginning on the flight’s second day (the MMT doesn’t meet on launch day) and continuing during the standard work week and also on weekends. For 14 straight days did I and others spend discussing the concerns and nuances of an orbiting space mission, watching over the hardware and always protecting the crew. At the end of the mission, because there are no strategic decisions left to make, it was typical for the MMT to not meet on entry day. On this day, the Space Shuttle Orbiter would convert from a spaceflight platform back into a dynamic aircraft, reenter Earth’s atmosphere, and glide to a stop on one of NASA’s long runways in Florida or California. On entry day, the MMT’s oversight was no longer required and from that point on, all responsibility fell to the flight control team.

Therefore, on Saturday, February 1, 2003, I finally had some relief from the daily grind of mission support, and I wasn’t wasting the opportunity. After two straight weeks of diligent work, I had the opportunity to sleep in a bit and enjoy a restful weekend.

Until the phone rang.

Drowsily, I crawled out of bed and wandered over to the telephone resting on my nightstand. Sleep still clouded my eyes, and I answered in a state of half awareness. On the other end I recognized the voice on my immediate boss, Rick Fitts, the division chief of the Systems Division. A burley, good-humored man with a thick but trimmed beard, Rick’s voice was solemn, quiet, and disturbingly unenthusiastic. Normally Rick was a bastion of optimism; he loved to laugh and with his wife Mary, who also worked in the Systems Division, embodied a joie de vivre that I always found infectious. But Rick’s tone that morning displayed a very different persona. Instead of cheeriness, this one was devastatingly serious.

Rick held his voice a low tone and he spoke slowly but clearly. He told me that just 30 minutes earlier we had lost communications with Columbia as the spacecraft and her crew were arcing over Texas on their reentry trajectory to KSC. It was now 30 minutes beyond the planned touchdown time and Columbia had never arrived. Furthermore, news shows on the TV were beginning to report smoking debris raining down on rural communities north of Houston.

My first thought of course was for the crew. Were they safe? I asked. Rick didn’t know. Did the vehicle come down somewhere? Again, he didn’t know. Well … what happened, I pleaded. Rick had no answers; it was simply too soon.

But as I think about that phone call today, I vividly remember one comment that Rick made, and I still recall his exact words. Before he hung up with me, Rick implored, “I sure hope it wasn’t the foam.”

I’ll return to this comment in a moment.

This was actually the second time in my life that I’ve experienced the immediate aftermath of a spaceflight tragedy. In January 1986 I was a senior at Embry-Riddle Aeronautical University, located in the humid and palm-laden coastal city of Daytona Beach, Florida. Residing only about 50 miles north of Cape Canaveral and KSC, students could often watch rocket launches from the comfort of the university’s campus. Whether we were watching Shuttles or expendable rockets, these massive machines on pillars of flame always took 15 or 30 seconds to arc out above the horizon before they became visible, but from our vantage point we could follow them on their fiery trajectories all the way until they passed beyond the far horizon or until the light from their burning engines became too diffuse to see.

On January 28, 1986, at 11:38 a.m., the Space Shuttle Challenger and crew of seven was launching on a mission designated as STS-51L to deploy one of NASA’s immense tracking and data relay communication satellites. Onboard as a crewmember was Teacher-on-Space participant Christa McAuliffe, a non-career astronaut and high school teacher from New Hampshire. At the time of launch I was sitting in a senior-level aircraft design class, one of about 20 students perched atop stools positioned in front of drafting tables, all focused on our senior project. This was before the advent of computer-aided design, so we all worked with sheets of vellum, mechanical pencils, and large, crumbly erasers.

In the midst of instruction from the professor, as we discussed weight and balance and center of gravity, the department secretary opened the door and cautiously peered inside. I noticed her cowering behind the door and thought it peculiar—the secretaries never interrupted a class in session—but then she shut the door again and disappeared, so I returned my attention back to the professor. Perhaps five minutes later she opened the door again and this time stepped into the room, announcing what had just happened. Space Shuttle Challenger had exploded during launch!2

Needless to say, all focus on whatever that day’s lesson was utterly lost. Presaging my experience with Columbia, the first question most in the class asked was what happened to the crew. Were they okay? Did they somehow escape the conflagration and glide the Orbiter back to the ground, or maybe even parachute into the ocean to await rescue?3 No one knew, but we bandied about elements of speculation on what was within the realm of possibility. After perhaps 20 minutes of chatter, the professor decided to cancel the remainder of that day’s class, and we all spilled out of the building and out onto the campus’s large parking lot.

Immediately, I looked upward. The image of the sky that morning is irrevocably scarred into my memory. It was clear and cold, the cloudless sky a luminescent shade of light blue. But a large shaft of smoke, now scattered somewhat by upper-level winds, thrust upward to a point maybe 50 degrees above the horizon. At the zenith, the smoke plumes reversed course and streamed back toward Earth until they met the horizon and disappeared from sight. The entire colossus reminded me of the shape of an umbrella, with the spokes a cloudy white imprinted on a background of irrepressible blue.

Now, 17 years later in my bedroom, swaying above my bed with disbelief, still holding the phone even after Rick had hung up, my thoughts again went to questions about the crew. Could they have somehow survived, against all odds? I wasn’t willing to declare them lost yet.

It’s important to keep in mind the incredulity of the news I had just heard from Rick. Shuttles do not arrive late and still land safely. Physics won’t allow it. An Orbiter returning from space is a pure glider, following a guided-ballistic trajectory at the whim of gravity. The vehicle cannot go around for another approach if it isn’t lined up well with the runway. That simply isn’t an option. Once committed to reentering Earth’s atmosphere, the Shuttle has only two possible outcomes: it either guides itself to its intended destination on a balancing dance between altitude, speed, and energy, or it is lost. There are no nuances on this or subtle variations; either the flight ends successfully, or it ends tragically.

Having spent 11 years as an active flight controller and supporting over 50 Shuttle missions on console, I rationally knew all of that. When I returned the phone to its cradle, against logic, I found myself still holding out hope that somehow, miraculously, the crew might have survived. But I also knew immediately that Columbia, the spaceship, was lost forever.

How does one deal with such events? Well, there is an immediate sense of loss and sorrow. The sorrow for the crew is obvious. Astronauts are not just cardboard cutouts of orange flight-suited icons, but real people, men and women with families, personalities, idiosyncrasies, and foibles; mothers and fathers, sisters and brothers, aunts and uncles, real human beings who, in many cases, became colleagues and friends to those of us who had the honor of working with them. They laughed, they complained, they ate lunch in the JSC cafeteria and drank beer over weekend barbeques with coworkers and family.

Now, the realization that seven of them might that very morning had died was not only a sobering thought but also one that my mind preferred not even believing, at least not right away.

But there was more to this than just the empathy of loss that myself and much of the rest of the nation and the world experienced. For those of us who worked in the Shuttle program, and maybe more specifically for us flight controllers, there was an immediate sense of failure. Plainly put, it felt like we didn’t do our job of protecting the crew. Somehow, in some way, consciously or unconsciously, we had failed. We failed the program, but more importantly we failed those seven astronauts. In the opening hours of this tragedy, the realization of that was vivid and apparent. It was and is today a feeling that, for me anyway, will never completely go away. Time and distance cannot expunge that sense of ultimate failure. It’ll be with me forever.

So, what did I do the rest of that terrible day? One thing I did not do was attempt to go in to the MCC that Saturday. I knew the procedures and protocols for mishaps and understood that the entire building would be locked down with no one allowed to go in or out. The team would be spending hours preserving data, records, logs, and anything else that might provide insight into what happened or offer evidence to an investigation.

Instead, I largely watched TV that day. As this was in the days before broad use of the internet was available, local and cable news was the best way to gain the latest information. I recall one shot from a roving news reporter and cameraperson of a street intersection somewhere north of Houston, in Nacogdoches or maybe San Augustine, Texas. On the road lay a smoking metal sphere, misshapen and partially collapsed. To my eye it was clearly a pressure vessel of some sort, possibly one of the dozens that resided inside of Columbia but now lay naked, bare and exposed, releasing tendrils of vapor and smoke from its seared surface. It didn’t require a forensic analysis to realize that what I was seeing was a part of the Space Shuttle Orbiter. Just an hour before this metallic sphere had been dutifully serving its purpose as one of thousands of parts that make up the spacecraft. Now, it lay on the ground, broken and bent and burned, releasing toxic fumes in a testament of the scorched journey it took to the place it now lay resting.

Scenes like this were replayed any number of times that day. CNN was all over it, as were the local and other national news sources. The unfolding tragedy struck a nerve with the nation who routinely were awed by spectacles from space but rarely understood the risks inherent in spaceflight. Now, there was no denying that the pursuit of exploration was indeed dangerous and that, sometimes, it effects a cost.

Now, what of Rick Fitt’s comment to me over the phone that morning. As you recall, almost immediately after the accident occurred and just before we hung up, Rick had glumly stated to me, “I sure hope it wasn’t the foam.” Let me explain the significance of this and why I bring it up here. As mentioned, one of my responsibilities as TA for Shuttle within the Systems Division was to coordinate and assemble our division’s Certificate of Flight Readiness (CoFR) package. Each division within MOD, plus all the various other organizations that made up the SSP, documented a certification prior to each Shuttle mission declaring that they were ready for the flight. That is, the products they produced were ready, the hardware and software was flight-qualified, all processes and procedures were in place, the crew and flight control team was trained, and together NASA and its associated contractors were prepared to conduct the mission. Our Systems Division CoFR would be integrated into the larger MOD CoFR, and the MOD CoFR into the overall SSP CoFR. It is a methodical, stage-wise process to ensure that nothing is overlooked and that all deviations from standard are known, understood, discussed, and accepted before proceeding to launch.

A week before February 1, while Columbia and the STS-107 mission was in orbit, many of us in the Systems Division—I, Rick, others in the division office, branch chiefs, group leads, and some of the discipline flight controllers assigned to the mission—gathered in our building’s main conference room to review the CoFR package for the mission that was to follow Columbia’s, that being STS-114 on Discovery. With a crew of seven, that mission was one of resupply and logistics to the International Space Station. Typically, the CoFR process analyzes the previous mission’s IFAs to ensure they are understood and posed no threats to the upcoming mission. Well, the previous mission would be Columbia’s, which was presently in space and ongoing, so instead we reviewed the IFAs from a few of the mission before Columbia’s, including STS-112 on Atlantis.

One of the anomalies on that mission was a large section of thermal protection foam that had broken away during launch and struck the large metallic ring at the base of one of the solid rocket boosters (SRB, called the aft skirt). While the impact posed no issues for that flight of Atlantis, after the boosters were recovered it was noticed that this foam impact had left a large dent in the aft skirt. At face value that may not sound so remarkable but consider that when a Space Shuttle is stacked, the process begins with bolting this aft skirt to the mobile launch platform. The SRB sections are then added, one by one, four segments in all, followed by a nose cone. With the two SRBs assembled, the ET is then slung between the two SRBs and, finally, the 100-ton Orbiter hung off the side of the tank. All of the massive weight of rocket and spacecraft (and eventually tons of propellant) bear its load on those two aft skirts. Purposedly, these skirts are tough, resolute, and very, very solid. For a piece of insulation foam to break off during flight and put a dent into one of these skirts, it must have imparted quite a punch!

As we ran through the STS-114 CoFR we reviewed these facts. We also knew that Columbia had incurred a similar foam strike, but of course we had no idea of the impending doom it was to cause when the spacecraft reentered Earth’s atmosphere at 25 times the speed of sound, generating searing temperatures upwards of 3,000°F.

Although no one knew this was going to happen, we did know that Columbia had been struck by foam, just as had Atlantis on the mission before, and just as had many Shuttle missions throughout the history of the program. These foam strikes were discussed as we navigated through our STS-114 CoFR package, at the very same time that Columbia orbited a hundred miles above us.

The remarkable thing to me, here, is that when the accident did happen a week later and Rick called me with the news, his mind apparently went immediately to the foam strike. Remember, this call took place less than an hour after the accident. There were no facts available to us, no tangible data, no real understanding of what had just happened and why. And yet, something drove Rick—experience, wisdom, intuition, something—to almost immediately fixate on the foam.

Was this simply a serendipitous or lucky guess? I don’t think so. Did Rick know something that the rest of us didn’t? I doubt that very much. But while a million things could have caused the tragedy, the foam strike was an obvious first place to begin looking.

Now, when a safety practitioner is taught how to conduct a mishap investigation, the importance of keeping an open mind and not jumping to conclusions is emphasized. That behavior is repeatedly mentioned again and again as a foundational tenant of mishap investigations. Don’t assume anything, follow all paths, and allow the data to dictate a solution. It’s a sensible philosophy and allows for even seemingly unlikely causes to be proven true based upon their merit. Plus, the process eliminates bias and favoritism. That is a bedrock belief that underlies good engineering.

But here’s the thing. While the eventual STS-107 accident fault tree included hundreds of potential causes, all of which had to be investigated before being ruled out, Rick’s mind immediately went to the foam. I’d guess that many others did as well.

Life Lesson for Engineers, Managers, and Leaders

Don’t discard the obvious; sometimes it is the right answer.

In the case of STS-107, that proved to be true. Of all the potential things that could have brought down Columbia, Rick’s initial guess turned out to be right.

This simple fact has sparked a realization within me. Whether investigating a NASA accident or probing some other mystery in life, sometimes the obvious answer is the right one. Life doesn’t always give us the ability to do a formal mishap investigation or construct a fault tree analysis for every conundrum or question we face—sometimes we have to figure things out for ourselves with no assistance. But in doing this myself, while I keep an open mind and seek out all possible answers, I also don’t discard an obvious one simply because it may be easily obtained. In my experience, there can be a tendency to discard an obvious answer because…well, because it’s obvious. I’ve witnessed an underlying human predisposition to assume that answers to complex problems also need to be complex. Sometimes they are, but sometimes they aren’t. I’ve learned to hold on to an obvious answer as part of the solution trade space and resist the temptation to ignore it with the assumption that, “that was too easy.”

Obvious answers, in these cases, are just as pertinent as complex ones.

For engineers it’s a bit more complicated because we seek out deterministic answers. We don’t rely on “gut feel” but instead place our trust in data, analysis, demonstratable facts, and other purely tangible evidence. And yet, sometimes, even engineers have a qualitative, even ephemeral, sense for what may be going on. And in that it’s important to at least listen to that sense. An obvious answer might prove to be true. While “obvious” isn’t rationale for eliminating other potential causes on the fault tree, neither should it be eliminated because it’s qualitatively derived.

For managers, particularly for one who place their faith in methodology (such as Earned Value Management or Schedule Risk Assessments), there may be the temptation to wait for the analysis to be performed before making decisions. On the surface of it, that’s certainly okay; analyses undoubtedly can inform and fill in gaps when information is not available. But as with engineers, managers need not always wait for extensive analyses to be completed before making decisions. Or similarly, neither do they always need extensive analysis if they are able to quantify acceptable risk of a decision based on a seemingly obvious answer.

And of course, the same is true for leaders, in whom responsibility is often placed to assess acceptable risk of a situation. Leadership is often an experiential task where judgement and proficiency are measured by the extent of one’s experience. It is this experience from which leaders can draw direction. But leaders are also often faced with situations outside of their personal experience base, where they need to rely on alternate sources for wisdom. As with the others, an obvious solution can be a temptation and feared to be little more than an easy or quick answer. But even leaders should consider the potential of an obvious solution.

1 In the mid-1990s, JSC expanded the original Mission Control. The historic three-story building was augmented with a six-story, block-like structure, within which were placed the new flight control rooms from which flight controllers would oversee space missions. That new addition is still
in use today.

2 Technically, Challenger was not destroyed in the explosion of the Shuttle’s external tank and release of the contained liquid oxygen and liquid hydrogen. The Orbiter, rather, was ripped apart by supersonic aerodynamic forces once it was liberated from the integrated stack. Whether by explosion or aerodynamic forces, Challenger was destroyed and the crew lost their lives.

3 At the time of STS-51L, Space Shuttle crews did not carry parachutes and there was no means to allow them to escape the Orbiter.

Chapter 2

STS-107 Flight Control Team

Life Can Place You in Circumstances You Didn’t Expect

Since the beginning of the Shuttle Program (and likely even well before), a cameraman wanders about the Flight Control Room (FCR, pronounced: Ficker) during every launch and reentry to document the event both in film/video and in still photography. There are striking photos that capture the faces of flight controllers at the very moment that Space Shuttle Challenger exploded, 73 seconds after launch. These images starkly reveal the utter shock, horror, and disbelief on the faces of the flight directors Jay Greene and Lee Briscoe and the two astronaut communicators, (CAPCOMS) Dick Covey and Fred Gregory. Dick Covey (who, coincidentally would fly two and a half years later the next Shuttle mission following Challenger, STS-26, in what NASA euphemistically referred to as Return to Flight) is captured in the exact moment he realizes disaster has occurred. Only seconds after the explosion, Covey glances toward a television monitor positioned beside him on the CAPCOM console and comprehends that the rapidly expanding cloud and tendrils of smoke exiting from it can only mean that Challenger had exploded, and worse, friends of his were inside that stricken spacecraft. It’s easy to distill his emotion of the moment, a mystifying slice of time in which his brain is likely refusing to accept the reality of the situation. His silent, contorted expression speaks louder than if he had shouted out words of petrified terror he must have been feeling. Moments like these are captured for history’s sake through the work of the NASA photographers and camera operators and they are truly historical.

STS-107 was no different. As Columbia and crew were preparing to return to Earth after a successful two-week science mission, LeRoy Cain and his Entry team were settling in on console to help them come home. As with all Shuttle missions, the flight controllers assigned to the Ascent and Entry team would practice, or “sim” (short for simulation), the timelines, procedures, and actions that the control team and, more importantly, the crew, would take in flight. A standard Shuttle mission training template included four launch (or Ascent) simulations, each lasting four hours, and three reentry (or Entry) sims of similar duration. With the assigned flight control team on console in the MCC and the astronaut crew themselves strapped into the Shuttle Mission Simulator in JSC’s Building 5, the integrated team would train for the upcoming mission.

These sims were overseen by the MOD training team, a cadre who were infamous for being nefarious and almost malicious in their training regimens. I participated in countless Ascent and Entry sims and never once did everything go according to plan. That wasn’t the point of all of this training. The point, rather, was to hone the combined crew and flight control team to act as would fine machinery, working together like the most precisely constructed pocket watch. During a sim, the training team would throw at us malfunctions and component failures, not just one or two but an entire host of them, and in combinations that necessitated the team to think as a group. Us flight controllers all believed that the training teams took some professional but sadistic pleasure in coming up with scenarios that really made us think and squirm and innovate. Whether true or not, the point of all this training was to prepare the crew and flight control team to handle complex failure scenarios such that the team would react quickly and not overanalyze a situation since these dynamic timeframes wouldn’t allow for long analysis.

On Saturday, February 1, 2003, the STS-107 Entry team came on console about six hours before landing and handed over from the previous shift. I experienced this exact handover—the final Orbit shift to the Entry shift—multiple times over my career I can tell you that, prepared or not, you knew that this was now showtime. There was no mistaking that after all that training, we were now going to do this for real! You would walk into the MCC with a certain energy about yourself that fell somewhere between exhilaration and expectation, anxiety, and confidence, and that agitation palpitated throughout the entire team. Although I’ve never flown in space, I’ve always imagined that the astronaut crews felt the same way.

LeRoy Cain led this team. LeRoy himself was a flight controller from the Systems Division, having begun his career as a GNC flight controller before being selected as a flight director, specializing in ascent and entry. When meeting him, the first thing one might notice about LeRoy was his physique—he was pencil thin, lithe and sinewy, the sort of person who would eat a small cup of yogurt for lunch and feel full. LeRoy was affable, friendly, but sternly professional in a no-nonsense way when on console.

Next to LeRoy that morning was Steve Stich, another flight director who was serving as “Weather Flight.” For obvious reasons, the predicted weather conditions at the Shuttle landing strip were a critical part in the decision to either deorbit or to wave off a landing attempt for another orbit or possibly even another day. Weather Flight was responsible for keeping a close eye on atmospheric conditions and advising the flight director who would make the decision whether to deorbit. This delegation of responsibility allowed Flight to focus her or his attention on the vehicle, the crew, and the management of the flight control team.1

Adjacent to the flight directors were CAPCOMs Charlie Hobaugh (who went by the moniker Scorch) and Bill Gregory (known as Borneo). As there were two flight directors on console for Ascent and Entry, also were there two CAPCOMs, one being considered prime and allowed to speak with the crew and the other serving in support, assisting the prime. Both Scorch and Borneo were experienced astronauts, with Hobaugh flying in space as pilot on STS-104 only six months before, and Gregory on STS-67 in 1995.2

The rest of the team who populated the FCR that day are less well known—most do not have Wikipedia pages dedicated to them—but were every bit as dedicated, experienced, and professional as those who led the Entry team. What’s more, they were colleagues of mine, friends, people I had worked with for years in fraternity and sorority in the shared experience of Shuttle flight control. On the left side of the FCR there was Dean Lenort who sat at the PROP console, all six foot three of him; Mike Sarafin at GNC, adorned with his trademark moustache.3 Behind them was Laura Hoppe at INCO, politely quiet but with an irrepressible humor. Behind Laura at the DPS (flight computers and associated avionics) was Jennifer Hagin, who would soon go on to lead the entire DPS group and show exquisite qualities of leadership.

Switching to the right-hand side of the FCR was positioned Jeff Kling at the MMACS console. Jeff was a contractor who worked for United Space Alliance (as did Dean at PROP) but was one of the most experienced flight controllers in the room. Quiet and reserved in person, Jeff was the paradigm of the flight control community—a prodigious systems expert, calm under pressure and capable of handling just about any situation. We’ll return to Jeff in a moment.

Next to Jeff was the EGIL console (my console position!) which was occupied that day my Mike Maher. Mike, a civil servant, had been an In-Flight Maintenance (IFM) flight controller (supporting MMACS) for years and had only moved over to the EGIL position when the IFM group was converted into an all-contractor position. Behind Mike was Katie Rogers at the EECOM console, a woman with a sharp, sometimes sarcastic sense of humor who always left me in stitches.

Along the front row of console directly beneath the large projection screens sat Bill Foster in GC, who was responsible for the Mission Control building and the space communications network. As of this writing, Bill still works in the GC group and is a committed fan of spaceflight history, taking on the ancillary role of historian of the MCC. Across from Bill sat Richard Jones at the FDO who, like Sarafin, would also go on to be a flight director, emulating LeRoy in the Ascent and Entry specialty. Next to Richard was Wayne Hensley at the GUIDANCE console.

There were also others who I didn’t know as well, those at the PAYLOADS console as well as FAO, responsible for planning and replanning the crew’s timeline. A SURGEON and PAO completed the team.

There is a vibrancy to the FCR that is difficult to explain. Inside that room permeates an energy that is apparent as soon as one enters. The subtle blue hues of the consoles and placidly bland light grey of the walls inside an airy, high-ceilinged enclosure promotes an ambiance of tranquilizing solitude. Even the rear of the room was adorned with foliage (albeit plastic foliage) which made the place feel like an atrium. When a person enters the first time, they might be struck by how quiet it is inside. The seemingly pleasant whispers of multiple conversations resemble that of a college library (flight controllers use a system of communication loops and headsets for all their conversations, so there is no need to “yell across the room”). Each controller is positioned in front of three large computer monitors displaying bright graphics and reams of numbers, immersing them in a mesmerizing glow of information. The most obvious sound is the stead hum of fans cooling the console computer monitors and the ceiling-mounted projectors.

But don’t let the illusion of soothing repose lull you into misperceiving the vibrancy of the room. This is the place where human spaceflight happens, along with all its excitement and risk. Within these walls, the activities occurring hundreds of miles in Earth orbit are focused intensely into a nexus of oversight. Beyond the physical distance between spacecraft and control center, little actually separates the two. The MCC has more insight into the functioning of the vehicle than even the onboard crew, and controllers within these hallowed halls are the only ones who can communicate with the orbiting astronauts, almost serving as additional crew. This relationship forges a tightly knit synergy that creates a kinship of shared trust in an inherently dangerous endeavor.

Returning to Earth after a successful Shuttle mission is just such an endeavor. When the Entry team comes on console some hours before the deorbit burn—the firing of rocket engines to slow the spacecraft from orbital velocities and begin its plunge back into Earth’s atmosphere—the Orbiter has not yet made the transition from spacecraft to atmospheric gliding vehicle. The crew would be woken before the Entry team arrives and begin consuming large quantities of water—a medical remediation that helped astronauts adjust to the onset of gravity after days in space—in addition to salt tablets.4

But activities supporting their return to Earth wait for the Entry team. Approximately four hours prior to the deorbit burn, the crew and team transitions to the Deorbit Prep checklist, a timelined sequence of procedures that prepares the ship for reentry. This includes the activation of additional computers which need to be powered on and added to the reduced set used for orbit operations. Navigation equipment, which had remained dormant since launch, are revitalized. The payload bay doors are closed and the Orbiter’s freon cooling system bypassed from radiators mounted on the inside surface of the doors and directed to the flash evaporator system which would ensure cooling through the reentry. The crew would don their cumbersome pumpkin-orange pressure suits, necessitating an acrobatic three-dimensional microgravity ballet just to squeeze into, and strap themselves into their seats Eventually, just before the command is given to fire the aft-facing Orbital Maneuvering System (OMS) rockets, the spacecraft’s auxiliary power units are activated, generating hydraulic pressure to move the vehicle’s control surfaces during aerodynamic flight.

When Columbia and crew were ready and the “Go for Deorbit” communicated by the CAPCOM, all was set to initiate a sequence of events that would be forever burnished into this team’s memory. The deorbit burn, a five-minute firing of the two large OMS engines attached to the aft end of the Orbiter, just above the three main engines, occurred without problems at 8:15 a.m. Eastern Standard Time. At the completion of the burn, Columbia was now on a trajectory that intercepted the surface of Earth, taking the spacecraft out of orbit and plunging the ship into the rarified air of the upper atmosphere.

It takes about 30 minutes from the deorbit burn before the spacecraft strikes the upper parts of the atmosphere, a milestone we called Entry Interface (EI). That time period is actually a quiet one for the flight control team. Doing all the previously mentioned reconfiguration and activation keeps both the Entry team and the crew quite busy. But once the deorbit burn takes place, with the crew strapped into their seats and all preparations complete, the FCR is normally quiet. Each flight controller would be monitoring their system or watching the trajectory of the Orbiter. These minutes of seeming solitude following a few hours of busy activity allows the team some quick moments of preparation before the time-critical, dynamic period from EI to touchdown begins, when events would unfold quickly. All the training the team endured was mostly focused on that timeframe—EI to touchdown—where in only 30 minutes the Orbiter would go from Mach 25 at an altitude of 400,000 feet to a dead stop on the runway.

The team was well prepared to follow that nominal timeline, as many had done previously on other Shuttle missions. But on this day, what transpired next happened very quickly and was, of course, completely unexpected. The team responded with professionalism and their reactions were commensurate with the rigorous training provided to them, but nothing, really, could totally prepare them for the events that were about to unfold.

After the deorbit burn, reentry was proceeding normally. NASA would routinely send aloft high-altitude balloons to gather information on upper-level winds, and LeRoy asked Richard Jones (FDO) if another balloon had yet been launched. Richard confirmed the balloon and mentioned that it might be 10 or 15 minutes before they received the data. Jeff Kling (MMACS) called to report a good Space Shuttle Main Engine hydraulic repressurization, a nominal reentry action taken by the crew for the three main engines.5 CAPCOM Scorch confirmed with the crew that we in the MCC saw the repress, to which Commander Rick Husband acknowledged. Mike Sarafin (GNC) confirmed another crew action, this one with the vehicle’s flight control power, and requested the crew enter a command into the flight computers to optimize the inertial platform. Again, Scorch informed the crew of this request and action was taken in the cockpit. Then Mike confirmed that the spacecraft was in a good attitude for EI. Finally, the crew transitioned the flight software to Ops 304, the code required to begin EI.

More chatter transpired on the flight director loop as LeRoy and Richard went back to a discussion on the balloons and the availability of the wind data. The talk was professional and unhurried, almost the sort of conversation two people would have discussing a sporting event by the coffee maker or water cooler. While Columbia was hurtling through space 20 times faster than a bullet, conditions inside the MCC were routine, but not cavalier. I can tell you from experience that even as events were occurring nominally per the checklists, there is always a light tension in the MCC during reentry. It’s a recognition that even a “nominal” reentry stands on the edge of a knife when returning to Earth. I always delt with this tension by studying the telemetry being displayed on my console monitors and by talking with my EGIL counterpart. Moving my mouth, I found, actually could alleviate some tension. The talk was focused on doing our job, on the passage of reentry event milestones, and on the performance of the systems for which we were responsible. Amiable chatter about, say, our plans after work, or the success or failure of a favorite sports team, were inappropriate for flight controllers during these critical mission operations. Everyone focuses on their job to the exclusion of all else.

This is exactly as MMACS/Jeff Kling did when he made his next call on the flight director loop:

KLING: Flight, MMACS. 

CAIN: Go ahead, MMACS. 

KLING: FYI, I’ve just lost four separate temperature transducers on the left side of the vehicle, hydraulic return temperatures. [pause] Two of them on system one and one in each of systems 2 and 3. 

CAIN: Four hyd return temps? 

KLING: To the left outboard and left inboard elevons [wing flaps]. 

What Jeff was reporting was not one of those standard mission events; this was decidedly off-nominal. Watching his telemetry with his backroom counterpart—scanning the reams of numbers that streamed out of Columbia, coursed through the communications network, entered the MCC and routed to his computer screens—Jeff saw something he could not immediately explain. He saw four temperature sensors blink out; that is, fail to zero, or go “off-scale low” using a flight controller’s terminology. Flight controllers would train for telemetry failures all the time. The training teams would routinely simulate computer and avionics failures that limits the insight we have into the spacecraft, causing us to get inventive with how we manage the systems. A single computer could remove a slew of data, but (and this is critical) those losses always came in a pattern in which flight controllers can find a commonality. That commonality makes it possible to diagnose the failure as that of the computer and not of the dozens of seemingly unrelated sensors. Unfortunately, for the four measurements that Jeff saw go to zero, there was no such commonality or single computer failure that could link the indications together. But being the exceptional flight controller he was, Jeff did recognize a connection—all four sensors resided in Columbia’s left wing.

Both Jeff and LeRoy then followed that train of logic:

CAIN: OK, is there anything common to them, DSC [Dedicated Signal Conditioner] or MDM [Multiplexer/Demultiplexer] or anything? I mean, you’re telling me you lost them all at exactly the same time.

KLING: No, not exactly. They were within probably four or five seconds of each other.

CAIN: OK, where are those? Where is that instrumentation located?

KLING: They’re—all four of them are located in the aft part of the left wing, right in front of the elevons, elevon actuators. And there is no commonality.

CAIN: No commonality.

As there was no single failure that would have resulted in the loss of these indications, this clearly would have perplexed Jeff and his backroom.6 It’s not hard for me to imagine the two of them struggling to figure out what could have caused four independent temperature sensors to fail beyond the commonality of electronics boxes or common path wiring. They would have scoured through schematics and drawings and quickly pulled out reference material, all the time recognizing that the spacecraft was still hurtling through the upper parts of Earth’s atmosphere at incredible speeds. As confusing as these measurement losses were, they couldn’t afford to take their eyes off of the rest of the mechanical systems they were responsible for. Finding a balance between failure diagnosis and continual monitoring of everything else was a skill only born by years and years of challenging training.

With this drama going on, the rest of the room continued with their nominal reporting. Mike Sarafin at the GNC console reported the flight computers were processing good aerodynamic data. Bill Foster at GC reported that the air-to-ground communications channels were enabled for the landing team in Florida.

But LeRoy next asked a question of Sarafin that, in my mind, was emblematic of the fact that he hadn’t forgotten about Jeff’s four temperature sensors and was continuing to try to figure out just what was going on:

CAIN: GNC, Flight.

SARAFIN: Flight, GNC.

CAIN: Everything look good to you, control and rates and everything is nominal, right?

SARAFIN: Control’s been stable through the rolls that we’ve done so far, Flight, we have good trims. I don’t see anything out of the ordinary.

CAIN: OK. MMACS, Flight.

KLING: Flight, MMACS.

CAIN: All other indications for your hydraulic system indications are good?

KLING: They’re all good, we’ve had good quantities all the way across.

CAIN: And the other temps are normal?

KLING: The other temps are normal, yes sir.

CAIN: And when you say you lost these, are you saying that they went to zero—

KLING: All four of them are off-scale low.

CAIN: —or off-scale low.7

KLING: And they were all staggered. They were, like I said, within several seconds of each other.

CAIN: OK.

This is what well trained flight controllers do. Flight controllers aren’t omniscient and don’t have crystal balls. They have to take the limited insight that the vehicle provides and find ways to fit that data to plausible scenarios. They struggle to make sense of situations as if trying to understand a story when 50 percent of the words in a book are redacted. It takes experience, ingenuity and a certain amount of cunning interrogation to conjecture with confidence the causes of something happening thousands of miles away, a hundred miles above and at speeds that dwarf the highest performing aircraft in existence.

FDO/Richard Jones started to dutifully report about the balloons when suddenly Jeff broke in again:

KLING: Flight, MMACS.

CAIN: Go.

KLING: We just lost tire pressure on left outboard and left inboard, both tires.

CAIN: Copy. Is it instrumentation, MMACS?

KLING: Flight, MMACS, those are also off, off-scale low.

Sitting next to the flight directors are two CAPCOMs. Incumbent in their role is to keep the crew informed about the state of the vehicle they were riding. Even with the limited insight the flight control team has on the ground, that insight still exceeds what the crew has onboard the spacecraft. For every 10 points of data that the MCC receives on the ground, perhaps one of those are available to the onboard crew. Fortunately, the tire pressure measurements were ones that were available to the crew. Scorch knew from his spaceflight experience that the loss of these measurements would ring an alarm onboard, and he wanted the crew to know that we were seeing the same on the ground. Scorch called up to the crew with that fact, and Rick Husband, the commander of STS-107, replied:

HUSBAND: Roger, uh bu...

Rick Husband’s voice faded into a cacophony of static. And that was the last communications ever to be received from the Space Shuttle Columbia.

Back in Mission Control, except for this unexpected loss of communications, the flight control team still didn’t know what was transpiring 40 miles above the piney woods of eastern Texas. What they did know was that with the communication loss, all telemetry from Columbia had ceased to be received and all the numbers displayed on the flight controllers’ consoles had mysteriously transitioned to blue. Normally, active telemetry is displayed in green, indicating conditions within normal limits. If a sensor measured something that was out of limits, it might turn yellow, or even red if the condition was critical. Blue numbers, on the other hand, simply meant the telemetry had gone “static.” This occurred every orbit when the Orbiter passed out of range of NASA’s constellation of Tracking and Data Relay Satellite (TDRS) communication satellites, a gap in coverage over the Indian Ocean. Static data was a daily, almost hourly occurrence while in orbital flight. But it was not expected during Entry.

Sometimes, due to the reentry trajectory and the heating caused by friction between the returning spacecraft and the atmosphere, a sheath of plasma would surround the vehicle which could temporarily block out communications. This occurred once when I was on console during STS-39 and the Loss Of Signal (LOS) lasted a whopping five minutes, resulting in a very anxious flight control team. Similarly, after Columbia commander’s last transmission, all the data in the control center went static. Given the circumstances, however, the STS-107 Entry team continued to do their jobs.

Laura Hoppe at the INCO console reported that “We’re taking a few [communications] hits here,” but related that the angle from Columbia to the TDRS satellite was “right up on top of the tail,” which meant that broken communication was not unexpected.

Now with a growing mystery to solve, LeRoy returned again to Jeff:

CAIN: MMACS, Flight.

KLING: Flight, MMACS.

CAIN: And there’s no commonality between all these tire pressure instrumentations and the hydraulic return instrumentations?

KLING: No, sir, there’s not. We’ve also lost the nose gear down talkback8 and the right main gear talkback.

CAIN: Nose gear and right main gear down talkbacks?

KLING: Yes, sir. 

Then Katie Rogers at the EECOM console reported more indications gone amiss:

ROGERS: And Flight, EECOM.

CAIN: EECOM?

ROGERS: I’ve got four temperature sensors on the bond-line data that are off- scale low.

I have to believe that at this point LeRoy knew something wrong was clearly going on with Columbia. Laura Hoppe (INCO) clarified that while poor communications with the TDRS satellite were not unexpected at this point in the trajectory, she didn’t expect a loss of signal as long as was being experienced. LeRoy checked with Bill Foster (GC) as to when we would expect UHF communications at the landing site (received directly at KSC and not routed through TDRS). Mike Sarafin (GNC) suggested to LeRoy that if there was any reason to expect controllability issues with Columbia that he should keep the contingency procedures handy. LeRoy went back to Laura, asking about the loss of comm, with Laura responding that pre-mission predictions indicated good comm at this point in the reentry.

Now, Scorch began a repeated call to Columbia, transmitted once every 30 seconds or so, that has since become synonymous with the accident. His repeated plea, offered vainly in the hope that maybe, just maybe, Columbia would respond, was a desperate attempt but one that followed protocol. In the event of loss of communications, call in the blind.9

HOBAUGH: Columbia, Houston, comm check. 

Richard Jones reported the balloon data had come in and fed the expected landing conditions to LeRoy. Jeff clarified that the tire pressure measurements had gone erratic before going to zero. Scorch called the crew, “Columbia, Houston, UHF comm check” again, and again, and again.

At this point it’s easy to tell from the video being taken in the FCR that LeRoy was quite concerned. This situation was beyond any scenario faced during a sim, maybe beyond anything anyone in the training division could have imagined. I can only guess at the powerlessness LeRoy and the team must have felt at this point. There was little they could do—whatever was happening with the crew and the spacecraft was happening far away from them and no commands sent from Mission Control to Columbia could fix the situation.

HOBAUGH: Columbia, Houston, UHF comm check.

Then, Bill Foster chillingly announced that MILA (Merritt Island Launch Annex, which contained much of the communications receivers at the KSC) was “not reporting any RF [radio frequency] at this time.” This meant that no signal was being received from Columbia at the landing site. Quickly, and just as agonizingly, he reported that there were no C-band hits, meaning that the ground radar at the landing site wasn’t picking up anything entering the airspace.

In the event of loss of communications, call in the blind.

HOBAUGH: Columbia, Houston, UHF comm check.

And on and on. The team was doing exactly what they had been trained to do—work the problem, don’t give up, forget fear, put anxiety away, exert every effort, and do what you’re trained to do. Succeed.

But finally, like a veterinarian easing a terminally injured animal into the solace of death by compassionately administering a potion that will ease the transition, Phil Engelauf leaned over the flight director console and whispered something to LeRoy. Phil was serving at the MOD console that day—a management console with no flight control responsibilities situated in the last row behind the flight director console. To Phil’s side was astronaut Ellen Ochoa, also there in a management capacity. It would be revealed later that Phil had just received a phone call informing him that local news sources were reporting multiple contrails over the region of Texas where Columbia had just been. The vehicle had broken up in flight.

With the controlled demeanor of a man who had just witnessed tragedy but still had a job to perform, LeRoy uttered three words, a simple statement whose underlying message to the flight control team, and to the world, still continues to resonate. “Lock the doors.”10

The rest is history. Even faced with the unimaginable, LeRoy and his team continued to perform professionally. The focus had now abruptly shifted from flight to recovery operations, but there was no hesitation, no wallowing in sorrow, no one struck silent due to disbelief. Instead, there was only professionalism, just like they were trained to do, just like Flight Director Jay Greene and his flight control team had done 17 years earlier when Challenger exploded. Even in tragedy, there was still work that needed to be done, and the STS-107 Entry team endeavored to do just that.

For my part, even today I still think back to that day in the FCR. Everyone in that room was a colleague, co-worker, or friend of mine, and everything that transpired there were things that I understood, the semantics and protocols that I lived for 11 years as a Space Shuttle flight controller. I had moved on to my position as TA for the Systems Division a short two years prior, but different choices or other motivations on my part could easily have placed me at the EGIL console that day instead of Mike Maher.

The good people who composed the STS-107 Entry team were miraculous engineers and dedicated professionals. But just like astronauts, outside of work they were also simple people, mothers and fathers, husbands and wives, people who ate ice cream and watched TV, took their children to soccer matches and got takeout when they lacked the energy to cook. They paid bills, bought cars, vacuumed their house’s rugs, and brushed their teeth. Some were married, others single or divorced, many had children and others none. They showed up to work in the morning and left in the afternoon, and all enjoyed the rest afforded by weekends (unless they were working console on a Shuttle flight).

These people were a cross section of America and carried with them the tapestries that weave individual lives. They were people who you could pass on the street and never notice. And then, one day in early February, they were forced headlong into history.

Life Lesson for Engineers, Managers, and Leaders

Life can place you in circumstances you didn’t expect.

My guess is none of my colleagues and friends on that team would have woken up that morning and considered that by the end of the day they would be a part of history. But, whether inconceivable or not, they were and today they are. Their presence in the FCR on February 1, 2003, is now irrevocably inscribed in the history of NASA and the world. While the convergence of their presence in the control center and historical events was outside their control, whether they choose to revel in their inclusion in history, reject it, or simply ignore the significance of their participation and quietly get on with their lives is entirely their choice. It can be an either honor or a bane, or sometimes both, for them to carry this knowledge.

Part of the MOD credo has long been, “To always be aware that suddenly and unexpectedly we may find ourselves in a role where our performance has ultimate consequences.” You can never predict when this might occur, but discipline, competence, confidence, responsibility, toughness, teamwork, and vigilance—the credo’s stated qualities essential to professional excellence—can help engineers, managers, and leaders be ready when the unexpected happens.

Figure 5. Members of the STS-107 Entry Team, in a pre-mission photograph in the Flight Control Room taken with all seven crew members. (NASA Image JSC2003-00010)

1 Today, Stich is the program manager for NASA’s Commercial Crew Program.

2 Scorch would go on to fly two more missions after the Columbia accident—STS-118 and STS-129, in 2007 and 2009, respectfully. STS-67 was Borneo’s lone space flight.

3 Mike also went on to be a flight director and later served as the mission manager of NASA’s Artemis I Moon mission.

4 Suzanne M. Fortney, Fluid-Loading Solutions and Plasma Volume (Washington, DC: NASA TP-3456), https://ntrs.nasa.gov/api/citations/19940019071/downloads/19940019071.pdf (accessed Jan. 21, 2025).

5 The Space Shuttle Main Engines, only used during launch, are not used in any capacity on entry. During the return to Earth, helium is used to purge the system and repressurize it prior to the Orbiter’s immersion back into the atmosphere.

6 In Mission Control, there is the “Front Room”—the room with banks of consoles and flight controllers fronted by large screen projections commonly seen on television and represented in movies. But in addition to the Front Room, a series of smaller, non-descript rooms—called “backrooms,” or Multi-Purpose Support Rooms, where other flight controllers reside. While the more experienced controllers populate the front room, less experienced personnel occupy the backroom positions, supporting the Front Room controllers. Front room and Backroom truly forge a partnership in safely overseeing NASA’s human spaceflight missions.

7 Each telemetry measurement on the Orbiter operated within a range, from highest value to lowest. If a measurement were to go below that lowest value, or the sensor lose power, it would go “off- scale low.”

8 A bi-level, or yes/no, indication on the forward landing gear. These talkbacks provide visual indications to the crew of the status of a piece of equipment. Nominal expected position would show up as a block of gray, off-nominal as a stripped banner called “barber-pole.”

9 Calling “in the blind” is a colloquialism used in flight control that traces its lineage back to aeronautics control. The term simply means a ground controller is continuing to communicate, but is unaware if the transmissions are being received.

10 Transcription based upon STS-107 mission audio.

Chapter 3

Crew Memorial Service at Johnson Space Center

Community Provides Comfort and Can Salve the Pain of Loss

The day after the accident, February 2, 2003, a Sunday, everyone at JSC received the following e-mail message:

The Johnson Space Center will host a memorial service for the crew of STS-107 at noon on Tuesday, Feb. 4. The memorial will bring the NASA family together to honor the courageous crew of STS-107. The service will only be open to badged civil servant and contractor employees due to logistics and security constraints. Employee spouses and children will not be able to attend due to space limitations.

The memorial will be held in the Center’s central mall area around Building 16. JSC civil service employees and badged contractor employees will be required to enter the designated area, between Building 4 and Building 7, at specific points for security screening. The program will commence at noon CST. Access points will open at approximately 9 a.m. and all attendees must be in place by 11 a.m. Since space is limited, attendance will be on a first-come-first-served basis. Please do not carry briefcases, purses, or other items which may delay the overall screening process.

Please watch for additional special notices with more details about the memorial service.

In Memoriam:

STS-107 Crewmembers:

Commander Rick Husband

Pilot William “Willie” McCool

Mission Specialist Michael Anderson

Mission Specialist David Brown

Mission Specialist Kalpana Chawla

Mission Specialist Laurel Clark

Payload Specialist Ilan Ramon1

The stated logistics and security constraints, in addition to the intentionally specific instructions to enter the designated area between two JSC buildings, was due to the expected presence of the president of the United States. George W. Bush was planning to travel to JSC to offer words in memoriam for the crew and to recognize the loss their families and their co-workers at JSC were enduring. In fact, his message was to be carried by national news organizations, so it was being crafted to empathize with the entirety of a grieving country.

This e-mail message was quickly followed by a more personalized message from JSC Center Director Lt. Gen. Jefferson D. Howell (retired), otherwise known as “Beak.”

We mourn the loss of the STS-107 crewmembers who were our beloved teammates, coworkers, and friends. There is no doubt that this tragedy has deeply affected all of us at JSC. It will be on our minds every minute for many days to come. However, we must persevere and forge ahead to ensure their loss was not in vain.

In this time of healing, it is up to us to carry on the legacy of their mission. We cannot fail them. We must look inward for our strength, while leaning on each other and our community for support.

Part of our healing process will be served through our ongoing support of the Expedition 6 crew aboard the International Space Station—we must ensure their continued success and safety. We must keep our watchful eye on them 24 hours a day, seven days a week, as we have capably done since we fully crewed the station more than two years ago. We will also look to our international partners to be at our side as we go forward.

We must reconfirm our commitment to NASA’s Missions of better understanding our home planet, exploring the universe, and inspiring the next generation of explorers. The STS-107 crewmembers exemplified that commitment. Their extraordinary work on orbit surely inspired many young minds and scientists around the world as they searched for new solutions to improve our lives here on Earth.

We know that human spaceflight is not an easy task, nor one without risk. However, it is one of the most noble endeavors in the history of humankind. Our President and Administrator understand this and have pledged an expeditious return to flight once we fully understand what transpired with Columbia and its crew.

The dream of going into space was made a reality by many of you here 40 years ago. We are known for solving challenging problems and expanding the limits of technology. This effort has been built upon the human spirit. A machine without a spirit does not go far. We at NASA have never had to worry about that as we have a huge wellspring of human spirit involved in all our efforts.

I am confident that we will persevere, and we will forever honor the memory of the STS-107 crew by continuing our dedication to space exploration and research. They carried our pride and the dreams of the world into orbit, and for them, we must carry on.

Beak sends.2

Inspiring words from a practiced leader. I’m sure Beak understood the importance of resilience and perseverance, in addition to focusing on the job at hand even in the presence of tragedy. Yet the further down one went in the hierarchy of JSC, the more personal the messaging became. On the following day, Monday, February 3, the following e-mail was distributed from the director of MOD, a gentleman (and former flight controller) named Jon Harpold:

We are all suffering and mourning the loss of Columbia and the STS-107 crew.

As is true with every manned space flight mission, most of us were involved at some level. This will be a difficult time for all of us and a time to help our team members and families.

But as we have demonstrated in the past, and will do in this case, we can and will move on with the help of each other, our loved ones, and our families and friends. We know that Rick and his crew, and others that have gone before them would want it no other way.

As we approach this first week back to work, we need to focus on each other and allow some time and space for a process that will be different for each of us.  As such, with the exception of ISS mission support, all previously planned technical meetings, planning meetings, and training are cancelled through Wednesday. 2/5.

MOD is planning to host with the Flight Director Office an open forum in the B30 Auditorium on Monday to provide some information and to answer questions about what we know so far, what activities are ongoing, and things we don’t know. There will be two sessions, one at 1400 and another at 1500.

A memorial service is being planned for Tuesday, time, and place at JSC has been sent out by the JSC today notification system.

We will hold some high-level strategy and planning meetings on Wednesday.

It is possible that we will plan for some activity starting on Thursday, possibly including some integrated simulation activity. We will stay close and informed and provide as much as we know when we know it.

Our initial priorities during the immediate next few days/weeks should be (1) continuing to provide excellent support for ISS mission and planning (we must be especially vigilant and detail oriented with our continuing planned ISS activities), (2) providing our expertise and support to the Columbia accident investigation activities/teams to allow a thorough professional review/investigation/resolution, and (3) planning for the resumption of the necessary training/simulation activities needed to continue our mission/manifest.

The center has activated a new web page to assist you and your co-workers within the JSC workforce as we deal with the Shuttle Columbia Tragedy. You may search this page for the latest information on the Center’s workforce activities, services, and resources related to this tragedy.3

Again, an emphasis on the grief we all were feeling. But Jon’s note was also a message to his troops that while we mourn, we needed to remain cognizant that other astronauts, those in space on the ISS, were still depending on us for their safety and for the success of their mission. For their sake, we couldn’t afford to take our eyes off the ball. True enough, and we within MOD certainly understood that message.

The import of Jon’s message certainly resonated for me. But when I read it, and reread it again and again, searching for some kind of solace in the face of the despair I felt, I focused on his opening words and noted a simple but powerful empathy of his admission.

“We are all suffering.”

In fact, we all were. Everyone I knew, all my colleagues at work, were suffering. It didn’t matter whether they knew the crew personally or had any direct role in the STS-107 mission. At times like these, those facts weren’t relevant. JSC was like one large spaceflight-connected family—civil servants and contractors, engineers and administrative staff, senior executives, and janitors—all were part of this communal entity called the Johnson Space Center, connected by a bond of kinship that was stronger than the most durable cement.

Figure 6. In memory of the Space Shuttle Columbia crewmembers who lost their lives on February 1, 2003, a massive collection of flowers, balloons, flags, signs, and other arrangements were placed at the Johnson Space Center sign at the main entrance to the center. (NASA Image JSC2003-E-04037)

In many ways, that suffering extended beyond the fence of the center and extended out into the surrounding Clear Lake community. Within hours of the accident, people of the local community began to place flowers outside the JSC front gate. Balloons and American flags accented these floral arrangements, some of them arranged professionally and other in a more hand-crafted style, bound together with bits of found ribbon and tissue paper. In no time, a makeshift memorial stood at the entrance to our home of human spaceflight, and over the following days and weeks it continued to grow. Each day when coming into work, every employee of JSC was reminded of the sorrow this accident had wrought on our NASA-inspired community.

I had heard that various branches and groups across JSC had gathered at work early that week simply to talk through what they knew about the accident or had read in the news, ponder about the future of the Shuttle program (which was understandably unknown at the time), and more importantly share the emotional impact that the accident was having on them, their friends, and their family. JSC leadership encouraged these discussions and broadly offered the services of the center’s Employee Assistance Program, who in response set up a schedule of sessions to converse with all JSC organizations on coping strategies and set up a webpage to communicate to employees’ activities, services, and resources.

The pain that this event provoked was tangible. We all could feel it. For me, the deep sorrow I felt and the difficult questions the accident provoked caused the muscles around my stomach to tighten as if I had eaten some molded food. When not at work, I walked around in a daze, questioning whether I could have done something that could have prevented this tragedy, raised some issue, motivated someone in authority to take some action. I suspect I wasn’t alone in this self-recrimination, that there was an abundance of second-guessing going on. While at home I would stare blankly at the TV and unenthusiastically prepare my meals. I’d distractedly listen to music, which in any other circumstances would rivet me with inspiration but at that time simply passed through me like air through a mesh screen. Lost is a good description of how I felt within the first few days after the accident. Maybe ungrounded too or possibly unmoored. My trust in our ability to protect our astronaut crews was completely shattered.

Perhaps providentially, my time at work didn’t allow much time for grieving. As you’ll read in Chapter 5, not more than 24 hours after Columbia broke up, I was back at work, providing a conduit to an operations community who were thirsting for information. I was back in Mission Control attending the recently created NASA Mishap Response Team, formed late on the afternoon of February 1. The MCC had been released from its lockdown and the building was once again open for admittance. I was busy, consumed even, with my TA duties and responsibilities.

Coming into work and supporting the accident recovery and investigation kept my mind off these musings of inadequacy and offered me the impression (accurately or inaccurately) that I was doing something to help. But those efforts only consumed 8 to 10 (sometimes 12) hours of the day, and when I was not at work, I no longer had a vessel in which to pour my suppressed energies.

A day or two after the accident I learned that a memorial service for Ilan Ramon, the Israeli astronaut who was aboard Columbia, was to be held at a synagogue only a few miles from JSC. The service was scheduled for 7:30 pm and the doors were open to everyone. I was not a member of the congregation—if fact, I had never been inside this synagogue—but it was a simple matter for me to attend, to personally grieve for the crew’s loss, and commiserate along with others in the Clear Lake, Texas, area.

So, I attended the service along with a colleague of mine from our EGIL flight control group. The weather was cool, but not terribly cold, just the normal mild chill of a Houston winter. When we arrived the parking lot was surprisingly full; so full that they were directing cars onto an adjacent grass lot, where we found a space and slid in between two pickup trucks.

We entered the building and were directed to the main sanctuary where the service would be conducted. To say the place was crowded would be a gross understatement—it was packed! Every seat in the pews had already been filled and the sides of the sanctuary were crowded with standing attendees. My colleague and I wandered to the back of the room and found a sliver of empty space between individuals who solemnly lined the rear wall.

Some of Ilan’s family, we were told, were in attendance that evening, sitting along the front row immediately beneath the dais. Resting on a wooden stand was two large portraits of Ilan—one in his astronaut launch/entry suit and the other capturing him in casual clothes brandishing a simple, confident smile. Ilan and his family had temporarily moved to Houston while he was training for the STS-107 mission, and they had rented a house in Clear Lake. There they served not as ambassadors of their country but as simply members of the local JSC community. This synagogue welcomed them as they would any member of the diaspora and in this place, I suspected the Ramon family found both familiarity and, perhaps, even a tad of solitude.

The service itself was, to me, a traditional Jewish service, replete with customary ritual and prayer. But when they got to the mourner’s kaddish, the part of the service when the congregation pays homage to those ancestors and loved ones who have passed, the ceremony had an obvious extra significance. There were sniffles and quiet cries, short bursts of tears, and lots and lots of hugging.

Beyond the religious aspects of this service, the thing that struck me was the number of people in attendance who were not part of this congregation and, very likely, not Jewish at all. On this day, it made absolutely no difference at all. These people, a hundred or more mourners, had gathered in common purpose to pay respects to a fallen comrade, and to commune so that they could share those feelings of loss with each other. At one point, I recall, everyone was requested to reach over to their neighbor and grasp hands. We formed a chain of humanity that spiraled and coursed through the room, down and back the pews and around the perimeter of the sanctuary. I could almost feel an energy travelling along that chain from person to person, an energy of sincere compassion and connection. The service was, if nothing else, a very human experience.

Then on February 4, a Tuesday, much of JSC gathered for their own remembrance ceremony. President Bush was in attendance as expected so security was extremely tight. The JSC campus seen from, say, a low flying airplane, has a logical configuration with buildings situated organically around two large ponds (we called them duck ponds although they were filled with large Koi fish and rarely hosted any ducks). Many of these office buildings, including the nine-story Building 1, the three-story Building 4-North (where my office resided), along with others such as Mission Control’s Building 30, were all constructed when the center first opened in 1964 as the Manned Spacecraft Center. Designed to meet an aesthetic paying homage to the space age, the architecture was reminiscent of science fiction depictions of a utilitarian government estate, using white concrete and glass for facades and steel and aluminum frames supporting the interior.4 The duck ponds were tastefully shaped, each covering maybe a quarter acre of open space, amorphously resembling large amoebas with sloped edges of small rocks and pebbles. The ponds separated our Systems Division offices in Building 4 and the MCC attached to Building 30, necessitating but a quick walk from one to the other. Whenever I was called to the control center I made this short trek, navigating around the duck ponds. I always found them an idyllic addition to an otherwise austere and pragmatic space center.

For this crew memorial ceremony, a perimeter of temporary chain-link fences was set up between buildings forming a supposedly impenetrable barrier around the central campus. Two entry points were established, resulting in twin choke points for those desiring admittance; but the security was necessary. NASA guards were positioned at both entrances to check everyone’s badges—as long as you had a NASA badge or one for a supporting contractor you were allowed through. In addition, the doors to every building were locked and, although I didn’t see any, I had heard rumors that snipers were positioned on a few rooftops to ensure the safety of the president of the United States.

I never heard an exact counting of the number of people who had assembled that day, but in 2001 JSC employed around 3,000 civil servants with an equal (or possibly greater) staff of contractors. Standing in the midst of that crowd, it certainly looked like all of them had shown up. Most people were dressed in “civies” —that is, not in work clothing, as all non-critical activities at the center had been cancelled for the day. Our function instead was to surround ourselves with family.

We maneuvered through the thicket of people and wandered to a grassy area not far from the window of my first floor Building 4 office which overlooked the duck ponds. I could see toward the other side, closer to Building 12, a small stage had been set up along with the familiar rectilinear podium used by the president whenever he gives a speech. A table covered in blue felt stood behind the podium, behind which sat the NASA Administrator Sean O’Keefe, some of the speakers, and a few other luminaries. Off to the side were positioned a couple rows of chairs where the families of the Columbia astronauts were seated. I thought it nice that the president and First Lady Laura Bush were seated with the families rather than with the NASA brass. I applauded whoever made that decision.

Figure 7. President George W. Bush and Laura Bush bow their heads in prayer during a memorial service at Johnson Space Center, February 4, 2003. Sitting with the President are Commander Rick Husband’s wife, Evelyn, and children Laura and Matthew. (White House photo by Paul Morse)

Each of us were handed folded paper pamphlets containing the agenda. The service would have four speakers. First up was Harold Robinson, Chaplin of the US Navy, to give a benediction. Next up was NASA Administrator O’Keefe, followed by Kent Rominger, at the time the chief of the Astronaut Office. The speeches were heartfelt, respectfully mournful, and laced with personal reflections of the crew. They also were laced with missives on the importance of NASA’s mission and on the ideals of spaceflight and human exploration. Rominger, who went by the moniker “Rommel,” offered a short but tremendously personal speech. He began by saying:

The world lost seven heroes. We lost seven family members. Coping with our tragic loss is going to be extremely difficult, but remembering the unique qualities of each and sharing our special memories will help us heal. That’s what I’d like to do today.

And continued with:

Every Shuttle crew form into a family as they go through months and, in the case of STS-107, years of training.5 But the STS-107 crew grew particularly close. They were a generous and caring bunch with a great sense of humor. As a matter of fact, they would bake birthday cakes for their instructors. They traveled around with a mascot, a small toy hamster that sang the “Kung Fu Fighting” song. The referred to the crew secretary as “The Great and Powerful Roz.” And they convinced her that if she would keep candy on her desk, she would see a lot more of them.

Figure 8. President George W. Bush addresses those assembled for the memorial service at Johnson Space Center. (NASA Image JSC2003-E-05938)

At the astronaut Christmas party this last December, the STS-107 crew and spouses were there in rare form. They had paper crowns; they were sporting STS-107 tattoos and truly were the life of the party.

As a matter of fact, anybody that passed within ten feet of their table was encouraged, if not physically helped, into a chair and immediately branded with an STS-107 tattoo. And at the time, I didn’t openly thank them for the tattoo square in the middle of my forehead, but I was very proud to be labeled one of them.

Rick, Willie, Mike, K.C., Laurel, David, and Ilan, I know you’re listening. Please know that you are in our hearts, and we will always smile when we think of you.6

When Rommel sat down, President Bush rose and walked to the podium. He was maybe 50 yards distant from where I stood but still within my view, wearing a dark suit, white shirt, and blue tie. There was no jumbotron or large screen to project his image, but there were loudspeakers, so his words were clear and audible. Although the president wasn’t considered “family” in the way Rommel or the throngs of JSC employees were, his tone was respectful, patient, and reserved. The president opened with:

Their mission was almost complete, and we lost them so close to home. The men and women of the Columbia had journeyed more than 6 million miles and were minutes away from arrival and reunion.

The loss was sudden and terrible, and for their families, the grief is heavy. Our nation shares in your sorrow and in your pride. And today we remember not only one moment of tragedy, but seven lives of great purpose and achievement.

To leave behind Earth and air and gravity is an ancient dream of humanity. For these seven, it was a dream fulfilled. Each of these astronauts had the daring and discipline required of their calling. Each of them knew that great endeavors are inseparable from great risks. And each of them accepted those risks willingly, even joyfully, in the cause of discovery.

Bush then went on to say a few words about each crew member. He offered tidbits of their wisdom, recollections from their childhood and family upbringing, and especially the intense love, passion, and dedication they all brought to the endeavor of human spaceflight. He reflected that,

Our whole nation was blessed to have such men and women serving in our space program. Their loss is deeply felt, especially in this place, where so many of you called them friends. The people in NASA are being tested once again. In your grief, you are responding as your friends would have wished—with focus, professionalism, and unbroken faith in the mission of this agency.

And then, perhaps for the benefit of the JSC audience or maybe for the benefit of history, the president recounted a conversation Columbia astronaut David Brown had with his brother prior to the launch of STS-107. David Brown’s brother asked him what would happen if something went wrong with the mission. Bush stated that David replied, “The program will go on” and then, in his own words, added “Captain Brown was correct: America’s space program will go on.”

Finally, in a rhetorical swirl, Bush concluded,

This cause of exploration and discovery is not an option we choose; it is a desire written in the human heart. We are that part of creation which seeks to understand all creation. We find the best among us, send them forth into unmapped darkness, and pray they will return. They go in peace for all mankind, and all mankind is in their debt.

Yet, some explorers do not return. And the loss settles unfairly on a few. The families here today shared in the courage of those they loved. But now they must face life and grief without them. The sorrow is lonely; but you are not alone. In time, you will find comfort and the grace to see you through. And in God’s own time, we can pray that the day of your reunion will come.

And to the children who miss your Mom or Dad so much today, you need to know they love you, and that love will always be with you. They were proud of you. And you can be proud of them for the rest of your life.

The final days of their own lives were spent looking down upon this Earth. And now, on every continent, in every land they could see, the names of these astronauts are known and remembered. They will always have an honored place in the memory of this country. And today I offer the respect and gratitude of the people of the United States.

May God bless you all.7

President Bush turned to the families seated behind him and silently acknowledged their presence and their sacrifice. Before he could return to his seat, the screech of four sleek NASA T-38 jet aircraft roared within earshot. The high-performance planes came arcing in from the distance and flew directly overhead at what seemed like no more than five hundred feet altitude. They made a low pass, circled around once, and came in again for a second pass. Just as they streamed over our heads, one of the jets peeled off and rose vertically, powering high into the sky and leaving the other three to continue their course out over Galveston Bay. This “Missing Man” formation is a common military salute to lost comrades and the fact that it was performed by four NASA astronauts made the spectacle even more poignant. I turned to my colleague and gave her a tight, unrestrained hug. I wasn’t alone. Such displays of emotions were completely acceptable between comrades on that day.

The Space Shuttle Program was, for better or worse, an endeavor firmly rooted in technological and industrial capability. It necessitated hundreds of companies to design, build, and supply thousands of components that collectively numbered in the millions of parts. These parts were certified to specified tolerances, assembled per reviewed procedures, maintained to accepted standards, and operated like a finely constructed watch (which, in many ways, the Shuttles were). And yet, for all this precision and as much passion those designers, builders, maintainers, and operators had for those wonderful, elegant machines, it was not the loss of the vehicle that we grieved for but the crew. It was the loss of seven human lives that made us cry, turned us sorrowful, and strove us to yearn for the salve that only community can offer.

Life Lesson for Engineers, Managers, and Leaders

Community provides comfort and can salve the pain of loss.

Grieving is hard. During times of loss, it’s common for people to feel alone, isolated, left only to themselves to find a path forward in their search for a place that doesn’t hurt quite so much. It’s true during such times that some people seek refuge in isolation, where they have the simplicity of only themselves to worry about. But the Columbia experience taught me a different lesson. Refuge is often necessary when recovering from loss, but refuge doesn’t need to equate to isolation. It’s possible to seek refuge in community, rather than in isolation.

Why is this important for engineers, managers, and leaders? It’s common for engineers to compartmentalize the ways we feel about a situation. Feelings are not something we bring into design reviews, and they don’t play in to how we certify hardware or software, nor are feelings recognized in hazard analyses or reliability reports. Managers are often surrounded by volumes of data and information, sometimes an avalanche of such, and these data can be invaluable in the course of making decisions. And leaders, for their part, may be expected to rise above their personal feelings and consider the larger good, the good of the group.

All of that is true. But it is nevertheless inescapable that engineers, managers, and leaders are in fact human and can be affected by emotions. The comfort we derive from others can be vital to our success.

I’ll give you an example. Recently, one of our NASA centers lost a colleague and friend when, tragically, the person took his own life. As word spread quickly, the impact this had on the center and within his project was both quick and devastating. Work on the project stopped for a while. The center had to contend with a workforce who had not just unexpectedly lost a colleague and friend but were struggling with the question of whether they could have done something—anything—that might have prevented such a sad eventuality. After the suicide a number of my engineering colleagues reached out to me, partly to inform me of the news as I had also worked with the person, but more so, I believe, simply to share their sorrow.

I have found that people, including engineers, managers, and leaders, often find comfort in community. They discover that pain doesn’t have to be borne alone, that sharing grief is a healthy refuge and offers a constructive path to healing. Doing so doesn’t eliminate the pain—maybe nothing will—but can ease it so that it’s possible to bear.

Nothing about loss is easy. Walking, crying, yelling, screaming—it’s all part of what we do. We cope with strength and hope and try our best not to crumble. It’s not easy and sometimes we break a little. That’s okay. It’s… human. And so are engineers, managers, and leaders.

1 JSC Office of Public Affairs, e-mail to JSC employees, Feb. 2, 2003.

2 Lt. Gen. Jefferson D. Howell (retired center director), e-mail to JSC employees, Feb. 2, 2003.

3 Jon Harpold, e-mail to JSC employees, Feb. 3, 2003.

4 This architectural style is evidently referred to as postmodern Brutalist architecture. For more information, see Dana Marks, “Why Are there so Many Brutalist Federal Buildings in Washington?” National Capital Planning Commission, https://www.ncpc.gov/news/item/52/ (accessed Jan. 21, 2025).

5 Because of persistent manifesting problems and the priority of ISS construction flights, STS-107—an Earth-orbital science mission—kept being pushed out of line. A Shuttle crew normally gets assigned to a mission a year or year and a half prior to launch, but the STS-107 crew had been a unit for much longer than standard.

6 Space Shuttle Program News Commemorative Edition, February 2003, Space Shuttle Program 2000-, Box 156, Record Number 18239, NASA Headquarters Archives, Washington, DC.

7 Ibid.

Chapter 4

A Call from the Boss (Part 2)

In God We Trust; All Others Bring Data (But Don’t Ignore Your Gut)

And that was it—they were truly gone. In what would be little more than an instant to most of us, seven brave and dedicated explorers had been lost. Over the course of my MOD career, I had the opportunity to work with numerous astronauts; some assigned to crews and others back in rotation awaiting their next assignment. Many were colleagues and a few I considered friends. Two or three were neighbors. But for whatever reason, due to the vagaries of crossed paths or other reason, I didn’t get to know the STS-107 crew well. We just ran in different JSC circles, I guess. Their loss was no less tragic to me and the impact of their sacrifice profound, but I can’t say that I had a close personal connection to any of that septet of heroes.

On the other hand, I had a very close connection to the ship on which they rode. Columbia was the mother ship of the Space Shuttle Orbiter fleet, the original reusable space vehicle. My association with her traces back to the days before coming to NASA as a flight controller. On April 12, 1981, the day that astronauts John Young and Robert Crippen ventured forth on the first Shuttle mission—STS-1—I was a high school junior. I recall my algebra class teacher pausing his lessons and asking us students to gather around a handheld radio he had brought with him. At that time, the United States hadn’t launched an astronaut into space in six years and spaceflight launches were still a very big deal for the country. We arranged our chairs loosely around the radio and sat silently but with profound awe as a commentator walked us and all the rest of his listeners through the countdown. At the moment of liftoff, my classroom erupted in a thunderclap of applause, cheers, and raucous shouts of both students and teacher.

Over the course of my flight control career, I worked 13 of Columbia’s missions. The vast majority of them were dedicated to science and research, the only outliers being the launch of the Chandra Space Telescope on STS-93 (where an electrical short five seconds into the flight almost resulted in an abort) and the Tethered Satellite System (TSS) on the mission of STS-75. That particular mission holds a special place in my heart as I had just earned my “front room” certification, where only the most senior flight controllers resided. The mission was dedicated to deployment and retrieval of an Italian-designed and -built satellite which would be spun out on a reel like a fishing lure, connected to a kilometers-long tether. The purpose of this experiment was to demonstrate the feasibility and viability of conducting electricity through this crazily long tether as it passed through Earth’s magnetic field. If the technology proved itself, it could have the potential of producing nearly limitless supplies of electricity to orbiting spacecraft.

I was working the Orbit 1 shift and Chuck Shaw, the mission’s lead, was our flight director. The process of deploying the spacecraft was mostly uneventful—releasing clamps that held it firmly in Columbia’s payload bay for launch, removing the brakes from the tether reel, and firing small cold-gas jets of compressed nitrogen to nudge the satellite from its mooring. Once clear of the payload bay the jets were fired again and the satellite began to slowly back away from Columbia.

The flight control room was very quiet. The PAYLOADS flight controller would interrupt the silence periodically to provide status reports to Chuck, who would consume the information with professional aplomb. Chuck was the lead flight director for a mission a few years earlier when the tethered satellite system was first attempted—on STS-46—with the experiment ending ignominiously early when the tether got stuck after unreeling only a hundred feet or so. I have no doubt that Chuck was apprehensive and likely caging his responses to PAYLOADS carefully so as to not jinx this mission too.

Having departed the payload bay the satellite passed one kilometer, and then two. Always connected to Columbia by the tether, it continued to unreel, past four kilometers and then six. At some point orbital mechanics effects took over and the satellite fell behind Columbia as it was above and thus moving slower than the Shuttle. This produced a tension in the tether, but designers and planners had expected this and engineered the tether to accommodate those forces. Then 10 kilometers, and then 12.

I and my backroom spent our time focusing on our job monitoring Columbia’s electrical systems, planning the management of the onboard cryogenics the fuel cells used to generate electricity, and other normal, almost mundane, tasks. We really weren’t paying much attention to the TSS, but still listening in to the communication loops as this experiment was a first in spaceflight history.

Suddenly, on the Air-to-Ground loop, and with almost excited panic in their voice, the crew barked to Mission Control the following words, “The tether just broke!”

The flight director’s console was behind me. Although I didn’t turn around, I did hear a single, distinct and sharp expletive emerge from the occupant’s mouth. Chuck cut off his cursing, paused a few seconds, and then studiously got back to work, telling the CAPCOM to acknowledge the crew’s report.

With the failure on STS-46 on his resume as lead flight director I have no doubt that Chuck’s heart sank the very instant he heard the words that the tether had broken. In hindsight, though, I found it fascinating that he allowed himself a few seconds for an emotional outburst but then reigned that in and returned to the professionalism that the flight director position demanded. He still had a job to do, and he did it.1

Although it’s still early in this book to start retreading old territory, before leaving this section on A Bad Day, I wanted to return quickly to the phone call I received on February 1, 2003, on the morning of the Columbia accident. As recounted in Chapter 1, very soon after it was realized that Columbia was lost, I received a phone call from my boss, Systems Division Chief Rick Fitts, who informed me of what had just transpired, stating, “I hope it wasn’t the foam.” In many ways, that question from Rick was transformational for me.

I’ve mentioned the MER, the Mission Evaluation Room, in Mission Control. While that area within Mission Control2 had humble beginnings, today it’s a cavernous room with white walls and high ceilings, lined with computer consoles and voice loop stations, where the cadre of subsystem managers and staff engineers can monitor the performance and condition of their hardware while the Shuttle is in flight. These engineers don’t have the same broad responsibilities as does the flight control team to a) ensure the safety of the crew, b) protect the vehicle (as a national asset), and c) successfully execute the mission’s objectives, but they do carry important responsibilities, albeit with a somewhat narrower perspective. The job of a subsystem manager is to provide technical expertise in the details of their subsystem of responsibility and to manage those components as critical assets of the program. For example, the fuel cell subsystem managers who I dealt with from the EGIL console had over two dozen fuel cells in the program’s itinerary. At any one time we had as many as four Space Shuttles capable of flying (when Challenger was lost, Endeavour was created to replace it). Each Orbiter needed three fuel cells, which if all installed would constitute 12 from the subsystem manager’s inventory. But fuel cells were commonly removed and replaced due to in-flight performance issues, upgrades, refurbishment, and other reasons. If only 12 were available within the entire inventory, then serious issues would arise if one should fail and need to be removed from the fleet for diagnosis and/or repair. These subsystem managers then had responsibility for oversight of their entire inventories—those that were installed on the vehicle, those that were back at the vendor for repair, and even those placed into ready-storage awaiting their next chance to fly again.

During missions, the subsystem managers would come into the MER on a daily basis and assess how their hardware was performing and whether they would need to note anything that could affect that component’s ability to perform its next mission. They would scour through the telemetered data—real time or recorded—and watch their hardware like protective parents. Sometimes a condition in flight might necessitate the component being removed once an Orbiter was back on the ground, and the subsystem managers would be responsible for those dispositions.

When their hardware failed or malfunctioned on-orbit, the subsystem managers would zip over to the MCC (if they weren’t already there) and diagnose the discrepant part as best as they could based on the sometimes-limited telemetry from the vehicle. We had pretty good insight on Shuttle components—instrumentation, sensors, measurements, readings, and so forth—but it was still less than what might have been available on a test stand or laboratory on the ground. Still, these systems experts were always called upon to weigh in on whether it was safe to proceed with the component’s operations and how, if necessary, those operations should deviate from normal in order to achieve that Shuttle mission’s objective. They knew the constraints on their hardware but given the cost and risk of simply getting to orbit, sometimes they would push the limits knowing that once back on the ground the hardware could be removed for detailed failure analysis.

One entered the MER through a door on the third floor of the MCC. The room extends far to the left and right, less so in the direction of depth. Once through the door, the opposite wall is not too distant, holding a broad American flag hung next to mission clocks counting mission elapsed time or a countdown sequence to the next critical milestone. The aesthetic of the windowless room was functional, creating a stuffy and professional atmosphere that replicated the demeanor of the engineers themselves. No one seemed to mind.

Figure 9. The Shuttle MER, or Mission Evaluation Room. (NASA Image JSC2004-E-50827)

Walk across the room and you’d intersect the MER manager’s console. While not as hierarchical or commanding the authority of a flight director, the MER manager still was a nexus of the room, maintaining an awareness over of the entire vehicle while receiving inputs from all the subsystem managers. The MER manager spoke for the room when a single voice was required, were responsible for approving the shift’s anomaly list, and held other managerial duties that kept the cadre of subsystem managers as a cohesive unit.

On the wall above the MER manager’s console was a placard or sign that, in one form or another, had graced the MER going back to the advent of the space program in the 1960s. Initially credited to management guru W. Edwards Deming, the eight-word phrase constituted the MER’s entire credo. It was a sacrosanct expression of engineering philosophy that encapsulated the basic, foundational philosophy of this technical community. Embossed in wood, painted white with black letters, were the following words:

In God we Trust, All Others Bring Data.

Rarely has any short sentence better codified the NASA engineer! It’s almost a manifesto for how we conduct engineering at NASA. In diagnosing malfunctions and determining an appropriate course of action, in understanding why a piece of hardware is operating flaky (that’s a technical term), and in making decisions affecting the vehicle or the crew, whether in-flight or after the mission is over, data is the necessary component. We are conditioned not to base decisions on guesswork, supposition, or analogy. “Gut feel,” we are told, doesn’t enter into the equation and we certainly don’t allow fear or other bias to motivate our technical decisions. Rather, we rely on data, on hard, demonstrable fact, to guide the decisions we make.

After the Columbia accident, there was rampant speculation around the potential that national assets (national security assets, to be specific) could be brought to bear to attempt to ascertain if Columbia’s wing was damaged. We knew insulation foam had been shed from the ET during ascent and struck the left wing, but for reasons explained below, we had no proof of damage. Interestingly, the question of calling in the use of national assets never came up during any discussion I attended with the Orbiter Project Office during the flight of STS-107, so I surmise that these discussions occurred behind closed doors.

Regardless, what I do know is that while Columbia and her crew were in space, we knew from ground tracking cameras that a large piece of foam (estimated at the time to be between one and two pounds) had struck the Orbiter somewhere around the left wing and had shattered after impact, leaving a spray of particulates in the spacecraft’s wake. We knew precisely when this occurred, and using the same imagery some experts were able to calculate the errant foam’s trajectory, velocity at impact, and even offer a rough idea of the rate at which this foam was tumbling. There were uncertainties in these estimates, of course, but even that could be calculated and caveated on presentation charts.

The early 2000s was a time in which digital photography and videography was just making its presence known. Prior to this, the cameras that tracked a Shuttle’s launch and ascent into orbit utilized celluloid films. After launch the film would need to be processed—a careful and sometimes laboriously painstaking task—and then be thoroughly analyzed for the appearance of anything anomalous. As long as the cameras were properly focused and could track the rising vehicle, the insight this ground-based imagery provided was rather good. For example, in this imagery of every Shuttle launch, if you look closely, you can see the Orbiter’s body flap—a control surface beneath the three main engines—actually flutter. The body flap doesn’t do much to control the vehicle during launch—those forces are primarily driven by the twin solid rocket boosters (SRBs) and gimbling of the three main engines—but the body flap does contribute some control authority.3 But during launch the imagery was often good enough to reveal the body flap shuddering under the forces of launch and aerodynamic pressures. At the time of STS-107, NASA was just making the transition from film to digital technologies. Not only did these new cameras have improved resolution, but because of the absence of film that required processing the data were made available to engineers much, much sooner. Rather than this important imagery being made available no sooner than the second or third day of a mission, now it was available almost right away.

I don’t remember which day of the STS-107 mission it was—sometime early in the mission, I suspect—but I do recall at one of the daily OPO meeting in the MER conference room we were shown the launch day imagery, and that imagery clearly revealed the foam strike. The room was crowded as normal, packed with management, technical managers, and other engineers (all the subsystems managers attended the daily meetings whether their hardware was being discussed or not). I sat along one side of the large main table, with countless others filling the seats along the edges of the gray-walled, high-ceilinged meeting room. As the videos from the ground trackers were projected on the large screen at the front of the room, we all watched captivated. It was not unknown for insulation foam to strike the Orbiter during ascent, but to actually watch it happen brought a visceral reality to the situation.

Each individual image in the sequence accounted for only a fraction of a second, but at the velocity the Shuttle was moving, in that fraction the foam would have moved 10 or more feet each frame. The entire conference room sat transfixed as we watched the debris liberate from its originating point near the ET’s bipod ramp, cascade downward (as the Shuttle accelerated upward), spin in accordance with aerodynamic laws, move precipitously toward the left wing and then go out of sight. The next image showed nothing, as if a cinematographer had inserted a pregnant pause just to increase the tension. Then, two images later we could see a foggy spray of debris exiting from beneath the left wing and fall back into the vehicle’s slipstream.

Critically, the ground imagery did not capture the impact itself, which was obscured by the left wing, nor could it precisely ascertain the location of the strike. All we knew was that this tumbling piece of foam approached the left wing and then went out of sight before, some frames later, it reappeared again transformed into a cloud of pulverized insulation. We didn’t have many specifics, but imagery is imagery, and the foam impact was undeniable.

It has been thoroughly accounted in other histories, such as the Columbia Accident Investigation Board Report, that while the STS-107 mission was in flight, concerns were raised about the potential damage this impact could have imparted to the Orbiter. In fact, a Debris Assessment Team was formed to assess this question. As recounted in a Wikibooks submission titled, “Professionalism/Rodney Rocha and Columbia”:

After analyzing the initial images of the foam collision, the engineers of the Debris Assessment Team determined that they needed more images; they couldn’t make proper calculations and assessments without more information. As co-chair of the Debris Assessment Team, Rodney Rocha had serious doubts about the safety of the flight and notified several others about his concerns. However, without sufficient data to prove that his concerns were legitimate, he experienced difficulty in validating his fears to his superiors.4

Rodney was at the time the chief engineer for the JSC Engineering Directorate’s Structural Engineering Division and chaired the Space Shuttle Loads and Dynamics Panel. A quietly unassuming but enormously dedicated career engineer, Rodney knew his chops when it came to the Orbiter’s structure and its inherent limitations.

During the mission I can recall conversations at the OPO meetings about uncertainties over the potential of damage on the Orbiter. Rodney was there. The most specific uncertainty was the location of the impact site, which was not revealed by the launch imagery. Uncertainties also revolved around the impact velocity and the size of the shed foam, but the impact location was the one that garnered the most attention. Could imagery have been obtained in flight that could have helped better quantify the potential for damage? Well, this necessitates a bit of a diversion from the story.

Each the Shuttle Orbiters was able to accommodate a long robotic arm called the Remote Manipulator System, or RMS, developed and provided to NASA by the Canadian Space Agency. The RMS was attached to the Orbiter on three releasable pedestals (at its elbow, wrist, and tip) mounted along the inner edge of the port side of the payload bay, and a fourth that was permanently attached to the arm at its shoulder. With the RMS, a Shuttle crew could extract large payloads from the payload bay (such as satellites), deploying them into space and even retrieving them for return to Earth. The RMS was used extensively during spacewalks, or Extra Vehicular Activity (EVA), maneuvering spacesuited astronauts to various points in and above the payload bay. The arm was an invaluable tool during the repair and maintenance of the Hubble Space Telescope and was critical to the success of construction of the International Space Station. Had an RMS been flown on STS-107 it would have been possible to deploy it, crane it over the left-hand side of the payload bay and, if not obtain views from the underside of the lefthand wing, at least get close enough to the leading edge to possibly resolve damage.

But STS-107 had a weight problem. Columbia, the first Shuttle, was the heaviest Orbiter of the fleet. Those Orbiters constructed after Columbia—Challenger, Discovery, Atlantis, and Endeavor—all instituted lessons from the construction of Columbia that allowed for efficiencies in their design and weight to be reduced. But Columbia herself was cumbrously heavy (which, coincidentally, was why Columbia never was assigned a mission to the ISS—it was too heavy to get to the space station’s orbit.) In addition, the payloads which Columbia carried—a double SpaceHab science laboratory and a large cryogenic pallet which provided consumables for STS-107’s extended, two-week-long mission—were one of the heaviest payload compliments ever to fly on a Space Shuttle, possibly the heaviest. To accommodate all this weight and still be able to get off the ground, certain compromises had to be made on this mission and one of those was to remove the RMS.

Could the crew, then, have simply looked out the window and seen any damage to the left wing? Unfortunately, no. An image taken out one of the two aft-facing windows from the crew cabin reveals that the area was obstructed by the side of the payload bay, frustratingly remaining out of view from the crew.

Figure 10. View from inside Columbia’s crew cabin taken during STS-107, reveals the area of the left wing thought to have been damaged was obstructed by the side of the payload bay. (NASA Image S107-E-05359)

After the accident I became aware that some imagery was taken of Columbia in orbit. This was from the Air Force Maui Optical & Supercomputing (AMOS) site, operated by the Maui Space Surveillance Complex in the Hawaiian Islands. On January 28, four days before Columbia returned to Earth, AMOS trained its infrared cameras on Columbia as the ship flew overhead. A series of images were taken, but the attitude Columbia was flying was with its payload bay facing Earth (this is the warmest attitude for an Orbiter to fly). As you can see from the photo below, once again, all areas on the left wing near what was believed to be the impact site were obscured.

Figure 11. Showing Columbia through telescope. (NASA Image JSC2003-E-13226)

I don’t recall seeing any of these during the mission, but I suspect they could have been available to the Debris Assessment Team. Regardless, even if they were, none of this was proof that the crew was in any danger. Whether it was the launch imagery, photos taken by the crew and downlinked to the ground, or even the AMOS imagery, none of it offered any conclusive evidence of damage to the vehicle, critical or otherwise.

Rodney Rocha was, thus, left in a conundrum. His concerns, perhaps influenced by conservatism but certainly informed by a career’s-worth of experience, left him uncomfortable with the situation. But he had no data to prove that his concerns were valid. Even assuming the worst-case damage location, the tools available to the MER at that time to assess damage to the Orbiter were rudimentary with broad inherent uncertainties and, thus, not irrefutably conclusive.5

Rodney hailed from the Engineering Directorate and I from MOD. We certainly crossed paths through our mutual support to the OPO, but I never had a conversation with him about his concerns over STS-107, either during the mission or even afterward. I know from historical accountings that Rodney tried to request additional imagery, which needed endorsement and approval from SSP leadership but given their confidence in previous Shuttle damage history, their belief that it was more of a maintenance issue than a safety one, and the lack of proof of damage, his requests were denied.

In God We Trust, All Others Bring Data.

I have always believed that Rodney’s gut was sending him very strong signals at the time. Something was telling him that things were not all right, conditions in this case were outside of experience, and that although he couldn’t prove it, the crew might be in danger. That voice, I always imagined, was yelling inside of Rodney to do more, to seek more knowledge in order to reduce uncertainty. To get more data! I can completely understand that voice, the one that says that although I can’t prove that something is amiss, my experience, accumulated knowledge, and even my gut is telling me loud and clear that something more needs to be done, that what I can’t see is just as important than what I can.

Which brings me, finally, back to Rick Fitts admonition of “I hope it wasn’t the foam.” If you remember, the call I received from Rick was scarcely an hour after the accident had occurred. At that time there were no hypothesis of what went wrong, no plausible theories that could explain why Columbia was lost, no fault trees or failure analysis. All we knew was that a terrible accident had just occurred, and seven brave souls had likely lost their lives. Even the telemetry from the Orbiter was cryptic, showing the loss of temperature and pressure sensors along with a few discrete indications in the vehicle’s left wing, but no more. Certainly, loss of some instrumentation couldn’t lead to the catastrophic breakup of a Shuttle inflight. The vehicle was in control at the time communications were lost and there were no indications from the crew that anything was seriously wrong.

And yet, only an hour after the accident, something within Rick led him to speculate on the foam. Something … ineffable, caused him to focus on the foam lost during ascent 14 days earlier. It wasn’t a conclusion supported by irrefutable fact, or data, but rather by something that transcended those.

Life Lesson for Engineers, Managers, and Leaders

In God we trust; All others bring data (But don’t ignore your gut).

Now, allow me to make one thing clear before I go on. In this Life Lesson I am not advocating that you replace data with some instinctual quality that gives you messages. Data are the foundation of good engineering and management. Intuition alone is not rationale to assess risk. Gut feel should not be incorporated into technical analyses or probability assessments. Margins above requirements should not be derived based upon an unquantifiable fear. All of those products of engineering and management assessment—risks, margins, and more—should be supported by hard facts based on testing, analysis, inspection, or demonstration for engineers and historical project performance for managers. Risks should be assessed by what we know, and the likelihood or consequence informed by what we don’t (e.g., uncertainties). None should be measured by what we might suspect, guess, or fear. Doing so would not be good engineering nor good management, and I certainly wouldn’t advocate for that.

So, what am I advocating for? Simply put, use the data but don’t ignore your gut. In doing that, you don’t have to move a gut feel to the top of the agenda or eliminate all the data-driven paths to a solution. But a gut feel is still something that, while less quantitative, can still be relevant. In my experience, a gut feel is born by experience and the judgement and wisdom that is derived by that experience. There’s an expression I like to use; it goes, “I’ve been to this movie before.” It’s an expression that an event or action appears to be recurrent. While I may not be able to give you the exact time or place that the previous events happened, I know that they have and that I’ve seen this sort of occurrence in the past. That is, to me, the similarity between past and present events is unmistakable. Sensing that one has been to this movie before is not provable by fact, but it is something that I have learned not to ignore.

The only downside, of course, is that gut feel provides tepid tangible rationale for immediate action. Rodney Rocha’s experience is an example.

One other example of this lesson: If flight controllers had a bible, it’d be the document titled Flight Rules. We had one for Shuttle. Collectively, the collection of our Shuttle flight rules was about a thousand pages long, requiring a three-inch binder to hold it all together, and was separated into the separate technical disciplines (e.g., electrical, environmental control, propulsion, in addition to sections on flight dynamics and other operations). Each flight control console position had a bookcase next to it that included all of our procedures, checklists, schematics, and other regalia necessary to perform our job. Flight Rules was normally on the top shelf.

Flight Rules guided almost every aspect of Shuttle flight operations. It dictated the actions, conditions, and cautions, that mediated everything that could or might happen during a Shuttle mission. Flight Rules described actions to take when a component malfunctions and communicate the boundaries that limit normal operations. It informed both to the crew and the flight control team the constraints and limitations of the hardware, from launch to touchdown. We operated the Shuttle within the boundaries of Flight Rules and only exceeded those when conditions or scenarios demanded it.

Each section in Flight Rules might contain a few dozen rules, maybe up to 50. The format taken was an evocation of the rule itself, written in CAPITAL LETTERS, that described the condition and the action to be taken. Following that, in lowercase italics, was a set of rationale that justified and substantiated the rule. The rationale could be certification or test data, previous flight history, information from the component manufacturer, or other technical information that, collectively, justified the rule.

Before being included in Flight Rules, all proposed rules were discussed and codified through a Flight Rules Control Board, where new rules or modifications to existing rules were presented and richly debated. Occasionally, a proposed new or altered rule would be rejected because the justification was insubstantial or not supported by available data, but more often than not rules got approved for inclusion in the document because there was adequate data to justify itself.

Sometimes, however, there simply isn’t much data to provide. Sometimes, a rule makes sense on its own merit, even without data, and from an operational viewpoint it simply makes sense. I myself authored one or two of those for my EGIL discipline. In these cases, the documented rationale was simple and enormously concise. Listed within the rationale under the new rule was an astonishingly short phrase that yet still conveyed a volume of information. In those cases, the rationale was very simply: Engineering Judgement.

While Engineering Judgement might not win a case in court, sometimes it’s all that is necessary to justify a flight rule. We humans are endowed with enormous gifts called sentience and intelligence. We can process through fantastically complex problems and, given experience and judgement, derive solutions. That processing may not be precisely repeatable, and those solutions may vary between engineer and engineer, or manager to manager, influenced by the particulars of our experiences. But even with that variability and imprecision, that voice in our head, or in our gut, shouldn’t be ignored. I’ve learned to listen to it and to trust it. I know that that voice is not infallible nor guaranteed to be correct, but I’ve found for me that more often than not it’s right on target. I’ve learned to use that voice in my decision-making process, and while that voice doesn’t make decisions for me, I use it to help inform me and guide me when decisions are made. Most importantly, I don’t ignore it.

Leaders are forced to deal with this all the time. Experience and their subconscious realization of patterns can create warning signals that something is not quite right. The challenge for leaders tends to be when and how to act on their gut feeling in such a manner that produces actionable results. When this occurs, it may not be a call for choosing a different path, but it should at least raise a flag that more information is needed. That information may be more data, but for leaders it may also be gathering more perspectives.

Another point for leaders: while engineers and managers should listen to their gut, leaders need to help their engineers and managers when those situations occur. As savvy and resourceful as engineers and managers are, it is still possible for them to run into roadblocks or challenges in locating the data they need. Leaders and decision-makers should not simply rely on their engineers and managers to figure out the path to more information on their own. Sometimes they need assistance. When and how to apply such guidance can be a subtle skill for decision-makers to recognize, but doing so can often provide the path to success.

1 The next 48 hours were also a fascinating time of hectic brainstorming and planning as the entire Shuttle program, including those us flight controllers on console, worked feverishly to ascertain if there was any way to retrieve the wayward TSS and return it to Earth. In the end NASA felt there was too much risk of the 12-kilometer-long tether wrapping itself around the Orbiter and potentially preventing the crew from returning home. It would have been neat to have tried, but I can’t argue with the logic of the decision.

2 In authoring this book, I learned something. I had assumed the MER had always been located in the MCC, only for an insightful and knowledgeable peer reviewer to correct me. Evidently, the MER first appeared in Building 45—an office building a short distance from the MCC—and only moved to the MCC in 1988. I didn’t know that!

3 It is primarily during reentry where the body flap comes into its own, providing a significant contribution to the vehicle’s ability to maintain control, along with the wing’s elevons.

4 Wikibooks contributors, “Professionalism/Rodney Rocha and Columbia,” Wikibooks, https://en.wikibooks.org/w/index.php?title=Professionalism/Rodney_Rocha_and_Columbia&oldid=4429262 (accessed December 6, 2024).

5 One tool, a spreadsheet-based tool named Crater, was available to the MER team. But Crater was never intended to be used for real-time, mission critical assessments.

Mishap Recovery

Chapter 5

The Mishap Response Team (Part 1)

In Times of Chaos, People Thirst for Information

February 2, 2003—a Sunday, the day after the accident—I decided to go into the MCC. It was a weekend, and I didn’t have any specific duties to perform, but the MCC had been released from its lockdown, and I was motivated to learn anything I could about the accident. But perhaps more so, I also wanted to help. I felt powerless just watching the news from home, neutered in my abilities to assist (which was a large portion of my integration-focused job). As the MOD representative to the Orbiter Project Office and the TA for the Systems Division, I wasn’t worried that my presence in the MCC would raise questions that day, but I also acknowledged to myself that we were only at the start of what would likely be a long investigation. I understood that the gears of that machinery were just beginning to turn, and not all the parts would be in place yet. The effort might lurch forward in fits and starts before eventually attaining a semblance of battle rhythm, but that yet lay in the future. Still, I thought I could be of assistance, even in some small way.

So, I came into work. I parked my car outside the MCC’s north entrance, walked into the lobby, inserted my badge into the card reader, and then passed through one of the three heavy metal doors that separated the outside world from the cloistered, windowless confines of the control center. I maneuvered past the two notoriously slow elevators and beyond a wall-mounted artist’s sketch from Apollo 16 (signed by John Young, the mission’s commander), up flights of stairs and on to the second floor. Once in the hallway of the “old” side of the MCC I had a choice—turn left, past the side-entrance to the FCR and across a partition to the “new” MCC, or go straight, beyond the main entrance to the FCR and on into the heart of the building. Exactly what I was expecting to find I didn’t know, but once in the hallway it was immediately apparent that this was no standard Sunday. Instead of empty corridors, the building appeared to be a hive of activity. People hurriedly rushed past me, short lines had formed in front of copy machines, and small groups huddled together, immersed in conversation. Many people were holding notebooks or thin stacks of paper. One hive of activity centered on an area at the end of the hallway, so I headed in that direction.

The hallway terminated at a large room with a single entrance. A placard outside the door marked the place as the “MCC Action Center.” This was the room where the Mission Management Team, or MMT, had used to meet to make strategic decisions while a Shuttle was in flight. When NASA constructed the new control center in the mid-1990s they also built a new room for the MMT to gather. This new room was significantly larger and more expansive, with an immense segmented table large enough to seat 30 participants. It was built with three large projection screens beyond the table and rows and rows of chairs to accommodate the engineers, support staff, and the myriad attendees who supported each MMT discussion. Sterile walls streaked with blue accents bordered clocks that displayed local time at various locations around the world, in addition to Mission Elapsed Time and countdowns to critical on-orbit events. A line of wooden plaques representing past Shuttle missions completed the décor.

The Action Center was, by necessity, much smaller and far more intimate, maybe a quarter the size of the newer MMT room. A large rectangular wooden table dominated the center of the room orbited by perhaps 15 to 20 cushioned chairs (adorned with brass rivets holding the fabric). One side of the room held three additional rows of cushioned chairs closely set together arm-to-arm, with boxes of electronics (desktop computers, audio/visual equipment) residing along the other wall. As with the new MMT room, wooden plaques in the shape of Shuttle mission patches ran along the top rim of ceiling. What the room lacked in elegance and comfort, it more than made up in being a testament to the history that had taken place within its walls.

As I wandered closer to the Action Center, I began to see familiar faces. I recognized Ron Dittemore, the SSP program manager, and Milt Heflin, chief of the Flight Directors Office. Ralph Roe, who managed the OPO, loitered along with a member of his team. MOD leadership and flight directors, members of the Shuttle Program Office, and other senior leadership hovered by the door to the Action Center, many holding Styrofoam cups filled with coffee. General Howell, the JSC center director, was there too. I saw Linda Ham, a former PROP flight controller and flight director who had chaired the MMT for STS-107, pop out of the room, hand something to an assistant, and then walk back inside. With so many senior leaders present, it was obvious to me that the meeting pertained to Columbia. I followed behind Linda, not knowing what to expect.

Eventually, other people wandered inside and took their seats. Many sat at the table with Linda at its head while the others, me included, contented ourselves by squeezing into the side chairs like packed sardines. Linda called the meeting to order and got right to business. Today’s discussion was to be less about figuring out what happened to Columbia and her crew and more about getting the NASA Accident Investigation Team (NAIT), as she called it, process rolling. As with all major NASA programs, the SSP had documented procedures for events such as these. Investigations are long, arduous affairs tapping the resources across not just the space agency but other government agencies as well, including the Federal Emergency Management Agency (FEMA), and even international entities when required. The procedures, codified in a Mishap Response and Contingency Action Plan, outlined the basic steps to be taken in the event of a catastrophic mishap. The formation of a NAIT was close to step one, and as these things go, this one was sure to be massive.

Figure 12. Linda Ham, chair of the STS-107 Mission Management Team (MMT) and the follow-on Mishap Response Team. (NASA Image JSC2003-E-09152)

Linda opened the meeting declaring that this wasn’t yet a full-blown recovery operation, but it was quickly becoming one. The initial focus on NASA’s effort was squarely on search and rescue; if there was any possibility that the crew survived reentry, no matter how improbable that may have seemed, every effort needed to be expended to find them quickly and provide assistance. Experts from FEMA and other government and local agencies joined the conversation via the telecom. Because Columbia broke up largely over Texas, the Lone Star State was also participating in search operations.

Major General Mike Kostelnik, the NASA Headquarters deputy associate administrator for Space Station and Space Shuttle, was up next.1 Calling in from Washington DC, he emphasized to the room that all of this was playing out on a world stage and that we need to get it right. He mentioned a press conference to be held at NASA Headquarters (HQ) with himself and his boss, former astronaut Bill Readdy, who was serving as associate administrator for spaceflight. They planned to focus on the long-term impacts of the accident on elements of NASA’s portfolio, such as the ISS. Then, later that day, SSP Ron Dittemore would follow with another press conference, this one discussing the more tactical aspects of “technical details and engineering.”

He mentioned that NASA Administrator Sean O’Keefe was on his way to the White House to brief the president and that Readdy (known by his moniker, “Reads”) was “on the front line” with the media. Kostelnik reported that President George H.W. Bush, the 41st president of the United States and father to the current president, was on-site to pick up spirits. Mike continued that the Office of Communication had been working with the Columbia families to generate a statement they planned to release to the public and that he has been amazed at the extent of offers for assistance that have flooded into HQ. Finally, he ended his report on a technical note, speculating how STS-107 would meld into the program’s Probabilistic Risk Assessments (PRA), which predict the likelihood of failures including the loss of crew and vehicle.

Speaking next was Dave Whittle, a long-time JSC safety engineer who was tasked with leading NASA’s formal Mishap Investigation Team (separate and more focused than the Mishap Response Team [MRT] that Linda chaired). The independent accident investigation board, likely the most visible investigative body to the public outside of NASA, hadn’t yet been formally established but astronaut John Casper had been assigned to get them up to speed with the technical aspects of the Shuttle system. Astronaut Dom Gorie had also been assigned as the MRT’s point of contact for the search and recovery of the crew and, if required, retrieval of any human remains, and astronaut Jim Weatherbee would lead that team (Weatherbee, he said, was reporting that the initial search area was 4 miles wide by 50 miles long, assisted by airborne, ground, and Global Positioning Satellite assets. The FBI was assisting but FEMA was running the show).

Dave continued by reporting that FEMA was active with the team, as was the Environmental Protection Agency (EPA), overseeing the collection and dispersal of the toxic elements and gases contained within the destroyed Orbiter. Initial operations were being established in the area between Ft. Worth and Lufkin, Texas. Amazingly, debris was already being brought in for NASA collection and temporarily stored in a hangar in Louisiana. As to the debris, Dave said the plan was to bag and tag each individual element for eventual transport to KSC.

Ron Dittemore then interjected that certain specific debris has been identified as of particular interest by Ralph Roe’s OPO team and Ron wanted to make sure that this debris was made immediately available for assessment. Dave acknowledged the comment and suggested Ralph forward to him the list of items of interest.

Dave then continued by mentioning that there were 12 FEMA teams already in the field, with the goal of eventually having as many as 30. Since the accident’s occurrence only 24 hours previously, over 80 people have gone to the hospital after touching debris, although none were admitted. The EPA had given NASA authority to remove debris that was considered nontoxic; however, the more hazardous items would necessitate special remediation. Over 500 National Guard members were to be deployed to assist the operations, most of them providing security. Dave concluded that, to date, the majority of debris sightings have come from local landowners in the thickets and piney woods of east Texas.

This sequential reporting then drifted to Ralph Roe. As the manager of the OPO, Ralph’s office was responsible for everything having to do with the Orbiter. The subsystem managers reported to Ralph, and the OPO controlled an annual budget of one billion dollars to manage the fleet of spacecraft. Standing at six foot three (maybe more) and with a deep baritone voice that sounded like boulders crashing down a hillside, Ralph could be an imposing figure. But even with his physical stature, Ralph was more often than not quiet and reserved, sparse in his comments and an open book only to his closest friends.2

Ralph began by stating that Problem Resolution Teams are the foundation of his organization. To that end they had stood up 14 sub-teams, all of which were already actively engaged. These included discipline expertise on data (as an example, he stated that some additional wheel well temperature data had already been found in the telemetry); a team on separated aerothermal effects looking at the reentry environment; one dedicated to vehicle reconstruction and another focused on the intricacies of math modeling; a team looking through all the KSC closeout photography of Columbia taken prior to the mission, even searching back to photography of her construction at the North American-Rockwell factory in Palmdale, California; one team developing a fault tree; one working to map a timeline of events via a three-dimensional representation of the Orbiter; a team looking at all historical launch debris events going back to STS-1; and finally a group investigating the integrated environment during reentry, with a particular focus on flight control.

Ralph was reading all of these from a piece of paper placed in front of him on the table. He then leaned back in his seat and offered, more extemporaneously, one last tidbit. Over the previous day and night, his team had been perusing all the downlinked telemetry. One thing brought to his attention was that during the latter stages of the reentry, before communications with Columbia was irrevocably lost, the telemetry indicated that two of the yaw thrusters on the forward portions of the Orbiter had been firing. That, in combination with the movement of the elevons, indicated that at that time the vehicle was fighting … something, trying to stay within control but already beginning to lose the fight. What Ralph meant was that just prior to breaking up, Columbia was still within control and pointed in the right direction, but forces were acting on her, causing the ship to react by firing thrusters and moving elevons in an attempt to counteract those forces (more on this in Chapter 12). What that told me, at a minimum, was that whatever happened to Columbia didn’t occur instantaneously but was a process affected over a span of minutes. The unspoken conclusion was that this was not an accident with no warning. I personally found that a sobering thought, and the room went completely quiet for a time. Others, evidently, had come to the same conclusion.

After the implications of Ralph’s information sunk in, four others followed with brief reports. Phil Engelauf of the Flight Directors Office (and the one who informed LeRoy Cain during the STS-107 reentry that numerous contrails were being reported, indicating Columbia had broken up), stated that MOD was working to provide virtual-reality visualization tools. Additionally, he opined that new insights might be available in the recorded telemetry by relaxing data error constraints. He explained that when telemetry comes into the MCC, algorithms check for error rates. If the errors exceed known constraints, then the data on all consoles goes static, meaning enough errors are present to suspect that the data is unreliable. If those algorithms were removed and the “raw” data was allowed to be processed, it was possible that additional information might be found. Phil said that work was underway. Lastly, he stated that reports had come in of sightings of debris coming off Columbia as far west as California and Nevada, and that MOD’s flight dynamics office was assisting in mapping all debris sightings with the vehicle’s trajectory.

Alex McCool, a longtime member of the Marshall Space Flight Center in Huntsville, Alabama, and the manager of Space Shuttle projects there (the Space Shuttle main engines, the external tanks, and solid rocket boosters) mentioned that they were working on fault trees.

Mike Wetmore of KSC reported that 50 to 60 people had already volunteered to assist in debris recovery and that KSC was working to designate a location to store the debris (the Shuttle Landing Facility, or SLF, hangar was being considered).

Astronaut Bob Cabana, the chief of the Flight Crew Operations Directorate, simply stated that the Columbia families were being supported around the clock.

Lastly, former astronaut Loren Shriver, now working for the program’s major contractor (USA, the United Space Alliance), offered his company’s training academy to facilitate the accident team members.

An hour or so of this discussion and after all the primaries had a chance to speak, the team broke and everyone filtered out of the Action Center. I recall being exhausted, partly because of the enormity of what I had just heard, but also in part because physical effort to hand-write copious notes of all that I had heard. These were the days before laptop computers, and while my office was equipped with a desktop computer, those were not particularly mobile. For this meeting I used the same log paper that we flight controllers used on console to keep track of events in our logbooks. Now, I found myself with about 10 pages of quickly scrawled notes. The effort made my hands cramp and the physicality of doing this for over an hour was immensely difficult.

But alternatively, I also felt exhilarated, because I now was in the possession of critical information that I knew my colleagues and supervisors in MOD would want to know. As that first MRT came to a close, I wasted no time in returning to my first-floor office in Building 4-North (I had to throw a bank of circuit breakers to turn the lights on in the building—remember, it was a Sunday), power on my computer, and draft up these notes. I then copied and pasted them into an e-mail and sent them out to MOD and Systems Division leadership. When I was content with the effort, I turned off my computer and left to go home to get some lunch.

Each day for the next few weeks I would return to the MCC’s Action Center to attend the MRT, scribe as much information as I could onto my lined flight controller log paper and send out the information to my leadership. This cycle of attending meetings, gathering pertinent information, and then communicating it was, in fact, a familiar part of my job as TA for the Systems Division. For example, as MOD rep to the OPO, each week I would attend their configuration control boards. Scheduled for each meeting would be an eclectic mix of topics, most pertaining to management of the Orbiter’s hardware—anomalies that occurred in flight or on the ground, certification issues, supply chain and vendor topics that affected the ability to keep the Orbiters flying, and so forth. Each was reviewed, discussed, debated, and dispositioned. Some of the agendas were light, containing only two or three topics, but others were daylong affairs lasting well into the late afternoon, covering as many as 8 or 10 different discussion items. The following day, now back in my office, I would take my notes, along with a stack of hardcopies of the presentations, and draft up minutes for my leadership and stakeholders in MOD. This sort of communication, accomplished for the purpose of awareness and integration, was a major component of my job. So, when I began doing the same for the MRT, it felt … well, comfortable and familiar. I was left with an inescapable impression that not only was I doing my job, but I was making a contribution to our agency’s response to this terrible accident. I felt like I was helping, even in a small way,

The pace of the MRT’s continued. On February 5, four days after the accident, NASA Administrator Sean O’Keefe joined the team via telecom.3 He led off by offering his thanks for the “extraordinary professionalism” shown by the teams and opined that “the typical chaos that walks hand-in-hand with these types of activities” was not in evidence. The administrator passed on word from President Bush that he was “most impressed” and concurred with the notion that respectful attention be focused on the families.

His voice then became somewhat more reflective, and he voiced his desire to make some observations and to reinforce a few of his previous statements. Administrator O’Keefe reiterated that our objective should be to allow facts to drive our conclusions. He suggested that we need to be very decisive and to lay out the facts but also warned against holding on to any favored approach, hypotheses, or theories and not to foreclose any direction. He directed us to, “Keep the aperture wide open.”

The administrator emphasized the need for NASA to maintain focus on the activities of the ISS crew presently orbiting overhead. He mentioned that he spoke with ISS commander Ken Bowersox (known by his astronaut moniker “Sox”) and wanted to pass along that Sox had replied “Don’t worry—we’re up here doing our mission and you know where to find us.”

O’Keefe concluded by informing us that he had received no limit on the amount of support from other federal agencies and left the discussion by adding “If you need extra help—ask!”4

Following the administrator, Ralph Roe came up for his daily report, and it was a lengthy one. He began with mentioned that his office had formed a new working group, one they titled the Vehicle Engineering Working Group, or VEWG, which would integrate and summarize the findings of all 14 sub-groups working specific investigation tasks. The VEWG was to meet Mondays through Saturdays at 3:00 p.m. in the Shuttle Program’s main conference room in Building 1 (e.g., JSC’s “Headquarters” building). I made plans to attend those in addition to the MRTs.

At the moment, Ralph reported, the work was concentrated on constructing a timeline of all the engineering data throughout Columbia’s last flight. As it stood, they were nearly 100 percent complete and although the product had been somewhat fluid over the last few days, it was stabilizing. Using the data from this timeline, their hope was to backward engineer a heat source and its location and attempt to reconstruct the loads on the vehicle. He added that beyond the failed sensors and indications of a left-wing drag on the vehicle, everything else in the telemetry showed nominal conditions. The team was assessing the entry profile and asking if anything was different on this flight. A fault tree was still under development, and they were attempting to link all possible causes of the accident. His team was also finalizing a list of recovered debris that could be critical to the investigation, the list to be forwarded to Barksdale Air Force Base in Louisiana, which was serving as a collection point for all the located debris.

Again, Ralph sat back and his next statements were offered unscripted. He lamented that, unfortunately, the agency had received dozens of hoaxes, many of them in the form of doctored digital images posted on the internet. Still, an equal or greater number of images have been received from private citizens of possible “real” debris on their land. He had been advised by the NASA General Counsel that these images need to be handled carefully as some citizens may be concerned about their property rights and worried that release of these images risked affecting the value of their land. The General Counsel, apparently, had insured Ralph that the issue was being addressed and that NASA wanted to make sure that confidentiality was maintained so that we continued to receive these kinds of images.

And so on. The cadence for these MRT meetings continued throughout February and well into the following month. I don’t recall getting the opportunity to take a break, but I’m sure it happened. In any event, breaks really weren’t relevant; this was a crisis moment, and we all were fully dedicated to discovering what had gone wrong with Columbia and preventing those circumstances from ever happening again. Rest, relaxation, leisure, pursuit of interests outside of work to take a breather from the grind of human spaceflight or to catch up on some sleep simply weren’t a priority in the face of investigating Columbia. If ever there was a time for all hands on deck, this was assuredly it!

So, the MRT pressed forward. MOD matched trajectories with the OPO’s timeline. MSFC developed fault trees including scenarios focused on the area of the ET bipod ramp where the foam was believed to have been liberated. KSC dispatched over 200 of their personnel into the field to search for debris and prepped the SLF Hangar to receive it.

Day after day the team met. Day after day NASA’s response got more organized. Day after day more information was gained, more debris was located, more images were being sent to us. We were still a long way away from understanding what brought about this tragedy, but the pieces were being put into place and the gears were slowly starting to turn. And as that process gained momentum, I would just as doggedly gather up the notes I took from the daily MRT meeting and send them out to MOD leadership.

And then, something very interesting began to happen. Five days after the accident, on February 6, 2003, I got an e-mail from astronaut Mike Fossum. He explained that he had been receiving my e-mail through the Flight Director’s office and appreciated that I had been taking them and sending out. As the lead CAPCOM (the astronaut in MCC who communicates with crews on orbit) for ISS Expedition #6, now in space aboard the ISS, Mike was the main source of information from the astronaut office to the expedition crew. He wrote to me, “Crew has requested technical information be sent to them to help them deal with the tragedy and their separation at this time. If you don’t mind, would you please add me to your distribution list? (These notes are excellent!)”5 Of course, I replied immediately that I’d be happy to add him to my distribution list.

As days turned into weeks, I received more requests. Other astronauts were added to distribution. Ralph and folks from his OPO asked to be included. I received requests from managers in the Shuttle Program, and even in the ISS Program. People at JSC not connected with Shuttle, or even with the center’s human spaceflight efforts, wanted information and evidently found solace in the notes I distributed. I even got a request from Jim Kennedy, the KSC center director.

Each time I hit SEND on these notes, the dynamics of e-mail networks were enacted. Beyond the distribution list that I used, I discovered that many recipients forwarded my notes to others, and then those forwarded them again, expanding into ever broader and broader networks. I honestly never really knew where and how far my notes got, whether within the boundaries of JSC or outside to other centers, or even outside NASA. Evidently, they even were sent into space!6

Life Lesson for Engineers, Managers, and Leaders

In times of chaos, people thirst for information.

I walked into the MCC Action Center on the day after the accident, not because I was specifically invited but because it was commensurate with my job (so I got in) and I wanted to contribute. But, more importantly, I also wanted to know what was going on. The knowledge I gained from that day’s meeting, and all the others that followed, allowed me to maintain a very close awareness of the events as they were transpiring. It kept me informed of both the activities and the thoughts of leaders and their direction to the teams following this catastrophic accident.

The hardest day for me was the day of the accident, not only because of the shock and horror it engendered but also because I was completely out of the loop, away from my colleagues, and left only to my guesses and fears. Those ambiguities inside an environment devoid of information made the situation worse for me. As one of the peer reviewers of this book put it, “Uncertainty drives fear; and elevates emotions above logic and analytics.” But as soon as I was able to get informed and raise my awareness, I was better able to cope with the tragedy. Knowledge was a prescription to the affliction of ignorance.

Given the requests I received during the days and weeks that followed, I suspect most people at JSC and elsewhere were sharing in that struggle. The notes I sent out were, I believe, a mechanism of personal recovery. Those people didn’t need to be in the Action Center themselves and hear the discussions; the information alone was sufficient. Those notes weren’t filled with expressions of courage, motivation, or wisdom, eschewing all of that for the raw technical details of a growing investigation. But they were a shield against ignorance, and that, I believe, was why so many found value in them. Knowing was synonymous with comfort.

This idea is true under normal circumstances, but especially true during times of extreme uncertainty. In such times, people—engineers included—thirst for information. Uncertainty simply compounds what is an already difficult situation.

I’ve spent much of my career doing technical integration at one level or another, from the smallest NASA group of flight controllers to the wide network of NASA centers as the chief engineer for aeronautics (my position today). My general tendency is to share information unless there is a specific, tangible, and justifiable reason to embargo it. Sometimes there is, and in those cases limiting or even withholding information is warranted. But in general, I adhere to a tenant of sharing what I have, or more specifically, what I know.

That practice takes on an even more critical role when we are facing catastrophe. The experience of Columbia made that even more apparent.

As engineers and managers, and especially for leaders, we are often privy to critical information. We may have access to and an understanding of events derived from this information. To me, ownership of that critical information comes with a responsibility. I’m certainly not suggesting including the entire population of the planet on distribution when we communicate information—that’d be silly. Or even every person in whatever organization we may reside in. But rather, we should always remain cognizant that the information we carry holds a special power in times of chaos, and that there is an aspect of responsibility in its possession.

1 Kostelnik’s, Whittle’s, Roe’s, Engelauf’s, McCool’s, Wetmore’s, Cabana’s & Shriver’s comments that follow are taken directly from my notes of the Feb. 3, 2003, Mishap Response Team. Steven Hirshorn, MRT notes, Feb. 3, 2003.

2 Ten years after these events Ralph and I would again cross paths when he was elevated to the position of NASA chief engineer and I to the job as chief engineer for aeronautics at NASA HQ, reporting directly to him.

3 From my notes of the Feb. 5, 2003, Mishap Response Team.

4 Ibid.

5 Mike Fossum, e-mail to author, Feb. 6, 2003.

6 I never did, however, see anything I wrote ever appear in the media, sourced or unsourced, whether on the ubiquitous NASA Watch website or other news or opinion outlets. Whatever the cause, I attribute that to the fact that most of us who worked at NASA took this accident personally. It impacted NASA as a family, demanding a certain amount of respect and reverence for the information. I believe it is possible that in the expansive network some of my notes did end up with media outlets, but if that occurred, I like to think that they respected the information as we did—as personal information—and didn’t publish it. I certainly don’t know that for a fact, but I do know that I never saw my words quoted or published in news or media articles.

Chapter 6

Debris Search in East Texas

When Disaster Strikes, People Want to Help

The majority of Columbia’s debris rained down over East Texas in a line that tracked from Ellis, just south of Dallas, passing through the towns of Nacogdoches, Hemphill, and St. Augustine, to a point southeast of Sabine, Texas, along the Louisiana boarder, and then on farther into that neighboring state. The swath was over 150 miles long and approximately 20 miles wide. Larger parts of the Shuttle like propellant tanks, avionics boxes, and pieces of Columbia’s structure were relatively easy to identify. But many if not most of the 84,000 pieces of debris that were eventually recovered—accounting for only 40 percent of the vehicle—rained down on these pastural lands as small pieces, particles, and even flecks of metal and other materials. The local weather radar actually captured much of the destruction as it rained down like a dispirited shower.

These areas of rural Texas and Louisiana, almost all privately owned land, consisted of scenic, rolling plains broken into rugged terrain, including the Angelina National Forest and the Toledo Bend Reservoir. A combination of piney woods and swamp bog, hot and humid in the summer and cool and damp during winter months, the most predominant feature of this part of the United States is the interlocking thickets of barbed shrubs that can make passage through these lands a harrowing experience.

Almost immediately after the accident, the Texas Army National Guard deployed their members to assist with the recovery. This initial cadre was then augmented with thousands of wildland firefighters dispatched from all over the country. And not least of which were the people of NASA, employees and colleagues volunteering their time and effort, in many cases risking injury and enduring backbreaking physical work that led some to utter exhaustion.

Figure 13. Radar data showing the plume of debris. (National Centers for Environmental Information)1

Calls went out within the Systems Division, the entire MOD and all over JSC and other NASA centers asking for volunteers to assist in the recovery efforts. The colleague of mine with whom I attended the memorial service to Ilan Ramon and who stood by me during the crew remembrance at JSC was one of the first to volunteer. As no Shuttle flights were being launched, we flight controllers focused all our efforts on Columbia and her crew’s recovery. I was, at the time, fully encumbered covering the MRT, the OPO’s various tiger teams and efforts back at JSC and I felt those tasks were also critical. The opportunity to volunteer for search operations was available to me but, busy as I was, I allowed it to pass me by. To this day there is a small part of me that still regrets that decision and worries that I failed to give my all.

Our division sent people to a number of locales. For example, our deputy division chief, Stan Schaefer, was posted to Barksdale Air Force Base in Louisiana and arrived there only four days after the accident. After arriving and setting up shop, he sent the following e-mail from his new temporary home:

Hi folks. I know you’ve been interested in what’s going on at Barksdale and I’ve been trying to find time to send you a note. I don’t have much time right now, but I want to send you a quick note. First, the picture in this slideshow labeled “wing” is really part of the body flap. The chewed-up tile sections is the result of SSME damage and is normal wear and tear.2 I believe it was shipped here to Barksdale last night, but I haven’t had a chance to get over to the hangar yet.

Things here vary between hectic and chaotic. Our job is to accept all of the debris reports and photos collected at JSC and then triage them and forward them to the field centers. We also work with the KSC guys to identify what we can. Most of the reports go to Lufkin and Carswell (Dallas) and the teams are deployed from there. Since we don’t have a field center in the West, we run all of the deployed teams for California, Arizona, and New Mexico.3 Over the last two days, we’ve had some great leads out there, but have yet to confirm shuttle debris.

I’ll probably be coming back Tuesday or Wednesday of next week after the relief team arrives. As you probably know, this effort is still in the growth stage with more and more effort being focused on West Texas, NM, AZ, and CA. Before I got here, I thought I understood the scope of the recovery effort, but I was just fooling myself. I’ve seen over a thousand photos and debris reports in just the last two days and I know I haven’t seen near all of them.4

But most volunteers got deployed into the wilds of east Texas to support the search teams. When they returned from the field after two weeks supporting the recovery efforts I was regaled of stories of an almost incomprehensibly difficult task. NASA personnel from both JSC and KSC were being sent into the area and available local hotel rooms filled quickly, becoming difficult to find. The NASA personnel and firefighter volunteers (many Native Americans from the western United States) who weren’t afforded the relative luxury of hotel rooms were forced to satisfy themselves by encamping in canvas army tents.

Their day would begin before sunrise when they would be offered a cup of coffee or perhaps a donut for breakfast and then meet their search group at a mustering station. Transport to the search grid would be provided where the day’s actual work would begin. The deployed searchers would line up in a straight line, separated by 6 to 10 feet, and walk methodically across the grid-like search area. If someone were to find an object—a sharp piece of metal, something that might be Shuttle tile insulation, or anything manmade ranging from possible Shuttle debris to simple refuse and trash—the location would be GPS marked and the object carefully removed, bagged, and collected. As mentioned, this rural area of east Texas was thick with prickly bushes and thorny flora, ripping both clothing and occasionally flesh with equal ease. Some of my NASA colleagues went through two pairs on of blue jeans in just the first week.

Figure 14. Shuttle debris being stored at Barksdale Air Force Base, Louisiana. The debris is collected and cataloged prior to shipment to the Kennedy Space Center, Florida. (NASA Image JSC2003-E-14383)

Figure 15. At Barksdale Air Force Base, members of the Mishap Investigation Team inspect and prepare debris for shipment to Kennedy Space Center. (National Archives Catalog ID 6633277)

This slow, painstaking march would continue throughout the entire day. Often, the vegetation was so thick that machetes were required simply to clear a path. As the sun arced high overhead and tracked back towards the western horizon, the team would call it a day and be transported back to the muster station. Dinner was mostly the fine cuisine from local fast-food restaurants and pizza joints. Exhausted, the NASA personnel would return to their hotels, shower off the grime accumulated on their bodies, and collapse into bed for what typically proved to be no more than four or five hours of sleep, knowing that the next morning they would need to awaken and repeat the process again.

It was hard for me to imagine what my co-workers were going through. A few, however, did send us dispatches from the field via e-mail, giving us a glimpse of their daily grind. One of the INCO’s (Instrumentation and Communication) flight controllers from our division, Steve Sides, sent the following missive:

Please let folks that will be coming up here know that some days can be very long. We follow behind the crews of 40 personnel as they look for shuttle parts. They ask us to determine it the part is shuttle or not. New folks will get a great education while working with the evidence room. I am completely pooped by the end of the day which starts at 6 AM with a briefing, followed by a trip to the work site. We usually get back to camp about 5 PM but sometimes it is a little later like it was today. We walk through some fields which are ankle deep mud but others are nice and dry. The logistics folks here provide some really nice waterproof boots along with a backpack, socks, 2 sets of Forestry Service uniforms and a pair of chaps. The chaps and boots are very important tools to wear in the field because some of the fields are covered with dense torn bushes. You might want to let folks also know that they should try to go by building 25 and try to pick up some stuff to give away to the crews they will be going out with each day. The NASA folks up here have some stuff to give out but that generally goes pretty fast. I have been giving the person that finds a confirmed piece of Shuttle debris a little thank you. It seems to motivate the troops well.6

Another flight controller from our division, this one from the DPS (Data Processing System) group sent out a more comprehensive summary, his focused as advice to anyone who may be inclined to volunteer for this physically difficult work:

The real deal on field deployment for debris search.

Mar 17, 2003

The following relates to Hemphill only.

Hotel situation: When you are selected for deployment you will receive a packet with pictures in it that will lead you to believe that you will be living in a tent for two weeks. 90% of everyone out here are US Forest service firefighters and they are the ones living in the tents.

You are responsible for booking your own hotel room for the duration that you are here. The bad news is that you need to book EARLY in order to get a room for 2 weeks. When I went I had less than one day’s notice and NO hotel chains had ANY rooms available. I ended up staying at a bed and breakfast that (apparently) no one else knew about. Also, there are very few hotels in Hemphill itself so you will most likely stay in Jasper (40 miles away) or San Augustine (20 miles away).

The Daily Routine: There is a 6:30 meeting at the ICP (Incident Command Post) that you will go to every morning. If you are living in Jasper (which most are) that is a 45-minute drive away… so you’ll get up around 5:20 am or so every day. At the 6:30 meeting, you will find out your assignment for the day (which team you are on, etc.)… and you do switch teams from time to time.

After you get your assignment, you’ll head out to the staging area and meet up with your group. You will get with your strike team leader and find out what grid you will be searching and get the game plan together. Then you’ll drive to the site and begin the search (more on exactly how you search later). You will search from about 8:00am to 5:00pm or so and will cover about 6–10 miles per day.

Then you’ll drive back to the ICP, take a look at the new debris that came in for that day, then drive home and get ready to do it all over again. I’ve heard that they will be implementing a mandatory 1 day off per 7 days for all USA (e.g., United Space Alliance, the Shuttle Program’s main contractor) and NASA employees but up until now there have been no days off at all unless you are feeling ill.

The Terrain: You’ll be hiking for 6–10 miles per day. You will be frequently crossing barbed wire, muddy bogs with foot deep mud, and ankle-deep swampy water. You will be crossing creeks multiple times per day and some of these things can be up to 10 or 15 feet across. We are many times crossing them by walking over fallen logs that bridge the shores of the creek. You will encounter several kinds of snakes (we saw 4 today) and some of these guys can be pretty big (2-inch diameter, 4 feet long). It is getting warmer during the day now (in the 80s) so bugs are coming out and wanting to bite…so bring the deep woods OFF! Probably the most dreaded type of hiking is the feared briars that we have out here. What is a briar, you ask? Well, imagine the thickest thorn bush you have ever seen, the kind where you can’t see past 2 or 3 feet in any direction, the stalks stretch over your head to about 6 feet height, and it is anywhere from a few feet to 50 feet deep. They don’t allow machetes out here anymore because one forest service person literally chopped his finger off, so you have to go through this stuff basically by using a stick to hold it down and your gloves to move it out of the way. And yes, it does hurt.

The Gear: Ok what will you be carrying and what will you be wearing. The packet that you’ll probably get will imply that you will need to immediately run out and buy a whole bunch of gear. Don’t do it! At Hemphill at least, they have provided everything that you’ll need. You’ll be issued boots, pants, raingear, shirts, chaps (to protect the legs from the thorns), gloves, safety glasses, a hardhat, and a backpack. In Lufkin you can check out a GPS, and I suggest that you do so. Do yourself a favor and tear up their gear and not your own stuff.

The Search and your role in it: As I mentioned NASA is about 10% of everyone here, the other 90% is either forest service or EPA contractors. Basically, here is how the search is conducted…you are on a team of about 20 people. On each team of 20 firefighters, there are 3 EPA contractors and one NASA person. By the way, out here there is no USA, we are all referred to as NASA.7 Basically, what happens is that the firefighters line up in a line and each person is spaced about 15–20 feet apart. These are the guys in the “front lines” blazing the trail. You do not walk in front of them—that is considered “poaching”… they are the searchers and your role is the technical expert. Now of course if they miss something then by all means you can grab it and bag it.

The EPA contractors and the NASA rep walk behind the firefighter crew by about 20 feet or so. When someone finds something, they yell “bingo” and the EPA and NASA guys make their way over there to see what they have found. Your job as the NASA guy is the following:

Decide if what is found is a shuttle part. If so, bag it. If not, don’t bag it. If you aren’t sure, bag it anyway and let someone else figure it out.

Assess if the item is possibly a pyrotechnic device. You will get a briefing and pictures describing them when you get here. If you are even the slightest bit concerned AT ALL, you will instruct the EPA guys to call in the pyro experts.

Assess if the item is anything of or relating to the crew. Any FDF (Flight Data File) with names, food canisters, flight suits, etc., are considered sensitive and are sent high priority back to Lufkin.

Basically, you will have fewer questions regarding search operations if you remember that you are there to do a, b, and c above and the crews are there to search. Now of course you will be looking too but that is the big picture. Also, you do not carry the parts out of the woods, the EPA does that. They are also the ones with the sniffers that will be used if you deem appropriate.

You may be wondering, “how will I know for sure if a part is from the shuttle?” I’ve gotten that question a lot. After you spend a few minutes in the collection center, you will see, touch, and feel the characteristic nature of what the parts look like. The aluminum is burned in a rather unnatural way and you will not have many problems identifying it. Also, before I left, I was under the impression by the pictures on the internal e-mail that the recovered debris was in rather large pieces. It turns out that many, many of the pieces recovered are only inches across, much smaller than one would think. Be prepared to look for small things.

Having said that, after over a month of searching we have hardly scratched the surface in the Hemphill area. There is a LOT of land left to cover. This message was not intended to discourage people from volunteering. Far from it. It is one of the most rewarding experiences of my life, and I think it is helping all of us here to work through what happened. I just wanted to send out my thoughts on what really goes on so that people are more informed before coming out here.8

During this time, what amazed me the most was the absolute willingness of people close to the accident, NASA or otherwise, to withstand grueling physical labor, little sleep, horrible food, wetness, dampness, dirty mud, torn clothing, and all the other indignities that come from backwoods search efforts, simply because they yearned to help. Yes, people were still grieving, and yes, the program was grounded until a root cause of the accident was determined, corrective actions put into place, and all was ready (programmatically and psychologically) to fly again. That meant little work for our Shuttle cadres to do except supporting the accident investigation. But supporting the investigation didn’t necessitate being a member of the search parties, and yet people were readily volunteering for the task. As many who volunteered from MOD, the SSP Office and JSC Engineering, many more from KSC lent their assistance to help support the effort.9

As mentioned, at this time I was busy attending the MRT meetings and communicating out the detailed minutes from my notes. I was also acting as liaison between the OPO and MOD and assisting in coordinating our support to the OPO’s detailed investigations … all of that and perhaps a dozen other tasks that needed to be performed as well. I was occupied and focused on the work at hand. The rest of the division, however, while also supporting the investigation in various capacities, had a bit more discretionary time on their hands. MOD was keeping the control center busy with flight controllers coming in every day to train. It was unknown when we would get back to flying again but managers felt that maintaining our flight controller’s proficiency—ready to support the call to work the next mission (whenever that would occur) and keeping our skills sharpened—was necessary. So, folks got to train, and they supported the data analysis reviews from the end of STS-107. But consequently, it wasn’t enough to fulfill their personal need to help.

It was like relief pitchers in a baseball game—ready, willing, and able to come onto the playing field and help their team win, sitting idly in the bullpen and waiting for the call from the manager. My Systems Division personnel were coming in every morning wanting to help in whatever way they could to peel back the mysteries that surrounded the accident and pay their respects to the lost crew through the diligence of dedicated work. Staying busy was motivating not only because of professional integrity but also as a coping mechanism to a horrible tragedy that shook all those who were involved.

Consequently, driving the few hours north from JSC into the rural thickets and piney woods of east Texas and acting as systems experts in support of the teams of forest fighters scouring every acre of land for Shuttle debris was a task many were eager to volunteer for. It took them away from families for two weeks or more at a time, but the act was one as serious and as heartfelt as an act of purification or absolution, The selflessness required to volunteer for these duties was, I always felt, part of the rationale for going.

Figure 16. Search parties on March 19, near the area where the OEX recorder was found. (NASA Images KSC-03PD1115 and KSC-03PD1107)

On Wednesday, March 19, day 46 of the search and recovery effort, a particular component of Columbia hardware was discovered that, more than anything else, was considered the “holy grail” of the recovery efforts. This was the OEX (Orbiter Experiment) recorder, or black box. All Shuttle Orbiters were outfitted with such recorders, installed not for mishap investigations as they are on commercial aircraft but to record the thousands of parameters of telemetry while the spacecraft is out of communications range for eventual downlink to MCC. This happens routinely once an orbit over the Indian Ocean when we go out of range of the TDRS constellation.10 But these tape recorders were recording all the time, except for when they were rewound after the data was transmitted to the ground. Because the OEX recorder was recording data all the way to the point that Columbia lost power, it might still have information vital to the investigation. Debris had been recovered not long after the search parties began that originated from locations in Columbia near the OEX recorder, but the box itself had remained stubbornly hidden. Until March 19, that is.

At the time it was found, one of the search party supervisors was quoted as reporting, “Although it was a little dirty and had a couple of holes in its case where connections had broken-off, it appeared to be in amazing shape. It had been recovered in a thick, young stand of pine on private land northwest of Bronson in San Augustine County. It had very little apparent damage.11

My colleague in our flight control group was in the field at the time. Although not with the crew that found the recorder, word spread quickly among the search parties and NASA personnel and just as quickly with those of us back at JSC. It was an epiphanic moment, almost as if hope had suddenly been restored. More than any other debris, I was told after my colleague returned that locating this piece of hardware buoyed the spirits of the exhausted searchers like a burst of oxygen into a stuffy room long depleted of air. It did the same for those of us back at JSC, but we had comfortable beds to sleep in, got reasonable quantities of sleep, homemade food, and the comforts of civilization. Those searchers, particularly the fire fighters living in tents and contending with cool, damp weather for weeks on end, needed this news even more that we did.

Figure 17. The recovered OEX recorder. (CAIB Photo)

Mishaps occur, whether we’re dealing with spaceflight or experimental aircraft flight testing, industrial engineering challenges building bridges, or any of the millions of technical endeavors we develop. It’s an unfortunate reality of the business and while much of our energies are directed solely to the pursuit of preventing accidents, the environment we work in is filled with inherent risks. We succeed when the fruits of our labors prove to be resilient, but sometimes we fail. We parlay in a risky business.

While all of that may be true, the sting of failure is all the more profound when human lives are lost. Nothing in my experience is as devastating to engineers, managers, and leaders as when their creations fail in the task of protecting the very people who inhabit them. And when that happens, on those dark days, when community salves the pain of loss (as discussed in Chapter 3), another mechanism I have found employees use for coping is in the act of helping.

Life Lesson for Engineers, Managers, and Leaders

When disaster strikes, people want to help.

My advice for these circumstances is to find ways to allow your team to contribute to the recovery from the mishap. The very act of lending a hand—of jumping into the lifeboat, grasping an oar, and pulling—is an important catharsis. Not that the only pursuit here is catharsis; after all, talented engineers, managers, and leaders are always needed to determine the cause of an accident and to formulate solutions to prevent such disasters from happening again. But the cathartic effect of assisting in these activities, even if it’s just making another pot of coffee for teams working late into the night, is a constructive way for people to process their grief. Recognizing this can be an important tool in helping your team recover from the anguish of loss.

1 National Centers for Environmental Information. https://www.ncei.noaa.gov/maps/radar/ (accessed May 29, 2024).

2 Around all three Shuttle Main Engines are a ring of tiles referred to as the dome heat shield. These tiles seal the interface between the engines and the aft compartment of the Orbiter and were routinely damaged and replaced each mission.

3 Long-range and infrared camera tracking Columbia captured evidence of debris shedding from the spacecraft over the western United States, long before the vehicle broke up over Texas. Search crews were positioned in those states as well.

4 Email from Stan Schaefer sent Feb. 7, 2003, and forwarded by Rick Fitts to the author on the same day.

5 JSC’s Building 2 housed the center’s public affairs outreach and education offices. It was common for JSC employees to pick up posters, stickers, and other such materials there for friends, family, schools, and use during external speaking events.

6 Steve Sides, e-mail forwarded to author by Rick Fitts, March 20, 2003.

7 This is a nod to the fact that civil servant or contractor, all were in this together.

8 Steve Sides, e-mail forwarded to author by Rick Fitts, March 20, 2003.

9 Mike Leinbach, the former KSC launch director, wrote an insightful and deeply personal book about the recovery efforts titled, Bringing Columbia Home. An excellent resource into this moment of history, I encourage you to read it.

10 These short periods of LOS, or Loss of Signal, is the time when flight controllers would unplug from console, use the restroom, refill mugs of coffee, retrieve our lunch from one of the refrigerators or simply stretch our legs.

11 Paul Keller, Searching for and Recovering the Space Shuttle Columbia (Washington, DC: USDA Forest Service, 2003), p. 36.

Chapter 7

Touring Columbia’s Debris in the Shuttle Landing Facility Hangar

Sometimes Seeing is Believing

The first mission I worked on Columbia was STS-35, a science-oriented spaceflight dedicated to astronomical research. In her payload bay was mounted a series of telescopes—collectively referred to as ASTRO-1—that would scan the heavens above the atmosphere in the ultraviolet and x-ray spectrums. I was not yet a certified flight controller, so I worked this mission as an on-the-job (or OJT) trainee, watching, learning, and getting the gibe of the flight control profession.

I was assigned to the Orbit 3 shift led by Flight Director Bob Castle. The Orbit 3 shift normally also had the responsibility of working the mission’s prelaunch—the timeframe from around 12 hours to 3 or 4 hours prior to launch when we would hand over to the Ascent shift. For me and my EGIL compadres, the prelaunch shift was exciting, as this was when the ship’s three onboard fuel cells would be started, a particularly important task for our console. But prelaunch was also the time when the large ET would be loaded with hundreds of thousands of gallons of cryogenic liquid-oxygen and liquid-hydrogen, which was also a neat process to watch.

Unbeknownst to me at the time, it would take STS-35 six months to get off the pad.

Originally set to launch in mid-May 1990, the date was reset for later in the month when a problem with Columbia’s freon coolant loop was discovered. After that was fixed, an attempt to launch was made but during tanking of the ET a minor hydrogen leak was discovered in the tail service mast of the mobile launch platform—the massive structure on which the entire Shuttle stack sat upon—in addition to a second, more minor leak in the 17-inch quick disconnect valve where cryogens exit the ET and enter the Orbiter. The leak in the 17-inch disconnect led to concentrations of hydrogen in Columbia’s aft compartment exceeding launch limitations, so the launch was scrubbed.

Hydrogen is nefarious for leaking. The minute size of the molecule is difficult to contain, and the chillingly cold cryogenic temperatures below –400°F make sealants almost ineffective. A mini-tanking test was called for on June 6 where the leak was again confirmed to the 17-inch disconnect, but unfortunately repair of that component couldn’t be performed at the pad, so the entire stack was rolled back to the cavernous Vehicle Assembly Building (VAB) where Columbia was demated and transferred back to her hangar.

More shifts worked on console with Castle and the Orbit 3 team, but still no launch.

After the 17-inch disconnect was replaced with one borrowed from Space Shuttle Endeavour, Columbia was rolled back to Pad A for a second time on August 9 with the intent of launching her on September 1. But two days before launch a computer on the ASTRO-1 payload was noted to have malfunctioned and had to be replaced. This moved the launch to September 6. When we gathered again on console for that launch attempt, high concentrations of hydrogen were again noted in the aft compartment and the launch was similarly scrubbed. Thorough analysis determined that the concentrations in the aft compartment were not the result of leakage in the 17-inch disconnect—therefore it was a separate problem—with the focus now shifted to a package of three recirculation pumps located in the aft compartment. These were replaced and retested and launch reset for September 18, but when that attempt was made the leak in the aft compartment frustratingly resurfaced. At this point, program management made the wise decision to stand down for a while to really think through what was going on.

A special “tiger team” was formed, which determined that the problem stemmed from a leaky seal on one of the three main engine’s pre-valves. This pre-valve system was routinely tested with gaseous nitrogen at ambient (room temperature) conditions and always passed the test, but when faced with cryogenically cold hydrogen, they began to leak.1 Now we had a culprit.

Columbia was moved from Launch Complex 39-A to 39-B in order to make room for Atlantis for its STS-36 mission (a classified one for the Department of Defense) and then moved back to the VAB for repairs to the pre-valves. A tanking test on October 14 verified that the repair was successful, and no leaks were noted in the aft compartment. Finally, on December 2, we tanked again but this time with the intention of launching Columbia, which occurred successfully just short of 2 p.m. local time.

Through all of these tanking tests and launch attempts the Orbit 3 team would gather and monitor conditions from our consoles in Houston. This was the first prelaunch shift I had worked (and only my third Shuttle flight in total), but by the end of it I was as experienced in prelaunch activities as any of the more seasoned flight controllers in our group!

But back to STS-107. Allow me to start with a few final statistics from the efforts to recover Columbia’s debris:

At liftoff, Columbia weighed a total 263,706 lbs. This included 231,622 pounds for Columbia herself, plus an additional 32,084 pounds for the primary payload she carried—the SpaceHab module and the cryogenic pallet needed to extend the mission’s duration to two weeks, both located in Columbia’s payload bay.

Columbia’s debris was shed over a zone of some 2,000 square miles in east Texas and western Louisiana, stretching in a swath 200 miles long by 20 miles wide, reaching from Dallas, Texas to Ft. Polk, Louisiana (it is known that some tile debris was shed over the western United States too but the exact weight of that is unknown).

Over the course of all searches during the spring of 2003, and a scattering of debris mailed to NASA after the searches were complete, a total of 81,710 separate pieces of Columbia and her payload were recovered, totaling 84,700 pounds. This constituted 38 percent by weight of the spacecraft and payload.

Less than 40 percent of Columbia was recovered. Where is the rest of the spacecraft? Well, only two plausible explanations are available. Either some elements of Columbia were vaporized in the searing reentry environment when the ship broke up, or the debris remains out there still to this day, now permanent residents of the deep thickets of thorny pine forests or immersed in the mud and ponds that scatter the landscape of the region, monuments where they landed to the dangers of human spaceflight.2

In May 2003, five of us from MOD were flown out to KSC and allowed to tour the debris being collected in the Shuttle Landing Facility’s hangar. This cavernous volume resides almost directly adjacent to the 15,000-foot grooved-concrete runway where Shuttles land once returning from space (and, coincidently, was Columbia’s intended destination on STS-107). We weren’t the first members of MOD to view the debris, but we all were, in one capacity or another, involved in the mishap investigation. It was believed we could benefit from seeing the debris for ourselves and NASA would pay the bill.

Early that morning we gathered at Ellington Field; a former Air Force Base turned into a commercial airfield. Home to JSC’s fleet of T-38 astronaut training aircraft as well as NASA’s three WB-57 reconnaissance planes and the bulbous Supper Guppy cargo plane, Ellington resides only a few miles from JSC, just northeast of the center. Out on the tarmac before us stood NASA’s Beechcraft Super King Air aircraft, primed and ready for flight. This twin-engine “business-class” airplane is one of the small cadre of aircraft NASA uses to shepherd executives on diplomatic missions or astronauts home to JSC at the end of their space missions. Today, the King Air was to support our mission!

Excited to begin the day’s adventure, we approached the folding stairway that led up into the plane by a NASA logistics operations person, had our badges checked, welcomed aboard, and then allowed to enter. While luxurious by most commercial standards, the plane smelled musty to me, almost as if it were a museum piece repurposed for operational use. Inside was a reasonably spacious interior filled with four wide, leather-covered swivel seats. Two of the seats bordered a flat, faux-wood table while the other two were arranged almost chaotically about the interior, seemingly installed haphazardly. In addition to the seats was a single comfortable-looking bench along one side and two other smaller seats permanently mounted onto the carpeted deck. Fake wood covered much of the walls giving the aircraft a “mod” 1970s feel to it. We took our seats, buckled our seat belts, and prepared for the two-hour flight to KSC.

The day was sunny, clear, and typically humid for a mid-spring day in Houston. Once we took off and were cruising towards KSC, the pilots came aft to greeted us as if the five of us were celebrities. We all shook hands courteously, glancing nervously at each other wondering who exactly was flying the plane. One of the pilots warned innocuously that the aircraft’s autopilot was acting up and had a tendency to suddenly switch itself off. He said it was no reason to worry, the condition was a minor inconvenience, but we might feel a few “excursions” from normal if it happened. Contentedly, both pilots returned to the cockpit, leaving the door open, which provided us a spectacular view of clouds over the Gulf of Mexico out the forward windows.

Sure enough, about halfway into our flight, the floor the aircraft suddenly dropped, and we entered a roller coaster–like mode of flight known as a phugoid. Phugoids are not particularly dangerous and are well understood; they are a longitudinal oscillation of altitude where the amount of height gained and lost is reduced each cycle until the mode dampens itself out. We “non-pilots” were all seated at the time (which was fortunate), but when the autopilot disengaged, I felt my stomach drop to my feet and blood rush from my head, followed by a sudden reversal and an increase in weight, and back and forth. The entire event lasted perhaps no more than 10 seconds, but I vividly recall the look on the face of one of our MOD flying companions, a former flight controller and then manager named Jimmy Spivey, as he held the seat’s armrests in a death grip of surprise and subdued fear. His eyes were wide and he smiled slightly, as if not quite sure if the sensation was thrilling or terrorizing. Fortunately, or not, we got to enjoy two more phugoids before the trip was over.

As we approached KSC the aircraft banked out over the Atlantic Ocean and flew over the VAB, the gargantuan hanger where the Apollo Saturn V rockets and Space Shuttles were stacked and prepared for flight. We aligned with the SLF runway and still hundreds of feet above the Florida coastline passed beyond the landing strip’s threshold. Being that we were still so high, I recall wondering if the pilots were going to make a pass over the runway and then circle back for a landing, but as the concrete passed beneath us, we descended, growing closer and closer to the ground. A normal approach for a commercial airliner would be far less than a hundred feet high at this point and I wondered what was wrong, why were we not already on the ground. Five seconds passed, then 10, and we were still above the runway. Twenty seconds, getting closer and closer, when finally, the King Air’s wheels kissed the hard surface and we were down. That small bit of drama, I would realize later, was purely due to the incredible length of the Shuttle runway. At around three miles in length, it was one of the longest runways in the world.3

The NASA King Air taxied over to the SLF hangar and shut down its engines. It was probably 10 o’clock in the morning and as we walked down the plane’s aluminum steps and onto solid ground we were immediately met by the warm, damp air of Florida’s late spring. The stroll to the hangar was only a hundred feet or so, which we crossed quickly, passing inside the hangar through a metal door in the side the building next to the expansive set of expansive sliding doors that lined the front of the hangar.

Once inside the first thing I noticed was the smell. It was, as with the King Air, a musty odor reminiscent of what I’ve experienced when touring old WWII battle ships; a mix of metal, oil, age and perhaps the hallowed ghosts of previous inhabitants long since passed. The scent in the SLF hangar had all those characteristics plus one more—the damp, earthy smell of mud. Perhaps the odor was inherent to the hangar itself due to its aging aluminum construction and fiberglass and fabric walls, but the earthy overtones were, I suspected, an artifact of the “found” state much of Columbia’s debris had when brought into the building.

Figure 18. Inside the SLF Hangar, looking towards the rear of the building. A general outline of Columbia can be seen mapped out in blue tape along the right-hand side of the picture, with the nose of the Orbiter at the bottom of the photo. A mapping of the left-hand wing, where the damage was believed to have originated, is toward the left-hand side of the photo. (NASA Image KSC-03PD1474)

A row of large eggshell fans lined the hangar-door wall that separated the facility from the runway. Evidently the curators wanted to keep the airflow moving within the confines of the hangar. I was sure that every piece placed onto the gridded floor had been evaluated for toxic chemicals and dangerous irritants, likely by the EPA, but the fans offered an extra layer of protection and, I assumed, also helped preserve the debris.

Laid out next to the fans were an entire series of spherical metallic tanks. Some of these globes were reasonably small, about the size of a basketball, but others were much larger, reaching three or four feet in diameter. The Shuttle Orbiter was outfitted with dozens of spherical pressure vessels, each containing the varieties of fluids and gases necessary to conduct the operations of traveling to, from, and in space. Hydrogen, oxygen, helium and nitrogen, ammonia, and hydrazine—a veritable chemist’s shop of elements and concoctions. Some were held at relatively low pressures while others pressurized to dozens or even hundreds of times that of Earth’s atmosphere (14.7 pounds per square inch, or psi). As we walked past this farm of pressure vessels, we noted that many appeared to be intact with pipes, tubes and wiring still emerging from their surface. And yet there were probably an equal number that had collapsed or ruptured, whether from Columbia’s high-energy breakup or the equally high-energy impact with the ground it wasn’t obvious. Most were caked with dirt and muds still adhered to their surface, but few showed signed of scorching or other temperature-related effects.

Figure 19. Some of Columbia’s pressure vessels, being moved from the SLF Hangar to their more permanent storage location in the Vehicle Assembly Building. (NASA Image KSC-03PD2610)

Moving further inside the hangar we came face-to-face with the extent of just how much debris was being displayed. The entire surface of the hangar’s concrete floor had been covered with a gridwork of yellow tape, providing a system of location-markers where pieces could then be positioned and catalogued should they need to be retrieved. A considerable portion of the floor was then also imposed by a general outline of Columbia, mapped out with blue tape, which demarked the basic physical parts of the Orbiter. Starting with the nose of the vehicle, the blue outlines followed the shape of the spacecraft to the crew module, the midbody and wings, back to the aft fuselage and finally to the tail. Within these general outlines pieces of debris were placed onto the hangar floor. I assumed this was a standard practice for vehicle reconstruction efforts where correct placement of recovered debris can be critical to determining the cause of an accident. Now, it is important to remember that while the floor of the SLF hangar is a two-dimensional construct of length and width, the actual Space Shuttle very much extended in three dimensions. How the reconstructors and investigators accommodated this wasn’t apparent to me, but I was confident their system accounted for that third dimension of height.

Of the 81,000 separate parts of Columbia that were eventually recovered, a total of 2,791 were displayed here on the SLF Hangar floor (the remaining 78,919 remained in storage).4 One of the first pieces that was clearly recognizable was the spacecraft’s nose gear. The nose gear was part of the Shuttle’s tricycle landing gear, consisting of a long shaft that ended with twin rollers onto which the tires were attached. The entire structure was perhaps four or five feet long, with tubes containing hydraulic fluids and electrical wires snaking down one side. The gear was positioned on its side, laid atop a few wooden boxes. A small bucket was placed below the shaft where it was actively catching drops of hydraulic fluid leaking from the mechanisms.5

Both tires were still attached but were obviously deflated and looked like they had been through a shredder. I moved back along the edge of the blue tape and came to the other two landing gear, the left and right main gear. If you recall from Chapter 2, it was in the well that contained the left main gear where indications of hot plasma flow entering the wing were first seen by Mission Control. I’m not sure what I expected to see, perhaps a slag of molten metal or other evidence of the gear being exposed to 3,000° hot gases, but none of that was in evidence. Yes, the gear looked dirty and somewhat beaten up, like it had been thrown from the back of a fast-moving pickup truck and struck a cinder-block wall (which in ways it had), but it looked largely intact except for the tires, which were partially shredded. If there was evidence of searing hot plasma on the structure, I didn’t notice it.

There were then some larger sections that came from the sides of Columbia’s fuselage, an area of the vehicle called the “Chine,” to which remained attached bits and pieces of the familiar thermal protection tiles that covered much of the vehicle’s exterior. A few of the tiles were intact but most were broken and shorn, the black silicon overcoat blown away to reveal the jagged edges of their white interior. In fact, everywhere I looked were indications of the massive energies involved in Columbia’s destruction. Metal was ripped, bent, broken, cleaved into ragged slivers, and wrenched apart from each other in a maelstrom of violent forces. It was almost unimaginable to realize what had transpired, what had taken this beautiful human spacecraft and torn it asunder. For some time, I and the others stood there and simply considered all of this, unable to speak, quietly considering what failure really looks like.

Eventually we moved back to the section marking the aft fuselage and tail of the Orbiter. Portions of the vertical tail were laid out horizontally on the floor, the right-side of the tail separated from the left. Each was perhaps five or six feet long and two or three feet wide. These sections were obviously destroyed pieces of wreckage, but even besides that they revealed one stark difference from each other. The black thermal protection tiles from the section of tail from the right-hand side of Columbia were largely intact, roughened up but mostly untouched. The sister portion from the left-hand side of Columbia was starkly different. It looked like it had withstood several shotgun blasts. The tiles were all completely mangled and many completely wrenched from the tail’s surface. We knew at that time that the damage to Columbia had begun on the left wing, but the evidence we now stood gazing at indicated a violent disintegration of the left wing. We speculated that portions of the wing may have come off during entry, flew backwards and struck the left-hand side of the vertical tail.6

Off to the side of the hangar, separated from the Orbiter’s outline, was a raised platform which approximated the left wing. Onto this dais were placed thermal protection tiles and underlying structure from Columbia’s left wing. This reconstruction placed this debris, after being positively confirmed to have come from the upper surface of the left wing, into the exact locations they would have been mounted on Columbia. Again, we knew approximately where the hot plasma had entered the wing, around the leading edge Reenforced Carbon-Carbon (RCC) panels #8 and 9, which is where the ET foam was estimated to have struck. The tiles on this reconstruction in the area just aft of panels 8 and 9 did indeed show evidence of hot gases. The technical term applied to the physical malformation of the tiles was “slumping,” and that’s exactly what they looked like, as if they were made of sand and had slumped from repeated application of waves on a beach. Right behind these RCC panels the tiles were melted like butter scooped onto a plate and then held above a hot flame. This clearly wasn’t impact damage of the sort we saw along other parts of the fuselage—this was clear evidence of the effects of searing hot gases.

Interestingly, we had seen pictures of these tiles after they had been discovered in the field and brought in for examination. I can recall viewing images of these very tiles in conference rooms back at JSC as the evidence of Columbia’s destruction slowly came in. In fact, these specific tiles became somewhat famous back at JSC for their slumping characteristics and I had seen pictures of them dozens of times. But seeing them for real, the actual tiles, sitting on the table right in front of me, made the impact on me more visceral. This wasn’t something being communicated via PowerPoint charts; no, these tiles were real. It was like viewing some important historical artifact in a museum for the first time, something that was imbued with legend and mystery, only imagined through tales and history books and now standing right in front of you, testaments to their reality. The effect was nothing less than sobering.

After 30 minutes in this area, we wandered over to the far side of the hangar floor. There stood an artifact of wreckage which I found surreal. Propped into a vertical position was one was the frames of the six forward windshields of Columbia. Each Space Shuttle Orbiter had twelve windows, ten on the flight deck and two (if you count the airlock window) on the middeck. Orbiter windows are a tempered fused-silica construct that are designed to withstand the rigors of spaceflight. Two panes of glass are used to contain the atmospheric pressure of the crew cabin, while a third outer pane provides thermal protection from the temperatures of ascent and reentry and also protection against impact from orbital debris while in flight.7 But now these six window frames—three from the left-hand side of the cockpit and three from the right—were devoid of glass. All that remained were the metal frames that formerly held the windows. The shape of these frames was familiar to everyone who worked in the Shuttle program, an outline reminiscent of aircraft forward windows but distinctive to the Shuttle Orbiters. There was no mistaking these window frames for anything else.

Again, I was struck by a weird sensation. Through these very windows gazed the four astronauts who rode on the flight deck during Columbia’s reentry—Commander Rick Husband, Pilot Willie McCool, and Mission Specialists Kalpana Chawla and Laurel Clark. It was these very windows inside Columbia that had protected the crew from the searing hot gases of entry—not some other windows, but these very ones. Instead of looking through them, I now gazed at them, propped up on a table, rendered free from the rest of the fuselage, singed by unimaginable temperatures and dropped onto the ground from a height of a hundred miles. But it wasn’t the journey these window frames took that I found sobering; it was the visceral connection they made to the crew, the seven brave souls who lost their lives in this tragedy. It was as if there was a connection somehow between me standing in the SLF hangar and Columbia’s crew who only a few months previously had gazed through the exact same window frames. The feeling left me … well, unsettled.

We were, of course, not allowed to touch the hardware, but I did take a few steps closer to look at the debris. Inside the frames I saw small pieces of broken glass. Each smaller than a sugar cube, these remnants of the shattered windows still held on, residing in the narrow crevices that once held the intact panes of glass. Here lay real window, not a simulation or prototype or a replacement held in storage, but the actual thing.8 Again, a chill ran down my spine and I shivered in response.

Figure 20. Media representatives taking photographs of the frames that contained Columbia’s forward cockpit windows. (NASA Image KSC-03PD2578)

You may ask what of the rest of the crew cabin? Except for these window frames, none of the crew module—the flight deck or middeck, the pressurized volume in which the crew lived—was anywhere to be seen on the floor of the hangar. That, actually, was due to a conscious decision by managers to show reverence for the crew and made out of respect for their families. Every portion of the recovered flight deck and middeck, of which there was considerable, was partitioned off from the rest of the debris into a section of the hangar only accessible to a few. A solid metal door separated this part of the building from the rest of the hangar floor and access was only granted to select members of the astronaut office and others given permission to participate in the cabin’s reconstruction. These extra precautions, sensitive to the crew and their families, I thought was an honorable action for NASA to take.

Life Lesson for Engineers, Managers, and Leaders

Sometimes seeing is believing.

We live in a professional environment where presentation material—viewgraphs when I first began my career, and today PowerPoint—predominates our communications. A person can spend their entire career without once getting their hands on the hardware that they are designing or operating. Case in point, over my 11-year career as an active flight controller in Mission Control and the following 5 years as the MOD rep to the OPO, the only time I actually “saw” Shuttle hardware was the above experience with Columbia’s wreckage. It would be another three years after I stopped supporting the Shuttle Program and moved on to NASA’s Constellation Program development before I actually got to see the Shuttle up close. In 2008 I had the opportunity to accompany one of MOD’s training teams in support of the STS-123 Terminal Countdown Demonstration Test (TCDT), a routine event for each Shuttle mission where the launch team and crew practice the last hours of a launch. Space Shuttle Discovery was, at the time, being prepared in the Orbiter Processing Facility for its next mission. We got an opportunity to crawl through the claustrophobically cramped volume of the ship’s aft compartment and donned protective “bunny suits” which allowed us entry into the crew cabin’s middeck and flight desk. What’s more, the STS-123’s vehicle, Endeavour, was sitting stacked out on launch pad 39-A. We stood on the Mobile Launch Platform right next to the stack, craning our necks upwards at the massive rocket we stood next to. We ascended the various levels of the launch gantry, including the crew access arm, which provided astronauts and supporting technicians entry into the Orbiter. I got close enough to Endeavour to touch it (which I didn’t), and the experience was both exhilarating and enormously educational. Why flight controllers weren’t routinely sent to KSC to see the vehicle they were expected to operate has always been a mystery to me. (For more on this STS-123 TCDT experience, see Chapter 11.)

In the case of Columbia, I had attended months of meetings discussing the spacecraft and its hardware, with graphic descriptions of the debris located in east Texas and western Louisiana and stacks of presentations, many of which included photographs of these parts. But nothing prepared me for actually seeing the debris with my own eyes. The violence that characterized the breakup of Columbia, the forces, accelerations, and temperatures that literally tore that spacecraft apart into hundreds of thousands of pieces, simply couldn’t be adequately communicated through PowerPoint charts. I literally needed to see it for myself to truly understand the scope of this accident.

That day spent in the SLF brought home to me the magnitude of this horrific event. I had to see it to truly believe it.

As engineers, managers, and leaders, we live in a steady cadence of design and programmatic reviews and trade studies. Many of these include copious amounts of computer-aided representations of hardware, detailed schematics, and other accoutrements of the engineering trade, as well as budget sand charts and detailed schedule waterfall assessments. And those can be helpful in characterizing an issue or conveying an idea. But my experience with the remnants of Columbia in the SLF and the example of Discovery and Endeavour during the STS-123 TCDT has also taught me that nothing beats seeing the actual hardware. Sometimes, you need to see something in order to believe it. And surprisingly I believe this is as true for managers as it is for engineers—get out occasionally and see what your team is producing or working on. Get your hands on it (if that’s allowed) and transcend the faux reality of PowerPoint. And for leaders, the concept is even more important!

I can only encourage you to get out there and see the systems you’re developing, managing, or operating. Don’t rely just on charts, graphics, or photographs—go see it for yourself. Especially on those rare and critical moments when supporting a mishap investigation. Charts, schematics, and photographs don’t always convey sufficiently the enormity of a situation but seeing it for yourself often can. Avail yourself of these opportunities and, in my mind, you (and your team) will be better for it.

1 Dave Seymour, STS-35 Scrub 3 Hydrogen Leak Analysis. (Washington, DC: NASA, TM-103548, July 1991), https://ntrs.nasa.gov/api/citations/19910020874/downloads/19910020874.pdf (accessed Jan. 16, 2025).

2 During production of this book, one peer reviewer recounted, “I received debris reports that were ultimately deemed unreachable. I also experienced challenges with understanding the observers’ description of falling debris and translating that into a search location. We did our best with the tools we had at the time, but knew some pieces would never be found or retrieved.”

3 Later that afternoon when we took off again to return to JSC, I remember looking out the window as the aircraft rotated and took to the skies. A hundred feet in altitude and we were still over the runway. Two hundred feet and we were still over the runway. Nearly 500 feet above the sandy beaches of central Florida and we were still over the runway. The SLF is a very, very long runway!

4 Steven Hirshorn, NAIT #7 meeting notes. May 15, 2003.

5 Amazingly, this fluid would continue to escape, drip after slow drip, for literally years after the accident.

6 Other evidence we learned later in the investigation would validate this hypothesis.

7 During the course of the Shuttle Program, changing out these outer windows after a mission because of orbital debris impact damage was a common occurrence.

8 The frames for Columbia’s windows 1 through 6 are now prominently on display at the KSC Visitor’s Center.

Chapter 8

The Mishap Response Team (Part 2)

During Disasters, It’s Amazing How Quickly Resources Can be Applied to a Problem

According to my notes, I supported the MRT beginning Monday, February 3, 2003—two days after the accident—and continued supporting it through Wednesday, June 4, 2003. It’s entirely possible that I supported even after that date, but that’s when my evidence runs out based on the notes I’ve kept. During that time between February and June 2003, the MRT was rechristened as the NASA Accident Investigation Team, or NAIT. During the life of these two forums, the main players came and went, but the intent and inherent purpose always remained the same. First, the MRT/NAIT promoted coordination and communication, especially in the early days of the investigation immediately after the accident. At that time no one really knew what was going on or how we should be going about an accident investigation of this magnitude. The MRT/NAIT enabled NASA to get a coordinated start, line up all the requisite functions that would be needed for exploring what happened to Columbia, and, ultimately, explaining why. It allowed for the beginnings of order from the state of complete disorder that follows most major incidents. When Columbia broke up over Texas it was as if we were a chess board with the pieces scattered haphazardly on the floor. The MRT/NAIT allowed us to place the pieces back onto the board and position them in their correct starting squares.

The MRT/NAIT also filled an important gap. As per the SSP’s Mishap Response and Contingency Action Plan, which was invoked immediately after the accident, a formal investigation board independent of the agency was called to be formed. This was achieved by the NASA administrator on the afternoon of the accident with the creation of the Columbia Accident Investigation Board, or the CAIB, chaired by retired Navy Admiral Harold W. Gehman Jr. The CAIB was populated by a diverse cadre of technical and organizational experts, emeritus from both Air Force and Navy safety centers, experienced Federal Aviation Administration accident investigation practitioners, tenured professors from academia including Stanford and MIT, and a single NASA representative, the center director of Ames Research Center. Collectively the CAIB was a capable and credible group with the background and experience necessary to assess truth from mystery.

Figure 21. Navy Admiral Harold W. Gehman, Jr. (ret.), Columbia Accident Investigation Board Chair. (CAIB Photo by Rick Stiles)

But the CAIB itself needed some help in getting organized, and all the while the clock was ticking on critical activities like communications with federal and local agencies such as FEMA and the EPA, and on the identification, collection, and recovery of debris. This also included the beginnings of analysis of all the clues that hopefully would lead to determination of root cause. It simply wasn’t possible to approach all of this activity slowly and to patiently take time to start the momentum moving forward. Things needed to be done right away, and those activities needed to be coordinated and managed. In this capacity, then, the MRT/NAIT filled that role admirably.

My daily walks to attend the MRT had nothing particularly spectacular about them as I navigated from one JSC government building to another, up a flight of stairs, and then down a hallway to the cramped confines of the MCC’s Action Center. But even within this staid normalcy, that walk also offered a kind of hallowed reverence for me. Halfway down that hallway within the MCC rose a set of tall, metal encased doors which marked the entrance to the FCR, or Flight Control Room. Originally referred to as the MOCR (clumsily pronounced Moe-Ker, an acronym for the Mission Operations Control Room) during the earlier days of the Gemini, Apollo, and Skylab programs, the volume of space behind those doors were to us flight controllers the very heart of the space program. NASA’s Gemini 4 mission in 1965 was the first spaceflight to be controlled from Houston (previous missions, including all those of Project Mercury, were controlled from Florida), and that tradition continued throughout both Gemini and Apollo, through the flights to Skylab and all Shuttle missions prior to 1996 when the “new” control center became operational. The MOCR/FCR was used for them all.

I had spent years working in that room and in some sense, it was just four walls containing a bunch of workstations. But in other very real ways, that place was something special, and entering it was not unlike the feeling one gets when crossing the threshold into a grand, ornate cathedral, awed by its splendor, magnificence, and history.

To gain entrance one had to insert a specially coded NASA badge into a card reader that hung sedately on the wall. A small light would transition from red to green, whereupon an audible “clack” would be heard as the locks on the door were released. At that point, it was a simple matter of grasping the doorknob and swinging the heavy portal open.

Once inside things would appear very similar to entering today’s stadium-seating movie theaters. You pass through the door and are inserted into a short, darkened hallway. Walls rise on either side but as you proceed farther into the room the rows of seats descend until they meet the floor. Turn to your left and there it is splayed in front of you—Mission Control.

Immediately inside the doors was a tall cabinet that provided the fuel that kept this locomotive moving—a coffee machine. The cabinet was government issue, perhaps six feet tall, made of dark aluminum and with high hinged doors that were perpetually kept open. Inside was the coffee machine with burners for twin carafes, slots for baskets containing the ground beans and filters, and an orange glowing power light that I never saw turned off. Water was decanted from two large blue plastic jugs that sat on the floor next to the cabinet, poured into a four-quart white plastic measuring container, and then emptied into the coffee maker. A small metallic box sat on the shelf next to the machine through which quarters or dollar bills could be slipped, and next to that were stacks of small Styrofoam cups. A single incandescent light was built into the top of the cabinet, enough to enable visibility but not so much as to disturb the quiet darkness that enveloped the entranceway.

Ordinarily one would walk past the coffee machine and round the corner where the four long rows of flight control consoles rose in tiers. However, if one were to continue on a straight path, walking past a series of windows that contained the SCA, or Sim Control Area, where the training team would gather during our frequent training simulations, you would reach another door, perched below an American Flag. Open that door—which contained no lock—and you would be thrust into the almost pitch-dark blackness of what was euphemistically known as the “bat cave.”

Along the front of the FCR were a series of screens that conveyed a multitude of information to the flight controllers in the room. The largest screen, prominently set in the center, was a 10-foot by 30-foot representation of Earth’s geography with an orbital map—known as a ground track—overlaid. The spacecraft’s position over Earth could be ascertained at any time with a simple glance, as would the path it would take over the next two orbits. To the side of the ground track map were four smaller, 10-foot by 10-foot square screens, onto which could be projected a variety of data, from spacecraft fault messages and crew computer keystrokes to flight dynamics information, and even video images being streamed from orbit.

The bat cave was the space that sat behind these five screens and contained the necessary projector equipment. As with any theater’s projectors, the room was kept dark so as to not pollute the image being projected. Each screen required its own projector, held atop metal scaffolding as if the place were no more than a construction site. A small, rubber-lined, and grooved ramp led down from the FCR onto the floor of the bat cave. On the opposite side was a second ramp back up to another door, through which one could access other portions of mission control. Early in my flight control career, before matriculating to a FCR position and working as one of the “back room” support positions, my route to the FCR was just as frequently through the bat cave as it was through the front entrance with the coffee machine.

Figure 22. View of the door to the “bat cave,” beneath the American flag. (NASA Image S71-17610)

But getting back to our story, during the investigation my path led me beyond the FCR and on to the Action Center. As previously mentioned in Chapter 5, just a day or two after the accident numerous offers of assistance were being made from other government agencies and President Bush himself ensured NASA (and the world) that all necessary resources would be made available. Dave Whittle’s report at the February 5 MRT meeting, four days after the accident, gives some insight into the crescendo of activities occurring in the field in support of recovery operations:

Mishap Investigation Team (MIT) – Dave Whittle

Have begun receiving debris (processing facility up and running). It is being logged and tagged:

Debris received includes an SSME actuator and a Landing Gear wheel and strut (not sure if it is LMLG or RMLG and it is “barely recognizable”), and a N2O4 tank (with fluid).1

Expect a jump in the number of reports of debris in California and Arizona.

Dryden personnel2 are being dispatched to California and KSC personnel to Phoenix, Arizona.

Have receive a truckload of debris from Louisiana yesterday.

All debris will be sniffed for contaminants and then cataloged.

357 bags of debris in storage.

60 teams in the field in Lufkin and 20 teams in Ft. Worth.

Local law enforcement personnel have been deputized to pick up minor debris.

Debris sightings in California and Arizona are increasing—may increase the level of support. These sightings match the vehicle’s ground track (i.e., credible).

FEMA declares the operation is past the planning stage and is up and running.

Shuttle Landing Facility (KSC) hanger is ready to receive the debris.3

What was remarkable to me was the speed with which FEMA was able to muster their support of this national tragedy. I suppose that I shouldn’t have been so surprised given that disaster recovery is the raison d’etre for FEMA and a thing they do on a routine basis, but it was rare for NASA to need assistance from “outside our fence” and the speed, efficiency, and professionalism with which this assistance was made available to us impressed me nonetheless.

But things were already evolving. At the beginning of the next day’s MRT, on February 6, 2003, chair Linda Ham voiced her appreciation for everyone’s hard and diligent work but then opined that the reporting seemed “in a rush yesterday”4 and wanted to apply some additional control. As such, she directed that future MRTs would have a slightly altered agenda with significant information presented first, beginning with Dave Whittle’s Mishap Investigation Team (MIT), then Ralph Roe’s OPO report and hardware release requests, followed by general status and all other reporting. Linda said she wanted us to take our time in the MRTs and that it was okay to defer items to the next day’s meeting if time was inadequate to discuss the item properly. She also mentioned that the primary MRT members would meet with Admiral Gehman and the CAIB in the main conference room on the top floor of JSC’s Building 1 (Room 966) and include concise, 10-minute presentations each from the astronaut office, MOD, JSC Engineering, Shuttle Program Integration Office, MSFC Project Offices, the Mishap Investigation Team, the Debris Sighting Team (investigating all visual evidence of debris being shed from Columbia prior to the ship breaking apart), the OPO, JSC Emergency Ops Center, and a few others.

Mike Kostelnik from NASA HQ gave the next report. From my notes:

HQ – Kostelnik

Discussed the general sensitivity for data preservation.

Discussed the ‘transition’ of authorities from SSP to the CAIB. This transition will be taking place over the weekend, after which the CAIB will be authorizing all approvals pertaining to the investigation (hardware and press releases) as well as making the large programmatic conclusions (i.e., the why’s of the accident).

Reviewed the subject of spokespersons for NASA. First, Dittemore [Ron Dittemore, Shuttle Program manager] has to approve all spokespersons. For those authorized to speak for the agency, he cautioned against hypothesizing. Let the systems experts detail the technical information that they are the experts on. All “conclusions” should be made via Adm. Gehman and the CAIB.

Let’s get all unambiguous facts out, but not offer “opinions.”

Sean O’Keefe and Bill Readdy will be at JSC tomorrow to talk with the CAIB.

Mr. Kostelnik voiced his desire to have inter-organizational relationships firmed up during this transition phase and to have clear and unambiguous direction for who’s in charge at every level. Fundamentally, NASA is responsible for analysis, field work, and data and the CAIB is responsible for conclusion.

Linda asked about Freedom of Information (FOI) releases. She stated that FOI requests coming to her will be routed to either HQ Legal or the MRT Legal people. Mr. Kostelnik replied that this process needs some work. Again, he emphasized that the CAIB provides the authority to release information and the MRT should hold items until that approval has been granted.

Lt. Col. Rick Burgess was in attendance—he was representing the CAIB. He discussed the release of impounded vendor equipment and asked Linda and Ralph Roe to formulate a process to control this. He asked there to be a single point of contact for all information and hardware release and that Adm. Gehman must sign for final approval. He utilized APUs as an example—APUs are not considered to have played any part in the accident; the hardware is sent to the vendor; later, questions arise pertaining to the APUs, but they have been altered by the vendor in some fashion. Basically, the CAIB wants to avoid the occurrence of this.

A question was raised about release of field work and Barksdale processing video. The press has requested some video of this. PAO/Kyle Herring stated that he has gotten requests for a standard press pool to come in a develop a “B Roll” of this work. All PAO requests will be worked through Kyle and HQ PAO and then approved by the CAIB. It was also mentioned that various news organizations are being contacted for video and images as they are taking thousands of pictures of debris.5

Dave Whittle then gave his commentary for the day. He began by stating that NASA was receiving more and more reports of sightings in California and Arizona, evidence Columbia was shedding parts on her reentry trajectory long before she eventually broke up. Astronaut John Casper, JSC’s director of safety and mission assurance, came on next and discussed their focus on early debris. He said nothing further west of Ft. Worth, Texas, has been “confirmed” but that they are utilizing NTSB (National Transportation Safety Board) radar to scour a primary interest swath 150 miles west of Ft. Worth. John stated that “the DOD is prioritizing their assets and which ones that we can utilize and getting the right people talking to each other.”6 He also mentioned that NASA would receive a side looking radar the next day (February 6) for help in searching the forests of east Texas.

Dave then continued with some striking information that a 600 lb. fuel pump had been located and will be transported directly to Florida (being too big to transition through Barksdale Air Force Base). Rain was apparently impeding with that day’s field operations, which were hampering planned flight operations [aircraft used in the search]. At this point, five days after the accident, Dave mentioned that about three dozen items had been delivered to Barksdale and over a thousand were in temporary storage in the Ft. Worth, Texas, collection site, but adding that no items on the Ralph Roe prioritized list had yet been found. Finally, Dave stated that so far 165 people have gone to the hospital from touching debris but that no exposure had yet been serious enough for anyone to be admitted.

On the February 7 MRT it was stated that “EPA has done a great job in packing the parts.” On the same day it was reported that “DOD is providing assistance in ways to search the more rural areas. Also using NTSB, Trajectory, FEMA.”7

On the February 10 MRT, Dave Whittle said that a significant military presence was in Lufkin to help with the effort.8 He also added that the Civilian Air Patrol (CAP) had been deployed to assist with the debris recovery, having investigated two reports of debris in Abilene, Texas (both of which turned out to be scrap sheet metal and not debris from Columbia), adding that a helicopter had been brought into Carswell Air Force Base near Ft. Worth for use by the CAP.

On February 11 the Mishap Investigation Team mentioned that “CAP (air patrol) [was] being utilized west for Ft. Worth. Getting points from NTSB on likelihood of finding debris” and that “A Louisiana guardsman was taken to the hospital after getting a whiff of something (think it was ammonia). Not admitted and is OK.” MOD’s Phil Engelauf, speaking during the OPO report, interestingly mentioned that NORAD (the DOD’s North American Aerospace Defense Command) had some analysis that indicated a part from Columbia might have departed the ship on the mission’s second day on orbit.9

On February 12 I recorded in my notes that the investigation team would be losing the National Guard in Lufkin, Texas, but might receive support from North Carolina National Guard and may get an augmentation of ~200 firefighters (this number was increased to ~1800 firefighters on the next day’s report).

On February 14 it was mentioned that FEMA would be providing full support for another 30 days and then would need to negotiate with NASA after that.

On February 18 word had been received that Navy divers were conducting operations in the murky, cold waters of Toledo Bend Reservoir and Lake Nacogdoches, and that there was a meeting with “NTSB, Boeing and SLF Hangar management... to determine if everything that can be done is being done.”10

All this interagency and local support activity arose in just the first two weeks after the accident, and much more was to follow.

I’ve never seen a complete listing of all the federal and local government and private organizations that took part or played a role in the recovery of Columbia. A simple internet search doesn’t reveal a comprehensive list and neither does the CAIB report, nor does Mike Leinbach and Jonathon Ward’s seminal book on Columbia’s recovery. Perhaps we will never know. But the MRT notes I took over the initial months of these operations reveal an enormously broad network of governmental, federal, state, and municipal organizations that included the resources of industry and corporate America as well. What’s more, hundreds (if not thousands) of individuals volunteered their time and efforts to this cause, from the landowners in east Texas and western Louisiana who graciously allowed searchers to canvas their property to the ranks of dedicated firefighters actually doing the searching. Private individuals sent in photographs of potential debris, and local communities facilitated the search with copious amounts of coffee, pizza, donuts, homemade pies and heaps of simple, rural empathy and hospitality. The true magnitude of these operations is unlikely to be fully quantified into numbers and spreadsheets, but the totality of what went into the effort assuredly rivaled many of humanity’s greatest disaster responses.

And what was even more remarkable to me was the speed in which these efforts were called into service and placed into the field. The Columbia accident, as well as the Space Shuttle Challenger accident 17 years before, were true national tragedies affecting not just the internecine NASA community but most everyone across both the nation and perhaps the entire world. These events were tragic in nature, as would be any circumstance in which seven people died, but the fact that this drama played out in full view of the public and the spectacularism and extraordinary nature of the event almost as if scripted from Hollywood, thrust the tragedy into a category that touched millions, if not billions, of people. There was something inherently personal in both accidents, as if those watching the news unfold on television impacted people as if the deceased astronauts were friends of the family, neighbors, familiar to the public in some fashion. People were horrified and touched by the sacrifice of the astronauts and were often eager to help.

That desire was true down in the coffee shops of Lufkin and Nacogdoches whose owners stayed up nights to ensure search parties had a hot cup when they set out early each morning. But the desire to help extended up throughout the government, to FEMA, the EPA, and the FBI, to the military services and the honorable soldiers and sailors who served, to desk bound bureaucrats who parsed policy and paperwork and yet still wanted to help.

Government is not renowned for its speed or efficiency (whether that is deserved or underserved I will leave for a separate argument), but in the case of Columbia all resources were made available almost immediately. Certainly, we flight controllers, engineers, and managers at JSC were ready and willing to commit ourselves to whatever was needed, but I still find it remarkable how quickly the same sentiment reverberated throughout our country’s government. All came together in common cause; barriers were lowered and boots were on the ground even before much of the debris had cooled from the flames of reentry.

Life Lesson for Engineers, Managers, and Leaders

During disasters, it’s amazing how quickly resources can be applied to a problem.

As of this writing I’ve worked for NASA for 35 years. I’ve seen my share of bureaucracy and ineffective policy, derived with good intentions but implemented poorly, become a hinderance to progress. It happens, whether in government or in industry, large or small offices. It can be maddening sometimes when a need is exigent and relief is difficult to apply. Rules and processes are turned into hurdles, some impossibly high and unscalable.

And yet, none of that seemed to matter when the need was searching for Columbia and her crew. I have no doubt that in this case there were hundreds, maybe thousands, of rules, policies and processes that were streamlined, avoided, negated, or possibly even ignored because NASA’s need was great, because the nation’s need was great.

And that says something to me about ourselves, about our country and perhaps about the human species in general. In times of the greatest need, it truly is possible to get resources where they need to be and to do it quickly. Disasters can be motivators of amazing accomplishments, as long as the will is there. Under the right circumstances, the impossible becomes possible, divisions melt away and common cause becomes a rallying cry that forges cooperation.

The STS-107 Columbia accident was exactly like that: a confluence of factors that allowed the miraculous to occur and to occur quickly. I have no doubt that massive organizational and logistical challenges were present that could have slowed down the search, but those involved prevented them from overwhelming the effort.

While this confluence of factors assisted in the case of Columbia, they aren’t there under all circumstances. But what these events showed me, the lesson that I get out of it, is that efficiency and the application of resources is entirely possible, and maybe even inherent in the right situation. It’s like a sleek, aerodynamic aircraft, built for speed and maneuverability, that is mounted with strakes and appendages that disturb the airflow, increasing drag and preventing it from being the super airplane that it can be. Within those impediments is a supersonic vehicle made for speed, but with the impediments it may be slow, lumbering, and cautious. The immediacy of disasters and the intent of goodwill can remove those barriers and reveal the clean, fast surface that resides beneath.

This realization is important for engineers, managers, and leaders to keep in mind and to know how to utilize strategically. Discarding policy and process haphazardly can be dangerous in its own right, as many of those are put into place through lessons learned and past experience. Rules and procedures should not normally be discarded or discounted. Yet, under extreme circumstances, in time of real need, an organization’s or effort’s “streamlined” supersonic aircraft can be made available with the right intent and applied under the right circumstances. These are rare occasions, but ones that can be brought to bear if the need is great.

1 LMLG is Left Main Landing Gear. RMLG is Right Main Landing Gear. N2O4 is Nitrogen Tetraoxide, the oxidizer used by the Orbiter’s maneuvering jets.

2 Employees at NASA’s Dryden Flight Research Center (now called Armstrong Flight Research Center), adjacent to Edwards Air Force Base in Mojave, California.

3 Steven Hirshorn, MRT notes, Feb. 5, 2003.

4 Steven Hirshorn, MRT notes, Feb. 6, 2003.

5 Steven Hirshorn, MRT notes, Feb. 6, 2003.

6 Ibid.

7 Steven Hirshorn, MRT notes, Feb. 7, 2003.

8 Ibid, Feb. 10, 2003.

9 Ibid, Feb. 11, 2003.

10 Ibid, Feb. 18, 2003.

Investigation

Chapter 9

Witnessing Columbia’s Disintegration

Failures Often Play Out Over Time

In the months and weeks that followed, as the jigsaw pieces of this vast, complex puzzle began to be filled in, the activities I supported slowly migrated from the MCC’s “Action Center”—where the MRT/NAIT had been meeting—to the sixth floor of JSC’s Building 1. This nine-story structure prominently stood above all other buildings on the JSC campus and was situated prominently close to the front entrance gate. Anyone entering the center was within sight of its imposing edifice, almost forced to gaze up and acknowledge that this building represented the center of gravity of the JSC. While the MCC was more famous and recognizable to the general public, the tall and rectangular Building 1, as the center’s “headquarters,” exuding power, influence, and authority, with the center director’s suite of offices sitting atop it all on the nineth floor like a garland of olive leaves.

Figure 23. JSC’s Building 1, its “headquarters” building. (NASA Image S93-41864)

The sixth floor was dedicated to the SSP Office, providing space for managers, support staff, and conference rooms. Like all buildings constructed when JSC first opened as the Manned Spacecraft Center in the 1960s, the interior of Building 1 was designed for utility rather than splendor. The walls were made of light metal and were reconfigurable, able to be shifted and moved so as to arrange and rearrange the interior layout as desired. If another office was needed, just move a few walls and create some new space! Each wall was painted a light beige-yellow, almost the color of eggnog, which was common to all buildings at JSC (including my home building of 4-North). A central shaft held four elevators, with four long hallways running the length of the rectangular structure, two traversing to the east and two to the west. Offices, restrooms and conference rooms then branched off these main hallways like twigs off a tree’s limbs.

Almost directly off the elevators was the Shuttle Program’s main meeting location, Conference Room 602C. To gain access one first entered a small, wooden-paneled vestibule where a secretary could check credentials of those entering (that is, make sure you were invited to the meeting!). Taking a quick left through a heavy locking door and you entered the conference room.

We had gathered this particular day to review a summary of all the imagery taken during Columbia’s entry which revealed pieces of the spacecraft coming off, being shed and left in its hypersonic slipstream. The room was one of the larger conference rooms in Building 1, perhaps only rivaled by the space on the building’s ninth floor used by the center director. But as conference rooms go, it was more than adequate. The interior walls were wood paneled and adorned with commemorations of each and every Shuttle mission. An enormous wooden table commandeered the center of the room with cushioned swivel chairs surrounding it adorned in a staid, earthtone fabric. All around the perimeter of the two long walls and the shorter entrance wall were rows of chairs, two or three deep, outfitted in the same neutral fabric but fixed, not able to swivel. The last wall hosted two large screens through which presentations or videos could be projected from a small, enclosed space behind them.

Paul Hill, one of MOD’s flight directors, had been charged two days after the accident with leading the effort of collecting all of this imagery and producing a summary timeline of these shedding events. As referenced by John Shannon, another flight director, at the February 3, 2003, MRT, “[We] will take sightings, corroborate them with other sightings (test credibility) and match with trajectory information. With this information, local authorities will be notified to search for debris. A great deal of concern will be areas west of Texas.”1 It was mentioned that over 1,000 video files were available to the reviewers.

At the February 6 MRT, Paul mentioned that his team has had 12 key photos/video images from the timeframe when Columbia was flying over California on her planned route to Florida. The resolution of this imagery was good enough to be able to propagate trajectory to 80,000 feet altitude. At that point, JSC’s Spaceflight Meteorology Group and local ground radar would be used to narrow the potential debris field. The goal, Paul continued, was to “shrink the footprint” where they could then further rely on Air Traffic Control (ATC) assets. All of this was to help locate any debris that might play usefully (or critically) in the accident investigation.2 Paul also stated that his team was fully interfacing with the United States Air Force (USAF) on sensor data (both ground- and space-based) and wants to use these facilities to look back all the way to Columbia’s deorbit engine burn.3

The next day, my MRT notes include the following report from Paul:

MOD – Hill

10 strong data sources of early sightings and 6 more that are still being investigated. Reports of 5 items of debris sighted when crossing the coast.

Looking at both sides of the trajectory from California through Flagstaff, AZ.

Several of these sightings overlap.

We have video in house and will be evaluating for trajectory propagation today; remaining video is enroute.

Weather radar analysis across the ground track is in work. USGS [United States Geological Survey] is evaluating entry signatures in seismic sensors.  Expectations are, however, low on these.

USAF has our trajectory data and timeline and is routing to various sensor sites for processing and analysis.

Weather data confirms SMG [Spaceflight Meteorology Group—JSC’s weather office]—assessment: no thunderstorms in the region and limited clouds.4

And it continued as more data and evidence were uncovered. Paul’s team reviewed a telescoping video of Columbia as the ship’s trajectory took it over Las Vegas. Reports were arriving of sightings of debris shedding over California. DOD (Department of Defense) visualization processes were leveraged to assist in clarifying the video and images; although the processes used were classified, the results would be made available to NASA. Infrasonic data from the NOAA (National Oceanographic and Atmospheric Administration) Los Alamos office was utilized. Each Orbiter vehicle, it was explained, has a classic infrasonic (relating to or denoting sound waves with a frequency below the lower limit of human audibility) signature. The team would be looking for off-nominal signatures compared to other similar data from some of Columbia’s previous missions. Apparently, initial results from this analysis showed something very significant occurring around the time Columbia passed over the California coast. Another piece of the jigsaw puzzle.

At the February 18 MRT it was mentioned that relative motion and ballistics analysis were being conducted for what was believed to be a debris shedding event (possibly a thermal tile) at GMT (Greenwich Mean Time) 13:54:33, which was a full five minutes before Columbia’s last transmission. The area on the ground, composed of the entire state of Arizona, was subdivided into nine different zones, each given a likelihood from 1 (high) to 4 (low) of finding a piece of debris. During this particular MRT, NASA Administrator O’Keefe was listening in and asked how the search would be conducted. Paul Hill replied that the FAA’s radar database would be researched to further reduce the size of the footprint—now an area 30 miles long by 5 miles wide in the “primary” zone, but areas significantly larger than that in a secondary (but still possible) zone and others. Then, aircraft assets would scan from the air and teams would be deployed to search on the ground.5

On February 19 Paul returned to the MRT to report that they had received new video from photographers in California and Flagstaff, Arizona, and were exploring these sightings for “luminosity and spectral comparison.” Another source revealed indications of a possible “flash” emanating from the spacecraft taken from Utah, just north of Flagstaff, which was directly under Columbia’s ground track. Some of the previously documented flashes (now being individually numbered for identification purposes) have had their footprints (e.g., ballistic coefficients) calculated and distributed to search parties in the field. It was also mentioned that there may have been sightings while Columbia was still out over the ocean and that these were also being investigated.

By the time we had gathered together in Room 602C that day, all of this information had been collected, processed, and analyzed. Shuttle debris was, in fact, found in some of these western states and were used in the investigation, particularly those for which their location on the vehicle could be positively identified. This information helped support the integrated timeline of events that was being constructed. It was hoped that enough information could be collected to map out the sequence of Columbia’s demise beginning with Entry Interface, the point at which the spacecraft began to enter Earth’s atmosphere and the start of reentry heating.

Shuttle Program Manager Ron Dittemore (also a former MOD flight controller and flight director) sat at the head of the long wooden table. I sat in one of the chairs aligned along one row that ran the length of the conference room. I remember Ron looking pensive, like some people do as a roller coaster pulls out from the gate and begins its long climb into the sky. I assumed at the time (and today as well) that Ron had seen all of this information before, perhaps the entire reel of video Paul was about to reveal. But the more I thought about it the more Ron’s introspective expression made sense. It was as if watching the opening credits of a horror movie, knowing that sensations of fright were impending. Only in this case, the imagery wouldn’t be fiction but quite real, horrifically real in fact, and that seven of the country’s bravest were riding on that roller coaster, unaware that doom and catastrophe were only minutes away. I think everyone in the room shared in that sensation.

What Paul presented was a montage of videos. Some were taken in the infrared, showing white heat sources against the black background of cold sky. Others were visual imagery enhanced to bring out details. If you recall, Columbia was to land in Florida early in the morning, only a bit after 9 am local at KSC. As the ship and her crew flew over the western United States the sun had not yet risen and, consequently, Columbia was in darkness.

The first video was when Columbia was still out over the Pacific Ocean. The imagery revealed the very distinctive white silhouette of Columbia, a hypersonic reentry vehicle with a sharpened nose and stubby lifting-body wings perched in front of a squat tail. The outline of the ship, like all Shuttle Orbiters, was as familiar to us as the shape of our own hands. Inserted at the bottom of the image was a time reference, counting off hours, minutes, seconds, and milliseconds. Also embossed in white letters on the image was the latitude, longitude and, if available, location from which the imagery was taken.

Columbia stayed solidly within the center of the image frame, whether due to the use of image-stabilized cameras of due to post-processing of the video wasn’t clear but didn’t really matter. Paul stood to the side of the projection screens holding a long wooden pointer, narrating the scene as time clocked by. At first all we could see was Columbia, but within a few seconds a small white “blip” appeared behind her wings and trailed off into the distance into her wake. That was, Paul mentioned, the first instance we knew of a piece of Columbia coming off during her reentry.

Several more videos followed, each correlated to the time of the event and Columbia’s location over the ground. She lost a part or two over the coast of California, then again each over Arizona, New Mexico, and finally Texas. Every time, a transient would shed from the amorphous outline of the spacecraft and retreat behind her and out of view. On two occasions, these events were associated by a bright flash or flare, which momentarily overcame the image and made the two—Columbia and the liberated part—appear as one.

What were the parts coming off of Columbia? Most likely they were thermal insulation tiles, the black silica protective blocks that covered most of the Orbiter’s outer surface. Whether these parts came from the front of Columbia’s left wing, from the rear where the elevons and hydraulic actuators were located, or other areas of the spacecraft was impossible to tell—there simply wasn’t enough resolution in the images to ascertain a specific location. The flares or flashes, it was surmised, were due to interaction with the rarified but superheated gases of the atmosphere, momentarily causing the part to become incandescent before leaving the field of view. The flashes stimulated expressions of amazement from many in the room, with frequent exhalations of “Whoa” and “My Gosh!”

When Paul finished, he replayed the roll of imagery a second time, and then once more so that the room was clear on the evidence that had been gathered. Although I didn’t take a poll of those in attendance, one thing became very clear in my mind—the disintegration of Columbia didn’t start with the loss of communication and Commander Husband’s last recorded comment. It actually began many minutes before that point, a thousand or more miles away from the east Texas woods where the majority of debris landed. Whatever was going on with the spacecraft was slowly playing out as soon she hit Earth’s atmosphere. Columbia began to disintegrate while she was still flying before anyone—the flight controllers on the ground or the crew within—had any inklings that something was amiss. Whatever process or physics had been acting on the ship, devouring it and reducing its integrity, began long before the ship’s final, catastrophic breakup. Heat and searing gasses had ostensibly found their way into Columbia’s left wing, wreaking havoc with its aluminum structure, and irrevocably sealing the destiny of the astronauts inside.

But to know that this was occurring 10 or more minutes before the final destruction was a sobering thought to me, certainly, and I suspect also to the rest of those in the room that day. Even if this information was available to those in Mission Control or the crew on the flight deck and middeck, it’s doubtful that anything could have been done. Columbia at that point was committed to reentering the atmosphere, trading energy for altitude, and nothing on the spacecraft was going to prevent the continuation of that plunge. The vehicle at this point was a hypersonic glider on a somewhat controlled ballistic trajectory, impotent to significantly altering its course or rising itself back above the atmosphere. And yet, while all of that is true, watching it happen, even months later, was a very cold reckoning.

Life Lesson for Engineers, Managers, and Leaders

Failures often play out over time.

Rarely, in my experience, are disasters without any previous warnings, occurring instantaneously and without pre-condition. That certainly can happen, of course, and does happen. One example might be the TWA Flight 800 disaster in 1996 when fuel vapors combined with an electrical ignition source in an empty fuel tank and led to the explosion of the aircraft on departure from the John F. Kennedy International Airport. Those events probably occurred within a split second and disaster ensued.

But take, for example, the case of the Space Shuttle Challenger tragedy. That tragedy was due to hot gases within one of the Shuttle’s solid rocket motors escaping, burning past the rubber O-rings in an aft field joint and allowing those destructive forces to degrade the structure that connected the SRB to the stack’s ET. Seventy-three seconds into the flight of STS-51L that attachment structure failed, rotating the giant SRB about its still-connected forward attachment point and causing the rupture of the ET. When the oxygen and hydrogen were released into the atmosphere they exploded and Challenger was destroyed. Aerodynamic forces rather than the resultant explosion are what actually tore Challenger apart, but the end result was the same, and seven astronauts died as a result.

But on that event, evidence of an impending disaster was first seen on tracking cameras (film, in those days) 15 seconds before the explosion, at a Mission Elapsed Time of 58 seconds into flight. That was when the flames escaping the aft field joint were first visible and it took only 15 seconds for those hot gases to reduce the integrity of the attachment point.

What’s more, video of the launch ignition itself, when the SRBs were first ignited and the stack still on the mobile launch platform, clearly show plumes of black smoke puffing from the exact same location on the aft field joint where 58 seconds later the flames would appear. The destruction of Challenger began not when the attachment mechanism broke and the ET breached, nor even when flames first began to erode the structural integrity of that mechanism. No, the failure began at launch, 73 seconds before Challenger’s final destruction. The end effects were instantaneous, but the causes began well before the final destructive outcome.

Another example—one of my favorites from the annals of NASA engineering—is the oxygen tank explosion on Apollo 13. It’s common knowledge that in April 1970, while enroute to the Moon, approximately 200,000 miles away from Earth, an oxygen tank exploded in the Service Module of the Apollo 13 spacecraft, initiating a harrowing fight for survival for the three astronauts aboard. Supported by the dedicated and tireless cadres on the ground in Mission Control and other locations, the crew safely returned home four days later after a swing around the Moon and an endurance test in a cold, darkened and nearly dead spacecraft. If you’ve watched Ron Howard’s 1995 film about the event, then you know all the basics.

But the genesis of that in-flight accident actually can be traced back to months, if not years, before the explosion. Years before, during construction of the Service Module at the North American Aviation facility near Los Angeles, a decision was made to modify the top of the oxygen tank, an area referred to as the “dome” where piping and electrical wires enter and exit the sealed pressure vessel, to increase its resistance to electromagnetic interference. Originally installed in the Apollo 10 Service Module, the tank was removed for the modification and reinstalled into the Service Module destined for Apollo 13.

Unfortunately, during tank removal, a slight mishap occurred. The oxygen tanks were mounted in a pie-shaped shelf, which fit into one of six triangular bays in the cylindrical Service module. These shelfs were attached by dozens of bolts surrounding the shelf’s perimeter. All the bolts were removed—except for one. After a fixture was lowered from the factory ceiling to lift the shelf out and the extraction began, the shelf lifted an inch or two but, still held affirm by that last remaining bolt, refused to move further. The fixture broke and the shelf slammed back into place. Inspections didn’t reveal any obvious damage, so the errant bolt was removed, the shelf and tank extracted, and the modifications were made prior to its reinstallation into the Apollo 13 spacecraft.

What wasn’t known at the time, however, was that a small Teflon tube inside the tank which connected the pipe used to fill the tank with cryogenic oxygen and another pipe within the tank has dislodged. Since the tank was already welded shut, none of this was known.

A month before the launch of Apollo 13, when the massive Saturn 5 rocket and the Apollo 13 spacecraft were sitting out on the pad at Launch Complex 39, a routine prelaunch test was conducted as verification that the rocket and spacecraft were ready for the mission. Part of this test was to fill the tanks of oxygen and hydrogen used by the vehicle’s fuel cells to generate electricity and water during the mission. Filling the damaged oxygen tank went as expected, but when it was time to drain the tanks again, this particular tank refused to give up its liquid oxygen. The problem, of course, was the dislodged Teflon tube inside the tank, which prevented the oxygen from flowing back out into the pipe and out of the spacecraft. Try as they might, the technicians running the operation couldn’t get the tank to drain. No problem, they decided, all they needed to do was to cycle on the heaters inside the oxygen tank—used to maintain pressure of the cryogen fluid during the mission—and the heat would “boil off” the super cold oxygen, eventually draining the tank. The heater was protected by a thermostat that would remove power before it got too hot, so the operation was considered safe.

But one thing was failed to be taken into account. Sometime previously, NASA had informed all of the contractors who built equipment used in the spacecraft to make sure everything was compatible with the 65 VDC power used at the KSC, in addition to the normal 28 VDC spacecraft power generated by the onboard fuel cells. All subcontractors and equipment providers complied, but for some reason the company that built the thermostats in the cryogenic oxygen tanks didn’t get the message. Thus, when the operation commenced to turn on the tank heaters in order to boil off the liquid oxygen, the thermostat was met with the higher voltage and fused closed.

The operation to boil off the oxygen took some hours. Technicians commanded the heaters on and assumed the thermostat would prevent temperatures from rising too high, which could damage the internal wiring, but with the thermostat fused closed, power was never removed. As a result, it is estimated that the heaters go up to 1,000°F, which damaged much of the Teflon insulation coating that surrounded the wiring. When the operation was completed, newly exposed wiring existed within a tank that contained pure oxygen. In other words, it was now a bomb just waiting to go off.

So, the initiating event of the Apollo 13 explosion could be traced back to events that occurred on the factory floor years before that fateful day 200,000 miles away from Earth. In fact, even during the mission, subtle indications of electrical shorting in the wiring inside the tank were evidenced to flight controllers in Mission Control, although the full extent of the scenario wasn’t caught until after the mission was over and a full investigation revealed the causes.

As for Columbia, when the ship went out of control and the violent destruction that ripped the spacecraft into tens of thousands of pieces and killed the crew was wrought, much of that happened quickly (reference Chapter 12 for more). But Columbia’s loss didn’t occur in an instant—it was a more slow, pernicious process that began 10, 15, or more minutes before when the ship first struck the outer layers of Earth’s atmosphere. And once initiated, it played out innocuously for some time before physics finally caught up and Columbia lost the battle.

Many things, in my experience, fail in this manner. Connecting the dots of seemingly separate indications is the challenge for all. For engineers, tracing backward from the time of an accident is necessary to identify the proximate causes that led to it. The same is true for managers, who, upon evaluation, may be able to see the signs of a pending management catastrophe in the making. Leaders especially need to stay cognizant of all the potential warning signs, both technically and organizationally, so that direction and resources can be shifted to best prevent the event from ever occurring.

But the point is that all of these cases reveal that these three disasters didn’t happen in a split second—they metastasized over time until the malevolency of failure finally succumbed all. This can provide engineers, managers, and leaders windows of opportunity to take action and ward off disaster. It is acknowledged that situations aren’t always this simple or clear cut, that failures can be hidden many layers deep and can portray themselves in other guises before they are fully revealed. Yes, life is not simple. But observant engineers, managers, and leaders often have these windows of opportunity available to them. The trick, of course, it to remain observant.

1 Steven Hirshorn, MRT notes, Feb. 3, 2003.

2 The idea was that ATC assets could provide a higher fidelity to the radar data than these 12 images.

3 Steven Hirshorn, MRT notes, Feb. 6, 2003.

4 Ibid, Feb 7, 2003.

5 Steven Hirshorn, MRT notes, Feb. 18, 2003.

Chapter 10

Two Views of the Impact Damage to Columbia’s Left Wing

Sometimes Seeing is Not Believing

Monday, February 10, 2003, was the first mention of a report from NORAD at the MRT that evidently showed that a piece of “something” may have departed from Columbia on the second day of the mission (what we colloquially called Flight Day 2). Paul Hill from MOD emphasized that the report was preliminary and that the Air Force was giving it top priority, hard at work on resolving whether any details could be gleaned from the information. Paul said whatever the object was asymmetrical and that it was estimated to be approximately 50 cm in size. Ralph Roe, the OPO manager, opined that a NASA Tiger Team should be created to investigate this object more. Paul continued that we did have downlink video from Columbia’s payload bay (Shuttle Orbiters had four close-circuit television cameras—named A through D—mounted in each corner of the payload bay) from a period about one hour after the reported time of the object’s departure, and those videos were under review. Paul concluded the topic by mentioning that telemetry recorded on the ground from the spacecraft’s Inertial Measurement Units (IMU) was being reviewed to see if any forces were imparted to Columbia at the recorded time, including crew activity. When an astronaut moves around within the Shuttle’s crew cabin they can push off with their feet and “fly” to another destination, then consequently stop their motion by landing on the opposite bulkhead. Amazingly, these slight perturbations on the vehicle can be measured by the IMUs. The purpose of analyzing this data was to ascertain if there might have been a force that, even very slightly, shook the vehicle and induced the errant object to depart from Columbia.1

The next day, February 11, Paul reported that the Air Force was still studying the data but that analysis from engineers within JSC gave confidence that the freely orbiting object could in fact be back-propagated directly to Columbia. That is, given its motion and trajectory, these engineers were confident that the object likely did originate from Columbia. The preliminary analysis still suggested an asymmetrical shape with a separation velocity that might have been in the range of between one and five meters per second. Refinement of the data showed a departing velocity closer to one meter per second, which was in family with previous Shuttle history of liberated objects.2

On February 12, the MRT reported that engineers at JSC had provided material densities to the Air Force of a variety of Orbiter objects such as the Reinforced Carbon-Carbon (RCC) wing leading edge panels, thermal blankets made of beta cloth, and other prospective objects, for better radar interpretation.3

Then, on February 24, a final report on this event was delivered to NASA. USSPACECOM (United States Space Command) determined that an unidentified object was released from Columbia at approximately GMT 17/15:59, or a Mission Elapsed Time (MET) of 1 day, 0 hours, and 29 minutes. Upon departing from Columbia, the debris was positively tracked until GMT 19/21:46, two days later, with reentry into Earth’s atmosphere projected to occur early the following day. The radar cross section varied over time, from 0.04 square meters to 0.5 square meters, indicating the object was tumbling, with an estimated rotational period of seven seconds, but the size of the object itself was estimated to be on the order of one to two feet. The analysis suggested that the object was released radially downward (that it, toward Earth). At the exact time of release, no Orbiter activities could be correlated with the event. No Reaction Control System (RCS) jets had fired at the time—the Orbiter was only using its small vernier thrusters to maintain attitude and the closest of those occurred five minutes after the estimated release time. The LiOH cannister changeout was resolved to have occurred an hour before, as was a purge of the vehicle’s fuel cells, which releases a small amount of extra oxygen and hydrogen through the electricity producing units to remove contaminants from their reaction cells (as with the LiOH change out, this is a routine on-orbit activity). There were no known crew operations at the time that could explain the liberation of the object—all crewmembers were awake and in the process of handing over from one team to the next (Columbia’s science mission necessitated 24-hour operations, so the crew was split into two 12-hour shifts, a Blue shift and a Red shift). The door to one of the small payloads mounted in the payload bay was opened at 17/06:05 GMT, approximately nine minutes before the estimated release time, but that imparted essentially no forces or rates into Columbia. The analysis team reviewed payload bay video scenes both before and after event, looking for any differences such as dislodged thermal blankets or covers, but this review showed no changes to Columbia either.4

All in all, beyond being confident that something was released from Columbia slightly over 24 hours into her mission and having an imprecise idea of the object’s size and motion, nothing could be correlated to Columbia or her crew that could explain why such an object was released.

So, why is all of this significant? During mid- to late-February 2003, NASA was still the early stages of its investigation into what happened to Columbia. Debris was still being recovered from the field, much of which hadn’t yet been transported to KSC or been analyzed by hardware and forensic experts. The fault tree—an analytical tool common to engineers that maps out all potential causes of an event and then methodically and objectively eliminates as many as possible until the root cause of the event is determined—was still being constructed. Nothing had yet been ruled out as a contributor to the destruction of Columbia and loss of her crew. And yet, even in these early stages, some things were known and a few were suspected. We knew that the left wing had been struck by a mass of bipod ramp foam from the ET during launch. We knew that various temperature and status sensors in the left wing had failed only minutes before we lost all communications with Columbia. We knew that at the time Columbia was slamming into the upper reaches of Earth’s atmosphere at reentry velocities that the frictional temperatures on the vehicle’s underside and leading edges were approaching 3,000°F. Taken together, it was logical to conclude that searing hot plasma had somehow entered Columbia’s left wing and wrought the destruction that was soon to follow.

Perhaps, then, as a result of the foam strike some portion of the left wing, a piece of Columbia’s structure perhaps, had been knocked loose, disrupted, broken, or wrenched free, and whatever that structure was had finally departed the vehicle about 24 hours into the mission of STS-107. There was, of course, no conclusive evidence that this had occurred, but the seeming confluence of these indicators certainly was on people’s minds. It was not unheard of for certain objects to break loose from the Orbiter’s while in flight, limited mostly to an occasional thermal blanket, but when those events occurred there was routinely some demonstrative evidence of it, an exposed area of underlying structure or other visual evidence which confirmed the source of the errant object. On STS-107, however, nothing within view seemed to be out of place.

As a component of the SSP, the OPO owned a portion of JSC Building 1’s sixth-floor real estate, occupying about one-fourth of the floor space along the west side of the building. Here the offices of the manager of the OPO, their deputy (or deputies) and administrative staff, MER managers, systems experts and others overlooked the JSC campus facing in the direction of the city of Houston. Just outside the main office suite of the OPO was a semi-large conference room, just a step from the OPO’s leadership, where the group held official meetings. The weekly Configuration Control Board was held in this conference room, where multitudes of technical topics would be brought to the OPO manager for decision on a weekly basis. The room was composed of a long wooden table which could accommodate perhaps 15 or 20 people. To each side rung rows of blue fabric-covered chairs—a single row along one wall and two rows deep on the other—offering seating for another 30 or so participants. A pair of six-foot projector screens hung on the short wall farthest from the OPO leadership’s office suite. Although the walls maintained the same bland eggnog coloration of the rest of the building, some color was added to this room with the inclusion of a lines of plaques that ran along the top of the walls just beneath the ceiling. Each plaque, blue in color with laser-etched white lettering, was dedicated to every individual Shuttle mission flown to date. They included a representation of the mission patch above statements of the launch date, landing date, and primary payload. At the completion of each Shuttle mission another plaque would be added to the row, which by the time of STS-107 filled two walls and had migrated onto a third.

Much of the activity and discussions I was involved with as the MOD representative to the Orbiter Project Office occurred in this conference room, designated Room 620.

As debris from Columbia was located, bagged, categorized, and eventually transported to KSC, a strong case emerged that the hot gases of reentry had gained access to Columbia’s left wing around the area of the leading edge’s RCC panels number 8 and 9. This is the area where the wing begins to extend outward from the Orbiter’s fuselage. RCC panels 1 through 6 largely run aft along the side of the fuselage. Beginning with Panel 7, they then are mounted in a more outbound direction, defining the front of the wing, until approximately Panels 19 or 20 when they resume an aft ward placement again all the way to the tips of the wing’s elevon control surfaces. Panels 8 and 9 were considered critical because this is where the wing’s leading edge would experience the hottest temperatures during reentry. This “bend” in the wing protected by Panels 8 and 9 was a critical juncture as airflow would cause the air molecules to impinge on the Orbiter more in this area than on any other. As such, the carbon composite from which the RCC panels were constructed was thicker in this area than on the other panels and considered more resilient to the high temperatures of reentry.

Because the airflow during entry was concentrated more on those panels than the others, it was only logical that the airflow during launch may have also directed the large piece of foam that came off the ET to be directed to this location as well. Aero-analysis, in fact, confirmed this prediction.

Most of Columbia’s Panels 8 and 9, unfortunately, were never recovered. However, we had confidence that the ET foam struck in the critical area of Panels 8 and 9. But that wasn’t all.

Figure 24. The approximate location of the foam impact is circled on this image from Columbia’s launch. (NASA Image KSC-03PP0142)

Portions of the upper surface of the wing in the area of Panels 8 and 9 had been recovered. A bent, twisted piece of wreckage from this area was discovered in a field, maybe five feet long and two feet wide, with portions of the Shuttle’s black protective thermal tiles still attached. The underlying aluminum structure showed clear evidence of extreme heating and melting. Splatters of liquified aluminum could be seen sprayed on the metal ribs and plating. Other portions were deformed in the way that only extreme heating can produce. Of course, when the vehicle broke up and aerodynamic forces ripped it apart, Columbia was still in the period of intense heating, and some of this evidence could be from exposure to those hot gases. However, the evidence revealed a directed flow of heating, not the sort of random scorching and melting one would expect for wreckage tumbling in free fall, but rather an impingement while Columbia was still intact in controlled flight.

But the real kicker came from a single piece of insulation tile which, over time, became endowed with a certain notoriety. Hundreds of Columbia’s tiles were located in the woods and forests of Texas and Louisiana. While every tile—tens of thousands of them were used on the Orbiters—were individually marked with identification numbers and all were unique in size and shape, not all tiles located could be positively identified due to the violent disassembly of Columbia. Many were found on the ground broken with the identifier numbers missing. But a few tiles along the upper surface of the left wing were identified and one, in particular, was critical to the investigation.5

This tile was located along the wing’s upper surface right at the transition from the RCC Panel to the start of the black thermal protection tiles. Most tiles in this location were never located, either having been consumed in the conflagration that surrounded Columbia or, more likely, simply never having been found in the field. But regardless, this one had been located, and once it was identified its proximity to Panels 8 and 9 was quickly realized. What’s more, the tile showed clear and unmistakable evidence of exposure to persistent high-temperature heating. The silica-based tile was melted, or in the terminology used for this piece of forensic evidence, it was “slumped” in the way that only a concentrated application of heat can produce. The configuration of this slumping was consistent with the theory that hot gases had entered in the region of Panels 8 and 9, coursing through the internal structure of the left wing but also, in part, escaping along the upper surface at the interface between RCC panel and tile and flowing aft over these components. Based on extensive analysis, this tile’s slumping effect was one of the most direct and compelling proof we had of what had happened to Columbia.

Nearly daily the OPO would gather in Conference Room 620, me included, and we would review this kind of evidence. As pieces of Columbia’s wreckage were brought in and their location on the Orbiter identified, photos would be taken, inserted into PowerPoint charts, and shared with the OPO via presentations. The found debris could come from all aspects of the spacecraft, and as it was early in the investigation, everything on the vehicle was suspect. We viewed tanks and avionics boxes, portions of structure, wire harnesses and twisted pieces down to the level of bolts and connectors, as long as they were identified. Almost never, however, did we view any portion of the crew module or, certainly, any aspect of the astronauts remains, although I do recall one photo of one of the crew members empty helmets, sitting on its side encased with mud and dirt, which sent a shiver down the spines of everyone present.

Over time, though, much of the presented images began to focus more and more on the RCC Panels, the mechanism that connected the panels to the leading-edge spar (called horse collars), the RCC “T-seals” that fill the gap between panels, and the thermal tiles surrounding this region of the left wing. Evidence supporting the hypothesis, in addition to various computer analyses, slowly but inevitably built a story of the destruction of Columbia that began with a liberated piece of ET foam striking the spacecraft’s left wing.

But in good engineering fashion, forensic evidence alone was insufficient to verify the hypothesis. We needed to do more. We needed to, in fact, test the hypothesis, not through analysis or computer modelling but through an actual demonstration that replicated the conditions as closely as possible. Only in this way could we be confident that we truly understood what had transpired.

To this end, the CAIB contracted with a company named the Southwest Research Institute, or SwRI, a research and development institution located near San Antonio, Texas. Founded in 1947, SwRI has supported both government and industry in advancing scientific understanding and applying technology and had the means to construct an apparatus to demonstrate what exactly could happen to an RCC panel when struck by a piece of ET Tank foam.6 The test was performed in July 2003.

Assigned to lead this effort from NASA was a JSC engineer named Justin Kerr who had managed JSC’s Hypervelocity Impact Test Facility, where he managed test programs to investigate the effects of micrometeoroid and orbital debris (MMOD) impacts on spacecraft critical hardware.7 With a football player’s physique and, at the time of the accident, over 11 years’ experience in high-velocity impacts on spacecraft, Justin was the right person for the job.

Figure 25. JSC engineers and subsystems managers Rodney Rocha (left), Don Curry (middle) along with Boeing engineer Darwin Moon, conversing in the Mission Evaluation Room (MER). (NASA Image JSC2006-E-26699)

Another person critical to this demonstration was Don Curry, a somewhat curmudgeonly yet brilliant and experienced engineer and the subsystem manager for the RCC panels, Don had been a member of JSC since its creation in 1962, working on programs from Mercury to Shuttle, and was considered NASA’s foremost expert on the RCC panels.

Portions of real RCC panels taken from Space Shuttle Discovery were attached to an aluminum fixture that approximated the shape and contours of Columbia’s left wing, duplicating the dimensions of the area of the “bend” near Panels 7 through 9. The panels then were attached, lined on their interior surface with networks of strain, pressure, and acceleration sensors to record every aspect of the test impact.

A 1.67-foot rectangular block of ET Tank foam was created specifically for this test. Nearly a foot long and six inches wide, the object approximated what was believed to be the size of foam shed from the bipod ramp during STS-107’s launch.8 The foam was loaded into a high-pressure pneumatic air gun which would shoot the object at the RCC panels at a speed approximating that which Columbia hit the foam—approximately 500 mph. (Note: Contrary to popular belief, which is that the foam broke off the ET bipod ramp, came down, and struck Columbia, engineering analysis indicated that Shuttle stack, still thrusting upwards, rose up and impacted the debris. At the time the foam was liberated the combined stack was travelling at approximately 1,500 mph. Almost immediately, air friction slowed the foam down to approximately 1,000 mph. However, Columbia was still accelerating upward due to the combined power of the twin SRBs and three main engines. When the foam and Columbia met, physics dictated that the only parameter that was important was the relative difference between the foam and Columbia, or a speed of approximately 500 mph).

When the test was performed, the results were nothing short of dramatic. The entire event from firing of the air gun to impact on the RCC panels occurred in less than a second, but after the smoke had cleared a large, gaping hole had formed in the panel.

Figure 26. The test apparatus setup at Southwest Research Institute, demonstrating whether a block of ET foam could damage the Orbiter’s wing leading edge RCC panels. After a test, the hole in the panel caused by the ET foam is visible. (CAIB Photos by Rick Stiles)

Two days after the demonstration both Justin and Don returned to JSC, and we all gathered in Conference Room 620 to review videos of the test. The images were projected onto both screens at the front of the room so everyone, seated on both sides of the conference room, would have a good view. At first the video was played in real time—that is, at full speed. We had all known what had occurred, but when the foam punched through the panel there were explanations of “Whoa!,” “That’s incredible.” “Oh my God.” and a few more, expletive-tinged phrases. Eyes were held wide in shock of the apparent ease with which this lightweight, seemingly innocuous, and airy foam could punch a hole through the hardened structure of a composite panel. It was difficult to believe and contradicted engineering sensibilities, and yet the evidence being projected onto the screen right in front of us was irrefutable. There was no denying what we had just seen. Apparently, as was mentioned in Chapter 7, seeing was believing.

Next, the video was played in super–slow motion, from the high-pressure release of the foam from the pneumatic air gun, through its trajectory in the air as it closed the distance to the test apparatus, and to the moment it struck RCC Panel 8. The foam didn’t strike through the panel; instead, it glanced off the composite panel, contorted, and deflected to fly off in a slightly different direction. But once it left the view of the camera, the area of RCC it had struck simply crumbled and fell inward, leaving a gaping hole as evidence of the strike.

We spent, literally, an hour or more playing and replaying the video again and again. We viewed it at various speeds, stopping it occasionally to get a closer look, and then resuming the motion. Justin Kerr narrated the events, pointing out aspects of the foam’s trajectory and the reaction of the panels to the strike. The repetition was neither annoying nor boring—we were all transfixed on the video, scanning it for all the engineering data it provided, speculating and making calculated guesses of the force involved, the mechanics of the dissolution of the panel, and the physics underlying the event. But through all of this conjecture, I distinctly remember Don Curry, the RCC Panels subsystem manager, sitting across the table from me and shaking his head. Don knew these panels better than anyone, more intimately than the company that produced them. This was his bread and butter and had been the focus of his career for years. Even given the video we all were watching; Don apparently wasn’t convinced what we saw was precisely what happened on Columbia. In Don’s mind, having his intimate knowledge of composites, the impregnated epoxy that held it all together and the material properties of the entire panel structure, Don felt it was much more likely that the foam impact produced cracks in the panel, not the gaping hole we saw in the SwRI test. Cracks would have been sufficient to allow the plasma of entry to enter the wing structure and wreak the havoc that destroyed Columbia. As far as I know, Don never wavered from his belief that the panels on Columbia were cracked, not punched through with a hole.

Almost immediately, the news media picked up the video roll and began playing it on CNN, the evening news, and other outlets. Newspapers across the country opened the following day with a spread of the hole in the RCC apparatus on the front page. Countless sources and organizations declared that the test was a “smoking gun,” irrefutable evidence of exactly what had happened to Columbia a minute and a half into its flight when it was struck by a piece of ET foam. Their clarion call was, essentially, “case closed.”

Given Don Currey’s perspectives, I wasn’t so sure.

Life Lesson for Engineers, Managers, and Leaders

Sometimes seeing is not believing.

Most of the opinions I heard or read on the root technical cause of the Columbia accident were generated by that single video and photographs taken after the SwRI test. The CAIB report, even, focused prominently on this test. To many, that video alone was sufficient. It was enough to declare the foam as the guilty culprit and a large hole in a RCC panel as the breach that led quickly to Columbia’s demise and the death of seven astronauts. I heard the same not just from outside NASA but from inside the agency as well, even from colleagues at JSC. Seeing, for them, was good enough.

Was it plausible that what happened in San Antonio was exactly what occurred to Columbia? Sure. Is it possible that the object NORAD had tracked departing from Columbia on Flight Day 2 was a section of an RCC panel that had been dislodged by the foam impact on launch day? Absolutely.

But I’ve been haunted to this day by the thoughts of Don Curry. Barely anyone on this planet was a greater expert in Shuttle RCC panels and, to Don, the video was apparently not sufficient to state unequivocally that a hole was punched into Columbia’s left wing. It was evidently more plausible to him that instead the impact cracked the RCC. Ultimately, I suppose, it doesn’t make a difference; in the end the ship’s protective outer coating was breached and searing hot gasses led to her destruction. But to me it does matter, and to engineers, managers, and leaders it should matter. We deal in likelihoods and probabilities all the time and rarely, if ever, are we offered unequivocal proof. And even in something as seemingly irrefutable as the SwRI video, we should always hold open the possibility that something else could be a cause, that alternate perspectives may still exist.

When building fault trees to explain a mishap, candidate scenarios are often scratched out because they prove to be implausible or remote.9 That’s fine. A systematic, methodical search that narrows down options to a final hypothesis is part of the process. As Arthur Conan Doyle alluded, in the guise of his protagonist Sherlock Holmes, When you have eliminated the impossible, whatever remains, however improbable, must be the truth. That may very well be true for gumshoe 19th-century investigators, and likely for fault trees investigators too. But caution should be applied when focusing too quickly on any one scenario without fully exhausting the others. That, to me, is the lesson here.

1 Steven Hirshorn, MRT notes, Feb. 10, 2003. Note: the analysis did show a small acceleration to the Orbiter around the time in question and was correlated to the crew changing out a Lithium-Hydroxide (or LiOH) cannister used to scrub the atmosphere in the crew cabin of carbon dioxide—a routine event that occurred twice a day.

2 Steven Hirshorn, MRT notes, Feb. 11, 2003.

3 Steven Hirshorn, MRT notes, Feb. 12, 2003.

4 Email from Judy Parnell to Steven Hirshorn et al. sent Feb. 13, 2003.

5 I mentioned this tile in Chapter 7, seeing it in all its infamy laid out onto the platform in the SLF’s hangar with other tiles originating from the left wing’s upper surface in the region of RCC Panels 8 and 9.

6 “History.” Southwest Research Institute. https://www.swri.org/who-we-are/history (accessed May 29, 2024).

7 “Justin H. Kerr – NESC Chief Engineer.” NASA. July 26, 2023. https://www.nasa.gov/nesc/team/Justin-Kerr (accessed Jan. 21, 2025).

8 Edwin L. Fasanella et al., “Test and Analysis Correlation of Form Impact onto Space Shuttle Wing Leading Edge RCC Panel 8.” NASA Document ID: 20040070935. March 5, 2004. https://ntrs.nasa.gov/api/citations/20040070935/downloads/20040070935.pdf (accessed Jan. 21, 2025).

9 Foam damage to the Orbiter was considered implausible by many prior to the accident. As recounted by a peer reviewer to this book, “I remember seeing an interview with Buzz Aldrin who mentioned the probability of that happening, but no one thought it was plausible. They thought it was ridiculous.”

Chapter 11

ET Foam Shedding

Hardware Talks, But You Have to Listen to Hear It

Throughout the history of the Space Shuttle Program, from the very first mission of STS-1 in 1980 straight through to the last flight, STS-135, in 2011—a period of 31 years—small pieces of the ET’s spray-on insulation foam continued to separate and come off while in flight (e.g., during ascent). An obvious question might be why use foam in the first place? Well, think of the ET as a giant thermos bottle. Inside it contained thousands of gallons of both liquid oxygen and liquid hydrogen, stored at cryogenically frigid temperatures and used to provide both the fuel (the thing that burns) and oxidizer (required for things to burn) for the three SSMEs, which help propel the spacecraft to orbit. The foam was an insulating barrier for the cryogens from the relatively warm temperatures of ambient air outside of the tank.

There was actually two layers of insulation designed into the ETs. One was a closed-cell polyurethane foam sprayed onto the tank’s entire acreage. Creatively referred to as Spray-On Foam Insulation (or SOFI, pronounced so-fee), this is what gave the tanks their recognizable burnt-orange color. In addition to the SOFI were areas of denser composite ablator, made of silicon resins and cork called Super Lightweight Ablator (or SLA, pronounced slaw). The SLA dissipated heat by eroding the ablator and was used on areas of the tank that were subjected to extreme heating, such as the aft dome near the main engine exhaust and on protuberances that were exposed to aerodynamic heating such as cable trays. And bipod ramps.1

Both the SOFI and the SLA were designed to meet multiple requirements. These two insulations needed to maintain the cryogen held inside the tanks at temperatures hundreds of degrees below zero Fahrenheit during both the tanking process on the launch pad and during the eight-and-a-half-minute climb to orbit; they had to be durable enough to withstand exposure to Florida’s tempestuous weather at the launch pad for as long as 180 days and, according to the NASA fact sheet, “sand, salt, fog, rain, solar radiation and even fungus.”2 On top of those mandates the insulations needed to be lightweight, for any additional weight carried to orbit was less payload that could be mounted in the Orbiter. The SLA was also used separately to protect certain critical areas on the twin SRBs, which provided most of the lift for the rocket’s first two-and-a-half-minute climb to orbit.

While the SOFI and SLA were undoubtedly elegant solutions from an engineering standpoint, fulfilling all of these difficult requirements, they nonetheless did have an unfortunate tendency to break off during flight. The vast majority was shed in very small pieces, nothing of the size or magnitude that came off and struck Columbia on STS-107. And the vast majority of what did come off never came anywhere near striking the spacecraft but instead was innocently deposited into the slipstream created by this enormous stack, only to flitter downward and most likely land innocuously in the ocean. Still, most Shuttle missions returned home with some amount of damage from particle strikes from the ET foam.

Here’s a story. I began my career in January 1990. While the first mission I was assigned to work (as an On-the-Job Trainee) was STS-31, the launch of the Hubble Space Telescope in April 1990, the previous mission, STS-32, was already on orbit during my first day as a flight controller in MOD. Only 14 months before that was the flight of STS-27 on Space Shuttle Atlantis in December 1988. STS-27 was the second Shuttle mission to fly following the Challenger accident and its four-day mission was dedicated to carrying a classified military payload for the DOD. Many of my flight controller colleagues had worked that mission and while they of course never spoke of the payload or mission objectives (those were classified), they were allowed to discuss the performance of their Shuttle systems. For example, while the power profile of electricity generated by the fuel cells could be classified as it might give insights into the payload activities on-orbit, should a fuel cell or a component of the fuel cell fail while in flight, the details of that failure could be documented in the Shuttle’s In-Flight Anomaly list and used along with all previous flight history in the tracking and management of the hardware. Therefore, some elements could not be discussed, but aspects that pertained to Orbiter itself could be. At the time, STS-27 had become somewhat infamous among flight controllers as the flight where debris shed from the ET and SRB almost prevented the crew’s safe return from space.

On that mission, approximately 85 seconds into flight, some amount of SLA broke off from the nose cap of that stack’s righthand SRB and struck the Orbiter. While this was known, the full extent of the damage was not realized until Atlantis returned home and was inspected. Significant damage to the black thermal tiles that protected the underside of the vehicle could be seen. Large gouges in the tiles revealed the white underlayer, indicating that the SLA struck the spacecraft’s underside at high speed, scraping along and past the tiles like buckshot scouring a wafer cookie. More than 700 tiles were damaged, and at least one was completely missing. Even during the mission, the crew was apparently aware that some amount of damage had been wrought to the underside of their vehicle, and they were concerned for their safety. The commander of the flight, Robert L. “Hoot” Gibson, apocryphally advised his crewmates to relax because, “No use dying all tensed-up.”3

Much later in the Shuttle Program small digital cameras were added to both the ET and the SRB, which provided truly spectacular video imagery of the ride to space. One of those cameras was focused on the intertank area of the ET, the section between the spherical liquid oxygen tank and the long cylindrical liquid hydrogen tank. Once the Shuttle rose above the atmosphere and was surrounded mostly by vacuum, maybe five or six minutes into a mission, this camera would routinely pick up undeniable evidence of an effect called “popcorning.” Like all areas of the ET, the intertank section was covered by SOFI. Once above the atmosphere, small pockets of trapped gas within the foam insulation began to heat and migrate to the surface. This had the effect of causing small pieces of foam, perhaps the size of a fingernail or smaller, to “pop” off the tank. Toward the latter portions of the ascent to orbit, you could actually discern the frequency of this popcorning increase and, true to the name, it looked like corn being heated in a frying pan and snapping into everyone’s favorite movie food. I had watched video of this from multiple Shuttle missions after the imagery was downlinked to the ground.

Figure 27. The intertank region of the external tank. (Photo taken by the author)

Again, the vast majority of this insulation shedding was harmless and never affected the Orbiter, nor did it put the crew in any danger. But of course, the crew was sometimes endangered, and this foam shedding was a condition the Shuttle was never certified to operate in. During the design and development of the Space Shuttle, and over the long years of its operations, nothing was ever supposed to hit the Orbiter, during ascent or at other times. Some of what did was unpreventable—for example, some of the outer thermal panes of the cockpit windows would routinely be replaced between missions due to impact with small orbital debris or micrometeoroids, which had a tendency to leave pockmarks in the transparent glass. That was the price of operating in low-Earth orbit and a known hazard, so the windows were designed to accommodate some amount of high-speed impact. But nothing else was supported to strike the spacecraft, and as such the Shuttle wasn’t designed for such an environment.4

Some years after the Columbia accident, I got to see some of this hardware close up. In February 2008, five years after the Columbia accident, I had the opportunity to travel to KSC with the STS-123 crew and training team and participate in that mission’s TCTD (Terminal Countdown Demonstration Test). The TCDT is fundamentally a dress rehearsal of the final hours of a launch countdown for the flight crew, the pad technicians, and the cadres of controllers in the Launch Control Center. No cryogens are loaded into the Shuttle and the vehicle is not in a launch configuration, but all the steps that would proceed in the last few hours prior to launch were simulated, including loading the crew into the spacecraft. TCDTs were held for every mission, and it was expected for that crew’s training team to accompany them to Florida for the event. At the time I had stopped supporting the Shuttle Program, my flight controller days then over, and was serving at the MOD lead engineer for the Constellation Program, yet my contacts and residence in the Flight Director’s Office allowed me the opportunity to tag along for this TCDT.

This trip proved to be one of the most technically insightful experiences of my entire career. Over the previous 18 years, I had rarely the opportunity to actually see and interact with the hardware for which I was responsible, both as a flight controller and as part of my duties as MOD rep to the OPO. Now, during this TCDT, I was presented with the opportunity to walk around Space Shuttle Discovery as she sat in Orbiter Processing Facility #3—her home while not in space. I could peer at and into the spacecraft as closely as I wanted, crawl through the claustrophobic aft engine compartment at the rear of the vehicle that held the manifolds and plumbing for the three main engines along with electrical buses, APUs, dozens of avionics, and miles of conduit. We donned the protective bunny suits and explored Discovery’s flight deck and middeck, little different from the simulators at JSC where I had spent so much time, but simply knowing that this wasn’t a mockup but the real deal made the experience so vitalizing.

Figure 28. The author on Discovery’s middeck, seated in front of the hatch leading to the spacecraft’s airlock on April 25, 2009. (Photo provided by author)

Figure 29. The author in the Commander’s seat on Discovery’s flight deck on April 25, 2009. (Photo provided by author)

But more relevant to this story, we also got to go out to Launch Complex 39-A where Space Shuttle Endeavour was mounted and pointed skyward, awaiting to be launched on the STS-123 mission. We loaded into cars and drove the same route the crew’s seven astronauts would take on launch day, paralleling the gravel-strewn crawler way to 39-A that most human spaceflight mission have taken since Apollo 8 in 1968 (the remainders launching from the adjacent pad 39-B). It was a foggy day and as we passed the guard gate at the pad’s perimeter and ascended the low sloping hill up to the launch pad itself, much of Endeavour and its ET and SRBs were obscured within wet, moisture-laden clouds. We parked our cars near the entrance to the elevator of the fixed gantry, entered the lift’s steel cage, and rode upwards, feeling much like astronauts.

Our first stop was at the level of the top of the Mobile Launch Platform (MLP), the enormous, square base on which the entire Shuttle stack was firmly affixed and transported from the cavernous and cathedral-like VAB out to the launch pad. About the size of a baseball infield, everything on the MLP is painted a dour, light, battleship gray, bland and industrial in appearance and, on that day, matching the overcast sky above.

We exited from the elevator out onto the platform and were immediately thrust into a world of rocketry. In addition to all the massive piping, electrical cable boxes, and guardrails situated above 30-foot drops to the concrete below, towering above us rose a giant rocket, pointed inevitably upward and reaching toward the sky as if it already knew and yearned for its orbital destination. I had never been this close to a rocket and the experience was nothing less than captivating.

Figure 30. A view of the underside of the ET. (Photo taken by the author)

Figure 31. Looking up at the ET and SRBs. (Photo taken by the author)

We walked directly underneath the bulbous aft dome of the ET and got within perhaps four feet of the towering SRB. Both were perched above a gaping square hole in the MLP, each SRB affixed to the platform only at four points. As I gazed at those attachment points, I recognized the almost incomprehensible stress and strain being transmitted through that metal and steel. Understand that when the combined Shuttle vehicle is stacked onto the MLP it all begins with the SRB’s aft skirt, the flaring structure that surrounds the SRB’s nozzles. This after-skirt is attached to the MLP by four bolts, each roughly a foot in diameter with bolts that are torqued to 75,000 pounds of force. Once the skirts are attached, four large booster segments are bolted atop it, each containing tons of aluminum-perchlorate propellant. The SRBs are finally capped by a nose cone containing parachutes and electronics, completing what unmistakably looks like a sleek rocket.

The stacking procedure would continue with the hanging of the large ET between the twin SRBs and attaching them together. Finally, the 100-ton Orbiter would be lowered and affixed to the ET at three attachment points—one just aft of the nose landing gear door and the two others much further aft towards the rear of the delta-wing belly. Only three points attach the spacecraft to the ET—no more and no less.

But what amazed me that morning was the realization that all of this—Orbiter, ET, twin SRBs—all that weight, was being supported by those eight bolts on the aft skirts, four on each SRB. Eight bolts in total held the entirety of this giant rocket. To be truthful, I was tempted to place my hand on one of those attachment points, called Hold Down posts, close my eyes, and discover if I could feel or somehow sense the loads and forces coursing through that steel. I didn’t, of course, but I almost felt as if in placing my hand flat on the metal that I could.

But back to ET foam. Here I was standing only a few feet from a colossally powerful rocket soaring high above me and as I craned my neck upward to take it all in, the textures and patterns within the ET could not be ignored. When the SOFI is applied it is sprayed on in much the same fashion as home insulation—from a handheld air gun with a technician swiping leftward and rightward, the foam first adhering closely but then blossoming outward like a dry sponge placed in water. The result is a somewhat lumpy texture, more akin to the surface of the moon than a flat, aerodynamic piece of glass one might expect on the outside of a rocket. Technicians can contour this initial surface, sanding it to a more unform grain and even removing entire protuberances and nonconformities. While seen from a distance the ET appears like a smooth bullet; up close it conveys a different impression—one more of art than of science. Up close it is curdy, unrefined, rough, and uneven. During launch, as the ET passes through the aerodynamic pressures of first subsonic and then supersonic regimes, it’s not hard to see why pieces can come flying off.

Later that day we ascended again in the gantry elevator and rose to the other levels of the launch pad. We exited on Level 195 (195 feet about the base of the pad) and traipsed out onto the access arm that led to the entrance of Space Shuttle Endeavor. At the tip of the arm was the eponymous white room, a small, climate-controlled environment where astronauts and technicians prepare the crew for final entry into their spacecraft. I had seen this room literally hundreds of times on close circuit camera during launch attempts. Here launch pad staff helped each crewmember don their parachute harnesses and affix their launch/entry helmets, the final acts of attire before bending down and crawling into the abode of life that will protect them for the next week or two on adventures in space. For us, we were allowed entrance into that hallowed white room one at a time to sit at the threshold of Endeavour and peer inside. We weren’t allowed to enter, but it didn’t matter—we were close enough.

Further up we exited the elevator and navigated within the fixed service structure that surrounded much of Endeavour. This housing forms a protective shell around the spacecraft, warding off the effects of rain, wind, and the surrounding environment. Here we could walk right up to the bipod itself, the forward structural mechanism that connects the Orbiter to the ET. Two solid struts jutted out from attachments points on each side of the tank and came together like a large “V” at a single juncture then attached to the Orbiter just aft of the Nose Landing Gear well. More importantly, though, I could easily see the area where the bipod ramps of SOFI had been applied up to and including Columbia’s STS-107 mission.

Removing the sloping trapezoidal section of foam from the bipod attachment points was one of the first design changes that came from the accident. SOFI was applied to these attachment points and then sculpted into a ramp, contoured from low-to-high in order to make the entire structure more aerodynamic. This aspect of ET design was original to the Shuttle and had flown on every mission beginning with STS-1. After Columbia was lost and the culprit traced back to this piece of foam,
analysis was performed to ascertain whether the foam ramp was in fact actually required. In the end, the analysis indicated we could live without it. The air flow and heating in this area would not be as smooth and controlled as it would with the ramp, but the underlying structure had sufficient margins such that it could be flown with no ramp and still be safe. See the figures on page 157 in Chapter 13 for before and after views.

This, to me, was the crux of this entire trip. Right there before my eyes was the area indicted as the cause of the accident. I know logically and cognitively that this was a different ET, attached to a different Orbiter spacecraft, and separated by five years from the day Columbia lifted off on her last mission. And yet, as if I had been transported to January 16, 2003, the day of STS-107’s launch, I stood staring at this area of modern technology as if it were speaking to me. Had I been standing here five years earlier, even prior to that fateful launch and looking at the bipod attaching Columbia to her propellant tank, I likely wouldn’t have paid it much attention. But now, after the accident, knowing that seven brave people had lost their lives in a horrible accident, the sight filled me with a sense of after-shadowing (the opposite of foreshadowing). Or rather, it was like viewing the Titanic in her berth in the Belfast shipyard where she was built, walking along her bow, and gazing at the starboard side of her frame near the waterline, knowing that shortly she would brush alongside an iceberg and irrevocably sink in only a few hours. Foreshadowing, after-shadowing, whatever the feeling, it was very real to me.

Life Lesson for Engineers, Managers, and Leaders

Hardware talks, but you have to listen to hear it.

Whether SOFI, SLA or other stuff, pieces were coming off the ET and SRBs since the very first Space Shuttle flight. Overall, the integrated system performed its intended function—delivering the Orbiter and crew to space—but the design never intended for foam shedding to happen. Nor was the Orbiter certified to withstand those impacts. Did the loss of Columbia and her crew through the release of the ET’s bipod ramp impacting and damaging the spacecraft’s left wing occur without warning? In my opinion, No. As I just stated, insulation had been coming off ETs and SRBs for years. The hardware was talking, giving us warning signs, but we didn’t listen to it.

Harken back to the Space Shuttle Challenger accident in 1986. The cause of that accident was searing hot gasses escaping past both O-ring seals in the case joint between rocket segments. These gases then impinged upon the rear attachment that connected the SRB to the ET, degrading the integrity of the attachment point until it failed, allowing the SRB to rotate about the still functioning forward attach point, rupturing the ET and causing a massive explosion. A quick search of SRB field joint O-ring performance will reveal that hot gas “blow-by” of the primary and even the secondary O-rings had occurred on numerous Shuttle launches prior to Challenger’s STS-51-L mission. The condition was known and most definitely outside of vehicle certification, but we continued to fly with the condition. The hardware was talking, giving us warning signs, but we didn’t listen to it.

Even in the case of the Apollo 13 Service Module oxygen tank explosion, which seemingly happened with no warning whatsoever, we learned at the end of the last chapter that it did not in fact occur without warning. The conditions that led up to the near calamitous explosion occurred both months and years prior to the explosion in a sequence of events that precipitated the accident. Was there any warning that an explosion could have resulted? Well, the normal procedure to remove liquid oxygen from the tank following the countdown test didn’t work and alternate procedures had to be employed, necessitating using the tank’s internal heaters to warm the fluid and boil it off. Was it determined why the normal procedure didn’t work (the Teflon tube inside the tank’s fill & drain line had become displaced)? No, it wasn’t, and we launched one month later with the tank’s actual condition still a mystery. The hardware was talking, giving us warning signs, but we didn’t listen to it.

All of this, I have noticed, applies to any of the cars I have owned over my lifetime. Rarely does a part simply up-and-fail with no warning. Typically, its performance degrades and gives peculiar indications that not everything is well. The indications may be subtle, but in hindsight I usually realize that they were there.

Hardware typically talks. It acts up, gets quirky, sometimes in inexplicable ways that are difficult to track down. But often it lets you know that all is not well. Or in engineering language, not per spec or within certification. The component in question may still be functioning and performing its task within acceptable limits, but the very fact that that performance is off kilter should raise an alarm in most engineers. If we are listening.

There is a seduction in an operational program, or a robotic mission close to launch, to declare acceptable performance as sufficient. “It’s doing its job,” one might say, “so we’re good to go, even if we don’t fully understand what’s going on.” That seduction is real, and managers and leaders can often get tempted to succumb to it. And one might get lucky and no harm comes from that ambiguity. But the consequences of not being lucky, when consciously knowing that something is strangely amiss and not taking action to fully understand the condition, can be great, even catastrophic.

The lesson here is, very simply, that when the hardware talks, please listen to it. Don’t ignore it or rationalize it away as innocuous, or worse, fall into the trap, as sociologist Diane Vaughn put it in her exploration of the Challenger tragedy, of Normalization of Deviance.5 This insidious omission of caution occurs when situations outside of our expectations get normalized as expected and insufficient caution is applied because nothing has yet gone wrong.

Hardware that is operating within specifications normally doesn’t talk; it just operates. But when it does begin to talk, now’s the time to pay attention.

For leaders, the same is true for organizations as it is for hardware to engineers. Organizations have ways of “talking” before total collapse. Warning signs can be given in the guise of repeated performance shortfalls, indications of personnel burnout, diminution of team chemistry, and other behavioral shortcomings. Just as hardware often talks before failing, organizations can as well. To paraphrase the above, when an organization begins to talk, now’s the time to pay attention. Be willing to listen to your organization, and to your people because, compared with hardware, people can actually talk!

1 “Return to Flight Area External Tank Protections System.” NASA-FS-2005-04-10-MSFC. April 2005. https://www.nasa.gov/wp-content/uploads/2016/08/114022main_tps_fs.pdf (accessed Jan 21, 2025).

2 Ibid.

3 Ben Evans, “‘Dying All Tensed-Up’: 30 Years Since the Troubled Secret Mission of STS-27,” AmericaSpace, 2018. https://www.americaspace.com/2018/12/09/dying-all-tensed-up-30-years-since-the-troubled-secret-mission-of-sts-27/ (accessed Jan. 22, 2025). The mission’s commander, Robert “Hoot” Gibson, also offers his thoughts on the incident in the following interview: https://historycollection.jsc.nasa.gov/JSCHistoryPortal/history/oral_histories/GibsonRL/GibsonRL_4-10-18.htm (accessed Jan. 22, 2025). Also, one of the astronauts on that crew, Mike Mullane, recounts some of this story in his interesting biopic Riding Rockets.

4 There are undoubtedly lessons right here for engineers, managers, and leaders, an accounting of which I’ll defer to other dedicated tomes.

5 Diane Vaughan, The Challenger Launch Decision: Risky Technology, Culture, and Deviance at NASA, Enlarged Edition (Chicago: University of Chicago Press, 1996).

Chapter 12

Valiant Attempts to Maintain Control

There is Both Power and Limits to Struggle

I worked more than 50 Shuttle missions over 11 years as an active flight controller: 17 flown on Discovery, 16 on Atlantis, 15 on Endeavour, and 13 on Columbia. Having worked on missions that spanned the Shuttle fleet (Challenger was lost before I began working at NASA), I can tell you that one gets to develop a feel for each vehicle. The Shuttle Orbiter’s may all look the same, and Discovery, Atlantis, and Endeavour did share many similarities (Columbia, being the first, was significantly heavier and didn’t share in some of the more advanced design features that graced her sister spacecraft), but in fact each vehicle developed their own mystique, almost a personality that differentiated each from the other, at least to us flight controllers.

Columbia, for better or worse, gained the reputation in our ranks of being “the clunker,” a term coined by one of my flight controller colleagues. I think this reputation was due to Columbia’s age relative to the rest of the fleet, but for some reason the ship just seemed to pick up more failure and anomalies than the other Orbiters. It was a qualitative assessment—I never checked with OPO and actually counted anomalies across the four spacecraft. And the reputation was undoubtedly unfair as all the ships shared many components in the fleet’s inventory, equipment like main engines, fuel cells, and certain avionics boxes being routinely swapped from one vehicle to another. But still, Columbia’s reputation persisted.

In April 1997 I arrived on console for another prelaunch shift, this one for the STS-83 mission on Columbia. As with many of her missions, this one was dedicated to science and microgravity experiments conducted in the bus-sized pressurized module known as Spacelab mounted in the Orbiter’s payload bay and connected to the crew cabin by a tunnel. The mission—anointed MSL-1, or the first Microgravity Science Laboratory—would spend 14 days on orbit with the crew split into two teams so that science could be conducted around the clock.

During prelaunch shifts the fuel cells were normally started before the majority of the flight control team got on console, so when I showed up the FCR was mostly empty. I logged into the console workstation, brought up my displays, arranged my checklists, got a cup of coffee and settled myself for a long, 10- or 11-hour shift.

After an hour or two the Launch Control Center (LCC) in Florida initiated the procedures to start the three fuel cells, a process of getting the units up to their operating temperatures before connecting them to their respective main buses, the Orbiter’s primary network to distribute electricity. Fuel Cell 1 started nominally and was connected to Main Bus A. The LCC next commanded the pumps and heaters in Fuel Cell 2 to turn on and when all was ready that unit was similarly connected to Main Bus B. All seemed in place, except out of the corner of my eye I noticed one minor parameter that was wildly out of the norm.

Excuse a brief deviation into technical minutiae for a moment. Each fuel cell consisted of 96 individual electricity-generating cells, separated into three sub-stacks of 32 cells each. Combine these 32 cells, each generating about one volt, and connect them together and the sub-stacks created the power for the Orbiter’s 28-volt equipment. Now, fuel cells combine gaseous oxygen and hydrogen together in the presence of a catalyst to produce electricity and water (which the crew can drink). But the oxygen and hydrogen are never directly exposed to each other lest bad things happen, like explosions (as occurred within the Hindenburg in 1937). Rather, inside the fuel cells the gases get married in the presence of the catalyst and a four-step chemical reaction forms molecules of water, in the process creating electrical currents which is fed to the rest of the ship. The fuel cell design feeds the oxygen and hydrogen to separate sides of the 96 reaction cell plates and, theoretically, the two gases never touch directly. But certain failure modes, like a pinhole leak in a reaction cell, can allow the gases to mix directly, and if concentrations get high enough an explosion can result.

Neither the fuel cells nor us flight controllers on the ground can monitor for this potentially catastrophic failure mode directly. Instead, we got a more rudimentary insight called sub-stack delta volts, which compared each side of the 32-cell sub-stacks—16 cells versus 16 cells—and provided to us a single comparison. The theory was that if this mixing was occurring—what we referred to as crossover—then some cells would lose voltage, and that would be apparent in the sub-stack delta volts. Our flight rules dictated that action would be taken to safe a fuel cell (deplete it of oxygen and hydrogen) should this measurement get above 150 millivolts. When Fuel Cell 2 was started and connected to Main B, it read to me something over 200 millivolts and that violated the launch commit criteria.

I recognized the deviation immediately and brought it to the attention of my backroom. No one had ever seen this condition before. When fuel cells are started the sub-stack delta volt readings are routinely in the region of 20 millivolts or so, sometimes a bit higher, but nowhere near the flight rule limit of 150. By this time our shift flight director—Rob Kelso—was also on console and I made him aware of the situation.

Over the course of our shift the reading slowly decreased until it was below 150 millivolts—no longer violating launch commit criteria—at which time our shift handed over to the Ascent Team. By the time of launch the measurement was well within limits and Columbia and her seven crewmembers were sent heavenward for their MSL-1 mission.

Unfortunately, after getting to orbit the measurement continued to trend and within a day it was again approaching 150 millivolts. Should it exceed that level, the flight rules were very clear that to prevent the potential of a catastrophic explosion the fuel cell shall be safed—depleted of its operating oxygen and hydrogen—and considered lost for the remainder of the mission. The flight rules were equally as adamant that with only two fuel cells remaining the mission was to be cut short and terminated as early as possible. And that was exactly what happened, necessitating a return to Earth after only three days after launch, well short of the planned two-week duration.

But the story for me doesn’t end there. The crew checklists include a powerdown procedure in the event that a second fuel cell should fail. The Orbiters can operate on two fuel cells with only minor concessions to power, but if left to a single fuel cell a much more draconian powerdown would need to be effected, requiring large reductions in powered equipment during reentry when landing aids and additional equipment was required. The Loss of 2 Fuel Cells powerdown reduced Orbiter operability to bare minimums and, since it had never been performed in flight before, it was felt necessary to validate the powerdown before the STS-83 crew prepared for reentry. Since Rob Kelso’s shift was not called to return after prelaunch until the fourth day of the two-week mission, the call went to me to participate in the validation session in the Shuttle simulator with another crew of astronauts.

We gathered in the Shuttle Mission Simulator, three astronauts and me, to practice this powerdown procedure and determine if any changes needed to be made. As the flight deck crew included four crewmembers, I was asked to perform the role of that fourth astronaut. We spent hours in the simulator, going through each step of the procedure with the help of a training team monitoring in their own room and playing the role of mission control. As we proceeded through each step of the checklist, we did indeed periodically catch moments when too much power was being demanded from the single fuel cell and the ship’s main bus voltage dropped below flight rule limits. Each time this happened we would circle the offending step, make an adjustment to the procedure, and then press on. In the end we ran through this long and cumbersome powerdown three, maybe four times, just to make sure we got it right.

When we were complete, I ran back to the Control Center, put on my headset, and settled myself into an unoccupied console in the FCR. While Columbia was in orbit and the STS-83 crew preparing to return home the next day, I reviewed our findings with the Entry Team. Some of them, I recall, raised concerns over the powerdown shutting off pieces of their equipment, sometimes requiring that function to operate with zero redundancy. But I was always confident in the knowledge that as uncomfortable as that may have made some of the Entry team, keeping all of the equipment powered would have assuredly undervolted the entire Shuttle electrical system, putting the entire vehicle and crew at risk.

After STS-83 landed, I couldn’t help but feel like I had experienced some great adventure. As portrayed in Ron Howard’s film Apollo 13, after the tank explosion and as mission control was endeavoring to return those astronauts safely to Earth, one of the scions of flight control—John Aaron—helped astronauts in the simulator review a similar powerdown procedure, running into some of the same bus undervolting conditions that we found.1 I felt, in a way, that I was able to carry on the heritage of flight control excellence as had John Aaron, separated on by 27 years.2

Back now to STS-107. Columbia did not have the kind of crash-worthy flight data recorder, more commonly known as a “black box,” that commercial aircraft fly with. None of the Orbiters did. But data from the spacecraft was continuously telemetered to the ground and recorded onboard the vehicles during periods when the spacecraft was out of communications range (and then downlinked when communications had been restored). A Modular Auxiliary Data System, or MADS, was installed in every Orbiter that followed Columbia, but being the first Space Shuttle, Columbia’s data recorder was different. Its recorder was referred to as the OEX, or Orbiter Experiment recorder. Both devices used reels of magnetic tape to record information, but the OEX was procured from a different manufacturer than the later, more advanced MADS recorders. Additionally, being the program’s “test” vehicle, Columbia was instrumented with considerably more sensors—strain, pressure, temperature, and others—used during the early flights to certify the vehicle for operations. While the names of the units were different, their functions and basic designs were quite similar.

After Columbia’s OEX recorder was located in the fields of east Texas and its data was analyzed, it was hoped that the information would provide a gold mine of insight into why the accident occurred. Regrettably, the recorded telemetry proved more useful in ruling out potential causes than in definitively proving beyond a doubt what brought Columbia down. What we saw in the recorded data validated the existing hypothesis that a breach in Columbia’s left wing and entry of hot plasma was the culprit; a hypothesis that had formed the day of the accident based on the existing real-time telemetry available to MCC flight controllers. However, the OEX data did prove useful in more fully developing that hypothesis and evolving it into a full-blown theory. Additionally, the data gave us some insights into Columbia’s and the crew’s last moments, which weren’t available real-time to controllers on February 1, 2003.

In April 2003, I and a few others had gathered in the building 1, eighth-floor office of Jon Harpold, the director of MOD. Investigators had only recently extracted the wealth of information fortunately still available from the OEX recorder and most were furiously parsing it for additional insights. Harpold joined NASA in 1964 and worked extensively on Apollo guidance systems before moving on to the Shuttle Program. Before being elevated to the director’s chair, Jon had been the chief of the Flight Design & Dynamics Division, the home within MOD for those responsible for how the Shuttle flew, the FDOs and the Guidance and Trajectory officers. The Flight Design & Dynamic Division produced all of the Shuttle’s ascent and reentry trajectories, managed the rendezvous planning and execution while in orbit, and provided the corporate knowledge on how this rocket/spacecraft/hypersonic reentry vehicle/glider behaved while swimming in the ocean of Earth’s atmosphere and in orbital spaceflight.

I had brought to the meeting a printout of some of the relevant telemetry recorded on the OEX just before the vehicle broke up. Jon took a quick look at the data points and pressed his finger to one specific parameter. It was the rotational rates sensed by Columbia’s three Inertial Measurement Units, or IMUs. These ingenious gyroscopic devices could sense where the Orbiter was pointed and at what rate it was moving about all three of its body axes—roll, pitch and yaw. It measured accelerations about an inertial reference frame—maintained by a series of gyroscopes—so that the crew and the ground always knew where the spacecraft was pointed.

Jon’s finger pressed against the number that represented the rate at which Columbia was yawing, the movement from side to side. It showed an increment slightly above 15 degrees per second, which was the highest range possible for these yaw sensors. That meant that Columbia was yawing at a rate of at least 15 degrees per second but could have been moving even faster. I remember Jon looking up with sorrow in his eyes and he said, “They were in a flat spin.” Think of Columbia as a frisbee spinning uncontrolled and you get a rough idea of what was occurring.

Using this OEX data and forensic analysis from the debris recovered from Columbia, the most likely explanation for the loss of control that led to the structural breakup of the spacecraft was that the searing hot gasses that entered the left wing impinged upon the triply redundant hydraulic systems that powered Columbia’s aero surfaces—the rudder and the four elevons (two inboard and two outboard) located on the wing’s trailing edges. Without hydraulic power there was no way to move these critical aerosurfaces and, as a result, the vehicle went out of control. This hypothesis was supported by a HYD PRESS fault messages generated by Columbia’s flight computers and annunciated to the crew. Once hydraulic pressure was compromised, there was no way Columbia’s computers or the crew could maintain control. Aerodynamic and thermal stresses did the rest of the work destroying the vehicle.

This all made sense to me and the logic was undeniable. In fact, much of these details are described in a NASA authorized study, the Columbia Crew Survival Investigation Report, which is publicly available.3 And yet, another thought or potential cause for the loss of control continued to haunt me. If you recall from Chapter 7, while viewing the collected debris from Columbia in the SLF Hangar at KSC, portions of the left-hand side of the rudder showed significantly more damage than the right-hand side, almost as if the thermal tile-covered structure had been blasted with pellets from a shotgun. It was easy to theorize that the left wing had been so structurally compromised from the plasma coursing through it that disintegration was inevitable. When that occurred hundreds or thousands of pieces of wing and all its internal components razed backwards and impacted the ship’s rudder and tail, littering them with shrapnel like the explosion of a grenade. I could easily see how the destruction of the left wing, or even a significant portion of it, could have put Columbia into a flat spin. Although there was no denying that at some point Columbia lost all hydraulic pressure, that scenario was, to me, equally as plausible.

Still, the demise of Columbia occurred not all at once but over several minutes, maybe tens of minutes. We know that ground-based imagery showed something was occurring along the left wing well prior to the breakup of the vehicle. As stated in the Columbia Crew Survival Investigation Report, at a GMT of 13:50:30, the first indications of entry heating were recorded on the OEX by temperature sensors in the aft fuselage—a normal occurrence. The vehicle rolled to the right, again a planned maneuver, designed to help the orbiting spacecraft, now interacting with the atmosphere, exchange energy and speed for altitude. These roll reversals occurred on every Shuttle mission and were part of the well-choreographed sequence of maneuvers required for descending back into Earth’s atmosphere.

At GMT 13:52:05, about seven minutes before the loss of signal with Columbia, postflight analysis determined the first clear indication of an off-nominal aerodynamics as something caused the yaw-moment to change. That is, something—most likely drag induced by the still rarefied atmosphere—was pulling Columbia’s nose slightly to the left (what is referred to as a yaw-moment). Nothing out of the ordinary was indicated to either the crew or the ground until a minute later at GMT 13:53:10 when four of the hydraulic return line temperature sensor readings went off-scale low.

Thirty-six seconds later, at GMT 13:53:46, the first observed incident of debris being shed from Columbia was noted by ground-based observers. This debris was most likely a piece of the left wing and was seen in the imagery just aft of the Orbiter’s envelope as a noticeably luminescent section of the plasma trail. Postflight analysis indicates the mass of the debris to be less than 8 pounds. Again, neither the crew nor the MCC were aware that this had occurred.

Figure 32. Columbia’s attitude as it was approaching the California coast. The “roll-right” position is normal for Shuttle reentries. (Figure 1.2-21 in NASA SP-2008-565)

Figure 33. Video capture of the first observed incident of debris being shed from Columbia’s left wing. The Orbiter is traveling from left to right in this image. (Figure 1.2-24 in NASA SP-2008-565)

At GMT 13:54:20, a slow elevon trim change begins, indicating that Columbia has noticed the additional drag and was attempting to compensate for it. Another piece of debris departed the vehicle at GMT 13:54:32. This debris event was far more significant, with an estimated mass of a few hundred pounds, but again no sensor or instrumentation picked this up nor was the crew aware that this debris was lost.

One second later, at GMT 13:54:33, two of the side-mounted reaction jets on Columbia’s nose (R3R & R2R)) began to fire. Whether this was due to an attempt to compensate for the increased drag from the left wing or other cause is not known. However, soon thereafter Columbia performed a role-reversal, moving from its initial left-wing up attitude to one where the left-wing was down, completing the attitude change by GMT 13:56:55. Again, this was a normal maneuver during the reentry profile. Less than a minute later, Columbia was passing above Kirtland Air Force Base in New Mexico when a ground-based camera from the Starfire Optical Range captured an image of the overflying spacecraft. While it had not been planned for Starfire to take this infrared image, they knew that Columbia’s trajectory took the spacecraft over the Air Force base and the opportunity to test their equipment was, evidently, too good to pass up.

Figure 34. Caption: Starfire infrared image as Columbia passed over Kirtland Air Force Base, New Mexico. This photo has been enhanced, and a wireframe representation of the Orbiter has been overlaid. (Figure 1.2-31 in NASA SP-2008-565)

The infrared image captured by the Starfire equipment showed a distinct “bulge” forward of the left wing (Note: the image was taken from the ground with the Orbiter flying above it, so while the bulge may appear on the right wing, it is in fact on the left). What, if anything, that bulge was could not be definitively determined, but it also could not be explained by the normal aerodynamic and aerothermal effects being experienced at the time.

After that, more debris was noted to come off of Columbia. Now her speed had reduced from Mach 25 to nearly Mach 20. At GMT 13:58:03 a sharp elevon trim increase was noted. At this point postflight reconstruction was confident that now Columbia was compensating for the increasingly asymmetric aerodynamic forces being imparted by the damaged left wing.

Twenty second later, a tile was shed from Columbia’s left wing that was located in Littlefield, Texas, the westernmost piece of recovered debris. Now more indications of problems were communicated to both the crew and flight controllers on the ground as the main landing gear tires in the left wing indicated they had lost pressure. This event was annunciated by fault messages available to the crew and to the MCC. OEX data indicated that the crew called up the Fault Summary display on their cockpit monitors (CRT3) to look at the message.

At GMT 13:59:30, both of those yaw reaction jets now began to fire continuously. It is normal for reaction control attitude jets to fire periodically throughout the complex sequence of reentry, including the roll reversals of the vehicle used to help bleed off energy from orbital velocities. What is unusual, however, was for these jets to fire continuously. Normally they would “pulse” or fire for short bursts, imparting momentary force to keep the vehicle in the desired attitude. But continuous firing of two Yaw jets indicated that the aerodynamic authority of the elevons, sufficient in the denser portions of the atmosphere but less powerful at Columbia’s attitude of approximately 200,000 feet, were insufficient to keep Columbia pointed in the right direction. That is, the damage to the left wing and the additional drag being imparted had now exceeded the ability of the elevon control surfaces to compensate. They needed help and the Yaw thrusters were the only things available.

Cockpit indicator lights would have revealed that the thrusters were firing continuously, but the jets themselves were just outside the forward windows and it’s possible that the Commander, Rick Husband, may have been able to visually note that the thrusters were firing.

At GMT 13:59:32, just a few seconds after the two jets began to fire continuously, we lost communication with Columbia. Commander Husband’s last acknowledgement of “Roger, uh [truncated mid-word] …”4 were the final words recorded from the crew, and at that moment all telemetry was lost in the MCC.

But at this time, while Columbia’s control systems were struggling valiantly to keep the ship in its desired attitude and messages had appeared on the crew’s displays that something was erroneous (loss of the left main gear tire pressures), all else appeared normal. The cabin temperature was a comfortable 71.6 degrees Fahrenheit and cabin pressure hovering at a normal seal-level pressure of 14.64 pounds per square inch. But the fact that the MCC no longer was receiving either telemetry or voice communications indicated that things were going awry very, very quickly.

The data stored in the OEX recorder has allowed us to piece together some of the last moments before Columbia broke up. At GMT 13:59:33, one second after the loss of signal, an FCS CH 4 (Flight Control System Channel Four) fault summary message was annunciated to the crew, indicating that this channel had been removed from the control loop, and this would have been accompanied by a Master Alarm in the cockpit. Four seconds later, at GMT 13:59:36, a third yaw reaction control jet (R4R) began to fire continuously, revealing that Columbia was losing the fight with the drag imposed by the left wing’s damage. One second later, at GMT 13:59:37, it is believed that control was lost, and Columbia began to spin or tumble. Postflight reconstruction has estimated that at this time Columbia pitched up, and in fact her contrail noted by ground cameras began to show a braided or corkscrew appearance, in addition to several bright “puffs” and a splitting of Columbia’s contrail into multiple plumes. It is believed this is when the left wing became structurally compromised and struck both the rudder and the OMS pod directly beneath it. The OMS Pods contained the large rocket nozzles used for major orbital maneuvers and for deorbiting the Orbiter at the end of the mission. These two large pods sat at the base of the rudder, one on each side, and contained both the engines and the propellant (fuel and oxidizer) used by the engines.

Figure 35. Simulation of the loss of control and the initial pitch-up attitude. (Figure 1.2-40 in NASA SP-2008-565)

A fourth yaw jet now began to fire, but it was too late. At GMT 13:59:46, ground observations indicated another debris event, this one believed to be the cover of the Left OMS Pod, followed six seconds later by the possible release of the pod itself. When this occurred, several Master Alarms began to howl in the crew cabin. The crew apparently took action to silence the alarm and the Commander’s hand controller was noted to have been moved. Beyond the alarms and messages, the attitude excursions would have been evident to the crew simply by looking out the windows. By this time, all hydraulic pressure had been lost (the lines holding the hydraulic fluid had been compromised, causing all fluid to leak out) and the three APUs that provide the hydraulic pressure indicated failures, resulting in more alarms and more messages. Most of the Left OMS sensors were now either off-scale low or meaningless.

At this time, at approximately GMT 14:00:05, amidst all this chaos, it appears that the pilot, Willie McCool, began to take action. First, he evidently tried to restart two of the APUs (APU 2 and 3) and he may have manually turned on the hydraulic Circulation Pumps, which while not an action on the restart procedure could provide some limited hydraulic pressure. This action shows a good systems knowledge on the part of the pilot.

At GMT 14:00:11 it is believed that the Left OMS propellant tank ruptured and two seconds later all electrical power to Columbia was lost. No more OEX data from that point on was available and it is believed that the vehicle now began to disintegrate. Death for the crew followed quickly thereafter.

But amazingly, even after Columbia began to break up and, ostensibly, the lights went out, the crew evidently didn’t give up. Some of the switch panels from the flight deck that were recovered indicated that switches had been thrown in additional attempts to fix whatever was going on. For example, switches for the spacecraft’s electrical system’s AC inverters and buses were found in the Off position. These switches are taken to the On position before launch and are never moved again until after the Orbiter is back at KSC and being powered down. However, these switches being found in the Off position would make sense if the crew was attempting to recover from the situation presented to them.

The OMS Pods contained a number of AC-powered valves used to separate fuel from oxidizer and to control the pressurant used to move the fluids from one place to another. When portions of the left wing violently struck the Left OMS Pod, it is likely that some of these valves or the electrical wires leading to them shorted. Electrical systems was my flight control discipline and over the course of 11 years in that job I got to know the malfunction and recovery procedures very, very well, having executed them in the Shuttle simulators more than a few times. Turning the AC inverters off and disconnecting them from the associated buses were all actions commensurate with working the checklist recovery procedures. Unless the switches were jostled and moved during the breakup of the spacecraft or impact with the ground, crew action is the only other plausible explanation for them being in the Off position. And given the scenario with the OMS Pod, it may even be likely.

I personally don’t doubt something like this happened. Such a large part of the preparation for any Shuttle mission, but particularly for the commander and pilot, was the hours upon hours of time spent in the simulator, handling malfunctions, working checklist procedures, using their minds and their intellect to figure out how to get out of challenging technological situations. It’s what they were trained to do. It’s what we flight controllers were trained to do. Any crew’s or flight control team’s intellectual muscle memory would be heightened by the time it came to fly because of those hours and hours of dedicated and rigorous training.

All in all, Columbia herself was valiantly struggling to maintain control, commanding elevon aerosurfaces and firing RCS thrusters all to counteract the increasing drag on a damaged left wing. And the crew themselves were valiantly struggling to maintain control of a rapidly deteriorating situation by throwing switches and working checklist procedures. Both crew and vehicle acted jointly in heroic action against the forces of destruction, a battle that they would both fight but, inevitably and tragically, lose.

Life Lesson for Engineers, Managers, and Leaders

There is both power and limits to struggle.

We are not powerless. We can and do produce systems to be resilient, reliable, and capable of performing complex missions even in the face of failure. That mission assurance aspect of our job is vital to the success of the things we design and operate. We often over-design, at least compared with the systems’ requirements and mission objectives, not because we’re trying to overachieve but because we understand the value of margins and of the probability that they may need to be called upon, even if we can’t always predict how and when. A system that’s resilient, that can continue to operate, perform its function, and accomplish its mission, is almost by definition an elegant system, if not by definition, then certainly at least by culture.

We have the power to accomplish amazing feats of science and engineering, create vehicles that continuously push envelopes, build buildings taller and more expansive then any previously constructed, and find solutions to problems that have vexed humanity since the stone age. There is power inherent in what we do and what we enable. But physics demands that there are also limitations.

Physics is the underlying foundation of all we do. While we can manipulate energy, we cannot create nor destroy it. While we can build machines that seem to defy physics, that appearance is an illusion. Under all circumstances, physics wins. End of story. We concede that fact because, ultimately, as much as we may wish it not to be true, there’s nothing we can do to alter it.

Our systems have limitations. They always will. Even our actions have limitations bounded by the laws of physics. That is not a failing on our part—it’s just life.

Struggles like those of the STS-107 crew and vehicle can be memorable and heroic and at the same time prove ultimately futile. Again, that’s not a failing on our part—it’s just life. Recognizing this, I would suggest, is the mark of a competent engineer, manager, or leader. Conscientious denial of these limitations is self-defeating. Pushing the envelope is fine, enabling resiliency is better, and producing the best systems we can is expected. But realize that after all of our work is complete, life still places limits on what we can accomplish. If we can accept that as a component of our work, feed that into our calculus and accept the reality of its truth, then all of our endeavors will benefit from it.

1 This was evidently a storytelling technique created by Ron Howard for the film, Apollo 13. But you hopefully get my point. See the interview of John Aaron for more details: https://historycollection.jsc.nasa.gov/JSCHistoryPortal/history/oral_histories/AaronJW/AaronJW_1-26-00.htm (accessed Jan. 22, 2025).

2 The same STS-83 crew would launch again on Columbia a few months later on the newly designated STS-94 mission and complete a successful full-duration, two-week research mission.

3 Columbia Crew Survival Investigation Report. (Washington, DC: NASA, SP-2008-565. Jan 1, 2009). https://www.nasa.gov/pdf/298870main_SP-2008-565.pdf (accessed Jan 22, 2025).

4 Harold W. Gehman, Jr. et al., Columbia Accident Investigation Board Report Vol. I, (Washington, DC: NASA, Aug. 2003), p. 43, https://ntrs.nasa.gov/api/citations/20030066167/downloads/20030066167.pdf (accessed Jan. 22, 2025).

Return to Flight

Chapter 13

Recertifying the Orbiters

Don’t Allow Problems to Get Too Large to be Effectively Dealt With

Once the CAIB released their findings and recommendations in August 2003, we had great confidence that we now knew what had occurred on the morning of February 1 high over the western and southern United States, a calamity that had reduced a venerable space vehicle to a swarm of jagged wreckage and, more importantly, taken the lives of seven brave explorers—husbands and wives, brothers and sisters, aunts and uncles, friends, and colleagues. We will never know with one hundred percent precision that the theories that were developed over seven months of intense efforts did in fact describe exactly what had occurred, but the explanations were plausible enough, supported by data and evidence, and certainly met the normal engineering expectations of three sigma (a statistical calculation in which the datum are within three standard deviations from a mean; generally providing 99.7 percent certainty). With this in hand, we now turned to our next challenge—determining what needed to be done to get the Space Shuttles flying again and then doing it.

It should be obvious to you now that Space Shuttle Orbiters were never designed to withstand impacts from material coming off the ET or SRBs, or by anything else for that matter. Much of the spacecraft’s external design focused on the thermal protection system (TPS) that enveloped its aluminum frame, a sheath of resiliency that allowed the ship to withstand the unbelievably hot fires of reentry. Most varieties of aluminum will melt at a temperature of 750ºF, but the hottest portions of the Orbiter would see temperatures approaching 3,000ºF—four times hotter than what is needed to melt the spacecraft’s metal structure. Those critical areas that would see the hottest temperatures—namely the nose cap and the leading edges of the wings—were protected by RCC panels, a composite-based structure resistant to oxidation and covered with a silicon carbide coating and a final protective layer of a glass sealant.1 Other areas that saw high heating were covered in felt reusable insulation blankets, with the remainder of the outer acreage of the Orbiters (the majority in fact) protected by more than 20,000 high-temperature reusable insulation tiles, each with its own specific size, shape, and geometry (and identification number).2 As an ensemble, these layers allowed the relatively delicate frame of the Orbiter to pierce Earth’s atmosphere at speeds approaching 25 times the speed of sound and to survive tens of minutes within a bubble of searing hot plasma.

But while the design for thermal protection was a good one, we now know that it was not impervious to debris strikes. Fixing that, of course, was a top priority for the program’s Return to Flight efforts, as it became known.

The very first action was to investigate whether the ET’s bipod ramps could be removed entirely. The application of insulating foam in the area of the bipod attachment point was intended to prevent ice from building up around the metal fitting (which, ironically, it was feared could break off and strike the Orbiter). The ramps were sculpted and shaped to allow for smooth aerodynamic flow when the stack was moving through the atmosphere, first subsonically and then supersonically. Analysis was performed by the ET Project Office at NASA’s MSFC in Huntsville, Alabama, which managed all ET efforts, revealing that this foam insulation could in fact be removed, but the metal attachment fittings would then require the addition of heaters to prevent ice formation. A seemingly easy fix! The fittings were modified with four rod heaters held within a copper plate and placed between the fitting and the tank’s outer layer of insulation. Standard engineering modeling and testing of the new configuration ensured that the design would work adequately and provide sufficient performance against the formation of ice.3 Initial problem solved.

Other changes were made to the spray-on application of insulation on both the ET and the SRBs, reducing the potential for bubble formation within the foam that could expand in flight and initiate the sequence of events that leads to insulation breaking free.

Figure 36. The bipod ramp of spray-on insulating foam that covers the attachment point of the bipod to the external tank. Compare this photo to Figure 37 below with no bipod ramp. (Figure 15 in Charles J. Camarda, “Space Shuttle Return-to-Flight Following the Columbia Tragedy.” NATO Science and Technology Lecture Series on “Hypersonic Flight Testing,” March 14, 2014).

Figure 37. Image taken by author (April 25, 2009) of the STS-123 stack, showing the updated configuration without the bipod ramps.

All this work was performed by the MSFC ET Project Office, and the OPO at JSC, which I supported, was largely uninvolved. Our simple requirement was that nothing strike the Orbiter; how the ET Project implemented that was up to them. Still, we were periodically briefed on the progress they were making at a few meetings in the OPO conference room in Building 1. Not surprisingly, we were interested stakeholders.

But even given these changes, which reduced but didn’t eliminate the risk of debris striking the Orbiter, we were still left with a spacecraft that was not certified for the environment within which it operated—namely, being hit.

But certifying a spaceflight system can be a complex, laborious, and often very, very expensive task. Certification draws upon sequences of tests, analysis, and demonstrations that reveal the resiliency of a system. In later years, when I was working as the chief engineer for aeronautics at NASA HQ, I learned that fully 25 percent of the costs to develop Boeing’s 787 Dreamliner commercial aircraft—several billions of dollars—was invested in navigating the prototype aircraft through the FAA’s certification process. Now, given the fact that once put into production these 787 aircraft would carry millions of passengers over the length of their use, the FAA establishes very high demands for reliability—on the order of one failure in 1,000,000,000 cases (or what we engineers refer to as nine nines of reliability, or a probability of 0.0000000001 of failure). In order to prove that a system can perform to that level of near perfection, years of testing is required with legions of engineers and technicians called upon to reach that consensus.

Of course, the Shuttle doesn’t carry millions of passengers, and every astronaut who crawls into the ships and straps in for launch understands that there is a large amount of inherent risk in the endeavor. Still, we certify even reusable spacecraft to stringent standards—perhaps not as stringent as the FAA’s but standards that still require a demanding certification program. While a resilient system is adept at withstanding failures that we can’t predict, it should be even more resilient to the ones we can.

Thus, considerable effort was put into the question of whether we could make the Orbiters more resilient to debris coming off the ET and SRBs, specifically to their protective covering of RCC panels and thermal insulation tiles. (More on the insulation tiles in Chapter 15.)

In September 2003, NASA released a status on the SSP’s effort to return to flight. The full intent of the agency was to comply with the CAIB’s recommendations marked “RTF” (for return to flight) before attempting to launch again.4

As one example, CAIB recommendation 3.2-1 reads, “Initiate an aggressive program to eliminate all External Tank Thermal Protection System debris-shedding at the source with particular emphasis on the region where the bipod struts attach to the External Tank. [RTF].”5 The NASA response to this recommendation listed seven remedial actions the agency was taking:

Forward Bipod Ramp—NASA has redesigned the ramp to eliminate the foam ramp and incorporate redundant heaters [Mentioned above].

LO2 Feedline Bellows (Ice)—Potential solutions are a bellows boot, drip lip and drain, or a purge ring.

Protuberance Air Load (PAL) Ramps—Potential solutions are to verify the current design; replace the ramps with a more controlled foam application technique; or eliminate the ramps altogether.

LH2/Intertank Flange Closeout—Potential solutions are performing a localized gas purge; sealing the flow path from the intertank joint to the foam; improving Thermal Protection System closeout to prevent voids; and improving procedures to minimize post-manufacturing foam damage.6

Foam Verification Reassessment—NASA is reassessing the Thermal Protection System verification rationale and data for all processes for applying foam to the External Tank. NASA will ensure that at least two employees attend all final closeouts and critical hand-spraying procedures to ensure proper processing.

Nondestructive Inspection (NDI) of Foam—NASA has initiated a long-term program to develop NDI techniques for foam for improved process verification.

Long-Term Activities—As part of the Shuttle Life Extension activities, NASA is evaluating potential long-term changes in the External Tank design to continue our aggressive program to eliminate debris shedding at the source.7

Throughout the latter portions of 2003 and well into 2004, members of the MSFC would tie into the OPO’s weekly telecons and give us a status on their progress. On a positive note, none of the above remediations appeared to be beyond MSFC’s abilities to effect change. Most required new processes and substantial analysis and testing to certify the new designs, but nothing fell into the barren engineering landscape of “unobtanium.”

As far as hardening of the Orbiter itself, CAIB recommendation 3.3-2 read, “Initiate a program designed to increase the Orbiter’s ability to sustain minor debris damage by measures such as improved impact-resistant Reinforced Carbon-Carbon and acreage tiles. This program should determine the actual impact resistance of current materials and the effect of likely debris strikes. [RTF]”8 In the response, NASA stated that 17 redesign candidates had been developed, ranging from near-term with low technical risk to very long-term with high technical risk. Of these, eight of the near-term options were selected for further study with detailed feasibility assessments for each option.9

Seven critical areas were identified for enhancement. These were: landing gear and ET door TPS and structure; wing leading edge (WLE) subsystem; vehicle carrier panels and attachments; critical area lower surface tile; elevon gap and cove TPS and seals; critical Orbiter Maneuvering System pod and vertical tail areas; and nose cap and chin panel subsystem.

To implement these improvements, the conceptual design candidates were developed and reviewed by the Program Requirements Control Board, which met in May 2003. The board, chaired by the SSP manager, decided on the following options (listed in order of priority):

WLE [Wing Leading Edge] Redesign—Options include WLE carrier panel and fastener redesign, spar insulation, and new WLE surface coating materials to provide additional protection against impact and plasma flow vulnerability.

Durable Tile—Complete development of tougher lower surface landing gear door and ET door periphery tiles, elevon leading edge and wing trailing edge carrier panel tiles and window frames, and acreage tile. Also, complete development of ballistic strain isolation pad material.

Landing Gear Door and ET Door Redesign—Options include upgrade of thermal barrier materials to provide better protection against high temperatures, and multiple thermal barrier backup capability to main landing gear doors (MLGDs).

Carrier Panel Upgrades to Eliminate Bonded Studs and Elevon Leading Edge Carrier Panel Installation Redesign—Redesign of carrier panel attachments to eliminate failure mode of structural bonds to ensure positive margins. Redesign access panels to improve protection against impacts and provide additional protection from plasma flow due to impact damage.

TPS Instrumentation—Define additional instrumentation needs, sensor types, and avionics modifications; determine requirements for data trending. Installation of an impact penetration instrumentation system to provide monitoring capability for potential ascent/micrometeoroid and orbital debris impacts.

White Toughened Unipiece Fibrous Insulation (TUFI) Tiles—Lessen impact damage susceptibility of certain upper surface tiles by replacing existing tile with white TUFI tile.

Vertical Tail Advanced Flexible Reusable Surface Insulation (AFRSI) High- Emittance Coating—Add high-admittance coating to existing AFRSI blankets to expand contingency low-alpha reentry trajectory limits.

Robust Reinforced Carbon-Carbon (RCC) Replacement Study—Apply new technologies to develop a more debris-tolerant material for the nose cone, chin panel, and WLE panels.10

All of these feasibility assessments, considering their design, engineering, manufacturing, and operational implications, came through the OPO’s decisional board for discussion and debate. Hours upon hours were spent around the conference room’s large table, the faces of engineers and subsystem managers seated at the table bouncing between images being projected onto the twin screens at the front of the room and the OPO manager at the head of the table. These were all lively discussions, fully technical in nature but often contentiously debated as opinions and perspectives had a tendency to clash. The demeanor was professional (mostly), and tempers rarely flared (although sometimes they did) as decades of collective knowledge and experience was simultaneously brought to bear on the challenges of RTF. Everyone knew what was at stake—ultimately, the lives of astronaut-explorers and the existence of national assets (e.g., the Shuttle). There was recognition that safety was the highest priority in all our endeavors, but also an acknowledgement that physics and, sometimes, money, were hurdles that sometimes couldn’t be overcome. An elegant engineering solution is always desired, but practical and realistic solutions could win out in the end. The chair of the board—originally Ralph Roe and then his replacement, Steve Poulos—would conscientiously listen to these discussions, often remaining silent so that participants felt free to speak their minds. Eventually, when the discussion was exhausted or time had run out for that particular topic, they would weigh in with a decision if one needed to be made (or sometimes not, needing a bit more time to think about it outside of board). Or, if the topic was simply a status, they would give the presenter a heartfelt “thank you.”

Figure 38. The seven critical TPS areas identified for enhancement. (Figure 3.3-2-1 in “NASA’s Implementation Plan for Return to Flight and Beyond” September 8, 2003)

But then there was the WLE redesign—the RCC panels. These hardened composite structures protected the most critical areas of the Orbiter, those that saw the highest temperatures during reentry. As you know by now, it was a breach of one of Columbia’s left-wing RCC Panels that allowed hot plasma to enter the wing structure and, fairly quickly, destroy the vehicle. Other areas of Orbiter structure were vulnerable to the high temperatures of reentry, but none moreso than those protected by RCC. And given the extraordinary task these panels needed to accomplish and the immaturity of composite and hardened thermal protection technologies at the time, modifications to their design proved to be nearly impossible. Analysis showed that certain portions of the panels, particularly their underside, could not withstand any damage at all and still protect the Orbiter’s underlying structure through the heat of reentry. Without a dedicated development program that could take years (decades?) and cost untold tens or hundreds of millions of dollars (or more?), we were left with using the panels we had.

The best that we could muster was to improve the way we inspected panels between flights to ensure that any strikes they did take place didn’t delaminate the underlying composite weave and render the panels ineffective. As mentioned, composite technologies of the nature of the RCC panels were still nascent in 2003, as were the techniques for evaluating them. Nondestructive Evaluation, or NDE, was widely available for determining the inherent capabilities of structures, techniques like x-ray and others that could reveal cracks, fatigue, and internal defects. But x-ray didn’t work too well on the composite RCC panels, only revealing the largest or most obvious imperfection. To solve this, entire new processes would need to be created.

Of these, “thermography, contact ultrasonics [similar to the technique obstetricians use to visualize an unborn baby], eddy current, and radiography were selected as the most promising techniques that could be developed in less than 12 months and used for on-vehicle inspection”—that is, the part being inspected wouldn’t need to be removed from the Orbiter.11 The Shuttle Program approved the budget for the development of these techniques. Ultimately, contact ultrasonics was deemed less promising than the other techniques and its development was discontinued.

Now, back to certification. In March of 2002, less than a year before the Columbia tragedy, the NASA HQ associate administrator for the Office of Space Flight, former astronaut Fred Gregory, commissioned a quick assessment called Shuttle 2020 to identify and prioritize future investments required to keep flying the Shuttle safely and effectively through the year 2020.12 While this assessment proved helpful in the generation of the program’s 2003 budget, a more formal study—the Shuttle Service Life Extension Project (SLEP)—was initiated to extend the life of the Shuttle assets from their planned certification of 2012 to 2020, at least.

While the RTF activities focused on remediating the causes of the Columbia accident, SLEP had a much broader ambition of extending the certification of the entire vehicle system, not just specific parts but of everything! All the structure, all the equipment, all the computers and avionics boxes, every value and pump, connector, and circuit breaker, down to the level of nuts and bolts and rivets. This wasn’t limited to just the flight hardware but also to infrastructure, facilities, special test equipment, and even to human capital (e.g., people). It was a massive effort to say the least and consumed a significant amount of time and effort in addition to those required to keep the fleet flying on a day-to-day basis.

From my perspective, the loss of Columbia put an end to the Shuttle 2020 efforts.13 Or at least I didn’t hear much about it after the accident. All hands were assuredly on-deck to discover the cause of the accident and to return to flight. Just launching again and flying the missions necessary to complete construction of the ISS was more than enough mandate for us working within or supporting the program.

But the certification efforts never strayed far from my mind. Certifying these new RTF designs would consume the better part of the next year and a half before we felt ready to attempt another Shuttle launch. And this was when I learned of a concern that would vex the program all the way to its retirement after the last Shuttle mission in 2011.

When anything on the vehicle was removed and replaced, whether due to repair, obsolescence or planned replacement, the operation was controlled by an installation drawing. The Shuttle was designed in the 1970s, long before the advent of computer-aided design. The company contracted with designing and manufacturing the Orbiter, North American-Rockwell, as well as those for the ET, SRBs, and everything else, utilized paper (or more precisely, engineering vellum) to document the designs. Thousands, perhaps tens of thousands, of drawings were produced, each characterizing a specific piece of hardware and showing the exact dimensions and tolerances of how it should be installed. These drawings represented the heart of Shuttle ground operations, for every time a component needed to be removed and replaced it was these drawings that mediated these critical operations.

But herein lay the problem. Sometimes, components change. It wasn’t unheard of for one part vendor to go out of business or decide it didn’t make financial sense for them to continue supporting Shuttle, and another vendor would need to be found. Occasionally when that happened, slight modifications to the part would be required. Perhaps the item would need to be made to very slightly different tolerances or made by different manufacturing techniques or even out of new materials. All of these, including the procedures by which the part was removed or installed, needed to be reflected in these drawings. This had been going on ever since the first Shuttle flight in 1981, and by 2003 the entire system had metastasized into somewhat of a mess.

Ideally, when a change to any of these drawings was made, the drawing would be updated to reflect that change. Seems simple enough, right? But given forces acting on the program like budget reductions and congressional pressures to increase efficiencies, these changes were directed into what were called Engineering Orders, or EOs, which would document the specific change. Because of this, over time, many installation drawings degenerated into a stack of papers starting with the original foundational drawing and adding in all the EOs appended to it. Therefore, when a technician needed to remove and replace a part, they needed to reference the original drawing plus an entire series of EOs, sometimes as many as 50! A cumbersome and inefficient task, to say the least.

During the RTF activities in 2004 and 2005 and lasting until I left my position as MOD rep to the OPO in 2006 to begin focusing on development of NASA’s Constellation Program, this was never resolved. While the problem certainly was recognized (and I always imagined it being the bane of Orbiter technicians at KSC), the challenge of gathering up all the EOs and incorporating the changes into the baseline drawing simply became too egregious. The costs that would be incurred by coalescing all of these installation drawing changes was simply more than the SSP could invest. When the program ended in 2011, this problem still existed.

Life Lesson for Engineers, Managers, and Leaders

Don’t allow problems to get too large to be effectively dealt with.

That experience taught me an important lesson that I carry with me even to this day. Don’t allow problems to get too large to be effectively dealt with. It can be tempting to defer a solution for a problem, to “kick the can down the road” to use a common colloquialism. Given the pressures placed on engineers, managers, and leaders to keep things moving and to meet cost, schedule, and technical requirements, it can sometimes be easy to allow conditions to fester and simply live with an inefficient situation. Sometimes that’s an acceptable solution as there’s only enough bandwidth and resources to be applied to our challenges. But doing so also carries risk, and that risk should be included in any decision calculus.

If the Shuttle Program was vigilant in incorporating EOs into the baseline drawing or the problem of accumulated EOs had been remediated earlier in the program, the situation might never have reached the point of being a problem (and countless installation technicians would inevitably have been much happier). But it wasn’t, and the condition eventually reached the point where remediation was no longer a viable option given the resources available to the program. At that point the situation could not be mitigated. It was too big to deal with and became a sea anchor to efforts to keep the Shuttle maintained throughout the rest of the program.

Deciding to deal with an issue often requires resources—money, workforce, facility time, new contracting, and so on. Nothing is free, and reducing risk by taking action has a cost. But those costs usually begin small and only grow over time. Allowing a situation to reach the point when action is no longer a viable option, when inaction becomes the solution, is in ways akin to failure. If inaction is consciously decided upon and the risks are incorporated into the decision, then it can be justifiable. But if inaction comes from a desire to avoid taking action, then leaders, managers, and engineers only have themselves to blame for the conditions it creates.

1 “NASA Facts: Reinforced Carbon-Carbon (RCC) Panels.” NASA/FS-2004-01-001-KSC (Rev 2006). https://www3.nasa.gov/centers/kennedy/pdf/167435main_RCCpanels08.pdf (accessed May 30, 2024).

2 “Space Shuttle Tiles.” NASA Aeronautics Research Mission Directorate Museum in a Box Grades 9–12. https://www.nasa.gov/wp-content/uploads/2023/06/shuttle-tiles-9-12v2.pdf?emrc=aa90c2 (accessed May 30, 2024).

3 “NASA Facts: External Tank Forward Bipod Fitting.” NASA-FS-2005-04-035-MSFC. April 2005 https://www.nasa.gov/wp-content/uploads/2016/08/114020main_et_bipod_fs.pdf (accessed Jan. 22, 2025).

4 Return to Flight Task Group, Final Report of the Return to Flight Task Group (Washington, DC: NASA, ID: 20050201800, July 1, 2005), p. 5. https://ntrs.nasa.gov/api/citations/20050201800/downloads/20050201800.pdf (accessed Jan. 22, 2025)

5 Harold W. Gehman, Jr. et al., Columbia Accident Investigation Board Report Volume 1. (Washington, DC: Columbia Accident Investigation Board, August 2003), p. 225. https://ntrs.nasa.gov/api/citations/20030066167/downloads/20030066167.pdf (accessed Jan. 22, 2025).

6 Also mentioned above.

7 “NASA’s Implementation Plan for Return to Flight and Beyond” (Sept. 8, 2003), p. xv.

8 Harold W. Gehman, Jr. et al., Columbia Accident Investigation Board Report Volume 1. (Columbia Accident Investigation Board: August 2003), p. 225.

9 “NASA’s Implementation Plan for Return to Flight and Beyond” (Sept. 8, 2003), p. xv.

10 “NASA’s Implementation Plan for Return to Flight and Beyond” (Sept. 8, 2003), p. 1-11 to 1-13.

11 Return to Flight Task Group, Final Report of the Return to Flight Task Group (Washington, DC: NASA, ID: 20050201800, July 1, 2005), p. 40.

12 Parker V. Counts, “Space Shuttle Program 2020 Assessment,” August 21, 2002.

13 Another viewpoint is that the George W. Bush administration’s Vision for Space Exploration, proposed in 2004 as a result of the Columbia accident, halted any idea of extending the program beyond 2010. That may be true politically, but I stand by my perspective.

Chapter 14

MMT Improvements

Train as You Fly (or, More Commonly Stated, Test as You Fly)

In the Spring of 1989, I jumped in my car and travelled from Austin, Texas, to JSC near Houston to meet the people with whom I was about to begin my professional career. I had just been hired into the EGIL flight controller group within the MOD, but at the time was still attending the University of Texas completing the last year of my master’s degree. On one midweek day my graduate advisor allowed me to skip his spacecraft design class for this sojourn to JSC.

The first person I met was Dick Brown (everyone called him “Brownie”) with whom I was to soon share an office. Brownie was set to retire within six months of my arrival and was completing the arc of an historic career. Not only was Brownie on Flight Director Gene Kranz’s Apollo 11 moon landing team, providing backroom support to EECOM John Aaron, but Brownie also was on console when the oxygen tank ruptured on Apollo 13. While having participated in history not just once but twice could tempt any person to overestimate their value, doing so was very much not Brownie’s way. His humility was equally matched by his graciousness, and those together allowed his technical brilliance to shine through. Brownie was the emeritus of the EGIL group—the oldest and most experienced within our small, immediate flight controller family. After I graduated, whether simply due to the luck of the draw or perhaps a touch of serendipity, I got to share an office with him for those last six months and reap his wisdom.

But on that day in late 1989, while still a graduate student and only getting a perfunctory tour, Brownie walked me into the Flight Control Room of Mission Control. We entered through a side door—not the large metallic door next to the coffee cabinet described earlier, but another door on the opposite side of the FCR. I had never been inside Mission Control before, least of all the main flight control room, so famous from history and imagery from the Apollo program, where Shuttle flights were also controlled. The sense of hallowed reverence filled me as we entered that side door, feeling much as one would when visiting a museum or historic site, coming to terms with the awed awareness that history occurred on the very ground upon which I was stepping. And yet, even feeling this veneration, I quickly came to realize that the FCR also was a place of work, a factory of production where people performed their jobs, earned their pay, drank their coffee, and left at the end of the day to return home to a hot dinner, only to repeat the cycle the next day and the day after that.

Figure 39. Gene Kranz’s Apollo 11 lunar landing team. Dick “Brownie” Brown is seated in the front row in the middle. (NASA Image S70-29751)

I followed Brownie into the room, walking past the lowest tiers of consoles. To my left were the large screens illuminated from the bat cave onto which was projected a map of Earth and the Shuttle’s sinusoidal ground track overlaid it. I had seen pictures of this screen, emblematic as it was of the role of Mission Control. On this day, however, the map of the world looked different, somewhat out of place, and it took me a few moments to figure out why. Every image I had seen of this caricature of Earth had the continents and countries outlined in light blue, but on this day the cartography was presented in a distinctly vibrant red. I would learn over the next weeks that this was simply due to the preference of the flight director on console that day—Rob Kelso—who preferred the hues of red. As flight director, he didn’t need any more reason.

To my right rose four rows of beige-metal consoles, each set slightly above and behind the others much like bleachers at a stadium. Behind these consoles sat ranks of flight controllers, spaced about 10 feet between them, staring at the monochrome monitors before them and surrounded by banks of indicator lights. Some of them were jotting notes in their flight controller logs or referencing looseleaf-bound books of procedures and checklists. A few were even sipping on cups of coffee. I was surprised to find the sound level in the room to be low, more a collection of hushed whispers than a raucous chorus. Each controller wore headsets that allowed them to speak directly to the support consoles in the back rooms or to other controllers in the FCR, eliminating the need to shout across the room. The one exception to this were some of the controllers speaking with Kelso, the flight director. Most spoke to him using the flight director’s communications loop, but every now and again a few comments were made “over the airways,” or simply spoken to each across the room without use of their headsets. Whatever happened to be going on with this mission in space, the mood in the room was quietly professional and matter of fact, as if these folks were discussing planning for a Sunday barbeque.

As Brownie and I walked to the other side of the room and watched the proceedings from there, he leaned close and mentioned that this was a simulation for the STS-30 mission. That Shuttle flight was to deploy the Magellan radar mapping spacecraft to chart the cloud-obscured landscape of the planet Venus.1 Kelso was the lead flight director for this mission and he and the controllers on console were assigned to the Orbit 2 shift, the shift that would guide the crew through deploying the Magellan probe only hours after launch. The simulation being conducted was one of the ways NASA prepared both the crew and the flight control teams for missions. A typical sim like this one for STS-30 would begin a few hours prior to deployment of the payload and last a few hours after, assuming the payload got deployed at all. The team would proceed through the nominal timeline, performing expected checklist activities at their expected times, as they would on the day when the Magellan spacecraft would actually be deployed. But of course, nominal wasn’t the point of these sims; off-nominal was de rigeur on these practice runs. The training team would routinely simulate malfunctions on the Orbiter, failures on the payload or deployment mechanism, an entire host of anomalous events that kept the crew and flight control team thinking, adapting, forcing them to innovate and still achieve success. Rigorous training was what we did to prepare us for the potential of unexpected events that could occur on the real day, in space and when mistakes can be costly. We trained, we practiced, we occasionally failed and learned from the experience, all to ensure that we were ready for anything when decisions were critical.

We trained to be proficient. We trained to stay sharp. Inevitably, we trained a lot!

And while deploying a mission’s primary payload was important to NASA and to the country (and often to humankind), decision-making was never as critical as during dynamic flight, specifically Ascent (during launch) and Entry (deorbiting and returning to Earth). During these timeframes events happen so quickly and speeds and energies are so high that often situations don’t allow for contemplation or consideration. Decisions need to be made—now! Ascent and Entry training could be rough, hectic, and constructed to test the mettle of even the most implacable flight controllers. But there was a reason for this seeming chaos and that was to forge the flight controller’s mindset to react rather than to contemplate. In time-critical situations when lives and national assets are at stake, a well-prepared team will almost intuitively know what to do when a component breaks or, simply, when the unexpected happens.

Training is how we prepared for Shuttle flights, and we did a lot of it.

Now, how is this relevant to Columbia and STS-107? Nothing the crew or flight control team could have done would have prevented the accident and the tragic loss of life. That’s true. But in the thoughts of the CAIB, training wasn’t reserved just for the crew and the flight control team, but the idea expanded to other aspects of mission oversight—namely, the Mission Management Team, or MMT.

CAIB recommendation 6.3-1 read:

Implement an expanded training program in which the Mission Management Team faces potential crew and vehicle safety contingencies beyond launch and ascent. These contingencies should involve potential loss of Shuttle or crew, contain numerous uncertainties and unknowns, and require the Mission Management Team to assemble and interact with support organizations across NASA/Contractor lines and in various locations.2

Why make a recommendation like this? The CAIB (and others) recognized that the MMT played a role in the conduct of NASA’s human spaceflight missions but that the present MMT construct didn’t require any training to prepare them for this task. All members of the MMT were experienced managers and senior leaders, and yet that experience was perhaps used as a conceit to alleviate them from the need to train. Throughout Shuttle missions the MMT made the strategic, high-level decisions that affected how these missions would progress while relegating all tactical decisions to the flight control team. But even these decisions played a critical role in whether a Shuttle mission was successful or not.

STS-107 was by most accounts a relatively benign and straightforward Shuttle science flight, and that idea remained throughout its mission up until the very end. Admittedly, nothing in human spaceflight is completely straightforward of course, but compared with some of the more complex missions that required rendezvous, payload retrievals, or extra-vehicular activities, STS-107 was more staid. Spacecraft and crew would get to orbit, conduct science for two weeks, then pack up and return home. The STS-107 MMT played a daily role in making essential strategic decisions throughout the flight, and many of those decisions were proper and appropriate. The greatest omission of the STS-107 MMT was likely their failure to call upon other national assets to ascertain the state of Columbia’s left wing while the ship was still in orbit.

Was the MMT honed to a fine edge through rigorous training and simulated trial prior to the catastrophe that befell the mission at its conclusion? No. Nor were any of the Mission Management Teams that guided Shuttle missions before STS-107. When tragedy did befall NASA, it was many from the MMT who were immediately called upon to respond. Were they prepared for this? Well, beyond the notion that no one can ever truly be prepared for an unexpected calamitous event, the answer is likely also no. Did they respond well in the immediately aftermath of the accident? I suspect they did as best as they were able, professionals one and all, each of them suddenly thrusted into an almost inconceivable circumstance.

But the CAIB was getting at something larger than appropriate real-time reaction. They were suggesting preparation in the form of dedicated, conscious efforts that could exercise the requisite skills by exposing the team to scenarios that were both harrowing and terrifying and yet still completely realistic.

In July 2005, a NASA task group charged with assessing the implementation of the 15 CAIB Return-to-Flight recommendations released their final report. This group, formed in July 2003 and co-led by former astronauts Richard Covey (who flew on the first flight after the Challenger accident) and Thomas Stafford (who flew on two Gemini and two Apollo missions), assessed the actions taken by NASA to implement the CAIB’s RTF recommendations. They conducted fact-finding activities, reviewed documents, held public meetings, and released a series of interim reports before concluding their efforts with this final report.3

On the subject of training for the MMT, the task group found that the Space Shuttle Program began to identify necessary changes to the MMT as early as May 2003, just three months after the accident, expanding the MMT membership, better defining the member’s responsibilities, and enhancing the formality of the MMT. But the CAIB felt that “the MMT become somewhat ad hoc and informal in nature.”4 Additionally, risk management, they suggested, should now be a major consideration at each MMT meeting, with each identified hazard required to have a clear risk assessment performed and presented to the MMT so the appropriate tradeoffs could be discussed and decided upon.

With respect to training, however, the Program established a policy that all MMT members (except those serving exclusively in an advisory capacity) would be required to complete a minimum set of trainings in order to attain initial qualification prior to performing any MMT duties. This was to be followed by an ongoing training program to maintain their status, which would be annually renewed.

Figure 40. A Shuttle Mission Management Team in session on the first floor of the Mission Control Center. (NASA Image JSC2005-E-32007)

In response, the SSP released a directive that defined generic training requirements for all MMT members, consisting of three basic varieties of training: (a) Courses and workshops; (b) MMT simulations; and (c) Self instruction. The NASA task group report stated:

Courses, workshops, and self-instruction materials were selected to strengthen individual expertise in human factors, critical decision making, and risk management of high-reliability systems. MMT training activities are well under way with several courses/workshops held at various NASA centers and 13 simulations completed, including an end-to-end contingency simulation and a simulation to address MMT actions related to Contingency Shuttle Crew Support (see Section 3.16, SSP-3). These simulations brought together the flight crew, flight control team, launch control team, engineering staff, outside agencies, and ISS and Space Shuttle MMT members to improve communication and teach better problem-recognition and decision-making skills.5

According to the task group, however, the program’s initial attempts at this were wide of the mark. The first MMT simulation was held over three days, December 3–5, 2003, with the task group observing that “Some of the training protocols were initially developed without clear objectives and techniques to assess the quality of the training.” Furthermore, they reported, “lessons learned from prior simulations were not incorporated in subsequent exercises.”6

I recall reading this report when it came out. I sat in my office and parsed through these sections on training and, to be honest, was flabbergasted. Coming from a background in MOD where rigorous training and preparation were minimal expectations of the job and inherent parts of the flight controller culture, it was difficult to understand how this first post-Columbia MMT simulation could have missed the target in these ways. What’s more, a sizable portion of Shuttle Program management had cut their teeth in the ranks of MOD earlier in their career where training wasn’t considered an additional task to a person’s job but rather an intrinsic part of their job. To be fair, not everyone’s formative years had been spent in MOD, especially those from the Engineering Directorate (the largest organization at JSC), but the expectation of training to ensure readiness was ingrained in many at JSC, perhaps because of the presence of MOD. Regardless, the result was less than satisfactory.

Over the course of the next year and a half, leading up to the flight of Discovery on STS-114 (the first Shuttle mission after the Columbia accident), the task group determined that the Mission Management Team had improved, making notable progress in addressing the CAIB’s concerns. In closing the report, they stated that the intent of the CAIB recommendation had in fact been met. Even an Office of Inspector General audit on this topic the following year concluded that the Shuttle Program took appropriate action in response to this CAIB recommendation.7

Life Lessons for Engineers, Managers, and Leaders

Train as you fly (or more commonly stated in engineering circles, Test as you fly).

The words “train as you fly” would routinely echo within the walls of Mission Control and inside the placid yellow hallways of the buildings that contained our flight controller offices. The premise is simple—train as if the simulations were actual missions. Don’t let up (or worse, give up) because a situation is too confounding to figure out or too complex to integrate, assuming that the stakes are low because it is just a simulation. No, rather, train as you fly. Put as much concentration, skills, integrity, and guile into your training regimen as you would during a real spaceflight mission because the testing you face in a sim will prepare you for other chaotic experiences when the stakes are much higher.

Our training as flight controllers was almost ruthless. Some enjoyed calling it sadistic. The training team took an almost perverse pleasure in trying to stump the flight control teams and the crew feverishly working in the simulators, purposedly forcing us all to contend with situations outside of our experience and making us all sweat. But in truth it wasn’t ruthless nor perverse because it built a resilience within our flight control cadres and with the astronauts, a resilience on console or in the cockpit that stuck with us on game day.

Train as you fly.

One famous anecdote of this practice was from the actual landing of the Lunar Module Eagle on Apollo 11. Famously onboard were Neil Armstrong and Buzz Aldrin at the controls, guiding the spacecraft closer and closer to the Moon. On console in Mission Control was Gene Kranz and his flight control team, Dick Brown among them. Several hundred feet about the lunar surface, still moving forward and downward at considerable speed, the vehicle’s computers announced a Program Alarm, a coded message that not all was well. Neil Armstrong reported the “1201 alarm” almost immediately to mission control, his voice ranging through a quarter of a million miles of vacuum before being intercepted on Earth. Both Neil and Buzz were perplexed by the presence of this rarely annunciated warning. Quickly this was followed by a “1202 alarm,” adding to the crew’s confusion.

With little time to spare as Eagle rapidly continued her descent, everyone was aware that a poor decision could mean aborting the lunar landing to the disappointment of the entire world watching on their televisions or listening on radios. Or worse, it could mean crashing into the surface, relegating Neil and Buzz to infamy as the first humans to die on the Moon.

Gene Kranz, as flight director, looked to the flight controller responsible for monitoring Eagle’s trajectory. Steve Bales, a young Iowan working at the GUIDO, or Guidance Officer, console, asked his backroom, Jack Garman, if he knew what the alarm was. In fact, Jack did know, but only because he had seen these exact same alarms codes—1201 and 1202—in a simulation some time previously. Jack had the wherewithal at the time to write down these and other apparently innocuous codes on a cheat sheet which he kept under the sheet of transparent plexiglass on his console. Because of that previous experience, Jack recognized that the alarms were indications that Eagle’s flight computer was getting overloaded, trying to process too much information. The rudimentary design of these flight computers allowed them to ignore lower priority tasks in favor of more critical ones. That is, the computer itself would sort out the data overload and continue to process the software routines that were most important, which at that times were those dedicated to guiding the spacecraft to a soft landing. At this intensely critical moment in humankind’s first lunar landing, Jack Garman recommended to Steve Bales that they continue with the descent, giving his front room an audible “Go.” Bales relayed that to Kranz, and the call was broadcast to the crew that Neil and Buzz could proceed. On that day, a historic day for humanity, simulation training saved the day.

Train as you fly.

Years later, after I left MOD and JSC for the imperious confines of NASA HQ in Washington DC, eventually finding my way to the NASA Office of Chief Engineer, this turn of words was similarly repeated, but with a small modification. Instead of the MOD credo of Train as you fly, I would often hear recited the engineering admonition of Test as you fly. While obviously slightly different, the meaning largely remained the same. In the engineering context, after a spacecraft, aircraft, or any flight vehicle is assembled, it proceeds through the series of ground testing, whether environmental, thermal, vibration, or other. The common notion is to test the system to the same or greater limits as it would be expected to see in flight, reflecting the mission profile. Countless lessons learned from flight vehicle failures has led NASA to adopt this approach, and it has proven to be a successful principle.

Test as you fly. Train as you fly. While one is applied to development and the other to operations, both are essential to success.

1 STS-30 launched on May 4, 1989, and landed four days later at Edwards Air Force Base, California.

2 Harold W. Gehman, Jr. et al., Columbia Accident Investigation Board Report Volume 1. (Washington, DC: Columbia Accident Investigation Board, August 2003), p. 226.

3 Return to Flight Task Group, Final Report of the Return to Flight Task Group (Washington, DC: NASA, ID: 20050201800, July 1, 2005).

4 Return to Flight Task Group, Final Report of the Return to Flight Task Group (Washington, DC: NASA, ID: 20050201800, July 1, 2005), p. 77.

5 Ibid, p. 79.

6 Ibid, p. 80.

7 Memo re: IG-06-011. https://oigforms.nasa.gov/docs/ig-06-011.pdf (accessed Jan. 22, 2025).

Chapter 15

On-Orbit Tile Repair

Separate the Challenging from the Futile

Some of the initial CAIB recommendations were discussed in Chapter 10, particularly those pertaining to hardening of the Orbiter’s thermal protection system; the carbon-composite RCC that lined the wing’s lead edges and other critical areas expected to see the highest temperatures during reentry. Also mentioned were the nearly 30,000 silica tiles that covered close to the entirety of the spacecraft. Without a complete redesign of spacecraft (which wasn’t feasible for a variety of reasons), necessitating going back to the drawing board and starting from scratch, most experts felt that NASA had reasonably done what it could to ensure these protective coatings were as resilient to debris strikes as possible. Combine those actions with the changes made to reduce the probability that pieces of insulation would come off the ET and SRBs during launch, and these critical CAIB recommendations were considered met.

Yet, there was really no way to reduce all the way to zero the likelihood of damage to the thermal protection system. Actually, there was one way, and that was to stay on the pad and never launch into space, which of course defeats the whole purpose of having a Space Shuttle. Short of that, though, as diligently as we may have worked to strengthen the insulation on the ET and SRBs and toughen the thermal protection on the Orbiter, there was always some possibility that these sorts of event could recur.

Then what?

Even if the likelihood of damage to the Orbiter while in flight was low, what would we do if it happened again nonetheless? That was also a challenge the CAIB handed to NASA. CAIB Recommendation 6.4-1: Thermal Protection System Inspection and Repair, beseeched NASA to do the following:

For missions to the International Space Station, develop a practicable capability to inspect and effect emergency repairs to the widest possible range of damage to the Thermal Protection System, including both tile and Reinforced Carbon-Carbon, taking advantage of the additional capabilities available when near to or docked at the International Space Station. For non-Station missions, develop a comprehensive autonomous (independent of Station) inspection and repair capability to cover the widest possible range of damage scenarios. Accomplish an on-orbit Thermal Protection System inspection, using appropriate assets and capabilities, early in all missions. The ultimate objective should be a fully autonomous capability for all missions to address the possibility that an International Space Station mission fails to achieve the correct orbit, fails to dock successfully, or is damaged during or after undocking.1

The RFT Group took a somewhat more limited view of the recommendation. Their mandated purview was to oversee and provide guidance to NASA leadership on all the activities leading up to the first post-Columbia mission, which would be STS-114. This mission always had been planned to be dedicated to resupplying the ISS, and the two-and-a-half-year downtime between visits to the Space Station made resupply even more critical. Thus, the provision concerning non-Station missions was, in the view of the RTF TG, not applicable to their charter. They did recommend that if a non-Station mission were to be added to the Shuttle manifest, such as one to service the Hubble Space Telescope, that a review should be conducted by the Aerospace Safety Advisory Panel, a congressionally authorized external body created in the wake of the Apollo 1 fire that killed three astronauts in 1967.

But the task group accepted the charge for the other provisions, going even further than the CAIB by stating that any option to repair must have gone through “formal design reviews, ground verification testing, procedure development and an integrated Design Certification Review.”2 These additional provisions would, it was hoped, ensure that if repair became necessary in an emergency situation, NASA would have confidence that the fix would behave as expected.

Repairing damaged tiles was not a new idea and the concept of flying a “tile repair kit” had been around since the beginning of the Shuttle program. Such a kit was intended to be flown on STS-1, the first Shuttle mission, with NASA even contracting with the aerospace giant Martin Marietta for the work. But development of a repair capability was eventually cancelled due to a combination of technical difficulties and a renewed confidence in the tiles themselves. The RCC panels were at the time considered sufficiently resilient and little effort went in to developing a capability to repair them.3

These CAIB recommendations on damage to the thermal protection system could effectively be separated into two areas of investigation: (a) inspecting the Orbiter while in flight, and (b) repairing any damage noted during those inspections. History would show that the first proved easier to implement than the second.

The first method chosen to inspect the Orbiter while in flight didn’t require any modifications to the spacecraft at all. Called the R-Bar Pitch Maneuver, this choreographed ballet of orbiting space vehicles would occur during the latter portions of the rendezvous with the Space Station. With the Orbiter hovering only 600 feet away along the vector connecting the Station with Earth (or radius-vector, also known as the R-bar), the crew of the Shuttle would command their flying ship to undergo a slow, serene, almost casual pirouette in space, a 360-degree flip that revealed all sides of the spacecraft to the observing crew onboard the ISS. The Station crew, outfitted with cameras with long, telephoto lenses, would photograph all the critical areas of the spinning shuttle, and then downlink to Mission Control those images, ensuring that if there was any damage it would be noted, assessed, analyzed, and determined safe for reentry (or not).

The R-Bar Pitch maneuver was a relatively simple and straightforward solution to ascertaining the integrity of the Shuttle’s thermal protection system, costing only the use of a bit more of the spacecraft’s propellant to orient into position, initiate, and then stop the rotation. When this was actually performed on all Shuttle missions to the ISS it was almost hypnotic to watch. Taking over 30 minutes to complete a full rotation, the maneuver was often downlinked live from orbit to Mission Control where observers on the ground (and at home) could also watch. I was often awestruck by the beauty of a winged vehicle flying in space at 17,500 mph, backdropped by the cloud-sprinkled landscape of Earth slowly passing beneath.

Another means put into place to detect if damage had occurred was the Wing Leading Edge Impact Detection System. This system of wireless sensors and accelerometers applied to the wing’s front spar was installed to detect the force of an impact on this critical area of the Orbiter. The system didn’t require any new technology to develop; it was relatively low-cost and easy to install, requiring very little power to operate (the sensors were battery powered). It was highly reliable, and its data could be accessed by the crew through one of the onboard laptop computers. The system, of course, would have to be certified as safe to fly on the Orbiters and to operate in the vacuum and temperatures of low-Earth orbit. It would also need to remain functional for years without the need for removal, replacement, or repair, as access to these devices was difficult and cumbersome. But all of these were straightforward engineering challenges, ones that the NASA technical team greedily accepted.

Figure 41. Figure showing the Wing Leading Edge Impact Detection System’s sensor locations. (Figure 3 in NASA Document ID 20110008039)

To augment both of these innovations and to acquire a closer scrutiny of areas where impact and damage may be suspected, particularly on the critical RCC panels, a large, sensor-laden inspection boom was also developed and flown on each post-Columbia mission. Developed by the same Canadian company who designed and built the Shuttle’s robot arm, or Remote Manipulator System (RMS), this 50-foot-long composite boom would be secured for launch on the opposite side of the payload bay from the RMS, mounted on pedestals that would hold the device firmly during launch and reentry but could be released when the boom was to be used. Called the Orbiter Boom Sensor System (OBSS), this innovative device more than made up for the lack of a creative name.

Two instrument packages were mounted at the tip of the boom. The first consisted of a combination of a Laser Dynamic Range Imager and an improved closed-circuit television camera. The second sensor included the Laser Camera System and a digital still camera. Together, these instruments could record at a resolution of a few millimeters using a slow scan rate of approximately 2.5 inches per second.4 Once grappled by the Shuttle’s RMS and released from the payload bay, the OBSS would be directed along both wing leading edge RCC panels. The crew would commence a slow scan of the safety-critical components, moving the boom at only a few inches per second. Scanning of both wings took some time, but the confidence in the integrity of the panels they provided proved to offer considerable relief to both those in Mission Control and to the onboard crew.

Figure 42. Renderings of the Orbiter Boom Sensor System. (NASA Images JSC2005-E-02444 and JSC2005-E-02441)

The OBSS was also outfitted with handrails so that the boom could be used to provide spacewalkers access to the vehicle’s underbelly—normally inaccessible—in the event that an in-flight repair was required. For more on this, read the story of astronaut Steve Robinson and his EVA in Chapter 17.

All of these devices and capabilities wended their way through the Orbiter Project Office’s meetings in Building 1’s conference room 620 over the course of late 2003, 2004, and eventually into 2005. Discussions on the development of these capabilities—from concept to design and eventually manufacture through certification for flight—came through the OPO’s board. Presented in the form of PowerPoint charts, I and many others dissected these solutions, focusing on whether they would adequately perform their intended function while at the same time avoid somehow damaging our precious Orbiter fleet. This dual charge was difficult at times to enable, resulting in passionate, dynamic, and sometimes caustic arguments, but the end game never shifted or deviated from the charge to get back to flying again. While differences of opinion occurred (inherent in most technical discussions), no one ever once questioned the purpose of all of this work or threw their hands up in frustration and admonished the futility of our efforts. An esprit de corps permeated all our efforts and all those across the SSP and the agency to learn from our mistakes and to get back to the endeavor of human exploration of space.

Which brings this story to the unfortunate lament of tile repair. As mentioned, the idea of carrying a kit onboard each mission that would allow for the in-flight repair of the broken or even completely missing thermal tiles reaches back to the very origins of the Shuttle program. Tiles were routinely damaged in flight, the brittle silica structure brilliantly impervious to high temperatures but failing to withstand being physically struck. Most returning spacecraft, once pulled into the Orbiter Processing Facility hangars to be readied for their next mission, necessitated removal of broken tiles and replacement with new ones before being launched again. The state of the Shuttle Atlantis following its return on STS-27, described in Chapter 11, is maybe the Program’s most egregious example of tile damage, but it was certainly not the only one. There were even a few close calls where tiles were nearly completely compromised down to the felt barrier that separated the tiles from the underlying aluminum structure, sometimes revealing heat-induced damage that could not be ignored. However, over dozens of Shuttle missions for the first 20 years of operations, none of these circumstances resulted in a complete burn-through of primary structure. The Orbiters always returned safely. Until STS-107.

After the accident, work began again on development of a capability that could be utilized by spacewalking astronauts to repair damaged tiles. A slew of prospective materials was investigated, many with a putty- or foam-like consistency that could be applied into the gaps of damaged tiles and smoothed over by pallets and scrapers, not unlike those used when installing flooring or bathroom tiles. The repair materials needed to be storable at room temperatures within the pressurized and climate-controlled environment of the crew cabin, be transferrable with the spacewalking astronauts into the vacuum of space, be applied such that the damaged area was filled and the underlying structure covered, be cured after being applied, and then be able to withstand the searing hot temperatures of reentry. That’s a lot of requirements to place onto a single material, but all were required.

Alternate means of repair were also investigated, such as overlaying the damaged tiles with a sheet of protective ablative material that when exposed to high temperatures would char away, dissolve and dissipate the heat much as the heat shields of the Apollo spacecraft did when smashing into the atmosphere at the end of their missions. While seemingly uncomplicated and ablative materials readily available, affixing such a repair to the already brittle Shuttle tiles and ensuring that it remain in place through the pressure forces of reentry proved to be tremendously challenging.

Methods to practice installation of these ablative covers and the foam-like fillers were investigated and demonstrated both in laboratory environments and in JSC’s large swimming pool, the Neutral Buoyancy Laboratory, when spacewalks were practiced and refined. The application proved to be more challenging than anyone expected given the peculiarities of the putty or foam, the need to ensure a smooth surface after application, and the physical constraints of spacesuited astronauts working in microgravity.

Figure 43. Onboard a KC-135 aircraft, astronaut Rex J. Walheim smooths and removes excess putty while rehearsing an EVA task for repairing damaged Shuttle tiles. The aircraft flew a series of special parabolas to practice the activity in free-fall. (NASA Image JSC2003-E-52340)

Eventually, it became clear that a certified capability of tile repair simply wouldn’t be available any time soon. Even the RTF Task Group’s assessment was similarly dour. In their report they stated, “Despite extensive efforts to develop TPS materials and techniques, the state-of-the-art in this area has yielded little technology to support the concept. As a result, continued effort does not hold promise of significant capabilities beyond those in hand.”5

More specifically, in Section 3.13.4, they conceded that tile repair would not be available when the Shuttle got back to flying:

The orbiter Thermal Protection System was never intended to be repaired on-orbit. Various repair capabilities were explored early during Space Shuttle development and again more recently, but it is highly unlikely that a comprehensive repair capability for all possible damage will become available for the remaining flights of the Space Shuttle Program. Tile and RCC repair have proven to be far more challenging than either the CAIB or NASA understood two years ago. Enormous effort has been expended in search of effective and operationally feasible repair capabilities, and far more is known today than before about the capabilities and vulnerabilities of the Orbiter Thermal Protection System. Nevertheless, the program is far from having a certifiable capability. Several innovative repair solutions for a limited range of potential damage are aggressively being pursued. Five such limited repair options will be carried on STS-114; however, much more testing and evaluation remain to be done. The options proposed by NASA have not yet achieved a level of maturity that the Task Group considers necessary to be defined as a capability and thus the intent of this recommendation has not been met. 6

For STS-114 at least, inspecting the exterior of the orbiter would have to suffice. The best that could be done is to fly a few candidate repair capabilities and certify that they would be “safe to fly”—that is, they would not be poisonous to the crew or caustic to the crew cabin. They would be used strictly as a contingency capability, providing astronauts with at least something in the event of damaged thermal protection.

Life Lessons for Engineers, Managers, and Leaders

Separate the challenging from the futile.

The results of these efforts to develop an in-orbit repair capability and make them available to be flown on STS-114 left me grossly disheartened and deeply disappointed. It wasn’t that I didn’t recognize the immense technical challenges incumbent in these pathfinding capabilities. They had a bevy of sometimes conflicting requirements and were expected to withstand incredibly harsh environments. Engineers at JSC and other NASA Centers, in academia and within industry labored for nearly two years to solve this engineering challenge by brainstorming, experimenting with different materials and apparatuses, testing promising concepts and demonstrating them in laboratories or simulated environments. But try as everyone might, certifiable solutions eluded us. I suppose part of my disappointment was facing the harsh reality that NASA’s ingenuity, decades of experience and corporate knowledge, and amazing can-do spirit still wasn’t enough to overcome this technical challenge. After all, we were the agency who routinely produced miracles. We returned the Apollo 13 crew safely to Earth after an oxygen tank exploded on their spacecraft, two hundred thousand miles away and heading toward the Moon. We were the agency who sent rovers to other planets, explored the solar system, our galaxy, and the universe. We developed groundbreaking technologies that altered the course of civil life, made scientific discoveries never before even pondered, won Nobel prizes, and were the envy of nations. And yet, on-orbit repair was something that appeared to be beyond us, outside our grasp and a problem surpassing our ability to solve. The very idea was unpalatable to me, and I know I wasn’t alone. Co-workers within MOD, the OPO, SSP, and beyond, voiced similar feelings of dismay and chagrin to me, and to others. The experience was an important lesson, not just on the limits of technical achievability but also on the importance of humility in our work and the need to ward off any sense of omnipotence. We are, after all, nothing more than human.

In pursuing great technical challenges, it is sometimes important to separate what may be challenging or difficult from what may be ultimately futile. Some technologies are not ready when you need them, so alternative paths need to be identified. Engineers can recognize when the technology is mature enough to implement, but leadership needs to consider priorities and resources. You can throw all the resources you have at a technical challenge but recognizing when and how to pull back (recognizing your limits) is a challenge for everyone.

As good as we are and accounting for the dizzying number of amazing accomplishments, some things may still be beyond our capability. To quote the Jean-Luc Picard, captain of the eponymous starship Enterprise (registry NCC-1701-D) from the television series Star Trek–The Next Generation, “It is possible to commit no mistakes and still lose. That is not a weakness; that is life.”

Pursuing the futile is, well, a futile pursuit. Knowing the difference between challenging and futile is not something one learns from a textbook but only through the successes and failures of life. This is true even in engineering. We can pursue the futile until our days on this world are exhausted and still not succeed. That is life. Over time, a good engineer, manager, or leader will recognize what is difficult from that which is not possible and pursue the possible. Or, more importantly, perhaps, recognize when a challenge enters the realm of futility. Futility does not need to be synonymous with failure, as great and meaningful advancements can be attained along the way even if the desired end product doesn’t meet its performance requirements. That is to say, there is no failure in trying.

1 Return to Flight Task Group, Final Report of the Return to Flight Task Group (Washington, DC: NASA, ID: 20050201800, July 1, 2005), p. 85.

2 Ibid.

3 Ibid, pp. 85–86.

4 “The Space Shuttle’s Return to Flight: Mission STS-114 Press Kit.” NASA, July 2005, p. 51.

5 Return to Flight Task Group, Final Report of the Return to Flight Task Group (Washington, DC: NASA, ID: 20050201800, July 1, 2005), p. 88.

6 Ibid, p. 92.

Chapter 16

STS-114 (Part 1)

Recognize When It’s Time to Move Forward (You Can Never Reduce Risk to Zero)

Over the length of my NASA career, I’ve had the opportunity to visit each of our agency’s field centers, geographically distributed across the nation—east, south, and west. The majority of them I’ve visited multiple times until each has become a familiar place. I spent 21 years at JSC, near Houston Texas, and (as of this writing) somewhat over 10 years at NASA HQ in Washington DC, so those two are perhaps the most familiar to me. But through the course of attending meetings or reviews, to see hardware or tour test facilities, or even simply to chat face-to-face with colleagues, my career in MOD also necessitated visits to the KSC in Florida and the MSFC near the Army’s Redstone Arsenal in Huntsville, Alabama. My time in the Science Mission Directorate mandated travel to the Goddard Space Flight Center (GSFC) in Greenbelt, Maryland, and the Jet Propulsion Laboratory outside Los Angeles in Pasadena, California. As chief engineer for aeronautics at NASA HQ I have had frequent need to venture to the agency’s four research centers, include the Ames Research Center (ARC) in Mountain View, California, near Silicon Valley; the Armstrong Flight Research Center (AFRC, formerly the Dryden Flight Research Center) in Mojave, California, abutting against the Edwards Air Force Base; the Glenn Research Center (GRC, formerly Lewis Research Center) near Cleveland, Ohio; and the Langley Research Center (LaRC) in Hampton, Virginia—NASA’s oldest center.

At each of these NASA locations, I have found a certain area, perhaps a spot or corner, where I can enjoy a momentary semblance of peace and tranquility. Within the hustle and bustle of daily technical meetings, design reviews, or facility visits, there is at least one location at each NASA center to which I can retreat; a quiet, relaxing area where I can get away from the activity, find some solitude, and attend to a bit of relaxation. At some centers that may be the facility’s cafeteria, a few of which have constructed cordoned-off areas specifically designed for contemplation or discrete one-on-one discussions. For the other centers, some may have a particularly well adorned conference room. One center has a room which I will attempt to commandeer as long as it is unoccupied with panoramic views of the center’s campus displayed outside the windows. But each has a place like these and over repeated visits I’ve utilized them all for comfort, downtime, and an opportunity to recharge.

Of all of these special places, my favorite was at the JSC, and as my center of residence for many years I got to harbor in this place rather frequently. When I was exhausted from hours in crowded meeting rooms, in need a bit of isolation to refresh from hectic runs of training simulations, or simply needed a bit of time to myself, this one place was my go-to location. It was my refuge. Unfortunately, it was also often occupied, assisting in its duties of advancing the nation’s human spaceflight efforts. But sometimes it was available, particularly in late afternoons. When it was unoccupied and I needed some solitude I’d retreat there, even for just a half an hour, and revive myself.

It may surprise you to learn that my favorite location was the Shuttle Mission Simulator.

Located in the windowless, three-story-tall frame of Building 5, just across from my resident office Building 4, the Shuttle Mission Simulator (or SMS as everyone called it) was a high-fidelity mockup of the Orbiter’s flight deck. It was used by astronaut crews throughout the 30-year history of the Shuttle program to train for upcoming missions, practice their procedures, and work through malfunctions with flight controllers in Mission Control, honing their skills to a fine precision. The SMS emulated all the functionalities of the spacecraft so that every switch, every circuit breaker, was simulated to perform the actual function as would the real vehicle. There were two SMSs, actually. One was a “Motion-base” simulator that included only the forward control stations of the cockpit on a platform that could rise, fall, and move on sets of stout hydraulic pistons and was used to train Shuttle crews of the dynamic portions of the flight, namely launch and reentry. The other was the “Fixed-base” simulator, which incorporated an exact replica of the entire flight deck—forward and aft stations—creating an environment that was indistinguishable from being inside the real spacecrafts (except for the presence of gravity, of course).

The path to the SMS was somewhat circuitous. Enter a door at the front of the Building 5 and you’re deposited into a small vestibule—almost like an airlock—with a card reader. Slide your badge into an accommodating slot, key in your Personal Identification Number, and assuming you had clearance for access the locks of a heavy metal door would release and you could proceed into the building.

A small lobby adorned with models of the old Gemini and Apollo training simulators led to a long, straight hallway. On each side of the hallway hung portraits of each Shuttle crew, beginning with the very first, STS-1, and chronologically tracking the history of the Shuttle Program through the decades and the sundry missions.

Turn left and walk down a second hallway (also lined with portraits of Shuttle crews) until a door was reached, leading to what could only be referred to as a high bay, perhaps 25- or 30-feet tall, which enclosed the two simulators—the Motion-base to the right and the Fixed-base slightly to the left.

Figure 44. The exterior of the Fixed-Base shuttle mission simulator, a.k.a. SMS. (Photo taken by the author)

Figure 45. The ladder reaching up into the SMS to access the flight deck. (Photo taken by the author)

The Fixed-base laid behind sheets of heavy, transparent plastic, which could be brushed aside to allow passage. At first, you’d find yourself in a low-fidelity mockup of the Orbiter’s middeck. This area was strewn with very un-Shuttle-like tables and chairs and was used more for discussions than actual immersive training. However, mounted next to the near-wall rose a plain metal ladder, rising nearly but not quite vertically, that entered a square hole in the ceiling. Climb up the ladder, squeeze yourself through the portal using only a single handhold set flush into the metallic floor, and, finally, you enter the realm of spaceflight.

If you’ve never been in the Shuttle’s flight deck before, there’s not a lot of room. The flight deck looks similar to many aircraft cockpits with two hard-mounted seats for the pilot and co-pilot (or in Shuttle vernacular, the Commander and Pilot) and copious banks of switches, dials, circuit breakers, and other controls along with monitors and displays. These surround the occupant with the entire volume covered with controls from the front to the sides to the rear and even above. The only surface that wasn’t overflowing with controls was its floor, which was a flat casting of shiny, smooth aluminum.

Generally, it’s dark in the SMS. Each of the 10 windows on the flight deck were backed by television monitors which could simulate (in polygon form) Earth or, through the aft-facing windows, the payload bay. But when not in operation the screens were jet black, providing no lighting from the windows. Some of the illumination inside the cabin came from a set of floodlights (with rotary dials to brighten or darken) and the monitors that displayed information to the crew. But the majority of lighting came from the backlit panel illumination, revealing the wording for each switch or circuit breaker inscribed on the flat panels and lit from behind. This panel illumination was normally turned down when I would enter the SMS, but I would dial it all up to maximum brightness, casting an amorphous bright amber glow into the entire volume.

Spinning fans were always turned on, circulating the cooled and dehumidified air to ensure heat was removed from all the lights, electronics, and people within the closed space. The entire volume was about as large as the interior of a car accounting for both the front and rear seats. There was perhaps four feet of open space between the commander’s and pilot’s chairs and the rear console of the aft flight deck, just enough room for a few people to stand slightly hunched over.

Taken all together—the cramped space, the darkness offset by lighted control panels, and the persistent hum of circulation fans—the SMS was like a womb to me: comforting, enveloping, and a tiny oasis away from people. It was technological in nature, but I found it as free from strife and disturbance as much as I would a pastoral mountain meadow.

Often, I would climb up and settle myself into either the Commander’s or Pilot’s seat. Sometimes I would bring work with me, parsing through presentation material for upcoming meetings or reviewing minutes of those already conducted. But other times I would bring a book with me, maybe a novel or a paperback story of some science fiction or fantasy tale I was reading, and in the white-noise quiet and semi-darkness, descend into the story and lose myself.

My need to periodically break away and gain a short respite of solace became ever more present as we entered the year 2005 and all the Return-to-Flight activities were reaching a fever pitch. Almost two years beyond the accident it was possible for us to take a look back at where we have been. We had recovered as much of Columbia’s wreckage as was possible. We had furthermore determined with high confidence the cause of the accident and identified mitigations and design changes to prevent such a tragedy from occurring again. And we had begun work on implementing those changes. It was a heady time of accomplishment and achievement, and also of some optimism that it just may be possible to get back to flying Shuttle missions again.

The CAIB’s final report included 29 separate recommendations covering the physical improvements to the Shuttle’s thermal protection system, imaging of the spacecraft while in flight, and gaining data on debris strikes. They also made recommendations on the SSP’s (and NASA’s at large) organization, decision-making, scheduling, and training, and touched on the recertification of the integrated architecture beyond Return-to-Flight. Add onto these formal recommendations the board’s 27 observations and 137 findings and the path for NASA improvement had been laid.

In response, NASA’s RTF Task Group focused their efforts on 15 of the 29 recommendations—those considered mandatory to launching Shuttles again—while tracking NASA’s implemented solutions. As for the remainder of the recommendations, the RTF Task Group transitioned them to the Aerospace Safety Advisory Panel, NASA’s congressionally authorized independent safety oversight body, which would continue to monitor the agency for compliance.

However, more oversight was to come. NASA Administrator O’Keefe also chartered a high-level team to shine a spotlight on those recommendations, observations, and findings from the CAIB report that might apply across the agency, even outside of the Shuttle Program. This team was to also identify a set of actions and to suggest leadership to implement those actions. To fulfill this duty, O’Keefe selected Al Diaz, the director of GSFC, to lead the team (and thus it forevermore became known as the Diaz Team), with members including Dr. Ghassem Asrar, the associate administrator for Earth Science at NASA HQ; Dr. Julian Earls, the director of GRC; Scott Hubbard, director of ARC; Jim Kennedy, the director of KSC; and Vicki Novak, associate administrator for human resources at NASA HQ. This team of senior leaders took a holistic view of NASA as an organization and aimed to provide recommendations on improving the agency systemically, beyond just the causes and remediations for the Columbia accident or corrections to the structure and operations of the Shuttle Program. Within the resultant report—the Diaz Report—recommendations were made which affected the very fiber of how NASA operates as an agency, across the 10 field centers, including every program and project, every technology development, and all activities inimical to NASA’s mission. One of the most significant of these improvements was the institution of an independent Technical Authority, independent of the authority held by the programs and projects, which would provide a check and balance on NASA decision-making on areas that pertain to safety and mission success (see Epilogue for more further thoughts on Technical Authority). The Diaz Report, as much as the CAIB, constituted one of the enduring legacies of the Columbia tragedy.

We had covered a lot of ground since that harrowing day in February 2003. Herculean efforts predominated and became routine for us. We sought answers to great mysteries and found them or at least found plausible explanations. We challenged the hardware to improve and challenged ourselves to improve, recognizing our deficiencies and organizational shortcomings and endeavoring to hold ourselves to a higher standard of excellence. We made the Shuttle components more resilient and gave ourselves capabilities to reveal damage that hadn’t been invented before the accident. We outlined what needed to be done in order to return the program to flying in space again and worked to accomplish those tasks.

But as we reached the closing days of this long journey, one question persisted throughout the ranks of those I worked with: Was it all enough?

Wayne Hale, a former flight controller and flight director with whom I flew a number of Shuttle missions on console (and who would go on to become SSP manager in 2005), and a friend of mine, was famous with MOD for offering the sage wisdom that the only way we would ever be 100 percent certain of safety was to stay on the launch pad and never launch into space. As the prominent American author and businessman, John A. Shedd, similarly stated, “A ship in a harbor is safe but that is not what ships are built for.”1

That sentiment permeated much of the cadres working to return the Shuttle to flight, including the astronaut crews themselves. During my years as a flight controller, I worked with many astronauts, both rookies and veterans, and I often heard comments from them that they all were aware of the risks inherent of riding a rocket. In fact, astronaut Mike Mullane, who flew on three Shuttle missions in the 1980s and early 1990s, authored a memoir of his astronaut experiences using just that title, Riding Rockets, in which he recounts in stark terms just how frightening a launch could be from the vantage of a seat inside the rocket. And yet, with clear knowledge of the risks involved, every astronaut I worked with accepted that risk and still flew on the Shuttle because, simply, important missions were to be had. Sending an orbiter into low-Earth orbit was a voyage with a destination, and the purpose of making that voyage was to accomplish important things for the nation and for humankind.

But the Columbia accident was like being thrown into a frozen lake, breaking through the ice-rimed surface, and plunging into breathtakingly cold waters. It was a shock to the system, emotionally as well as physically, and few of us were absolutely confident that they were ready to traipse out again onto the icy lake. Nothing is ever certain, and questions remained. Maybe there were other causes of the accident that we didn’t identify? Perhaps we needed to run more tests to make sure our solutions were bulletproof? Had we reduced the risk of a similar accident to a point that we were confident the crew would survive?

While persistent questions like these can be nerve-wracking, such doubt can also be helpful in that it promotes rigor. But too much rigor seldom gets one to their goal; that is, it’s possible to be paralyzed by doubt. At no time would we be able to guarantee success or to reduce the risk of what befell Columbia all the way to zero. That just wasn’t possible, and most of us recognized that neither was it necessary in order to get back to flying. What we could do, however—what we would do—was to reduce those risks to a level that was acceptable. Not all the way to zero, but to within an acceptable level, and proceed forward into this dangerous territory with acknowledgement and with our eyes wide open. At some point then, the question became, have we done that?

The Certification of Flight Readiness process, or CoFR, was a detailed, methodical, and systemic exploration of the readiness to go launch and execute the mission plan. CoFR was used for every Shuttle mission—both before and after STS-107—and is still used today, remaining one of the bedrock tactics of human spaceflight safety and mission success. It starts small, at the level of components and individuals, assessing whether the condition of that piece of hardware or preparation of that operator or technician is sufficient to risk human lives to a mission. From those discrete levels the CoFR process gathers those parts into subsystems and those individuals into groups, ascertaining the readiness of the collections. Then it integrates subsystems into systems and groups into divisions or entire organizations. At each step the focus gets less tactical and more strategic, until it all comes together at the penultimate Flight Readiness Review where all the major element leaders of the program are in attendance. The timing of these readiness statements was well choreographed and aligned with a repeatable schedule which could be duplicated from mission to mission.

Our first post-Columbia flight was to be STS-114 on Space Shuttle Discovery. That mission, identified and manifested before Columbia ever flew on STS-107, was dedicated to resupplying the ISS. The crew had been assigned, including a new three-person Station crew (the ISS crew on orbit would return on Discovery) and the payloads manifested. As mentioned back in Chapter 1, our MOD Systems Division was immersed in the STS-114 CoFR process in January 2003 when Columbia was in orbit. After the accident, all planning for STS-114 of course ceased, but after two years of work and as we began entertaining the possibility of flying again, the STS-114 planning process was reinitiated. The original ISS crew who would launch with Discovery had found other means to get to the station (namely on the Russian Soyuz spacecraft) and had served their tour while the Shuttles were grounded, so those three slots were opened to other members of the astronaut corps.

The crew needed to be retrained on new mission parameters, namely those dedicated to verifying the capabilities developed during the hiatus, activities like the Rendezvous Pitch Maneuver and inspections with the Orbiter Boom Sensor System. A number of this crew were involved in the development of these new capabilities. Some of them would on occasion drop by the OPO meetings in Conference Room 620 as we discussed the development and certification of this new hardware. They would ask insightful questions and peer deeply but politely into the development challenges and hurdles we faced. Never, however, did they offer any doubts of their own, always maintaining a positive view and a belief in the ingenuity of NASA engineers. It did not miss my, or anyone’s, attention that no one in that room had more to lose then the crew of STS-114 themselves, and their faith in us was as much motivation for us to persevere as was anything that could be offered.

Eventually, most felt that we were ready to launch again. So did I. The only area where I felt we didn’t meet minimum success criteria was in our failure to certify an in-orbit tile repair capability. STS-114 was to fly kits that could be used in the event of an emergency, but they wouldn’t be certified capable of performing their function—that is, to protect an area of the orbiter where tile had been damaged, warding off the extreme temperatures of reentry and preventing searing gases from burning through the aluminum structure. Instead, the kits would only be certified as “safe to fly” and not damage the spacecraft and crew. But as to whether they could be used or not was an unanswered question. As mentioned in the previous chapter, that capability eluded NASA’s and industry’s best efforts and the omission was a glaring one to me.

The Flight Readiness Review was held on June 30, 2005, at which time a launch date of July 13 was approved. As new NASA Administrator Michael Griffin at the time stated to the media, “After a vigorous, healthy discussion, our team has come to a decision: we’re ready to go …. The past two-and-a-half years have resulted in significant improvements that have greatly reduced the risk of flying the Shuttle. But we should never lose sight of the fact that space flight is risky.2 Former astronaut Bill Ready, NASA’s associate administrator for space operations, also offered his thoughts, stating, “Today’s decision is an important milestone in returning the Shuttle to service for the country. Our technical and engineering teams are continuing their in-depth preparations to ensure that Eileen [Collins, the mission’s commander] and her crew have a successful mission.”3

While the accolades from the agency’s senior leadership were comforting, another aspect of our readiness to return to flight was weighing on a number of us. About a week before launch, in the late afternoon of a sultry Houston summer, I gathered at Boondoggles Pub, only a few miles from JSC, with a few old flight controller friends and a few new colleagues from the OPO. We were not senior leadership, rather just a bunch of mid-career managers and engineers with considerable experience in the Shuttle Program. But while our decisions didn’t dictate the course of the agency, we also were not oblivious to the shortcomings of our beloved organization.

Beyond the hardware-related causes of the Columbia accident—the failed ET foam and compromised RCC Panels—the CAIB shined a brilliantly hot spotlight on some pernicious organizational deficiencies in how NASA operated and made decisions. In NASA parlance, the ET foam striking Columbia’s wing leading edge RCC panels, causing a breach that allowed hot plasma to enter the wing structure during reentry and eventually lead to the disintegration of the spacecraft and her crew, were the proximate causes of the accident. From the Office of Safety and Mission Assurance’s policy document, proximate causes are defined as:

The event that occurred, including any conditions existing immediately before the undesired outcome, directly resulted in its occurrence, and if eliminated or modified, would have prevented it. Also, known as direct cause.4

But while proximate causes are what is seen, noticed, heard, or experienced, the underlying reasons for an accident such as Columbia are frequently unseen unless uncovered. Often, it is not simply the hardware that fails but is the organizational deficiencies, culture, or practices that lead to the failure of the hardware. These organizational deficits metastasize into the practices and behaviors that contribute to the eventual failure of the hardware, and these root causes often lie in the shadows, noxiously hiding (sometimes in plain sight) until the cancerous effects of omission, complacency, and normalization of deviance penetrates our practices and result in eventual tragedy.

The CAIB was not shy in pointing out these deficiencies in Shuttle Program culture. And there weren’t just one or two failings, but an entire set of them. These infirmities included:

A reliance on past success.

The lack of testing to understand system performance.

The stifling of professional differences of opinion that prevented communication.

A lack of integrated management across program elements.

Evolution of an informal chain of command allowing for decision-making outside of the formal process.5

As we sat on the wooden benches of an outdoor picnic table on Boondoggle’s concrete front porch, sipping our ales, beers, and cocktails, my colleagues and I solemnly discussed these points. None of us had yet entered into NASA at the time of the Challenger accident in 1986, yet each of us worked with people still in the agency who did experience that horrific January day. The NASA culture each of us knew had, we thought, been informed by the experience and lessons of Challenger, building resilience into our decision-making that we trusted. And yet, each of the five organizational elements above identified by the CAIB could be similarly tracked as root causes of the Challenger accident as well as Columbia. How much, then, had we really learned?

While JSC is a large center with thousands of people working there, word still gets around. We all were aware of Rodney Rocha’s frustrated attempts to convince someone—anyone—in Program leadership to gain some in-flight imagery of Columbia and ascertain definitively whether there was a safety issue or not. No one did, of course, and NASA processes at the time didn’t afford Rodney a path to elevate his concerns above those who denied his requests. Although I worked with Rodney through the OPO, he and I did not directly discuss his efforts during STS-107. But I did hear from colleagues of his who knew much more intimately and directly of Rodney’s pleas and the deep antipathy he received in return. While we learned after the accident that some queries to those who operate the associated national assets were considered, it was clear that not enough was done. Rodney’s concerns were stifled and organizational barriers prevented elevation to others who might have taken action. An allegiance to hierarchy and procedure had replaced deference to an engineer’s technical expertise.

So, we sat, and drank, and lamented. Our waitress came to deliver another round and, overhearing our conversation, asked us about it. While she didn’t work at NASA, those who lived in close proximity to JSC all formed the broader community of suburban villages and towns that surrounded the center. We were all family. Obligingly, we shared with her our thoughts, and she listened generously and tenderly but also with a keen, analytical interest. After a few minutes of discourse, she needed to attend to her duties and left us to ponder the true state of NASA.

Life Lesson for Engineers, Managers, and Leaders

Recognize when it’s time to move forward (You can never reduce risk to zero).

Contrary to popular and somewhat misapplied belief, the role of the engineer is not to reduce the risk of a thing or event to zero. Rather, it’s to reduce the risk to acceptable levels. No one short of a deity can actually reduce all risk to zero, and the best we simple mortals can do to that end is to avoid the activity completely, which is counterproductive. While we can’t reduce all risk to zero, we do however try very hard to mitigate or control what we can and minimize both the likelihood and consequence of a catastrophic event to levels that we can live with. All projects have finite budgets and limited schedules, and managers have to be selective in where to invest effort. Once we can reduce risk to acceptable levels, we can then allow ourselves to move on to other risks which we deem as unacceptable and then reduce those as well. It is a continuous effort until, by the time we launch or fly, we are comfortable in the system’s ability to successfully perform the mission.

The trick in all this, of course, is determining what acceptable is and where it resides. That’s not always easy; in fact, it can be balefully difficult. There is no science known to mankind that can purely quantify the level of acceptable risk because, inherently, an acceptable level is a subjective thing. Each person’s level of acceptability may be different, based on intent, desire, values, and past experiences. Everyone wants an endeavor to be successful, but the level of risk that remains unmitigated—the residual risk—can be a source of vibrant and rich debate, and even conflict.

The situation can be even more confounding for leaders as they have the additional challenges of politics, team dynamics, and influence (sometimes measured in decibels). Knowing when to continue studying a problem, how deep or how broad to address the issue is definitely a challenge; and one that can result in “paralysis by analysis.” Good leadership can leverage the engineer’s viewpoints to minimize potential biases while also encouraging new ways of thinking. Or said differently, do not expect to turn the ship overnight; instead set realistic expectations and work toward common, achievable goals.

Eventually, however, decisions must be made, as time and resources are not unlimited. Those can be difficult decisions to make and rarely are done without some disagreement. But it is incumbent in every engineer, manager, and leader to know when it’s time to move forward. As perviously stated, A ship in a harbor is safe but that is not what ships are built for. Our mission is to explore, to invent, to research and unlock the mysteries of physics and the universe, to push the envelope of what has previously been possible and expand the boundaries of what humans can achieve. All of those endeavors include some amount of risk and knowing when to call it “good enough” and start the mission is a skill for all to develop.

1 John A. Shedd, Salt from my Attic. (The Mosher Press: Portland, Maine, 1928), p. 20.

2 “NASA Gives Go for Space Shuttle Return to Flight.” Air Force, June 30, 2005. https://www.af.mil/News/Article-Display/Article/134050/nasa-gives-go-for-space-shuttle-return-to-flight/ (accessed Jan. 27, 2025).

3 Ibid.

4 “NPR 8621.1D Appendix A. Terms and Definitions.” Office of Safety and Mission Assurance: July 6, 2020.

5 Harold W. Gehman, Jr. et al., Columbia Accident Investigation Board Report Volume 1. (Washington, DC: Columbia Accident Investigation Board, August 2003), p. 9.

Chapter 17

STS-114 (Part 2)

Failure is a Part of Life, but Don’t Forget to Celebrate Good Work and Success

On July 26, 2005, 882 days since the Columbia accident, we were ready to launch. I was back in the control center, once again sitting behind a console and adorned with headset, listening in to the NASA test director’s loop streamed from Launch Control at the KSC in Florida. Discovery rose atop the launch pad at Complex 39-A, billows of vapor and released gases swirling around her bulbous but elegant form. Fueling of the ET had begun at 12:48 a.m. that morning. Now, some hours later, the seven crew members had already entered the spacecraft, assisted by ground technicians and other astronauts, secured into their seats by five-point harnesses that held them firmly and snug. There they awaited their release from the bonds of Earth and the shackles of gravity.

I was on console but not in the Flight Control Room nor surrounded by fellow flight controllers. I was in the MER, occupying the SPAN, or Spacecraft Analysis, console, and my companions were the managers, engineers, and subsystem owners of the program’s OPO and JSC’s Engineering Directorate. The MER is considerably larger than the Flight Control Room that hosts us flight controllers and, given the extra volume, an inordinate number of people were on hand. Undoubtedly this was due to the return-to-flight nature of STS-114 and the fact that we hadn’t launched a Shuttle in two and a half years. A palpable anticipatory energy spun through the entire room, electric to all attendants, as we waited patiently through the long countdown.

The SPAN console position traced its lineage back to at least the Apollo program, perhaps earlier. In those programs, SPAN had been the primary interface between the flight controllers and the industry managers and engineers who had built the spacecraft and rockets we operated. Over the years that direct interaction between builders and operators diminished, and by 2003 all communications with industry engineers now occurred purely through the MER engineers. If the flight controllers had a question about a certain piece of hardware’s pedigree or history, they would send a note (referred to as a chit) to the person at the SPAN console who would pass the query to their counterparts in the MER, who would then forward the question on to the industry partner. As a result, while the SPAN console was still maintained, it was now largely relegated to the function of liaison between the flight control team and the MER. I had volunteered to work the mission of STS-114, in addition to my MOD-to-OPO duties, and my offer was accepted by MOD leadership.

Few technical issues of any significance were being worked, and the launch remained on track for 3:50 p.m. The biggest uncertainty remained the weather, with forecasters predicting possible showers and thunderstorms in the area as launch time approach. But the probability of launching was estimated to be 80 percent, so we remained hopeful.

I had been through countdowns numerous times before during my flight controller career, working the prelaunch shift on 15 Shuttle missions and the Ascent shift on another 10, each of these requiring any number of countdowns before successfully launching. The cadence of a Shuttle countdown is slow, methodical, and very, very precise. It followed a printed checklist called the S0007 (pronounced soo-seven) which ran to over 3,000 pages, necessitating three separate loose-leaf binders at the SPAN console (and most others). Nothing was rushed during a countdown and each step was checked and rechecked to ensure patient and accurate compliance. Although much of the countdown was automated, it still required the eyeballs of hundreds of controllers to verify that steps were performed correctly and that the combined Orbiter, ET, SRB, SSMEs, launch pad, ground infrastructure were in the right configuration to commit to launch.

At T–9 minutes, things get very real. That was the point that the Ground Launch Sequencer (GLS)—a collection of computers at the launch complex—takes over and we get into the final minutes of the countdown. During much of the countdown the hours of waiting seemed little different than any other task, but when we get into the terminal count and the GLS takes over, my heart rate would always increase noticeably. At this point, the pad technicians and support astronauts had departed the pad area for more safe environs, ready to race back should an emergency occur. The hatch to the spacecraft had been closed and locked and no one except for the astronauts sitting on top of hundreds of thousands of gallons of rocket propellant were anywhere near the explosive mixture. Astronauts had told me that this time frame was a period of loneliness and isolation for them, as if they were fodder in an arena awaiting the release of a ferocious beast, with cordons of crowds watching from the elevated, protected bleachers.

The crew access arm retracted away from the orbiter at approximately T–7 minutes, and at T–5 minutes another milestone is reached. At this point Launch Control directs the pilot to start the spacecraft’s three auxiliary power units (APUs), which help control the vehicle during ascent and reentry. While the countdown still can be stopped after this point if everything is not perfect, the orbiter carries just a finite amount of fuel for the APUs. If a scrub were to be called here there would not be enough APU fuel to perform an entire mission, resulting in a lengthy delay of days before another attempt could be made. Starting the APUs was a commitment to trying to launch and a recognition that things were in fact going well.

At three and a half minutes before launch valves close forcing the Orbiter’s three fuel cells to begin feeding off the cryogenic supply carried aboard the spacecraft rather than from ground sources. At T–50 seconds, ground power is backed off and the fuel cells now were powering the entire vehicle.

T–30 seconds is the point when the GLS hands off to the Orbiter’s own computers. A final Go for launch is announced. Now my heart would really begin to race. Of all the computers and sequences—which for hours had moderated this complex and choreographed series of actions—when the Orbiter’s onboard computers take over even the LCC hasn’t the ability to stop the launch or call for an abort. Only the Orbiter’s computers could do that now. At this point, only a problem with the SRBs or with one of the three main engines could prevent a launch. With my heart close to palpitation, I would slide closer to my console, peer intently at the hundreds of telemetry parameters on my computer monitor, watching with the intensity of a hungry lion stalking a wily prey for any indication of an anomalous signature, breathe deeply, and maybe even say a prayer.

Three main engines would roar to life, sending billowing clouds of steam and water vapor to one side of the launch pad. The Shuttle stack, firmly bolted to the pad by hold-down posts on the SRB’s aft skirts, would slowly lean forward with the millions of pounds of force of the three engines, stop, and then return to a perfectly vertical position again. At that exact moment the enormity of the SRB would ignite. Those bolts would release and, paraphrasing more than a few astronauts I knew, You knew you were going somewhere!

LIFTOFF!!!!

And then something very unexpected to me happened. See, in the conduct of the hundreds of ascent simulations I had worked over the years, at the moment of liftoff only a single voice would be heard over the communication loops. That was of the Data Processing System (DPS) flight controller, who while monitoring the onboard computers would receive a bit of telemetry indicating that the orbiter had sensed the liftoff and would announce “Liftoff confirmed.” No other voices would be heard in the room. There would be no banter, no idle chit chat. The flight control room is all business, invoking a protocol of reporting only when necessary. During simulations the training team would routinely insert systems failures which would necessitate communication to the flight director and, when action was required, requests to the launching crew to throw switches or input commands into the flight computers. But during real launches, when failures were rare, the FCR was normally intensely quiet, not unlike a library, and except for limited discussion over the headset with my backroom counterpart and the CAPCOM reporting nominal milestones to the crew, all else was silent.

But not in the MER. At the exact moment of liftoff, when Discovery had broken her connection with Earth and blasted skyward on pillars of flame, the entire room erupted into a wild chorus of cheers and applause. It was cacophonous and nearly deafening, and I was unprepared for the noise. Annoyed, I was tempted to scream out louder than the combined hoots and hollers of 200 people for them to “quiet down!” but I resisted that temptation. This moment, I realized, after two and a half years of exhaustingly hard work, was for them too.

Hope had been restored. And then just as quickly, it was tempered. We saw something fall away from the ET. Small cameras the size of a thumb had been mounted on the ET and were being downlinked live, offering those on the ground (including the MER) with real-time views of the combined stack as it rose thunderously off the pad. One camera was mounted high on the ET looking downward with Discovery clearly in the field of view. As the Shuttle gains speed, the combination of velocity and air density create a pressure on the vehicle which continues to increase until the stack has risen above most of the denser atmosphere. This dynamic pressure, what we call q-bar, is one of the mechanisms believed to cause foam to break away from the ET and SRBs and potentially strike the orbiter. And now we were watching it live.

Two minutes into the flight and a few seconds after the SRB had completed their job and had separated from the still attached orbiter and ET, a large piece of foam was seen tumbling away from the ET and towards our precious spacecraft. But whether by aerodynamics or the grace of a higher being, the section of errant foam passed beneath the orbiter, never touching Discovery’s sensitive surface, and continued downward and out of sight, likely landing somewhere in the Atlantic Ocean offshore from Cape Canaveral.

“Did you see that,” a number of voices in the MER exclaimed in surprise and disbelief. Whether spoken as a question or a statement I couldn’t tell, but the murmur spread throughout the room like the foreboding haunt of past ghosts. During the rest of Discovery’s and the STS-114 crew’s ascent into space, the MER transformed from the party-like the celebration of a beloved college football team after a game-winning touchdown to the solemn reverence more akin to a congregation in a house of worship during a memorial service. That is, the whole room hushed up!

After Discovery reached orbit and analysis was performed on the launch video, the portion of ET Tank foam believed to have broken away was traced to the protuberance air load (PAL) ramp on the side of the external fuel tank. The ramp is there to smooth the airflow over a cable tray and two pressurization lines as the tank goes supersonic. The errant section was estimated to have measured 36 by 11 by 6 inches—sizable proportions!—and weighed about one pound, or about half as much as the foam blamed for striking Columbia.1 In fact, imagery of the ET taken by the crew after separation revealed multiple areas where foam insulation was missing.

Figure 46. Imagery taken by the STS-114 crew after separation of the external tank, showing the region of the protuberance air load ramp where foam debris had separated. (NASA Image JSC2005-E-30569)

Yes, Discovery had safely made it to orbit and the crew was well. But … damnit, after all that work to prevent debris coming off the ET and SRBs—two and a half years of difficult and challenging analysis and redesign, dozens of presentations and hundreds of discussions, hours of technician training and intensive post-application inspections—after all of that, we still hadn’t licked the problem. Foam was still coming off these things, despite our very best efforts. The feeling of letdown, that we had let ourselves down, was palpable.

Still, STS-114 on Discovery showed just how unlucky Columbia’s mission of STS-107 was. A large consideration for the damage that resulted on Columbia was the relative speed of the debris striking Columbia. Because the STS-107 debris came off earlier in flight, lower down in the denser atmosphere, the foam decelerated from the stack’s velocity of nearly 1,500 mph to just 1,000 mph, all in a half a second. That deceleration caused a difference of 500 mph between the debris and the still ascending Columbia. On STS-114, the debris was shed higher in the thinning atmosphere and the foam had thus decelerated less. If it did strike Discovery, the relative difference in speed might have prevented damage.

During the post-launch press conference, Flight Director John Shannon, representing MOD, conjectured on what it would mean if this debris did hit Discovery. He opined that, “It’s bad. I’ll tell you that right now, it’s bad. You could hit the Orbiter and so that’s why you’ve got to go fix it.”2

The problem, however, was that we did not have anything presently on the drawing boards to replace the ET’s PAL ramps. Also at the press conference was Administrator Mike Griffin. Discussing the launch video, Mike stated,

The guys are going to take a professional look at every frame of footage that we have from every camera that we have. Because as I’ve said, these are test flights right now. The primary object under test is the external tank and all of the design changes we made to that tank so we’d never have a repeat of (Columbia).3

But as dour as the mood was, some still tried to kindle the spirit of optimism and, perhaps, invoke a touch of realism. STS-114 astronaut Stephen Robinson, a veteran of three previous spaceflights and who served as one of the crew’s mission specialists and the lead spacewalker, reflected philosophically during a prelaunch press conference on the nature of the environment he was about to enter and on the technological Return-to-Flight advantages NASA’s engineers and managers had provided him and the crew. Waxing openly, he said,

There will be debris, there will be some damage, I’m convinced of that. If there isn’t, that’ll be great but I’ll sure be surprised. I would be very surprised if it’s critical damage, damage that won’t allow us to fly home on. But here’s the thing. We’ll know it. We won’t have to wonder. We’ll know it. We’ll have the technology now for the first time on this mission to take a look at it with all the cameras and sensors. This is the way we verify all the engineering that’s been done. So, we’ll get to look at our bird before we come home. Then, on top of that, if the worst on worst on worst happens and we do have critical damage, the space station will (be available for safe haven), we won’t have to risk our lives coming back through the atmosphere. This is what gives me tremendous confidence and makes me feel very lucky I’m flying now.4

On the third day of the mission, prior to docking with the Space Station and hovering 600 feet directly below that great ship, Commander Collins placed Discovery into the correct attitude with the payload bay facing the ISS and began the Rendezvous Pitch Maneuver, that slow, languid pirouette in space that allowed the Station crew to photograph every inch of surface of the arriving Shuttle. We watched the drama unfold live on TV, captivated by the spectacle. Since ISS construction began in 1998 it had become common to see views of the approaching orbiter as it inched closer and closer in preparation for docking. In those approaches, however, only one side faced the Station, that of the open payload bay, where the docking mechanism was located. The slow, careful rendezvous was always fun to watch as it was relatively rare to capture our spacecraft in flight, but had it been a cardboard cutout of the Shuttle one would barely know the difference as only this one aspect of the beautiful machine was presented to the cameras. However, on this flight, we would get to see our beloved vehicle from many different angles, and that was truly exciting.

The pitch, or a 360-degree flip, of the spacecraft was performed slowly and precisely. It was difficult to tell when it actually began, although the crew confirmed the start of the maneuver over the air-to-ground communication channel and flight controllers in Mission Control concurred as they watched the spacecraft’s Inertial Measurement Units reveal an infinitesimal rotation rate. The contradiction of watching this orbiting spacecraft, moving at 17,500 miles per hour, rotating with the speed of a snail (or slower) promoted a sense of cognitive dissonance that only occurs with the relative speed contradictions of spaceflight. But after five or so minutes of frustratingly slow rotation, we could begin to see the aft end of the orbiter. Main Engine bells came into view, looking like circular cars that crashed into a wall and formed a perfect triangle. Two smaller OMS engine nozzles, red against the blackness of thermal protection tiles, could also be seen, jutting like wine goblets. It was also fascinating to watch because no one ever had seen the Orbiter’s in flight from this vantage point before.

Eventually, the engines rotated out of view and ploddingly, as if unhurried, the bottom of Discovery began to reveal itself, like a shy child cautiously emerging from behind a curtain. At first it showed its smooth, blackened surface from an oblique angle, but as time moved forward the full profile of the Shuttle’s hypersonic outline took form. That very same outline, exact in all dimensions, was included in the STS-107 mission patch, and I suspect I was not the only one to take notice of that fact. From what I could see on TV there didn’t appear to be any obvious damage to Discovery’s underside, but I knew that the ISS crew was rapidly scanning the surface using cameras with long telephoto lenses. If any damage was present, it would be revealed once their images were downlinked to the MCC.

But there was another element here that enraptured me. Beyond the spectacle of the Shuttle itself or the sensation of reality watching it live, there was also an immense beauty that was being revealed here, an evocative majesty that this ballet in the vacuum of space suggested. The movement of rotation was smooth, effortless, without friction or turbulence. As Discovery flipped end-over-end, behind and below her rotated a serene landscape, a wash of reds, ochres, salmons and beiges, other colors that were reflected from the Russian and Central Asian landscape. One only had to overlay this scene with the orchestral splendor of Johann Straus’s Blue Danube, almost as if it were a scene taken from the movie 2001: A Space Odyssey. It was human spaceflight at its most majestic, a resplendent reminder that, while dangerous, spaceflight could also be engrossingly beautiful.

Another of the primary objectives of the STS-114 mission was to grasp the Orbiter Boom Sensor System by the Shuttle’s robotic arm and use it to scan Discovery’s wing leading edge RCC panels to ensure their integrity to withstand the forces and heat of reentry. While the sensors attached to the tip of the OBSS were pathfinding from a Shuttle perspective, sensors and scanners on the ground could provide significantly better insight (particularly through the use of x-ray and thermography) than those on the OBSS, but we had what we had, and it was a leap over the (essentially) nothing we had flown with before. In practice, the scanning of the RCC panels proved to be … well, pretty boring. The scan-rate of the instruments was such that the astronaut crew could only move the arm at a pace of two or three inches per second, so the entire process of imaging both wings required a few hours. To most of us on the ground, each panel looked pretty much the same. I recall sitting at the SPAN console in the MER and watching the video downlink from orbit, watching on the TV perched atop the console’s blue metallic frame. At first it was fascinating just to see our toils being put to use and the fruits of imagery they produced. But after 30 minutes or so the novelty wore off and the rest of the procedure became an intrigue into how well the crew was “flying” the RMS and whether it would contact any of the panels. A morbid thought, perhaps, but spaceflight is like that.

Of course, the engineers responsible for the RCC Panels no doubt remained captivated throughout, and of course we all were aware of the PAL ramp foam that came off during launch, so clearing the leading edges of defect was truly an important milestone. Providentially, we saw nothing of concern that might impede the spacecraft’s reentry, and all of us blew a sigh of relief when that was confirmed.

Figure 47. Space Shuttle Discovery on STS-114, taken during the R-Bar Pitch Maneuver by astronauts aboard the International Space Station. (NASA Image ISS011-E-11227)

Figure 48. The underside of Space Shuttle Discovery on STS-114, taken during the R-Bar Pitch Maneuver by astronauts aboard the International Space Station. (NASA Image ISS011-E-11263)

But as with other great acts of drama and suspense, this mission was not yet done offering surprises. While the images taken during the Rendezvous Pitch Maneuver didn’t indicate anything of great concern, NASA was cautious about committing the crew to reentry and wanted to exonerate even the slightest blemish. One item that was noticed from the photos the ISS crew took was a “gap filler” protruding out from between two tiles on the spacecraft’s underside near the bay of the Nose Landing Gear. As mentioned in Chapter 15, gap fillers were small pieces of felt padding used to close the cavity between adjacent thermal protection tiles. While the tiles themselves were adhered to structure using epoxy, the gap fillers only relied on the compressional forces between the tiles to be held into place. Sometimes, vibration or shock could knock them loose, and they could end up protruding a few centimeters above the surface of the tiles. Protruding gap fillers had been noted on the ground on many missions after an orbiter returned, so we knew the phenomena could occur. This one on STS-114 was right beside the forward attachment point between the Orbiter and the ET—the same location where the Bipod connects the two structures together for launch. This area of RCC was called the Arrowhead due to its unique pentagonal shape, rather like the home plate on a baseball field. Once a Shuttle attained orbit, explosive charges were detonated at the connection point to separating the orbiter from the ET. It was felt that that shock had jarred one of the gap fillers loose and caused it to project above the surface of the ship’s belly.

While this may seem like a minor concern, it actually resulted in significant discussion. While Discovery’s crew was in orbit transferring equipment and supplies to the ISS, we met in the MER conference room to debate whether action needed to be taken on this errant gap filler. The concern had to do with the temperatures the protective tiles would see during reentry. Even at the extreme hypersonic velocities during the early stages of reentry, commonly at speeds of Mach 20–22, the airflow along the belly of the Orbiter is smooth and undisturbed, what aerodynamicists call laminar flow. The tiles were designed for laminar flow and could withstand the temperatures expected at those incredibly high speeds. But stick something up into that flow such as, let’s say, a protruding gap filler, and the smooth flow could be disturbed and become turbulent, which could produce significantly higher reentry temperatures.

We met in the MER’s conference room where numerous predictions were projected onto the large screen hung on the far wall. Experts debated what could be expected from the condition we had on-orbit and whether there was any significant risk of unexpectedly high temperatures burning through the tile and the underlying structure, which could be catastrophic. Remember, coming on the heels of STS-107, NASA was approaching any thought of a burn-through or a compromised structure during reentry with extreme caution. Eventually, after significant debate and discussion, our engineers simply couldn’t say with confidence that the condition was survivable to a high probability. Enough uncertainty remained in the question given the unknowns of this high-speed regime. This message was brought to the MMT who, in the end, decided to place one of Discovery’s astronauts on the tip of the OBSS, maneuver both to the underside of Discovery, and have the spacewalker attempt to remove the protruding gap filler.

Nothing like this was planned prior to launch, but the logic of it resonated with many of us. Doing this could also be a proof of concept of our ability to access areas of the orbiter that previously were unreachable, and proving concepts was a major goal of the entire STS-114 mission. Lead spacewalker Stephen Robinson was the obvious choice to participate in this 200-mile-high carnival ride. This would be the third spacewalk of the mission—the first dedicated to demonstrating tile repair techniques while on-orbit and the second focused on removing a failed Control Moment Gyro on the ISS for return to Earth.

Figure 49. Astronaut Stephen K. Robinson, STS-114 mission specialist, anchored to a foot restraint on the International Space Station’s Canadarm2 during the mission’s third EVA. (NASA Image S114-E-6642)

Figure 50. Robinson, on the end of the station’s Canadarm2 (out of frame), slowly and cautiously makes his way to the underside of Space Shuttle Discovery to remove gap fillers from between the Orbiter’s heat-shielding tiles. (NASA Image S114-E-6221)

In the end it took little effort for Steve to remove the gap filler, requiring less force than predicted. We could watch nearly the entire process unfold on Steve’s helmet camera, and although the communication was less than ideal (the Orbiter was never designed to have spacewalkers operating beneath it), it was sufficient for us to watch live as the filler was removed. At that moment, holding the tiny section of felt in Steve’s gloved hand, I pondered if he was wondering where exactly to put it. I still wonder that to this day.

Discovery landed safely back on Earth on August 9, 2005—13 days, 21 hours, 32 minutes, and 48 seconds after it launched. To me and perhaps to others, it was a pyrrhic victory. We successfully returned the Space Shuttle to flight, proving that we could get to orbit and back safely. We resupplied the Space Station and validated any number of new techniques and processes developed since the horrific end of the STS-107 mission. But we hadn’t solved the problem of foam coming off the ET during launch, and because of that it would be nearly a year before we would try again on STS-121, essentially a test flight–like repeat of STS-114, launching in July 2006.

Were we happy? Sure. Were we satisfied? Speaking for myself, not in the least. In the years I’ve spent in NASA, solving problems has been one of the hallmarks of our agency and a particularly critical reason why it’s such a fun place to work. Resigning to failure, or at least something short of full success, is anathema to our culture. Yes, sometimes we can’t change the laws of physics or alter the universal gravitational constant (two favorite quotes of mine on the implausibility of things from Star Trek), but gosh it only took eight years to move from sending an astronaut into space for the first time to landing a man on the Moon. We’ve sent emissaries to the far edges of the solar system and fleets of rovers to explore the surface of Mars (even a helicopter!). We NASA engineers and managers live for the challenge of doing what no one has ever done before. It’s in our DNA and is a source of pride.

Yet, as dissatisfying as STS-114 was for me, it felt good to again be an active player in the arena of human spaceflight. That was undeniable. We would improve (and we did), but if nothing else this mission embodied the spirit of that’s what ships are for, and for me at least, that part was very, very satisfying!

Life Lesson for Engineers, Managers, and Leaders:

Failure is a part of life, but don’t forget to celebrate good work and success.

Engineering, management, and leadership (and life in general) is full of both successes and failures. Many are difficult to predict, and some occur when least expected. But both do happen with wonderful, or appalling, frequency, depending on the outcome. When these life events take place, I can suggest recognizing that both are facts of life.

When failures take place, and especially if the failure was preventable, then, by all means, fix what would have prevented it. Change the design, alter your processes, or even the entire culture if that’s what is required to prevent a similar instance. But also recognize the Picard idiom that, “It is possible to commit no mistakes and still lose. That is not a weakness; that is life.”

In the sorts of endeavors NASA leads, we live with the possibility of failure all the time. We work and strive to reduce the likelihood that failure may rear its unwelcome and distasteful head. We add in redundancy and dictate required reliability, all in pursuit of ensuring a successful mission even if failure makes an appearance. We certify parts and systems to rigorous standards and test our systems in relevant environments in order to establish confidence that the things we build will be successful. And yet, life proves that we may still fail. If we overlooked something or, worse, ignored a potential hazard, then the failure is on us. But if we tried our best with positive intentions, worked and strived with all our effort to achieve success and still failed … well, then I revert to what Picard said.

STS-114 was like that. I don’t believe we committed any mistakes and yet we still lost the pursuit of preventing foam from breaking off the ET, at least for that mission. How we dealt with failure is just as important than how we dealt with success. The Shuttle Program’s decision to remain grounded for nearly another year to give engineers and technicians more time to perfect their solutions was the right decision. I have no doubt about that.

On the flip side, one could applaud the many successes that came out of STS-114 as well. Significant hardware changes were made and many of them proved to perform exactly as expected. Additionally, the STS-114 mission validated any number of organizational changes to NASA’s governance that added rigor to our technical and decision-making processes.

Both of these attributes were the result of literally years of dedicated work by hundreds if not thousands of engineers, managers, and leaders. Return to Flight was a herculean effort, equal in both magnitude and importance to NASA. It was the culmination of much of that work, and even if all the accomplishments during the mission didn’t meet full success, taken collectively it still remains an awesome achievement. The cheer I heard in the MER at the moment of liftoff was a recognition of all of that. And it was deserved. Or maybe more correctly, it was earned. Even in the face of failures, we can still celebrate good work and the successes we earn.

1 External Tank Tiger Team. STS-114 External Tank Tiger Team Report: Preliminary Status and Data Package (Washington, DC: NASA, 2005).

2 William Harwood, “STS-114 Mission Archive (Final)” CBS News/Kennedy Space Center, July 28, 2005. https://www.cbsnews.com/network/news/space/STS-114_Archive.html (accessed Jan. 23, 2025).

3 Ibid.

4 Ibid.

Epilogue

What We Leave Behind

Technical Authority, Formal Dissent, and Emotional Scarring That Will Last a Lifetime

One’s destination is never a place, but a new way of seeing things.

Henry Miller

It is good to have an end to journey toward; but it is the journey that matters in the end.

Ernest Hemingway

The long arc of NASA’s voyage from the unexpected tragedy of STS-107 to the union of successes and failures of STS-114 spanned a sometimes ecstatic, sometimes frustrating, but always engrossing two and a half years of tireless work. As with most voyages, this one was populated with stomach-churning lows and spirit-lifting highs, mixing both difficult challenges and celebration-worthy triumphs into what was for most of us a very personal experience. It certainly was for me.

When I reflect upon living through such times, I receive in return a suite of emotions. Foremost, I feel honored to have had the opportunity to witness all of this history from such a close vantage point. Alongside that appreciation is a glow of pride for having contributed to that history and assisted NASA in overcoming mountainous challenges, even if only in some small way. This period was filled with the exasperation of unanswered questions and the satisfaction of knowledge as the pieces of the puzzle were put into place. When it began there was confounding mystery and a litany of questions, but we found answers to many of those questions—some definitive, and others that necessitated some speculation but informed by evidence and sufficiently quantified to the point where the answer was accepted as truth (it was “good enough”).

There was exhaustion, letdown, dismay, irritation, annoyance, and weariness, all existing side-by-side with delight, encouragement, gratification, and cheer. If one were to concoct an aperitif of feelings using a recipe that prioritized variety, this period in our lives would have fulfilled that order admirably.

But for me, this entire multiyear journey all began with one horrific day when NASA—all of us—failed to protect the lives of seven brave explorers who entrusted us with that sacred responsibility. And what I felt that day largely defined the entire experience of Columbia and STS-107 to me. Of all the emotions I was to experience, the ones I remember the most were a searing grief of loss and an overwhelming disbelief in our collective failure. Those two sentiments combined to nearly incapacitate me for a few days, as I’m sure it did to others, forcing me to ignore the hurt coursing through me and, for a while, simply focus on doing my job. It was pure survival instinct, much like marathon runners do when pain intrudes their body and they have to push through it, mind conquering body, in order to finish the physical ordeal. But this was worse; a marathon runner’s body can recovery from 26 miles of pounding, but the grief and disbelief that I experienced on February 1, 2003, left a permanent scar, one that I will carry for my entire life.

If you’ve ever read J.R.R. Tolkien’s The Lord of the Rings, or even seen Peter Jackson’s Academy Award–winning movies, there is a powerful scene near the end of the book as narrated by the character Frodo Baggins. He and his Hobbit companions had spent 13 months of the road, forced away from their beloved Shire and experiencing hardship they could have barely imagined. They had overcome the personification of evil and had won, with Frodo successfully destroying the One Ring and ending a reign of terror beset upon their world. But when Frodo and his hobbit companions return to the Shire, he realized that something had changed within him. As penned by Tolkien:

How do you pick up the threads of an old life? How do you go on, when in your heart you begin to understand ... there is no going back? There are some things that time cannot mend. Some hurts that go too deep, that have taken hold.1

That was how I felt after the Columbia accident. I was irrevocably changed by the tragedy, and I knew it.

NASA as an organization was also irrevocably changed. The agency that emerged from the crucible of STS-107 was altered in some very significant and structural ways. In A Renewed Commitment to Excellence: An Assessment of the NASA Agency-wide Applicability of the Columbia Accident Investigation Board, otherwise known as the Diaz Report, Al Diaz and his team offered a series of recommendations to strengthen NASA’s organization and decision-making apparatus. Taking guidance from the CAIB report, the Diaz team recognized that during the STS-107 mission, as they put it, “The failure to convey the urgency of engineering concerns was caused, at least in part, by organizational structure and spheres of authority.” 2 In this they were recognizing that the concerns held by some within JSC Engineering that Columbia’s thermal protection shield might have been dangerously compromised had not been elevated sufficiently, and that no action was taken on the suggestion to request other national assets to image Columbia while in flight, thereby gaining crucial data on the ship’s ability to withstand reentry.

Recognizing this shortcoming, Diaz’s team sought to understand the causes that led to the shunning of these important concerns. Elaborating further, they observed that “The evidence that supports the organizational causes also led the [CAIB] to conclude that NASA’s current organization, which combines in the Shuttle Program all authority and responsibility for schedule, cost, manifest, safety, technical requirements, and waivers to technical requirements, is not an effective check-and-balance to achieve safety and mission assurance.”3

Allow me to dissect this a bit further. What the CAIB and the Diaz team were pulling on was that 100 percent of the authority within the Shuttle Program went through the program manager. That included authority to dictate schedule, to determine which payloads got manifested for launch, to set budgets, and to direct where resources were spent. It also included the authority to establish and enforce standards for safety, and to decide when and where technical specifications would be enforced and when they would not. All decisions went through the program manager. While this stove piping might seem efficient, it also opens the door to the need to trade each of the factors as commodities and to assign priorities between technical, safety, cost, and schedule concerns. For example, one apocryphal story related to me more recently by a senior agency official pertained to decisions around the time of the Columbia accident to reduce the budget for its Safety organization given other budget pressures on the program. As the Shuttle Program managed its Safety budget, the decision was final.

The Diaz report highlighted this deficiency, stating, “Organizations must be structured to provide a more effective balance between program schedule/cost management and technical review authority,” also adding, “Additional restructuring of the independence of Safety and Mission Assurance is critical.”4

Given these structural defects, the team recommended that NASA establish technical authorities independent of the programs. Specifically, they stipulated in Recommendation #23 that NASA

Establish an independent Technical Engineering Authority that is responsible for technical requirements and all waivers to them, and will build a disciplined, systematic approach to identifying, analyzing, and controlling hazards throughout the life cycle of the Shuttle System. The independent technical authority does the following as a minimum:

Develop and maintain technical standards for all Space Shuttle Program projects and elements,

Be the sole waiver-granting authority for all technical standards,

Conduct trend and risk analysis at the sub-system, system, and enterprise levels,

Own the failure mode, effects analysis and hazard reporting systems,

Conduct integrated hazard analysis,

Decide what is and is not an anomalous event,

Independently verify launch readiness, and

Approve the provisions of the recertification program called for in (CAIB).

The Technical Engineering Authority should be funded directly from NASA Headquarters and should have no connection to or responsibility for schedule or program cost.5

And in Recommendation #24:

NASA Headquarters Office of Safety and Mission Assurance should have direct line authority over the entire Space Shuttle Program safety organization and should be independently resourced.6

I’ll slow down for a moment and summarize what was being recommended. First, they were saying that some organizational separation was required between the program office and the organizations responsible for safety and technical oversight (namely, Engineering). In effect, the program office had too much authority, with decision-making over all aspects centralized with the program manager. In the rationale for the recommendation to establish an independent engineering technical authority, Diaz and his team discussed that no programs should have the ability to waive technical standards or compromise a standard without the review and approval of an appropriate engineering authority. The ability of the program office, which makes decisions on funding allocations and schedule implications, to also be the decision authority of which technical standards to abide by and which to not, was unpalatable to the Diaz team. They inferred an inherent conflict of interest in such centralized authority and felt that this concentration contributed to the conditions that led to the accident.

The also opined that safety of astronaut crews was paramount, more important than meeting manifest schedules or budget considerations. Their recommendation to have the Shuttle Safety Office report directly to safety officials at NASA HQ rather than the SSP manager would allow for considerable independence and freedom for that office to focus on safety without the complicating pressures of programmatic considerations.

And perhaps most importantly, the Diaz report recognized that funding for both capabilities—engineering and safety—should be independent to the Shuttle Program, instead coming from separate sources directly from NASA HQ. This independence, it was hoped, would extinguish the potential that schedule or cost pressures on the program could metastasize into compromises on technical integrity or safety.

These changes were consequential, structural, and quite simply a very big deal for NASA. They are perhaps obvious now in hindsight, but when enacted in 2005 they represented a fundamental rethinking of how NASA does business.

How was all of this received? Well, from my imbedded position with MOD I certainly didn’t note any significant dissent or push back from the authorities in the program office, nor from the engineering and safety officials who would now be granted not only new independence and authority but also a heightened responsibility. I surmise that the program office officials, most being former engineers, all saw a reasonable logic in these changes, and while this restructuring did reduce the universal authority of the program, absolutely no one wanted to ever repeat the calamity of the Columbia accident. From my perch I noted fairly broad saluting and little to no rebuttal.

Such structural changes to an organization rarely get implemented overnight. There is usually a period of transition when both the organization and individuals adopt, learn, and adapt. But the change was a good one and appreciably strengthened NASA’s resilience and decision-making abilities.

How did this get implemented and how has it been working? With the recommendations, NASA created three technical authorities. The first was Engineering, responsible for independent technical oversight of all programs and projects and the sole authority to tailor or waive engineering standards and specifications. The second was a Safety and Mission Assurance (S&MA) Office, responsible for independent oversight of all aspects of safety, reliability, maintainability, and quality assurance. Each of these were given independent reporting chains beginning with project chief engineers and project chief safety officers and then reporting up through their NASA center engineering or S&MA supervisors. From there technical concerns can directly reach the ears of the NASA chief engineer or the NASA chief of safety & mission assurance, both of whom reside at NASA Headquarters and who report not to program authorities but directly to the NASA associate administrator, the highest-ranking civil servant in the agency below only the politically appointed NASA administrator and deputy administrator. These separate and independent reporting chains ensure that technical and safety concerns can rise to the highest levels of the agency and avoid competition with the programmatic pressures of cost and schedule.

The third technical authority is one dedicated to Health and Medical. While smaller than the other two TAs, the Health & Medical Technical Authority is populated by medical doctors and certified physicians, with a purview of independent oversight of the health of flight crews, be they astronauts or experimental and research aircraft pilots.

All three TAs operate separately enforcing their own policies and procedures, and yet also operate collectively as cousins or siblings to each other in a common cause of independent assessment and technical oversight of NASA’s efforts.

Does it all work? Perhaps I’m biased, given my current position as chief engineer for aeronautics at NASA HQ and a link in the tech authority chain, but even conceding that point I would wager that TA has worked well for NASA, improved our resilience, and increased our rate of success not only in human spaceflight but in the robotic exploration of our planet and the solar system, experimental and research aircraft, and in developing and maturing pathfinding technologies. In practice, when a program or project is initiated, a chief engineer (CE) and chief safety officer (CSO) is identified (along with a chief medical officer, or CMO, if applicable) who are responsible for managing the project’s technical teams and maintaining ownership of the technical baselines. This is done from the residency of the engineering or safety organizations, who pay for the chief’s salaries and provide supervisory management, with the individuals then matrix into the project’s or program’s organizational structure. While they are part of the project’s or program’s leadership team, their independence allows for objective technical interpretations that are free from the pressures of schedule and cost.

Does that mean that CEs, CSOs and CMOs remain oblivious of the project’s programmatic challenges? No, it does not; most CEs, CSOs and CMOs are well aware of the complicated and interconnected challenges that face all project managers and operate in partnership with those managers in finding solutions that coincide with acceptable risk. But should serious technical concerns become evident during a project’s life, their independence provides them with the latitude to keep their focus on technical success and, if need be, elevate those concerns without fear of being quashed by other pressing programmatic constraints.

Which brings us to another critical element of this new NASA governance developed after STS-107. That is the Formal Dissent. Originally referred to as Dissenting Opinion, Formal Dissent provides the means for anyone in the agency to elevate a decision for reconsideration if the decision is deemed to be contrary to the best interests of NASA. Formal Dissent was instantiated into NASA governance to promote the concept of “voice” within the agency, where concerns over a decision would be heard for potential reconsideration, going all the way up to the NASA administrator if necessary. This injunction against the conditions that may prevent decision-makers from hearing critical information, open to anyone in the agency, has proven to be a strong testament that all viewpoints are to be heard.

Formal Dissent first requires a decision to be made. Normally this is done by a program or project manager, who makes a decision based upon counsel provided to them combined with their own best judgement. Historically, Formal Dissents have been used by those on the technical side who believe that the decision exceeds acceptable risk for safety or mission success. However, Formal Dissent is not reserved only for technical dissenters but is open to anyone who requests a decision be elevated to a higher authority for reconsideration.

Formal Dissent carries responsibilities for both the decision-maker and for the dissenter. The decision-maker needs to allow that a dissent is possible for the decision she or he made and to provide an environment that is safe for alternate perspective to be voiced. On the other hand, dissenters need to bring a tangible, defensible case driven by data as rationale for the decision to be overturned. Dissenters cannot move forward with a simple disagreement or notions that something is a bad idea—the onus is on them to back up the dissent with data.

Should a formal dissent be initiated, the process guarantees the dissenter an opportunity to be heard. But the process does not guarantee satisfaction. A decision for reconsideration on a Formal Dissent can end up overturning the original decision, or it can validate that original decision at which time no change is affected.

I’ve seen Formal Dissent implemented a few times and have been involved in a couple. In my opinion, it works. It provides avenues for alternate perspectives to be heard, whomever the advocate is, with no prejudice or malice. It is, to me, one of the best practices NASA has as an agency.

Both of these—Technical Authority and Formal Dissent—are invoked by NASA governance and policy and have become an inherent part of NASA culture. But the memory of Columbia continues to permeate through NASA even today, more than 20 years after the tragedy. Every year in late January NASA holds a Day of Remembrance. These special events at each NASA center recognize the sacrifice of astronaut crews who gave their lives in the service of the agency, namely Apollo 1, Challenger, and Columbia. With solemnity as a foundation, the Day of Remembrance acknowledges the steep price of hubris and complacency and reminds all NASA personnel that vigilance and a healthy organizational culture are our best defenses against future accidents. They spotlight the caustic effects of organizational silence—a pernicious sociological phenomena where people can be hesitant to speak up (voice dissent), whether due to peer pressure, organizational culture, perceptions of management expectations, fear of consequences of speaking up, self-diminishment, or other causes. Organizational silence has been known to have embargoed important information that may have led to better decisions being made, and the absence of which may have been a potential direct cause of undesirable consequences. NASA even has instituted mandatory training on the causes of organizational silence in efforts to eradicate it from the agency. Similarly, reviews of the conditions and decisions (or lack of decisions) that led to the Apollo 1, Challenger, and Columbia accidents have become common annual training requirements for all employees.

A lot has changed since we lost Columbia. A lot indeed.

Ralph Roe, manager of the OPO at the time of STS-107 and later the NASA chief engineer (my boss in my present position of chief engineer for aeronautics), remarked in 2021—a year before he retired—on the similarity in time between these calamitous accidents. Consider this: Three Apollo astronauts were lost in the Apollo 1 fire in 1967, seven perished on Challenger in 1986, and another seven with Columbia in 2003. The time span between the first two was 19 years, and between the second two it was 17 years. In the year 2021, Ralph cautioned that it had been 18 years since Columbia was lost and based on past history, we were due for another accident. No one wanted that to happen, of course, but Ralph didn’t bring up his point as an acknowledgement of some preordained fate but rather as a warning that vigilance was required to prevent more loss of life. Every 17 to 20 years, whether due to personnel turnover, corporate memory loss, or the simple creep of complacency, NASA experiences a major accident. As I write this in the year 2025 that clock is still ticking, but the more we advance it safely the more we might be able to entertain the notion that we’ve learned our lesson. It’s a spirited thought, and a hopeful one to me. But I acknowledge that to keep those hands moving requires constant vigilance and acknowledgement of our past mistakes. No one wants that clock to stop or to reset to zero. Everyone wants it to keep on ticking, and within that desire is a recognition that we ourselves are the ones who control that potential.

Which brings me back to why Columbia left such a consequential impact on me, both professionally and personally.

When I was hired by NASA in January 1990, into MOD and Mission Control, the cadres of flight controllers were still populated by veterans of the early space program. Such luminaries as Gene Kranz and Glynn Lunney, two of NASA’s first flight directors, still worked there. In Chapter 14 you met my first mentor, Dick “Brownie” Brown, also a veteran of our country’s first forays into the unknowns of space travel. But there were others, names of dedicated professionals that are not as well-known, such as John Temple, Jim Saultz (my branch chief) and Paul Joyce (my section head). Each of these people could trace their lineage to the predecessors of Apollo, to efforts like Projects Gemini and Mercury, or even before those to the predecessor of NASA, the National Advisory Committee or Aeronautics. While I was a young 25-year-old kid just out of graduate school, these engineers had experienced many of NASA’s triumphs and tragedies of the 1960s, from the low of the Apollo 1 fire to the high of the first Moon landing. At the time of the Apollo 1 fire, I was barely two years old, and while 20+ years later I would receive the honor of working with all of these people, the blackboards in our offices were still inscribed by Gene Kranz’s immortal charge of being Tough and Competent. But their scars predated mine by decades. Apollo 1 really wasn’t my accident.

Then came the loss of Challenger and the STS-51L crew in January 1986. In that accident, now we were at least talking about the Space Shuttle, the spacecraft I knew so well. As with my flight controller colleagues who experienced the shock of Apollo 1, many who were on console when Challenger was destroyed still actively worked in Mission Control when I hired. Bob Castle, for example, was the INCO (Instrumentation and Communications) flight controller that day. By the time I came around Bob had ascended to the role of flight director, and I worked any number of missions with Bob, many more simulations, and interacted with him weekly in meetings and discussions. Another flight controller, Tony Ceccacci, was at the PROP (Propulsion) console that cold January morning. Tony was a peer when I got certified and, interestingly, in need of a change, transferred to my own EGIL group for some years, serving side-by-side with me. Eventually, Tony also ascended to the ranks of the flight directors.

In Chapter 1, I mentioned witnessing the immediate aftermath of the Challenger explosion while an undergraduate at Embry-Riddle Aeronautical University in Daytona Beach, Florida. Seeing the contrails left by Challenger’s debris helplessly descending back to Earth made this Shuttle accident very, very real for me. It wasn’t an account I read about in a book or imagery from a documentary—I saw it with my own eyes.

And yet, even with that, there was a separation between me and that accident. I was not yet NASA, not yet a flight controller, more of a bystander than a participant. Again, Challenger wasn’t my accident.

But Columbia was different. Now, NASA was my employer and flight control was my career. Those whom fate brought together onto the STS-107 Entry Team on February 1, 2003, were each and every one friends and colleagues of mine. But beyond that connection, there were many other tendrils of sinew that connected me to the Columbia accident. The consoles they sat at, the checklists they read from, the protocols that guided their actions, were all intimately familiar to me from 11 years working Shuttle flights on console in Mission Control. The management teams, including the OPO and the MMT, were forums and groups in which I had participated. When Columbia broke apart high over central and eastern Texas, I was a cog in the Shuttle hierarchy, part of the machinery that made the entire operation work. Furthermore, the astronaut crews that flew on that spacecraft were not just caricatures who grace magazine covers or newspaper front pages, representations of heroism who might one day be portrayed by Hollywood actors. To me, astronauts were real people, with families, who joked and cried, laughed, and got angry, just like everyone else. A few of them I got to call friend, and I valued those relationships as one would with any co-worker.

In essence, Columbia was my accident. It was an event for which I could take ownership. NASA’s failures were my own failures, and I did not try to escape that unpleasant knowledge. And this sentiment wasn’t just due to proximity or immersion but derived from a loss that I took personally. As part of that machinery, if the machinery failed the STS-107 crew then I had also failed the STS-107 crew. It was as simple as that.

As I write this book—part memoir, part historical accounting, part advice for engineers, managers, and leaders—more than 20 years after the Columbia tragedy, I feel that loss and complicity as viscerally today as I did when receiving that call from Rick Fitts on the morning of the accident. I can’t remove those feelings or eliminate the memories; they are part of what makes me the person I am today, and I will carry them my entire life. But removing or eliminating such feelings isn’t the point for me. What is the point is to take those lessons and forge them into a responsibility to take action to prevent an event like that from happening again.

Columbia and STS-107 catalyzed in me a new sense of responsibility. Its aftereffects placed me on a new mission, one that I pursue in whatever positions I have attained since my flight controller days (including as NASA HQ chief engineer for aeronautics). I bring the lessons of Columbia along with me to every task I perform, every duty I take on. My radar is continuously set on high in terms of identifying signs of organizational silence in our programs and projects or any other indication that concerns are not being heard. I remain dutiful to the pursuit of avoidance of the deficiencies that contributed to the loss of Columbia and her crew, to the corrosive effects of complacency and normalization of deviance. That, for me, is Columbia’s legacy. It is the vigilance that two and a half years of disaster, recovery, investigation, and return to flight has instilled in me so that NASA, or even simply so that I, don’t have to experience such loss again.

Is NASA better for the Columbia accident? Well, I’d concede that seven lives is much too high a price to pay for any organizational improvement. So maybe a more appropriate question would be: Given that it did happen, did NASA improve itself? I think we have. I fervently believe that an independent technical authority and a process for formal dissent does make us more resilient, more fair, more equitable, allowing us to make better decisions, and be better as an agency responsible for missions of exploration and discovery accountable to our country and to the world.

I sincerely hope that future generations of NASA explorers don’t have to live through similar experiences in order to maintain that dedication to purpose and integrity. Keeping Columbia, along with Challenger and Apollo 1, at the forefront of our collective culture certainly can help in that pursuit. Never forget has become an axiom applied to many causes and it fits to the cause of NASA vigilance. If we remember the mistakes of the past, inculcate them into our culture, and ward off the influences that could stimulate their return, then I believe that NASA’s future is bright indeed.

I’ll conclude this book with one final quote, this one from Martin Luther King Jr. While he wrote these elegant and inspiring words in the context of the civil rights movement, I believe that they apply to NASA as well:

If you can’t fly, run; if you can’t run, walk; if you can’t walk, crawl; but by all means keep moving.7

1 J.R.R. Tolkien, The Lord of the Rings. (London, England: HarperCollins, 1955).

2 Al Diaz, A Renewed Commitment to Excellence: An Assessment of the NASA Agency-wide Applicability of the Columbia Accident Investigation Board Report, (Washington, DC: NASA), Jan. 30, 2004, p. 43. https://www.nasa.gov/pdf/55691main_Diaz_020204.pdf (accessed Jan. 27, 2025).

3 Ibid, p. 43.

4 Ibid, p. 15.

5 Ibid, pp. A-9–A-10.

6 Ibid, p. A-10.

7 “Spelman Messenger, Vol. 76, No. 3, May 1960.” Spelman Messenger, 1960–1969. Spelman College, May 1, 1960. Spelman College Archives. http://hdl.handle.net/20.500.12322/sc.001.messenger:1960.02

Acknowledgments

It takes a village to create a book, and this one is no different. Many people had a hand in this story’s creation and deserve recognition.

I’d like to sincerely thank James Franz, NASA intern at Glenn Research Center. James volunteered his time as part of his internship duties to assist with the laborious investigative work of finding references to the numerous photos, graphics, and images used in the book. Many of the original images I selected were from websites with unofficial sources and were inimical to publishing with the NASA History Office. James’s tireless work in locating official source images and those which met the post-production requirements of the History Office is appreciated beyond measure. To put it simply, this book couldn’t have been produced without James’s participation.

I’d also like to thank Ms. Becky Murray, the deputy engineering director at NASA’s Kennedy Space Center, for her technical peer review of the manuscript. Becky was a NASA employee at KSC at the time of STS-107 and, as do I, she carries many personal memories of those days. Becky’s technical corrections greatly added to the credibility of the information discussed in this book.

Thanks to Adam Franklin Greenstone, NASA’s counsel for ethics in the Office of the General Counsel, for your legal advice.

To the outstanding production team in the NASA Headquarters Communications Support Services Center, thank you to copyeditor Andrew Cooke, designer Ronna Goldman, printing specialist Vee Talbet, and production managers Garrett Shea and Bill Keeter.

Thank you sincerely to the four individuals who peer reviewed the manuscript. As the peer review process ensures anonymity, I don’t know who you are, but I can tell you that your inputs, considerations, perspectives, and suggestions immensely improved the contents of this book.

To the NASA History Office, my particular thanks to Stephen Garber (now retired from NASA) and James Anderson for all your assistance and advice in helping shepherd this book through the process.

About the Author

In 2025 Steven R. Hirshorn celebrated his 35th anniversary with the National Aeronautics and Space Administration (NASA). Presently he works at NASA Headquarters in Washington, DC within the Office of Chief Engineer where he serves as the chief engineer for aeronautics.

After earning a bachelor of science degree in aeronautical engineering from Embry-Riddle Aeronautics University in 1986, and a master of science degree in aerospace engineering at The University of Texas–Austin in 1989, Hirshorn began his NASA career at the Johnson Space Center in Houston, joining the ranks of Space Shuttle flight controllers in the eponymous Mission Control. Serving as an EGIL, he supported 55 Space Shuttle missions on console over the next 11 years, including 10 launches and landings, on some of the Space Shuttle Program’s most historic missions. Working side-by-side with many of the Mission Control veterans of the Apollo Moon landing missions who were approaching retirement, he was able to gain from the experience and wisdom of these space-age greats.

In the 2000s, Hirshorn served first as the Mission Operations representative to the Space Shuttle Orbiter project (as recounted in this book) and later as the Mission Operations lead engineer for the Constellation Program (the predecessor to today’s Artemis Program). At the end of both the Shuttle and Constellation Programs in 2011, he moved to NASA HQ, initially serving in one of NASA’s aeronautics research programs. In 2013, he transferred to the Office of Chief Engineer, where he still resides, as the chief engineer for aeronautics.

Most recently, in 2020, Hirshorn served a detail in NASA’s Science Mission Directorate as the assistant deputy associate administrator for programs, overseeing NASA’s portfolio of nearly 100 science missions both in development and operations.

Hirshorn resides in Mt. Airy, Maryland, with his wife Robin, their two cats (Eowyn and Freya), two koi ponds, and an acre of forested bliss. In addition to his NASA and authoring pursuits, he enjoys mountaineering, international travel, and shares a passion for all things Star Trek (which offers so many life lessons).

Acronym List

AC Alternating Current

AMOS Air Force Maui Optical & Supercomputing

APU auxiliary power unit

ATC Air Traffic Control

CAIB Columbia Accident Investigation Board

CAP Civilian Air Patrol

CE chief engineer

CoFR Certificate of Flight Readiness

CMO chief medical officer

CSO chief safety officer

DC Direct Current

DOD Department of Defense

DPS Data Processing System

DSC Dedicated Signal Conditioner

EI Entry Interface

EO Engineering Order

EPA Environmental Protection Agency

ET external tank

EVA Extra Vehicular Activity

FAO Flight Activities Officer

FCR Flight Control Room

FEMA Federal Emergency Management Agency

FOD Flight Operations Directorate

FOI Freedom of Information

GLS Ground Launch Sequencer

HQ Headquarters

ICP Incident Command Post

IFA In-Flight Anomaly

IFM In-Flight Maintenance

IMU Inertial Measurement Unit

JSC Johnson Space Center

KSC Kennedy Space Center

LCC Launch Control Center

LMLG Left Main Landing Gear

LOS Loss Of Signal

MADS Modular Auxiliary Data System

MCC Mission Control Center

MDM Multiplexer/Demultiplexer

MER Mission Evaluation Room

MET Mission Elapsed Time

MILA Merritt Island Launch Annex

MIT Mishap Investigation Team

MLP Mobile Launch Platform

MMOD micrometeoroid and orbital debris

MMT Mission Management Team

MOCR Mission Operations Control Room

MOD Mission Operations Directorate

MRT Mishap Response Team

MSL Microgravity Science Laboratory

NAIT NASA Accident Investigation Team

NDE Nondestructive Evaluation

NDI Nondestructive Inspection

NOAA National Oceanographic and Atmospheric Administration

NORAD North American Aerospace Defense Command

NTSB National Transportation Safety Board

OBSS Orbiter Boom Sensor System

OEX Orbiter Experiment

OMS Orbital Maneuvering System

OPO Orbiter Project Office

OTJ on-the-job

PAL protuberance air load

PAO Public Affairs Officer

PRA Probabilistic Risk Assessments

psi per square inch

RCC Reenforced Carbon-Carbon

RCS Reaction Control System

RF radio frequency

RMLG Right Main Landing Gear

RMS Remote Manipulator System

RTF Return to Flight

SCA Sim Control Area

SLA Super Lightweight Ablator

SLEP Shuttle Service Life Extension Project
SLF Shuttle Landing Facility

SMG Spaceflight Meteorology Group

SOFI Spray-On Foam Insulation

SRB solid rocket booster

SSP Space Shuttle Program

SwRI Southwest Research Institute

TA technical assistant

TCDT Terminal Countdown Demonstration Test

TDRS Tracking and Data Relay Satellite

TPS Thermal Protection System

TSS Tethered Satellite System

TUFI Toughened Unipiece Fibrous Insulation

USA United Space Alliance

USAF United States Air Force

USGS United States Geological Survey

USSPACECOM United States Space Command

VAB Vehicle Assembly Building

VEWG Vehicle Engineering Working Group

WLE wing leading edge

The NASA History Series

Reference Works, NASA SP-4000

Grimwood, James M. Project Mercury: A Chronology. NASA SP-4001, 1963.

Grimwood, James M., and Barton C. Hacker, with Peter J. Vorzimmer. Project Gemini Technology and Operations: A Chronology. NASA SP-4002, 1969.

Link, Mae Mills. Space Medicine in Project Mercury. NASA SP-4003, 1965.

Astronautics and Aeronautics, 1963: Chronology of Science, Technology, and Policy. NASA SP-4004, 1964.

Astronautics and Aeronautics, 1964: Chronology of Science, Technology, and Policy. NASA SP-4005, 1965.

Astronautics and Aeronautics, 1965: Chronology of Science, Technology, and Policy. NASA SP-4006, 1966.

Astronautics and Aeronautics, 1966: Chronology of Science, Technology, and Policy. NASA SP-4007, 1967.

Astronautics and Aeronautics, 1967: Chronology of Science, Technology, and Policy. NASA SP-4008, 1968.

Ertel, Ivan D., and Mary Louise Morse. The Apollo Spacecraft: A Chronology, Volume I, Through November 7, 1962. NASA SP-4009, 1969.

Morse, Mary Louise, and Jean Kernahan Bays. The Apollo Spacecraft: A Chronology, Volume II, November 8, 1962–September 30, 1964. NASA SP-4009, 1973.

Brooks, Courtney G., and Ivan D. Ertel. The Apollo Spacecraft: A Chronology, Volume III, October 1, 1964–January 20, 1966. NASA SP-4009, 1976.

Ertel, Ivan D., and Roland W. Newkirk, with Courtney G. Brooks. The Apollo Spacecraft: A Chronology, Volume IV, January 21, 1966–July 13, 1974. NASA SP-4009, 1978.

Astronautics and Aeronautics, 1968: Chronology of Science, Technology, and Policy. NASA SP-4010, 1969.

Newkirk, Roland W., and Ivan D. Ertel, with Courtney G. Brooks. Skylab: A Chronology. NASA SP-4011, 1977.

Van Nimmen, Jane, and Leonard C. Bruno, with Robert L. Rosholt. NASA Historical Data Book, Volume I: NASA Resources, 1958–1968. NASA SP-4012, 1976; rep. ed. 1988.

Ezell, Linda Neuman. NASA Historical Data Book, Volume II: Programs and Projects, 1958–1968. NASA SP-4012, 1988.

Ezell, Linda Neuman. NASA Historical Data Book, Volume III: Programs and Projects, 1969–1978. NASA SP-4012, 1988.

Gawdiak, Ihor, with Helen Fedor. NASA Historical Data Book, Volume IV: NASA Resources, 1969–1978. NASA SP-4012, 1994.

Rumerman, Judy A. NASA Historical Data Book, Volume V: NASA Launch Systems, Space Transportation, Human Spaceflight, and Space Science, 1979–1988. NASA SP-4012, 1999.

Rumerman, Judy A. NASA Historical Data Book, Volume VI: NASA Space Applications, Aeronautics and Space Research and Technology, Tracking and Data Acquisition/Support Operations, Commercial Programs, and Resources, 1979–1988. NASA SP-4012, 2000.

Rumerman, Judy A. NASA Historical Data Book, Volume VII: NASA Launch Systems, Space Transportation, Human Spaceflight, and Space Science, 1989–1998. NASA SP-2009-4012, 2009.

Rumerman, Judy A. NASA Historical Data Book, Volume VIII: NASA Earth Science and Space Applications, Aeronautics, Technology, and Exploration, Tracking and Data Acquisition/Space Operations, Facilities and Resources, 1989–1998. NASA SP-2012-4012, 2012.

No SP-4013.

Astronautics and Aeronautics, 1969: Chronology of Science, Technology, and Policy. NASA SP-4014, 1970.

Astronautics and Aeronautics, 1970: Chronology of Science, Technology, and Policy. NASA SP-4015, 1972.

Astronautics and Aeronautics, 1971: Chronology of Science, Technology, and Policy. NASA SP-4016, 1972.

Astronautics and Aeronautics, 1972: Chronology of Science, Technology, and Policy. NASA SP-4017, 1974.

Astronautics and Aeronautics, 1973: Chronology of Science, Technology, and Policy. NASA SP-4018, 1975.

Astronautics and Aeronautics, 1974: Chronology of Science, Technology, and Policy. NASA SP-4019, 1977.

Astronautics and Aeronautics, 1975: Chronology of Science, Technology, and Policy. NASA SP-4020, 1979.

Astronautics and Aeronautics, 1976: Chronology of Science, Technology, and Policy. NASA SP-4021, 1984.

Astronautics and Aeronautics, 1977: Chronology of Science, Technology, and Policy. NASA SP-4022, 1986.

Astronautics and Aeronautics, 1978: Chronology of Science, Technology, and Policy. NASA SP-4023, 1986.

Astronautics and Aeronautics, 1979–1984: Chronology of Science, Technology, and Policy. NASA SP-4024, 1990.

Astronautics and Aeronautics, 1985: Chronology of Science, Technology, and Policy. NASA SP-4025, 1988.

Noordung, Hermann. The Problem of Space Travel: The Rocket Motor. Edited by Ernst Stuhlinger and J. D. Hunley, with Jennifer Garland. NASA SP-4026, 1995.

Gawdiak, Ihor Y., Ramon J. Miro, and Sam Stueland. Astronautics and Aeronautics, 1986–1990: A Chronology. NASA SP-4027, 1997.

Gawdiak, Ihor Y., and Charles Shetland. Astronautics and Aeronautics, 1991–1995: A Chronology. NASA SP-2000-4028, 2000.

Orloff, Richard W. Apollo by the Numbers: A Statistical Reference. NASA SP-2000-4029, 2000.

Lewis, Marieke, and Ryan Swanson. Astronautics and Aeronautics: A Chronology, 1996–2000. NASA SP-2009-4030, 2009.

Ivey, William Noel, and Marieke Lewis. Astronautics and Aeronautics: A Chronology, 2001–2005. NASA SP-2010-4031, 2010.

Buchalter, Alice R., and William Noel Ivey. Astronautics and Aeronautics: A Chronology, 2006. NASA SP-2011-4032, 2010.

Lewis, Marieke. Astronautics and Aeronautics: A Chronology, 2007. NASA SP-2011-4033, 2011.

Lewis, Marieke. Astronautics and Aeronautics: A Chronology, 2008. NASA SP-2012-4034, 2012.

Lewis, Marieke. Astronautics and Aeronautics: A Chronology, 2009. NASA SP-2012-4035, 2012.

No SP-4036.

Flattery, Meaghan. Astronautics and Aeronautics: A Chronology, 2010. NASA SP-2013-4037, 2014.

No SP-4038, 4039, or 4040.

Siddiqi, Asif A. Beyond Earth: A Chronicle of Deep Space Exploration, 1958–2016. NASA SP-2018-4041, 2018.

Management Histories, NASA SP-4100

Rosholt, Robert L. An Administrative History of NASA, 1958–1963. NASA SP-4101, 1966.

Levine, Arnold S. Managing NASA in the Apollo Era. NASA SP-4102, 1982.

Roland, Alex. Model Research: The National Advisory Committee for Aeronautics, 1915–1958. NASA SP-4103, 1985.

Fries, Sylvia D. NASA Engineers and the Age of Apollo. NASA SP-4104, 1992.

Glennan, T. Keith. The Birth of NASA: The Diary of T. Keith Glennan. Edited by J. D. Hunley. NASA SP-4105, 1993.

Seamans, Robert C. Aiming at Targets: The Autobiography of Robert C. Seamans. NASA SP-4106, 1996.

Garber, Stephen J., ed. Looking Backward, Looking Forward: Forty Years of Human Spaceflight Symposium. NASA SP-2002-4107, 2002.

Mallick, Donald L., with Peter W. Merlin. The Smell of Kerosene: A Test Pilot’s Odyssey. NASA SP-4108, 2003.

Iliff, Kenneth W., and Curtis L. Peebles. From Runway to Orbit: Reflections of a NASA Engineer. NASA SP-2004-4109, 2004.

Chertok, Boris. Rockets and People, Volume I. NASA SP-2005-4110, 2005.

Chertok, Boris. Rockets and People: Creating a Rocket Industry, Volume II. NASA SP-2006-4110, 2006.

Chertok, Boris. Rockets and People: Hot Days of the Cold War, Volume III. NASA SP-2009-4110, 2009.

Chertok, Boris. Rockets and People: The Moon Race, Volume IV. NASA SP-2011-4110, 2011.

Laufer, Alexander, Todd Post, and Edward Hoffman. Shared Voyage: Learning and Unlearning from Remarkable Projects. NASA SP-2005-4111, 2005.

Dawson, Virginia P., and Mark D. Bowles. Realizing the Dream of Flight: Biographical Essays in Honor of the Centennial of Flight, 1903–2003. NASA SP-2005-4112, 2005.

Mudgway, Douglas J. William H. Pickering: America’s Deep Space Pioneer. NASA SP-2008-4113, 2008.

Wright, Rebecca, Sandra Johnson, and Steven J. Dick. NASA at 50: Interviews with NASA’s Senior Leadership. NASA SP-2012-4114, 2012.

Hirshorn, Steven R. Ascension: Life Lessons from the Space Shuttle Columbia Tragedy for Engineers, Managers, and Leaders. NASA SP-2025-4115, 2025.

Project Histories, NASA SP-4200

Swenson, Loyd S., Jr., James M. Grimwood, and Charles C. Alexander. This New Ocean: A History of Project Mercury. NASA SP-4201, 1966; rep. ed. 1999.

Green, Constance McLaughlin, and Milton Lomask. Vanguard: A History. NASA SP-4202, 1970; rep. ed. Smithsonian Institution Press, 1971.

Hacker, Barton C., and James M. Grimwood. On the Shoulders of Titans: A History of Project Gemini. NASA SP-4203, 1977; rep. ed. 2002.

Benson, Charles D., and William Barnaby Faherty. Moonport: A History of Apollo Launch Facilities and Operations. NASA SP-4204, 1978.

Brooks, Courtney G., James M. Grimwood, and Loyd S. Swenson, Jr. Chariots for Apollo: A History of Manned Lunar Spacecraft. NASA SP-4205, 1979.

Bilstein, Roger E. Stages to Saturn: A Technological History of the Apollo/Saturn Launch Vehicles. NASA SP-4206, 1980 and 1996.

No SP-4207.

Compton, W. David, and Charles D. Benson. Living and Working in Space: A History of Skylab. NASA SP-4208, 1983.

Ezell, Edward Clinton, and Linda Neuman Ezell. The Partnership: A History of the Apollo-Soyuz Test Project. NASA SP-4209, 1978.

Hall, R. Cargill. Lunar Impact: A History of Project Ranger. NASA SP-4210, 1977.

Newell, Homer E. Beyond the Atmosphere: Early Years of Space Science. NASA SP-4211, 1980.

Ezell, Edward Clinton, and Linda Neuman Ezell. On Mars: Exploration of the Red Planet, 1958–1978. NASA SP-4212, 1984.

Pitts, John A. The Human Factor: Biomedicine in the Manned Space Program to 1980. NASA SP-4213, 1985.

Compton, W. David. Where No Man Has Gone Before: A History of Apollo Lunar Exploration Missions. NASA SP-4214, 1989.

Naugle, John E. First Among Equals: The Selection of NASA Space Science Experiments. NASA SP-4215, 1991.

Wallace, Lane E. Airborne Trailblazer: Two Decades with NASA Langley’s 737 Flying Laboratory. NASA SP-4216, 1994.

Butrica, Andrew J., ed. Beyond the Ionosphere: Fifty Years of Satellite Communications. NASA SP-4217, 1997.

Butrica, Andrew J. To See the Unseen: A History of Planetary Radar Astronomy. NASA SP-4218, 1996.

Mack, Pamela E., ed. From Engineering Science to Big Science: The NACA and NASA Collier Trophy Research Project Winners. NASA SP-4219, 1998.

Reed, R. Dale. Wingless Flight: The Lifting Body Story. NASA SP-4220, 1998.

Heppenheimer, T. A. The Space Shuttle Decision: NASA’s Search for a Reusable Space Vehicle. NASA SP-4221, 1999.

Hunley, J. D., ed. Toward Mach 2: The Douglas D-558 Program. NASA SP-4222, 1999.

Swanson, Glen E., ed. “Before This Decade Is Out…” Personal Reflections on the Apollo Program. NASA SP-4223, 1999.

Tomayko, James E. Computers Take Flight: A History of NASA’s Pioneering Digital Fly-By-Wire Project. NASA SP-4224, 2000.

Morgan, Clay. Shuttle-Mir: The United States and Russia Share History’s Highest Stage. NASA SP-2001-4225, 2001.

Leary, William M. “We Freeze to Please”: A History of NASA’s Icing Research Tunnel and the Quest for Safety. NASA SP-2002-4226, 2002.

Mudgway, Douglas J. Uplink-Downlink: A History of the Deep Space Network, 1957–1997. NASA SP-2001-4227, 2001.

No SP-4228 or 4229.

Dawson, Virginia P., and Mark D. Bowles. Taming Liquid Hydrogen: The Centaur Upper Stage Rocket, 1958–2002. NASA SP-2004-4230, 2004.

Meltzer, Michael. Mission to Jupiter: A History of the Galileo Project. NASA SP-2007-4231, 2007.

Heppenheimer, T. A. Facing the Heat Barrier: A History of Hypersonics. NASA SP-2007-4232, 2007.

Tsiao, Sunny. “Read You Loud and Clear!” The Story of NASA’s Spaceflight Tracking and Data Network. NASA SP-2007-4233, 2007.

Meltzer, Michael. When Biospheres Collide: A History of NASA’s Planetary Protection Programs. NASA SP-2011-4234, 2011.

Conway, Erik M., Donald K. Yeomans, and Meg Rosenburg. A History of Near-Earth Objects Research. NASA SP-2022-4235, 2022.

No SP-4236.

Gainor, Christopher. Not Yet Imagined: A Study of Hubble Space Telescope Operations. NASA SP-2020-4237, 2020.

Niebur, Susan M., with David W. Brown (Editor). NASA’s Discovery Program: The First Twenty Years of Competitive Planetary Exploration. NASA SP-2023-4238, 2023.

Center Histories, NASA SP-4300

Rosenthal, Alfred. Venture into Space: Early Years of Goddard Space Flight Center. NASA SP-4301, 1985.

Hartman, Edwin P. Adventures in Research: A History of Ames Research Center, 1940–1965. NASA SP-4302, 1970.

Hallion, Richard P. On the Frontier: Flight Research at Dryden, 1946–1981. NASA SP-4303, 1984.

Muenger, Elizabeth A. Searching the Horizon: A History of Ames Research Center, 1940–1976. NASA SP-4304, 1985.

Hansen, James R. Engineer in Charge: A History of the Langley Aeronautical Laboratory, 1917–1958. NASA SP-4305, 1987.

Dawson, Virginia P. Engines and Innovation: Lewis Laboratory and American Propulsion Technology. NASA SP-4306, 1991.

Dethloff, Henry C. “Suddenly Tomorrow Came…”: A History of the Johnson Space Center, 1957–1990. NASA SP-4307, 1993.

Hansen, James R. Spaceflight Revolution: NASA Langley Research Center from Sputnik to Apollo. NASA SP-4308, 1995.

Wallace, Lane E. Flights of Discovery: An Illustrated History of the Dryden Flight Research Center. NASA SP-4309, 1996.

Herring, Mack R. Way Station to Space: A History of the John C. Stennis Space Center. NASA SP-4310, 1997.

Wallace, Harold D., Jr. Wallops Station and the Creation of an American Space Program. NASA SP-4311, 1997.

Wallace, Lane E. Dreams, Hopes, Realities. NASA’s Goddard Space Flight Center: The First Forty Years. NASA SP-4312, 1999.

Dunar, Andrew J., and Stephen P. Waring. Power to Explore: A History of Marshall Space Flight Center, 1960–1990. NASA SP-4313, 1999.

Bugos, Glenn E. Atmosphere of Freedom: Sixty Years at the NASA Ames Research Center. NASA SP-2000-4314, 2000.

Bugos, Glenn E. Atmosphere of Freedom: Seventy Years at the NASA Ames Research Center. NASA SP-2010-4314, 2010. Revised version of NASA SP-2000-4314.

Bugos, Glenn E. Atmosphere of Freedom: Seventy Five Years at the NASA Ames Research Center. NASA SP-2014-4314, 2014. Revised version of NASA SP-2000-4314.

No SP-4315.

Schultz, James. Crafting Flight: Aircraft Pioneers and the Contributions of the Men and Women of NASA Langley Research Center. NASA SP-2003-4316, 2003.

Bowles, Mark D. Science in Flux: NASA’s Nuclear Program at Plum Brook Station, 1955–2005. NASA SP-2006-4317, 2006.

Wallace, Lane E. Flights of Discovery: An Illustrated History of the Dryden Flight Research Center. NASA SP-2007-4318, 2007. Revised version of NASA SP-4309.

Arrighi, Robert S. Revolutionary Atmosphere: The Story of the Altitude Wind Tunnel and the Space Power Chambers. NASA SP-2010-4319, 2010.

Lee, J. Lawrence with Robert S. Arrighi. Wind Tunnels of the NACA and NASA. NASA SP-2025-4320, 2025.

No SP-4321.

Gelzer, Christian, ed. NASA Armstrong Flight Research Center’s Contributions to the Space Shuttle Program. NASA SP-2020-4322, 2022.

General Histories, NASA SP-4400

Corliss, William R. NASA Sounding Rockets, 1958–1968: A Historical Summary. NASA SP-4401, 1971.

Wells, Helen T., Susan H. Whiteley, and Carrie Karegeannes. Origins of NASA Names. NASA SP-4402, 1976.

Anderson, Frank W., Jr. Orders of Magnitude: A History of NACA and NASA, 1915–1980. NASA SP-4403, 1981.

Sloop, John L. Liquid Hydrogen as a Propulsion Fuel, 1945–1959. NASA SP-4404, 1978.

Roland, Alex. A Spacefaring People: Perspectives on Early Spaceflight. NASA SP-4405, 1985.

Bilstein, Roger E. Orders of Magnitude: A History of the NACA and NASA, 1915–1990. NASA SP-4406, 1989.

Logsdon, John M., ed., with Linda J. Lear, Jannelle Warren Findley, Ray A. Williamson, and Dwayne A. Day. Exploring the Unknown: Selected Documents in the History of the U.S. Civil Space Program, Volume I: Organizing for Exploration. NASA SP-4407, 1995.

Logsdon, John M., ed., with Dwayne A. Day and Roger D. Launius. Exploring the Unknown: Selected Documents in the History of the U.S. Civil Space Program, Volume II: External Relationships. NASA SP-4407, 1996.

Logsdon, John M., ed., with Roger D. Launius, David H. Onkst, and Stephen J. Garber. Exploring the Unknown: Selected Documents in the History of the U.S. Civil Space Program, Volume III: Using Space. NASA SP-4407, 1998.

Logsdon, John M., ed., with Ray A. Williamson, Roger D. Launius, Russell J. Acker, Stephen J. Garber, and Jonathan L. Friedman. Exploring the Unknown: Selected Documents in the History of the U.S. Civil Space Program, Volume IV: Accessing Space. NASA SP-4407, 1999.

Logsdon, John M., ed., with Amy Paige Snyder, Roger D. Launius, Stephen J. Garber, and Regan Anne Newport. Exploring the Unknown: Selected Documents in the History of the U.S. Civil Space Program, Volume V: Exploring the Cosmos. NASA SP-2001-4407, 2001.

Logsdon, John M., ed., with Stephen J. Garber, Roger D. Launius, and Ray A. Williamson. Exploring the Unknown: Selected Documents in the History of the U.S. Civil Space Program, Volume VI: Space and Earth Science. NASA SP-2004-4407, 2004.

Logsdon, John M., ed., with Roger D. Launius. Exploring the Unknown: Selected Documents in the History of the U.S. Civil Space Program, Volume VII: Human Spaceflight: Projects Mercury, Gemini, and Apollo. NASA SP-2008-4407, 2008.

Siddiqi, Asif A., Challenge to Apollo: The Soviet Union and the Space Race, 1945–1974. NASA SP-2000-4408, 2000.

Hansen, James R., ed. The Wind and Beyond: Journey into the History of Aerodynamics in America, Volume 1: The Ascent of the Airplane. NASA SP-2003-4409, 2003.

Hansen, James R., ed. The Wind and Beyond: Journey into the History of Aerodynamics in America, Volume 2: Reinventing the Airplane. NASA SP-2007-4409, 2007.

Hansen, James R., and Jeremy R. Kinney eds. The Wind and Beyond: Journey into the History of Aerodynamics in America, Volume 3: Other Paths, Other Flyways. NASA SP-2007-4409, 2021.

Hogan, Thor. Mars Wars: The Rise and Fall of the Space Exploration Initiative. NASA SP-2007-4410, 2007.

Vakoch, Douglas A., ed. Psychology of Space Exploration: Contemporary Research in Historical Perspective. NASA SP-2011-4411, 2011.

Ferguson, Robert G. NASA’s First A: Aeronautics from 1958 to 2008. NASA SP-2012-4412, 2013.

Vakoch, Douglas A., ed. Archaeology, Anthropology, and Interstellar Communication. NASA SP-2013-4413, 2014.

No SP-4414.

Asner, Glen R., and Stephen J. Garber. Origins of 21st-Century Space Travel: A History of NASA’s Decadal Planning Team and the Vision for Space Exploration, 1999–2004. NASA SP-2019-4415, 2019.

No SP-4416 or 4417.

Spencer, Alex M, ed. A Wartime Necessity: The National Advisory Committee for Aeronautics (NACA) and Other National Aeronautical Research Organizations’ Efforts at Innovation During World War II. NASA SP-2024-4418, 2024.

Launius, Roger D. NACA to NASA to Now: The Frontiers of Air and Space in the American Century. NASA SP-2022-4419, 2022.

Monographs in Aerospace History, NASA SP-4500

Launius, Roger D., and Aaron K. Gillette, comps. Toward a History of the Space Shuttle: An Annotated Bibliography. Monographs in Aerospace History, No. 1, 1992.

Launius, Roger D., and J. D. Hunley, comps. An Annotated Bibliography of the Apollo Program. Monographs in Aerospace History, No. 2, 1994.

Launius, Roger D. Apollo: A Retrospective Analysis. Monographs in Aerospace History, No. 3, 1994.

Hansen, James R. Enchanted Rendezvous: John C. Houbolt and the Genesis of the Lunar-Orbit Rendezvous Concept. Monographs in Aerospace History, No. 4, 1995.

Gorn, Michael H. Hugh L. Dryden’s Career in Aviation and Space. Monographs in Aerospace History, No. 5, 1996.

Powers, Sheryll Goecke. Women in Flight Research at NASA Dryden Flight Research Center from 1946 to 1995. Monographs in Aerospace History, No. 6, 1997.

Portree, David S. F., and Robert C. Trevino. Walking to Olympus: An EVA Chronology. Monographs in Aerospace History, No. 7, 1997.

Logsdon, John M., moderator. Legislative Origins of the National Aeronautics and Space Act of 1958: Proceedings of an Oral History Workshop. Monographs in Aerospace History, No. 8, 1998.

Rumerman, Judy A., comp. U.S. Human Spaceflight: A Record of Achievement, 1961–1998. Monographs in Aerospace History, No. 9, 1998.

Portree, David S. F. NASA’s Origins and the Dawn of the Space Age. Monographs in Aerospace History, No. 10, 1998.

Logsdon, John M. Together in Orbit: The Origins of International Cooperation in the Space Station. Monographs in Aerospace History, No. 11, 1998.

Phillips, W. Hewitt. Journey in Aeronautical Research: A Career at NASA Langley Research Center. Monographs in Aerospace History, No. 12, 1998.

Braslow, Albert L. A History of Suction-Type Laminar-Flow Control with Emphasis on Flight Research. Monographs in Aerospace History, No. 13, 1999.

Logsdon, John M., moderator. Managing the Moon Program: Lessons Learned from Apollo. Monographs in Aerospace History, No. 14, 1999.

Perminov, V. G. The Difficult Road to Mars: A Brief History of Mars Exploration in the Soviet Union. Monographs in Aerospace History, No. 15, 1999.

Tucker, Tom. Touchdown: The Development of Propulsion Controlled Aircraft at NASA Dryden. Monographs in Aerospace History, No. 16, 1999.

Maisel, Martin, Demo J. Giulanetti, and Daniel C. Dugan. The History of the XV-15 Tilt Rotor Research Aircraft: From Concept to Flight. Monographs in Aerospace History, No. 17, 2000. NASA SP-2000-4517.

Jenkins, Dennis R. Hypersonics Before the Shuttle: A Concise History of the X-15 Research Airplane. Monographs in Aerospace History, No. 18, 2000. NASA SP-2000-4518.

Chambers, Joseph R. Partners in Freedom: Contributions of the Langley Research Center to U.S. Military Aircraft of the 1990s. Monographs in Aerospace History, No. 19, 2000. NASA SP-2000-4519.

Waltman, Gene L. Black Magic and Gremlins: Analog Flight Simulations at NASA’s Flight Research Center. Monographs in Aerospace History, No. 20, 2000. NASA SP-2000-4520.

Portree, David S. F. Humans to Mars: Fifty Years of Mission Planning, 1950–2000. Monographs in Aerospace History, No. 21, 2001. NASA SP-2001-4521.

Thompson, Milton O., with J. D. Hunley. Flight Research: Problems Encountered and What They Should Teach Us. Monographs in Aerospace History, No. 22, 2001. NASA SP-2001-4522.

Tucker, Tom. The Eclipse Project. Monographs in Aerospace History, No. 23, 2001. NASA SP-2001-4523.

Siddiqi, Asif A. Deep Space Chronicle: A Chronology of Deep Space and Planetary Probes, 1958–2000. Monographs in Aerospace History, No. 24, 2002. NASA SP-2002-4524.

Merlin, Peter W. Mach 3+: NASA/USAF YF-12 Flight Research, 1969–1979. Monographs in Aerospace History, No. 25, 2001. NASA SP-2001-4525.

Anderson, Seth B. Memoirs of an Aeronautical Engineer: Flight Tests at Ames Research Center: 1940–1970. Monographs in Aerospace History, No. 26, 2002. NASA SP-2002-4526.

Renstrom, Arthur G. Wilbur and Orville Wright: A Bibliography Commemorating the One-Hundredth Anniversary of the First Powered Flight on December 17, 1903. Monographs in Aerospace History, No. 27, 2002. NASA SP-2002-4527.

No monograph 28.

Chambers, Joseph R. Concept to Reality: Contributions of the NASA Langley Research Center to U.S. Civil Aircraft of the 1990s. Monographs in Aerospace History, No. 29, 2003. NASA SP-2003-4529.

Peebles, Curtis, ed. The Spoken Word: Recollections of Dryden History, The Early Years. Monographs in Aerospace History, No. 30, 2003. NASA SP-2003-4530.

Jenkins, Dennis R., Tony Landis, and Jay Miller. American X-Vehicles: An Inventory — X-1 to X-50. Monographs in Aerospace History, No. 31, 2003. NASA SP-2003-4531.

Renstrom, Arthur G. Wilbur and Orville Wright: A Chronology Commemorating the One-Hundredth Anniversary of the First Powered Flight on December 17, 1903. Monographs in Aerospace History, No. 32, 2003. NASA SP-2003-4532.

Bowles, Mark D., and Robert S. Arrighi. NASA’s Nuclear Frontier: The Plum Brook Research Reactor. Monographs in Aerospace History, No. 33, 2004. NASA SP-2004-4533.

Wallace, Lane, and Christian Gelzer. Nose Up: High Angle-of-Attack and Thrust Vectoring Research at NASA Dryden, 1979–2001. Monographs in Aerospace History, No. 34, 2009. NASA SP-2009-4534.

Matranga, Gene J., C. Wayne Ottinger, Calvin R. Jarvis, and D. Christian Gelzer. Unconventional, Contrary, and Ugly: The Lunar Landing Research Vehicle. Monographs in Aerospace History, No. 35, 2006. NASA SP-2004-4535.

McCurdy, Howard E. Low-Cost Innovation in Spaceflight: The History of the Near Earth Asteroid Rendezvous (NEAR) Mission. Monographs in Aerospace History, No. 36, 2005. NASA SP-2005-4536.

Seamans, Robert C., Jr. Project Apollo: The Tough Decisions. Monographs in Aerospace History, No. 37, 2005. NASA SP-2005-4537.

Lambright, W. Henry. NASA and the Environment: The Case of Ozone Depletion. Monographs in Aerospace History, No. 38, 2005. NASA SP-2005-4538.

Chambers, Joseph R. Innovation in Flight: Research of the NASA Langley Research Center on Revolutionary Advanced Concepts for Aeronautics. Monographs in Aerospace History, No. 39, 2005. NASA SP-2005-4539.

Phillips, W. Hewitt. Journey into Space Research: Continuation of a Career at NASA Langley Research Center. Monographs in Aerospace History, No. 40, 2005. NASA SP-2005-4540.

Rumerman, Judy A., Chris Gamble, and Gabriel Okolski, comps. U.S. Human Spaceflight: A Record of Achievement, 1961–2006. Monographs in Aerospace History, No. 41, 2007. NASA SP-2007-4541.

Peebles, Curtis. The Spoken Word II: Recollections of Dryden History Beyond the Sky. Monographs in Aerospace History, No. 42, 2011. NASA SP-2011-4542.

Dick, Steven J., Stephen J. Garber, and Jane H. Odom. Research in NASA History. Monographs in Aerospace History, No. 43, 2009. NASA SP-2009-4543.

Merlin, Peter W. Ikhana: Unmanned Aircraft System Western States Fire Missions. Monographs in Aerospace History, No. 44, 2009. NASA SP-2009-4544.

Fisher, Steven C., and Shamim A. Rahman. Remembering the Giants: Apollo Rocket Propulsion Development. Monographs in Aerospace History, No. 45, 2009. NASA SP-2009-4545.

Gelzer, Christian. Fairing Well: From Shoebox to Bat Truck and Beyond, Aerodynamic Truck Research at NASA’s Dryden Flight Research Center. Monographs in Aerospace History, No. 46, 2011. NASA SP-2011-4546.

Renee M. Rottner. Making the Invisible Visible: A History of the Spitzer Infrared Telescope Facility (1971–2003). Monographs in Aerospace History, No. 47, 2017. NASA SP-2017-4547.

Arrighi, Robert. Pursuit of Power: NASA’s Propulsion Systems Laboratory No. 1 and 2. Monographs in Aerospace History, No. 48, 2012. NASA SP-2012-4548.

Goodrich, Malinda K., Alice R. Buchalter, and Patrick M. Miller, comps. Toward a History of the Space Shuttle: An Annotated Bibliography, Part 2 (1992–2011). Monographs in Aerospace History, No. 49, 2012. NASA SP-2012-4549.

Ta, Julie B., and Robert C. Treviño. Walking to Olympus: An EVA Chronology, 1997–2011, Vol. 2. Monographs in Aerospace History, No. 50, 2016. NASA SP-2016-4550.

No monograph 51.

Gelzer, Christian. The Spoken Word III: Recollections of Dryden History; The Shuttle Years. Monographs in Aerospace History, No. 52, 2013. NASA SP-2013-4552.

Ross, James C. NASA Photo One. Monographs in Aerospace History, No. 53, 2013. NASA SP-2013-4553.

Launius, Roger D. Historical Analogs for the Stimulation of Space Commerce. Monographs in Aerospace History, No. 54, 2014. NASA SP-2014-4554.

Buchalter, Alice R., and Patrick M. Miller, comps. The National Advisory Committee for Aeronautics: An Annotated Bibliography. Monographs in Aerospace History, No. 55, 2014. NASA SP-2014-4555.

Chambers, Joseph R., and Mark A. Chambers. Emblems of Exploration: Logos of the NACA and NASA. Monographs in Aerospace History, No. 56, 2015. NASA SP-2015-4556.

Alexander, Joseph K. Science Advice to NASA: Conflict, Consensus, Partnership, Leadership. Monographs in Aerospace History, No. 57, 2017. NASA SP-2017-4557.

Logsdon, John M., Going Beyond: The Space Exploration Initiative and the Challenges of Organizational Change at NASA. Monographs in Aerospace History, No. 58, 2024. NASA SP-2024-4558.

Buono, Stephen S., Governing the Moon: A History. Monographs in Aerospace History, No. 59, 2024. NASA SP-2024-4559.

Electronic Media, NASA SP-4600

Remembering Apollo 11: The 30th Anniversary Data Archive CD-ROM. NASA SP-4601, 1999.

Remembering Apollo 11: The 35th Anniversary Data Archive CD-ROM. NASA SP-2004-4601, 2004. This is an update of the 1999 edition.

The Mission Transcript Collection: U.S. Human Spaceflight Missions from Mercury Redstone 3 to Apollo 17. NASA SP-2000-4602, 2001.

Shuttle-Mir: The United States and Russia Share History’s Highest Stage. NASA SP-2001-4603, 2002.

U.S. Centennial of Flight Commission Presents Born of Dreams — Inspired by Freedom. NASA SP-2004-4604, 2004.

Of Ashes and Atoms: A Documentary on the NASA Plum Brook Reactor Facility. NASA SP-2005-4605, 2005.

Taming Liquid Hydrogen: The Centaur Upper Stage Rocket Interactive CD-ROM. NASA SP-2004-4606, 2004.

Fueling Space Exploration: The History of NASA’s Rocket Engine Test Facility DVD. NASA SP-2005-4607, 2005.

Altitude Wind Tunnel at NASA Glenn Research Center: An Interactive History CD-ROM. NASA SP-2008-4608, 2008.

A Tunnel Through Time: The History of NASA’s Altitude Wind Tunnel. NASA SP-2010-4609, 2010.

Launching the Future of Flight: The National Advisory Committee for Aeronautics. NASA SP-2025-4610, 2025.

Conference Proceedings, NASA SP-4700

Dick, Steven J., and Keith Cowing, eds. Risk and Exploration: Earth, Sea and the Stars. NASA SP-2005-4701, 2005.

Dick, Steven J., and Roger D. Launius. Critical Issues in the History of Spaceflight. NASA SP-2006-4702, 2006.

Dick, Steven J., ed. Remembering the Space Age: Proceedings of the 50th Anniversary Conference. NASA SP-2008-4703, 2008.

Dick, Steven J., ed. NASA’s First 50 Years: Historical Perspectives. NASA SP-2010-4704, 2010.

Billings, Linda, ed. 50 Years of Solar System Exploration: Historical Perspectives. NASA SP-2021-4705, 2021.

Societal Impact, NASA SP-4800

Dick, Steven J., and Roger D. Launius. Societal Impact of Spaceflight. NASA SP-2007-4801, 2007.

Dick, Steven J., and Mark L. Lupisella. Cosmos and Culture: Cultural Evolution in a Cosmic Context. NASA SP-2009-4802, 2009.

Dick, Steven J. Historical Studies in the Societal Impact of Spaceflight. NASA SP-2015-4803, 2015.

Back Cover

This sunrise was captured from the crew cabin of the Space Shuttle Columbia during the STS-107 mission. (NASA Image 0301002)

On February 1, 2003, the Space Shuttle Columbia and her crew of seven astronauts were lost as STS-107 broke apart during reentry mere minutes before Columbia was scheduled to land at Kennedy Space Center. The Columbia orbiter had suffered a catastrophic failure due to a breach that occurred during launch when falling foam from the External Tank struck the Reinforced Carbon-Carbon panels on the underside of the left wing. The tragedy grounded the Space Shuttle Program and initiated a thorough mishap investigation with a congressionally authorized board. After two and a half years and crucial technical and organizational changes at NASA, the Space Shuttle Program returned to flight.

In this firsthand account, author Steven R. Hirshorn details his experiences over those two and a half years as the operations representative to the Space Shuttle Orbiter Project Office, charged along with others with the safety and success of NASA’s human spaceflight endeavors. With technical expertise and deeply human, visceral insight, Hirshorn’s perspective on these events as they unfolded—and the lessons he draws from them—provide insight for the engineers, managers, and leaders now charged with advancing human spaceflight today.

Steven R. Hirshorn

Author, Chief Engineer for Aeronautics

Office of the Chief Engineer

NASA Headquarters

Image courtesy of Steven Hirshorn

Front Cover

A low-angle perspective captures the Space Shuttle Columbia as it lifts off June 1992. (NASA Image STS050-S-040)

National Aeronautics and Space Administration

Office of Communications

NASA History Office

Washington, DC 20546

www.nasa.gov

NASA SP-2025-4115