A conversation with Jeremy Frank, Principle Investigator for the Autonomous Systems and Operations Project at NASA’s Ames Research Center in Silicon Valley.
Transcript
Matthew C. Buffington (Host): You are listening to the NASA in Silicon Valley podcast, episode 26. A common mathematical problem involves the famous traveling salesperson problem. Basically, a salesperson has to visit a number of houses in various locations and has to work out the shortest route possible that still hits all of the houses. Seems simple enough, but this problem gets exponentially more complex and difficult with more houses to visit and possible roads taken. Computer scientists and NASA have been working on variations of this problem, trying to optimize precious time and resources. One such researcher is today’s guest, Jeremy Frank, the Principle Investigator for the Autonomous Systems and Operations Project. We talk about his work in autonomy and how NASA delivers innovative solutions that dramatically improve the agency’s capabilities while also supporting the innovation economy. Without further delay, here is Jeremy Frank.
[Music]
Host: Tell us a little bit about how you came to Silicon Valley, how you ended up joining NASA. Tell us about yourself.
Jeremy Frank: Sure. So, I actually came here because someone who used to work here came and gave a talk when I was in graduate school, and I actually ended up working with that guy for about three summers before I joined as a contractor in 1997.
Host: Are you originally from California?
Jeremy Frank: No, I moved here to go to school, and I went to school in southern California for four years and then I came to northern California and went to school for another five or so years.
Host: Wow, lot of school.
Jeremy Frank: Lot of school.
Host: Has it always been STEM?
Jeremy Frank: Always.
Host: Always interested in, was it like, were you always interested in science in general, or is it more engineering?
Jeremy Frank: Math and computers, all math and computers, although NASA was not actually on my radar screen early on. So, that was a surprise. But I’ve loved working here, I’ve worked a lot of different things since I’ve been here.
Host: So, when you first came on, was it as a Ph.D. student or something like that?
Jeremy Frank: Yeah.
Host: What did you first start working on?
Jeremy Frank: So, I work in computer science, and I’m interested in problems that are hard for computers even to solve. And so, without getting way down into the weeds, these problems are hard because if you think about just sorting a bunch of numbers, you can sort numbers pretty fast. If you have N numbers, you can sort them in less than N-squared individual operations — in fact even fewer than that. So, that’s fast. But if you imagined that you had N things that you wanted to say arrange into the shortest possible way to deliver packages in N cities, we as computer scientists don’t know how to do that in less than N-factorial. N-factorial is in fact N to the N, 2 to the -N, et cetera, et cetera. So, N to the N is a lot worse than N-squared.
Host: I’ve heard this described as almost the salesman dilemma.
Jeremy Frank: The traveling salesperson’s problem.
Host: Yes, that’s what I’ve seen.
Jeremy Frank: So, there are bunches of problems like the traveling salesperson problem. Many of them are scheduling problems or planning problems. There are also problems in things like identifying a fault, like in a circuit that’s faulty, or if I’ve lost a bunch of equipment, and I don’t know exactly what happened, to take all that equipment out. Some of those fault management diagnosis problems can also be problems that take exponential time. And so, a lot of these problems NASA cares very deeply about, so those are the classes of things I worked on when I first got here, in a very purely mathematical and computer science sense.
Host: Okay, so it’s more than just necessarily brute force, trying every single possible solution to that traveling salesman dilemma. It’s, I have to step back, is there a more creative way to do this? Do it like in bunches, or I guess it’s a problem that’s still being worked out. It hasn’t been solved yet.
Jeremy Frank: It hasn’t been solved. What people do is, they say I have a clever technique that’s just a little smarter than searching everything. And those clever techniques, there are a variety of different ones. There are basically educated guess techniques, there are educated guess techniques that get folded in between searching every possibility. There’s, I can rule a bunch of things out because of what I’ve discovered while I’ve been trying to do my search, and the list goes on. And these are things that Ph.D. students in computer science continue to work on to this day.
We here at NASA, our job is to take the best and coolest of those ideas and to say, can I apply that to my spacecraft mission operations problem and do it better than I’ve ever done it before? So, to some extent, that’s been the shift in my career and the career in a lot of other people here.
Host: It’s like, find out what that cutting edge, the forward of research and thinking, and now let’s give it to the engineers, or work with the engineers to figure out how this can actually solve these problems that we are having.
Jeremy Frank: That’s right, and to expound on that difference a little bit, if I’m a math person, I think in terms of, if I had an infinite number of things, which of course I don’t…
Host: Yes, budget being one of them.
Jeremy Frank: How does this scale? Does it scale as N-factorial over 282 trillion, which means that it’s better than just N-factorial? Mathematicians get all excited about that.
Host: And everybody else, their eyes glaze over — well, the public.
Jeremy Frank: An astronomer who says, I’ve got 32 targets tonight. I don’t care about how does it work when it’s N-factorial as N goes to a million, that’s not interesting to me.
Host: What can I deal with right now?
Jeremy Frank: This is my problem, put these things to work, and by the way, you’ve got 30 minutes to build me a schedule. Okay, and I’m supposed to do this on some commodity computer, I don’t have infinite compute power either. So, that’s the difference between pure mathematics and practical applications, engineering, operations.
Host: Okay, so you’re working in that nexus of all those things, powers combined.
Jeremy Frank: That’s where we are. And in particular, the niche area that I apply this now is human spaceflight.
Host: Okay, so did you start off working on the mathematics side and now you’ve moved into how to apply this to humans in space?
Jeremy Frank: Yeah, that’s the way it worked, humans in space.
Host: So, when you think of humans in space, obviously you think of life support, dealing with the radiation, how you keep a delicate carbon-based lifeform alive in the harshest of conditions. How does math play into that?
Jeremy Frank: How does math play into it? There are a couple ways. The first part of the problem, it’s good you mentioned life support. We mentioned that some of these problems of, tell me what went wrong, these are hard problems, they can be. And so, if the general case of tell me what went wrong is a hard problem, and I imagine my astronaut on the way to Mars has to figure out what went wrong by themselves — the control center’s too far wrong, they can’t respond in time.
Host: There’s a delay, even.
Jeremy Frank: Well, delay, out of communication. When Mars and earth are on the opposite side of the sun, they can’t talk at all for two weeks. So, there could be a period of time where the crew’s really on their own when we go to Mars. I need this kind of software, not just the math/brainy stuff, but it needs to be packaged in such a way that a human being can understand what it’s saying at the end of the day. That’s where it all comes together. That’s just one of many ways.
Host: So, what are some of the stuff that you’re elbows-deep in right now? What’s some of the cool stuff that you’re working on?
Jeremy Frank: We have a project which really is all about how are we going to send people to Mars and how are we going to keep them safe and how are we going to keep them effective? So, we take things like those cool diagnosis algorithms to deal with spacecraft faults. We also look at the problem of, if a crewperson has to schedule their own day, that can be a couple of hundred activities and there can be a lot of constraints. So, it’s a little bit like that traveling salesman problem. So we ask, what of these things do we have to put to work to help the crew figure out how they can arrange their day with all the constraints they have, how do they choose what activities to do because maybe you can’t do everything, how do you make sure it all goes in the right place? These two activities can’t overlap, these two activities have to be separated by a minimum amount of time. That’s another place where it comes in.
Host: You can’t avoid it, the traveling salesman, it keeps coming back.
Jeremy Frank: The traveling salesman is even in space.
Host: He’s got a long way to go.
Jeremy Frank: That’s right. But if you think then about things like what can go wrong, every piece of equipment onboard a spacecraft has to get power from somewhere. So, looking to the power system and asking, how does the power system configured to distribute power to all those components? Has something been turned off so I don’t need to worry about it, or was it turned on and something commanded it off so I don’t need to worry about it, or did I just lose 2/3 of the powered pieces of my spacecraft and what happened? It’s highly unlikely that every single one of them decided to pack it in all by themselves completely independently, so it’s more likely that the power system took it all with it, and something happened to the power system. That’s the place where that diagnosis problem comes into play.
But if I look at things like the water recycling system onboard my spacecraft, if that’s not behaving properly, I have to think about, where’s all the water, how’s the water moving from one place to another, how’s the water being processed, similarly how is the water system and the air system integrated? Is the water system packing in because the power system has broken and it’s taken out part of my life support system, or is there a dirty filter, is there scum growing in the tanks and it’s clogging filters and valves are sticking — what’s going on? So, the diagnosis part of the problem also requires understanding enough about how those systems work in order to inform it.
Host: And some of your work also falls into some of the autonomy, or autonomous…?
Jeremy Frank: That’s right. All of this, you could think of all the same technologies living in the space mission control center today. So, instead of an astronaut doing this task, I might have a flight controller doing it. But, if the flight controller’s doing it, they’re waiting for the information to come from the spacecraft over the air-to-ground link, into the control center, and then it’s now the flight controller’s job to look at all that, to use those algorithms, and then to gather themselves together and to say, this is what we think is going on, and this is what we need to tell the crew. That’s the way it’s done today. Tomorrow, when that spacecraft is five light-minutes from earth, 10 light-minutes from earth –
Host: It’s great with the ISS, this International Space Station, relatively really close, and then when you start heading out to Mars, there’s a delay.
Jeremy Frank: It’s not going to be quite so simple.
Host: Even at the speed of light, it’s still going to be a delay.
Jeremy Frank: Light minutes, it’s just insane, yes. But, take all that technology, all that brain power. There’s a minimum of six people operating the ISS every single day, and usually it’s more like 12 or 15, and sometimes it’s as many as 30. We imagine there will be four crew on the way to Mars, four. So, all that expertise, all that knowledge, it needs to be compressed greatly.
Host: And as much assistance as you can give them.
Jeremy Frank: That’s right. And we don’t need to solve the entire problem of the control center onboard, but in the worst case, when they’ve got nothing, they’re out of communication, that’s the class of help we think we need. So, all that autonomy part of the problem is, how much of that control center smarts, whether it’s software, human beings, or a combination of the two, how much of that do we need to put on that Martian-bound spacecraft?
Host: Okay, so you figure some of like — “easier tasks” isn’t necessarily the right way to put it — but there may be some things that can be made autonomous, so they don’t have to be in the weeds of some things, but you can hand it over to software or robots or vehicles or whatever?
Jeremy Frank: That’s right. The autonomous element is the human being and all of their smart software on their spacecraft, doing the job without asking for help from mission control. And so, one of the big questions is, how autonomous do they really have to be? Once you figure that part out, the second part of the question is, does the human need to be a participant in this, or can the software and the spacecraft do the job all by themselves, or is it going to be some combination of the two?
Host: As you figure this stuff out, there’s any number of ways this could be applicable to daily life.
Jeremy Frank: I think that’s exactly right. So, one obvious way is, if you think about, let’s pick on the Navy — so, submarine crews do this today, all the time. Large submarines go and travel without talking to anybody for months at a time. So, that’s an obvious application. There’s a second one, which is remote medicine, or telemedicine. If you think, one of the terrible things that could happen onboard your spacecraft is someone is injured or they get sick.
Host: You break a leg.
Jeremy Frank: Break a leg, hit your head, I have a headache and it’s just not going away and I don’t know what to do.
Host: Just imagine getting sick in close quarters.
Jeremy Frank: That’s right. This is a big cause for concern. And so, think about the packaging of autonomous medical diagnostics technology to help understand what exactly is your problem, and once you’ve figured out what’s going wrong, what is it that you do about it? So, there are numerous places that don’t have access to great big hospital systems throughout the world. Think about the technology to help that. A variety of exploratory — people who are off in, say, harsh conditions like on mountaintops or in jungles, to be able to use a lot of this technology. You’re absolutely right, there are a lot of ways in which this all plays together. Even things like a remote power station or a remote water processing plant.
Host: I was going to go right into, you think of places on this planet where power is a precious resource, water, and if we can figure out how to be life- or self-sustaining and supporting in space, clearly there’s application to other countries, other areas.
Jeremy Frank: Even things as innocuous as the self-driving car revolution, if you think about, these cars are very complex today. They have multiple computer systems, they have a complex avionic system.
Host: They have LIDAR lasers on top.
Jeremy Frank: Eventually they’re going to have all of those things. So, looking at even just better diagnostics for the car that’s not autonomous — I don’t want to see an idiot light, I want to know what’s gone wrong. I want to have a sense of…
Host: Tell me what’s wrong, tell me what I’m ignoring.
Jeremy Frank: You need to take this to the shop right now versus you can take this to the shop eventually, but really you need to take it in.
Host: Or when they fixed the car, they left the light on, and so you get to go all the way back just for them to turn it off.
Jeremy Frank: The other place that NASA is exploring that has a very good crossover with a lot of this is in aviation. There’s a lot of question about, how is it that a smaller pilot or cockpit crew can fly a commercial-class aircraft today. Some of the same kind of diagnostics considerations are already being employed for aircraft, mostly in control centers and in maintenance, but you could also think about those things as being useful on an airplane in flight.
Host: Just even thinking of our own cars, apart from even being autonomous, cruise control. It’s more features of a souped-up cruise control, all the way to you call it up on your phone and hop in and go. There’s a whole range.
Jeremy Frank: That’s the ideal, that’s the ideal.
Host: Obviously, there’s the human exploration component and how it applies to real life, but are you working on any probes, or that first-wave scouting of autonomy, or even going places where the human literally can’t go?
Jeremy Frank: Well, we actually get to learn a lot from what’s already being done in this area. If you think about our future Martian mission, we’ve been sending spacecraft to Mars since 1976 with the Viking landing, so a lot of our informed ideas about what must be done and what doesn’t need to be done come from the way that Viking and all of its successors, right up to Spirit, Opportunity and Curiosity, as well as all the Martian orbiters. If you look at how they work today, they work on a cycle where they build a plan and they send that plan to the spacecraft, and then they wait for the spacecraft to report back about how things went.
That planning cycle, that daily planning cycle, it’s not that different than what’s done on the International Space Station today. While the crew’s asleep, the control center in Houston and all the other control centers around the world are building the crew’s plan for the next day. So, that’s great news, because it means the crew doesn’t need to figure out how to plan their own activities for a month ahead of time.
Host: They wake up with a to-do list.
Jeremy Frank: And they’re going to wake up with a to-do list on Mars, and we know that because of the way the JPL runs their missions. But if you take a look at some of the autonomy-enabling technology we’re building for human space exploration, the part of it that differs from what we already know how to do for probes that go to not just Mars but elsewhere, like Pluto, it’s the human interface because of the presence of the astronaut, that’s a big difference.
The second big difference is the presence of a life support system, because we have a human crew and we need to keep them alive — most Martian missions don’t need that. All Martian mission of the past don’t need it. There is actually a very intriguing element of Mars 2020, which is oxygen extraction. So, even now, we’re starting to see some of those future missions doing things that are a little bit more human spaceflight-like. So, that’s changing.
And then the last one is just a matter of scale. The International Space Station, the U.S. part of the International Space Station alone sends at least 180,000 distinct pieces of information to the control center all the time. So, the scale is much larger than anything we have sent further out.
Host: And then trying to sort that out and make sense of that data is an entirely different thing.
Jeremy Frank: That’s right. So, it really puts into perspective the potential for huge problems when we have a crew, an astronaut crew, do that job. We know they can’t do the whole thing. So the question is, what kind of help do we give them?
Host: Going back even to the self-driving cars, it goes to a thing of, we know how humans drive, and oftentimes it’s not very well. Or it’s like, humans text, they eat food, they have their kid crying in the back, they get sleepy — a robot doesn’t text or eat McDonalds while driving down the road.
Jeremy Frank: That’s exactly right, but if I’m battery-powered, I’ll run out of juice exactly the same way a human-rated car does. There’s this issue that when I have those cars working together…
Host: Like a hive mentality.
Jeremy Frank: That’s right. Even if they were all communicating with each other, which maybe they could, there’s still going to be imperfect information, there’s going to be delay, there’s going to be lag — there’s going to be lag of communication, lag of processing, and there’s going to be the unexpected event — there’s a pothole I didn’t expect. So, all those things are interesting problems in how groups of robots are going to work together.
But to come back to the human spaceflight part of the problem, because the humans will have some idea of what they want to do that will be imperfectly understood by the automated system, that puts an interesting premium on all this automated reasoning that we want to do. Take planning for example. I can build a plan and present that to the crew, and the crew might say, that plan’s great, but it didn’t take into account something I would like to do, so I am going to ignore that plan and do something else. If we don’t build our planning systems properly, they’ll be totally freaked out by that, and it’s a bad situation.
So it’s a nice thing, and when I was studying computer science, that wasn’t a problem that I had really thought about. And even now, there’s a small subset of people who are thinking about that as something to worry about.
Host: In your work, you’re kind of in that nexus of supercomputing, autonomy, spaceflight. So, how does the supercomputing help and augment or play into all of these different factors together?
Jeremy Frank: Well, what we want to do to the best of our ability is build a plan the human wants. However, we need to be able to roll with what the human says they would like out of a plan. So, here’s one example of how you might do that — don’t build a plan the human can’t change around. That’s a pretty straightforward one.
Host: Yeah, they want to be able to have the flexibility to change it up.
Jeremy Frank: That’s right, but humans make mistakes. And so if a human does something that the system says, you gave me a constraint that you’re not supposed to do that — either you’re smarter than me, and you have to tell me it’s all right in this circumstance and I won’t bother you with that anymore…
Host: But at least let me give you a heads-up.
Jeremy Frank: But I need to give you the heads-up. So, there’s that facet of things. Another one is, as you specify a problem of building a plan, there could be a lot of different solutions. To give you an example, suppose I tell you your goals for today are to go to the grocery store, to go to work for at least four hours, to take a run, and to meet your bud for lunch. Well, I didn’t tell you exactly how to order all those things. There may be some you can’t do at the same time. So, you know maybe you can’t eat lunch and have a run at the same time, so that’s obvious. But you can go to the store at the very beginning of the day if you’re buying nonperishables and nothing that’s cold, or maybe you have to go at the end of the day because you have to hit the ATM first. But, you can see there are a lot of plans. Without presupposing which plan is the best, the system should be able to generate new plans quickly, or to generate say the top five plans and let you pick one.
So, there are a variety of these kinds of considerations. Many of these are well understood in computer science, some of them not so well. But the point is, look at those tools we have in the computer science toolbox, ask the people who are the consumers of plans, when we give you plans, what do you want to see, how do you want to see them? How do you want to manipulate them, how do you want to reinvoke the system and ask for a new plan? So, that’s a big part of what we do, and with diagnosis it’s very much the same problem.
Host: Help me help you.
Jeremy Frank: Help me help you, but it’s a two-way street. We need to think about it from, what are computer science tools in the toolbox we can use to help? What do we use for human interfaces, how is it that we present the information to people? And when we get really lucky, we encounter a new, interesting technical problem, and we go back to computer science and say, we need some help with this, because this is new and we’re not totally sure how to solve this. So, that’s fun.
Host: This is crazy, and how it cross-sections with all the different missions and structures of NASA. It’s like how all this stuff can work together.
Jeremy Frank: Before a couple of years ago, I really did not have a good sense of just multidisciplinary it all is.
Host: So, for anybody who’s listening who wants to get more information on this, there’s a whole wide range. Probably the best place is to go to NASA.gov and search around.
Jeremy Frank: So I’m in what’s called the intelligent systems division at NASA, so if you search for the intelligent systems division at NASA Ames Research Center, my work and a lot of the other work we do that’s in a similar spirit can be found there. Within the human spaceflight part of NASA, I am part of what is called the Advanced Exploration Systems Program, and so you can search for the Advanced Exploration Systems Program. And the project I run that this is all a part of is called the Autonomous Systems and Operations Project. And so, you will ultimately be able to find out some more about what we’ve done there.
Host: Excellent. And also for anybody who’s listening, if you want to use the smartphone, the computer in your pocket, we’re on Twitter @NASAAmes, we’re using #NASASiliconValley. Thank you so much for coming, Jeremy, this has been great.
Jeremy Frank: You’re welcome, thank you.
[End]