Ellis Ratner
University of California, Berkeley
Building robots that act safely around astronauts and as intended when deployed with limited operator supervision depends on designing good reward functions for these robots to optimize. Unfortunately, designing reward functions is difficult, and hence rewards are often misspecified. This can elicit undesirable behavior, and even cause harm to the robot, or astronauts around it. We propose to develop robots that actively query humans for information about the desired reward function when needed, so as to complete their task safely and as intended with minimal supervision. To that end, we will develop an active reward design framework enabling the robot to query the human for information about the desired reward through various modalities, such as demonstrations, corrections, preferences, or full reward functions. We plan to experimentally validate this framework on both simulated and real robotic platforms in two settings. The first is a robot that operates in close proximity to human astronauts, such as Astrobee, that is set to be deployed on the International Space Station. The second is a robot, such as the K-REX planetary rover, operating on a distant planet so that operator supervision is necessarily limited. In this case, the robot will learn through a limited number of queries, constrained, for example, by the bandwidth available.