HET Smartphone-MM (Smartphone-MM) - 12.13.17

Overview | Description | Applications | Operations | Results | Publications | Imagery

ISS Science for Everyone

Science Objectives for Everyone
Smartphone-MM takes advantage of smartphone technology and the SPHERES autonomous navigation satellites’ capabilities to demonstrate remote operations of robots flying 230 miles above the Earth.  The experiment evaluates the capability of consumer electronics to perform vision-based navigation and reduces risk in technology development for a robotic free-flyer.
Science Results for Everyone
Information Pending

The following content was provided by Terry Fong, Ph.D., and is maintained in a database by the ISS Program Science Office.
Experiment Details

OpNom: SPHERES Smartphone

Principal Investigator(s)
Terry Fong, Ph.D., NASA Ames Research Center, Moffett Field, CA, United States

Information Pending

Information Pending

Sponsoring Space Agency
National Aeronautics and Space Administration (NASA)

Sponsoring Organization
Technology Demonstration Office (TDO)

Research Benefits
Information Pending

ISS Expedition Duration
March 2014 - March 2015; September 2017 - February 2018

Expeditions Assigned

Previous Missions
Information Pending

^ back to top

Experiment Description

Research Overview

  • Demonstrates use cases and robotics tasks that may be used by crew or mission controllers.  Allows robot to off-load routine or hazardous tasks from crew so that crew can focus on science tasks (as opposed to maintenance).  Provides new asset and increased situational awareness for mission control.  Explores new methods of navigating a free flyer on ISS; required for a next generation robotic free flyer.  Matures technology for implementation on a next generation robotic free flyer.


The Human Exploration Telerobotics SPHERES Smartphone project is pairing the SPHERES robotics platform with a new in-development smartphone that performs visual odometry. The SPHERES platform has a successful history of operations onboard parabolic flights and the International Space Station (ISS). It is a small satellite that propels itself by releasing small bursts of CO2 from an internal tank. The satellite navigates prior rigging of its environment with ultrasound beacons. SPHERES measures time-of-flight of those beacons’ signal through an array of microphones found on its surface. The collation of these measurements defines the pose of the satellite. This system works well but limits SPHERES operation to just a small volume defined by the parameter of the beacons. HET plans to implement smartphone technology that will enable SPHERES-Smartphone to determine satellite pose without the need for beacons.

Smartphone-MM Description
Like the first HET Smartphone, the new Smartphone runs on the Android operating system.  It is a prototype smartphone in development by a 3rd party vendor.  What makes this Smartphone so unique is that it implements visual odometry, which is the measurement of pose by the tracking of features or landmarks it sees through its sensor suite. This suite is composed of a fisheye lens camera, a narrow angle IR and RGB camera, an infrared projector, and finally an additional forward facing camera. At the time of this writing the primary workhorse is the fisheye lens. The Smartphone contains dedicated IC logic that tracks features between sequential observations of the fisheye lens. These captured features plus the measurements from a 120 Hz Inertial Measurement Unit (IMU) are fed to a Kalman filter that outputs an estimate of the current pose.

The infrared projector and IR and RGB camera are used in conjunction to capture occasional dense 3D measurements of the surroundings. They work in a similar fashion as the popular Kinect sensor used on the Microsoft Xbox 360. The projector and camera are calibrated to each other so that their baseline and focal length are well known. The projector casts a static image on to the world that is composed of scattered dots. This texture is unique in that its design enables an algorithm to localize a small crop of the texture in the context of the entire image. This texture is infrared and invisible to the naked eye but not to the narrow angle IR and RGB camera. Its IR observation allows every patch of the projected image to be triangulated by finding the intersection between cast rays of the projector and camera creating dense 3D models. These models are not used for navigation directly. In application, these dense measurements are taken sparingly so that only occasional snapshots of new scenes are captured. Offline from the Smartphone, these observations are merged together to make a large contiguous map. This map can then be loaded back on to the Smartphone to 1) establish a coordinate system for navigation, and 2) allow it to localize to better precision than visual odometry can do alone in real time.

The Smartphone also has a UBS 3.0, a USB On The Go, and a mini HDMI slot.
The current prototype includes a front facing camera and proximity sensor, and on the back, a rear camera (IR and regular RGB), a LED, IR projector, and a fish eye camera.  The HET Smartphone will be based on the next prototype version.  The next version will differ from the current version in the following ways:

• It will have a different form factor.  The phone will be wider (as much as twice as wide); all sensors on the rear of the phone will be moved to the front.  This will allow all of the sensors and the touch screen to face out from satellite when mounted.
• It will not include a GSM radio.  This design modification will mitigate RFI/EMI hazards.
• A different battery will power it.  The Smartphone will be designed and manufactured to run off batteries that are already certified for ISS (AA battery pack, or a lithium ion battery used for ISS cameras (TBD)).
• The screen will include protective covering.  This will mitigate the hazard associated with a broken touch screen.
• The front camera and proximity sensor will be removed.

ISS Testing
The objective of the ISS testing is to demonstrate and assess the navigation and localization capabilities of the Smartphone without the use of the SPHERES beacon system.  To meet this objective, the SPHERES Smartphone will fly several trajectories in increasing complexity and length.

The Smartphone must first generate a 3-D map of the interior of the ISS U.S. orbital segment (USOS).  The crew will post an augmented reality (AR) code (similar to a QR code) on a fixed surface in the ISS.  This will provide a reference point and coordinate system for the Smartphone.  The crew will then scan the interior of the USOS with the Smartphone (SPHERES satellite not required for this activity).  After the session, the data will be downlinked, and the 3-D map will be constructed on the ground.  The map will then be used to plan the routes for the subsequent demonstration.

Prior to the demonstration, the crew will re-post the AR code in the same exact location as it was first used.  The Smartphone will identify the AR code, then synchronize its position and pose with respect to the ISS and its map.  The first route the Smart SPHERES will fly will be within the beacon volume.  This will allow the SPHERES to calculate position and pose, though that data will not be used for navigation.  It will be used later for comparison between the SPHERES beacon and Smartphone localization solutions.

The second SPHERES Smartphone route will leave the beacon volume.  The satellite will fly across the JEM, into Node2, then turn and face down the long axis of the U.S. Lab.

The third route will start in the JEM, and this time the satellite will fly to the aft end of the lab, and attempt to bring a designated target (for example, the Audio Terminal Unit (ATU) located near the aft hatch.  This will demonstrate navigation required to conduct a IVA inspection.

The fourth and final route will start and end in the JEM.  The satellite will fly to the US Lab and hold position for one minute.  This will demonstrate navigation required to conduct an environmental survey in the USOS.  The satellite will then return to its starting point in the JEM.

^ back to top


Space Applications
The Space Application for this investigation is the development of IVA or EVA robotic free flyers that can support current or future HEOMD missions.

Earth Applications
The Earth Application for this investigation has yet to be identified.

^ back to top


Operational Requirements and Protocols

ISS Mapping session:
• 1 crewmember for 1 hr
• 1 Smartphone-MM
• 1 SSC for uploading software to Smartphone-MM prior to activity, and for download of data after the session
• Real-time Ku-Band and S-Band for voice and video support
Navigation demonstration:
• 1 month after the ISS Mapping session
• 1 crewmember for 4 hrs
• 1 SPHERES Facility, 2 Smartphone-MM, 2 SSC for SPHERES GUI and uploading software to Smartphone-MM prior to activity, and for download of data after the session
• Real-time Ku-Band and S-Band for voice and video support

ISS Mapping session:
• Crew sets up Smartphone-MM (batteries & settings)
• Crew loads new application from SSC onto Smartphone-MM
• Crew moves through entire ISS USOS while pointing the Smartphone-MM at all forward, aft port, starboard, overhead and deck racks and hatches
• Crew downloads data from Smartphone-MM to SSC, then stows Smartphone-MM
• Data downlinked for payload developer
Navigation demonstration:
• Crew sets up 1 SPHERES facility and 2 Smartphone-MM (1 smartphone is connected to satellite)
• Crew uses smartphone touch screen to launch satellite test runs in JEM, N2 and US Lab
• Crew carries 2nd smartphone as a remote e-stop
• Crew downloads data from Smartphone-MM to SSC, then stows SPHERES and Smartphone-MM
• Data downlinked for payload developer

^ back to top

Decadal Survey Recommendations

Information Pending

^ back to top

Results/More Information

Information Pending

^ back to top

Related Websites

^ back to top


This close-up view shows three basketball-sized free-flying satellites called Synchronized Position Hold, Engage, Reorient, Experimental Satellites (SPHERES) flying inside the Destiny laboratory of the International Space Station (NASA image). 

+ View Larger Image

SPHERES and HET Smartphone attached (NASA/ARC Image).

+ View Larger Image

Modified Smartphone with Modified Battery Pack (NASA/ARC Image). 

+ View Larger Image