Sidekick: Investigating Immersive Visualization Capabilities (Sidekick) - 11.22.16

Overview | Description | Applications | Operations | Results | Publications | Imagery

ISS Science for Everyone

Science Objectives for Everyone
Crew members on the International Space Station (ISS) have plenty of devices and reference guides to help them complete tasks, but none is as immersive and simple to use as the new Sidekick tool. Sidekick: Investigating Immersive Visualization Capabilities (Sidekick) is a hands-free, wearable remote assistance system that enables high-definition 3-D holograms mixed with real-time views, enabling new ways to communicate and work. Sidekick improves efficiency for crew tasks such as science experiments, maintenance and operations on the ISS.
Science Results for Everyone
Information Pending

The following content was provided by Jeffrey Norris, and is maintained in a database by the ISS Program Science Office.
Experiment Details

OpNom:

Principal Investigator(s)
Jeffrey Norris, Jet Propulsion Laboratory, CA, United States

Co-Investigator(s)/Collaborator(s)
Victor Luo, Jet Propulsion Laboratory, CA, United States

Developer(s)
Microsoft Corporation, Redmond, WA, United States
NASA Jet Propulsion Laboratory, Pasadena, CA, United States
NASA Johnson Space Center, Houston, TX, United States

Sponsoring Space Agency
National Aeronautics and Space Administration (NASA)

Sponsoring Organization
Technology Demonstration Office (TDO)

Research Benefits
Earth Benefits, Scientific Discovery

ISS Expedition Duration
September 2015 - March 2016

Expeditions Assigned
45/46

Previous Missions
Information Pending

^ back to top

Experiment Description

Research Overview

  • Sidekick: Investigating Immersive Visualization Capabilities (Sidekick) is a commercial crew-worn immersive procedural reference and remote assistance (guider) system.
  • Sidekick provides the crew information and assistance when they are in the middle of completing a task utilizing high-definition holograms.
  • Sidekick tests the possibility of increasing crew efficiency in the daily execution of science and operations.

Description

Sidekick: Investigating Immersive Visualization Capabilities (Sidekick) is a crew worn immersive procedural reference and remote assistance (guider) system developed to provide the crew information and assistance when and where they need it. It enables high-definition holograms to come to life in their world, seamlessly integrating with the physical places, spaces, and things. This is a mixed reality experience in which holograms are mixed with the real world enabling new ways to communicate and work. The goal of Sidekick is to increase the efficiency of crew activities related to the daily execution of science, utilization and operations aboard the ISS.
 
Using Sidekick provides flexibility to assist the crew with remote expert support from the flight control team, Payload Operations Integration Center (POIC), system expert or payload developer to support mission operations. Procedures are displayed as holograms such that they can be inserted when and where they are needed to support a given operation or task. The flight control team and POIC have the capability and opportunity to see what the crew sees from their perspective. The experts on the ground assist the crew by marking up their world with directions or pointers.
 
Sidekick can be operated in the two modes, Standalone Mode (procedure reference) and Remote Expert Mode. The Sidekick application in Standalone Mode is intended to be an additive tool to assist crew members with complex procedures or to support procedures requiring multiple crew members. The Sidekick application has all procedures pre-loaded within the application and the on-orbit wireless and ground network are not required. The Sidekick application, which provides the holographic procedure viewing capability is not intended to replace any existing ISS procedure viewing applications (International Procedure Viewer, etc.). The Remote Expert Mode is a video teleconference capability that allows the crew to have direct support from the flight control team, POIC, system expert or payload developer to support ISS mission objectives.
 
Initial remote expert mode is intended for the Operations Support Officer (OSO) Console with expansion to the POIC and other Flight Control Team (FCT) Consoles as required. The Remote Expert Mode requires the Sidekick Device, on-orbit wireless communications, KU-Band connectivity and Internet access to provide this capability.

^ back to top

Applications

Space Applications
Sidekick provides hands-free, heads-up assistance to crew members in space, enabling new ways to communicate with each other and with support personnel on Earth. High-definition holograms integrate with physical places and objects, allowing crew members to see schematics or diagrams in 3-D to help them complete tasks. Sidekick operates in two modes, allowing it to work as a simple reference tool or as a teleconferencing tool.

Earth Applications
The technology enabling Sidekick has many potential uses on Earth, from construction and design projects to meetings to immersive play.

^ back to top

Operations

Operational Requirements and Protocols

The checkout of Remote Expert Mode requires a scheduled video teleconference with the ground and also coordination with MCC for the initial teleconference with each of two devices and two crew members for a one hour call each to verify the quality of the video and audio of the device and annotations.

Crew member powers on the device. Crew places and adjusts the Sidekick device on head. Crew carries out tasks as designed.

^ back to top

Decadal Survey Recommendations

Information Pending

^ back to top

Results/More Information

Information Pending

^ back to top

Related Websites

^ back to top


Imagery

image The Sidekick hardware. Image courtesy of Microsoft.
+ View Larger Image


image
Operational concept and implementation of the Sidekick investigation. Image courtesy of NASA.

+ View Larger Image


image
NASA and Microsoft engineers test Project Sidekick on NASA's Weightless Wonder C9 jet. Project Sidekick uses Microsoft HoloLens to provide virtual aid to astronauts working on the International Space Station. Image courtesy of NASA.

+ View Larger Image