Skip to main content

Missouri State University home page Missouri State University Libraries

Microsoft Hololens: Hololens in Science

Microsoft Hololens mixed reality device at Meyer Library

Science

FEBRUARY 20, 2016 7:46 AM

Microsoft HoloLens in space: Making science fiction (mixed) reality

By  / Technical Fellow, Microsoft

In December, a shuttle resupply mission successfully reached the International Space Station. Among the cargo were two Microsoft HoloLens devices for use as a part of NASA’s Sidekick project. The goal of Sidekick is to enable station crews with assistance when and where they need it. According to NASA, this new capability could reduce crew training requirements and increase the efficiency at which astronauts can work in space.

We were thrilled to see some early pictures today of astronaut Scott Kelly with HoloLens at the International Space Station!

Scott Kelly using Microsoft HoloLens at the international space station.

To provide a little background on the project, Sidekick has two modes of operation. The first is “Remote Expert Mode,” which uses Skype, to allow a ground operator to see what a crew member sees, provide real-time guidance, and draw annotations into the crew member’s environment to coach him or her through a task. Until now, crew members have relied on written and voice instructions when performing complex repair tasks or experiments.

The second mode is “Procedure Mode,” which augments standalone procedures with animated holographic illustrations displayed on top of the objects with which the crew is interacting. This capability could lessen the amount of training that future crews will require and could be an invaluable resource for missions deep into our solar system, where communication delays complicate difficult operations.

 

In order to prepare for the mission, and what it would be like to use HoloLens at the International Space Station, NASA had a chance to experiment with it quite a bit at the Aquarius underwater research station as a part of NASA Extreme Environment Mission Operations NEEMO. Below are a few pictures of Astronaut Luca Parmitano using HoloLens at the underwater facility.

 
 
 
 
 

And we’re happy to report that HoloLens is mission operational at NASA’s Jet Propulsion Laboratory – exploring Mars using holograms of Mars Rover images.

We couldn’t be more thrilled about the work we are doing with NASA – I can’t wait to see more from the crew at the International Space Station!

UPDATED FEBRUARY 22, 2016 11:01 AM

Science

 
A screen view from OnSight, a software tool developed by JPL in collaboration with Microsoft

 

 

 

 
 
Scientist Katie Stack Morgan examines rover images on her computer.
 
 
 
 
 
A screen view from OnSight, a software tool developed by NASA's Jet Propulsion Laboratory in collaboration with Microsoft. OnSight uses real rover data to create a 3-D simulation of the Martian environment where mission scientists can "meet" to discuss rover operations. Image credit: NASA/JPL-Caltech
› Larger image

NASA and Microsoft have teamed up to develop software called OnSight, a new technology that will enable scientists to work virtually on Mars using wearable technology called Microsoft HoloLens.

Developed by NASA's Jet Propulsion Laboratory in Pasadena, California, OnSight will give scientists a means to plan and, along with the Mars Curiosity rover, conduct science operations on the Red Planet.

"OnSight gives our rover scientists the ability to walk around and explore Mars right from their offices," said Dave Lavery, program executive for the Mars Science Laboratory mission at NASA Headquarters in Washington. "It fundamentally changes our perception of Mars, and how we understand the Mars environment surrounding the rover."

OnSight will use real rover data and extend the Curiosity mission's existing planning tools by creating a 3-D simulation of the Martian environment where scientists around the world can meet. Program scientists will be able to examine the rover's worksite from a first-person perspective, plan new activities and preview the results of their work firsthand.

"We believe OnSight will enhance the ways in which we explore Mars and share that journey of exploration with the world," said Jeff Norris, JPL's OnSight project manager.

Until now, rover operations required scientists to examine Mars imagery on a computer screen, and make inferences about what they are seeing. But images, even 3-D stereo views, lack a natural sense of depth that human vision employs to understand spatial relationships.

The OnSight system uses holographic computing to overlay visual information and rover data into the user's field of view. Holographic computing blends a view of the physical world with computer-generated imagery to create a hybrid of real and virtual.

To view this holographic realm, members of the Curiosity mission team don a Microsoft HoloLens device, which surrounds them with images from the rover's Martian field site. They then can stroll around the rocky surface or crouch down to examine rocky outcrops from different angles. The tool provides access to scientists and engineers looking to interact with Mars in a more natural, human way.

"Previously, our Mars explorers have been stuck on one side of a computer screen. This tool gives them the ability to explore the rover's surroundings much as an Earth geologist would do field work here on our planet," said Norris.

The OnSight tool also will be useful for planning rover operations. For example, scientists can program activities for many of the rover's science instruments by looking at a target and using gestures to select menu commands.

The joint effort to develop OnSight with Microsoft grew from an ongoing partnership to investigate advances in human-robot interaction. The JPL team responsible for OnSight specializes in systems to control robots and spacecraft. The tool will assist researchers in better understanding the environment and workspace of robotic spacecraft -- something that can be quite challenging with their traditional suite of tools.

JPL plans to begin testing OnSight in Curiosity mission operations later this year. Future applications may include Mars 2020 rover mission operations, and other applications in support of NASA's journey to Mars.

JPL manages the Mars Science Laboratory Project for NASA's Science Mission Directorate in Washington, and built the project's Curiosity rover.

Learn more about NASA's journey to Mars at:

http://www.nasa.gov/mars