Tensor Holography – 3D Imagery on a Smartphone

April 2021

A new method called tensor holography could enable the creation of holograms for virtual reality, 3D printing and medical imaging that can run on a smartphone. Daniel Ackerman, writing in MIT News, explains how.

Despite years of development and commercial push, virtual reality (VR) headsets have yet to supersede TVs or computer monitors as the preferred screen for viewing moving images. One reason could be that VR can make users feel sick. Nausea and eye strain can result because VR creates an illusion of 3D viewing although the user is in fact staring at a fixed-distance 2D display – an issue referred to as vergence-accommodation conflict (see HN March 2021).

The solution for better 3D visualisation could lie in remaking holograms for the digital age – which would allow people to see and sense images in a way that is most realistic, comfortable and natural. Holograms offer a shifting perspective of the recorded world based on the viewer’s position, and they allow the eye to adjust focal depth to alternately focus on foreground and background.

Researchers have long sought to make computer-generated holograms, but the process has traditionally required a supercomputer to churn through mathematical simulations, which is time-consuming and can yield less-than-photorealistic results. Now, MIT (Massachusetts Institute of Technology) researchers have developed a new way to produce holograms almost instantly — and the deep learning-based method is so efficient that it can run on a laptop in almost real-time, the researchers say.

‘People previously thought that with existing consumer-grade hardware, it was impossible to do real-time 3D holography computations,’ says Liang Shi, the study’s lead author and a PhD student in MIT’s Department of Electrical Engineering and Computer Science (EECS). ‘It’s often been said that commercially available holographic displays will be around in 10 years, yet this statement has been around for decades.’

Shi believes the new approach, which the team calls ‘tensor holography,’ will finally bring that elusive 10-year goal within reach. The advance could fuel a spillover of holography into fields like VR and 3D printing.

Shi worked on the study, published in Nature, with his advisor and co-author Wojciech Matusik. Other co-authors include Beichen Li of EECS and the Computer Science and Artificial Intelligence Laboratory at MIT, as well as former MIT researchers Changil Kim (now at Facebook) and Petr Kellnhofer (now at Stanford University).

The quest for better 3D

A typical lens-based photograph encodes the brightness of each light wave – a photo can faithfully reproduce a scene’s colours, but it ultimately yields a flat image. In contrast, a hologram encodes both the brightness and phase of each light wave.

Anyone reading this who has recorded an optical hologram will appreciate the technical challenges involved, whereas computer-generated holography sidesteps these challenges by simulating the optical setup. But the process can be a computational slog.

‘Because each point in the scene has a different depth, you can’t apply the same operations for all of them,’ says Shi. ‘That increases the complexity significantly.’

Directing a clustered supercomputer to run these physics-based simulations could take seconds or minutes for a single holographic image. Plus, existing algorithms don’t model occlusion with photorealistic precision. Occlusion is a strong visual depth cue: objects closer to the viewer block, or occlude, objects that are more distant along the line of sight. So, Shi’s team took a different approach: letting the computer teach physics to itself.

They used deep learning to accelerate computer-generated holography, allowing for real-time hologram generation. The team designed a convolutional neural network – a processing technique that uses an array of functions that can be trained to change (a tensor) so that they can roughly mimic how humans process visual information. Training a neural network typically requires a large, high-quality dataset, which didn’t previously exist for 3D holograms.

The team built a custom database of 4,000 pairs of computer-generated images. Each pair matched a picture – including colour and depth information for each pixel – with its corresponding hologram.

To create the holograms in the new database, the researchers used scenes with complex and variable shapes and colours, with the depth of pixels distributed evenly from the background to the foreground, and with a new set of physics-based calculations to handle occlusion. That approach resulted in photorealistic training data.

Next, the algorithm got to work.

By learning from each image pair, the tensor network tweaked the parameters of its own calculations, successively enhancing its ability to create holograms. The fully optimised network operated orders of magnitude faster than physics-based calculations. That efficiency surprised the team itself.

‘We are amazed at how well it performs,’ says Matusik. Within milliseconds, tensor holography can craft holograms from images with depth information – which is provided by typical computer-generated images and can be calculated from a multicamera setup or LiDAR sensor (both are standard on some new smartphones).

This advance paves the way for real-time 3D holography. What’s more, the compact tensor network requires less than 1 MB of memory. ‘It’s negligible, considering the tens of gigabytes available on the latest cell phone’, he says.

A considerable leap

Real-time 3D holography would enhance a slew of systems, from VR to 3D printing. The team says the new system could help immerse VR viewers in more realistic scenery, while eliminating eye strain and other side effects of long-term VR use. The technology could be deployed on displays that modulate the phase of light waves. Currently, most affordable consumer-grade displays modulate only brightness, though the cost of phase-modulating displays would fall if widely adopted.

Three-dimensional holography could also boost the development of volumetric 3D printing, the researchers say. This technology could prove faster and more precise than traditional layer-by-layer 3D printing, since volumetric 3D printing allows for the simultaneous projection of the entire 3D pattern.

Other applications include microscopy, visualisation of medical data, and the design of surfaces with unique optical properties.

Also in this issue:

  • Is Medical Holography a Market, or a Group of Applications?
  • Apple’s Patent Interest in Holography
  • News in Brief
  • New Photochromic Materials Could Have Applications in Holography
  • Writing in 3D 
  • A Review of SPIE Practical Holography XXXV: Displays, Materials and Applications
  • Diffracting and Reflecting Light – A Potent Combination for Protecting Banknotes
  • Holograms Help to Study Plankton and Ecology

Other articles