Engineers at Rice University (Houston, Texas, USA) have developed a new three-dimensional (3-D) camera technology, aimed at collecting the spatial and spectral information required for applications, such as surface wear and corrosion detection.
Known as the Hyperspectral Stripe Projector (HSP),1 the compact technology is being developed in the lab of Kevin Kelly, an associate professor of electrical and computer engineering. According to Kelly, the lab could enable 3-D spectroscopy on the fly with a system that combines the HSP, a monochrome sensor array, and sophisticated programming to give users a more complete picture of an object’s shape and composition.
“We’re getting four-dimensional information from an image, three spatial and one spectral, in real time,” Kelly says. “Other people use multiple modulators and, thus, require bright light sources to accomplish this, but we found we could do it with a light source of normal brightness and some clever optics.”
Portable 3-D Imaging
According to the researchers, HSP takes a cue from portable 3-D imaging techniques that are already in consumers’ hands—like facial identification systems in smartphones and body trackers in gaming systems—and adds a way to pull broad spectral data from every pixel captured. This compressed data is reconstructed into a 3-D map, with spectral information that can incorporate hundreds of colors and be used to reveal, not only the shape of an object, but also its material composition.
“Regular RGB [red, green, blue] cameras basically give you only three spectral channels,” says Yibo Xu, a Rice alumna who collaborated on the project. “But a hyperspectral camera gives us spectra in many, many channels. We can capture red at around 700 nm and blue at around 400 nm, but we can also have bandwidths at every few nanometers or less between. That gives us fine spectral resolution and a fuller understanding of the scene.”
“HSP simultaneously encodes the depth and hyperspectral measurements in a very simple and efficient way, allowing the use of a monochrome camera instead of an expensive hyperspectral camera as typically used in similar systems,” adds Xu, who earned her doctorate in 2019 and is now a machine learning and computer vision research engineer at Samsung Research America Inc. (Mountain View, California, USA). She developed both the hardware and reconstruction software as part of her thesis in Kelly’s lab.
Digital Micromirror Device
As the team explains it, HSP uses an off-the-shelf digital micromirror device (DMD) to project patterned stripes that look something like colorful bar codes onto a surface. Sending the white-light projection through a diffraction grating separates the overlapping patterns into colors. Each color is reflected back to the monochrome camera, which assigns a numerical gray level to that pixel.
Each pixel can have multiple levels, one for every color stripe it reflects. These are recombined into an overall spectral value for that part of the object.
“We use a single DMD and a single grating in HSP,” Xu says. “The novel optical design of folding the light path back to the same diffraction grating and lens is what makes it really compact. The single DMD allows us to keep the light we want and throw away the rest.”
These finely tuned spectra can reach beyond visible light, the engineers explain, since what they reflect back to the sensor as multiplexed fine-band spectra can be used to identify the material’s chemical composition.
At the same time, distortions in the pattern are reconstructed into 3-D point clouds. Essentially, this is a picture of the target, but with much more data than a plain snapshot could provide.
Future System Developments
Kelly envisions HSP built into car headlights that can see the difference between an object and a person. “It could never get confused between a green dress and a green plant, because everything has its own spectral signature,” he says.
He believes the lab will eventually incorporate ideas from Rice’s groundbreaking single-pixel camera to further reduce the size of the device and adapt it for compressive video capture, as well.
The research was funded by an investment from the U.S. National Science Foundation (Alexandria, Virginia, USA).
Reference
1 “3D Camera Earns Its Stripes at Rice,” Rice University News and Media Relations, Sept. 24, 2020, https://news.rice.edu/2020/09/24/3d-camera-earns-its-stripes-at-rice/ (Oct. 19, 2020).