Introducing the spectroscopic analysis tool detector dog. This is an AI-enhanced sensor that can “sniff and find” target objects in real-time.
Spectral imaging tools (cameras that capture colors beyond the RGB spectrum visible to our eyes) are essential for gathering information about the material and structural properties of objects. Combined with machine learning, these provide a powerful pipeline for identifying features for real-world applications such as semiconductor manufacturing, contaminant tracking, and crop monitoring. Researchers at the Department of Energy’s Lawrence Berkeley National Laboratory (Berkeley Lab) have eliminated data processing bottlenecks that have long plagued the performance of spectral imaging techniques by incorporating AI algorithms into the camera’s sensors themselves. The result is an intelligent sensor that can identify chemicals and characterize materials quickly and efficiently.
“We focused on improving the speed, resolution, and power efficiency of existing spectral machine vision technologies by more than two orders of magnitude,” said Ali Javey, the scientist who led the research. science Survey reporting devices. Javey is a senior faculty member at Berkeley Lab and a professor of materials science and engineering at the University of California, Berkeley. This research was conducted in close collaboration with Aydogan Ozcan at UCLA.
“We focused on increasing the speed, resolution, and power efficiency of existing spectral machine vision technologies by more than two orders of magnitude.”
– Ali Jabey
This sensor design shows how new capabilities can be built into the semiconductor device itself to improve efficiency and utility, enabling a new class of AI vision hardware.
Building algorithms using light
Today’s spectral imaging technologies include separate sensor modules and computational modules. The sensor first captures a stack of images, each corresponding to a specific color. The dense image stack is then sent to a digital processor for further calculations and produces an object identification result. That’s where the problem arises.
“The sensor must collect and send much more data to the digital processor than a typical camera, about 10 to 100 times more,” said Dehui Zhang, a postdoctoral fellow in Berkeley Lab’s Department of Materials Science and lead author of the study. As a result, sensor and computer hardware are often overloaded, making object recognition tasks very slow and power-consuming.
Instead, the Berkeley Lab team developed a sensor that performs AI calculations and spectral analysis during the image capture (or light detection) process itself.
“Light detection can be seen as an automatic physical calculation process,” Zhang explains. When light hits the sensor, its intensity is automatically mapped to the strength of an electrical current. The sensor’s responsiveness to light can be easily adjusted, giving researchers a tuning knob to choose which spectral features to emphasize and which to suppress. The current leaving the sensor and read by the circuit therefore serves as an inference about the spectral content of the image.
“We demonstrated that the calculation process is mathematically similar to algorithms typically used for digital machine learning,” Zhang said. This similarity made it possible to use the sensor as a machine learning computer, performing machine learning calculations on the incident light itself.
train the machine
An AI or machine vision model must first learn what to identify. This means “showing” enough examples of the spectral signature of interest. For example, infrared patterns from real and artificial leaves. Or pixels in an image that belong to tree bark with a similar color to bird feathers — meaning you can find these features in an untried test case.
In the training step, the researchers showed the sensor dozens of images of colorful birds surrounded by forest. Rather than inspecting every pixel in each image, the sensor “sniffed” randomly selected pixels, each labeled as belonging to the bird or unwanted background. An external computer sent electrical signals commanding the sensor to “identify the bird” or “identify the background,” and recorded the sensor’s output for each command. The software then determined the best combination of commands to teach the sensor to highlight the bird’s area while suppressing everything else.
The test step used a combination of commands developed during training to show the sensor a new image and tell it to find the bird. The sensor gave a positive output signal only for pixels belonging to birds. This result means that the sensor has learned from the example to identify the target object even if it belongs to an image it has never seen before.
“The most exciting part for me is the concept of giving intelligence to sensors,” Jabee says. Regular sensors only collect raw environmental information, leaving the intelligent perception tasks to digital processors.
By co-designing semiconductor materials, devices, and algorithms, the team enabled the sensor to learn and compute without the need for digital post-processing of the data.
But the applications of this technology go far beyond bird identification. Researchers have experimentally demonstrated several other interesting possibilities using black phosphorus photodiodes, which can detect mid-infrared light with tunable responsivity. They were able to determine the thickness of an oxide layer in a semiconductor sample that a major manufacturing company needed to be perfectly uniform, as well as the hydration state of leaves of various plants, segmenting objects in optical images, and identifying transparent chemicals in petri dishes.
“I’m also optimistic about the future of devices like this being used in broader applications,” Jabee said. In the future, smart sensors could find applications not only in spectral machine vision, but also “other advanced optical sensing and beyond.”
This research was funded by the U.S. Department of Energy’s Office of Basic Energy Sciences. This research received support from DOE’s Microelectronics Energy Efficiency Research Center for Advanced Technologies, one of DOE’s three Microelectronics Science Research Centers.
For information about licensing this technology, please contact the University of California, Berkeley.
###
Lawrence Berkeley National Laboratory (Berkeley Lab) is engaged in groundbreaking research focused on discovery science and solutions for an abundant and reliable energy supply. The laboratory’s expertise spans materials, chemistry, physics, biology, earth and environmental sciences, mathematics, and computing. Researchers from around the world utilize the Institute’s world-class scientific facilities to conduct their own pioneering research. Founded in 1931 on the belief that the biggest problems are best tackled by teams, Berkeley Lab and its scientists have won 17 Nobel Prizes. Berkeley Lab is a multiprogram national laboratory managed by the University of California for the U.S. Department of Energy’s Office of Science.
The DOE Office of Science is the largest supporter of basic research in the physical sciences in the United States, working to address some of the most pressing challenges of our time. For more information, visit energy.gov/science.
