Compact meta-spectral image sensor for mobile applications

We have demonstrated a compact and efficient metasurface-based spectral imager for use in the near-infrared range. The spectral imager was created by fabricating dielectric multilayer filters directly on top of the CMOS image sensor. The transmission wavelength for each spectral channel was selected by embedding a Si nanopost array of appropriate dimensions within the multilayers on the corresponding pixels, and this greatly simplified the fabrication process by avoiding the variation of the multilayer-film thicknesses. The meta-spectral imager shows high efficiency and excellent spectral resolution up to 2.0 nm in the near-infrared region. Using the spectral imager, we were able to measure the broad spectra of LED emission and obtain hyperspectral images from wavelength-mixed images. This approach provides ease of fabrication, miniaturization, low crosstalk, high spectral resolution, and high transmission. Our findings can potentially be used in integrating a compact spectral imager in smartphones for diverse applications.

In-sensor optoelectronic computing using electrostatically doped silicon

Complementary metal-oxide-semiconductor (CMOS) image sensors are a visual outpost of many machines that interact with the world. While they presently separate image capture in front-end silicon photodiode arrays from image processing in digital back-ends, efforts to process images within the photodiode array itself are rapidly emerging, in hopes of minimizing the data transfer between sensing and computing, and the associated overhead in energy and bandwidth. Electrical modulation, or programming, of photocurrents is requisite for such in-sensor computing, which was indeed demonstrated with electrostatically doped, but non-silicon, photodiodes. CMOS image sensors are currently incapable of in-sensor computing, as their chemically doped photodiodes cannot produce electrically tunable photocurrents. Here we report in-sensor computing with an array of electrostatically doped silicon p-i-n photodiodes, which is amenable to seamless integration with the rest of the CMOS image sensor electronics. This silicon-based approach could more rapidly bring in-sensor computing to the real world due to its compatibility with the mainstream CMOS electronics industry. Our wafer-scale production of thousands of silicon photodiodes using standard fabrication emphasizes this compatibility. We then demonstrate in-sensor processing of optical images using a variety of convolutional filters electrically programmed into a 3 × 3 network of these photodiodes.

A Novel, Automated, and Real-Time Method for the Analysis of Non-Human Primate Behavioral Patterns Using a Depth Image Sensor

By virtue of their upright locomotion, similar to that of humans, motion analysis of non-human primates has been widely used in order to better understand musculoskeletal biomechanics and neuroscience problems. Given the difficulty of conducting a marker-based infrared optical tracking system for the behavior analysis of primates, a 2-dimensional (D) video analysis has been applied. Distinct from a conventional marker-based optical tracking system, a depth image sensor system provides 3-D information on movement without any skin markers. The specific aim of this study was to develop a novel algorithm to analyze the behavioral patterns of non-human primates in a home cage using a depth image sensor. The behavioral patterns of nine monkeys in their home cage, including sitting, standing, and pacing, were captured using a depth image sensor. Thereafter, these were analyzed by observers’ manual assessment and the newly written automated program. We confirmed that the measurement results from the observers’ manual assessments and the automated program with depth image analysis were statistically identical.