Mnemonic-Opto-Synaptic Transistor for In-Sensor Vision System

A mnemonic-opto-synaptic transistor (MOST) that has triple functions is demonstrated for an in-sensor vision system. It memorizes a photoresponsivity that corresponds to a synaptic weight as a memory cell, senses light as a photodetector, and performs weight updates as a synapse for machine vision with an artificial neural network (ANN). Herein the memory function added to a previous photodetecting device combined with a photodetector and a synapse provides a technical breakthrough for realizing in-sensor processing that is able to perform image sensing and signal processing in a sensor. A charge trap layer (CTL) was intercalated to gate dielectrics of a vertical pillar-shaped transistor for the memory function. Weight memorized in the CTL makes photoresponsivity tunable for real-time multiplication of the image with a memorized photoresponsivity matrix. Therefore, these multi-faceted features can allow in-sensor processing without external memory for the in-sensor vision system. In particular, the in-sensor vision system can enhance speed and energy efficiency compared to a conventional vision system due to the simultaneous preprocessing of massive data at sensor nodes prior to ANN nodes. Recognition of a simple pattern was demonstrated with full sets of the fabricated MOSTs. Furthermore, recognition of complex hand-written digits in the MNIST database was also demonstrated with software simulations.

Investigation of Timing Behavior and Jitter in a Smart Inertial Sensor Debugging Architecture

Due to upcoming higher integration levels of microprocessors, the market of inertial sensors has changed in the last few years. Smart inertial sensors are becoming more and more important. This type of sensor offers the benefit of implementing sensor-processing tasks directly on the sensor hardware. The software development on such sensors is quite challenging. In this article, we propose an approach for using prerecorded sensor data during the development process to test and evaluate the functionality and timing of the sensor firmware in a repeatable and reproducible way on the actual hardware. Our proposed Sensor-in-the-Loop architecture enables the developer to inject sensor data during the debugging process directly into the sensor hardware in real time. As the timing becomes more critical in future smart sensor applications, we investigate the timing behavior of our approach with respect to timing and jitter. The implemented approach can inject data of three 3-DOF sensors at 1.6 kHz. Furthermore, the jitter shown in our proposed sampling method is at least three times lower than using real sensor data. To prove the statistical significance of our experiments, we use a Gage R&R analysis, extended by the assessment of confidence intervals of our data.

Advanced sun-sensor processing and design for super-resolution performance

The performance of conventional and parametric super-resolution algorithms for estimating sun position in a spacecraft sun-sensor was analyzed. Widely employed in other applications, parametric algorithms were examined to evaluate increase in system performance without affecting the cost of the sensor system. Using a simplified model of detector illumination simulations provided quantitative comparisons of algorithm performance. Simple sensor re-design was examined by using genetic algorithms as a heuristic to optimize the illumination pattern for a single axis digital sun-sensor. Findings show that, multiple narrow peak patterns provide subpixel accuracy in resolving the sun-angle. The optimal illumination pattern can be implemented by fabricating a replacement aperture mask for the sensor and this change can be made at a minimal cost. The super-resolution algorithms were tested with a component noise model and image degradation due to Earth albedo effects were examined. Parametric algorithms display very good performance throughout the test regime. The improvements are substantial enough to validate this approach worthy of future study.

Advanced sun-sensor processing and design for super-resolution performance

The performance of conventional and parametric super-resolution algorithms for estimating sun position in a spacecraft sun-sensor was analyzed. Widely employed in other applications, parametric algorithms were examined to evaluate increase in system performance without affecting the cost of the sensor system. Using a simplified model of detector illumination simulations provided quantitative comparisons of algorithm performance. Simple sensor re-design was examined by using genetic algorithms as a heuristic to optimize the illumination pattern for a single axis digital sun-sensor. Findings show that, multiple narrow peak patterns provide subpixel accuracy in resolving the sun-angle. The optimal illumination pattern can be implemented by fabricating a replacement aperture mask for the sensor and this change can be made at a minimal cost. The super-resolution algorithms were tested with a component noise model and image degradation due to Earth albedo effects were examined. Parametric algorithms display very good performance throughout the test regime. The improvements are substantial enough to validate this approach worthy of future study.

Elk Audio OS

As the Internet of Musical Things (IoMusT) emerges, audio-specific operating systems (OSs) are required on embedded hardware to ease development and portability of IoMusT applications. Despite the increasing importance of IoMusT applications, in this article, we show that there is no OS able to fulfill the diverse requirements of IoMusT systems. To address such a gap, we propose the Elk Audio OS as a novel and open source OS in this space. It is a Linux-based OS optimized for ultra-low-latency and high-performance audio and sensor processing on embedded hardware, as well as for handling wireless connectivity to local and remote networks. Elk Audio OS uses the Xenomai real-time kernel extension, which makes it suitable for the most demanding of low-latency audio tasks. We provide the first comprehensive overview of Elk Audio OS, describing its architecture and the key components of interest to potential developers and users. We explain operational aspects like the configuration of the architecture and the control mechanisms of the internal sound engine, as well as the tools that enable an easier and faster development of connected musical devices. Finally, we discuss the implications of Elk Audio OS, including the development of an open source community around it.

Dynamic Temperature Management of Near-Sensor Processing for Energy-Efficient High-Fidelity Imaging

Vision processing on traditional architectures is inefficient due to energy-expensive off-chip data movement. Many researchers advocate pushing processing close to the sensor to substantially reduce data movement. However, continuous near-sensor processing raises sensor temperature, impairing imaging/vision fidelity. We characterize the thermal implications of using 3D stacked image sensors with near-sensor vision processing units. Our characterization reveals that near-sensor processing reduces system power but degrades image quality. For reasonable image fidelity, the sensor temperature needs to stay below a threshold, situationally determined by application needs. Fortunately, our characterization also identifies opportunities—unique to the needs of near-sensor processing—to regulate temperature based on dynamic visual task requirements and rapidly increase capture quality on demand. Based on our characterization, we propose and investigate two thermal management strategies—stop-capture-go and seasonal migration—for imaging-aware thermal management. For our evaluated tasks, our policies save up to 53% of system power with negligible performance impact and sustained image fidelity.