Long term facial parts tracking in thermal imaging for uncooperative emotion recognition

Author(s):  
Eslam Mostafa ◽  
Aly Farag ◽  
Ahmed Shalaby ◽  
Asem Ali ◽  
Travis Gault ◽  
...  
2021 ◽  
Author(s):  
◽  
Matt Cryer

<p>Colloidal semiconductor nanocrystals (NCs) with bandgaps less than 1 eV allow the development of mid wave infrared (MIR) sensitive detectors that exploit the benefits of colloidal materials, primarily bandgap selection and solution deposition. Additionally, the electrical behaviour of these films can be examined for characteristics that can increase the functionality of NC based detectors.  The production of devices that are designed to be competitive as ultra-low-cost, room temperature MIR detectors, operating with photonic, rather than thermal detection is detailed. The evolution of the colloidal synthesis, spray deposition methods, substrate materials and post deposition treatments used here lead to highly robust and high performing devices. These devices demonstrate a “colour” sensitivity down to 300 nm in the MIR (≈10 % of scale), with superior responsivities for this class of device, up to 0.9 AW⁻¹, and competitive specific detectivity up to 8 × 10⁹ Jones at 200 Hz and 300 K. Furthermore, these devices utilise a cheap and robust substrate material that allows operation after deformation up to 45 ° without degradation over many cycles. These devices offer a template for ultra-low-cost MIR detectors with performance that rivals microbolometers but with better measurement speed and spectral sensitivity. As such these devices showcase the key advantages of using colloidal NCs in MIR applications.  Planar and fully air processed thin film devices that demonstrate photo-induced memristive behaviour and can be used as a transistors, photode-tectors or memory devices are investigated. Following long term (60 h) air exposure, unpackaged NC films develop reliable memristive characteristics in tandem with temperature, gate and photoresponse. On/off ratios of more than 50 are achieved and the devices show long term stability, producing repeatable metrics over days of measurement. The on/off behaviour is shown to be dependent on previous charge flow and carrier density, implying memristive rather than switching behaviour. These observations are described within a long term trap filling model. This work represents an advance in the integration of NC films into electronic devices, which may lead to the development of multi-functional electronic components.  Building on the previous work the steps taken to move from a planar device, that works well in controlled conditions, to a multi-pixel sensor that can demonstrate MIR video imaging at room temperature in a noisy environment are shown. This is achieved with a 15 pixel detector that consists only of a polymer substrate and solution patterned NC pixels. This device can detect a 373 K object with the device at 298 K in a noisy environment. This performance is enabled by photogain at 5 V bias that reaches a maximum External Quantum Efficiency (EQE) of 1940 ± 290 % for a pixel with a 3.3 µm bandgap. Through the use of four separate bandgaps it is shown that “multicolour” thermal imaging systems can deliver another layer of information, on top of intensity, to the user. The behaviour of the system is examined under use and it is shown that the photoconductive device behaves as expected with regards to bias, and that trap enabled gain is sensitive to total incident flux, more than the spectral energy distribution of the target. Finally, it is shown that solution patterned QD fabrication methods can deliver electrical reproducibility between pixels that is sufficient to allow an imaging plane of multiple pixels.  The somewhat neglected tin chalcogenide semiconductor nanocrystals are investigated and inverse MIR detection at room temperature is demonstrated with planar, solution and airprocessed PbSnTe and SnTe QD devices. The detection mechanism is shown to be mediated by an interaction between MIR radiation and the vibrational stretches of adsorbed hydroxyl species at the oxdised NC surface. Devices are shown to possess mAW⁻¹ responsivity via a reduction in film conductance due to MIR radiation and, unlike classic MIR photoconductors, are unaffected by visible wavelengths. As such these devices offer the possibility of MIR thermal imaging that has an intrinsic solution to the blinding caused by higher energy light sources.  In summary, it is shown that semiconductor NCs with an all ambient fully solution processed deposition and ligand exchange procedure can be used to create simple, robust and cheap devices that are beginning to demonstrate metrics on par with current commercial thermal detector systems. It is also shown that these devices can under certain circumstances demonstrate novel behaviours that offer the prospects of enhanced or novel functionality.</p>


2021 ◽  
Author(s):  
◽  
Matt Cryer

<p>Colloidal semiconductor nanocrystals (NCs) with bandgaps less than 1 eV allow the development of mid wave infrared (MIR) sensitive detectors that exploit the benefits of colloidal materials, primarily bandgap selection and solution deposition. Additionally, the electrical behaviour of these films can be examined for characteristics that can increase the functionality of NC based detectors.  The production of devices that are designed to be competitive as ultra-low-cost, room temperature MIR detectors, operating with photonic, rather than thermal detection is detailed. The evolution of the colloidal synthesis, spray deposition methods, substrate materials and post deposition treatments used here lead to highly robust and high performing devices. These devices demonstrate a “colour” sensitivity down to 300 nm in the MIR (≈10 % of scale), with superior responsivities for this class of device, up to 0.9 AW⁻¹, and competitive specific detectivity up to 8 × 10⁹ Jones at 200 Hz and 300 K. Furthermore, these devices utilise a cheap and robust substrate material that allows operation after deformation up to 45 ° without degradation over many cycles. These devices offer a template for ultra-low-cost MIR detectors with performance that rivals microbolometers but with better measurement speed and spectral sensitivity. As such these devices showcase the key advantages of using colloidal NCs in MIR applications.  Planar and fully air processed thin film devices that demonstrate photo-induced memristive behaviour and can be used as a transistors, photode-tectors or memory devices are investigated. Following long term (60 h) air exposure, unpackaged NC films develop reliable memristive characteristics in tandem with temperature, gate and photoresponse. On/off ratios of more than 50 are achieved and the devices show long term stability, producing repeatable metrics over days of measurement. The on/off behaviour is shown to be dependent on previous charge flow and carrier density, implying memristive rather than switching behaviour. These observations are described within a long term trap filling model. This work represents an advance in the integration of NC films into electronic devices, which may lead to the development of multi-functional electronic components.  Building on the previous work the steps taken to move from a planar device, that works well in controlled conditions, to a multi-pixel sensor that can demonstrate MIR video imaging at room temperature in a noisy environment are shown. This is achieved with a 15 pixel detector that consists only of a polymer substrate and solution patterned NC pixels. This device can detect a 373 K object with the device at 298 K in a noisy environment. This performance is enabled by photogain at 5 V bias that reaches a maximum External Quantum Efficiency (EQE) of 1940 ± 290 % for a pixel with a 3.3 µm bandgap. Through the use of four separate bandgaps it is shown that “multicolour” thermal imaging systems can deliver another layer of information, on top of intensity, to the user. The behaviour of the system is examined under use and it is shown that the photoconductive device behaves as expected with regards to bias, and that trap enabled gain is sensitive to total incident flux, more than the spectral energy distribution of the target. Finally, it is shown that solution patterned QD fabrication methods can deliver electrical reproducibility between pixels that is sufficient to allow an imaging plane of multiple pixels.  The somewhat neglected tin chalcogenide semiconductor nanocrystals are investigated and inverse MIR detection at room temperature is demonstrated with planar, solution and airprocessed PbSnTe and SnTe QD devices. The detection mechanism is shown to be mediated by an interaction between MIR radiation and the vibrational stretches of adsorbed hydroxyl species at the oxdised NC surface. Devices are shown to possess mAW⁻¹ responsivity via a reduction in film conductance due to MIR radiation and, unlike classic MIR photoconductors, are unaffected by visible wavelengths. As such these devices offer the possibility of MIR thermal imaging that has an intrinsic solution to the blinding caused by higher energy light sources.  In summary, it is shown that semiconductor NCs with an all ambient fully solution processed deposition and ligand exchange procedure can be used to create simple, robust and cheap devices that are beginning to demonstrate metrics on par with current commercial thermal detector systems. It is also shown that these devices can under certain circumstances demonstrate novel behaviours that offer the prospects of enhanced or novel functionality.</p>


2013 ◽  
Vol 2013 ◽  
pp. 1-10 ◽  
Author(s):  
I. Bosch ◽  
A. Serrano ◽  
L. Vergara

This paper presents the next step in the evolution of multi-sensor wireless network systems in the early automatic detection of forest fires. This network allows remote monitoring of each of the locations as well as communication between each of the sensors and with the control stations. The result is an increased coverage area, with quicker and safer responses. To determine the presence of a forest wildfire, the system employs decision fusion in thermal imaging, which can exploit various expected characteristics of a real fire, including short-term persistence and long-term increases over time. Results from testing in the laboratory and in a real environment are presented to authenticate and verify the accuracy of the operation of the proposed system. The system performance is gauged by the number of alarms and the time to the first alarm (corresponding to a real fire), for different probability of false alarm (PFA). The necessity of including decision fusion is thereby demonstrated.


2019 ◽  
Vol 67 (3) ◽  
pp. 180
Author(s):  
Gregory S. Watson ◽  
David W. Green ◽  
Jolanta A. Watson

Most reptiles exhibit no parental care and aggressive behaviour towards heterospecific predators has rarely been recorded in the natural environment. Several species of the subfamily Egerniinae are amongst the most highly social of all squamate reptiles, exhibiting stable social aggregations and high levels of long-term social and genetic monogamy. We have examined Cunningham’s skinks, Egernia cunninghami, over a three-year period during late January and early February (total 32 days) in the alpine region of New South Wales using video and thermal imaging. Four birthing sessions were witnessed during our field studies of social aggregations of skinks. Our observations monitored skink encounters, in the presence of offspring, with an eastern brown snake, Pseudonaja textilis (two separate encounters, one recorded by video/imaging) and 12 encounters with the Australian magpie, Gymnorhina tibicen. All events were associated with aggressive chasing and/or attack by adult skinks. The first snake encounter involved the active targeting of a recently born juvenile with the mother of the juvenile attacking the snake (running towards the snake, biting and remaining attached for several seconds). The second encounter (the following year) comprised two adult skinks attacking and biting a snake, Pseudonaja textilis. All magpie encounters resulted in chases by adult skinks.


2009 ◽  
Author(s):  
Pierre Dumouchel ◽  
Najim Dehak ◽  
Yazid Attabi ◽  
Réda Dehak ◽  
Narjès Boufaden
Keyword(s):  

2016 ◽  
Vol 228-229 ◽  
pp. 315-326 ◽  
Author(s):  
Donald M. Aubrecht ◽  
Brent R. Helliker ◽  
Michael L. Goulden ◽  
Dar A. Roberts ◽  
Christopher J. Still ◽  
...  

Sensors ◽  
2021 ◽  
Vol 21 (7) ◽  
pp. 2344
Author(s):  
Nhu-Tai Do ◽  
Soo-Hyung Kim ◽  
Hyung-Jeong Yang ◽  
Guee-Sang Lee ◽  
Soonja Yeom

Emotion recognition plays an important role in human–computer interactions. Recent studies have focused on video emotion recognition in the wild and have run into difficulties related to occlusion, illumination, complex behavior over time, and auditory cues. State-of-the-art methods use multiple modalities, such as frame-level, spatiotemporal, and audio approaches. However, such methods have difficulties in exploiting long-term dependencies in temporal information, capturing contextual information, and integrating multi-modal information. In this paper, we introduce a multi-modal flexible system for video-based emotion recognition in the wild. Our system tracks and votes on significant faces corresponding to persons of interest in a video to classify seven basic emotions. The key contribution of this study is that it proposes the use of face feature extraction with context-aware and statistical information for emotion recognition. We also build two model architectures to effectively exploit long-term dependencies in temporal information with a temporal-pyramid model and a spatiotemporal model with “Conv2D+LSTM+3DCNN+Classify” architecture. Finally, we propose the best selection ensemble to improve the accuracy of multi-modal fusion. The best selection ensemble selects the best combination from spatiotemporal and temporal-pyramid models to achieve the best accuracy for classifying the seven basic emotions. In our experiment, we take benchmark measurement on the AFEW dataset with high accuracy.


Sign in / Sign up

Export Citation Format

Share Document