Automatic, Illumination-Invariant and Real-Time Green-Screen Keying Using Deeply Guided Linear Models

The conventional green screen keying method requires users’ interaction to guide the whole process and usually assumes a well-controlled illumination environment. In the era of “we-media”, millions of short videos are shared online every day, and most of them are produced by amateurs in relatively poor conditions. As a result, a fully automatic, real-time, and illumination-robust keying method would be very helpful and commercially promising in this era. In this paper, we propose a linear model guided by deep learning prediction to solve this problem. The simple, yet effective algorithm inherits the robustness of the deep-learning-based segmentation method, as well as the high matting quality of energy-minimization-based matting algorithms. Furthermore, thanks to the introduction of linear models, the proposed minimization problem is much less complex, and thus, real-time green screen keying is achieved. In the experiment, our algorithm achieved comparable keying performance to the manual keying software and deep-learning-based methods while beating other shallow matting algorithms in terms of accuracy. As for the matting speed and robustness, which are critical for a practical matting system, the proposed method significantly outperformed all the compared methods and showed superiority over all the off-the-self approaches.

Visual Performance under Varying Illumination Conditions while using an Indigenously Developed Wrist Wearable Computer

Ambient illumination conditions have significant impact on users’ visual performance while carrying out onscreen reading tasks on visual display units, especially smaller screen sizes. Present study assessed the visual performance responses of different ambient illumination levels during onscreen reading on Wrist Wearable Computer (WWC) developed for the command-control-communication between the control room and the soldiers operating in remote locations. Ten (10) Indian Infantry soldiers performed two different types of loud reading tasks on the display of WWC under three different ambient illumination (mean ±SEM) conditions namely, Indoor controlled (450.00±10.00 lx), Outdoor daylight (11818.7±582.91 lx) and Indoor dark (0.12±0.03 lx) environments. While reading, participants wore an eye tracking glass which recorded the eye movement responses. Visualisation techniques were used to predict the association of illumination levelof surrounding with visual performance of the user. Subjective legibility rating was also applied to understand participants’ preferences towards physical attributes of the onscreen information and illumination level. Results indicated that illumination had a significant effect on eye movement parameters like fixation frequency, fixation duration and scanpath length while completing the tasks. Overall, participants performed better under indoor controlled illumination conditions in terms of fixation profile and scanpath length, apart from improved subjective legibility ratings as compared to other two illumination conditions. Future research attempts need to be directed towards the optimum performance of the display across wide range of ambient illumination conditions and to establish how the display of indigenously developed wearable computer performs in comparison to other such displays available across the globe.

OpenEDS2020 Challenge on Gaze Tracking for VR: Dataset and Results

This paper summarizes the OpenEDS 2020 Challenge dataset, the proposed baselines, and results obtained by the top three winners of each competition: (1) Gaze prediction Challenge, with the goal of predicting the gaze vector 1 to 5 frames into the future based on a sequence of previous eye images, and (2) Sparse Temporal Semantic Segmentation Challenge, with the goal of using temporal information to propagate semantic eye labels to contiguous eye image frames. Both competitions were based on the OpenEDS2020 dataset, a novel dataset of eye-image sequences captured at a frame rate of 100 Hz under controlled illumination, using a virtual-reality head-mounted display with two synchronized eye-facing cameras. The dataset, which we make publicly available for the research community, consists of 87 subjects performing several gaze-elicited tasks, and is divided into 2 subsets, one for each competition task. The proposed baselines, based on deep learning approaches, obtained an average angular error of 5.37 degrees for gaze prediction, and a mean intersection over union score (mIoU) of 84.1% for semantic segmentation. The winning solutions were able to outperform the baselines, obtaining up to 3.17 degrees for the former task and 95.2% mIoU for the latter.

I spy with my hyperspectral eye: unique reflectance database of plastics and riverbank-harvested litter

<p><span>Airborne and spaceborne remote sensing (RS) collecting hyperspectral imagery provides a promising way forward for detection and monitoring of riverine and marine plastic pollution. However, a major challenge in the application of RS techniques is the lack of fundamental understanding of spectral signatures of floating plastic debris at multiple scales. Recent work emphasised the case for open-access hyperspectral reflectance reference libraries of commonly used polymer items. In this paper, we present a high-resolution hyperspectral image database of a unique mix of (i) 40 virgin macroplastic items, (ii) organic material of plant leaves, tree leaves and riparian vegetation, and (iii) 50 items of riverbank-harvested macrolitter including plastics and other anthropogenic debris. We used a double camera setup that covered the VIS-SWIR range from 400-1700 nm in a dark room experiment with controlled illumination. The cameras scanned the samples floating in water and captured high-resolution images in 336 spectral bands. From these images we identified diagnostic absorption features for different materials, item categories, and plastic polymers. The identification was done by applying a linear discriminant analysis to the spectra, allowing the creation of combined band indices distinguishing between the different item types and polymer categories. We present reflectance spectra of all items in our image dataset, complemented by easy-to-interpret visual representations of derived indices. We demonstrate the importance of high-resolution reference reflectance libraries, to (i) further optimise existing remote sensing monitoring techniques, and (ii) contribute towards the development of future plastic monitoring and classification missions.</span></p>

Linking Tissue Damage to Hyperspectral Reflectance for Non-Invasive Monitoring of Apple Fruit in Orchards

Reflected light carries ample information about the biochemical composition, tissue architecture, and physiological condition of plants. Recent technical progress has paved the way for affordable imaging hyperspectrometers (IH) providing spatially resolved spectral information on plants on different levels, from individual plant organs to communities. The extraction of sensible information from hyperspectral images is difficult due to inherent complexity of plant tissue and canopy optics, especially when recorded under ambient sunlight. We report on the changes in hyperspectral reflectance accompanying the accumulation of anthocyanins in healthy apple (cultivars Ligol, Gala, Golden Delicious) fruits as well as in fruits affected by pigment breakdown during sunscald development and phytopathogen attacks. The measurements made outdoors with a snapshot IH were compared with traditional “point-type” reflectance measured with a spectrophotometer under controlled illumination conditions. The spectra captured by the IH were suitable for processing using the approaches previously developed for “point-type” apple fruit and leaf reflectance spectra. The validity of this approach was tested by constructing a novel index mBRI (modified browning reflectance index) for detection of tissue damages on the background of the anthocyanin absorption. The index was suggested in the form of mBRI = (R640−1 + R800−1) − R678−1. Difficulties of the interpretation of fruit hyperspectral reflectance images recorded in situ are discussed with possible implications for plant physiology and precision horticulture practices.

Linking Tissue Damage to Hyperspectral Reflectance for Non-Invasive Monitoring of Apple Fruit in Orchards

Reflected light carries ample information about biochemical composition, tissue architecture, and physiological condition of plants. Recent technical progress brought about affordable imaging hyperspectrometers (IH) providing spatially resolved spectral data on plants. The extraction of sensible information from hyperspectral reflectance images is difficult due to inherent complexity of plant tissue and canopy optics, especially when recorded by IH under ambient sunlight. We aimed at obtaining a deeper insight into plant optics as perceived by IH since there is a high demand for algorithms for fruit harvesting and grading systems equipped with computer vision and robotic systems capable of working in orchard. We report on the characteristic changes in hyperspectral reflectance accompanying the accumulation of anthocyanins in healthy fruit, pigment breakdown during sunscald and phytopathogen attacks. The measurements made outdoors with a snapshot IH were compared with traditional “point” reflectance measured with a conventional spectrophotometer under controlled illumination conditions. Most of the spectral features and patterns of plant reflectance were evident in the IH-derived reflectance images. As a step forward, a novel index for highlighting tissue damages on the background of the anthocyanin absorption, BRI-M = (1/Rorange – 1/Rred + 1/RNIR), is suggested. Difficulties of the interpretation of fruit hyperspectral reflectance images recorded in situ are discussed with possible implications for plant physiology and precision horticulture practices.