Artificial Eyes with Emotion and Light Responsive Pupils for Realistic Humanoid Robots

This study employs a novel 3D engineered robotic eye system with dielectric elastomer actuator (DEA) pupils and a 3D sculpted and colourised gelatin iris membrane to replicate the appearance and materiality of the human eye. A camera system for facial expression analysis (FEA) was installed in the left eye, and a photo-resistor for measuring light frequencies in the right. Unlike previous prototypes, this configuration permits the robotic eyes to respond to both light and emotion proximal to a human eye. A series of experiments were undertaken using a pupil tracking headset to monitor test subjects when observing positive and negative video stimuli. A second test measured pupil dilation ranges to high and low light frequencies using a high-powered artificial light. This data was converted into a series of algorithms for servomotor triangulation to control the photosensitive and emotive pupil dilation sequences. The robotic eyes were evaluated against the pupillometric data and video feeds of the human eyes to determine operational accuracy. Finally, the dilating robotic eye system was installed in a realistic humanoid robot (RHR) and comparatively evaluated in a human-robot interaction (HRI) experiment. The results of this study show that the robotic eyes can emulate the average pupil reflex of the human eye under typical light conditions and to positive and negative emotive stimuli. However, the results of the HRI experiment indicate that replicating natural eye contact behaviour was more significant than emulating pupil dilation.

PupilEXT: Flexible Open-Source Platform for High-Resolution Pupillometry in Vision Research

The human pupil behavior has gained increased attention due to the discovery of the intrinsically photosensitive retinal ganglion cells and the afferent pupil control path’s role as a biomarker for cognitive processes. Diameter changes in the range of 10–2 mm are of interest, requiring reliable and characterized measurement equipment to accurately detect neurocognitive effects on the pupil. Mostly commercial solutions are used as measurement devices in pupillometry which is associated with high investments. Moreover, commercial systems rely on closed software, restricting conclusions about the used pupil-tracking algorithms. Here, we developed an open-source pupillometry platform consisting of hardware and software competitive with high-end commercial stereo eye-tracking systems. Our goal was to make a professional remote pupil measurement pipeline for laboratory conditions accessible for everyone. This work’s core outcome is an integrated cross-platform (macOS, Windows and Linux) pupillometry software called PupilEXT, featuring a user-friendly graphical interface covering the relevant requirements of professional pupil response research. We offer a selection of six state-of-the-art open-source pupil detection algorithms (Starburst, Swirski, ExCuSe, ElSe, PuRe and PuReST) to perform the pupil measurement. A developed 120-fps pupillometry demo system was able to achieve a calibration accuracy of 0.003 mm and an averaged temporal pupil measurement detection accuracy of 0.0059 mm in stereo mode. The PupilEXT software has extended features in pupil detection, measurement validation, image acquisition, data acquisition, offline pupil measurement, camera calibration, stereo vision, data visualization and system independence, all combined in a single open-source interface, available at https://github.com/openPupil/Open-PupilEXT.

Low-Complexity Pupil Tracking for Sunglasses-Wearing Faces for Glasses-Free 3D HUDs

This study proposes a pupil-tracking method applicable to drivers both with and without sunglasses on, which has greater compatibility with augmented reality (AR) three-dimensional (3D) head-up displays (HUDs). Performing real-time pupil localization and tracking is complicated by drivers wearing facial accessories such as masks, caps, or sunglasses. The proposed method fulfills two key requirements: low complexity and algorithm performance. Our system assesses both bare and sunglasses-wearing faces by first classifying images according to these modes and then assigning the appropriate eye tracker. For bare faces with unobstructed eyes, we applied our previous regression-algorithm-based method that uses scale-invariant feature transform features. For eyes occluded by sunglasses, we propose an eye position estimation method: our eye tracker uses nonoccluded face area tracking and a supervised regression-based pupil position estimation method to locate pupil centers. Experiments showed that the proposed method achieved high accuracy and speed, with a precision error of <10 mm in <5 ms for bare and sunglasses-wearing faces for both a 2.5 GHz CPU and a commercial 2.0 GHz CPU vehicle-embedded system. Coupled with its performance, the low CPU consumption (10%) demonstrated by the proposed algorithm highlights its promise for implementation in AR 3D HUD systems.

Fast pupil tracking based on the study of a boundary-stepped image model and multidimensional optimization Hook-Jives method

A new fast method for pupil detection and eyetracking real time is being developed based on the study of a boundary-step model of a grayscale image by the Laplacian-Gaussian operator and finding a new proposed descriptor of accumulated differences (point identifier), which displays a measure of the equidistance of each point from the boundaries of some relative monotonous area (for example, the pupil of the eye). The operation of this descriptor is based on the assumption that the pupil in the frame is the most rounded monotonic region with a high brightness difference at the border, the pixels of the region should have an intensity less than a predetermined threshold (but the pupil may not be the darkest region in the image). Taking into account all of the above characteristics of the pupil, the descriptor allows achieving high detection accuracy of its center and size, in contrast to methods based on threshold image segmentation, based on the assumption of the pupil as the darkest area, morphological methods (recursive morphological erosion), correlation or methods that investigate only the boundary image model (Hough transform and its variations with two-dimensional and three-dimensional parameter spaces, the Starburst algorithm, Swirski, RANSAC, ElSe). The possibility of representing the pupil tracking problem as a multidimensional unconstrained optimization problem and its solution by the Hook-Jeeves non-gradient method, where the function expressing the descriptor is used as the objective function, is investigated. In this case, there is no need to calculate the descriptor for each point of the image (compiling a special accumulator function), which significantly speeds up the work of the method. The proposed descriptor and method were analyzed, and a software package was developed in Python 3 (visualization) and C ++ (tracking kernel) in the laboratory of the Physics and Mathematics Faculty of Kamchatka State University of Vitus Bering, which allows illustrating the work of the method and tracking the pupil in real time.

Detection of Parkinson's Disease Through Automated Pupil Tracking of the Post-illumination Pupillary Response

Parkinson's disease (PD) is one of the most common neurodegenerative disorders, but it is often diagnosed after the majority of dopaminergic cells are already damaged. It is critical to develop biomarkers to identify the disease as early as possible for early intervention. PD patients appear to have an altered pupillary response consistent with an abnormality in photoreceptive retinal ganglion cells. Tracking the pupil size manually is a tedious process and offline automated systems can be prone to errors that may require intervention; for this reason in this work we describe a system for pupil size estimation with a user interface to allow rapid adjustment of parameters and extraction of pupil parameters of interest for the present study. We implemented a user-friendly system designed for clinicians to automate the process of tracking the pupil diameter to measure the post-illumination pupillary response (PIPR), permit manual corrections when needed, and continue automation after correction. Tracking was automated using a Kalman filter estimating the pupil center and diameter over time. The resulting system was tested on a PD classification task in which PD subjects are known to have similar responses for two wavelengths of light. The pupillary response is measured in the contralateral eye to two different light stimuli (470 and 610 nm) for 19 PD and 10 control subjects. The measured Net PIPR indicating different responsiveness to the wavelengths was 0.13 mm for PD subjects and 0.61 mm for control subjects, demonstrating a highly significant difference (p &lt; 0.001). Net PIPR has the potential to be a biomarker for PD, suggesting further study to determine clinical validity.

Discordant phenotypes in twins with infantile nystagmus

Infantile nystagmus (IN) may result from aetiologies including albinism and FRMD7 mutations. IN has low prevalence, and twins with IN are rare. Whilst discordant presentation has been previously reported for IN, we present for the first time the comprehensive assessment of diagnostically discordant monozygotic twins. From a cohort of over 2000 patients, we identified twins and triplets discordant for nystagmus. Using next-generation sequencing, high-resolution infra-red pupil tracking and optical coherence tomography, we characterised differences in genotype and phenotype. Monozygotic twins (n = 1), dizygotic twins (n = 3) and triplets (n = 1) were included. The monozygotic twins had concordant TYR variants. No causative variants were identified in the triplets. Dizygotic twins had discordant variants in TYR, OCA2 and FRMD7. One unaffected co-twin demonstrated sub-clinical nystagmus. Foveal hypoplasia (FH) was noted in four of five probands. Both co-twins of the monozygotic pair and triplets displayed FH. In three families, at least one parent had FH without nystagmus. FH alone may be insufficient to develop nystagmus. Whilst arrested optokinetic reflex pathway development is implicated in IN, discordant twins raise questions regarding where differences in development have arisen. In unaffected monozygotes therefore, genetic variants may predispose to oculomotor instability, with variable expressivity possibly responsible for the discordance observed.

Deep-Learning-Based Pupil Center Detection and Tracking Technology for Visible-Light Wearable Gaze Tracking Devices

In this study, for the application of visible-light wearable eye trackers, a pupil tracking methodology based on deep-learning technology is developed. By applying deep-learning object detection technology based on the You Only Look Once (YOLO) model, the proposed pupil tracking method can effectively estimate and predict the center of the pupil in the visible-light mode. By using the developed YOLOv3-tiny-based model to test the pupil tracking performance, the detection accuracy is as high as 80%, and the recall rate is close to 83%. In addition, the average visible-light pupil tracking errors of the proposed YOLO-based deep-learning design are smaller than 2 pixels for the training mode and 5 pixels for the cross-person test, which are much smaller than those of the previous ellipse fitting design without using deep-learning technology under the same visible-light conditions. After the combination of calibration process, the average gaze tracking errors by the proposed YOLOv3-tiny-based pupil tracking models are smaller than 2.9 and 3.5 degrees at the training and testing modes, respectively, and the proposed visible-light wearable gaze tracking system performs up to 20 frames per second (FPS) on the GPU-based software embedded platform.

Portable device for presbyopia correction with optoelectronic lenses driven by pupil response

A novel portable device has been developed and built to dynamically, and automatically, correct presbyopia by means of a couple of opto-electronics lenses driven by pupil tracking. The system is completely portable providing with a high range of defocus correction up to 10 D. The glasses are controlled and powered by a smartphone. To achieve a truly real-time response, image processing algorithms have been implemented in OpenCL and ran on the GPU of the smartphone. To validate the system, different visual experiments were carried out in presbyopic subjects. Visual acuity was maintained nearly constant for a range of distances from 5 m to 20 cm.