scholarly journals Enabling Real-Time 3D Display of Lifelike Fingerspelling in a Web App

2020 ◽  
pp. 38-44
Author(s):  
Jami Montgomery ◽  
John McDonald ◽  
Eric Gong ◽  
Souad Baowidan ◽  
Rosalee Wolfe
Keyword(s):  
Real Time ◽  
Sign Language ◽  
Mobile Technologies ◽  
3D Display ◽  
3D Rendering ◽  
Natural Motion ◽  
Software Platforms ◽  
Web App

AbstractFingerspelling receptive skills remain among the most difficult aspects of sign language for hearing people to learn due to the lack of access to practice tools that reproduce the natural motion of human signing. This problem has been exacerbated in recent years by the move from desktop to mobile technologies which has rendered prior software platforms less accessible to general users. This paper explores a web-enabled 3D rendering architecture that enables real-time fingerspelling on a human avatar that can address these issues. In addition it is capable of producing more realistic motion than prior efforts that were video-based and provides greater interactivity and customization that will support further enhancements to self-practice tools for fingerspelling reception.

scholarly journals SURG-03. IMMERSIVE VIRTUAL REALITY APPLICATIONS IN NEUROSURGICAL ONCOLOGY

2020 ◽  
Vol 22 (Supplement_3) ◽  
pp. iii461-iii461
Author(s):  
Andrea Carai ◽  
Angela Mastronuzzi ◽  
Giovanna Stefania Colafati ◽  
Paul Voicu ◽  
Nicola Onorini ◽  
...  
Keyword(s):  
Virtual Reality ◽  
Real Time ◽  
New Technologies ◽  
Young Patients ◽  
Spatial Relationship ◽  
3D Models ◽  
Immersive Virtual Reality ◽  
Global Improvement ◽  
3D Rendering ◽  
Before And After

Abstract Tridimensional (3D) rendering of volumetric neuroimaging is increasingly been used to assist surgical management of brain tumors. New technologies allowing immersive virtual reality (VR) visualization of obtained models offer the opportunity to appreciate neuroanatomical details and spatial relationship between the tumor and normal neuroanatomical structures to a level never seen before. We present our preliminary experience with the Surgical Theatre, a commercially available 3D VR system, in 60 consecutive neurosurgical oncology cases. 3D models were developed from volumetric CT scans and MR standard and advanced sequences. The system allows the loading of 6 different layers at the same time, with the possibility to modulate opacity and threshold in real time. Use of the 3D VR was used during preoperative planning allowing a better definition of surgical strategy. A tailored craniotomy and brain dissection can be simulated in advanced and precisely performed in the OR, connecting the system to intraoperative neuronavigation. Smaller blood vessels are generally not included in the 3D rendering, however, real-time intraoperative threshold modulation of the 3D model assisted in their identification improving surgical confidence and safety during the procedure. VR was also used offline, both before and after surgery, in the setting of case discussion within the neurosurgical team and during MDT discussion. Finally, 3D VR was used during informed consent, improving communication with families and young patients. 3D VR allows to tailor surgical strategies to the single patient, contributing to procedural safety and efficacy and to the global improvement of neurosurgical oncology care.

Enabling Real-time Sign Language Translation on Mobile Platforms with On-board Depth Cameras

2021 ◽  
Vol 5 (2) ◽  
pp. 1-30
Author(s):  
HyeonJung Park ◽  
Youngki Lee ◽  
JeongGil Ko
Keyword(s):  
Real Time ◽  
Sign Language ◽  
Data Augmentation ◽  
Language Translation ◽  
Mobile Platforms ◽  
Depth Cameras ◽  
Language Data ◽  
In The Wild ◽  
Environmental Robustness ◽  
Cloud Servers

In this work we present SUGO, a depth video-based system for translating sign language to text using a smartphone's front camera. While exploiting depth-only videos offer benefits such as being less privacy-invasive compared to using RGB videos, it introduces new challenges which include dealing with low video resolutions and the sensors' sensitiveness towards user motion. We overcome these challenges by diversifying our sign language video dataset to be robust to various usage scenarios via data augmentation and design a set of schemes to emphasize human gestures from the input images for effective sign detection. The inference engine of SUGO is based on a 3-dimensional convolutional neural network (3DCNN) to classify a sequence of video frames as a pre-trained word. Furthermore, the overall operations are designed to be light-weight so that sign language translation takes place in real-time using only the resources available on a smartphone, with no help from cloud servers nor external sensing components. Specifically, to train and test SUGO, we collect sign language data from 20 individuals for 50 Korean Sign Language words, summing up to a dataset of ~5,000 sign gestures and collect additional in-the-wild data to evaluate the performance of SUGO in real-world usage scenarios with different lighting conditions and daily activities. Comprehensively, our extensive evaluations show that SUGO can properly classify sign words with an accuracy of up to 91% and also suggest that the system is suitable (in terms of resource usage, latency, and environmental robustness) to enable a fully mobile solution for sign language translation.

Sign in / Sign up

Export Citation Format

Share Document