scholarly journals SPHERE: A novel approach to 3D and active sound localization

2020 ◽  
Author(s):  
V. Gaveau ◽  
A. Coudert ◽  
R. Salemme ◽  
E. Koun ◽  
C. Desoche ◽  
...  
Keyword(s):  
Virtual Reality ◽  
Real Time ◽  
Sound Localization ◽  
Sound Source ◽  
Spatial Hearing ◽  
Active Listening ◽  
Head Mounted Display ◽  
Real Time Kinematic ◽  
Novel Approach ◽  
Listening Behavior

AbstractIn everyday life, localizing a sound source in free-field entails more than the sole extraction of monaural and binaural auditory cues to define its location in the three-dimensions (azimuth, elevation and distance). In spatial hearing, we also take into account all the available visual information (e.g., cues to sound position, cues to the structure of the environment), and we resolve perceptual ambiguities through active listening behavior, exploring the auditory environment with head or/and body movements. Here we introduce a novel approach to sound localization in 3D named SPHERE (European patent n° WO2017203028A1), which exploits a commercially available Virtual Reality Head-mounted display system with real-time kinematic tracking to combine all of these elements (controlled positioning of a real sound source and recording of participants’ responses in 3D, controlled visual stimulations and active listening behavior). We prove that SPHERE allows accurate sampling of the 3D spatial hearing abilities of normal hearing adults, and it allowed detecting and quantifying the contribution of active listening. Specifically, comparing static vs. free head-motion during sound emission we found an improvement of sound localization accuracy and precisions. By combining visual virtual reality, real-time kinematic tracking and real-sound delivery we have achieved a novel approach to the study of spatial hearing, with the potentials to capture real-life behaviors in laboratory conditions. Furthermore, our new approach also paves the way for clinical and industrial applications that will leverage the full potentials of active listening and multisensory stimulation intrinsic to the SPHERE approach for the purpose rehabilitation and product assessment.

scholarly journals Reaching to sounds in virtual reality: A multisensory-motor approach to re-learn sound localisation

2020 ◽  
Author(s):  
Chiara Valzolgher ◽  
Grègoire Verdelet ◽  
Romeo Salemme ◽  
Luigi Lombardi ◽  
Valerie Gaveau ◽  
...  
Keyword(s):  
Virtual Reality ◽  
Visual Feedback ◽  
Sound Source ◽  
Head Movement ◽  
Spatial Hearing ◽  
Head Movements ◽  
Active Listening ◽  
Movement Behaviour ◽  
Sound Localisation

ABSTRACTWhen localising sounds in space the brain relies on internal models that specify the correspondence between the auditory input reaching the ears and initial head-position with coordinates in external space. These models can be updated throughout life, setting the basis for re-learning spatial hearing abilities in adulthood. This is particularly important for individuals who experience long-term auditory alterations (e.g., hearing loss, hearing aids, cochlear implants) as well as individuals who have to adapt to novel auditory cues when listening in virtual auditory environments. Until now, several methodological constraints have limited our understanding of the mechanisms involved in spatial hearing re-learning. In particular, the potential role of active listening and head-movements have remained largely overlooked. Here, we overcome these limitations by using a novel methodology, based on virtual reality and real-time kinematic tracking, to study the role of active multisensory-motor interactions with sounds in the updating of sound-space correspondences. Participants were immersed in a virtual reality scenario showing 17 speakers at ear-level. From each visible speaker a free-field real sound could be generated. Two separate groups of participants localised the sound source either by reaching or naming the perceived sound source, under binaural or monaural listening. Participants were free to move their head during the task and received audio-visual feedback on their performance. Results showed that both groups compensated rapidly for the short-term auditory alteration caused by monaural listening, improving sound localisation performance across trials. Crucially, compared to naming, reaching the sounds induced faster and larger sound localisation improvements. Furthermore, more accurate sound localisation was accompanied by progressively wider head-movements. These two measures were significantly correlated selectively for the Reaching group. In conclusion, reaching to sounds in an immersive visual VR context proved most effective for updating altered spatial hearing. Head movements played an important role in this fast updating, pointing to the importance of active listening when implementing training protocols for improving spatial hearing.HIGHLIGHTS- We studied spatial hearing re-learning using virtual reality and kinematic tracking- Audio-visual feedback combined with active listening improved monaural sound localisation- Reaching to sounds improved performance more than naming sounds- Monaural listening triggered compensatory head-movement behaviour- Head-movement behaviour correlated with re-learning only when reaching to sounds

scholarly journals Real-Time Application for Generating Multiple Experiences from 360° Panoramic Video by Tracking Arbitrary Objects and Viewer’s Orientations

2020 ◽  
Vol 10 (7) ◽  
pp. 2248
Author(s):  
Syed Hammad Hussain Shah ◽  
Kyungjin Han ◽  
Jong Weon Lee
Keyword(s):  
Virtual Reality ◽  
Real Time ◽  
User Study ◽  
Limited Area ◽  
Head Mounted Display ◽  
Head Mounted Displays ◽  
Panoramic Video ◽  
Interesting Object ◽  
Technical Evaluation ◽  
Arbitrary Objects

We propose a novel authoring and viewing system for generating multiple experiences with a single 360° video and efficiently transferring these experiences to the user. An immersive video contains much more interesting information within the 360° environment than normal videos. There can be multiple interesting areas within a 360° frame at the same time. Due to the narrow field of view in virtual reality head-mounted displays, a user can only view a limited area of a 360° video. Hence, our system is aimed at generating multiple experiences based on interesting information in different regions of a 360° video and efficient transferring of these experiences to prospective users. The proposed system generates experiences by using two approaches: (1) Recording of the user’s experience when the user watches a panoramic video using a virtual reality head-mounted display, and (2) tracking of an arbitrary interesting object in a 360° video selected by the user. For tracking of an arbitrary interesting object, we have developed a pipeline around an existing simple object tracker to adapt it for 360° videos. This tracking algorithm was performed in real time on a CPU with high precision. Moreover, to the best of our knowledge, there is no such existing system that can generate a variety of different experiences from a single 360° video and enable the viewer to watch one 360° visual content from various interesting perspectives in immersive virtual reality. Furthermore, we have provided an adaptive focus assistance technique for efficient transferring of the generated experiences to other users in virtual reality. In this study, technical evaluation of the system along with a detailed user study has been performed to assess the system’s application. Findings from evaluation of the system showed that a single 360° multimedia content has the capability of generating multiple experiences and transfers among users. Moreover, sharing of the 360° experiences enabled viewers to watch multiple interesting contents with less effort.

scholarly journals Short-term effects of sound localization training in virtual reality

2017 ◽  
Author(s):  
Mark A. Steadman ◽  
Chungeun Kim ◽  
Jean-Hugues Lestang ◽  
Dan F. M. Goodman ◽  
Lorenzo Picinali
Keyword(s):  
Sound Localization ◽  
Sound Source ◽  
Game Design ◽  
Transfer Functions ◽  
Polar Angle ◽  
Active Listening ◽  
Head Tracking ◽  
Localization Accuracy ◽  
Design Elements ◽  
Listening Group

ABSTRACTHead-related transfer functions (HRTFs) capture the direction-dependant way that sound interacts with the head and torso. In virtual audio systems, which aim to emulate these effects, non-individualized, generic HRTFs are typically used leading to an inaccurate perception of virtual sound location. Training has the potential to exploit the brain’s ability to adapt to these unfamiliar cues. In this study, three virtual sound localization training paradigms were evaluated; one provided simple visual positional confirmation of sound source location, a second introduced game design elements (“gamification”) and a final version additionally utilized head-tracking to provide listeners with experience of relative sound source motion (“active listening”). The results demonstrate a significant effect of training after a small number of short (12-minute) training sessions, which is retained across multiple days. Gamification alone had no significant effect on the efficacy of the training, but active listening resulted in a significantly greater improvements in localization accuracy. In general, improvements in virtual sound localization following training generalized to a second set of non-individualized HRTFs, although some HRTF-specific changes were observed in polar angle judgement for the active listening group. The implications of this on the putative mechanisms of the adaptation process are discussed.

scholarly journals Virtual Reality Application of the Fortress Al Zubarah in Qatar Including Performance Analysis of Real-Time Visualisation

2021 ◽  
Author(s):  
Thomas Kersten ◽  
Daniel Drenkhan ◽  
Simon Deggim
Keyword(s):  
Virtual Reality ◽  
Real Time ◽  
Virtual Worlds ◽  
3D Models ◽  
Frame Rate ◽  
Computer Hardware ◽  
Head Mounted Display ◽  
Digital World ◽  
Cad Models ◽  
Game Engines

AbstractTechnological advancements in the area of Virtual Reality (VR) in the past years have the potential to fundamentally impact our everyday lives. VR makes it possible to explore a digital world with a Head-Mounted Display (HMD) in an immersive, embodied way. In combination with current tools for 3D documentation, modelling and software for creating interactive virtual worlds, VR has the means to play an important role in the conservation and visualisation of cultural heritage (CH) for museums, educational institutions and other cultural areas. Corresponding game engines offer tools for interactive 3D visualisation of CH objects, which makes a new form of knowledge transfer possible with the direct participation of users in the virtual world. However, to ensure smooth and optimal real-time visualisation of the data in the HMD, VR applications should run at 90 frames per second. This frame rate is dependent on several criteria including the amount of data or number of dynamic objects. In this contribution, the performance of a VR application has been investigated using different digital 3D models of the fortress Al Zubarah in Qatar with various resolutions. We demonstrate the influence on real-time performance by the amount of data and the hardware equipment and that developers of VR applications should find a compromise between the amount of data and the available computer hardware, to guarantee a smooth real-time visualisation with approx. 90 fps (frames per second). Therefore, CAD models offer a better performance for real-time VR visualisation than meshed models due to the significant reduced data volume.

scholarly journals Enabling real-time applications in Wi-Fi networks

2019 ◽  
Vol 15 (5) ◽  
pp. 155014771984531 ◽  
Author(s):  
Dmitry Bankov ◽  
Evgeny Khorov ◽  
Andrey Lyakhov ◽  
Mark Sandal
Keyword(s):  
Mathematical Model ◽  
Virtual Reality ◽  
Remote Control ◽  
Real Time ◽  
Packet Loss ◽  
Loss Ratio ◽  
High Importance ◽  
Real Time Traffic ◽  
Novel Approach ◽  
Real Time Applications

Being of high importance, real-time applications, such as online gaming, real-time video streaming, virtual reality, and remote-control drone and robots, introduce many challenges to the developers of wireless networks. Such applications pose strict requirements on the delay and packet loss ratio, and it is hardly possible to satisfy them in Wi-Fi networks that use random channel access. The article presents a novel approach to enable real-time communications by exploiting an additional radio. This approach was recently proposed by us in the IEEE 802.11 Working Group and attracted much attention. To evaluate its gain and to study how real-time traffic coexists with the usual one, a mathematical model is designed. The numerical results show that the proposed approach allows decreasing the losses and delays for the real-time traffic by orders of magnitude, while the throughput for the usual traffic is reduced insignificantly in comparison to existing networks.

Virtual Reality as Communication Tool: A Sociocognitive Analysis

1999 ◽  
Vol 8 (4) ◽  
pp. 462-468 ◽  
Author(s):  
Giuseppe Riva
Keyword(s):  
Virtual Reality ◽  
Real Time ◽  
Communication Tool ◽  
Communication Interface ◽  
Interactive Communication ◽  
Communication Medium ◽  
Head Mounted Display ◽  
The Media ◽  
Single User

Virtual reality (VR) is usually described by the media as a particular collection of technological hardware: a computer capable of 3-D real-time animation, a head-mounted display, and data gloves equipped with one or more position trackers. However, this focus on technology is somewhat disappointing for communication researchers and VR designers. To overcome this limitation, this paper describes VR as a communication tool: a communication medium in the case of multiuser VR and a communication interface in single-user VR. The consequences of this approach for the design and the development of VR systems are presented, together with the methodological and technical implications for the study of interactive communication via computers.

Medical Assessment Test of Extrapersonal Neglect Using Virtual Reality: A Preliminary Study

2020 ◽  
Author(s):  
Daniel Lanzoni ◽  
Andrea Vitali ◽  
Daniele Regazzoni ◽  
Caterina Rizzi
Keyword(s):  
Virtual Reality ◽  
Software Architecture ◽  
Real Time ◽  
Virtual Environment ◽  
Research Work ◽  
Medical Personnel ◽  
Medical Assessment ◽  
Cerebral Lesions ◽  
Head Mounted Display ◽  
Preliminary Study

Abstract The research work presents a preliminary study to create a virtual reality platform for the medical assessment of spatial extrapersonal neglect, a syndrome affecting human awareness of a hemi-space that may be caused by cerebral lesions. Nowadays, the extrapersonal neglect is assessed by using real objects positioned in the space around the patient, with a poor capability of repetition and data gathering. Therefore, the aim of this research work is the introduction of a virtual reality solution based on consumer technology for the assessment of the extrapersonal neglect. By starting from the needs of the involved medical personnel, an online serious-game platform has been developed, which permits to perform a test and a real-time evaluation by means of objective data tracked by exploited technologies, i.e. an HTC Vive Pro head mounted display and ad-hoc IT solutions. The test is based on a virtual environment composed by a table on which twenty objects have been placed, ten on the right side and ten on the left side. The whole 3D virtual environment has been developed using low-cost and free development tools, such as Unity and Blender. The interaction with the virtual environment is based on voice recognition technology, therefore the patient interact with the application by pronouncing the name of each object aloud. The VR application has been developed according to an online gaming software architecture, which permits to share the 3D scene by exploiting a Wi-Fi hotspot network. Furthermore, the on-line gaming software architecture allows sending and receiving data between the doctor’s laptop and the VR system used by the patient on another laptop. The therapist can see through his/her personal computer a real time faithful replica of the test performed by the patient in order to have a fast feedback on patient’s field of view orientation during the evaluation of 3D objects. A preliminary test has been carried out to evaluate the ease of use for medical personnel of the developed VR platform. The big amount of recorded data and the possibility to manage the selection of objects when the voice commands are not correctly interpreted has been greatly appreciated. The review of the performed test represents for doctors the possibility of objectively reconstructing the improvements of patients during the whole period of the rehabilitation process. Medical feedback highlighted how the developed prototype can already be tested involving patients and thus, a procedure for enrolling a group of patients has been planned. Finally, future tests have been planned to compare the developed solution with the Caterine Bergero Scale to define a future standardization.

scholarly journals Towards a Synesthesia Laboratory: Real-time Localization and Visualization of a Sound Source for Virtual Reality Applications

2018 ◽  
Vol 14 (1) ◽  
Author(s):  
Ahmet Kose ◽  
Aleksei Tepljakov ◽  
Sergei Astapov ◽  
Dirk Draheim ◽  
Eduard Petlenkov ◽  
...  
Keyword(s):  
Virtual Reality ◽  
Real Time ◽  
Sound Source
Sign in / Sign up

Export Citation Format

Share Document