scholarly journals Audio–visual object removal in 360-degree videos

2020 ◽  
Vol 36 (10-12) ◽  
pp. 2117-2128
Author(s):  
Ryo Shimamura ◽  
Qi Feng ◽  
Yuki Koyama ◽  
Takayuki Nakatsuka ◽  
Satoru Fukayama ◽  
...  

Abstract We present a novel concept audio–visual object removal in 360-degree videos, in which a target object in a 360-degree video is removed in both the visual and auditory domains synchronously. Previous methods have solely focused on the visual aspect of object removal using video inpainting techniques, resulting in videos with unreasonable remaining sounds corresponding to the removed objects. We propose a solution which incorporates direction acquired during the video inpainting process into the audio removal process. More specifically, our method identifies the sound corresponding to the visually tracked target object and then synthesizes a three-dimensional sound field by subtracting the identified sound from the input 360-degree video. We conducted a user study showing that our multi-modal object removal supporting both visual and auditory domains could significantly improve the virtual reality experience, and our method could generate sufficiently synchronous, natural and satisfactory 360-degree videos.

PLoS ONE ◽  
2021 ◽  
Vol 16 (10) ◽  
pp. e0258103
Author(s):  
Andreas Bueckle ◽  
Kilian Buehling ◽  
Patrick C. Shih ◽  
Katy Börner

Working with organs and extracted tissue blocks is an essential task in many medical surgery and anatomy environments. In order to prepare specimens from human donors for further analysis, wet-bench workers must properly dissect human tissue and collect metadata for downstream analysis, including information about the spatial origin of tissue. The Registration User Interface (RUI) was developed to allow stakeholders in the Human Biomolecular Atlas Program (HuBMAP) to register tissue blocks—i.e., to record the size, position, and orientation of human tissue data with regard to reference organs. The RUI has been used by tissue mapping centers across the HuBMAP consortium to register a total of 45 kidney, spleen, and colon tissue blocks, with planned support for 17 organs in the near future. In this paper, we compare three setups for registering one 3D tissue block object to another 3D reference organ (target) object. The first setup is a 2D Desktop implementation featuring a traditional screen, mouse, and keyboard interface. The remaining setups are both virtual reality (VR) versions of the RUI: VR Tabletop, where users sit at a physical desk which is replicated in virtual space; VR Standup, where users stand upright while performing their tasks. All three setups were implemented using the Unity game engine. We then ran a user study for these three setups involving 42 human subjects completing 14 increasingly difficult and then 30 identical tasks in sequence and reporting position accuracy, rotation accuracy, completion time, and satisfaction. All study materials were made available in support of future study replication, alongside videos documenting our setups. We found that while VR Tabletop and VR Standup users are about three times as fast and about a third more accurate in terms of rotation than 2D Desktop users (for the sequence of 30 identical tasks), there are no significant differences between the three setups for position accuracy when normalized by the height of the virtual kidney across setups. When extrapolating from the 2D Desktop setup with a 113-mm-tall kidney, the absolute performance values for the 2D Desktop version (22.6 seconds per task, 5.88 degrees rotation, and 1.32 mm position accuracy after 8.3 tasks in the series of 30 identical tasks) confirm that the 2D Desktop interface is well-suited for allowing users in HuBMAP to register tissue blocks at a speed and accuracy that meets the needs of experts performing tissue dissection. In addition, the 2D Desktop setup is cheaper, easier to learn, and more practical for wet-bench environments than the VR setups.


2014 ◽  
Vol 602-605 ◽  
pp. 1689-1692
Author(s):  
Cong Lin ◽  
Chi Man Pun

A novel visual object tracking method for color video stream based on traditional particle filter is proposed in this paper. Feature vectors are extracted from coefficient matrices of fast three-dimensional Discrete Cosine Transform (fast 3-D DCT). The feature, as experiment showed, is very robust to occlusion and rotation and it is not sensitive to scale changes. The proposed method is efficient enough to be used in a real-time application. The experiment was carried out on some common used datasets in literature. The results are satisfied and showed the estimated trace follows the target object very closely.


Author(s):  
Robin Horst ◽  
Ramtin Naraghi-Taghi-Off ◽  
Linda Rau ◽  
Ralf Dörner

AbstractEvery Virtual Reality (VR) experience has to end at some point. While there already exist concepts to design transitions for users to enter a virtual world, their return from the physical world should be considered, as well, as it is a part of the overall VR experience. We call the latter outro-transitions. In contrast to offboarding of VR experiences, that takes place after taking off VR hardware (e.g., HMDs), outro-transitions are still part of the immersive experience. Such transitions occur more frequently when VR is experienced periodically and for only short times. One example where transition techniques are necessary is in an auditorium where the audience has individual VR headsets available, for example, in a presentation using PowerPoint slides together with brief VR experiences sprinkled between the slides. The audience must put on and take off HMDs frequently every time they switch from common presentation media to VR and back. In a such a one-to-many VR scenario, it is challenging for presenters to explore the process of multiple people coming back from the virtual to the physical world at once. Direct communication may be constrained while VR users are wearing an HMD. Presenters need a tool to indicate them to stop the VR session and switch back to the slide presentation. Virtual visual cues can help presenters or other external entities (e.g., automated/scripted events) to request VR users to end a VR session. Such transitions become part of the overall experience of the audience and thus must be considered. This paper explores visual cues as outro-transitions from a virtual world back to the physical world and their utility to enable presenters to request VR users to end a VR session. We propose and investigate eight transition techniques. We focus on their usage in short consecutive VR experiences and include both established and novel techniques. The transition techniques are evaluated within a user study to draw conclusions on the effects of outro-transitions on the overall experience and presence of participants. We also take into account how long an outro-transition may take and how comfortable our participants perceived the proposed techniques. The study points out that they preferred non-interactive outro-transitions over interactive ones, except for a transition that allowed VR users to communicate with presenters. Furthermore, we explore the presenter-VR user relation within a presentation scenario that uses short VR experiences. The study indicates involving presenters that can stop a VR session was not only negligible but preferred by our participants.


2020 ◽  
Vol 1 (1) ◽  
pp. 62-70
Author(s):  
Amir H Sadeghi ◽  
Wouter Bakhuis ◽  
Frank Van Schaagen ◽  
Frans B S Oei ◽  
Jos A Bekkers ◽  
...  

Abstract Aims Increased complexity in cardiac surgery over the last decades necessitates more precise preoperative planning to minimize operating time, to limit the risk of complications during surgery and to aim for the best possible patient outcome. Novel, more realistic, and more immersive techniques, such as three-dimensional (3D) virtual reality (VR) could potentially contribute to the preoperative planning phase. This study shows our initial experience on the implementation of immersive VR technology as a complementary research-based imaging tool for preoperative planning in cardiothoracic surgery. In addition, essentials to set up and implement a VR platform are described. Methods Six patients who underwent cardiac surgery at the Erasmus Medical Center, Rotterdam, The Netherlands, between March 2020 and August 2020, were included, based on request by the surgeon and availability of computed tomography images. After 3D VR rendering and 3D segmentation of specific structures, the reconstruction was analysed via a head mount display. All participating surgeons (n = 5) filled out a questionnaire to evaluate the use of VR as preoperative planning tool for surgery. Conclusion Our study demonstrates that immersive 3D VR visualization of anatomy might be beneficial as a supplementary preoperative planning tool for cardiothoracic surgery, and further research on this topic may be considered to implement this innovative tool in daily clinical practice. Lay summary Over the past decades, surgery on the heart and vessels is becoming more and more complex, necessitating more precise and accurate preoperative planning. Nowadays, operative planning is feasible on flat, two-dimensional computer screens, however, requiring a lot of spatial and three-dimensional (3D) thinking of the surgeon. Since immersive 3D virtual reality (VR) is an upcoming imaging technique with promising results in other fields of surgery, we aimed in this study to explore the additional value of this technique in heart surgery. Our surgeons planned six different heart operations by visualizing computed tomography scans with a dedicated VR headset, enabling them to visualize the patient’s anatomy in an immersive and 3D environment. The outcomes of this preliminary study are positive, with a much more reality-like simulation for the surgeon. In such, VR could potentially be beneficial as a preoperative planning tool for complex heart surgery.


2021 ◽  
Author(s):  
Valentin Holzwarth ◽  
Johannes Schneider ◽  
Joshua Handali ◽  
Joy Gisler ◽  
Christian Hirt ◽  
...  

AbstractInferring users’ perceptions of Virtual Environments (VEs) is essential for Virtual Reality (VR) research. Traditionally, this is achieved through assessing users’ affective states before and after being exposed to a VE, based on standardized, self-assessment questionnaires. The main disadvantage of questionnaires is their sequential administration, i.e., a user’s affective state is measured asynchronously to its generation within the VE. A synchronous measurement of users’ affective states would be highly favorable, e.g., in the context of adaptive systems. Drawing from nonverbal behavior research, we argue that behavioral measures could be a powerful approach to assess users’ affective states in VR. In this paper, we contribute by providing methods and measures evaluated in a user study involving 42 participants to assess a users’ affective states by measuring head movements during VR exposure. We show that head yaw significantly correlates with presence, mental and physical demand, perceived performance, and system usability. We also exploit the identified relationships for two practical tasks that are based on head yaw: (1) predicting a user’s affective state, and (2) detecting manipulated questionnaire answers, i.e., answers that are possibly non-truthful. We found that affective states can be predicted significantly better than a naive estimate for mental demand, physical demand, perceived performance, and usability. Further, manipulated or non-truthful answers can also be estimated significantly better than by a naive approach. These findings mark an initial step in the development of novel methods to assess user perception of VEs.


i-com ◽  
2020 ◽  
Vol 19 (2) ◽  
pp. 67-85
Author(s):  
Matthias Weise ◽  
Raphael Zender ◽  
Ulrike Lucke

AbstractThe selection and manipulation of objects in Virtual Reality face application developers with a substantial challenge as they need to ensure a seamless interaction in three-dimensional space. Assessing the advantages and disadvantages of selection and manipulation techniques in specific scenarios and regarding usability and user experience is a mandatory task to find suitable forms of interaction. In this article, we take a look at the most common issues arising in the interaction with objects in VR. We present a taxonomy allowing the classification of techniques regarding multiple dimensions. The issues are then associated with these dimensions. Furthermore, we analyze the results of a study comparing multiple selection techniques and present a tool allowing developers of VR applications to search for appropriate selection and manipulation techniques and to get scenario dependent suggestions based on the data of the executed study.


Sensors ◽  
2021 ◽  
Vol 21 (2) ◽  
pp. 397
Author(s):  
Qimeng Zhang ◽  
Ji-Su Ban ◽  
Mingyu Kim ◽  
Hae Won Byun ◽  
Chang-Hun Kim

We propose a low-asymmetry interface to improve the presence of non-head-mounted-display (non-HMD) users in shared virtual reality (VR) experiences with HMD users. The low-asymmetry interface ensures that the HMD and non-HMD users’ perception of the VR environment is almost similar. That is, the point-of-view asymmetry and behavior asymmetry between HMD and non-HMD users are reduced. Our system comprises a portable mobile device as a visual display to provide a changing PoV for the non-HMD user and a walking simulator as an in-place walking detection sensor to enable the same level of realistic and unrestricted physical-walking-based locomotion for all users. Because this allows non-HMD users to experience the same level of visualization and free movement as HMD users, both of them can engage as the main actors in movement scenarios. Our user study revealed that the low-asymmetry interface enables non-HMD users to feel a presence similar to that of the HMD users when performing equivalent locomotion tasks in a virtual environment. Furthermore, our system can enable one HMD user and multiple non-HMD users to participate together in a virtual world; moreover, our experiments show that the non-HMD user satisfaction increases with the number of non-HMD participants owing to increased presence and enjoyment.


Leonardo ◽  
1999 ◽  
Vol 32 (4) ◽  
pp. 261-268 ◽  
Author(s):  
Matthew Kirschenbaum

This paper documents an interactive graphics installation entitled Lucid Mapping and Codex Transformissions in the Z-Buffer. Lucid Mapping uses the Virtual Reality Modeling Language to explore textual and narrative possibilities within three-dimensional (3D) electronic environments. The author describes the creative rationale and technical design of the work and places it within the context of other applications of 3D text and typography in the digital arts and the scientific visualization communities. The author also considers the implications of 3D textual environments on visual language and communication, and discriminates among a range of different visual/ rhetorical strategies that such environments can sustain.


2013 ◽  
Vol 791-793 ◽  
pp. 1436-1440
Author(s):  
Ling Hang Yang

With the development of computer hardware and software technology, virtual reality technology of computer has been widely used in various fields. Virtual teaching process is one of the main applications of virtual reality computer technology. Tennis is one of the most common sports. Tennis process mainly includes the process of catching a ball, serving a ball and hitting a ball. Virtual process of tennis system must establish an accurate numerical simulation model to calculate the mechanical impedance during the arm movement of human. According to this, it builds a model of the mechanical impedance of human arm in tennis virtual system using three-dimensional simulation software in this paper and gets the curve of mechanical impedance through the simulation. Finally, the article compares calculation results with the theoretical results and concludes that the theoretical results and simulation results are basically consistent which provide a theoretical reference for the design of the development of virtual system for the human.


Sign in / Sign up

Export Citation Format

Share Document