Dual mode imaging in mid infrared with thermal signal reconstruction for innovative diagnostics of the “Monocromo” by Leonardo da Vinci

Dual mode imaging in the mid infrared band, a joint use of thermography and quasi-thermal reflectography, was recently proposed as a full field diagnostic tool in cultural heritage. Here we discuss for the first time, to the best of our knowledge, a detailed application of such non destructive technique to the diagnostics of frescoes, with an emphasis on the location of detachments. We also investigate the use of a thermographic method based on TSR (thermal signal reconstruction), in a long pulse stimulus scheme, as well as the spatial registration of thermal images after post-processing analysis to their visible counterpart, so as to obtain a fine resolution diagnostic map. As an exemplar case study, we report about the application of dual mode imaging with a 500 $${\upmu }\hbox {m}$$ μ m pixel size at object plane on the “Monocromo”, a fresco by Leonardo da Vinci located in the Sforza Castle (Milan, Italy). Our technique was used to guide the conservators during the restoration works, opening new perspectives in artwork diagnostics.

A Mixed Reality Interface for a Digital Twin Based Crane

Digital twin technology empowers the digital transformation of the industrial world with an increasing amount of data, which meanwhile creates a challenging context for designing a human–machine interface (HMI) for operating machines. This work aims at creating an HMI for digital twin based services. With an industrial crane platform as a case study, we presented a mixed reality (MR) application running on a Microsoft HoloLens 1 device. The application, consisting of visualization, interaction, communication, and registration modules, allowed crane operators to both monitor the crane status and control its movement through interactive holograms and bi-directional data communication, with enhanced mobility thanks to spatial registration and tracking of the MR environment. The prototype was quantitatively evaluated regarding the control accuracy in 20 measurements following a step-by-step protocol that we defined to standardize the measurement procedure. The results suggested that the differences between the target and actual positions were within the 10 cm range in three dimensions, which were considered sufficiently small regarding the typical crane operation use case of logistics purposes and could be improved with the adoption of robust registration and tracking techniques in our future work.

Combined Mass Spectrometry and Histopathology Imaging for Perioperative Tissue Assessment in Cancer Surgery

Mass spectrometry is an effective imaging tool for evaluating biological tissue to detect cancer. With the assistance of deep learning, this technology can be used as a perioperative tissue assessment tool that will facilitate informed surgical decisions. To achieve such a system requires the development of a database of mass spectrometry signals and their corresponding pathology labels. Assigning correct labels, in turn, necessitates precise spatial registration of histopathology and mass spectrometry data. This is a challenging task due to the domain differences and noisy nature of images. In this study, we create a registration framework for mass spectrometry and pathology images as a contribution to the development of perioperative tissue assessment. In doing so, we explore two opportunities in deep learning for medical image registration, namely, unsupervised, multi-modal deformable image registration and evaluation of the registration. We test this system on prostate needle biopsy cores that were imaged with desorption electrospray ionization mass spectrometry (DESI) and show that we can successfully register DESI and histology images to achieve accurate alignment and, consequently, labelling for future training. This automation is expected to improve the efficiency and development of a deep learning architecture that will benefit the use of mass spectrometry imaging for cancer diagnosis.

Superimposing holograms on real world objects using HoloLens 2 and its depth camera

Augmented Reality glasses such as HoloLens 2 may provide visual guidance during surgical interventions. To superimpose the holograms on real world objects (RWO), for instance a patient, spatial registration is required. In this work, we propose an approach to automatically register a hologram to the according RWO. To this end, the framework utilizes the depth camera of HoloLens 2 to acquire the point cloud (PC) of the RWO. A novel and recently published PC registration algorithm allows to register the PC of the RWO and the hologram after a rough initial placement without any need for pre-processing or outlier removal. The approach is evaluated by measuring displacements between certain known positions of the hologram and the RWO. The first metric relies on measuring points using an optically tracked stylus while the second is based on visually perceived positions. The median displacements were 22.3 mm, 35.6 mm, and 13.3 mm for the x-, y-, and z-axes in the first metric and 8.1 mm, 4.3 mm, and 11.9 mm for the second metric. Even though the accuracy is not yet adequate for many surgical interventions, the framework provides an initial step for a convenient marker less registration of holograms to an RWO.

An Optics-guided Robotic System for Dental Implant Surgery

At present, dental implant surgery mainly relies on the clinical experience of the doctor and the assistance of preoperative medical imaging. However, there are some problems in dental implant surgery, such as narrow space, sight obstruction, inaccurate positioning, and high requirements of doctors' proficiency. Therefore, a dental implant robot system (DIRS) guided by optical navigation is developed in this study, with an x-shaped tool and an irregular pentagonal tracer are designed for spatial registration and needle tip positioning strategy respectively. The coordinate system of each unit in DIRS is unified through system calibration, spatial registration, and needle tip positioning strategy. Then the surgical path is planned on the Computed Tomography (CT) images in the navigation software before operation. The automatic positioning method and the auxiliary positioning method can be used in the operation to achieve accurate positioning and assist doctors to complete the operation. The errors of spatial registration, needle tip positioning strategy, and the overall accuracy of the system were evaluated respectively, and the results showed that they all met the needs of clinical surgery. This study preliminarily verified the feasibility of the precise positioning method for dental surgery robots and provided certain ideas for subsequent related research.