Pitch-Cluster-Map Based Daily Sound Recognition for Mobile Robot Audition

This paper presents a sound identification method for a mobile robot in home and office environments. We propose a short-term sound recognition method using Pitch-Cluster-Maps (PCMs) sound database (DB) based on a Vector Quantization approach. A binarized frequency spectrum is used to generate PCMs codebook, which describes a variety of sound sources, not only voice, from short-term sound input. PCMs sound identification requires several tens of milliseconds of sound input, and is suitable for mobile robot applications in which conditions are continuously and dynamically changing. We implemented this in mobile robot audition system using a 32-channel microphone array. Robot noise reduction and sound source tracking using our proposal are applied to robot audition system, and we evaluate daily sound recognition performance for separated sound sources from a moving robot.

Download Full-text

Simultaneous localization of mobile robot and multiple sound sources using microphone array

2009 IEEE International Conference on Robotics and Automation ◽

10.1109/robot.2009.5152813 ◽

2009 ◽

Cited By ~ 5

Author(s):

Jwu-Sheng Hu ◽

Chen-Yu Chan ◽

Cheng-Kang Wang ◽

Chieh-Chih Wang

Keyword(s):

Mobile Robot ◽

Microphone Array ◽

Sound Sources

Download Full-text

Simultaneous Localization of a Mobile Robot and Multiple Sound Sources Using a Microphone Array

Advanced Robotics ◽

10.1163/016918610x538525 ◽

2011 ◽

Vol 25 (1-2) ◽

pp. 135-152 ◽

Cited By ~ 23

Author(s):

Jwu-Sheng Hu ◽

Chen-Yu Chan ◽

Cheng-Kang Wang ◽

Ming-Tang Lee ◽

Ching-Yi Kuo

Keyword(s):

Mobile Robot ◽

Microphone Array ◽

Sound Sources

Download Full-text

Three ring microphone array for 3D sound localization and separation for mobile robot audition

2005 IEEE/RSJ International Conference on Intelligent Robots and Systems ◽

10.1109/iros.2005.1545095 ◽

2005 ◽

Cited By ~ 32

Author(s):

Y. Tamai ◽

Y. Sasaki ◽

S. Kagami ◽

H. Mizoguchi

Keyword(s):

Mobile Robot ◽

Sound Localization ◽

Microphone Array ◽

3D Sound ◽

Robot Audition

Download Full-text

Sound Source Tracking Control of a Mobile Robot Using a Microphone Array

Journal of Institute of Control Robotics and Systems ◽

10.5302/j.icros.2012.18.4.343 ◽

2012 ◽

Vol 18 (4) ◽

pp. 343-352 ◽

Cited By ~ 5

Author(s):

Jong-Ho Han ◽

Sun-Sin Han ◽

Jang-Myung Lee

Keyword(s):

Mobile Robot ◽

Sound Source ◽

Tracking Control ◽

Microphone Array ◽

Source Tracking

Download Full-text

Sound Localization and Separation for Mobile Robot Tele-Operation by Tri-Concentric Microphone Array

Journal of Robotics and Mechatronics ◽

10.20965/jrm.2007.p0281 ◽

2007 ◽

Vol 19 (3) ◽

pp. 281-289 ◽

Cited By ~ 8

Author(s):

Yoko Sasaki ◽

◽

Saori Masunaga ◽

Simon Thompson ◽

Satoshi Kagami ◽

...

Keyword(s):

Mobile Robot ◽

Mobile Robots ◽

Auditory System ◽

Sound Localization ◽

Source Localization ◽

Microphone Array ◽

Visual Presentation ◽

Sound Source Localization ◽

Robot System ◽

Sound Sources

The paper describes a tele-operated mobile robot system which can perform multiple sound source localization and separation using a 32-channel tri-concentric microphone array. Tele-operated mobile robots require two main capabilities: 1) audio/visual presentation of the robot’s environment to the operator, and 2) autonomy for mobility. This paper focuses on the auditory system of a tele-operated mobile robot in order to improve both the presentation of sound sources to the operator and also to facilitate autonomous robot actions. The auditory system is based on a 32-channel distributed microphone array that uses highly efficient directional design for localizing and separating multiple moving sound sources. Experimental results demonstrate the feasibility of inter-person distant communication through the tele-operated robot system.

Download Full-text

Multi-Modal Fusion Emotion Recognition Method of Speech Expression Based on Deep Learning

Frontiers in Neurorobotics ◽

10.3389/fnbot.2021.697634 ◽

2021 ◽

Vol 15 ◽

Author(s):

Dong Liu ◽

Zhiyong Wang ◽

Lifeng Wang ◽

Longxi Chen

Keyword(s):

Feature Extraction ◽

Deep Learning ◽

Emotion Recognition ◽

Short Term Memory ◽

Selection Process ◽

Recognition Performance ◽

Multimodal Fusion ◽

Recognition Method ◽

Short Term ◽

Term Memory

The redundant information, noise data generated in the process of single-modal feature extraction, and traditional learning algorithms are difficult to obtain ideal recognition performance. A multi-modal fusion emotion recognition method for speech expressions based on deep learning is proposed. Firstly, the corresponding feature extraction methods are set up for different single modalities. Among them, the voice uses the convolutional neural network-long and short term memory (CNN-LSTM) network, and the facial expression in the video uses the Inception-Res Net-v2 network to extract the feature data. Then, long and short term memory (LSTM) is used to capture the correlation between different modalities and within the modalities. After the feature selection process of the chi-square test, the single modalities are spliced to obtain a unified fusion feature. Finally, the fusion data features output by LSTM are used as the input of the classifier LIBSVM to realize the final emotion recognition. The experimental results show that the recognition accuracy of the proposed method on the MOSI and MELD datasets are 87.56 and 90.06%, respectively, which are better than other comparison methods. It has laid a certain theoretical foundation for the application of multimodal fusion in emotion recognition.

Download Full-text