This article describes how to use heterogeneous information in speech enhancement. In most of the current speech enhancement systems, clean speeches are recovered only from the signals collected by acoustic microphones, which will be greatly affected by the acoustic noises. However, heterogeneous information from different kinds of sensors, which is usually called the “multi-stream,” are seldom used in speech enhancement because the speech waveforms cannot be recovered from the signals provided by many kinds of sensors. In this article, the authors propose a new model-based multi-stream speech enhancement framework that can make use of the heterogeneous information provided by the signals from different kinds of sensors even when some of them are not directly related to the speech waveform. Then a new speech enhancement scheme using the acoustic and throat microphone recordings is also proposed based on the new speech enhancement framework. Experimental results show that the proposed scheme outperforms several single-stream speech enhancement methods in different noisy environments.
Not 3D Re-ID: Simple Single Stream 2D Convolution for Robust Video Re-identification
