A joint manifold leaning-based framework for heterogeneous upstream data fusion

A joint manifold learning fusion (JMLF) approach is proposed for nonlinear or mixed sensor modalities with large streams of data. The multimodal sensor data are stacked to form joint manifolds, from which the embedded low intrinsic dimensionalities are discovered for moving targets. The intrinsic low dimensionalities are mapped to resolve the target locations. The JMLF framework is tested on digital imaging and remote sensing image generation scenes with mid-wave infrared (WMIR) data augmented with distributed radio-frequency (RF) Doppler data. Eight manifold learning methods are explored to train the system with the neighborhood preserving embedding showing promise for robust target tracking using video–radio-frequency fusion. The JMLF method shows a 93% improved accuracy as compared to a standard target tracking (e.g., Kalman-filter based) approach.