Novelty Detection: A Perspective from Natural Language Processing

The quest for new information is an inborn human trait and has always been quintessential for human survival and progress. Novelty drives curiosity, which in turn drives innovation. In Natural Language Processing (NLP), Novelty Detection refers to finding text that has some new information to offer with respect to whatever is earlier seen or known. With the exponential growth of information all across the web, there is an accompanying menace of redundancy. A considerable portion of the web contents are duplicates, and we need efficient mechanisms to retain new information and filter out redundant ones. However, detecting redundancy at the semantic level and identifying novel text is not straightforward because the text may have less lexical overlap yet convey the same information. On top of that, non-novel/redundant information in a document may have assimilated from multiple source documents, not just one. The problem surmounts when the subject of the discourse is documents, and numerous prior documents need to be processed to ascertain the novelty/non-novelty of the current one in concern. In this work, we build upon our earlier investigations for document-level novelty detection and present a comprehensive account of our efforts towards the problem. We explore the role of pre-trained Textual Entailment (TE) models to deal with multiple source contexts and present the outcome of our current investigations. We argue that a multi-premise entailment task is one close approximation towards identifying semantic-level non-novelty. Our recent approach either performs comparably or achieves significant improvement over the latest reported results on several datasets and across several related tasks (paraphrasing, plagiarism, rewrite). We critically analyze our performance with respect to the existing state-of-the-art and show the superiority and promise of our approach for future investigations. We also present our enhanced dataset TAP-DLND 2.0 and several baselines to the community for further researchon document-level novelty detection.

Modelling novelty detection in the thalamocortical loop

In complex natural environments, sensory systems are constantly exposed to a large stream of inputs. Novel or rare stimuli, which are often associated with behaviorally important events, are typically processed differently than the steady sensory background, which has less relevance. Neural signatures of such differential processing, commonly referred to as novelty detection, have been identified on the level of EEG recordings as mismatch negativity and the level of single neurons as stimulus-specific adaptation. Here, we propose a multi-scale recurrent network with synaptic depression to explain how novelty detection can arise in the whisker-related part of the somatosensory thalamocortical loop. The architecture and dynamics of the model presume that neurons in cortical layer 6 adapt, via synaptic depression, specifically to a frequently presented stimulus, resulting in reduced population activity in the corresponding cortical column when compared with the population activity evoked by a rare stimulus. This difference in population activity is then projected from the cortex to the thalamus and amplified through the interaction between neurons of the primary and reticular nuclei of the thalamus, resulting in spindle-like, rhythmic oscillations. These differentially activated thalamic oscillations are forwarded to cortical layer 4 as a late secondary response that is specific to rare stimuli that violate a particular stimulus pattern. Model results show a strong analogy between this late single neuron activity and EEG-based mismatch negativity in terms of their common sensitivity to presentation context and timescales of response latency, as observed experimentally. Our results indicate that adaptation in L6 can establish the thalamocortical dynamics that produce signatures of SSA and MMN and suggest a mechanistic model of novelty detection that could generalize to other sensory modalities.

A gradient of electrophysiological novelty responses along the human hippocampal long axis.

The hippocampus is implicated in novelty detection, thought to be important for regulating entry of information into long-term memory. Whether electrophysiological responses to novelty differ along the human hippocampal long axis is currently unknown. By recording from electrodes implanted longitudinally in the hippocampus of epilepsy patients, here we show a gradual increase of theta frequency oscillatory power from anterior to posterior in response to unexpected stimuli, superimposed on novelty responses common to all long axis portions. Intracranial event-related potentials (iERPs) were larger for unexpected vs. expected stimuli and demonstrated a polarity inversion between the hippocampal head (HH) and body (HB). We observed stronger theta coherence between HH and hippocampal tail (HT) than between HB and HT, similarly for expected and unexpected stimuli. This was accompanied by theta and alpha traveling waves with surprisingly variable direction of travel characterized by a ~180 degree phase lag between hippocampal poles. Interestingly, this phase lag showed a pronounced phase offset between anterior and middle (HH-HB) hippocampal portions coinciding anatomically with a drop in theta coherence and the novelty iERP polarity inversion. Our findings indicate common response properties along the hippocampal long axis to unexpected stimuli, as well as a multifaceted, non-uniform engagement along the long axis for novelty processing.

Autoencoder-Based Semantic Novelty Detection: Towards Dependable AI-Based Systems

Many autonomous systems, such as driverless taxis, perform safety-critical functions. Autonomous systems employ artificial intelligence (AI) techniques, specifically for environmental perception. Engineers cannot completely test or formally verify AI-based autonomous systems. The accuracy of AI-based systems depends on the quality of training data. Thus, novelty detection, that is, identifying data that differ in some respect from the data used for training, becomes a safety measure for system development and operation. In this study, we propose a new architecture for autoencoder-based semantic novelty detection with two innovations: architectural guidelines for a semantic autoencoder topology and a semantic error calculation as novelty criteria. We demonstrate that such a semantic novelty detection outperforms autoencoder-based novelty detection approaches known from the literature by minimizing false negatives.