Topological Uncertainty: Monitoring Trained Neural Networks through Persistence of Activation Graphs

Although neural networks are capable of reaching astonishing performance on a wide variety of contexts, properly training networks on complicated tasks requires expertise and can be expensive from a computational perspective. In industrial applications, data coming from an open-world setting might widely differ from the benchmark datasets on which a network was trained. Being able to monitor the presence of such variations without retraining the network is of crucial importance. In this paper, we develop a method to monitor trained neural networks based on the topological properties of their activation graphs. To each new observation, we assign a Topological Uncertainty, a score that aims to assess the reliability of the predictions by investigating the whole network instead of its final layer only as typically done by practitioners. Our approach entirely works at a post-training level and does not require any assumption on the network architecture, optimization scheme, nor the use of data augmentation or auxiliary datasets; and can be faithfully applied on a large range of network architectures and data types. We showcase experimentally the potential of Topological Uncertainty in the context of trained network selection, Out-Of-Distribution detection, and shift-detection, both on synthetic and real datasets of images and graphs.

A meta-analysis on uncertainty monitoring in four non-primate animal species: Pigeons, rats, large-billed crows, and bees

Humans have the metacognitive capacity to be aware of what they do and do not know. While uncertainty monitoring has long been regarded as uniquely human, researchers in search of the polygenetic root of this ability have gathered evidence that primate species possess functional features parallel to humans. However, there were no systematic studies that quantitively take into account of extant data for these non-primate animals. Through a meta-analysis, we collected published data reported in 11 articles from 55 individual non-primate animals spanning over four species on the opt-out paradigm, the most prevailing paradigms used to test nonhuman animals uncertainty monitoring. We used chosen-forced advantage and opt-out rate to quantify animals performance results for computing the aggregated effect size for this literature. We found that these four NPA species process a significantly positive effect size for both scores and identified the moderators that have contributed to the inconsistencies across these studies. Implications for theories on metacognition are discussed.

The Neural System of Metacognition Accompanying Decision-Making in The Prefrontal Cortex

Decision-making is usually accompanied by metacognition, through which a decision maker monitors the decision uncertainty and consequently revises the decision, even prior to feedback. However, the neural mechanisms of metacognition remain controversial: one theory proposes that metacognition coincides the decision-making process; and another addresses that it entails an independent neural system in the prefrontal cortex (PFC). Here we devised a novel paradigm of “decision-redecision” to investigate the metacognition process in redecision, in comparison with the decision process. We here found that the anterior PFC, including dorsal anterior cingulate cortex (dACC) and lateral frontopolar cortex (lFPC), were exclusively activated after the initial decisions. dACC was involved in decision uncertainty monitoring, whereas lFPC was involved in decision adjustment controlling, subject to control demands of the tasks. Our findings support that the PFC is essentially involved in metacognition and further suggest that functions of the PFC in metacognition are dissociable.

Infants ask for help when they know they don’t know

Uncertainty monitoring is a core property of metacognition, allowing individuals to adapt their decision-making strategies depending on the state of their knowledge. Although it has been argued that other animals share these metacognitive abilities, only humans seem to possess the ability to explicitly communicate their own uncertainty to others. It remains unknown whether this capacity is present early in development, or whether it emerges later with the ability to verbally report one’s own mental states. Here, using a nonverbal memory-monitoring paradigm, we show that 20-month-olds can monitor and report their own uncertainty. Infants had to remember the location of a hidden toy before pointing to indicate where they wanted to recover it. In an experimental group, infants were given the possibility to ask for help through nonverbal communication when they had forgotten the toy location. Compared with a control group in which infants had no other option but to decide by themselves, infants given the opportunity to ask for help used this option strategically to improve their performance. Asking for help was used selectively to avoid making errors and to decline difficult choices. These results demonstrate that infants are able to successfully monitor their own uncertainty and share this information with others to fulfill their goals.