Optical flow estimation using the Fisher–Rao metric

The optical flow in an event camera is estimated using measurements in the address event representation (AER). Each measurement consists of a pixel address and the time at which a change in the pixel value equalled a given fixed threshold. The measurements in a small region of the pixel array and within a given window in time are approximated by a probability distribution defined on a finite set. The distributions obtained in this way form a three dimensional family parameterized by the pixel addresses and by time. Each parameter value has an associated Fisher–Rao matrix obtained from the Fisher–Rao metric for the parameterized family of distributions. The optical flow vector at a given pixel and at a given time is obtained from the eigenvector of the associated Fisher–Rao matrix with the least eigenvalue. The Fisher–Rao algorithm for estimating optical flow is tested on eight datasets, of which six have ground truth optical flow. It is shown that the Fisher–Rao algorithm performs well in comparison with two state of the art algorithms for estimating optical flow from AER measurements.

A long time ago, in a galaxy far, far away: How temporal are episodic contents?

A prominent feature of mental event (i.e. ‘episodic’) simulations is their temporality: human adults can generate episodic representations directed towards the past or the future. The ability to entertain event representations with different temporal orientations allows these representations to play various cognitive roles. Here, we investigated how the temporal orientation of imagined events relates to the contents (i.e. ‘what is happening’) of these events. Is the temporal orientation of an episode part of its contents? Or are the processes for assigning temporality to an event representation distinct from those generating its contents? In three experiments (N = 360), we asked participants to generate and later recall a series of imagined events differing in (1) location (indoors vs. outdoors), (2) time of day (daytime vs. nighttime), (3) temporal orientation (past vs. future), and (4) weekday (Monday vs. Friday). We then tested to what extent successful recall of episodic content (i.e. (1) and (2)) would predict recall of temporality and/or weekday information. Results showed that while recall of temporal orientation was predicted by content recall, weekday recall was not. However, temporal orientation was only weakly integrated with episodic contents. This finding suggests that episodic simulations are unlikely to be intrinsically temporal in nature. Instead, similar to other forms of temporal information, temporal orientation might be determined from such contents by reconstructive post-retrieval processes. These results have implications for how the human ability to ‘mentally travel’ in time is cognitively implemented.

The auditory brain in action: Intention determines predictive processing in the auditory system—A review of current paradigms and findings

According to the ideomotor theory, action may serve to produce desired sensory outcomes. Perception has been widely described in terms of sensory predictions arising due to top-down input from higher order cortical areas. Here, we demonstrate that the action intention results in reliable top-down predictions that modulate the auditory brain responses. We bring together several lines of research, including sensory attenuation, active oddball, and action-related omission studies: Together, the results suggest that the intention-based predictions modulate several steps in the sound processing hierarchy, from preattentive to evaluation-related processes, also when controlling for additional prediction sources (i.e., sound regularity). We propose an integrative theoretical framework—the extended auditory event representation system (AERS), a model compatible with the ideomotor theory, theory of event coding, and predictive coding. Initially introduced to describe regularity-based auditory predictions, we argue that the extended AERS explains the effects of action intention on auditory processing while additionally allowing studying the differences and commonalities between intention- and regularity-based predictions—we thus believe that this framework could guide future research on action and perception.

The neuromorphic Mosaic: re-configurable in-memory small-world graphs

Thanks to their non-volatile and multi-bit properties, memristors have been extensively used as synaptic weight elements in neuromorphic architectures. However, their use to define and re-program the network connectivity has been overlooked. Here, we propose, implement and experimentally demonstrate Mosaic, a neuromorphic architecture based on a systolic array of memristor crossbars. For the first time, we use distributed non-volatile memristors not only for computation, but also for routing (i.e., to define the network connectivity). Mosaic is particularly well-suited for the implementation of re-configurable small-world graphical models, with dense local and sparse global connectivity - found extensively in the brain. We mathematically show that, as the networks scale up, the Mosaic requires less memory than in conventional memristor approaches. We map a spiking recurrent neural network on the Mosaic to solve an Electrocardiogram (ECG) anomaly detection task. While the performance is either equivalent or better than software models, the advantage of the Mosaic was clearly seen in respective one and two orders of magnitude reduction in energy requirements, compared to a micro-controller and address-event representation-based processor. Mosaic promises to open up a new approach to designing neuromorphic hardware based on graph-theoretic principles with less memory and energy.

TrackAER: Real-Time Event-Based Particle Tracking

Event-based cameras (Lichtsteiner et al., 2008; Posch et al., 2010; Gallego et al., 2020) operate fundamentally different from frame-based cameras: Each pixel of the sensor array reacts asynchronously to relative brightness changes creating a sequential stream of events in address-event representation (AER). Each event is defined by a microsecond-accurate time stamp, the pixel position and a binary polarity indicating a relative increase or decrease of light intensity. Thus, event-based cameras only sense changes in a scenery while effectively suppressing static, redundant information. This renders the camera technology promising also for flow diagnostics. In established approaches like PIV or PTV vast amounts of data are generated, only for a large part of redundant information to be eliminated in data post-processing. In contrast, eventbased cameras effectively compress the data stream already at the source. To make full use of this potential, new data processing algorithms are needed since event-based cameras do not generate conventional framebased data. This work utilizes an event-based camera to identify and track flow tracers such as helium-filled soap bubbles (HFSBs) with real-time visual feedback in measurement volumes of the order of several cubic meters.

Causal Reasoning and Event Cognition as Evolutionary Determinants of Language Structure

The aim of this article is to provide an evolutionarily grounded explanation of central aspects of the structure of language. It begins with an account of the evolution of human causal reasoning. A comparison between humans and non-human primates suggests that human causal cognition is based on reasoning about the underlying forces that are involved in events, while other primates hardly understand external forces. This is illustrated by an analysis of the causal cognition required for early hominin tool use. Second, the thinking concerning forces in causation is used to motivate a model of human event cognition. A mental representation of an event contains two vectors representing a cause as well as a result but also entities such as agents, patients, instruments and locations. The fundamental connection between event representations and language is that declarative sentences express events (or states). The event structure also explains why sentences are constituted of noun phrases and verb phrases. Finally, the components of the event representation show up in language, where causes and effects are expressed by verbs, agents and patients by nouns (modified by adjectives), locations by prepositions, etc. Thus, the evolution of the complexity of mental event representations also provides insight into the evolution of the structure of language.