Compression of Deep Learning Models for Text: A Survey

In recent years, the fields of natural language processing (NLP) and information retrieval (IR) have made tremendous progress thanks to deep learning models like Recurrent Neural Networks (RNNs), Gated Recurrent Units (GRUs) and Long Short-Term Memory (LSTMs) networks, and Transformer [ 121 ] based models like Bidirectional Encoder Representations from Transformers (BERT) [ 24 ], Generative Pre-training Transformer (GPT-2) [ 95 ], Multi-task Deep Neural Network (MT-DNN) [ 74 ], Extra-Long Network (XLNet) [ 135 ], Text-to-text transfer transformer (T5) [ 96 ], T-NLG [ 99 ], and GShard [ 64 ]. But these models are humongous in size. On the other hand, real-world applications demand small model size, low response times, and low computational power wattage. In this survey, we discuss six different types of methods (Pruning, Quantization, Knowledge Distillation (KD), Parameter Sharing, Tensor Decomposition, and Sub-quadratic Transformer-based methods) for compression of such models to enable their deployment in real industry NLP projects. Given the critical need of building applications with efficient and small models, and the large amount of recently published work in this area, we believe that this survey organizes the plethora of work done by the “deep learning for NLP” community in the past few years and presents it as a coherent story.

Scene Context Impairs Perception of Semantically Congruent Objects

Visual scene context is well-known to facilitate the recognition of scene-congruent objects. Interestingly, however, according to predictive-processing accounts of brain function, scene congruency may lead to reduced (rather than enhanced) processing of congruent objects, compared with incongruent ones, because congruent objects elicit reduced prediction-error responses. We tested this counterintuitive hypothesis in two online behavioral experiments with human participants ( N = 300). We found clear evidence for impaired perception of congruent objects, both in a change-detection task measuring response times and in a bias-free object-discrimination task measuring accuracy. Congruency costs were related to independent subjective congruency ratings. Finally, we show that the reported effects cannot be explained by low-level stimulus confounds, response biases, or top-down strategy. These results provide convincing evidence for perceptual congruency costs during scene viewing, in line with predictive-processing theory.

Differences in the transient responses of individual glaciers: a case study of the Cascade Mountains of Washington State, USA

Mountain glaciers have response times that govern retreat due to anthropogenic climate change. We use geometric attributes to estimate individual response times for 383 glaciers in the Cascade mountain range of Washington State, USA. Approximately 90% of estimated response times are between 10 and 60 years, with many large glaciers on the short end of this distribution. A simple model of glacier dynamics shows that this range of response times entails consequential differences in recent and ongoing glacier changes: glaciers with decadal response times have nearly kept pace with anthropogenic warming, but those with multi-decadal response times are far from equilibrium, and their additional committed retreat stands well beyond natural variability. These differences have implications for changes in glacier runoff. A simple calculation highlights that transient peaks in area-integrated melt, either at the onset of forcing or due to variations in forcing, depend on the glacier's response time and degree of disequilibrium. We conclude that differences in individual response times should be considered when assessing the state of a population of glaciers and modeling their future response. These differences in response can arise simply from a range of different glacier geometries, and the same basic principles can be expected in other regions as well.

Mental rotation of hands and objects in ageing and Parkinson’s disease: Implications for understanding motor imagery

Motor imagery (MI) supports motor learning and performance, having the potential to be a useful tool for neurorehabilitation. However, MI ability may be impacted by ageing and neurodegeneration, which could limit its therapeutic effectiveness. MI is often assessed through a hand laterality task (HLT), whereby laterality judgements are typically slower for hands presented at orientations corresponding to physically more difficult postures (a “biomechanical constraint” effect). Performance is also found to differ between back and palm views of the hand, suggesting the differential involvement of visual and sensorimotor strategies. While older adults are generally found to be slowed and show increased biomechanical effects, few studies have examined the effects of both ageing and Parkinson’s disease (PD).The present study compared healthy younger (YA), healthy older (OA) and PD groups on HLT performance from both palm and back views, as well as an object-based (letter) mental rotation task. OA and PD groups were slower than YA, particularly when judging laterality from the back view, and exhibited increased biomechanical constraint effects for the palm. While response times were generally similar between OA and PD groups, the PD group showed reduced accuracy in the back view. Moreover, object rotation was slower and less accurate only in the PD group. The results indicate that different mechanisms are involved in mental rotation of hands viewed from the back or palm, consistent with previous findings, and demonstrate particular effects of ageing and PD when judging the back view. Alongside findings from studies of explicit MI, this suggests a greater alteration of visual than kinaesthetic MI with ageing and neurodegeneration, with additional impairment of object-based visual imagery in PD. The findings are also discussed in relation to different perspectives in MI and the integration of visual and kinaesthetic representations.

Neural Substrates of the Drift-Diffusion Model in Brain Disorders

Many studies on the drift-diffusion model (DDM) explain decision-making based on a unified analysis of both accuracy and response times. This review provides an in-depth account of the recent advances in DDM research which ground different DDM parameters on several brain areas, including the cortex and basal ganglia. Furthermore, we discuss the changes in DDM parameters due to structural and functional impairments in several clinical disorders, including Parkinson's disease, Attention Deficit Hyperactivity Disorder (ADHD), Autism Spectrum Disorders, Obsessive-Compulsive Disorder (OCD), and schizophrenia. This review thus uses DDM to provide a theoretical understanding of different brain disorders.

Ferroelectric-based synapses and neurons for neuromorphic computing

The shift towards a distributed computing paradigm, where multiple systems acquire and elaborate data in real-time, leads to challenges that must be met. In particular, it is becoming increasingly essential to compute on the edge of the network, close to the sensor collecting data. The requirements of a system operating on the edge are very tight: power efficiency, low area occupation, fast response times, and on-line learning. Brain-inspired architectures such as Spiking Neural Networks (SNNs) use artificial neurons and synapses that simultaneously perform low-latency computation and internal-state storage with very low power consumption. Still, they mainly rely on standard complementary metal-oxide-semiconductor (CMOS) technologies, making SNNs unfit to meet the aforementioned constraints. Recently, emerging technologies such as memristive devices have been investigated to flank CMOS technology and overcome edge computing systems' power and memory constraints. In this review, we will focus on ferroelectric technology. Thanks to its CMOS-compatible fabrication process and extreme energy efficiency, ferroelectric devices are rapidly affirming themselves as one of the most promising technology for neuromorphic computing. Therefore, we will discuss their role in emulating neural and synaptic behaviors in an area and power-efficient way.

Attentional Orienting in Front and Rear Spaces in a Virtual Reality Discrimination Task

Recent studies on covert attention suggested that the visual processing of information in front of us is different, depending on whether the information is present in front of us or if it is a reflection of information behind us (mirror information). This difference in processing suggests that we have different processes for directing our attention to objects in front of us (front space) or behind us (rear space). In this study, we investigated the effects of attentional orienting in front and rear space consecutive of visual or auditory endogenous cues. Twenty-one participants performed a modified version of the Posner paradigm in virtual reality during a spaceship discrimination task. An eye tracker integrated into the virtual reality headset was used to make sure that the participants did not move their eyes and used their covert attention. The results show that informative cues produced faster response times than non-informative cues but no impact on target identification was observed. In addition, we observed faster response times when the target occurred in front space rather than in rear space. These results are consistent with an orienting cognitive process differentiation in the front and rear spaces. Several explanations are discussed. No effect was found on subjects’ eye movements, suggesting that participants did not use their overt attention to improve task performance.

Tectonic controls on Quaternary landscape evolution in the Ventura basin, southern California, USA, quantified using cosmogenic isotopes and topographic analyses

The quantification of rates for the competing forces of tectonic uplift and erosion has important implications for understanding topographic evolution. Here, we quantify the complex interplay between tectonic uplift, topographic development, and erosion recorded in the hanging walls of several active reverse faults in the Ventura basin, southern California, USA. We use cosmogenic 26Al/10Be isochron burial dating and 10Be surface exposure dating to construct a basin-wide geochronology, which includes burial dating of the Saugus Formation: an important, but poorly dated, regional Quaternary strain marker. Our ages for the top of the exposed Saugus Formation range from 0.36 +0.18/−0.22 Ma to 1.06 +0.23/−0.26 Ma, and our burial ages near the base of shallow marine deposits, which underlie the Saugus Formation, increase eastward from 0.60 +0.05/−0.06 Ma to 3.30 +0.30/−0.41 Ma. Our geochronology is used to calculate rapid long-term reverse fault slip rates of 8.6−12.6 mm yr−1 since ca. 1.0 Ma for the San Cayetano fault and 1.3−3.0 mm yr−1 since ca. 1.0 Ma for the Oak Ridge fault, which are both broadly consistent with contemporary reverse slip rates derived from mechanical models driven by global positioning system (GPS) data. We also calculate terrestrial cosmogenic nuclide (TCN)-derived, catchment-averaged erosion rates that range from 0.05−1.14 mm yr−1 and discuss the applicability of TCN-derived, catchment-averaged erosion rates in rapidly uplifting, landslide-prone landscapes. We compare patterns in erosion rates and tectonic rates to fluvial response times and geomorphic landscape parameters to show that in young, rapidly uplifting mountain belts, catchments may attain a quasi-steady-state on timescales of <105 years even if catchment-averaged erosion rates are still adjusting to tectonic forcing.

Targeted Memory Reactivation during Mind-Wandering in Younger and Older Adults

Mind-wandering (MW) is a universal cognitive process that is estimated to comprise ~30% of our everyday thoughts. Despite its prevalence, the functional utility of MW remains a scientific blind spot. The present study sought to investigate whether MW serves a functional role in cognition. Specifically, we investigated whether MW contributes to memory consolidation processes, and if age differences in the ability to reactivate episodic memories during MW may contribute to age-related declines in episodic memory. Younger and older adults encoded paired associates, received targeted reactivation cues during an interval filled with a task which promotes MW, and were tested on their memory for the cued and uncued stimuli from the initial encoding task. Thought probes were presented during the retention (MW) interval to assess participants’ thought contents. Across three experiments, we compared the effect of different cue modalities (i.e., auditory, visual) on cued recall performance, and examined both correct retrieval response times as well as accuracy. Across experiments, there was evidence that stimuli that were cued during the MW task were correctly retrieved more quickly than uncued stimuli and that this effect was more robust for younger adults than older adults. Additionally, the more MW a participant reported during the retention interval, the stronger the cueing effect they produced during retrieval. The results from these experiments are interpreted within a retrieval facilitation framework wherein cues serve to reactivate the earlier traces during MW, and this reactivation benefits retrieval speed for cued items as compared to uncued items.