Compression of Deep Learning Models for Text: A Survey

In recent years, the fields of natural language processing (NLP) and information retrieval (IR) have made tremendous progress thanks to deep learning models like Recurrent Neural Networks (RNNs), Gated Recurrent Units (GRUs) and Long Short-Term Memory (LSTMs) networks, and Transformer [ 121 ] based models like Bidirectional Encoder Representations from Transformers (BERT) [ 24 ], Generative Pre-training Transformer (GPT-2) [ 95 ], Multi-task Deep Neural Network (MT-DNN) [ 74 ], Extra-Long Network (XLNet) [ 135 ], Text-to-text transfer transformer (T5) [ 96 ], T-NLG [ 99 ], and GShard [ 64 ]. But these models are humongous in size. On the other hand, real-world applications demand small model size, low response times, and low computational power wattage. In this survey, we discuss six different types of methods (Pruning, Quantization, Knowledge Distillation (KD), Parameter Sharing, Tensor Decomposition, and Sub-quadratic Transformer-based methods) for compression of such models to enable their deployment in real industry NLP projects. Given the critical need of building applications with efficient and small models, and the large amount of recently published work in this area, we believe that this survey organizes the plethora of work done by the “deep learning for NLP” community in the past few years and presents it as a coherent story.

Small but visible: Predicting rare bryophyte distribution and richness patterns using remote sensing-based ensembles of small models

In Canadian boreal forests, bryophytes represent an essential component of biodiversity and play a significant role in ecosystem functioning. Despite their ecological importance and sensitivity to disturbances, bryophytes are overlooked in conservation strategies due to knowledge gaps on their distribution, which is known as the Wallacean shortfall. Rare species deserve priority attention in conservation as they are at a high risk of extinction. This study aims to elaborate predictive models of rare bryophyte species in Canadian boreal forests using remote sensing-derived predictors in an Ensemble of Small Models (ESMs) framework. We hypothesize that high ESMs-based prediction accuracy can be achieved for rare bryophyte species despite their low number of occurrences. We also assess if there is a spatial correspondence between rare and overall bryophyte richness patterns. The study area is located in western Quebec and covers 72,292 km2. We selected 52 bryophyte species with <30 occurrences from a presence-only database (214 species, 389 plots in total). ESMs were built from Random Forest and Maxent techniques using remote sensing-derived predictors related to topography and vegetation. Lee’s L statistic was used to assess and map the spatial relationship between rare and overall bryophyte richness patterns. ESMs yielded poor to excellent prediction accuracy (AUC > 0.5) for 73% of the modeled species, with AUC values > 0.8 for 19 species, which confirmed our hypothesis. In fact, ESMs provided better predictions for the rarest bryophytes. Likewise, our study revealed a spatial concordance between rare and overall bryophyte richness patterns in different regions of the study area, which have important implications for conservation planning. This study demonstrates the potential of remote sensing for assessing and making predictions on inconspicuous and rare species across the landscape and lays the basis for the eventual inclusion of bryophytes into sustainable development planning.

A Reflection on Postwar Neoclassical Economics: The Shift from General Equilibrium Theory to the New Microeconomic Theories

The theoretical focus of neoclassical economics experienced a significant change in the 1970s–1980s. General equilibrium theory lost its dominant position in theoretical economic studies, with its role of setting the research agenda taken over by what this article calls the “new microeconomic theories,” principally decision theory, game theory, and mechanism design. Mainstream economists, post-Keynesians, and historians of economic thought each give a different explanation of the hows and whys of that change, but all miss some critical methodological implications. That change, as this article discusses, shows that neoclassical economics has turned from “grand theory” toward “small models” with empirically delimited utility and that the ideology of marketism lacks a valid scientific foundation. This interpretation can help illuminate the deeper dynamics of the postwar development of neoclassical economics and provide insights for a new political economy that can come to grips with political-economic practices that cannot be fully grasped by the neoclassical tradition.

The impact of sustainability on fire safety

<p>There is a growing demand for building green buildings that are perceived to have benefits environmentally through promoting recycling, energy efficiency and efficient use of resources. The green movement has also led to innovative technologies that are focused on reducing cost. However, the fire safety industry has concerns with the use of certain technologies that create passages for smoke and fire to spread such as passive ventilation or materials that can burn severely and release large amount of toxins. The benefit of this research is to determine which features are high risk and are commonly used. The aim of this research is to investigate whether sustainable or green features have an influence on fire safety in commercial buildings and determine which feature or features would have the most significant implications for building safety in regards to tenability. A detailed investigation was done on passive ventilation such as double skin facade and the thesis also briefly discusses other green features and their implications. There were two methods used to collect data. The first was a qualitative study done through sending out surveys to fire engineers to rate and rank the most significant features that have negative implications for fire safety in reference to the New Zealand Building Code Fire Safety Section criteria and objectives. Then, a one hour interview was carried out to determine the reason behind the engineers’ choice and their perceptions. The results from the surveys and the interviews were that double skin facade and atrium were ranked the most significant. The surveys established double skin facade has the highest ranking in terms of the worst feature, and the fire engineers reinforced that double skin facade needs to be studied as there is not enough research that have gone into this feature. While atrium issues are known and mitigation measures are well developed. A subsequent analysis for only double skin facade is conducted using Fire Dynamics Simulator (FDS) because little literature is found in regards to fire safety and double skin facade. FDS was used to simulate 14 small models and 2 large models for the best and worst scenarios of DSF. Each of the 14 models, one to three parameters are changed as part of the sensitivity study to determine which parameter have the most and least effect on fire safety in term of Carbon Monoxide (CO) and visibility. The issues the engineers raised and the mitigation measures were modelled, because the engineers had stated their opinions not facts. The output results from FDS illustrated that it is essential that the system shuts off in a fire event to prevent smoke spread to upper floors, which is the same mitigation measure that were emphasised at the interviews.</p>

The impact of sustainability on fire safety

<p>There is a growing demand for building green buildings that are perceived to have benefits environmentally through promoting recycling, energy efficiency and efficient use of resources. The green movement has also led to innovative technologies that are focused on reducing cost. However, the fire safety industry has concerns with the use of certain technologies that create passages for smoke and fire to spread such as passive ventilation or materials that can burn severely and release large amount of toxins. The benefit of this research is to determine which features are high risk and are commonly used. The aim of this research is to investigate whether sustainable or green features have an influence on fire safety in commercial buildings and determine which feature or features would have the most significant implications for building safety in regards to tenability. A detailed investigation was done on passive ventilation such as double skin facade and the thesis also briefly discusses other green features and their implications. There were two methods used to collect data. The first was a qualitative study done through sending out surveys to fire engineers to rate and rank the most significant features that have negative implications for fire safety in reference to the New Zealand Building Code Fire Safety Section criteria and objectives. Then, a one hour interview was carried out to determine the reason behind the engineers’ choice and their perceptions. The results from the surveys and the interviews were that double skin facade and atrium were ranked the most significant. The surveys established double skin facade has the highest ranking in terms of the worst feature, and the fire engineers reinforced that double skin facade needs to be studied as there is not enough research that have gone into this feature. While atrium issues are known and mitigation measures are well developed. A subsequent analysis for only double skin facade is conducted using Fire Dynamics Simulator (FDS) because little literature is found in regards to fire safety and double skin facade. FDS was used to simulate 14 small models and 2 large models for the best and worst scenarios of DSF. Each of the 14 models, one to three parameters are changed as part of the sensitivity study to determine which parameter have the most and least effect on fire safety in term of Carbon Monoxide (CO) and visibility. The issues the engineers raised and the mitigation measures were modelled, because the engineers had stated their opinions not facts. The output results from FDS illustrated that it is essential that the system shuts off in a fire event to prevent smoke spread to upper floors, which is the same mitigation measure that were emphasised at the interviews.</p>

The present is the key to the paleo-past: Charles R. Knight’s reconstruction of extinct beasts for the Field Museum, Chicago

ABSTRACT Although he was legally blind, Charles R. Knight (1874–1953) established himself as the premier paleontological artist in the early 1900s. When the Field Museum, Chicago, commissioned a series of large paintings to document the evolution of life, Knight was the obvious choice. Knight considered himself an artist guided by science; he researched and illustrated living animals and modern landscapes to better understand and represent extinct life forms within their paleoecosystems. Knight began the process by examining fossil skeletons; he then constructed small models to recreate the animals’ life anatomy and investigate lighting. Once details were finalized, Knight supervised assistants to transfer the study painting to the final mural. The Field Museum mural process, a monumental task of translating science into public art, was accompanied by a synergistic tension between Knight, who wanted full control over his artwork, and the museum’s scientific staff; the correct position of an Eocene whale’s tail—whether uplifted or not—documents a critical example. Although modern scientific understanding has rendered some of Knight’s representations obsolete, the majority of his 28 murals remain on display in the Field Museum’s Evolving Planet exhibit. Museum educators contrast these murals with contemporary paleontological knowledge, thereby demonstrating scientific progress for better public understanding of the nature of science.

Ensemble of Small Models As a Tool For Alien Invasive Species Management Planning: Evaluation of Vespa Velutina (Hymenoptera: Vespidae) Under Mediterranean Island Conditions

Ecological niche models have proved to be a powerful tool in assessing invasiveness risk of alien species, allowing the optimization of control strategies. Vespa velutina (Hymenoptera: Vespidae) is an invasive species with strong ecological, economical and health impacts in Europe after it first report in France in 2004. It was detected for the first time in a Mediterranean island (Mallorca, Balearic Islands, Spain) in 2015, where a single nest was found in the northwest of the island. Immediately a control plan was implemented. In this study, we analysed 30 occurrence data in Mallorca island to assess the suitability distribution predicted to Mediterranean island conditions using ensemble of small models. We obtained high values of AUC (0.9165), Somers’ D (0.8331), Boyce (0.7611) and TSS (0.7754) as quality parameters of the final ensembled model. We show for the first time that there are suitable areas where this species can expand and stablish, mainly in steeper slopes and low isothermality zones. Likewise, the distribution suitability of V. velutina for other Mediterranean islands (Ibiza, Formentera, Menorca, Corsica, Sardinia, Sicily, Crete and Cyprus) was also explored, showing potentially suitable zones. This study provides valuable information regarding the areas in the Mediterranean islands under risk of invasion, and it could be used by both scientists and managers for an early detection and control of the invasive species due to its cost-effectiveness in terms of conservation.

A TensorFlow Approach to Data Analysis for Time Series Forecasting in the Energy-Efficiency Realm

Thanks to advances in smart metering devices (SM), the electricity sector is undergoing a series of changes, among which it is worth highlighting the ability to control the response to all events that occur in the electricity grid with the intention of making it more smart. Predicting electricity consumption data is a key factor for the energy sector in order to create a completely intelligent electricity grid that optimizes consumption and forecasts future energy needs. However, it is currently not enough to give a prediction of energy consumption (EC), but it is also necessary to give the prediction as fast as possible so that the grid can operate in the shortest possible time. An approach for developing EC prediction systems is introduced here by the use of artificial neural networks (ANN). Differently from other research studies on the subject, a divide-and-conquer strategy is used so that the target system’s execution switches from one to another specialized small models that forecast the EC of a building within the time range of one hour. By simultaneously processing a large amount of data and models, a consequence of implementing them in parallel with TensorFlow on GPUs, the training procedure proposed here increases the performance of the classic time series prediction methods, which are based on ANN. Leveraging the latest generation of ANN techniques and new GPU-based architectures, correct EC predictions can be obtained and, as the experimentation carried out in this work shows, such predictions can be obtained quickly. The obtained results in this study show a promising way for speeding up big data processing of building’s monitoring data to achieve energy efficiency.

Efficient Binarized Convolutional Layers for Visual Inspection Applications on Resource-Limited FPGAs and ASICs

There has been a recent surge in publications related to binarized neural networks (BNNs), which use binary values to represent both the weights and activations in deep neural networks (DNNs). Due to the bitwise nature of BNNs, there have been many efforts to implement BNNs on ASICs and FPGAs. While BNNs are excellent candidates for these kinds of resource-limited systems, most implementations still require very large FPGAs or CPU-FPGA co-processing systems. Our work focuses on reducing the computational cost of BNNs even further, making them more efficient to implement on FPGAs. We target embedded visual inspection tasks, like quality inspection sorting on manufactured parts and agricultural produce sorting. We propose a new binarized convolutional layer, called the neural jet features layer, that learns well-known classic computer vision kernels that are efficient to calculate as a group. We show that on visual inspection tasks, neural jet features perform comparably to standard BNN convolutional layers while using less computational resources. We also show that neural jet features tend to be more stable than BNN convolution layers when training small models.