Accelerating DNN Training Through Selective Localized Learning

Training Deep Neural Networks (DNNs) places immense compute requirements on the underlying hardware platforms, expending large amounts of time and energy. We propose LoCal+SGD, a new algorithmic approach to accelerate DNN training by selectively combining localized or Hebbian learning within a Stochastic Gradient Descent (SGD) based training framework. Back-propagation is a computationally expensive process that requires 2 Generalized Matrix Multiply (GEMM) operations to compute the error and weight gradients for each layer. We alleviate this by selectively updating some layers' weights using localized learning rules that require only 1 GEMM operation per layer. Further, since localized weight updates are performed during the forward pass itself, the layer activations for such layers do not need to be stored until the backward pass, resulting in a reduced memory footprint. Localized updates can substantially boost training speed, but need to be used judiciously in order to preserve accuracy and convergence. We address this challenge through a Learning Mode Selection Algorithm, which gradually selects and moves layers to localized learning as training progresses. Specifically, for each epoch, the algorithm identifies a Localized→SGD transition layer that delineates the network into two regions. Layers before the transition layer use localized updates, while the transition layer and later layers use gradient-based updates. We propose both static and dynamic approaches to the design of the learning mode selection algorithm. The static algorithm utilizes a pre-defined scheduler function to identify the position of the transition layer, while the dynamic algorithm analyzes the dynamics of the weight updates made to the transition layer to determine how the boundary between SGD and localized updates is shifted in future epochs. We also propose a low-cost weak supervision mechanism that controls the learning rate of localized updates based on the overall training loss. We applied LoCal+SGD to 8 image recognition CNNs (including ResNet50 and MobileNetV2) across 3 datasets (Cifar10, Cifar100, and ImageNet). Our measurements on an Nvidia GTX 1080Ti GPU demonstrate upto 1.5× improvement in end-to-end training time with ~0.5% loss in Top-1 classification accuracy.

Linking K-12 STEM Pedagogy to Local Contexts: A Scoping Review of Benefits and Limitations

Student engagement and learning in science, technology, engineering and mathematics (STEM) fields in primary and secondary schools is increasingly being emphasized as the importance of STEM skills for future careers is realized. Localized learning has been identified as a group of pedagogical approaches that may enhance learning in STEM by making the relevance of STEM clear to students and providing stronger connections to students’ lives and contexts. This paper reports on a scoping review that was conducted to identify the benefits and limitations of localized learning in primary and secondary school STEM disciplines. A secondary aim of the review was to identify strategies that increase the effectiveness of localized learning these disciplines. Following literature searches of four databases, 1923 articles were identified. Twenty-five studies met the inclusion criteria. Potential benefits of localized learning included increases in enjoyment of STEM, improvements in learning, more positive STEM career aspirations, and development of transferable skills. The main challenges of these pedagogical approaches were time restrictions and lack of community involvement. Strategies for enhancing the impact of localized pedagogy included professional development for teachers (in STEM content knowledge, integration of localized pedagogy, and capacity to address socio-scientific issues), integration of technology, whole-school implementation of the pedagogical approach, and integration of the wider community into STEM education. These findings provide support for localized learning as an effective pedagogical approach to enhance STEM learning in schools, while emphasizing the critical roles of teachers and communities in supporting students to realize the relevance of STEM in their lives.

Localized learning

Purpose This paper aims to review the latest management developments across the globe and pinpoint practical implications from cutting-edge research and case studies. Design/methodology/approach This briefing is prepared by an independent writer who adds his/her own impartial comments and places the articles in context. Findings One of the more light-hearted interpretations of how to define organizational culture is to simply say. “It’s the way we do things around here”. This is illuminating and frustrating in equal measure, as while it does contain a kernel of truth - understanding how and why people take the positions and actions they do is central to the question of culture – it is also rather glib and is simply true of everywhere you might ask that question. It also points to a certain wariness and even defiance on behalf of the people answering the question in such a way, as if to challenge the newcomer into accepting how their world operates, and that it is never going to change. Practical implications This paper provides strategic insights and practical thinking that have influenced some of the world’s leading organizations. Originality/value The briefing saves busy executives and researchers hours of reading time by selecting only the very best, most pertinent information and presenting it in a condensed and easy-to-digest format.

Online Learning of Single- and Multivalued Functions with an Infinite Mixture of Linear Experts

We present a supervised learning algorithm for estimation of generic input-output relations in a real-time, online fashion. The proposed method is based on a generalized expectation-maximization approach to fit an infinite mixture of linear experts (IMLE) to an online stream of data samples. This probabilistic model, while not fully Bayesian, can efficiently choose the number of experts that are allocated to the mixture, this way effectively controlling the complexity of the resulting model. The result is an incremental, online, and localized learning algorithm that performs nonlinear, multivariate regression on multivariate outputs by approximating the target function by a linear relation within each expert input domain and that can allocate new experts as needed. A distinctive feature of the proposed method is the ability to learn multivalued functions: one-to-many mappings that naturally arise in some robotic and computer vision learning domains, using an approach based on a Bayesian generative model for the predictions provided by each of the mixture experts. As a consequence, it is able to directly provide forward and inverse relations from the same learned mixture model. We conduct an extensive set of experiments to evaluate the proposed algorithm performance, and the results show that it can outperform state-of-the-art online function approximation algorithms in single-valued regression, while demonstrating good estimation capabilities in a multivalued function approximation context.