Explaining Valence Asymmetries in Value Learning: A Reinforcement Learning Account

To understand how acquired value impacts how we perceive and process stimuli, psychologists have developed the Value Learning Task (VLT; e.g., Raymond & O’Brien, 2009). The task consists of a series of trials in which participants attempt to maximize accumulated winnings as they make choices from a pair of presented images associated with probabilistic win, loss, or no-change outcomes. Despite the task having a symmetric outcome structure for win and loss pairs, people learn win associations better than loss associations (Lin, Cabrera-Haro, & Reuter-Lorenz, 2020). This asymmetry could lead to differences when the stimuli are probed in subsequent tasks, compromising inferences about how acquired value affects downstream processing. We investigate the nature of the asymmetry using a standard error-driven reinforcement learning model with a softmax choice rule. Despite having no special role for valence, the model yields the asymmetry observed in human behavior, whether the model parameters are set to maximize empirical fit, or task payoff. The asymmetry arises from an interaction between a neutral initial value estimate and a choice policy that exploits while exploring, leading to more poorly discriminated value estimates for loss stimuli. We also show how differences in estimated individual learning rates help to explain individual differences in the observed win-loss asymmetries, and how the final value estimates produced by the model provide a simple account of a post-learning explicit value categorization task.

Hubungan motivasi belajar dengan hasil belajar pekerjaan dasar otomotif siswa jurusan TKR SMK Negeri 1 lolak

ARTIKEL HUBUNGAN MOTIVASI BELAJAR DENGAN HASIL BELAJAR PEKERJAAN DASAR OTOMOTIF SISWA JURUSAN TKR SMK NEGERI 1 LOLAK Dr. Eng. Zuldesmi, ST, M, Eng, David O. Mapaliey, ST, M. Eng. Liharman Pintubatu ABSTRAK Pendidikan merupakan salah satu proses kegiatan pembentukan sikap kepribadian, keterampilan dan meningkatkan potensi diri setiap orang untuk menghadapi masa depan. Pada umumnya sikap kepribadian siswa ditentukan oleh pendidikan, pengalaman, dan latihan-latihan yang dilalui sejak kecil. Keadaan di SMK Negeri 1 Lolak bahwa, motivasi belajar siswa masih rendah dalam pembelajaran Pekerjaan dasar otomotif (PDO) terlihat dari aktifitas didalam kelas, kurang antusias dalam belajar dan mengerjakan soal saat diberikan guru, tidak mengerti apa yang akan dipelajari, dan tidak memahami mengapa sesuatu itu perlu dipelajari yang akhirnya kegiatan belajar-mengajar kurang efisien, siswa tidak kondusif pada saat guru menjelaskan. Dengan demikian tujuan yang hendak dicapai dalam penelitian ini yaitu Mengetahui hubungan motivasi belajar dengan hasil mengajar pekerjaan dasar otomotif Siswa Jurusan TKR SMK Negeri 1 Lolak Tahun Pembelajaran 2019/2020. Metode penelitian yang digunakan adalah metode penelitian kualitatif. Dan hasil penelitian: diperoleh hasil Tingkat motivasi belajar siswa bernilai 3,09 dengan nilai rata-rata tertinggi menunjukkan pada ulet dan tidak putus asa 3,29, nilai rata-rata terendah berada pada tekun dalam menghadapi tugas 2,94. Hal ini menunjukkan bahwa motivasi belajar siswa di SMK Negeri 1 Lolak dikategorikan baik. “Kata kunci: Hubungan motivasi dengan hasil belajar pekerjaan dasar otomotif THE RELATIONSHIPS BETWEEN LEARNING MOTIVATION WITH THE RESULT OF LEARNING THE BASIC AUTOMOTIVE WORK OF TKR STUDENTS IN SMK NEGERI 1 LOLAK Dr. Eng. Zuldesmi, st, m, eng, David o. mapaliey, st, m. eng Liharman Pintubatu ABSTRACT Education is one of the processes of shaping students' personality, skills and improving each person's potential for what lies ahead. student's personality is generally determined by knowledge, experiences, and exercises passed through from infancy. The motivations of students in SMK NEGERI 1 LOLAK in learning basic automotive work is still low. It is seen from the activities in class, the students are often lack of enthusiasm for learning and working on the duty that given by the teacher, they also do not understand what will be learned, and not understanding why it needs to be learned, and those make teaching-learning process is less efficient. Thus the goal that must be achieved in this study is underlying the relationship between motivation of learning with the result of teachingbasic automotive work of TKR students SMK NEGERI 1 LOLAK, 2019/2020. The research method used is qualitative research. And research results: obtained results of the 3.09 value levels of students' learning rates with the highest average rates show on 3.29, the lowest average value being on diligent in facing duties 2.94. This shows that students' learning motivation in SMK Negeri 1 Lolak categorized well. Keywords: The relationship of learning motivations with the result of learning basic automotive work.

Confirmation Bias in the Course of Instructed Reinforcement Learning in Schizophrenia-Spectrum Disorders

A large body of research attributes learning deficits in schizophrenia (SZ) to the systems involved in value representation (prefrontal cortex, PFC) and reinforcement learning (basal ganglia, BG) as well as to the compromised connectivity of these regions. In this study, we employed learning tasks hypothesized to probe the function and interaction of the PFC and BG in patients with SZ-spectrum disorders in comparison to healthy control (HC) subjects. In the Instructed Probabilistic Selection task (IPST), participants received false instruction about one of the stimuli used in the course of probabilistic learning which creates confirmation bias, whereby the instructed stimulus is overvalued in comparison to its real experienced value. The IPST was administered to 102 patients with SZ and 120 HC subjects. We have shown that SZ patients and HC subjects were equally influenced by false instruction in reinforcement learning (RL) probabilistic task (IPST) (p-value = 0.441); however, HC subjects had significantly higher learning rates associated with the process of overcoming cognitive bias in comparison to SZ patients (p-value = 0.018). The behavioral results of our study could be hypothesized to provide further evidence for impairments in the SZ-BG circuitry; however, this should be verified by neurofunctional imaging studies.

Behavioral flexibility is manipulatable and it improves flexibility and problem solving in a new context

Behavioral flexibility, the ability to adapt behavior to new circumstances, is thought to play an important role in a species' ability to successfully adapt to new environments and expand its geographic range. However, flexibility is rarely directly tested in species in a way that would allow us to determine how flexibility works and predictions a species' ability to adapt their behavior to new environments. We use great-tailed grackles (a bird species) as a model to investigate this question because they have rapidly expanded their range into North America over the past 140 years. We attempted to manipulate grackle flexibility using colored tube reversal learning to determine whether flexibility is generalizable across contexts (touchscreen reversal learning and multi-access box), whether it is repeatable within individuals and across contexts, and what learning strategies grackles employ. We found that we were able to manipulate flexibility: birds in the manipulated group took fewer trials to pass criterion with increasing reversal number, and they reversed a color preference in fewer trials by the end of their serial reversals compared to control birds who had only one reversal. Flexibility was repeatable within individuals (reversal), but not across contexts (from reversal to multi-access box). The touchscreen reversal experiment did not appear to measure what was measured in the reversal learning experiment with the tubes, and we speculate as to why. One third of the grackles in the manipulated reversal learning group switched from one learning strategy (epsilon-decreasing where they have a long exploration period) to a different strategy (epsilon-first where they quickly shift their preference). A separate analysis showed that the grackles did not use a particular strategy earlier or later in their serial reversals. Posthoc analyses using a model that breaks down performance on the reversal learning task into different components showed that learning to be attracted to an option (phi) more consistently correlated with reversal performance than the rate of deviating from learned attractions that were rewarded (lambda). This result held in simulations and in the data from the grackles: learning rates in the manipulated grackles doubled by the end of the manipulation compared to control grackles, while the rate of deviation slightly decreased. Grackles with intermediate rates of deviation in their last reversal, independently of whether they had gone through the serial reversal manipulation, solved fewer loci on the plastic and wooden multi-access boxes, and those with intermediate learning rates in their last reversal were faster to attempt a new locus on both multi-access boxes. This investigation allowed us to make causal conclusions rather than relying only on correlations: we manipulated reversal learning, which caused changes in a different flexibility measure (multi-access box switch times) and in an innovativeness measure (multi-access box loci solved), as well as validating that the manipulation had an effect on the cognitive ability we think of as flexibility. Understanding how behavioral flexibility causally relates to other traits will allow researchers to develop robust theory about what behavioral flexibility is and when to invoke it as a primary driver in a given context, such as a rapid geographic range expansion. Given our results, flexibility manipulations could be useful in training threatened and endangered species in how to be more flexible. If such a flexibility manipulation was successful, it could then change their behavior in this and other domains, giving them a better chance of succeeding in human modified environments.

Multiscale regression on unknown manifolds

<abstract><p>We consider the regression problem of estimating functions on $ \mathbb{R}^D $ but supported on a $ d $-dimensional manifold $ \mathcal{M} ~~\subset \mathbb{R}^D $ with $ d \ll D $. Drawing ideas from multi-resolution analysis and nonlinear approximation, we construct low-dimensional coordinates on $ \mathcal{M} $ at multiple scales, and perform multiscale regression by local polynomial fitting. We propose a data-driven wavelet thresholding scheme that automatically adapts to the unknown regularity of the function, allowing for efficient estimation of functions exhibiting nonuniform regularity at different locations and scales. We analyze the generalization error of our method by proving finite sample bounds in high probability on rich classes of priors. Our estimator attains optimal learning rates (up to logarithmic factors) as if the function was defined on a known Euclidean domain of dimension $ d $, instead of an unknown manifold embedded in $ \mathbb{R}^D $. The implemented algorithm has quasilinear complexity in the sample size, with constants linear in $ D $ and exponential in $ d $. Our work therefore establishes a new framework for regression on low-dimensional sets embedded in high dimensions, with fast implementation and strong theoretical guarantees.</p></abstract>

RALR: Random Amplify Learning Rates for Training Neural Networks

It has been shown that the learning rate is one of the most critical hyper-parameters for the overall performance of deep neural networks. In this paper, we propose a new method for setting the global learning rate, named random amplify learning rates (RALR), to improve the performance of any optimizer in training deep neural networks. Instead of monotonically decreasing the learning rate, we expect to escape saddle points or local minima by amplifying the learning rate between reasonable boundary values based on a given probability. Training with RALR rather than conventionally decreasing the learning rate achieves further improvement on networks’ performance without extra consumption. Remarkably, the RALR is complementary with state-of-the-art data augmentation and regularization methods. Besides, we empirically study its performance on image classification tasks, fine-grained classification tasks, object detection tasks, and machine translation tasks. Experiments demonstrate that RALR can bring a notable improvement while preventing overfitting when training deep neural networks. For example, the classification accuracy of ResNet-110 trained on the CIFAR-100 dataset using RALR achieves a 1.34% gain compared with ResNet-110 trained traditionally.

Uncertainty Quantification and Optimization of Deep Learning for Fracture Recognition

Automatic fracture recognition from borehole images or outcrops is applicable for the construction of fractured reservoir models. Deep learning for fracture recognition is subject to uncertainty due to sparse and imbalanced training set, and random initialization. We present a new workflow to optimize a deep learning model under uncertainty using U-Net. We consider both epistemic and aleatoric uncertainty of the model. We propose a U-Net architecture by inserting dropout layer after every "weighting" layer. We vary the dropout probability to investigate its impact on the uncertainty response. We build the training set and assign uniform distribution for each training parameter, such as the number of epochs, batch size, and learning rate. We then perform uncertainty quantification by running the model multiple times for each realization, where we capture the aleatoric response. In this approach, which is based on Monte Carlo Dropout, the variance map and F1-scores are utilized to evaluate the need to craft additional augmentations or stop the process. This work demonstrates the existence of uncertainty within the deep learning caused by sparse and imbalanced training sets. This issue leads to unstable predictions. The overall responses are accommodated in the form of aleatoric uncertainty. Our workflow utilizes the uncertainty response (variance map) as a measure to craft additional augmentations in the training set. High variance in certain features denotes the need to add new augmented images containing the features, either through affine transformation (rotation, translation, and scaling) or utilizing similar images. The augmentation improves the accuracy of the prediction, reduces the variance prediction, and stabilizes the output. Architecture, number of epochs, batch size, and learning rate are optimized under a fixed-uncertain training set. We perform the optimization by searching the global maximum of accuracy after running multiple realizations. Besides the quality of the training set, the learning rate is the heavy-hitter in the optimization process. The selected learning rate controls the diffusion of information in the model. Under the imbalanced condition, fast learning rates cause the model to miss the main features. The other challenge in fracture recognition on a real outcrop is to optimally pick the parental images to generate the initial training set. We suggest picking images from multiple sides of the outcrop, which shows significant variations of the features. This technique is needed to avoid long iteration within the workflow. We introduce a new approach to address the uncertainties associated with the training process and with the physical problem. The proposed approach is general in concept and can be applied to various deep-learning problems in geoscience.

Cyclical Learning Rates (CLR’s) for Improving Training Accuracies and Lowering Computational Cost

Prediction of different lung pathologies using chest X-ray images is a challenging task requiring robust training and testing accuracies. In this article, one-class classifier (OCC) and binary classification algorithms have been tested to classify 14 different diseases (atelectasis, cardiomegaly, consolidation, effusion, edema, emphysema, fibrosis, hernia, infiltration, mass, nodule, pneumonia, pneumothorax and pleural-thickening). We have utilized 3 different neural network architectures (MobileNetV1, Alexnet, and DenseNet-121) with four different optimizers (SGD, Adam, and RMSProp) for comparing best possible accuracies. Cyclical learning rate (CLR), a tuning hyperparameters technique was found to have a faster convergence of the cost towards the minima of cost function. Here, we present a unique approach of utilizing previously trained binary classification models with a learning rate decay technique for re-training models using CLR’s. Doing so, we found significant improvement in training accuracies for each of the selected conditions. Thus, utilizing CLR’s in callback functions seems a promising strategy for image classification problems.

Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularization

Preconditioning and Regularization Enable Faster Reinforcement Learning Natural policy gradient (NPG) methods, in conjunction with entropy regularization to encourage exploration, are among the most popular policy optimization algorithms in contemporary reinforcement learning. Despite the empirical success, the theoretical underpinnings for NPG methods remain severely limited. In “Fast Global Convergence of Natural Policy Gradient Methods with Entropy Regularization”, Cen, Cheng, Chen, Wei, and Chi develop nonasymptotic convergence guarantees for entropy-regularized NPG methods under softmax parameterization, focusing on tabular discounted Markov decision processes. Assuming access to exact policy evaluation, the authors demonstrate that the algorithm converges linearly at an astonishing rate that is independent of the dimension of the state-action space. Moreover, the algorithm is provably stable vis-à-vis inexactness of policy evaluation. Accommodating a wide range of learning rates, this convergence result highlights the role of preconditioning and regularization in enabling fast convergence.