scholarly journals Information Flows of Diverse Autoencoders

Entropy ◽  
2021 ◽  
Vol 23 (7) ◽  
pp. 862
Author(s):  
Sungyeop Lee ◽  
Junghyo Jo
Keyword(s):  
Information Theory ◽  
Deep Learning ◽  
Mutual Information ◽  
Information Transmission ◽  
Test Data ◽  
Learning Process ◽  
Information Flows ◽  
Compression Phase ◽  
Entropy Functional ◽  
Last Stage

Deep learning methods have had outstanding performances in various fields. A fundamental query is why they are so effective. Information theory provides a potential answer by interpreting the learning process as the information transmission and compression of data. The information flows can be visualized on the information plane of the mutual information among the input, hidden, and output layers. In this study, we examine how the information flows are shaped by the network parameters, such as depth, sparsity, weight constraints, and hidden representations. Here, we adopt autoencoders as models of deep learning, because (i) they have clear guidelines for their information flows, and (ii) they have various species, such as vanilla, sparse, tied, variational, and label autoencoders. We measured their information flows using Rényi’s matrix-based α-order entropy functional. As learning progresses, they show a typical fitting phase where the amounts of input-to-hidden and hidden-to-output mutual information both increase. In the last stage of learning, however, some autoencoders show a simplifying phase, previously called the “compression phase”, where input-to-hidden mutual information diminishes. In particular, the sparsity regularization of hidden activities amplifies the simplifying phase. However, tied, variational, and label autoencoders do not have a simplifying phase. Nevertheless, all autoencoders have similar reconstruction errors for training and test data. Thus, the simplifying phase does not seem to be necessary for the generalization of learning.

scholarly journals Comparative performance of mutual information and transfer entropy for analyzing the balance of information flow and energy consumption at synapses

2020 ◽  
Author(s):  
Mireille Conrad ◽  
Renaud B Jolivet
Keyword(s):  
Information Theory ◽  
Mutual Information ◽  
Information Flow ◽  
Experimental Finding ◽  
Brain Regions ◽  
Transfer Entropy ◽  
Information Flows ◽  
Experimental Conditions ◽  
Information Theoretic ◽  
Thalamic Relay

AbstractInformation theory has become an essential tool of modern neuroscience. It can however be difficult to apply in experimental contexts when acquisition of very large datasets is prohibitive. Here, we compare the relative performance of two information theoretic measures, mutual information and transfer entropy, for the analysis of information flow and energetic consumption at synapses. We show that transfer entropy outperforms mutual information in terms of reliability of estimates for small datasets. However, we also show that a detailed understanding of the underlying neuronal biophysics is essential for properly interpreting the results obtained with transfer entropy. We conclude that when time and experimental conditions permit, mutual information might provide an easier to interpret alternative. Finally, we apply both measures to the study of energetic optimality of information flow at thalamic relay synapses in the visual pathway. We show that both measures recapitulate the experimental finding that these synapses are tuned to optimally balance information flowing through them with the energetic consumption associated with that synaptic and neuronal activity. Our results highlight the importance of conducting systematic computational studies prior to applying information theoretic tools to experimental data.Author summaryInformation theory has become an essential tool of modern neuroscience. It is being routinely used to evaluate how much information flows from external stimuli to various brain regions or individual neurons. It is also used to evaluate how information flows between brain regions, between neurons, across synapses, or in neural networks. Information theory offers multiple measures to do that. Two of the most popular are mutual information and transfer entropy. While these measures are related to each other, they differ in one important aspect: transfer entropy reports a directional flow of information, as mutual information does not. Here, we proceed to a systematic evaluation of their respective performances and trade-offs from the perspective of an experimentalist looking to apply these measures to binarized spike trains. We show that transfer entropy might be a better choice than mutual information when time for experimental data collection is limited, as it appears less affected by systematic biases induced by a relative lack of data. Transmission delays and integration properties of the output neuron can however complicate this picture, and we provide an example of the effect this has on both measures. We conclude that when time and experimental conditions permit, mutual information – especially when estimated using a method referred to as the ‘direct’ method – might provide an easier to interpret alternative. Finally, we apply both measures in the biophysical context of evaluating the energetic optimality of information flow at thalamic relay synapses in the visual pathway. We show that both measures capture the original experimental finding that those synapses are tuned to optimally balance information flowing through them with the concomitant energetic consumption associated with that synaptic and neuronal activity.

scholarly journals Confounding Ghost Channels and Causality: A New Approach to Causal Information Flows

2021 ◽  
Author(s):  
Nihat Ay
Keyword(s):  
Information Theory ◽  
Mutual Information ◽  
Causal Chain ◽  
Information Flows ◽  
Basic Requirement ◽  
Conditional Mutual Information ◽  
New Approach ◽  
Causal Information ◽  
Causal Nature ◽  
The Given

AbstractInformation theory provides a fundamental framework for the quantification of information flows through channels, formally Markov kernels. However, quantities such as mutual information and conditional mutual information do not necessarily reflect the causal nature of such flows. We argue that this is often the result of conditioning based on σ-algebras that are not associated with the given channels. We propose a version of the (conditional) mutual information based on families of σ-algebras that are coupled with the underlying channel. This leads to filtrations which allow us to prove a corresponding causal chain rule as a basic requirement within the presented approach.

THE EFFECTIVENESS OF QUANTUM READING MODEL BY PRINTING MEDIA WITH YEN BASIS IN READING CONCEPT LEARNING TO STUDENTS OF GRADE 4TH IN SUMBER REGENCY, CIREBON, SCHOOL ACADEMIC YEAR 2013/2014

2017 ◽  
Vol 3 (2) ◽  
pp. 594
Author(s):  
Mintarsih Danureja ◽  
Tati Hartati
Keyword(s):  
Elementary School ◽  
Test Data ◽  
Concept Learning ◽  
Learning Process ◽  
Alternative Technique ◽  
Purposive Sampling ◽  
Trouble Shooting ◽  
Competence Test ◽  
Academic Year ◽  
Year 2013

This research is intended to know the effectiveness of quantum reading model by printing media with yen basis in reading concept learning to students of grade 4th elementary school. The problems are: profile of students reading concept competences, the learning process of reading concept, the learning process of reading concept by quantum reading model by printing media with yen basis, the effectiveness of quantum reading model by printing media with yen basis in reading concept learning, and students response of quantum reading model by printing media with yen basis.The trouble-shooting is by use quantum reading model by printing media with yen basis by assumption: if students need to do something therefore will be easily to motivate them in learning and get a better result.Population in this research is Grade 4th Students of elementary school at Sumber Regency, Cirebon. The sample is determined by purposive sampling. The objects in this research are quantum reading model by printing media with yen basis and students reading concept competences. Instruments in this research are sheets of questionnaire, sheets of learning observation, and reading concept competence test. Data that are collected are learning process and data of students reading concept competences. Collected data are analyzed by descriptive statistic analyze.Result of this research are: (1) the competences of students reading concept are good, (2) the learning process are done with students actively, (3) quantum reading model by printing media with yen basis can increase students activity and learning result, (4) quantum reading model by printing media with yen basis is effective to use in reading concept learning, and (5) the students response to reading concept learning by use quantum reading model by printing media with yen basis is very good.Base on the result of this research, get to be proposed for teacher to use quantum reading model by printing media with yen basis to increase students reading concept competence as one of alternative technique in learning, because it is able to motivate, grow the students interest, and increase students creativities to increase students reading concept competences.

scholarly journals Representing Deep Neural Networks Latent Space Geometries with Graphs

Algorithms ◽  
2021 ◽  
Vol 14 (2) ◽  
pp. 39
Author(s):  
Carlos Lassance ◽  
Vincent Gripon ◽  
Antonio Ortega
Keyword(s):  
Machine Learning ◽  
Neural Networks ◽  
Deep Learning ◽  
Objective Function ◽  
Learning Process ◽  
Deep Neural Networks ◽  
State Of The Art ◽  
The Core ◽  
Learning Tasks ◽  
Latent Space

Deep Learning (DL) has attracted a lot of attention for its ability to reach state-of-the-art performance in many machine learning tasks. The core principle of DL methods consists of training composite architectures in an end-to-end fashion, where inputs are associated with outputs trained to optimize an objective function. Because of their compositional nature, DL architectures naturally exhibit several intermediate representations of the inputs, which belong to so-called latent spaces. When treated individually, these intermediate representations are most of the time unconstrained during the learning process, as it is unclear which properties should be favored. However, when processing a batch of inputs concurrently, the corresponding set of intermediate representations exhibit relations (what we call a geometry) on which desired properties can be sought. In this work, we show that it is possible to introduce constraints on these latent geometries to address various problems. In more detail, we propose to represent geometries by constructing similarity graphs from the intermediate representations obtained when processing a batch of inputs. By constraining these Latent Geometry Graphs (LGGs), we address the three following problems: (i) reproducing the behavior of a teacher architecture is achieved by mimicking its geometry, (ii) designing efficient embeddings for classification is achieved by targeting specific geometries, and (iii) robustness to deviations on inputs is achieved via enforcing smooth variation of geometry between consecutive latent spaces. Using standard vision benchmarks, we demonstrate the ability of the proposed geometry-based methods in solving the considered problems.

SMILE: Mutual Information Learning for Integration of Single-cell Omics Data

2021 ◽  
Author(s):  
Yang Xu ◽  
Priyojit Das ◽  
Rachel Patton McCord
Keyword(s):  
Deep Learning ◽  
Mutual Information ◽  
Single Cell ◽  
Learning Algorithm ◽  
Cellular Systems ◽  
Supplementary Information ◽  
Omics Data ◽  
Learning Approaches ◽  
Rna Seq ◽  
Integrate Data

Abstract Motivation Deep learning approaches have empowered single-cell omics data analysis in many ways and generated new insights from complex cellular systems. As there is an increasing need for single cell omics data to be integrated across sources, types, and features of data, the challenges of integrating single-cell omics data are rising. Here, we present an unsupervised deep learning algorithm that learns discriminative representations for single-cell data via maximizing mutual information, SMILE (Single-cell Mutual Information Learning). Results Using a unique cell-pairing design, SMILE successfully integrates multi-source single-cell transcriptome data, removing batch effects and projecting similar cell types, even from different tissues, into the shared space. SMILE can also integrate data from two or more modalities, such as joint profiling technologies using single-cell ATAC-seq, RNA-seq, DNA methylation, Hi-C, and ChIP data. When paired cells are known, SMILE can integrate data with unmatched feature, such as genes for RNA-seq and genome wide peaks for ATAC-seq. Integrated representations learned from joint profiling technologies can then be used as a framework for comparing independent single source data. Supplementary information Supplementary data are available at Bioinformatics online. The source code of SMILE including analyses of key results in the study can be found at: https://github.com/rpmccordlab/SMILE.

scholarly journals Deep Learning Using Multiple Degrees of Maximum-Intensity Projection for PET/CT Image Classification in Breast Cancer

Tomography ◽  
2022 ◽  
Vol 8 (1) ◽  
pp. 131-141
Author(s):  
Kanae Takahashi ◽  
Tomoyuki Fujioka ◽  
Jun Oyama ◽  
Mio Mori ◽  
Emi Yamaga ◽  
...  
Keyword(s):  
Breast Cancer ◽  
Deep Learning ◽  
Image Classification ◽  
Test Data ◽  
Characteristic Curve ◽  
Maximum Intensity Projection ◽  
Maximum Intensity ◽  
Training Data ◽  
Ct Image ◽  
Pet Ct

Deep learning (DL) has become a remarkably powerful tool for image processing recently. However, the usefulness of DL in positron emission tomography (PET)/computed tomography (CT) for breast cancer (BC) has been insufficiently studied. This study investigated whether a DL model using images with multiple degrees of PET maximum-intensity projection (MIP) images contributes to increase diagnostic accuracy for PET/CT image classification in BC. We retrospectively gathered 400 images of 200 BC and 200 non-BC patients for training data. For each image, we obtained PET MIP images with four different degrees (0°, 30°, 60°, 90°) and made two DL models using Xception. One DL model diagnosed BC with only 0-degree MIP and the other used four different degrees. After training phases, our DL models analyzed test data including 50 BC and 50 non-BC patients. Five radiologists interpreted these test data. Sensitivity, specificity, and area under the receiver operating characteristic curve (AUC) were calculated. Our 4-degree model, 0-degree model, and radiologists had a sensitivity of 96%, 82%, and 80–98% and a specificity of 80%, 88%, and 76–92%, respectively. Our 4-degree model had equal or better diagnostic performance compared with that of the radiologists (AUC = 0.936 and 0.872–0.967, p = 0.036–0.405). A DL model similar to our 4-degree model may lead to help radiologists in their diagnostic work in the future.

scholarly journals UPAYA MENINGKATKAN KEAKTIFAN DAN HASIL BELAJAR SISWA MELALUI PEMBELAJARAN KONTEKSTUAL MATERI SISTEM GERAK PADA MANUSIA

2018 ◽  
Vol 7 (1) ◽  
Author(s):  
Wahyu Widayaningsih
Keyword(s):  
Action Research ◽  
Test Data ◽  
Learning Process ◽  
Teaching And Learning ◽  
Group Discussion ◽  
Human Movement ◽  
Contextual Learning ◽  
Contextual Approach ◽  
Movement System ◽  
Written Test

ABSTRACTProblems which were raised in this research are the low students’ activity and their poor achievements during the teaching and learning process, especially in human movement system topic. Hence, the researcher implemented contextual approach with group discussion to stimulate students’ active activity and improve their scores. This study utilized classroom action research in which there were four steps for each cycle, namely planning, acting, observing, and reflecting. Data were divided into two, primary (researcher’s action) and secondary (documentation). Those data were collected through interview, students’ worksheets, teacher’s observation and written test. Data were analyzed descriptively. The results showed that the students’ activity from cycle 1 to 3 respectively are 71.88 %, 84.38 %, and 90.62 %; meanwhile the average scores of students’ achievement are 55.31; 69.55; and 79.38 respectively. It can be concluded that by implementing contextual approach with group discussion, students’ activity and achievements during teaching and learning the human movement system topic are improved. Keywords: contextual learning, human movement system, student’s achievement, students‘ activity ABSTRAKPermasalahan yang diangkat dalam penelitian ini adalah rendahnya keaktifan dan hasil belajar siswa pada materi sistem gerak pada manusia, sehingga peneliti menerapkan pendekatan kontekstual dengan diskusi kelompok untuk merangsang siswa lebih aktif dan hasil belajar siswa lebih meningkat. Penelitian ini merupakan penelitian tindakan kelas dimana setiap siklus terdiri dari empat tahap yaitu perencanaan, pelaksanaan, pengamatan dan refleksi. Sumber data dalam penelitian ini meliputi data primer yaitu peneliti yang melakukan tindakan dan siswa yang menerima tindakan dan sumber data sekunder yang berupa data dokumentasi. Data diperoleh melalui wawancara, lembar aktivitas siswa, observasi kinerja guru dan tes tertulis. Teknik analisis yang digunakan yaitu metode analisis deskriptif. Hasil ditunjukkan dengan persentase keaktifan siswa; hasil siklus 1, 2 dan 3 berturut-turut adalah 71,88 %, 84,38 %, dan 90,62 %. Sedangkan rata-rata hasil belajar siswa hasil siklus 1,2 dan 3 berturut-turut 55,31; 69,55; dan 79,38. Berdasarkan hasil tersebut, disimpulkan bahwa dengan menerapkan pembelajaran kontekstual dengan diskusi kelompok, keaktifan siswa dan hasil belajar siswa pada materi sistem gerak pada manusia meningkat. Kata kunci:  pembelajaran kontekstual, sistem gerak pada manusia, hasil belajar siswa, keaktifan siswa

scholarly journals A novel mutual information estimator to measure spike train correlations in a model thalamocortical network

2018 ◽  
Vol 120 (6) ◽  
pp. 2730-2744 ◽  
Author(s):  
Ekaterina D. Gribkova ◽  
Baher A. Ibrahim ◽  
Daniel A. Llano
Keyword(s):  
Mutual Information ◽  
Information Transmission ◽  
Spike Train ◽  
Calcium Current ◽  
Thalamic Reticular Nucleus ◽  
Reticular Nucleus ◽  
Sensory Signals ◽  
Layer 4 ◽  
The Impact ◽  
Current Conductance

The impact of thalamic state on information transmission to the cortex remains poorly understood. This limitation exists due to the rich dynamics displayed by thalamocortical networks and because of inadequate tools to characterize those dynamics. Here, we introduce a novel estimator of mutual information and use it to determine the impact of a computational model of thalamic state on information transmission. Using several criteria, this novel estimator, which uses an adaptive partition, is shown to be superior to other mutual information estimators with uniform partitions when used to analyze simulated spike train data with different mean spike rates, as well as electrophysiological data from simultaneously recorded neurons. When applied to a thalamocortical model, the estimator revealed that thalamocortical cell T-type calcium current conductance influences mutual information between the input and output from this network. In particular, a T-type calcium current conductance of ~40 nS appears to produce maximal mutual information between the input to this network (conceptualized as afferent input to the thalamocortical cell) and the output of the network at the level of a layer 4 cortical neuron. Furthermore, at particular combinations of inputs to thalamocortical and thalamic reticular nucleus cells, thalamic cell bursting correlated strongly with recovery of mutual information between thalamic afferents and layer 4 neurons. These studies suggest that the novel mutual information estimator has advantages over previous estimators and that thalamic reticular nucleus activity can enhance mutual information between thalamic afferents and thalamorecipient cells in the cortex. NEW & NOTEWORTHY In this study, a novel mutual information estimator was developed to analyze information flow in a model thalamocortical network. Our findings suggest that this estimator is a suitable tool for signal transmission analysis, particularly in neural circuits with disparate firing rates, and that the thalamic reticular nucleus can potentiate ascending sensory signals, while thalamic recipient cells in the cortex can recover mutual information in ascending sensory signals that is lost due to thalamic bursting.

Sign in / Sign up

Export Citation Format

Share Document