scholarly journals The unbiased estimation of the fraction of variance explained by a model

2021 ◽  
Vol 17 (8) ◽  
pp. e1009212
Author(s):  
Dean A. Pospisil ◽  
Wyeth Bair
Keyword(s):  
Confidence Intervals ◽  
Signal To Noise Ratio ◽  
Unbiased Estimation ◽  
Data Sets ◽  
Neural Data ◽  
Neural Recordings ◽  
Quality Metric ◽  
Neural Noise ◽  
Biological Phenomena ◽  
Image Position

The correlation coefficient squared, r2, is commonly used to validate quantitative models on neural data, yet it is biased by trial-to-trial variability: as trial-to-trial variability increases, measured correlation to a model’s predictions decreases. As a result, models that perfectly explain neural tuning can appear to perform poorly. Many solutions to this problem have been proposed, but no consensus has been reached on which is the least biased estimator. Some currently used methods substantially overestimate model fit, and the utility of even the best performing methods is limited by the lack of confidence intervals and asymptotic analysis. We provide a new estimator, r ^ ER 2, that outperforms all prior estimators in our testing, and we provide confidence intervals and asymptotic guarantees. We apply our estimator to a variety of neural data to validate its utility. We find that neural noise is often so great that confidence intervals of the estimator cover the entire possible range of values ([0, 1]), preventing meaningful evaluation of the quality of a model’s predictions. This leads us to propose the use of the signal-to-noise ratio (SNR) as a quality metric for making quantitative comparisons across neural recordings. Analyzing a variety of neural data sets, we find that up to ∼ 40% of some state-of-the-art neural recordings do not pass even a liberal SNR criterion. Moving toward more reliable estimates of correlation, and quantitatively comparing quality across recording modalities and data sets, will be critical to accelerating progress in modeling biological phenomena.

scholarly journals The unbiased estimation of the fraction of variance explained by a model

2020 ◽  
Author(s):  
Dean A. Pospisil ◽  
Wyeth Bair
Keyword(s):  
Correlation Coefficient ◽  
Confidence Intervals ◽  
Signal To Noise Ratio ◽  
Unbiased Estimation ◽  
Data Sets ◽  
Neural Data ◽  
Neural Recordings ◽  
Quality Metric ◽  
Neural Noise ◽  
Biological Phenomena

AbstractThe correlation coefficient squared, r2, is often used to validate quantitative models on neural data. Yet it is biased by trial-to-trial variability: as trial-to-trial variability increases, measured correlation to a model’s predictions decreases; therefore, models that perfectly explain neural tuning can appear to perform poorly. Many solutions to this problem have been proposed, but some prior methods overestimate model fit, the utility of even the best performing methods is limited by the lack of confidence intervals and asymptotic analysis, and no consensus has been reached on which is the least biased estimator. We provide a new estimator, , that outperforms all prior estimators in our testing, and we provide confidence intervals and asymptotic guarantees. We apply our estimator to a variety of neural data to validate its utility. We find that neural noise is often so great that confidence intervals of the estimator cover the entire possible range of values ([0, 1]), preventing meaningful evaluation of the quality of a model’s predictions. We demonstrate the use of the signal-to-noise ratio (SNR) as a quality metric for making quantitative comparisons across neural recordings. Analyzing a variety of neural data sets, we find ~ 40% or less of some neural recordings do not pass even a liberal SNR criterion.Author SummaryQuantifying the similarity between a model and noisy data is fundamental to the verification of advances in scientific understanding of biological phenomena, and it is particularly relevant to modeling neuronal responses. A ubiquitous metric of similarity is the correlation coefficient. Here we point out how the correlation coefficient depends on a variety of factors that are irrelevant to the similarity between a model and data. While neuroscientists have recognized this problem and proposed corrected methods, no consensus has been reached as to which are effective. Prior methods have wide variation in their precision, and even the most successful methods lack confidence intervals, leaving uncertainty about the reliability of any particular estimate. We address these issues by developing a new estimator along with an associated confidence interval that outperforms all prior methods.

scholarly journals Functional Imaging with Higher-Dimensional Electrical Data Sets

2018 ◽  
Vol 26 (6) ◽  
pp. 18-27 ◽  
Author(s):  
P. De Wolf ◽  
Z. Huang ◽  
B. Pittenger ◽  
A. Dujardin ◽  
M. Febvre ◽  
...  
Keyword(s):  
Functional Imaging ◽  
Data Sets ◽  
Higher Dimensional ◽  
Image Position ◽  
Electrical Data

Abstract

Review of Metrics and Assignment of Confidence Intervals for Health Management of Gas Turbine Engines

2008 ◽  
Author(s):  
Craig R. Davison ◽  
Jeff W. Bird
Keyword(s):  
Confidence Intervals ◽  
Health Management ◽  
Diagnostic Technique ◽  
Turbine Engines ◽  
Data Sets ◽  
Gas Turbine Engines ◽  
Diagnostic Systems ◽  
Operating Characteristics ◽  
Hypothetical Data ◽  
Test Instance

The development and evaluation of new diagnostic systems requires statistically-based methods to measure performance. Various metrics are in use by developers and users of diagnostic systems. Current metrics practices are reviewed, including receiver operating characteristics, confusion matrices, Kappa coefficients and various entropy techniques. A set of metrics is then proposed for assessment of diverse gas path diagnostic systems. The use of bootstrap statistics to compare metric results is developed, and demonstrated for a set of hypothetical data sets with a range of relevant characteristics. The bootstrap technique allows the expected range of the metric to be assessed without assuming a probability distribution. A method is proposed to develop confidence intervals for the calculated metrics. The application of a confidence interval could prevent a good diagnostic technique being discarded because of a lower value metric in one test instance. The strengths and weaknesses of the various metrics with derived confidence intervals are discussed. Recommendations are made for further work.

Automated phase attribute-based picking applied to reflection seismics

Geophysics ◽  
2016 ◽  
Vol 81 (2) ◽  
pp. V141-V150 ◽  
Author(s):  
Emanuele Forte ◽  
Matteo Dossi ◽  
Michele Pipan ◽  
Anna Del Ben
Keyword(s):  
Signal To Noise Ratio ◽  
Phase Method ◽  
Data Sets ◽  
Reflection Seismic ◽  
Instantaneous Phase ◽  
Whole Process ◽  
Specific Analysis ◽  
Reflection Seismics ◽  
Selection Of ◽  
Energy Package

We have applied an attribute-based autopicking algorithm to reflection seismics with the aim of reducing the influence of the user’s subjectivity on the picking results and making the interpretation faster with respect to manual and semiautomated techniques. Our picking procedure uses the cosine of the instantaneous phase to automatically detect and mark as a horizon any recorded event characterized by lateral phase continuity. A patching procedure, which exploits horizon parallelism, can be used to connect consecutive horizons marking the same event but separated by noise-related gaps. The picking process marks all coherent events regardless of their reflection strength; therefore, a large number of independent horizons can be constructed. To facilitate interpretation, horizons marking different phases of the same reflection can be automatically grouped together and specific horizons from each reflection can be selected using different possible methods. In the phase method, the algorithm reconstructs the reflected wavelets by averaging the cosine of the instantaneous phase along each horizon. The resulting wavelets are then locally analyzed and confronted through crosscorrelation, allowing the recognition and selection of specific reflection phases. In case the reflected wavelets cannot be recovered due to shape-altering processing or a low signal-to-noise ratio, the energy method uses the reflection strength to group together subparallel horizons within the same energy package and to select those satisfying either energy or arrival time criteria. These methods can be applied automatically to all the picked horizons or to horizons individually selected by the interpreter for specific analysis. We show examples of application to 2D reflection seismic data sets in complex geologic and stratigraphic conditions, critically reviewing the performance of the whole process.

scholarly journals T-SNE visualization of large-scale neural recordings

2016 ◽  
Author(s):  
George Dimitriadis ◽  
Joana Neto ◽  
Adam R. Kampff
Keyword(s):  
Large Scale ◽  
New Technologies ◽  
Dimensional Space ◽  
Clustering Algorithms ◽  
Brain Regions ◽  
Data Sets ◽  
Neural Recordings ◽  
Sorting Problem ◽  
Feature Spaces ◽  
High Degree

AbstractElectrophysiology is entering the era of ‘Big Data’. Multiple probes, each with hundreds to thousands of individual electrodes, are now capable of simultaneously recording from many brain regions. The major challenge confronting these new technologies is transforming the raw data into physiologically meaningful signals, i.e. single unit spikes. Sorting the spike events of individual neurons from a spatiotemporally dense sampling of the extracellular electric field is a problem that has attracted much attention [22, 23], but is still far from solved. Current methods still rely on human input and thus become unfeasible as the size of the data sets grow exponentially.Here we introduce the t-student stochastic neighbor embedding (t-sne) dimensionality reduction method [27] as a visualization tool in the spike sorting process. T-sne embeds the n-dimensional extracellular spikes (n = number of features by which each spike is decomposed) into a low (usually two) dimensional space. We show that such embeddings, even starting from different feature spaces, form obvious clusters of spikes that can be easily visualized and manually delineated with a high degree of precision. We propose that these clusters represent single units and test this assertion by applying our algorithm on labeled data sets both from hybrid [23] and paired juxtacellular/extracellular recordings [15]. We have released a graphical user interface (gui) written in python as a tool for the manual clustering of the t-sne embedded spikes and as a tool for an informed overview and fast manual curration of results from other clustering algorithms. Furthermore, the generated visualizations offer evidence in favor of the use of probes with higher density and smaller electrodes. They also graphically demonstrate the diverse nature of the sorting problem when spikes are recorded with different methods and arise from regions with different background spiking statistics.

scholarly journals The unbiased estimation of r2 between two sets of noisy neural responses

2021 ◽  
Author(s):  
Dean Pospisil ◽  
Wyeth A Bair
Keyword(s):  
Correlation Coefficient ◽  
Pearson Correlation ◽  
Unbiased Estimation ◽  
Pearson Correlation Coefficient ◽  
Neural Responses ◽  
Neural Data ◽  
Tuning Curves ◽  
The Relationship

The Pearson correlation coefficient squared, r2, is often used in the analysis of neural data to estimate the relationship between neural tuning curves. Yet this metric is biased by trial-to-trial variability: as trial-to-trial variability increases, measured correlation decreases. Major lines of research are confounded by this bias, including the study of invariance of neural tuning across conditions and the similarity of tuning across neurons. To address this, we extend the estimator, r̂2ER, developed for estimating model-to-neuron correlation to the neuron-to-neuron case. We compare the estimator to a prior method developed by Spearman, commonly used in other fields but widely overlooked in neuroscience, and find that our method has less bias. We then apply our estimator to the study of two forms of invariance and demonstrate how it avoids drastic confounds introduced by trial-to-trial variability.

scholarly journals New processing tools for weak and/or spatially overlapped macromolecular diffraction patterns

1999 ◽  
Vol 55 (10) ◽  
pp. 1733-1741 ◽  
Author(s):  
Dominique Bourgeois
Keyword(s):  
Signal To Noise Ratio ◽  
Data Sets ◽  
Signal To Noise ◽  
Weighting Method ◽  
Time Resolved ◽  
Accurate Evaluation ◽  
Profile Fitting ◽  
Integration Program ◽  
Diffraction Patterns ◽  
New Processing

Tools originally developed for the treatment of weak and/or spatially overlapped time-resolved Laue patterns were extended to improve the processing of difficult monochromatic data sets. The integration programPrOWallows deconvolution of spatially overlapped spots which are usually rejected by standard packages. By using dynamically adjusted profile-fitting areas, a carefully built library of reference spots and interpolation of reference profiles, this program also provides a more accurate evaluation of weak spots. In addition, by using Wilson statistics, it allows rejection of non-redundant strong outliers such as zingers, which otherwise may badly corrupt the data. A weighting method for optimizing structure-factor amplitude differences, based on Bayesian statistics and originally applied to low signal-to-noise ratio time-resolved Laue data, is also shown to significantly improve other types of subtle amplitude differences, such as anomalous differences.

Contrast Signal to Noise Ratio

2021 ◽  
Vol 2021 (17) ◽  
pp. 186-1-186-6
Author(s):  
Robin Jenkin
Keyword(s):  
Autonomous Vehicles ◽  
Signal To Noise Ratio ◽  
Focal Length ◽  
Signal To Noise ◽  
Quality Metric ◽  
Camera Systems ◽  
Image Quality Metric ◽  
Recognition Of Objects ◽  
Noise Ratio ◽  
Supply Information

The detection and recognition of objects is essential for the operation of autonomous vehicles and robots. Designing and predicting the performance of camera systems intended to supply information to neural networks and vision algorithms is nontrivial. Optimization has to occur across many parameters, such as focal length, f-number, pixel and sensor size, exposure regime and transmission schemes. As such numerous metrics are being explored to assist with these design choices. Detectability index (SNRI) is derived from signal detection theory as applied to imaging systems and is used to estimate the ability of a system to statistically distinguish objects [1], most notably in the medical imaging and defense fields [2]. A new metric is proposed, Contrast Signal to Noise Ratio (CSNR), which is calculated simply as mean contrast divided by the standard deviation of the contrast. This is distinct from contrast to noise ratio which uses the noise of the image as the denominator [3,4]. It is shown mathematically that the metric is proportional to the idealized observer for a cobblestone target and a constant may be calculated to estimate SNRI from CSNR, accounting for target size. Results are further compared to Contrast Detection Probability (CDP), which is a relatively new objective image quality metric proposed within IEEE P2020 to rank the performance of camera systems intended for use in autonomous vehicles [5]. CSNR is shown to generate information in illumination and contrast conditions where CDP saturates and further can be modified to provide CDP-like results.

Classification of productive profile using brain signals: Deep learning and neuroscience meets the mining industry

2021 ◽  
Author(s):  
Ana Siravenha ◽  
Walisson Gomes ◽  
Renan Tourinho ◽  
Sergio Viademonte ◽  
Bruno Gomes
Keyword(s):  
Deep Learning ◽  
Iron Ore ◽  
Signal To Noise Ratio ◽  
Mining Industry ◽  
Eeg Signals ◽  
Neural Data ◽  
Technical Staff ◽  
Brain Signals ◽  
Mining Complex

Classification of electroencephalography (EEG) signals is a complex task. EEG is a non-stationary time process with low signal to noise ratio. Among many methods usedfor EEG classification, those based on Deep Learning (DL) have been relatively successful in providing high classification accuracies. In the present study we aimed at classify resting state EEGs measured from workers of a mining complex. Just after the EEG has been collected, the workers undergonetraining in a 4D virtual reality simulator that emulates the iron ore excavation from which parameters related to their performance were analyzed by the technical staff who classified the workers into four groups based on their productivity. Twoconvolutional neural networks (ConvNets) were then used to classify the workers EEG bases on the same productivity label provided by the technical staff. The neural data was used in three configurations in order to evaluate the amount of datarequired for a high accuracy classification. Isolated, the channel T5 achieved 83% of accuracy, the subtraction of channels P3 and Pz achieved 99% and using all channels simultaneously was 99.40% assertive. This study provides results that add to the recent literature showing that even simple DL architectures are able to handle complex time series such as the EEG. In addition, it pin points an application in industry with vast possibilities of expansion.

Sign in / Sign up

Export Citation Format

Share Document