scholarly journals Learning stochastic closures using ensemble Kalman inversion

2021 ◽  
Vol 5 (1) ◽  
Author(s):  
Tapio Schneider ◽  
Andrew M Stuart ◽  
Jin-Long Wu
Keyword(s):  
Inverse Problem ◽  
Large Scale ◽  
Time Series Data ◽  
Gaussian Process Regression ◽  
Real Data ◽  
Series Data ◽  
Sufficient Statistics ◽  
Atmospheric Sciences ◽  
Pattern Modeling ◽  
Complex Phenomena

Abstract Although the governing equations of many systems, when derived from first principles, may be viewed as known, it is often too expensive to numerically simulate all the interactions they describe. Therefore, researchers often seek simpler descriptions that describe complex phenomena without numerically resolving all the interacting components. Stochastic differential equations (SDEs) arise naturally as models in this context. The growth in data acquisition, both through experiment and through simulations, provides an opportunity for the systematic derivation of SDE models in many disciplines. However, inconsistencies between SDEs and real data at short time scales often cause problems, when standard statistical methodology is applied to parameter estimation. The incompatibility between SDEs and real data can be addressed by deriving sufficient statistics from the time-series data and learning parameters of SDEs based on these. Here, we study sufficient statistics computed from time averages, an approach that we demonstrate to lead to sufficient statistics on a variety of problems and that has the secondary benefit of obviating the need to match trajectories. Following this approach, we formulate the fitting of SDEs to sufficient statistics from real data as an inverse problem and demonstrate that this inverse problem can be solved by using ensemble Kalman inversion. Furthermore, we create a framework for non-parametric learning of drift and diffusion terms by introducing hierarchical, refinable parameterizations of unknown functions, using Gaussian process regression. We demonstrate the proposed methodology for the fitting of SDE models, first in a simulation study with a noisy Lorenz ’63 model, and then in other applications, including dimension reduction in deterministic chaotic systems arising in the atmospheric sciences, large-scale pattern modeling in climate dynamics and simplified models for key observables arising in molecular dynamics. The results confirm that the proposed methodology provides a robust and systematic approach to fitting SDE models to real data.

scholarly journals How to Identify Varying Lead–Lag Effects in Time Series Data: Implementation, Validation, and Application of the Generalized Causality Algorithm

Algorithms ◽  
2020 ◽  
Vol 13 (4) ◽  
pp. 95 ◽  
Author(s):  
Johannes Stübinger ◽  
Katharina Adler
Keyword(s):  
Time Series ◽  
Large Scale ◽  
Structural Breaks ◽  
Time Series Data ◽  
Consumer Price Index ◽  
Real Data ◽  
Linear Mapping ◽  
Series Data ◽  
Lag Effects ◽  
Silver Metal

This paper develops the generalized causality algorithm and applies it to a multitude of data from the fields of economics and finance. Specifically, our parameter-free algorithm efficiently determines the optimal non-linear mapping and identifies varying lead–lag effects between two given time series. This procedure allows an elastic adjustment of the time axis to find similar but phase-shifted sequences—structural breaks in their relationship are also captured. A large-scale simulation study validates the outperformance in the vast majority of parameter constellations in terms of efficiency, robustness, and feasibility. Finally, the presented methodology is applied to real data from the areas of macroeconomics, finance, and metal. Highest similarity show the pairs of gross domestic product and consumer price index (macroeconomics), S&P 500 index and Deutscher Aktienindex (finance), as well as gold and silver (metal). In addition, the algorithm takes full use of its flexibility and identifies both various structural breaks and regime patterns over time, which are (partly) well documented in the literature.

The Classic Maya Collapse: Testing Class Conflict Hypotheses

1980 ◽  
Vol 45 (2) ◽  
pp. 246-267 ◽  
Author(s):  
Robert L. Hamblin ◽  
Brian L. Pitcher
Keyword(s):  
Large Scale ◽  
Time Series Data ◽  
Classic Maya ◽  
Class Conflict ◽  
Series Data ◽  
Geographical Patterns ◽  
Historical Societies ◽  
Testing Class ◽  
Lowland Area ◽  
Maya Collapse

Several lines of archaeological evidence are presented in this paper to suggest the existence of class warfare among the Classic Maya and of issues that historically have been associated with class conflict. This evidence indicates that class warfare may have halted the rule of the monument-producing, or Classic, elites and precipitated the depopulation of the lowland area. The theory is evaluated quantitatively by testing for time-related mathematical patterns that have been found to characterize large-scale conflicts in historical societies. The information used in the evaluation involves the time series data on the duration of rule by Classic elites as inferred from the production of monuments with Long Count dates at a sample of 82 ceremonial centers. The analyses confirm that the Maya data do exhibit the temporal and geographical patterns predicted from the class conflict explanation of the Classic Maya collapse. Alternative predictions from the other theories are considered but generally not found to be supported by these data.

scholarly journals Evidence Graphs: Supporting Transparent and FAIR Computation, with Defeasible Reasoning on Data, Methods and Results

2021 ◽  
Author(s):  
Sadnan Al Manir ◽  
Justin Niestroy ◽  
Maxwell Adam Levinson ◽  
Timothy Clark
Keyword(s):  
Time Series ◽  
Large Scale ◽  
Time Series Data ◽  
Predictive Analytics ◽  
Defeasible Reasoning ◽  
Series Data ◽  
Inference Rules ◽  
Deep Networks ◽  
Evidence Graph ◽  
Over Time

Introduction: Transparency of computation is a requirement for assessing the validity of computed results and research claims based upon them; and it is essential for access to, assessment, and reuse of computational components. These components may be subject to methodological or other challenges over time. While reference to archived software and/or data is increasingly common in publications, a single machine-interpretable, integrative representation of how results were derived, that supports defeasible reasoning, has been absent. Methods: We developed the Evidence Graph Ontology, EVI, in OWL 2, with a set of inference rules, to provide deep representations of supporting and challenging evidence for computations, services, software, data, and results, across arbitrarily deep networks of computations, in connected or fully distinct processes. EVI integrates FAIR practices on data and software, with important concepts from provenance models, and argumentation theory. It extends PROV for additional expressiveness, with support for defeasible reasoning. EVI treats any com- putational result or component of evidence as a defeasible assertion, supported by a DAG of the computations, software, data, and agents that produced it. Results: We have successfully deployed EVI for very-large-scale predictive analytics on clinical time-series data. Every result may reference its own evidence graph as metadata, which can be extended when subsequent computations are executed. Discussion: Evidence graphs support transparency and defeasible reasoning on results. They are first-class computational objects, and reference the datasets and software from which they are derived. They support fully transparent computation, with challenge and support propagation. The EVI approach may be extended to include instruments, animal models, and critical experimental reagents.

scholarly journals Differentially Private Autocorrelation Time-Series Data Publishing Based on Sliding Window

2021 ◽  
Vol 2021 ◽  
pp. 1-10
Author(s):  
Jing Zhao ◽  
Shubo Liu ◽  
Xingxing Xiong ◽  
Zhaohui Cai
Keyword(s):  
Time Series ◽  
Privacy Protection ◽  
Large Scale ◽  
Differential Privacy ◽  
Time Series Data ◽  
Sliding Window ◽  
Data Publishing ◽  
Series Data ◽  
Data Publication ◽  
Autocorrelation Time

Privacy protection is one of the major obstacles for data sharing. Time-series data have the characteristics of autocorrelation, continuity, and large scale. Current research on time-series data publication mainly ignores the correlation of time-series data and the lack of privacy protection. In this paper, we study the problem of correlated time-series data publication and propose a sliding window-based autocorrelation time-series data publication algorithm, called SW-ATS. Instead of using global sensitivity in the traditional differential privacy mechanisms, we proposed periodic sensitivity to provide a stronger degree of privacy guarantee. SW-ATS introduces a sliding window mechanism, with the correlation between the noise-adding sequence and the original time-series data guaranteed by sequence indistinguishability, to protect the privacy of the latest data. We prove that SW-ATS satisfies ε-differential privacy. Compared with the state-of-the-art algorithm, SW-ATS is superior in reducing the error rate of MAE which is about 25%, improving the utility of data, and providing stronger privacy protection.

A deep learning method for data recovery in sensor networks using effective spatio-temporal correlation data

Sensor Review ◽  
2019 ◽  
Vol 39 (2) ◽  
pp. 208-217 ◽  
Author(s):  
Jinghan Du ◽  
Haiyan Chen ◽  
Weining Zhang
Keyword(s):  
Deep Learning ◽  
Sensor Networks ◽  
Large Scale ◽  
Time Series Data ◽  
Temporal Correlation ◽  
Data Recovery ◽  
Series Data ◽  
Learning Method ◽  
Content Type ◽  
Spatio Temporal

Purpose In large-scale monitoring systems, sensors in different locations are deployed to collect massive useful time-series data, which can help in real-time data analytics and its related applications. However, affected by hardware device itself, sensor nodes often fail to work, resulting in a common phenomenon that the collected data are incomplete. The purpose of this study is to predict and recover the missing data in sensor networks. Design/methodology/approach Considering the spatio-temporal correlation of large-scale sensor data, this paper proposes a data recover model in sensor networks based on a deep learning method, i.e. deep belief network (DBN). Specifically, when one sensor fails, the historical time-series data of its own and the real-time data from surrounding sensor nodes, which have high similarity with a failure observed using the proposed similarity filter, are collected first. Then, the high-level feature representation of these spatio-temporal correlation data is extracted by DBN. Moreover, to determine the structure of a DBN model, a reconstruction error-based algorithm is proposed. Finally, the missing data are predicted based on these features by a single-layer neural network. Findings This paper collects a noise data set from an airport monitoring system for experiments. Various comparative experiments show that the proposed algorithms are effective. The proposed data recovery model is compared with several other classical models, and the experimental results prove that the deep learning-based model can not only get a better prediction accuracy but also get a better performance in training time and model robustness. Originality/value A deep learning method is investigated in data recovery task, and it proved to be effective compared with other previous methods. This might provide a practical experience in the application of a deep learning method.

scholarly journals Regional September Sea Ice Forecasting with Complex Networks and Gaussian Processes

2020 ◽  
Vol 35 (3) ◽  
pp. 793-806
Author(s):  
William Gregory ◽  
Michel Tsamados ◽  
Julienne Stroeve ◽  
Peter Sollich
Keyword(s):  
Sea Ice ◽  
Time Series Data ◽  
Gaussian Process Regression ◽  
Forecast Model ◽  
Arctic Sea Ice ◽  
The Arctic ◽  
Series Data ◽  
Lead Times ◽  
Sea Ice Concentration ◽  
Sea Ice Extent

Abstract Reliable predictions of the Arctic sea ice cover are becoming of paramount importance for Arctic communities and industry stakeholders. In this study pan-Arctic and regional September mean sea ice extents are forecast with lead times of up to 3 months using a complex network statistical approach. This method exploits relationships within climate time series data by constructing regions of spatiotemporal homogeneity (i.e., nodes), and subsequently deriving teleconnection links between them. Here the nodes and links of the networks are generated from monthly mean sea ice concentration fields in June, July, and August; hence, individual networks are constructed for each respective month. Network information is then utilized within a linear Gaussian process regression forecast model, a Bayesian inference technique, in order to generate predictions of sea ice extent. Pan-Arctic forecasts capture a significant amount of the variability in the satellite observations of September sea ice extent, with detrended predictive skills of 0.53, 0.62, and 0.81 at 3-, 2-, and 1-month lead times, respectively. Regional forecasts are also performed for nine Arctic regions. On average, the highest predictive skill is achieved in the Canadian Archipelago, Beaufort, Chukchi, East Siberian, Laptev, and Kara Seas, although the ability to accurately predict many of these regions appears to be changing over time.

scholarly journals AstroCatR: a mechanism and tool for efficient time series reconstruction of large-scale astronomical catalogues

2020 ◽  
Vol 496 (1) ◽  
pp. 629-637
Author(s):  
Ce Yu ◽  
Kun Li ◽  
Shanjiang Tang ◽  
Chao Sun ◽  
Bin Ma ◽  
...  
Keyword(s):  
Time Series ◽  
High Performance ◽  
Large Scale ◽  
Extrasolar Planets ◽  
Time Series Data ◽  
Series Data ◽  
Data Sets ◽  
Observation Data ◽  
Data Volume ◽  
And Performance

ABSTRACT Time series data of celestial objects are commonly used to study valuable and unexpected objects such as extrasolar planets and supernova in time domain astronomy. Due to the rapid growth of data volume, traditional manual methods are becoming extremely hard and infeasible for continuously analysing accumulated observation data. To meet such demands, we designed and implemented a special tool named AstroCatR that can efficiently and flexibly reconstruct time series data from large-scale astronomical catalogues. AstroCatR can load original catalogue data from Flexible Image Transport System (FITS) files or data bases, match each item to determine which object it belongs to, and finally produce time series data sets. To support the high-performance parallel processing of large-scale data sets, AstroCatR uses the extract-transform-load (ETL) pre-processing module to create sky zone files and balance the workload. The matching module uses the overlapped indexing method and an in-memory reference table to improve accuracy and performance. The output of AstroCatR can be stored in CSV files or be transformed other into formats as needed. Simultaneously, the module-based software architecture ensures the flexibility and scalability of AstroCatR. We evaluated AstroCatR with actual observation data from The three Antarctic Survey Telescopes (AST3). The experiments demonstrate that AstroCatR can efficiently and flexibly reconstruct all time series data by setting relevant parameters and configuration files. Furthermore, the tool is approximately 3× faster than methods using relational data base management systems at matching massive catalogues.

Sign in / Sign up

Export Citation Format

Share Document