SURROGATE DATA PATHOLOGIES AND THE FALSE-POSITIVE REJECTION OF THE NULL HYPOTHESIS

It is shown that inappropriately constructed random phase surrogates can give false-positive rejections of the surrogate null hypothesis. Specifically, the procedure erroneously indicated the presence of deterministic, nonlinear structure in a time series that was constructed by linearly filtering normally distributed random numbers. It is shown that the erroneous identification was due to numerical errors in the estimation of the signal's Fourier transform. In the example examined here, the introduction of data windowing into the algorithm eliminated the false-positive rejection of the null hypothesis. Additional guidelines for the use of surrogates are considered, and the results of a comparison test of random phase surrogates, Gaussian scaled surrogates and iterative surrogates are presented.

Download Full-text

Nonlinear analysis of magnetospheric data Part I. Geometric characteristics of the AE index time series and comparison with nonlinear surrogate data

Nonlinear Processes in Geophysics ◽

10.5194/npg-6-51-1999 ◽

1999 ◽

Vol 6 (1) ◽

pp. 51-65 ◽

Cited By ~ 23

Author(s):

G. P. Pavlos ◽

M. A. Athanasiu ◽

D. Kugiumtzis ◽

N. Hatzigeorgiu ◽

A. G. Rigas ◽

...

Keyword(s):

Time Series ◽

State Space ◽

Null Hypothesis ◽

Surrogate Data ◽

Statistical Testing ◽

Geometrical Parameters ◽

Geometric Characteristics ◽

Geometrical Characteristics ◽

Ae Index ◽

Index Time Series

Abstract. A long AE index time series is used as a crucial magnetospheric quantity in order to study the underlying dynainics. For this purpose we utilize methods of nonlinear and chaotic analysis of time series. Two basic components of this analysis are the reconstruction of the experimental tiine series state space trajectory of the underlying process and the statistical testing of an null hypothesis. The null hypothesis against which the experimental time series are tested is that the observed AE index signal is generated by a linear stochastic signal possibly perturbed by a static nonlinear distortion. As dis ' ' ating statistics we use geometrical characteristics of the reconstructed state space (Part I, which is the work of this paper) and dynamical characteristics (Part II, which is the work a separate paper), and "nonlinear" surrogate data, generated by two different techniques which can mimic the original (AE index) signal. lie null hypothesis is tested for geometrical characteristics which are the dimension of the reconstructed trajectory and some new geometrical parameters introduced in this work for the efficient discrimination between the nonlinear stochastic surrogate data and the AE index. Finally, the estimated geometric characteristics of the magnetospheric AE index present new evidence about the nonlinear and low dimensional character of the underlying magnetospheric dynamics for the AE index.

Download Full-text

Theoretical Consideration on the Application of Continuous Wavelet Transform to Time-Series Processing for Magnetotelluric Data

10.21203/rs.3.rs-72251/v1 ◽

2020 ◽

Author(s):

Hiroki Ogawa ◽

Yuki Hama ◽

Koichi Asamori ◽

Takumi Ueda

Keyword(s):

Time Series ◽

Fourier Transform ◽

Data Processing ◽

Time Series Data ◽

Series Data ◽

Short Time Fourier Transform ◽

Numerical Errors ◽

Spectral Transform ◽

Frequency Components ◽

Short Time

Abstract In the magnetotelluric (MT) method, the responses of the natural electromagnetic fields are evaluated by transforming time-series data into spectral data and calculating the apparent resistivity and phase. The continuous wavelet transform (CWT) can be an alternative to the short-time Fourier transform, and the applicability of CWT to MT data has been reported. There are, however, few cases of considering the effect of numerical errors derived from spectral transform on MT data processing. In general, it is desirable to adopt a window function narrow in the time domain for higher-frequency components and one in the frequency domain for lower-frequency components. In conducting the short-time Fourier transform, because the size of the window function is fixed unless the time-series data are decimated, there might be difference between the calculated MT responses and the true ones due to the numerical errors. Meanwhile, CWT can strike a balance between the resolution of the time and frequency domains by magnifying or reducing the wavelet, according to the value of frequency. Although the types of wavelet functions and their parameters influence the resolution of time and frequency, those calculation settings of CWT are often determined empirically. In this study, focusing on the frequency band between 0.001 Hz and 10 Hz, we demonstrated the superiority of utilizing CWT in MT data processing and determined its proper calculation settings in terms of restraining the numerical errors caused by the spectral transform of time-series data. The results obtained with the short-time Fourier transform accompanied with gradual decimation of the time-series data, called cascade decimation, were compared with those of CWT. The shape of the wavelet was changed by using different types of wavelet functions or their parameters, and the respective results of data processing were compared. Through these experiments, this study indicates that CWT with the complex Morlet function with its wavelet parameter k set to 6 ≤ k < 10 will be effective in restraining the numerical errors caused by the spectral transform.

Download Full-text

TESTING TIME-SERIES STATIONARITY AGAINST AN ALTERNATIVE WHOSE MEAN IS PERIODIC

Macroeconomic Dynamics ◽

10.1017/s1365100500020034 ◽

2001 ◽

Vol 5 (3) ◽

pp. 380-412 ◽

Cited By ~ 10

Author(s):

Melvin A. Hinich ◽

Phillip Wild

Keyword(s):

Time Series ◽

Fourier Transform ◽

Real World ◽

Null Hypothesis ◽

Data Sets ◽

Testing Time ◽

Economic Data ◽

The Mean ◽

Important Form ◽

Real World Problems

We develop a test of the null hypothesis that an observed time series is a realization of a strictly stationary random process. Our test is based on the result that the kth value of the discrete Fourier transform of a sample frame has a zero mean under the null hypothesis. The test that we develop will have considerable power against an important form of nonstationarity hitherto not considered in the mainstream econometric time-series literature, that is, where the mean of a time series is periodic with random variation in its periodic structure. The size and power properties of the test are investigated and its applicability to real-world problems is demonstrated by application to three economic data sets.

Download Full-text

ON THE RELIABILITY OF THE SURROGATE DATA TEST FOR NONLINEARITY IN THE ANALYSIS OF NOISY TIME SERIES

International Journal of Bifurcation and Chaos ◽

10.1142/s0218127401003061 ◽

2001 ◽

Vol 11 (07) ◽

pp. 1881-1896 ◽

Cited By ~ 31

Author(s):

D. KUGIUMTZIS

Keyword(s):

Time Series ◽

Null Hypothesis ◽

Volterra Series ◽

Simulated Data ◽

Real Data ◽

Surrogate Data ◽

Largest Lyapunov Exponent ◽

Real World Data ◽

Original Time ◽

False Nearest Neighbors

In the analysis of real world data, the surrogate data test is often performed in order to investigate nonlinearity in the data. The null hypothesis of the test is that the original time series is generated from a linear stochastic process possibly undergoing a nonlinear static transform. We argue against reported rejection of the null hypothesis and claims of evidence of nonlinearity based on a single nonlinear statistic. In particular, two schemes for the generation of surrogate data are examined, the amplitude adjusted Fourier transform (AAFT) and the iterated AAFT (IAFFT) and many nonlinear discriminating statistics are used for testing, i.e. the fit with the Volterra series of polynomials and the fit with local average mappings, the mutual information, the correlation dimension, the false nearest neighbors, the largest Lyapunov exponent and simple nonlinear averages (the three point autocorrelation and the time reversal asymmetry). The results on simulated data and real data (EEG and exchange rates) suggest that the test depends on the method and its parameters, the algorithm generating the surrogate data and the observational data of the examined process.

Download Full-text

TESTING FOR NONLINEARITY OF THE CONTRACTION SEGMENTS IN UTERINE ELECTROMYOGRAPHY

International Journal of Bifurcation and Chaos ◽

10.1142/s0218127400001870 ◽

2000 ◽

Vol 10 (12) ◽

pp. 2785-2790 ◽

Cited By ~ 11

Author(s):

N. RADHAKRISHNAN ◽

JAMES D. WILSON ◽

CURTIS LOWERY ◽

PAM MURPHY ◽

HARI ESWARAN

Keyword(s):

Fourier Transform ◽

Null Hypothesis ◽

Uterine Contraction ◽

Nonlinear Function ◽

Nonlinear Process ◽

Surrogate Data ◽

Correlated Noise ◽

Uterine Contractility ◽

Parametric Testing ◽

Uterine Electromyography

In this report, we test for possible nonlinearity of the contraction segments interspersed in a uterine electromyography (EMG), recorded externally with abdominal electrodes. There have been several reports in which the uterine contractility had been assumed to be an auto-regressive process and others have hypothesized it as a nonlinear process and possibly chaotic. The surrogate data testing was used successfully to detect nonlinear behavior of physiological systems. However, there have been case studies, which discuss spurious identification of nonrandom structures. The proper choice of the null hypothesis and discriminant statistics plays a crucial role in the surrogate data testing. We have chosen the approximate entropy as the discriminant statistic for our tests. The null hypothesis addressed here is that the uterine contraction is a linearly correlated noise transformed by a nonlinear function. We applied the Amplitude Adjusted Fourier Transform (AAFT) and the Iterated Amplitude Adjusted Fourier Transform (IAAFT) tests to the uterine contraction data. The Kolmogorov Smirnov (D) statistics identified the discriminant values of the surrogates to be from a Gaussian distribution. Parametric testing showed a very low significance value, (~2σ), which indicated the absence of nonrandom structure in the contraction segment.

Download Full-text

Data Preprocessing

10.1093/oso/9780198782933.003.0006 ◽

2018 ◽

Author(s):

Ray Huffaker ◽

Marco Bittelli ◽

Rodolfo Rosa

Keyword(s):

Stochastic Process ◽

Time Series ◽

Signal Processing ◽

Real World ◽

Null Hypothesis ◽

Time Series Data ◽

Nonlinear Behavior ◽

Surrogate Data ◽

Series Data ◽

Linear Behavior

Successful reconstruction of a shadow attractor provides preliminary empirical evidence that a signal isolated from observed time series data may be generated by deterministic dynamics. However, because we cannot reasonably expect signal processing to purge the signal of all noise in practice, and because noisy linear behavior can be visually indistinguishable from nonlinear behavior, the possibility remains that noticeable regularity detected in a shadow attractor may be fortuitously reconstructed from data generated by a linear-stochastic process. This chapter investigates how we can test this null hypothesis using surrogate data testing. The combination of a noticeably regular shadow attractor, along with strong statistical rejection of fortuitous regularity, increases the probability that observed data are generated by deterministic real-world dynamics.

Download Full-text

Streamflow dynamics at the Puget Sound, Washington: application of a surrogate data method

Nonlinear Processes in Geophysics ◽

10.5194/npg-12-461-2005 ◽

2005 ◽

Vol 12 (4) ◽

pp. 461-469 ◽

Cited By ~ 5

Author(s):

N. She ◽

D. Basketfield

Keyword(s):

Time Series ◽

Null Hypothesis ◽

Puget Sound ◽

Original Data ◽

Surrogate Data ◽

Linear Process ◽

Central Basin ◽

Daily Streamflow ◽

Streamflow Data ◽

Dynamic Nonlinearity

Abstract. Recent progress in nonlinear dynamic theory has inspired hydrologists to apply innovative nonlinear time series techniques to the analysis of streamflow data. However, regardless of the method employed to analyze streamflow data, the first step should be the identification of underlying dynamics using one or more methods that could distinguish between linear and nonlinear, deterministic and stochastic processes from data itself. In recent years a statistically rigorous framework to test whether or not the examined time series is generated by a Gaussian (linear) process undergoing a possibly nonlinear static transform is provided by the method of surrogate data. The surrogate data, generated to represent the null hypothesis, are compared to the original data under a nonlinear discriminating statistic in order to reject or approve the null hypothesis. In recognition of this tendency, the method of "surrogate data" is applied herein to determine the underlying linear stochastic or nonlinear deterministic nature of daily streamflow data observed from the central basin of Puget Sound, and as applicable, distinguish between the static or dynamic nonlinearity of the data in question.

Download Full-text

TESTING FOR NONLINEARITY IN SHANGHAI STOCK MARKET

International Journal of Modern Physics B ◽

10.1142/s021797920402597x ◽

2004 ◽

Vol 18 (17n19) ◽

pp. 2720-2724 ◽

Cited By ~ 2

Author(s):

HAIYAN WANG ◽

LONGKUN TANG

Keyword(s):

Time Series ◽

Stock Market ◽

Null Hypothesis ◽

Quantitative Method ◽

Multivariate Time Series ◽

Surrogate Data ◽

Significance Test ◽

Test Statistic ◽

Gaussian Random Process ◽

Linear And Nonlinear

In this paper, we apply IAAFT to generate surrogate time series of measured multivariate time series. A quantitative method to detect nonlinearity in multivariate time series is proposed using the generalized redundancy and linear redundancy as the significance test statistic. The null hypothesis of a multivariate linear Gaussian random process is tested using the multivariate surrogate data. The validity of this method is demonstrated using two types models (linear and nonlinear) and applied to Shanghai stock market.

Download Full-text