SSTRED: Data- and metadata-processing pipeline for CHROMIS and CRISP

Context. Data from ground-based, high-resolution solar telescopes can only be used for science with calibrations and processing, which requires detailed knowledge about the instrumentation. Space-based solar telescopes provide science-ready data, which are easier to work with for researchers whose expertise is in the interpretation of data. Recently, data-processing pipelines for ground-based instruments have been constructed. Aims. We aim to provide observers with a user-friendly data pipeline for data from the Swedish 1-meter Solar Telescope (SST) that delivers science-ready data together with the metadata needed for proper interpretation and archiving. Methods. We briefly describe the CHROMospheric Imaging Spectrometer (CHROMIS) instrument, including its (pre)filters, as well as recent upgrades to the CRisp Imaging SpectroPolarimeter (CRISP) prefilters and polarization optics. We summarize the processing steps from raw data to science-ready data cubes in FITS files. We report calibrations and compensations for data imperfections in detail. Misalignment of Ca II data due to wavelength-dependent dispersion is identified, characterized, and compensated for. We describe intensity calibrations that remove or reduce the effects of filter transmission profiles as well as solar elevation changes. We present REDUX, a new version of the MOMFBD image restoration code, with multiple enhancements and new features. It uses projective transforms for the registration of multiple detectors. We describe how image restoration is used with CRISP and CHROMIS data. The science-ready output is delivered in FITS files, with metadata compliant with the SOLARNET recommendations. Data cube coordinates are specified within the World Coordinate System (WCS). Cavity errors are specified as distortions of the WCS wavelength coordinate with an extension of existing WCS notation. We establish notation for specifying the reference system for Stokes vectors with reference to WCS coordinate directions. The CRIsp SPectral EXplorer (CRISPEX) data-cube browser has been extended to accept SSTRED output and to take advantage of the SOLARNET metadata. Results. SSTRED is a mature data-processing pipeline for imaging instruments, developed and used for the SST/CHROMIS imaging spectrometer and the SST/CRISP spectropolarimeter. SSTRED delivers well-characterized, science-ready, archival-quality FITS files with well-defined metadata. The SSTRED code, as well as REDUX and CRISPEX, is freely available through git repositories.

Download Full-text

Extending theFermi-LAT Data Processing Pipeline to the Grid

Journal of Physics Conference Series ◽

10.1088/1742-6596/396/3/032121 ◽

2012 ◽

Vol 396 (3) ◽

pp. 032121 ◽

Cited By ~ 1

Author(s):

S Zimmer ◽

L Arrabito ◽

T Glanzman ◽

T Johnson ◽

C Lavalley ◽

...

Keyword(s):

Data Processing ◽

Processing Pipeline

Download Full-text

The NGST long-wave hyperspectral imaging spectrometer: sensor hardware and data processing

10.1117/12.669546 ◽

2006 ◽

Author(s):

John Shepanski ◽

Stephanie Sandor-Leahy

Keyword(s):

Data Processing ◽

Hyperspectral Imaging ◽

Imaging Spectrometer ◽

Long Wave

Download Full-text

THE GWAC DATA PROCESSING AND MANAGEMENT SYSTEM

Revista Mexicana de Astronomía y Astrofísica Serie de Conferencias ◽

10.22201/ia.14052059p.2021.53.36 ◽

2021 ◽

Vol 53 ◽

pp. 174-179

Author(s):

Y. Xu ◽

L. P. Xin ◽

X. H. Han ◽

H. B. Cai ◽

L. Huang ◽

...

Keyword(s):

Data Processing ◽

User Interfaces ◽

Management System ◽

Scientific Data ◽

Data Archiving ◽

Processing Pipeline ◽

Advanced Technologies ◽

Single Frame ◽

Moonless Night ◽

Processing Requirement

GWAC will have been built an integrated FOV of 5,000 degree2 and have already built 1,800 square degree2. The limit magnitude of a 10-second exposure image in the moonless night is 16R. In each observation night, GWAC produces about 0.7TB of raw data, and the data processing pipeline generates millions of single frame alerts. We describe the GWAC Data Processing and Management System (GPMS), including hardware architecture, database, detection-filtering-validation of transient candidates, data archiving, and user interfaces for the check of transient and the monitor of the system. GPMS combines general technology and software in astronomy and computer field, and use some advanced technologies such as deep learning. Practical results show that GPMS can fully meet the scientific data processing requirement of GWAC. It can online accomplish the detection, filtering and validation of millions of transient candidates, and feedback the final results to the astronomer in real-time. During the observation from October of 2018 to December of 2019, we have already found 102 transients.

Download Full-text

Technical design: data processing pipeline in eHealth

10.1016/b978-0-12-823818-9.00023-7 ◽

2022 ◽

pp. 259-283

Author(s):

Patrick Schneider ◽

Fatos Xhafa

Keyword(s):

Data Processing ◽

Technical Design ◽

Design Data ◽

Processing Pipeline

Download Full-text

The Solar Orbiter SPICE instrument

Astronomy and Astrophysics ◽

10.1051/0004-6361/201935574 ◽

2020 ◽

Vol 642 ◽

pp. A14 ◽

Cited By ~ 7

Author(s):

◽

M. Anderson ◽

T. Appourchaux ◽

F. Auchère ◽

R. Aznar Cuadrado ◽

...

Keyword(s):

High Resolution ◽

Data Processing ◽

Extreme Ultraviolet ◽

High Resolution Imaging ◽

Concept Design ◽

Performance Measurements ◽

Imaging Spectrometer ◽

Scientific Success ◽

Resolution Imaging ◽

Orbiter Mission

Aims. The Spectral Imaging of the Coronal Environment (SPICE) instrument is a high-resolution imaging spectrometer operating at extreme ultraviolet wavelengths. In this paper, we present the concept, design, and pre-launch performance of this facility instrument on the ESA/NASA Solar Orbiter mission. Methods. The goal of this paper is to give prospective users a better understanding of the possible types of observations, the data acquisition, and the sources that contribute to the instrument’s signal. Results. The paper discusses the science objectives, with a focus on the SPICE-specific aspects, before presenting the instrument’s design, including optical, mechanical, thermal, and electronics aspects. This is followed by a characterisation and calibration of the instrument’s performance. The paper concludes with descriptions of the operations concept and data processing. Conclusions. The performance measurements of the various instrument parameters meet the requirements derived from the mission’s science objectives. The SPICE instrument is ready to perform measurements that will provide vital contributions to the scientific success of the Solar Orbiter mission.

Download Full-text

Optimizing a Whole-Genome Sequencing Data Processing Pipeline for Precision Surveillance of Health Care-Associated Infections

Microorganisms ◽

10.3390/microorganisms7100388 ◽

2019 ◽

Vol 7 (10) ◽

pp. 388 ◽

Cited By ~ 1

Author(s):

Weihua Huang ◽

Guiqing Wang ◽

Changhong Yin ◽

Donald Chen ◽

Abhay Dhand ◽

...

Keyword(s):

Health Care ◽

Data Processing ◽

Whole Genome Sequencing ◽

Genome Sequencing ◽

Structural Changes ◽

Essential Element ◽

Whole Genome Sequencing Data ◽

Whole Genome ◽

Infection Control Program ◽

Processing Pipeline

The surveillance of health care-associated infection (HAI) is an essential element of the infection control program. While whole-genome sequencing (WGS) has widely been adopted for genomic surveillance, its data processing remains to be improved. Here, we propose a three-level data processing pipeline for the precision genomic surveillance of microorganisms without prior knowledge: species identification, multi-locus sequence typing (MLST), and sub-MLST clustering. The former two are closely connected to what have widely been used in current clinical microbiology laboratories, whereas the latter one provides significantly improved resolution and accuracy in genomic surveillance. Comparing to a broadly used reference-dependent alignment/mapping method and an annotation-dependent pan-/core-genome analysis, we implemented our reference- and annotation-independent, k-mer-based, simplified workflow to a collection of Acinetobacter and Enterococcus clinical isolates for tests. By taking both single nucleotide variants and genomic structural changes into account, the optimized k-mer-based pipeline demonstrated a global view of bacterial population structure in a rapid manner and discriminated the relatedness between bacterial isolates in more detail and precision. The newly developed WGS data processing pipeline would facilitate WGS application to the precision genomic surveillance of HAI. In addition, the results from such a WGS-based analysis would be useful for the precision laboratory diagnosis of infectious microorganisms.

Download Full-text

High performance distributed data processing pipeline for Chinese Spectral RadioHeliograph

2015 1st URSI Atlantic Radio Science Conference (URSI AT-RASC) ◽

10.1109/ursi-at-rasc.2015.7303172 ◽

2015 ◽

Author(s):

Feng Wang ◽

Hui Deng ◽

Wei Wang

Keyword(s):

Data Processing ◽

High Performance ◽

Distributed Data ◽

Processing Pipeline ◽

Distributed Data Processing

Download Full-text

3D prediction of surface-related and interbed multiples

Geophysics ◽

10.1190/1.2159062 ◽

2006 ◽

Vol 71 (1) ◽

pp. V1-V6 ◽

Cited By ~ 10

Author(s):

Moshe Reshef ◽

Shahar Arad ◽

Evgeny Landa

Keyword(s):

Data Processing ◽

Time Domain ◽

Final Section ◽

Real Data ◽

Computational Procedure ◽

Detailed Knowledge ◽

Geological Model ◽

Multiple Attenuation ◽

The Time Domain ◽

Kinematic Properties

Multiple attenuation during data processing does not guarantee a multiple-free final section. Multiple identification plays an important role in seismic interpretation. A target-oriented method for predicting 3D multiples on stacked or migrated cubes in the time domain is presented. The method does not require detailed knowledge of the subsurface geological model or access to prestack data and is valid for both surface-related and interbed multiples. The computational procedure is based on kinematic properties of the data and uses Fermat's principle to define the multiples. Since no prestack data are required, the method can calculate 3D multiples even when only multi-2D survey data are available. The accuracy and possible use of the method are demonstrated on synthetic and real data examples.

Download Full-text