Scientific Data Management and Application in High Energy Physics

2019 ◽  
pp. 92-104
Author(s):  
Gang Chen ◽  
Yaodong Cheng
Keyword(s):  
Data Management ◽  
High Energy Physics ◽  
High Energy ◽  
Scientific Data ◽  
Scientific Data Management ◽  
Energy Physics

scholarly journals TECHNOLOGIES FOR LARGE DATA MANAGEMENT IN SCIENTIFIC COMPUTING

2014 ◽  
Vol 25 (02) ◽  
pp. 1430001 ◽  
Author(s):  
ALBERTO PACE
Keyword(s):  
Data Management ◽  
High Energy Physics ◽  
Hadron Collider ◽  
Scientific Research ◽  
Large Data ◽  
High Energy ◽  
Worldwide Distribution ◽  
Systematic Collection ◽  
Energy Physics

In recent years, intense usage of computing has been the main strategy of investigations in several scientific research projects. The progress in computing technology has opened unprecedented opportunities for systematic collection of experimental data and the associated analysis that were considered impossible only few years ago. This paper focuses on the strategies in use: it reviews the various components that are necessary for an effective solution that ensures the storage, the long term preservation, and the worldwide distribution of large quantities of data that are necessary in a large scientific research project. The paper also mentions several examples of data management solutions used in High Energy Physics for the CERN Large Hadron Collider (LHC) experiments in Geneva, Switzerland which generate more than 30,000 terabytes of data every year that need to be preserved, analyzed, and made available to a community of several tenth of thousands scientists worldwide.

scholarly journals JAZELLE: An enhanced data management system for high energy physics

1990 ◽  
Author(s):  
A. S. Johnson ◽  
M. I. Briedenbach ◽  
H. Hissen ◽  
P. F. Kunz ◽  
D. J. Sherden ◽  
...  
Keyword(s):  
Data Management ◽  
Management System ◽  
High Energy Physics ◽  
High Energy ◽  
Data Management System ◽  
Energy Physics

scholarly journals Rucio beyond ATLAS: experiences from Belle II, CMS, DUNE, EISCAT3D, LIGO/VIRGO, SKA, XENON

2020 ◽  
Vol 245 ◽  
pp. 11006 ◽  
Author(s):  
Mario Lassnig ◽  
Martin Barisits ◽  
Paul J Laycock ◽  
Cédric Serfon ◽  
Eric W Vaandering ◽  
...  
Keyword(s):  
Data Management ◽  
High Energy Physics ◽  
High Energy ◽  
Data Management System ◽  
Neutrino Experiment ◽  
Dark Matter Search ◽  
Long Term Storage ◽  
Belle Ii ◽  
Physics Experiment ◽  
Energy Physics

For many scientific projects, data management is an increasingly complicated challenge. The number of data-intensive instruments generating unprecedented volumes of data is growing and their accompanying workflows are becoming more complex. Their storage and computing resources are heterogeneous and are distributed at numerous geographical locations belonging to different administrative domains and organisations. These locations do not necessarily coincide with the places where data is produced nor where data is stored, analysed by researchers, or archived for safe long-term storage. To fulfil these needs, the data management system Rucio has been developed to allow the high-energy physics experiment ATLAS at LHC to manage its large volumes of data in an efficient and scalable way. But ATLAS is not alone, and several diverse scientific projects have started evaluating, adopting, and adapting the Rucio system for their own needs. As the Rucio community has grown, many improvements have been introduced, customisations have been added, and many bugs have been fixed. Additionally, new dataflows have been investigated and operational experiences have been documented. In this article we collect and compare the common successes, pitfalls, and oddities that arose in the evaluation efforts of multiple diverse experiments, and compare them with the ATLAS experience. This includes the high-energy physics experiments Belle II and CMS, the neutrino experiment DUNE, the scattering radar experiment EISCAT3D, the gravitational wave observatories LIGO and VIRGO, the SKA radio telescope, and the dark matter search experiment XENON.

scholarly journals MetaCat - metadata catalog for data management systems

2021 ◽  
Vol 251 ◽  
pp. 02048
Author(s):  
Mandrichenko Igor
Keyword(s):  
Data Management ◽  
High Energy Physics ◽  
Workflow Management ◽  
High Energy ◽  
Scientific Data ◽  
Data Item ◽  
Data Management System ◽  
Metadata Management ◽  
Replica Management ◽  
Recording Conditions

Metadata management is one of three major areas of scientific data management along with replica management and workflow management. Metadata is the information describing the data stored in a data item, a file or an object. It includes the data item provenance, recording conditions, format and other attributes. MetaCat is a metadata management database designed and developed for High Energy Physics experiments. As a component of a data management system, it’s main objectives are to provide efficient metadata storage and management and fast data selection functionality. MetaCat is required to work on the scale of 100 million files (or objects) and beyond. The article will discuss the functionality of MetaCat and technological solutions used to implement the product.

scholarly journals Exploring the virtues of XRootD5: Declarative API

2021 ◽  
Vol 251 ◽  
pp. 02063
Author(s):  
Michal Simon ◽  
Andrew Hanushevsky
Keyword(s):  
Data Management ◽  
Software Quality ◽  
High Energy Physics ◽  
Positive Impact ◽  
Building Blocks ◽  
High Energy ◽  
Storage Solutions ◽  
Numerous Data ◽  
Client Side ◽  
Energy Physics

Across the years, being the backbone of numerous data management solutions used within the WLCG collaboration, the XRootD framework and protocol became one of the most important building blocks for storage solutions in the High Energy Physics (HEP) community. The latest big milestone for the project, release 5, introduced multitude of architectural improvements and functional enhancements, including the new client side declarative API, which is the main focus of this study. In this contribution, we give an overview of the new client API and we discuss its motivation and its positive impact on overall software quality (coupling, cohesion), readability and composability.

scholarly journals Data management for high energy physics experiments — Preliminary proposals

1987 ◽  
Vol 45 (1-3) ◽  
pp. 379-383
Author(s):  
Frank Olken ◽  
Stewart C. Loken ◽  
Doron Rotem ◽  
Arie Shoshani ◽  
Thomas G. Trippe
Keyword(s):  
Data Management ◽  
High Energy Physics ◽  
High Energy ◽  
Physics Experiments ◽  
Energy Physics

High-energy physics and the nature of matter

1965 ◽  
Vol 86 (8) ◽  
pp. 589-590
Author(s):  
E.V. Shpol'skii
Keyword(s):  
High Energy Physics ◽  
High Energy ◽  
Energy Physics ◽  
Nature Of Matter
Sign in / Sign up

Export Citation Format

Share Document