scholarly journals Evolution of the open-source data management system Ru-cio for LHC Run-3 and beyond ATLAS

2019 ◽  
Vol 214 ◽  
pp. 04054
Author(s):  
Martin Barisits ◽  
Thomas Beermann ◽  
Joaquin Bogado ◽  
Vincent Garonne ◽  
Tomas Javurek ◽  
...  
Keyword(s):  
Data Management ◽  
Open Source ◽  
Management System ◽  
High Energy Physics ◽  
High Energy ◽  
Full Scale ◽  
Data Management System ◽  
Distributed Data ◽  
Open Source Data ◽  
Level Data

Rucio, the distributed data management system of the ATLAS experiment already manages more than 400 Petabytes of physics data on the grid. Rucio was incrementally improved throughout LHC Run-2 and is currently being prepared for the HL-LHC era of the experiment. Next to these improvements the system is currently evolving into a full-scale generic data management system for application beyond ATLAS, or even beyond high-energy physics. This contribution focuses on the development roadmap of Rucio for LHC Run-3, such as event level data management, generic meta-data support and increased usage of networks and tapes. At the same time Rucio is evolving beyond the original ATLAS requirements. This includes additional authentication mechanisms, generic database compatibility, deployment and packaging of the software stack in containers, and a project paradigm shift to a full-scale open source project..

scholarly journals JAZELLE: An enhanced data management system for high energy physics

1990 ◽  
Author(s):  
A. S. Johnson ◽  
M. I. Briedenbach ◽  
H. Hissen ◽  
P. F. Kunz ◽  
D. J. Sherden ◽  
...  
Keyword(s):  
Data Management ◽  
Management System ◽  
High Energy Physics ◽  
High Energy ◽  
Data Management System ◽  
Energy Physics

scholarly journals Experience of a low-maintenance distributed data management system

2014 ◽  
Vol 513 (3) ◽  
pp. 032095 ◽  
Author(s):  
Wataru Takase ◽  
Yoshimi Matsumoto ◽  
Adil Hasan ◽  
Francesca Di Lodovico ◽  
Yoshiyuki Watase ◽  
...  
Keyword(s):  
Data Management ◽  
Management System ◽  
Data Management System ◽  
Distributed Data ◽  
Distributed Data Management

UHF RFID proof of concept (POC) with open-source ILS at Universiti Sains Malaysia (USM) Libraries

2015 ◽  
Vol 49 (2) ◽  
pp. 135-150 ◽  
Author(s):  
JingHuey Khor ◽  
Widad Ismail ◽  
Mohd Nasir Md Rashid ◽  
A. Akla Wan Ismail ◽  
Muhammad Qayum Omar ◽  
...  
Keyword(s):  
Data Management ◽  
Open Source ◽  
Management System ◽  
Data Management System ◽  
Proof Of Concept ◽  
Rfid Technology ◽  
Uhf Rfid ◽  
Content Type ◽  
Library Automation ◽  
Detection Technologies

Purpose – Universiti Sains Malaysia (USM) uses different detection technologies and integrated library systems (ILS) at its four libraries located at different sites. Hence, ubiquitous data management system is required in the four libraries to enable real-time data sharing and tracking among the libraries. The paper aims to discuss these issues. Design/methodology/approach – Proof of concept is used to verify the performance of integration between different detection technologies including barcode, high frequency (HF) and ultra high frequency (UHF) radio frequency identification (RFID) as well as Integrated Library Management Utility (ILMU) and Koha ILSs. Five pilot tests are performed to investigate the speed of the borrowing and returning transactions, the efficiency of security gate to detect non-borrowed items, and the reliability of the ubiquitous data management system. Findings – The HF RFID technology and the proprietary library system have been used in many libraries. The current study proves that the integration of the UHF RFID technology and the Koha open-source ILS, which is named as Total Open-Source Library Automation Solution, can be a novel solution and can serve as a guide for other libraries. Originality/value – In the current phase of library data management system, it is common to integrate various technologies in improving the efficiency of the library system. But, the combination of Koha with UHF RFID into a single ILS for intra-library automation system has not been proven. This shows the importance of the proposed proven proof of concept that proves such integration is suitable to be implemented at USM libraries and other libraries.

scholarly journals Integrating a dynamic data federation into the ATLAS distributed data management system

2019 ◽  
Vol 214 ◽  
pp. 07009
Author(s):  
Frank Berghaus ◽  
Tobias Wegner ◽  
Mario Lassnig ◽  
Marcus Ebert ◽  
Cedric Serfon ◽  
...  
Keyword(s):  
Data Management ◽  
Management System ◽  
Data Management System ◽  
Distributed Data ◽  
Dynamic Data ◽  
Distributed Data Management ◽  
Data Federation ◽  
Other Information ◽  
Multiple Copies ◽  
Set Up

Input data for applications that run in cloud computing centres can be stored at remote repositories, typically with multiple copies of the most popular data stored at many sites. Locating and retrieving the remote data can be challenging, and we believe that federating the storage can address this problem. In this approach, the closest copy of the data is used based on geographical or other information. Currently, we are using the dynamic data federation, Dynafed, a software solution developed by CERN IT. Dynafed supports several industry standard interfaces, such as Amazon S3, Microsoft Azure and HTTP with WebDAV extensions. Dynafed functions as an abstraction layer under which protocol-dependent authentication details are hidden from the user, requiring the user to only provide an X509 certificate. We have set up an instance of Dynafed and integrated it into the ATLAS distributed data management system, Rucio. We report on the challenges faced during the installation and integration.

scholarly journals Monitoring the atlas distributed data management system

2008 ◽  
Vol 119 (7) ◽  
pp. 072027
Author(s):  
R Ricardo ◽  
B Miguel ◽  
G Benjamin ◽  
G Vincent ◽  
L Mario ◽  
...  
Keyword(s):  
Data Management ◽  
Management System ◽  
Data Management System ◽  
Distributed Data ◽  
Distributed Data Management

scholarly journals Enabling Data Intensive Science on Supercomputers for High Energy Physics R&D Projects in HL-LHC Era

2020 ◽  
Vol 226 ◽  
pp. 01007
Author(s):  
Alexei Klimentov ◽  
Douglas Benjamin ◽  
Alessandro Di Girolamo ◽  
Kaushik De ◽  
Johannes Elmsheuser ◽  
...  
Keyword(s):  
Data Storage ◽  
Management System ◽  
High Energy Physics ◽  
Hadron Collider ◽  
High Energy ◽  
Data Management System ◽  
Workload Management ◽  
Data Intensive ◽  
Community Needs ◽  
Energy Physics

The ATLAS experiment at CERN’s Large Hadron Collider uses theWorldwide LHC Computing Grid, the WLCG, for its distributed computing infrastructure. Through the workload management system PanDA and the distributed data management system Rucio, ATLAS provides seamless access to hundreds of WLCG grid and cloud based resources that are distributed worldwide, to thousands of physicists. PanDA annually processes more than an exabyte of data using an average of 350,000 distributed batch slots, to enable hundreds of new scientific results from ATLAS. However, the resources available to the experiment have been insufficient to meet ATLAS simulation needs over the past few years as the volume of data from the LHC has grown. The problem will be even more severe for the next LHC phases. High Luminosity LHC will be a multiexabyte challenge where the envisaged Storage and Compute needs are a factor 10 to 100 above the expected technology evolution. The High Energy Physics (HEP) community needs to evolve current computing and data organization models in order to introduce changes in the way it uses and manages the infrastructure, focused on optimizations to bring performance and efficiency not forgetting simplification of operations. In this paper we highlight recent R&D projects in HEP related to data lake prototype, federated data storage and data carousel.

Sign in / Sign up

Export Citation Format

Share Document