scholarly journals C3PO - A Dynamic Data Placement Agent for ATLAS Distributed Data Management

2017 ◽  
Vol 898 ◽  
pp. 062012 ◽  
Author(s):  
T Beermann ◽  
M Lassnig ◽  
M Barisits ◽  
C Serfon ◽  
V Garonne ◽  
...  
Keyword(s):  
Data Management ◽  
Data Placement ◽  
Distributed Data ◽  
Dynamic Data ◽  
Distributed Data Management

scholarly journals Integrating a dynamic data federation into the ATLAS distributed data management system

2019 ◽  
Vol 214 ◽  
pp. 07009
Author(s):  
Frank Berghaus ◽  
Tobias Wegner ◽  
Mario Lassnig ◽  
Marcus Ebert ◽  
Cedric Serfon ◽  
...  
Keyword(s):  
Data Management ◽  
Management System ◽  
Data Management System ◽  
Distributed Data ◽  
Dynamic Data ◽  
Distributed Data Management ◽  
Data Federation ◽  
Other Information ◽  
Multiple Copies ◽  
Set Up

Input data for applications that run in cloud computing centres can be stored at remote repositories, typically with multiple copies of the most popular data stored at many sites. Locating and retrieving the remote data can be challenging, and we believe that federating the storage can address this problem. In this approach, the closest copy of the data is used based on geographical or other information. Currently, we are using the dynamic data federation, Dynafed, a software solution developed by CERN IT. Dynafed supports several industry standard interfaces, such as Amazon S3, Microsoft Azure and HTTP with WebDAV extensions. Dynafed functions as an abstraction layer under which protocol-dependent authentication details are hidden from the user, requiring the user to only provide an X509 certificate. We have set up an instance of Dynafed and integrated it into the ATLAS distributed data management system, Rucio. We report on the challenges faced during the installation and integration.

scholarly journals Experience of a low-maintenance distributed data management system

2014 ◽  
Vol 513 (3) ◽  
pp. 032095 ◽  
Author(s):  
Wataru Takase ◽  
Yoshimi Matsumoto ◽  
Adil Hasan ◽  
Francesca Di Lodovico ◽  
Yoshiyuki Watase ◽  
...  
Keyword(s):  
Data Management ◽  
Management System ◽  
Data Management System ◽  
Distributed Data ◽  
Distributed Data Management

scholarly journals Integration of Rucio in Belle II

2021 ◽  
Vol 251 ◽  
pp. 02057
Author(s):  
Cédric Serfon ◽  
Ruslan Mashinistov ◽  
John Steven De Stefano ◽  
Michel Hernández Villanueva ◽  
Hironori Ito ◽  
...  
Keyword(s):  
Data Management ◽  
Smooth Transition ◽  
Distributed Data ◽  
Simulation Data ◽  
Migration Strategy ◽  
Distributed Data Management ◽  
Order Of Magnitude ◽  
Belle Ii ◽  
Work Done ◽  
Data Management Software

The Belle II experiment, which started taking physics data in April 2019, will multiply the volume of data currently stored on its nearly 30 storage elements worldwide by one order of magnitude to reach about 340 PB of data (raw and Monte Carlo simulation data) by the end of operations. To tackle this massive increase and to manage the data even after the end of the data taking, it was decided to move the Distributed Data Management software from a homegrown piece of software to a widely used Data Management solution in HEP and beyond : Rucio. This contribution describes the work done to integrate Rucio with Belle II distributed computing infrastructure as well as the migration strategy that was successfully performed to ensure a smooth transition.

Sign in / Sign up

Export Citation Format

Share Document