Puncturable Search: Enabling Authorized Search in Cross-data Federation

The implementation of cache systems in the computing model of HEP experiments enables to accelerate access to hot data sets by scientists, opening new scenarios of data distribution and enable to exploit the paradigm of storage-less sites. In this work, we present a study for the creation of an http data-federation ecosystem with caching functionality. We created plug-in integrated in the logic of a DPM Storage, able to reproduce a cache behaviour, taking advantage from the new feature introduced in the last version of Disk Pool Manager, called volatile-pool. Then we used Dynafed as lightweight federation system to aggregate a set of standard Grid Storage together with the caching system. With the designed setup, clients asking for a file present on the Data-Grid are automatically redirected to the cache, if the cache is the closest storage, thanks to the action of the geo-plugin run by Dynafed. As proof of the concept, we tested the whole system in a controlled environment within the Belle II computing infrastructure using a set of files located in production Storage Elements. Preliminary results demonstrate the proper functionality of the logic and encourage continuing the work.

Download Full-text

DataONE: A Data Federation with Provenance Support

Lecture Notes in Computer Science - Provenance and Annotation of Data and Processes ◽

10.1007/978-3-319-40593-3_28 ◽

2016 ◽

pp. 230-234 ◽

Cited By ~ 2

Author(s):

Yang Cao ◽

Christopher Jones ◽

Víctor Cuevas-Vicenttín ◽

Matthew B. Jones ◽

Bertram Ludäscher ◽

...

Keyword(s):

Data Federation

Download Full-text

Integrating a dynamic data federation into the ATLAS distributed data management system

EPJ Web of Conferences ◽

10.1051/epjconf/201921407009 ◽

2019 ◽

Vol 214 ◽

pp. 07009

Author(s):

Frank Berghaus ◽

Tobias Wegner ◽

Mario Lassnig ◽

Marcus Ebert ◽

Cedric Serfon ◽

...

Keyword(s):

Data Management ◽

Management System ◽

Data Management System ◽

Distributed Data ◽

Dynamic Data ◽

Distributed Data Management ◽

Data Federation ◽

Other Information ◽

Multiple Copies ◽

Set Up

Input data for applications that run in cloud computing centres can be stored at remote repositories, typically with multiple copies of the most popular data stored at many sites. Locating and retrieving the remote data can be challenging, and we believe that federating the storage can address this problem. In this approach, the closest copy of the data is used based on geographical or other information. Currently, we are using the dynamic data federation, Dynafed, a software solution developed by CERN IT. Dynafed supports several industry standard interfaces, such as Amazon S3, Microsoft Azure and HTTP with WebDAV extensions. Dynafed functions as an abstraction layer under which protocol-dependent authentication details are hidden from the user, requiring the user to only provide an X509 certificate. We have set up an instance of Dynafed and integrated it into the ATLAS distributed data management system, Rucio. We report on the challenges faced during the installation and integration.

Download Full-text