Distributed processing and analysis of physics data in the D0 SAM system at Fermilab

Terekhov, Igor V.

Fermi National Accelerator Laboratory

关键词: Management;    99 General And Miscellaneous//Mathematics, Computing, And Information Science;    Physics;    Design;    71 Classical And Quantum Mechanics, General Physics;   

DOI  :  10.2172/785150
RP-ID  :  FERMILAB-TM-2156
RP-ID  :  AC02-76CH03000
RP-ID  :  785150

美国|英语

来源: UNT Digital Library

【 摘 要 】

SAM (Sequential Access through Meta-data) is the data access system for the D0 high energy physics (HEP) experiment at Fermilab. The system is being developed and used to handle the Petabyte-scale experiment data. The D0 applications, like virtually all HEP applications, are data-intensive, which poses special problems for the data management and job control facilities in the distributed environment. The fundamental problem is to bring the user applications and the data together, and SAM attacks the problems from both sides. First, we describe how the system moves the data through the distributed disk cache. Second, we describe how SAM interacts with the batch system to synchronize parallel user jobs with the data availability. All the design solutions herein have been implemented in a real system that handles the mission-critical data of the D0 experiment; thus, we present our work from the standpoint of real experience.