Archival Services and Technologies for Scientific Data

20th International Conference on Computing in High Energy and Nuclear Physics

物理学;计算机科学

Meyer, Jörg^1 ; Hardt, Marcus^1 ; Streit, Achim^1 ; Van Wezel, Jos^1

Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany^1

关键词: Current practices;    Large-scale data management;    Long term archival;    Science community;    Scientific community;    Scientific data;    Software infrastructure;    Technical tools;   

Others  :  https://iopscience.iop.org/article/10.1088/1742-6596/513/6/062033/pdf
DOI  :  10.1088/1742-6596/513/6/062033

学科分类：计算机科学（综合）

来源: IOP

【 摘 要 】

After analysis and publication, there is no need to keep experimental data online on spinning disks. For reliability and costs inactive data is moved to tape and put into a data archive. The data archive must provide reliable access for at least ten years following a recommendation of the German Science Foundation (DFG), but many scientific communities wish to keep data available much longer. Data archival is on the one hand purely a bit preservation activity in order to ensure the bits read are the same as those written years before. On the other hand enough information must be archived to be able to use and interpret the content of the data. The latter is depending on many also community specific factors and remains an areas of much debate among archival specialists. The paper describes the current practice of archival and bit preservation in use for different science communities at KIT for which a combination of organizational services and technical tools are required. The special monitoring to detect tape related errors, the software infrastructure in use as well as the service certification are discussed. Plans and developments at KIT also in the context of the Large Scale Data Management and Analysis (LSDMA) project are presented. The technical advantages of the T10 SCSI Stream Commands (SSC-4) and the Linear Tape File System (LTFS) will have a profound impact on future long term archival of large data sets.