Understanding Lustre Internals

Wang, Feiyi ; Oral, H Sarp ; Shipman, Galen M ; Drokin, Oleg ; Wang, Di ; Huang, He

Oak Ridge National Laboratory

关键词: 99 General And Miscellaneous//Mathematics, Computing, And Information Science;    Documentation;    Supercomputers;    Parallel Processing;    L Codes;   

DOI  :  10.2172/951297
RP-ID  :  ORNL/TM-2009/117
RP-ID  :  DE-AC05-00OR22725
RP-ID  :  951297

美国|英语

来源: UNT Digital Library

【 摘 要 】

Lustre was initiated and funded, almost a decade ago, by the U.S. Department of Energy (DoE) Office of Science and National Nuclear Security Administration laboratories to address the need for an open source, highly-scalable, high-performance parallel filesystem on by then present and future supercomputing platforms. Throughout the last decade, it was deployed over numerous medium-to-large-scale supercomputing platforms and clusters, and it performed and met the expectations of the Lustre user community. As it stands at the time of writing this document, according to the Top500 list, 15 of the top 30 supercomputers in the world use Lustre filesystem. This report aims to present a streamlined overview on how Lustre works internally at reasonable details including relevant data structures, APIs, protocols and algorithms involved for Lustre version 1.6 source code base. More importantly, it tries to explain how various components interconnect with each other and function as a system. Portions of this report are based on discussions with Oak Ridge National Laboratory Lustre Center of Excellence team members and portions of it are based on our own understanding of how the code works. We, as the authors team bare all responsibilities for all errors and omissions in this document. We can only hope it helps current and future Lustre users and Lustre code developers as much as it helped us understanding the Lustre source code and its internal workings.