Metadata models for ad hoc queries on terabyte-scale scientific simulations

Journal of the Brazilian Computer Society

Critchlow, Terence¹  Center for Applied Scientific Computing Lawrence Livermore National Laboratory, Livermore, U.S.A.¹  Musick, Ron¹  Snapp, Robert R.¹  Lee, Byung S.¹  University of Vermont Burlington, Vermont, U.S.A.¹ 

关键词: scientific databases;    ad hoc queries.;   

DOI  :  10.1590/S0104-65002002000100002

学科分类：农业科学（综合）

来源: Springer U K

【 摘 要 】

We present our approach to enabling approximate ad hoc queries on terabyte-scale mesh data generated from large scientific simulations through the extension and integration of database, statistical, and data mining techniques. There are several significant barriers to overcome in achieving this objective. First, large-scale simulation data is already at the multi-terabyte scale and growing quickly, thus rendering traditional forms of interactive data exploration and query processing untenable. Second, a priori knowledge of user queries is not available, making it impossible to tune special-purpose solutions. Third, the data has spatial and temporal aspects, as well as arbitrarily high dimensionality, which exacerbates the task of finding compact, accurate, and easy-to-compute data models.Our approach is to preprocess the mesh data to generate highly compressed, lossy models that are used in lieu of the original data to answer users' queries. This approach leads to interesting challenges. The model (equivalently, the content-oriented metadata) being generated must be smaller than the original data by at least an order of magnitude. Second, the metadata representation must contain enough information to support a broad class of queries. Finally, the accuracy and speed of the queries must be within the tolerances required by users. In this paper we give an overview of ongoing development efforts with an emphasis on extracting metadata and using it in query processing.

【 授权许可】

Unknown