Automated SQL query generation for file search operations in a scale out file system

Keeton, Kimberly ; Sombrio, Evandro ; Nunes, Leandro ; Veitch, Alistair ; Zangaro, Sebastian

HP Development Company

关键词: SQL query generation;    eventually consistent database;    scale out file system;    file system search;    REST API;   

RP-ID  :  HPL-2014-73

学科分类：计算机科学（综合）

美国|英语

来源: HP Labs

【 摘 要 】

The explosive growth in unstructured (file) data in today's IT systems causes significant information management and compliance issues. For example, system administrators need to quickly and efficiently find files that match a given criteria, applications need to "tag" files with custom metadata and query that metadata, utilities need to efficiently determine which files have changed and are in need of backup, and legal staff need to find files that meet e- discovery criteria. In today's huge file systems, comprising billions of files and petabytes of data, these operations can be extremely difficult and time consuming. HP's StoreAll with ExpressQuery provides a scale-out retentionenabled file system for use in archival applications, coupled with a scalable embedded database to accelerate metadata queries. StoreAll's REST API enables users to assign custom metadata to files, and to query system and custom metadata from Express Query efficiently. Queries through the REST API offer over 30,000-fold speedups over file system scans for common file management queries; this speedup increases as the size of the file system grows. This paper summarizes the issues we addressed in translating REST API requests into efficient SQL queries to the Express Query database.