FY08 LDRD Final Report LOCAL: Locality-Optimizing Caching Algorithms and Layouts

Lindstrom, P

关键词: ALGORITHMS;    COMPRESSION;    DATA ANALYSIS;    PROCESSING;    SIMULATION;    STORAGE;   

DOI  :  10.2172/979439
RP-ID  :  LLNL-TR-410985
PID  :  OSTI ID: 979439
Others  :  TRN: US201010%%822

学科分类：社会科学、人文和艺术（综合）

美国|英语

来源: SciTech Connect

【 摘 要 】

This project investigated layout and compression techniques for large, unstructured simulation data to reduce bandwidth requirements and latency in simulation I/O and subsequent post-processing, e.g. data analysis and visualization. The main goal was to eliminate the data-transfer bottleneck - for example, from disk to memory and from central processing unit to graphics processing unit - through coherent data access and by trading underutilized compute power for effective bandwidth and storage. This was accomplished by (1) designing algorithms that both enforce and exploit compactness and locality in unstructured data, and (2) adapting offline computations to a novel stream processing framework that supports pipelining and low-latency sequential access to compressed data. This report summarizes the techniques developed and results achieved, and includes references to publications that elaborate on the technical details of these methods.