Efficient Detection of Large Scale Redundancy in Enterprise File Systems

Forman, George ; Eshghi, Kave ; Suermondt, Jaap

HP Development Company

关键词: data mining;    min-hashing;    set sketches;    directory similarity and deduplication;    file systems;    scalability;    storage management.;   

RP-ID  :  HPL-2008-30R2

学科分类：计算机科学（综合）

美国|英语

来源: HP Labs

【 摘 要 】

In order to catch and reduce waste in the exponential demand for disk storage, we have developed a technology based on set sketches that enables enterprise storage managers to efficiently detect approximate duplication of large directory hierarchies, e.g. unnecessary mirroring by uncoordinated employees or departments. Identifying these duplicate or near duplicate hierarchies allows appropriate action to be taken at a high level, e.g. coordinate and consolidate multiple copies in one location.