FASTA/Q data compressors for MapReduce-Hadoop genomics: space and time savings made easy

BMC Bioinformatics

Raffaele Giancarlo¹  Giuseppe Cattaneo²  Umberto Ferraro Petrillo³  Francesco Palini³ 

[1] Dipartimento di Informatica, Università di Salerno, Fisciano, Italy;Dipartimento di Matematica ed Informatica, Università di Palermo, Palermo, Italy;Dipartimento di Scienze Statistiche, Università di Roma - La Sapienza, Rome, Italy;

关键词: MapReduce;    Hadoop;    Sequence analysis;    Data compression;   

DOI  :  10.1186/s12859-021-04063-1

来源: Springer

【 摘 要 】

BackgroundStorage of genomic data is a major cost for the Life Sciences, effectively addressed via specialized data compression methods. For the same reasons of abundance in data production, the use of Big Data technologies is seen as the future for genomic data storage and processing, with MapReduce-Hadoop as leaders. Somewhat surprisingly, none of the specialized FASTA/Q compressors is available within Hadoop. Indeed, their deployment there is not exactly immediate. Such a State of the Art is problematic.ResultsWe provide major advances in two different directions. Methodologically, we propose two general methods, with the corresponding software, that make very easy to deploy a specialized FASTA/Q compressor within MapReduce-Hadoop for processing files stored on the distributed Hadoop File System, with very little knowledge of Hadoop. Practically, we provide evidence that the deployment of those specialized compressors within Hadoop, not available so far, results in better space savings, and even in better execution times over compressed data, with respect to the use of generic compressors available in Hadoop, in particular for FASTQ files. Finally, we observe that these results hold also for the Apache Spark framework, when used to process FASTA/Q files stored on the Hadoop File System.ConclusionsOur Methods and the corresponding software substantially contribute to achieve space and time savings for the storage and processing of FASTA/Q files in Hadoop and Spark. Being our approach general, it is very likely that it can be applied also to FASTA/Q compression methods that will appear in the future.AvailabilityThe software and the datasets are available at https://github.com/fpalini/fastdoopc

【 授权许可】

CC BY