Data, Meet Compute: NASA's Cumulus Ingest Architecture

Quinn, Patrick

关键词: EARTH OBSERVING SYSTEM (EOS);    EOS DATA AND INFORMATION SYSTEM;    DATA SYSTEMS;    INFORMATION SYSTEMS;    EARTH SCIENCES;    COMMUNICATING;    TIME SERIES ANALYSIS;    INFORMATION DISSEMINATION;    NASA PROGRAMS;    FORMAT;    STEP FUNCTIONS;   

RP-ID  :  GSFC-E-DAA-TN50937

美国|英语

来源: NASA Technical Reports Server

【 摘 要 】

NASA's Earth Observing System Data and Information System (EOSDIS) houses nearly 30PBs of critical Earth Science data and with upcoming missions is expected to balloon to between 200PBs-300PBs over the next seven years. In addition to the massive increase in data collected, researchers and application developers want more and faster access - enabling complex visualizations, long time-series analysis, and cross dataset research without needing to copy and manage massive amounts of data locally. NASA has looked to the cloud to address these needs, building its Cumulus system to manage the ingest of diverse data in a wide variety of formats into the cloud. In this talk, we look at what Cumulus is from a high level and then take a deep dive into how it manages complexity and versioning associated with multiple AWS Lambda and ECS microservices communicating through AWS Step Functions across several disparate installations