Segment backup: datacenter-disaster tolerance for stream processing systems

Stream processing systems;Fault tolerance

Shen, Juanli ; Winslett ; Marianne Southall

关键词: Stream processing systems;    Fault tolerance;   

Others  :  https://www.ideals.illinois.edu/bitstream/handle/2142/97603/SHEN-THESIS-2017.pdf?sequence=1&isAllowed=y

美国|英语

来源: The Illinois Digital Environment for Access to Learning and Scholarship

【 摘 要 】

We design a datacenter-disaster tolerance solution called segment backup for stream processing systems. During regular running, segment backup inserts barriers into the normal tuple stream to indicate the backup versions and ensure node-level synchronization of multiple input streams. Additionally, segment backup materializes the partial results at some intermediate nodes to reduce reprocessing work after failures. The simple restarting logic can construct global consistency by connecting the segment consistency with the help of the materialized partial results. This method gives each segment flexibility to roll back to its latest version. We implement a prototype stream processing system with segment backup. Our experimental results show that segment backup can shorten the recovery delay with acceptable overhead during normal execution.