A framework for evaluating epidemic forecasts

BMC Infectious Diseases

Technical Advance

Prithwish Chakraborty¹  Farzaneh Sadat Tabataba²  Madhav Marathe²  Naren Ramakrishnan²  Srinivasan Venkatramanan³  Jiangzhuo Chen³  Bryan Lewis³ 

[1] Computer Science Department, Virginia Tech, 2202 Kraft Drive, 24060, Blacksburg/Virginia, USA;Computer Science Department, Virginia Tech, 2202 Kraft Drive, 24060, Blacksburg/Virginia, USA;Network Dynamics and Simulation Science Laboratory (NDSSL), Biocomplexity Institute, Virginia Tech, 1015 Life Science Cir, 24061, Blacksburg/Virginia, USA;Network Dynamics and Simulation Science Laboratory (NDSSL), Biocomplexity Institute, Virginia Tech, 1015 Life Science Cir, 24061, Blacksburg/Virginia, USA;

关键词: Epidemic forecasting;    Error Measure;    Performance evaluation;    Epidemic-Features;    Ranking;   

DOI  :  10.1186/s12879-017-2365-1

 received in 2016-07-28, accepted in 2017-03-29,  发布年份 2017

来源: Springer

【 摘 要 】

BackgroundOver the past few decades, numerous forecasting methods have been proposed in the field of epidemic forecasting. Such methods can be classified into different categories such as deterministic vs. probabilistic, comparative methods vs. generative methods, and so on. In some of the more popular comparative methods, researchers compare observed epidemiological data from the early stages of an outbreak with the output of proposed models to forecast the future trend and prevalence of the pandemic. A significant problem in this area is the lack of standard well-defined evaluation measures to select the best algorithm among different ones, as well as for selecting the best possible configuration for a particular algorithm.ResultsIn this paper we present an evaluation framework which allows for combining different features, error measures, and ranking schema to evaluate forecasts. We describe the various epidemic features (Epi-features) included to characterize the output of forecasting methods and provide suitable error measures that could be used to evaluate the accuracy of the methods with respect to these Epi-features. We focus on long-term predictions rather than short-term forecasting and demonstrate the utility of the framework by evaluating six forecasting methods for predicting influenza in the United States. Our results demonstrate that different error measures lead to different rankings even for a single Epi-feature. Further, our experimental analyses show that no single method dominates the rest in predicting all Epi-features when evaluated across error measures. As an alternative, we provide various Consensus Ranking schema that summarize individual rankings, thus accounting for different error measures. Since each Epi-feature presents a different aspect of the epidemic, multiple methods need to be combined to provide a comprehensive forecast. Thus we call for a more nuanced approach while evaluating epidemic forecasts and we believe that a comprehensive evaluation framework, as presented in this paper, will add value to the computational epidemiology community.

【 授权许可】

CC BY   
© The Author(s) 2017

【 图 表 】

12864_2016_2871_Article_IEq15.gif

12885_2015_Article_1803_TeX2GIF_IEq2.gif

12864_2017_4179_Article_IEq28.gif

12864_2015_2192_Article_IEq19.gif

12864_2017_3916_Article_IEq1.gif

12864_2016_2880_Article_IEq26.gif

12864_2017_4132_Article_IEq12.gif

12864_2017_4004_Article_IEq17.gif

12864_2017_4228_Article_IEq2.gif

12864_2017_4226_Article_IEq2.gif

12864_2017_4132_Article_IEq6.gif

12864_2016_2392_Article_IEq1.gif

12864_2017_4186_Article_IEq31.gif

12864_2016_2821_Article_IEq60.gif

12864_2017_3771_Article_IEq2.gif

12864_2017_4004_Article_IEq9.gif

12864_2017_4004_Article_IEq8.gif

12888_2016_811_Article_IEq1.gif

12864_2017_4186_Article_IEq25.gif

12864_2015_2055_Article_IEq5.gif

12864_2017_4030_Article_IEq17.gif

12864_2016_3001_Article_IEq3.gif

12864_2016_3477_Article_IEq11.gif

12864_2015_2296_Article_IEq24.gif

12864_2016_2821_Article_IEq47.gif

12864_2017_4030_Article_IEq13.gif

12864_2017_4030_Article_IEq12.gif

12880_2015_Article_54_TeX2GIF_IEq16.gif

12864_2016_2796_Article_IEq9.gif

12864_2017_4030_Article_IEq10.gif

12864_2016_2913_Article_IEq9.gif

12864_2016_2682_Article_IEq48.gif

12864_2016_3098_Article_IEq57.gif

12864_2017_3527_Article_IEq3.gif

12888_2017_1504_Article_IEq1.gif

12864_2017_4030_Article_IEq5.gif

12864_2017_4030_Article_IEq4.gif

12864_2016_3477_Article_IEq2.gif

12864_2016_2821_Article_IEq35.gif

12864_2017_4020_Article_IEq42.gif

12888_2016_848_Article_IEq2.gif

12888_2016_848_Article_IEq1.gif

12864_2015_2296_Article_IEq158.gif

12864_2016_3098_Article_IEq45.gif

12864_2017_4020_Article_IEq36.gif

12864_2016_3098_Article_IEq43.gif