Evaluation of an open forecasting challenge to assess skill of West Nile virus neuroinvasive disease prediction

Parasites & Vectors

Research

John M. Humphreys¹  Praachi Das²  Andrew S. Freedman²  Ilia Rochlin³  Devin Kirk⁴  Mallory J. Harris⁴  Morgan P. Kain⁴  Erin A. Mordecai⁴  Nicole Nova⁴  Brandon D. Hollingsworth⁵  Nicholas DeFelice⁶  Evan L. Ray⁷  Marco Hamins-Puértolas⁸  Johnny A. Uelmen⁹  Christopher M. Barker^1,10  Matteo Marcantonio^1,11  Laura D. Kramer^1,12  Alexander C. Keyel^1,13  Charles B. Beard^1,14  Marc Fischer^1,14  Sarabeth Mathis^1,14  Randall J. Nett^1,14  J. Erin Staples^1,14  Michael A. Johansson^1,15  Marissa L. Childs^1,16  Karen M. Holcomb^1,17  Morgan E. Gorris^1,18  Emily M. X. Reed^1,19  Lee W. Cohnstaedt^2,20  Dave Osthus^2,21 

[1] Agricultural Research Service, United States Department of Agriculture, Sidney, MT, USA;Biomathematics Graduate Program, North Carolina State University, Raleigh, NC, USA;Center for Vector Biology, Rutgers University, New Brunswick, NJ, USA;Department of Biology, Stanford University, Stanford, CA, USA;Department of Entomology, Cornell University, Ithaca, NY, USA;Department of Environmental Medicine and Public Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA;Department of Global Health, Icahn School of Medicine at Mount Sinai, New York, NY, USA;Department of Mathematics and Statistics, Mount Holyoke College, South Hadley, MA, USA;Department of Medicine, University of California, San Francisco, CA, USA;Department of Pathobiology, University of Illinois at Urbana-Champaign, Urbana, IL, USA;Department of Pathology, Microbiology, and Immunology, School of Veterinary Medicine, University of California, Davis, CA, USA;Department of Pathology, Microbiology, and Immunology, School of Veterinary Medicine, University of California, Davis, CA, USA;Evolutionary Ecology and Genetics Group, Earth & Life Institute-UCLouvain, Louvain-La-Neuve, Belgium;Division of Infectious Diseases, Wadsworth Center, New York State Department of Health, Albany, NY, USA;Division of Infectious Diseases, Wadsworth Center, New York State Department of Health, Albany, NY, USA;Department of Atmospheric and Environmental Sciences, University at Albany, Albany, NY, USA;Division of Vector-Borne Diseases, Centers for Disease Control and Prevention, Fort Collins, CO, USA;Division of Vector-Borne Diseases, Centers for Disease Control and Prevention, San Juan, PR, USA;Emmett Interdisciplinary Program in Environment and Resources, Stanford University, Stanford, CA, USA;Global Systems Laboratory, National Atmospheric and Oceanic Administration, Boulder, CO, USA;Division of Vector-Borne Diseases, Centers for Disease Control and Prevention, Fort Collins, CO, USA;Information Systems and Modeling, Los Alamos National Laboratory, Los Alamos, NM, USA;Invasive Species Working Group, Global Change Center, Fralin Life Sciences Institute, Virginia Tech, Blacksburg, NC, USA;National Bio- and Agro-Defense Facility, Agricultural Research Service, United States Department of Agriculture, Manhattan, KS, USA;Statistical Sciences Group, Los Alamos National Laboratory, Los Alamos, NM, USA;

关键词: Calibration;    Discriminatory power;    Forecasting;    Logarithmic score;    Multi-model assessment;    West Nile virus;    West Nile neuroinvasive disease;    United States;   

DOI  :  10.1186/s13071-022-05630-y

 received in 2022-08-24, accepted in 2022-12-20,  发布年份 2022

来源: Springer

【 摘 要 】

BackgroundWest Nile virus (WNV) is the leading cause of mosquito-borne illness in the continental USA. WNV occurrence has high spatiotemporal variation, and current approaches to targeted control of the virus are limited, making forecasting a public health priority. However, little research has been done to compare strengths and weaknesses of WNV disease forecasting approaches on the national scale. We used forecasts submitted to the 2020 WNV Forecasting Challenge, an open challenge organized by the Centers for Disease Control and Prevention, to assess the status of WNV neuroinvasive disease (WNND) prediction and identify avenues for improvement.MethodsWe performed a multi-model comparative assessment of probabilistic forecasts submitted by 15 teams for annual WNND cases in US counties for 2020 and assessed forecast accuracy, calibration, and discriminatory power. In the evaluation, we included forecasts produced by comparison models of varying complexity as benchmarks of forecast performance. We also used regression analysis to identify modeling approaches and contextual factors that were associated with forecast skill.ResultsSimple models based on historical WNND cases generally scored better than more complex models and combined higher discriminatory power with better calibration of uncertainty. Forecast skill improved across updated forecast submissions submitted during the 2020 season. Among models using additional data, inclusion of climate or human demographic data was associated with higher skill, while inclusion of mosquito or land use data was associated with lower skill. We also identified population size, extreme minimum winter temperature, and interannual variation in WNND cases as county-level characteristics associated with variation in forecast skill.ConclusionsHistorical WNND cases were strong predictors of future cases with minimal increase in skill achieved by models that included other factors. Although opportunities might exist to specifically improve predictions for areas with large populations and low or high winter temperatures, areas with high case-count variability are intrinsically more difficult to predict. Also, the prediction of outbreaks, which are outliers relative to typical case numbers, remains difficult. Further improvements to prediction could be obtained with improved calibration of forecast uncertainty and access to real-time data streams (e.g. current weather and preliminary human cases).Graphical Abstract

【 授权许可】

CC BY   
© This is a U.S. Government work and not under copyright protection in the US; foreign copyright protection may apply 2023

【 图 表 】

Fig. 1 

41116_2022_35_Article_IEq68.gif

Fig. 5

40854_2022_443_Article_IEq20.gif

41116_2022_35_Article_IEq64.gif

41116_2022_35_Article_IEq60.gif

Fig. 1