Challenges of COVID-19 Case Forecasting in the US, 2020-2021.
During the COVID-19 pandemic, forecasting COVID-19 trends to support planning and response was a priority for scientists and decision makers alike. In the United States, COVID-19 forecasting was coordinated by a large group of universities, companies, and government entities led by the Centers for D...
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Article |
| Language: | English |
| Published: |
Public Library of Science (PLoS)
2024-05-01
|
| Series: | PLoS Computational Biology |
| Online Access: | https://doi.org/10.1371/journal.pcbi.1011200 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1850128186274217984 |
|---|---|
| author | Velma K Lopez Estee Y Cramer Robert Pagano John M Drake Eamon B O'Dea Madeline Adee Turgay Ayer Jagpreet Chhatwal Ozden O Dalgic Mary A Ladd Benjamin P Linas Peter P Mueller Jade Xiao Johannes Bracher Alvaro J Castro Rivadeneira Aaron Gerding Tilmann Gneiting Yuxin Huang Dasuni Jayawardena Abdul H Kanji Khoa Le Anja Mühlemann Jarad Niemi Evan L Ray Ariane Stark Yijin Wang Nutcha Wattanachit Martha W Zorn Sen Pei Jeffrey Shaman Teresa K Yamana Samuel R Tarasewicz Daniel J Wilson Sid Baccam Heidi Gurung Steve Stage Brad Suchoski Lei Gao Zhiling Gu Myungjin Kim Xinyi Li Guannan Wang Lily Wang Yueying Wang Shan Yu Lauren Gardner Sonia Jindal Maximilian Marshall Kristen Nixon Juan Dent Alison L Hill Joshua Kaminsky Elizabeth C Lee Joseph C Lemaitre Justin Lessler Claire P Smith Shaun Truelove Matt Kinsey Luke C Mullany Kaitlin Rainwater-Lovett Lauren Shin Katharine Tallaksen Shelby Wilson Dean Karlen Lauren Castro Geoffrey Fairchild Isaac Michaud Dave Osthus Jiang Bian Wei Cao Zhifeng Gao Juan Lavista Ferres Chaozhuo Li Tie-Yan Liu Xing Xie Shun Zhang Shun Zheng Matteo Chinazzi Jessica T Davis Kunpeng Mu Ana Pastore Y Piontti Alessandro Vespignani Xinyue Xiong Robert Walraven Jinghui Chen Quanquan Gu Lingxiao Wang Pan Xu Weitong Zhang Difan Zou Graham Casey Gibson Daniel Sheldon Ajitesh Srivastava Aniruddha Adiga Benjamin Hurt Gursharn Kaur Bryan Lewis Madhav Marathe Akhil Sai Peddireddy Przemyslaw Porebski Srinivasan Venkatramanan Lijing Wang Pragati V Prasad Jo W Walker Alexander E Webber Rachel B Slayton Matthew Biggerstaff Nicholas G Reich Michael A Johansson |
| author_facet | Velma K Lopez Estee Y Cramer Robert Pagano John M Drake Eamon B O'Dea Madeline Adee Turgay Ayer Jagpreet Chhatwal Ozden O Dalgic Mary A Ladd Benjamin P Linas Peter P Mueller Jade Xiao Johannes Bracher Alvaro J Castro Rivadeneira Aaron Gerding Tilmann Gneiting Yuxin Huang Dasuni Jayawardena Abdul H Kanji Khoa Le Anja Mühlemann Jarad Niemi Evan L Ray Ariane Stark Yijin Wang Nutcha Wattanachit Martha W Zorn Sen Pei Jeffrey Shaman Teresa K Yamana Samuel R Tarasewicz Daniel J Wilson Sid Baccam Heidi Gurung Steve Stage Brad Suchoski Lei Gao Zhiling Gu Myungjin Kim Xinyi Li Guannan Wang Lily Wang Yueying Wang Shan Yu Lauren Gardner Sonia Jindal Maximilian Marshall Kristen Nixon Juan Dent Alison L Hill Joshua Kaminsky Elizabeth C Lee Joseph C Lemaitre Justin Lessler Claire P Smith Shaun Truelove Matt Kinsey Luke C Mullany Kaitlin Rainwater-Lovett Lauren Shin Katharine Tallaksen Shelby Wilson Dean Karlen Lauren Castro Geoffrey Fairchild Isaac Michaud Dave Osthus Jiang Bian Wei Cao Zhifeng Gao Juan Lavista Ferres Chaozhuo Li Tie-Yan Liu Xing Xie Shun Zhang Shun Zheng Matteo Chinazzi Jessica T Davis Kunpeng Mu Ana Pastore Y Piontti Alessandro Vespignani Xinyue Xiong Robert Walraven Jinghui Chen Quanquan Gu Lingxiao Wang Pan Xu Weitong Zhang Difan Zou Graham Casey Gibson Daniel Sheldon Ajitesh Srivastava Aniruddha Adiga Benjamin Hurt Gursharn Kaur Bryan Lewis Madhav Marathe Akhil Sai Peddireddy Przemyslaw Porebski Srinivasan Venkatramanan Lijing Wang Pragati V Prasad Jo W Walker Alexander E Webber Rachel B Slayton Matthew Biggerstaff Nicholas G Reich Michael A Johansson |
| author_sort | Velma K Lopez |
| collection | DOAJ |
| description | During the COVID-19 pandemic, forecasting COVID-19 trends to support planning and response was a priority for scientists and decision makers alike. In the United States, COVID-19 forecasting was coordinated by a large group of universities, companies, and government entities led by the Centers for Disease Control and Prevention and the US COVID-19 Forecast Hub (https://covid19forecasthub.org). We evaluated approximately 9.7 million forecasts of weekly state-level COVID-19 cases for predictions 1-4 weeks into the future submitted by 24 teams from August 2020 to December 2021. We assessed coverage of central prediction intervals and weighted interval scores (WIS), adjusting for missing forecasts relative to a baseline forecast, and used a Gaussian generalized estimating equation (GEE) model to evaluate differences in skill across epidemic phases that were defined by the effective reproduction number. Overall, we found high variation in skill across individual models, with ensemble-based forecasts outperforming other approaches. Forecast skill relative to the baseline was generally higher for larger jurisdictions (e.g., states compared to counties). Over time, forecasts generally performed worst in periods of rapid changes in reported cases (either in increasing or decreasing epidemic phases) with 95% prediction interval coverage dropping below 50% during the growth phases of the winter 2020, Delta, and Omicron waves. Ideally, case forecasts could serve as a leading indicator of changes in transmission dynamics. However, while most COVID-19 case forecasts outperformed a naïve baseline model, even the most accurate case forecasts were unreliable in key phases. Further research could improve forecasts of leading indicators, like COVID-19 cases, by leveraging additional real-time data, addressing performance across phases, improving the characterization of forecast confidence, and ensuring that forecasts were coherent across spatial scales. In the meantime, it is critical for forecast users to appreciate current limitations and use a broad set of indicators to inform pandemic-related decision making. |
| format | Article |
| id | doaj-art-d7c7eeeba3c3436d9f9bb67e5878ebf7 |
| institution | OA Journals |
| issn | 1553-734X 1553-7358 |
| language | English |
| publishDate | 2024-05-01 |
| publisher | Public Library of Science (PLoS) |
| record_format | Article |
| series | PLoS Computational Biology |
| spelling | doaj-art-d7c7eeeba3c3436d9f9bb67e5878ebf72025-08-20T02:33:26ZengPublic Library of Science (PLoS)PLoS Computational Biology1553-734X1553-73582024-05-01205e101120010.1371/journal.pcbi.1011200Challenges of COVID-19 Case Forecasting in the US, 2020-2021.Velma K LopezEstee Y CramerRobert PaganoJohn M DrakeEamon B O'DeaMadeline AdeeTurgay AyerJagpreet ChhatwalOzden O DalgicMary A LaddBenjamin P LinasPeter P MuellerJade XiaoJohannes BracherAlvaro J Castro RivadeneiraAaron GerdingTilmann GneitingYuxin HuangDasuni JayawardenaAbdul H KanjiKhoa LeAnja MühlemannJarad NiemiEvan L RayAriane StarkYijin WangNutcha WattanachitMartha W ZornSen PeiJeffrey ShamanTeresa K YamanaSamuel R TarasewiczDaniel J WilsonSid BaccamHeidi GurungSteve StageBrad SuchoskiLei GaoZhiling GuMyungjin KimXinyi LiGuannan WangLily WangYueying WangShan YuLauren GardnerSonia JindalMaximilian MarshallKristen NixonJuan DentAlison L HillJoshua KaminskyElizabeth C LeeJoseph C LemaitreJustin LesslerClaire P SmithShaun TrueloveMatt KinseyLuke C MullanyKaitlin Rainwater-LovettLauren ShinKatharine TallaksenShelby WilsonDean KarlenLauren CastroGeoffrey FairchildIsaac MichaudDave OsthusJiang BianWei CaoZhifeng GaoJuan Lavista FerresChaozhuo LiTie-Yan LiuXing XieShun ZhangShun ZhengMatteo ChinazziJessica T DavisKunpeng MuAna Pastore Y PionttiAlessandro VespignaniXinyue XiongRobert WalravenJinghui ChenQuanquan GuLingxiao WangPan XuWeitong ZhangDifan ZouGraham Casey GibsonDaniel SheldonAjitesh SrivastavaAniruddha AdigaBenjamin HurtGursharn KaurBryan LewisMadhav MaratheAkhil Sai PeddireddyPrzemyslaw PorebskiSrinivasan VenkatramananLijing WangPragati V PrasadJo W WalkerAlexander E WebberRachel B SlaytonMatthew BiggerstaffNicholas G ReichMichael A JohanssonDuring the COVID-19 pandemic, forecasting COVID-19 trends to support planning and response was a priority for scientists and decision makers alike. In the United States, COVID-19 forecasting was coordinated by a large group of universities, companies, and government entities led by the Centers for Disease Control and Prevention and the US COVID-19 Forecast Hub (https://covid19forecasthub.org). We evaluated approximately 9.7 million forecasts of weekly state-level COVID-19 cases for predictions 1-4 weeks into the future submitted by 24 teams from August 2020 to December 2021. We assessed coverage of central prediction intervals and weighted interval scores (WIS), adjusting for missing forecasts relative to a baseline forecast, and used a Gaussian generalized estimating equation (GEE) model to evaluate differences in skill across epidemic phases that were defined by the effective reproduction number. Overall, we found high variation in skill across individual models, with ensemble-based forecasts outperforming other approaches. Forecast skill relative to the baseline was generally higher for larger jurisdictions (e.g., states compared to counties). Over time, forecasts generally performed worst in periods of rapid changes in reported cases (either in increasing or decreasing epidemic phases) with 95% prediction interval coverage dropping below 50% during the growth phases of the winter 2020, Delta, and Omicron waves. Ideally, case forecasts could serve as a leading indicator of changes in transmission dynamics. However, while most COVID-19 case forecasts outperformed a naïve baseline model, even the most accurate case forecasts were unreliable in key phases. Further research could improve forecasts of leading indicators, like COVID-19 cases, by leveraging additional real-time data, addressing performance across phases, improving the characterization of forecast confidence, and ensuring that forecasts were coherent across spatial scales. In the meantime, it is critical for forecast users to appreciate current limitations and use a broad set of indicators to inform pandemic-related decision making.https://doi.org/10.1371/journal.pcbi.1011200 |
| spellingShingle | Velma K Lopez Estee Y Cramer Robert Pagano John M Drake Eamon B O'Dea Madeline Adee Turgay Ayer Jagpreet Chhatwal Ozden O Dalgic Mary A Ladd Benjamin P Linas Peter P Mueller Jade Xiao Johannes Bracher Alvaro J Castro Rivadeneira Aaron Gerding Tilmann Gneiting Yuxin Huang Dasuni Jayawardena Abdul H Kanji Khoa Le Anja Mühlemann Jarad Niemi Evan L Ray Ariane Stark Yijin Wang Nutcha Wattanachit Martha W Zorn Sen Pei Jeffrey Shaman Teresa K Yamana Samuel R Tarasewicz Daniel J Wilson Sid Baccam Heidi Gurung Steve Stage Brad Suchoski Lei Gao Zhiling Gu Myungjin Kim Xinyi Li Guannan Wang Lily Wang Yueying Wang Shan Yu Lauren Gardner Sonia Jindal Maximilian Marshall Kristen Nixon Juan Dent Alison L Hill Joshua Kaminsky Elizabeth C Lee Joseph C Lemaitre Justin Lessler Claire P Smith Shaun Truelove Matt Kinsey Luke C Mullany Kaitlin Rainwater-Lovett Lauren Shin Katharine Tallaksen Shelby Wilson Dean Karlen Lauren Castro Geoffrey Fairchild Isaac Michaud Dave Osthus Jiang Bian Wei Cao Zhifeng Gao Juan Lavista Ferres Chaozhuo Li Tie-Yan Liu Xing Xie Shun Zhang Shun Zheng Matteo Chinazzi Jessica T Davis Kunpeng Mu Ana Pastore Y Piontti Alessandro Vespignani Xinyue Xiong Robert Walraven Jinghui Chen Quanquan Gu Lingxiao Wang Pan Xu Weitong Zhang Difan Zou Graham Casey Gibson Daniel Sheldon Ajitesh Srivastava Aniruddha Adiga Benjamin Hurt Gursharn Kaur Bryan Lewis Madhav Marathe Akhil Sai Peddireddy Przemyslaw Porebski Srinivasan Venkatramanan Lijing Wang Pragati V Prasad Jo W Walker Alexander E Webber Rachel B Slayton Matthew Biggerstaff Nicholas G Reich Michael A Johansson Challenges of COVID-19 Case Forecasting in the US, 2020-2021. PLoS Computational Biology |
| title | Challenges of COVID-19 Case Forecasting in the US, 2020-2021. |
| title_full | Challenges of COVID-19 Case Forecasting in the US, 2020-2021. |
| title_fullStr | Challenges of COVID-19 Case Forecasting in the US, 2020-2021. |
| title_full_unstemmed | Challenges of COVID-19 Case Forecasting in the US, 2020-2021. |
| title_short | Challenges of COVID-19 Case Forecasting in the US, 2020-2021. |
| title_sort | challenges of covid 19 case forecasting in the us 2020 2021 |
| url | https://doi.org/10.1371/journal.pcbi.1011200 |
| work_keys_str_mv | AT velmaklopez challengesofcovid19caseforecastingintheus20202021 AT esteeycramer challengesofcovid19caseforecastingintheus20202021 AT robertpagano challengesofcovid19caseforecastingintheus20202021 AT johnmdrake challengesofcovid19caseforecastingintheus20202021 AT eamonbodea challengesofcovid19caseforecastingintheus20202021 AT madelineadee challengesofcovid19caseforecastingintheus20202021 AT turgayayer challengesofcovid19caseforecastingintheus20202021 AT jagpreetchhatwal challengesofcovid19caseforecastingintheus20202021 AT ozdenodalgic challengesofcovid19caseforecastingintheus20202021 AT maryaladd challengesofcovid19caseforecastingintheus20202021 AT benjaminplinas challengesofcovid19caseforecastingintheus20202021 AT peterpmueller challengesofcovid19caseforecastingintheus20202021 AT jadexiao challengesofcovid19caseforecastingintheus20202021 AT johannesbracher challengesofcovid19caseforecastingintheus20202021 AT alvarojcastrorivadeneira challengesofcovid19caseforecastingintheus20202021 AT aarongerding challengesofcovid19caseforecastingintheus20202021 AT tilmanngneiting challengesofcovid19caseforecastingintheus20202021 AT yuxinhuang challengesofcovid19caseforecastingintheus20202021 AT dasunijayawardena challengesofcovid19caseforecastingintheus20202021 AT abdulhkanji challengesofcovid19caseforecastingintheus20202021 AT khoale challengesofcovid19caseforecastingintheus20202021 AT anjamuhlemann challengesofcovid19caseforecastingintheus20202021 AT jaradniemi challengesofcovid19caseforecastingintheus20202021 AT evanlray challengesofcovid19caseforecastingintheus20202021 AT arianestark challengesofcovid19caseforecastingintheus20202021 AT yijinwang challengesofcovid19caseforecastingintheus20202021 AT nutchawattanachit challengesofcovid19caseforecastingintheus20202021 AT marthawzorn challengesofcovid19caseforecastingintheus20202021 AT senpei challengesofcovid19caseforecastingintheus20202021 AT jeffreyshaman challengesofcovid19caseforecastingintheus20202021 AT teresakyamana challengesofcovid19caseforecastingintheus20202021 AT samuelrtarasewicz challengesofcovid19caseforecastingintheus20202021 AT danieljwilson challengesofcovid19caseforecastingintheus20202021 AT sidbaccam challengesofcovid19caseforecastingintheus20202021 AT heidigurung challengesofcovid19caseforecastingintheus20202021 AT stevestage challengesofcovid19caseforecastingintheus20202021 AT bradsuchoski challengesofcovid19caseforecastingintheus20202021 AT leigao challengesofcovid19caseforecastingintheus20202021 AT zhilinggu challengesofcovid19caseforecastingintheus20202021 AT myungjinkim challengesofcovid19caseforecastingintheus20202021 AT xinyili challengesofcovid19caseforecastingintheus20202021 AT guannanwang challengesofcovid19caseforecastingintheus20202021 AT lilywang challengesofcovid19caseforecastingintheus20202021 AT yueyingwang challengesofcovid19caseforecastingintheus20202021 AT shanyu challengesofcovid19caseforecastingintheus20202021 AT laurengardner challengesofcovid19caseforecastingintheus20202021 AT soniajindal challengesofcovid19caseforecastingintheus20202021 AT maximilianmarshall challengesofcovid19caseforecastingintheus20202021 AT kristennixon challengesofcovid19caseforecastingintheus20202021 AT juandent challengesofcovid19caseforecastingintheus20202021 AT alisonlhill challengesofcovid19caseforecastingintheus20202021 AT joshuakaminsky challengesofcovid19caseforecastingintheus20202021 AT elizabethclee challengesofcovid19caseforecastingintheus20202021 AT josephclemaitre challengesofcovid19caseforecastingintheus20202021 AT justinlessler challengesofcovid19caseforecastingintheus20202021 AT clairepsmith challengesofcovid19caseforecastingintheus20202021 AT shauntruelove challengesofcovid19caseforecastingintheus20202021 AT mattkinsey challengesofcovid19caseforecastingintheus20202021 AT lukecmullany challengesofcovid19caseforecastingintheus20202021 AT kaitlinrainwaterlovett challengesofcovid19caseforecastingintheus20202021 AT laurenshin challengesofcovid19caseforecastingintheus20202021 AT katharinetallaksen challengesofcovid19caseforecastingintheus20202021 AT shelbywilson challengesofcovid19caseforecastingintheus20202021 AT deankarlen challengesofcovid19caseforecastingintheus20202021 AT laurencastro challengesofcovid19caseforecastingintheus20202021 AT geoffreyfairchild challengesofcovid19caseforecastingintheus20202021 AT isaacmichaud challengesofcovid19caseforecastingintheus20202021 AT daveosthus challengesofcovid19caseforecastingintheus20202021 AT jiangbian challengesofcovid19caseforecastingintheus20202021 AT weicao challengesofcovid19caseforecastingintheus20202021 AT zhifenggao challengesofcovid19caseforecastingintheus20202021 AT juanlavistaferres challengesofcovid19caseforecastingintheus20202021 AT chaozhuoli challengesofcovid19caseforecastingintheus20202021 AT tieyanliu challengesofcovid19caseforecastingintheus20202021 AT xingxie challengesofcovid19caseforecastingintheus20202021 AT shunzhang challengesofcovid19caseforecastingintheus20202021 AT shunzheng challengesofcovid19caseforecastingintheus20202021 AT matteochinazzi challengesofcovid19caseforecastingintheus20202021 AT jessicatdavis challengesofcovid19caseforecastingintheus20202021 AT kunpengmu challengesofcovid19caseforecastingintheus20202021 AT anapastoreypiontti challengesofcovid19caseforecastingintheus20202021 AT alessandrovespignani challengesofcovid19caseforecastingintheus20202021 AT xinyuexiong challengesofcovid19caseforecastingintheus20202021 AT robertwalraven challengesofcovid19caseforecastingintheus20202021 AT jinghuichen challengesofcovid19caseforecastingintheus20202021 AT quanquangu challengesofcovid19caseforecastingintheus20202021 AT lingxiaowang challengesofcovid19caseforecastingintheus20202021 AT panxu challengesofcovid19caseforecastingintheus20202021 AT weitongzhang challengesofcovid19caseforecastingintheus20202021 AT difanzou challengesofcovid19caseforecastingintheus20202021 AT grahamcaseygibson challengesofcovid19caseforecastingintheus20202021 AT danielsheldon challengesofcovid19caseforecastingintheus20202021 AT ajiteshsrivastava challengesofcovid19caseforecastingintheus20202021 AT aniruddhaadiga challengesofcovid19caseforecastingintheus20202021 AT benjaminhurt challengesofcovid19caseforecastingintheus20202021 AT gursharnkaur challengesofcovid19caseforecastingintheus20202021 AT bryanlewis challengesofcovid19caseforecastingintheus20202021 AT madhavmarathe challengesofcovid19caseforecastingintheus20202021 AT akhilsaipeddireddy challengesofcovid19caseforecastingintheus20202021 AT przemyslawporebski challengesofcovid19caseforecastingintheus20202021 AT srinivasanvenkatramanan challengesofcovid19caseforecastingintheus20202021 AT lijingwang challengesofcovid19caseforecastingintheus20202021 AT pragativprasad challengesofcovid19caseforecastingintheus20202021 AT jowwalker challengesofcovid19caseforecastingintheus20202021 AT alexanderewebber challengesofcovid19caseforecastingintheus20202021 AT rachelbslayton challengesofcovid19caseforecastingintheus20202021 AT matthewbiggerstaff challengesofcovid19caseforecastingintheus20202021 AT nicholasgreich challengesofcovid19caseforecastingintheus20202021 AT michaelajohansson challengesofcovid19caseforecastingintheus20202021 |