Awesome

FluSight 2024-2025

This repository is designed to collect forecast data for the 2024-2025 FluSight collaborative exercise run by the US CDC. This project collects forecasts for weekly new hospitalizations due to confirmed influenza. Anyone interested in using these data for additional research or publications is requested to contact flusight@cdc.gov for information regarding attribution of the source forecasts.'

Nowcasts and Forecasts of Confirmed Influenza Hospitalizations Admissions During the 2024-2025 Influenza Season

Influenza-related hospitalizations are a major contributor to the overall burden of influenza in the United States. Accurate predictions of influenza hospital admissions will help support public health planning and interventions during the 2024-2025 season as COVID-19, RSV, and other respiratory pathogens continue to circulate. CDC will coordinate a collaborative nowcasting and forecasting challenge for weekly laboratory confirmed influenza hospital admissions during the 2024-2025 influenza season, currently planned to begin November 20, 2024. Each week during the challenge (November through May 31st), participating teams will be asked to provide national and jurisdiction-specific (all 50 states, Washington DC, and Puerto Rico) probabilistic nowcasts and forecasts of the weekly number of confirmed influenza hospital admissions during the preceding week, the current week, and the following three weeks. This predicted timespan is planned to include the four weeks after the most recent hospital admissions data are officially released by CDC (details here). Prediction activities may begin or end later depending on reported influenza activity and availability of data. Teams can but are not required to submit predictions for all week horizons or for all locations. Predictions will be compared with the number of confirmed influenza admissions from the NHSN Weekly Hospital Respiratory Dataset. Previously collected influenza data from the 2020-2021, 2022-2023, and 2023-2024 influenza seasons and the number of hospitals reporting these data each day are included in the NHSN Weekly Hospital Respiratory Dataset as well.

Dates: The Challenge Period will begin November 20, 2024, and will run until May 31, 2025. Participants are asked to submit weekly nowcasts and forecasts by 11PM Eastern Time each Wednesday (herein referred to as the Forecast Due Date). The Forecast Due Date has been designated based on the release of hospitalization data on Wednesdays. Weekly submissions (including file names) will be specified in terms of the reference date, which is the Saturday following the Forecast Due Date. The reference date is the last day of the epidemiological week (EW) (Sunday to Saturday) containing the Forecast Due Date.

Prediction Targets

Participating teams are asked to provide national and jurisdiction-specific (all 50 states, Washington DC, and Puerto Rico) predictions for one primary target, quantile predictions for weekly laboratory confirmed influenza hospital admissions, and three other potential targets: category probability predictions for the direction and magnitude of changes in hospitalization rates per 100k population, probability predictions of peak week of laboratory confirmed influenza hospitalizations, and quantile predictions for peak incidence of laboratory confirmed influenza hospital admissions. All targets are optional. Additionally, teams may submit sample trajectories for the primary target of weekly laboratory confirmed influenza hospital admissions in addition to quantile format predictions.

For the primary target, teams will submit quantile nowcasts and forecasts of the weekly number of confirmed influenza hospital admissions for the epidemiological week (EW) ending on the reference date as well as the three following weeks. Hindcasts may also be submitted for the preceding week (see Note below). Please note that starting in the 2024-2025 season, FluSight will strongly recommend integer submissions for incidence forecasts. Teams can but are not required to submit forecasts for all weekly horizons or for all locations. The evaluation data for forecasts will be the weekly number of confirmed influenza admissions data from the NHSN Weekly Hospital Respiratory Dataset. We will use the specification of EWs defined by the CDC, which run Sunday through Saturday. The target end date for a prediction is the Saturday that ends an EW of interest, and can be calculated using the expression: target end date = reference date + horizon * (7 days).

There are standard software packages to convert from dates to epidemic weeks and vice versa (e.g. MMWRweek and lubridate for R and pymmwr and epiweeks for Python).

Note: Since hospitalization admission data for the preceding week will be provided on the Wednesday deadline, weekly hospital admission targets with a horizon of -1 will not be scored in summary evaluations nor included in visualizations. However, teams are encouraged to submit targets with a horizon of -1 to aid in detecting potential calibration issues.

Sample trajectories:

In addition to the quantile forecasts for incident hospital admissions, this season teams may submit samples for 0- to 3- week ahead forecasts. We use the term “model task” below to refer to a prediction for a specific horizon, location, and reference date. For teams submitting samples, the FluSight hub will require exactly 100 samples for each model task. We request that samples only be submitted when they are structured into temporally connected samples across horizons (i.e., samples should not be submitted that are solely drawn from the distribution of quantile forecasts). In particular, a common sample ID (specified in the ‘output_type_id’ field) will be used in multiple rows of the submission file with values of target date.

Details on formatting and the submission procedure are provided in the model-output subfolder README file and include details on specification of ‘output_type_id’.

Teams submitting sample trajectories corresponding to the primary forecast target will be requested to include information on how the trajectories were constructed in their metadata file (e.g., the primary output of forecasts which are then aggregated into quantile distributions). Teams should note that for this pilot submission target samples should retain an element of the temporal structure of 0 to 3-week ahead forecasts, rather than just submitting posthoc samples generated from their quantile forecasts.

Rate-trend categories:

The objective of the optional rate trend target is to characterize the trajectory of confirmed influenza hospital admissions as "large increase", "increase", "stable", "decrease" or "large decrease" over the 1- to 4 -week forecast period following the most recent official hospital admissions data. Predictions for these targets will be in the form of probabilities for each rate trend category, and will be submitted in the same file as a team's weekly hospital admissions forecasts using a target name of "wk flu hosp rate change".

Rate-trend categories are defined by binning state-level changes in weekly hospital admission incidence on a rate scale (counts per 100k people). A change is defined as the difference between the finalized reported weekly hospitalization rates in the EW ending on the target end date and the baseline EW ending one week prior to the reference date. At the time that nowcasts and forecasts are generated, this baseline week will be the most recent week for which the official data released on healthdata.gov include reported hospital admission values for at least some days (see Figures 1 and 2). Let $t$ denote the reference date and $y_s$ denote the finalized hospitalization rate in units of counts/100k population on the week ending on date $s$. The change in hospitalization rates at a weekly horizon $h$ is rate_change = $y_{t+h*7} - y_{t}$ .

The date ranges used in these calculations are illustrated in an example in Table 2. Corresponding count changes are based on state-level population sizes (i.e., count_change = rate_change*state_population / 100,000). See the locations.csv file in auxiliary-data/ for the population sizes that will be used to calculate rates.

Rate thresholds separating categories of change (e.g., separating "stable" trends from "increase" trends) will be the same across states, but are translatable into counts using the state's population size (see locations.csv, in the auxiliary-data subfolder of this repository). Any week pairs with a difference of fewer than 10 hospital admissions will be classified as having a "stable" trend. Specific rate-difference thresholds for changes have been developed for each prediction horizon, based on past distributions observed in FluSurv-NET and HHS-Protect. These are provided below in the model-outputs directory README file.

Note: The threshold levels for rate_change that define the categories have been modified slightly for the 2024-25 season. These updated thresholds are derived from the distribution of previously observed data and are expected to capture a slightly larger range of increases and decreases, that previously would have been considered stable.

Seasonal Forecast Targets:

Teams may also submit probability forecasts for peak week. Peak week will be defined as the epidemiologic week with the highest count of confirmed influenza hospital admissions during the 2024-2025 season. In the case multiple weeks observe the same highest count of confirmed influenza hospital admissions for a particular location, peak week forecasts for that location will not be scored. Based on analysis of previously observed influenza hospital admission data, we do not anticipate that this will happen frequently if at all. Separately, teams may submit quantile forecasts predicting peak incidence of influenza hospitalizations. Peak Incidence is defined as the highest count of confirmed influenza hospital admissions reach during any epidemiological week of the 2024-2025 season.

The objective of these seasonal targets is to provide actionable and intuitive forecasts for public health decision makers. These targets will characterize the season as a whole and predict the intensity and timing of the highest severity segment of the season to expand public health utility.

All forecast targets will be evaluated based on reported values for weekly influenza admissions in the NHSN Hospital Respiratory Dataset provided on healthdata.gov (commonly referred to as HHS Protect). As such, forecasting teams are encouraged to consider uncertainty in values as they are reported in real time. We emphasize that predictions of the rate trend targets will be evaluated based on changes in reported hospitalization rates based on finalized data at the end of the season.

If you have questions about this season’s FluSight Collaboration, please reach out to Rebecca Borchering, Sarabeth Mathis, and the CDC FluSight team (flusight@cdc.gov).

Acknowledgments

This repository follows the guidelines and standards outlined by the hubverse, which provides a set of data formats and open source tools for modeling hubs.