Awesome

The methods here are guaranteed to have coverage for any, possibly <i>adversarial sequence</i>. We take a <b>control systems outlook</b> on performing this task, introducing a method called <a style="text-decoration:none !important;" href="https://arxiv.org/abs/2307.16895" alt="arXiv">Conformal PID Control</a>.

Several methods are implemented herein, including online quantile regression (quantile tracking/P control), adaptive conformal prediction, and more!

The one exception is the COVID experiment. For that experiment, you must first run the jupyter notebook in <code>conformal-time-series/tests/datasets/covid-ts-proc/statewide /death-forecasting-perstate-lasso-qr.ipynb</code>. It requires the <code>deaths.csv</code> data file, which you can download from <a style="text-decoration:none !important;" href="https://drive.google.com/file/d/1p_l3bKJjypmJDmIZ0tqrwIZWDo8JDjvY/view?usp=sharing" alt="arXiv">this Drive link</a>.

The first three arguments, <code>scores</code>, <code>alpha</code>, and <code>lr</code>, are <i>required</i> arguments for all methods. The first argument, <code>scores</code>, expects a numpy array of conformal scores. The second argument, <code>alpha</code>, is the desired miscoverage. Finally, the third argument, <code>lr</code>, is the learning rate. (In our paper, this is $\eta$, and in the language of control, this is $K_p$.)

The rest of the arguments listed are required arguments specific to the given method. The argument <code>ahead</code> determines how many steps ahead the prediction is made --- for example, if <code>ahead=4</code>, that means we are making 4-step-ahead predictions (one step is defined by the resolution of the input array <code>scores</code>). The function of <code>*args</code> and <code>**kwargs</code> is to allow methods to take arguments given in a dictionary form.

All methods should <code>return</code> a dictionary of results that includes the method name and the sequence of $q_{t}$. In the quantile example case, the dictionary should look like the following, where <code>qs</code> is a numpy array of quantiles the same length as <code>scores</code>: <code>results = {"method": "Quantile", "q" : qs}</code> Methods that follow this formatting will be able to be processed automatically by our testing infrastructure.

As background, this is part of our little testing infrastructure for online conformal. The infrastructure spawns a parallel process for every dataset, making it efficient to test one method on all datasets with only one command (the command to run the tests is <code>bash run_tests.sh</code>, and to plot the results is <code>bash make_plots.sh</code>).

The infrastructure works like this.

First, download your dataset and put it in <code>tests/datasets</code>. Then, edit the <code>tests/datasets.py</code> file to add a name for your dataset and some processing code for it. The dataset should be a <code>pandas</code> dataframe with a valid <code>datetime</code> index (it has to be evenly spaced, and correctly formatted with no invalid values), and at least one column simply titled <code>y</code>. This column represents the target value.

Alternatively, including a column titled <code>forecasts</code> or <code>scorecasts</code> will cause the infrastructure to use these forecasts/scorecasts instead of the ones it would have produced on its own. This is useful if you have defined a good forecaster/scorecaster outside our framework, and you just want to use our code to run conformal on top of that. Extra columns can be used to add information for more complex forecasting/scorecasting strategies.

After executing these two steps, you should be able to run <code>python base_test.py configs/your_dataset.yaml</code> and the results will be computed! Alternatively, you can just execute the bash scripts.