Awesome
Installation Guide
- Python version 3.10.10 is used for the experiments in a virtual environment. This is due to 3.11 being too recent for aws-fortuna.
- To create an environment:
python3.10 -m venv "environment_name"
- (Optional) For simplicity in kernel usage, you can create a labeled kernel:
python3.10 -m ipykernel install --user --name=thesis-3.10
- To activate the environment:
source environment_name/bin/activate
- To deactivate the environment:
deactivate
- To remove the environment:
rm -rf /path/to/environment
- To create an environment:
- Install the required libraries from the requirements.txt
- use:
pip install -r /path/to/requirements.txt
- use:
NOTE: For clarity, notebooks are inside folders for each experiment. When running experiments notebooks, one might need to move the notebook to the main directory. This is due to dependencies on the modules.
Experiments:
-
Main experiments
- there are 5 datasets (synthetic, google, Euro Stoxx, power, temperature)
- there are 3 seeds (seeds 100, 300, 500)
- for every dataset for every seed a notebook is created which generates results (15 in total)
- to reproduce the experiments you have to run every notebook in
main_results/notebooks
- to reproduce the experiments you have to run every notebook in
- the results are stored in results folders within the
main_results
folder and can be inspected usingresults.ipynb
-
Ablation experiments
- ACI grid search
- The file
ACI_grid_search.ipynb
is the file in which grid search is performed - Using
aci_optimized_comparison.ipynb
, the parameters found in the previous grid search can be evaluated on the test set (random seed 300) - These results are exported and can be inspected using results_aci_grid_search.ipynb
- The file
- Quantile tuning
- Experiments on synthetic data can be found in
quantile_tuning-SYNTHETIC.ipynb
- Experiments on google data can be found in
quantile_tuning.ipynb
- Experiments on synthetic data can be found in
- Multi-step forecasting
- Experiments and results can be found in
multi-step-forecasting.ipynb
- Experiments and results can be found in
- ACI grid search
-
Data
- Real-life data can be found within the data folder
- Synthetic data is found in the synthetic_data folder
If you use this repository, please cite the associated thesis:
@mastersthesis{Prinzhorn2023,
author = {Derck Walther Eward Prinzhorn},
title = {Benchmarking conformal prediction methods for time series regression},
school = {University of Amsterdam},
year = {2023},
type= {Bachelor's Thesis}
}