Home

Awesome

AutoTimes (Large Language Models for Time Series Forecasting)

Official implementation: AutoTimes: Autoregressive Time Series Forecasters via Large Language Models. [Slides], [Poster]

<p align="center"> <img src="./figures/icon.png" alt="" align=center /> </p>

Time Series Forecasting: AutoTimes convert LLMs to autoregressive time series forecasters. Unlike previous methods, the forecaster can accommodate arbitrary-length lookback and predictions.

Zero-Shot Forecasting: AutoTimes takes advantage of LLM's general-purposed token transition as the future extrapolation of time series, demonstrating good performance without downstream samples.

In-Context Forecasting: We propose in-context forecasting for the first time, where time series prompts can be incorporated into the input context to enhance forecasting.

Easy-to-Use: AutoTimes is compatiable with any decoder-only large language models, demonstrating generality and proper scaling behavior.

Updates

:triangular_flag_on_post: News (2024.10): An introduction of our works in available [Slides]. See you at NeurIPS 2024!

:triangular_flag_on_post: News (2024.10): AutoTimes has been accepted by NeurIPS 2024. A revised version (25 Pages) is now available, including prompt engineering of in-context forecasting, adaptation cost evaluations, textual embeddings of metadata, and low-rank adaptation techique.

:triangular_flag_on_post: News (2024.08): Recent work (code) has also raised questions about previous non-autoregressive LLM4TS methods. We conduct ablations here, highlighting AutoTimes can truly utilize LLMs. Instead of adopting LLMs in a BERT-style, the general-purpose token transition is transferable among time series and natural language.

<p align="center"> <img src="./figures/illustration.png" alt="" align=center /> </p>

:triangular_flag_on_post: News (2024.2) Scripts for the above tasks in our paper are all available.

Introduction

🌟 While prevalent LLM4TS methods adapt LLMs as encoder-only and non-autoregressive forecasters, we propose to keep consistent with the inherent autoregressive property and model architecture.

<p align="center"> <img src="./figures/motivation.png" alt="" align=center /> </p>

💪 We aim to fully revitalize LLMs as foundation models for time series forecasting, including multi-step forecasting, zero-shot capability, in-context forecasting, and multimodal utilization.

🏆 AutoTimes achieves state-of-the-art performance with 0.1% trainable parameters and over 5× training/inference speedup compared to advanced LLM-based forecasters.

Usage

  1. Install Pytorch and necessary dependencies.
pip install -r requirements.txt
  1. Put the datasets [Google Drive] [Tsinghua Cloud] under the folder ./dataset/.

  2. Download the large language models from Hugging Face. The default LLM is LLaMA-7B, you can change the llm_ckp_dir in run.py to use other LLMs.

    For example, if you download and put the LLaMA directory successfully, the directory structure is as follows:

    • data_provider
    • dataset
    • llama
      • config.json
      • pytorch_model-00001-of-00002.bin
      • pytorch_model-00002-of-00002.bin
      • ...
    • ...
    • run.py
  3. Using the position embedding from textual timestamps. Note that we have provided the embeddings of the given datasets in the download links, which are generated by LLaMA, suffixed by {dataset_name}.pt. If you want to generate the embeddings from your customized datasets, please refer to the following codes:

# preprocess timestamps to generate text embedding
python ./preprocess.py --gpu 0 --dataset ETTh1
  1. Train and evaluate the model. We provide all the above tasks under the folder ./scripts/.
# the default large language model is LLaMA-7B

# long-term forecasting
bash ./scripts/time_series_forecasting/long_term/AutoTimes_ETTh1.sh

# short-term forecasting
bash ./scripts/time_series_forecasting/short_term/AutoTimes_M4.sh

# zero-shot forecasting
# it's worth noting that sM4_tM3 utilizes models trained
# on short-term, you should run AutoTimes_M4 first
bash ./scripts/zero_shot_forecasting/sM4_tM3.sh
bash ./scripts/zero_shot_forecasting/sM3_tM4.sh

# in-context forecasting
bash ./scripts/in_context_forecasting/M3.sh

# try on other large language models
bash ./scripts/method_generality/opt.sh

Due to the simple tokenization and the frozen of LLM blocks, AutoTimes is highly compatiable with LLMs. For example, it requires only 15min for AutoTime to repurpuse LLaMA-7B on ETTh1 on one single RTX 3090-24G.

A Usage Example

See predict.ipynb for a simple training and inference workflow.

Overall Approach

<p align="center"> <img src="./figures/method.png" alt="" align=center /> </p>

Capability

<p align="center"> <img src="./figures/comparison.png" alt="" align=center /> </p>
Non-AutoregressiveAutoregressive
TrainingTrained with specific lookback-forecast lengthsTrained with the context length with each generated token being supervised
One-step ForecastingApplicable only on fixed lookback-forecast lengthsFlexible on scenarios less than the context length like large language models
Rolling ForecastingHas to drop the lookback series because of the fixed input lengthCan prolong the lookback horizon until the total length exceeds the context length

Time Series Forecasting

Towards the versatility of foundation models, we establish a novel one-for-all benchmark: a single forecaster is trained on one dataset and subsequently utilized for all prediction lengths.

<p align="center"> <img src="./figures/one-for-all_results.png" alt="" align=center /> </p>

Zero-Shot Forecasting

We evaluate the performance under the transfer learning scenario, where the forecaster is first trained on a source domain and then evaluated on the unseen target domain.

<p align="center"> <img src="./figures/zeroshot_results.png" alt="" align=center /> </p>

In-Context Forecasting

We concatenate time series prompts with lookback series and feed them as the context of the forecaster, termed in-context forecasting.

<p align="center"> <img src="./figures/formulation.png" alt="" align=center /> </p>

Benefiting from time series prompts from the target domain, AUTOTIMES achieves consistent promotions (averaged 13.3% SMAPE reduction) compared with zero-shot forecasting.

<p align="center"> <img src="./figures/in-context.png" alt="" align=center /> </p>

Model Generality

We evaluate the generality and efficiency on other LLMs, demonstrating improved performance with the increase of parameters that validates the scaling law.

<p align="center"> <img src="./figures/param.png" alt="" height = "300" align=center /> </p>

Method Efficiency

Not only does AutoTime achieve more acurate predcitions but its training and reasoning time is also greatly reduced, bringing over 5× speedup on average.

<p align="center"> <img src="./figures/adaption_efficiency.png" alt="" align=center /> </p>

Showcases

We investigate different prompt retrieval strategies. Insightful results are provided to reveal the influence of using time series prompts for interactive prediction.

<p align="center"> <img src="./figures/showcases.png" alt="" align=center /> </p> <p align="center"> <img src="./figures/subway_icf.png" alt="" align=center /> </p>

Citation

If you find this repo helpful, please cite our paper.

@article{liu2024autotimes,
  title={AutoTimes: Autoregressive Time Series Forecasters via Large Language Models},
  author={Liu, Yong and Qin, Guo and Huang, Xiangdong and Wang, Jianmin and Long, Mingsheng},
  journal={arXiv preprint arXiv:2402.02370},
  year={2024}
}

Acknowledgement

We appreciate the following GitHub repos a lot for their valuable code and efforts.

Contact

If you have any questions or want to use the code, feel free to contact: