Home

Awesome

<p align="center"> <img src="docs/imgs/hq.png"> </p>

Tests DOI paper DOI software

HyperQueue is a tool designed to simplify execution of large workflows (task graphs) on HPC clusters. It allows you to execute a large number of tasks in a simple way, without having to manually submit jobs into batch schedulers like Slurm or PBS. You specify what you want to compute and HyperQueue automatically asks for computational resources and dynamically load-balances tasks across all allocated nodes and resources. HyperQueue can also work without Slurm/PBS as a general distributed task execution engine.

Documentation

If you find a bug or a problem with HyperQueue, please create an issue. For more general discussion or feature requests, please use our discussion forum. If you want to chat with the HyperQueue developers, you can use our Zulip server.

If you use HyperQueue in your research, please consider citing it.

This image shows how HyperQueue can work on a distributed cluster that uses Slurm or PBS:

<img src="docs/imgs/architecture-bg.png" width="500" alt="Architecture of HyperQueue deployed on a Slurm/PBS cluster" />

Features

Installation

If you want to try the newest features, you can also download a nightly build.

Submitting a simple task

What's next?

Check out the documentation.

You can find FAQ (frequently asked questions) here.

HyperQueue team

We are a group of researchers working at IT4Innovations, the Czech National Supercomputing Center. We welcome any outside contributions.

Publications

If you want to cite HyperQueue, you can use the following BibTex entry:

@article{hyperqueue,
  title = {HyperQueue: Efficient and ergonomic task graphs on HPC clusters},
  journal = {SoftwareX},
  volume = {27},
  pages = {101814},
  year = {2024},
  issn = {2352-7110},
  doi = {https://doi.org/10.1016/j.softx.2024.101814},
  url = {https://www.sciencedirect.com/science/article/pii/S2352711024001857},
  author = {Jakub Beránek and Ada Böhm and Gianluca Palermo and Jan Martinovič and Branislav Jansík},
  keywords = {Distributed computing, Task scheduling, High performance computing, Job manager}}
}

Acknowledgement

License

MIT