Awesome

Hivemind: decentralized deep learning in PyTorch

Codecov

Hivemind is a PyTorch library for decentralized deep learning across the Internet. Its intended usage is training one large model on hundreds of computers from different universities, companies, and volunteers.

Key Features

Distributed training without a master node: Distributed Hash Table allows connecting computers in a decentralized network.
Fault-tolerant backpropagation: forward and backward passes succeed even if some nodes are unresponsive or take too long to respond.
Decentralized parameter averaging: iteratively aggregate updates from multiple workers without the need to synchronize across the entire network (paper).
Train neural networks of arbitrary size: parts of their layers are distributed across the participants with the Decentralized Mixture-of-Experts (paper).

To learn more about the ideas behind this library, see the full list of our papers below.

Example Use Cases

This section lists projects that leverage hivemind for decentralized training. If you have successfully trained a model or created a downstream repository with the help of our library, feel free to submit a pull request that adds your project to this list.

Petals (webpage, code) — a decentralized platform for inference and fine-tuning of 100B+ language models.
Training Transformers Together (webpage, code) — a NeurIPS 2021 demonstration that trained a collaborative text-to-image Transformer model.
CALM (webpage, code) — a masked language model trained on a combination of Arabic datasets.
sahajBERT (blog post, code) — a collaboratively pretrained ALBERT-xlarge for the Bengali language.
PyTorch Lightning Integration (docs). Integration into PyTorch Lightning allows adapting your existing pipelines to training over slow network with unreliable peers.

Installation

Before installing, make sure that your environment has Python 3.8+ and PyTorch 1.9.0 or newer. They can be installed either natively or with Anaconda.

You can get the latest release with pip or build hivemind from source.

With pip

If your versions of Python and PyTorch match the requirements, you can install hivemind from pip:

pip install hivemind

Also, if you want to use blockwise 8-bit compression from bitsandbytes during data transfer, you can install it with pip install hivemind[bitsandbytes]. After that, you can use the BlockwiseQuantization class in hivemind.compression

From source

To install hivemind from source, simply run the following:

git clone https://github.com/learning-at-home/hivemind.git
cd hivemind
pip install .

If you would like to verify that your installation is working properly, you can install with pip install .[dev] instead. Then, you can run the tests with pytest tests/.

By default, hivemind uses the precompiled binary of the go-libp2p-daemon library. If you face compatibility issues or want to build the binary yourself, you can recompile it by running pip install . --global-option="--buildgo". Before running the compilation, please ensure that your machine has a recent version of Go toolchain (1.15 or 1.16 are supported).

System requirements

Linux is the default OS for which hivemind is developed and tested. We recommend Ubuntu 18.04+ (64-bit), but other 64-bit distros should work as well. Legacy 32-bit is not recommended.
macOS is partially supported. If you have issues, you can run hivemind using Docker instead. We recommend using our Docker image.
Windows 10+ (experimental) can run hivemind using WSL. You can configure WSL to use GPU by following sections 1–3 of this guide by NVIDIA. After that, you can simply follow the instructions above to install with pip or from source.

Documentation

The quickstart tutorial walks through installation and a training a simple neural network with several peers.
examples/albert contains the starter kit and instructions for training a Transformer masked language model collaboratively.
The Mixture-of-Experts tutorial covers the usage of Decentralized Mixture-of-Experts layers.
API reference and additional tutorials are available at learning-at-home.readthedocs.io

If you have any questions about installing and using hivemind, feel free to ask them in our Discord chat or file an issue.

Contributing

Hivemind is currently at the active development stage, and we welcome all contributions. Everything, from bug fixes and documentation improvements to entirely new features, is appreciated.

If you want to contribute to hivemind but don't know where to start, take a look at the unresolved issues. Open a new issue or join our chat room in case you want to discuss new functionality or report a possible bug. Bug fixes are always welcome, but new features should be preferably discussed with maintainers beforehand.

If you want to start contributing to the source code of hivemind, please see the contributing guidelines first. To learn more about other ways to contribute, read our guide.

Citation

If you found hivemind or its underlying algorithms useful for your research, please cite the following source:

@misc{hivemind,
  title = {{H}ivemind: {D}ecentralized {D}eep {L}earning in {P}y{T}orch},
  author = {Max Ryabinin and Alexander Borzunov and Michael Diskin and Anton Gusev and Denis Mazur and Vsevolod Plokhotnyuk and Alexey Bukhtiyarov and Pavel Samygin and Anton Sinitsin and Artem Chumachenko},
  month = apr,
  year = 2020,
  address = {Online},
  url = {https://github.com/learning-at-home/hivemind}
}

Also, you can cite the paper that inspired the creation of this library (prototype implementation of hivemind available at mryab/learning-at-home):

@inproceedings{ryabinin2020crowdsourced,
  title = {Towards Crowdsourced Training of Large Neural Networks using Decentralized Mixture-of-Experts},
  author = {Ryabinin, Max and Gusev, Anton},
  year = 2020,
  booktitle = {Advances in Neural Information Processing Systems},
  volume = 33,
  url = {https://proceedings.neurips.cc/paper/2020/file/25ddc0f8c9d3e22e03d3076f98d83cb2-Paper.pdf}
}

<details> <summary>Additional publications</summary>

"Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devices"

@inproceedings{ryabinin2021moshpit,
  title = {Moshpit SGD: Communication-Efficient Decentralized Training on Heterogeneous Unreliable Devices},
  author = {Ryabinin, Max and Gorbunov, Eduard and Plokhotnyuk, Vsevolod and Pekhimenko, Gennady},
  year = 2021,
  booktitle = {Advances in Neural Information Processing Systems},
  volume = 34,
  url = {https://proceedings.neurips.cc/paper/2021/file/97275a23ca44226c9964043c8462be96-Paper.pdf}
}

"Distributed Deep Learning in Open Collaborations"

@inproceedings{diskin2021distributed,
  title = {Distributed Deep Learning In Open Collaborations},
  author = {Michael Diskin and Alexey Bukhtiyarov and Max Ryabinin and Lucile Saulnier and Quentin Lhoest and Anton Sinitsin and Dmitry Popov and Dmitriy Pyrkin and Maxim Kashirin and Alexander Borzunov and Albert Villanova del Moral and Denis Mazur and Ilia Kobelev and Yacine Jernite and Thomas Wolf and Gennady Pekhimenko},
  year = 2021,
  booktitle = {Advances in Neural Information Processing Systems},
  url = {https://openreview.net/forum?id=FYHktcK-7v}
}

"Secure Distributed Training at Scale"

@inproceedings{gorbunov2022secure,
  title = {Secure Distributed Training at Scale},
  author = {Gorbunov, Eduard and Borzunov, Alexander and Diskin, Michael and Ryabinin, Max},
  year = 2022,
  month = {17--23 Jul},
  booktitle = {Proceedings of the 39th International Conference on Machine Learning},
  series = {Proceedings of Machine Learning Research},
  volume = 162,
  url = {https://proceedings.mlr.press/v162/gorbunov22a.html}
}

"Training Transformers Together"

@misc{borzunov2022training,
  title = {Training Transformers Together},
  author = {Alexander Borzunov and Max Ryabinin and Tim Dettmers and Quentin Lhoest and Lucile Saulnier and Michael Diskin and Yacine Jernite and Thomas Wolf},
  year = 2022,
  eprint = {2207.03481},
  archiveprefix = {arXiv},
  primaryclass = {cs.LG}
}

"Petals: Collaborative Inference and Fine-tuning of Large Models"

@inproceedings{borzunov-etal-2023-petals,
  title = {Petals: Collaborative Inference and Fine-tuning of Large Models},
  author = {Borzunov, Alexander  and Baranchuk, Dmitry  and Dettmers, Tim  and Ryabinin, Max  and Belkada, Younes  and Chumachenko, Artem  and Samygin, Pavel  and Raffel, Colin},
  year = 2023,
  month = jul,
  booktitle = {Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 3: System Demonstrations)},
  publisher = {Association for Computational Linguistics},
  address = {Toronto, Canada},
  pages = {558--568},
  doi = {10.18653/v1/2023.acl-demo.54},
  url = {https://aclanthology.org/2023.acl-demo.54},
  editor = {Bollegala, Danushka  and Huang, Ruihong  and Ritter, Alan},
}

"SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient"

@inproceedings{ryabinin2023swarm,
  title = {{SWARM} Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient},
  author = {Ryabinin, Max and Dettmers, Tim and Diskin, Michael and Borzunov, Alexander},
  year = 2023,
  month = {23--29 Jul},
  booktitle = {Proceedings of the 40th International Conference on Machine Learning},
  publisher = {PMLR},
  series = {Proceedings of Machine Learning Research},
  volume = 202,
  pages = {29416--29440},
  url = {https://proceedings.mlr.press/v202/ryabinin23a.html},
  editor = {Krause, Andreas and Brunskill, Emma and Cho, Kyunghyun and Engelhardt, Barbara and Sabato, Sivan and Scarlett, Jonathan},
  pdf = {https://proceedings.mlr.press/v202/ryabinin23a/ryabinin23a.pdf}
}

"Distributed Inference and Fine-tuning of Large Language Models Over The Internet"

@inproceedings{borzunov2023distributed,
  title = {Distributed Inference and Fine-tuning of Large Language Models Over The Internet},
  author = {Alexander Borzunov and Max Ryabinin and Artem Chumachenko and Dmitry Baranchuk and Tim Dettmers and Younes Belkada and Pavel Samygin and Colin Raffel},
  year = 2023,
  booktitle = {Thirty-seventh Conference on Neural Information Processing Systems},
  url = {https://openreview.net/forum?id=XmN7ZNbUAe}
}

</details>

We also maintain a list of related projects and acknowledgements.