Awesome

Training DALL-E with volunteers from all over the Internet

This repository is a part of the NeurIPS 2021 demonstration <a href="https://training-transformers-together.github.io/">"Training Transformers Together"</a>. In this demo, we train a model similar to <a target="_blank" href="https://openai.com/blog/dall-e/">OpenAI DALL-E</a> — a Transformer "language model" that generates images from text descriptions. Training happens collaboratively — volunteers from all over the Internet contribute to the training using hardware available to them. We use <a target="_blank" href="https://laion.ai/laion-400-open-dataset/">LAION-400M</a>, the world's largest openly available image-text-pair dataset with 400 million samples. Our model is based on the <a target="_blank" href="https://github.com/lucidrains/DALLE-pytorch">dalle‑pytorch</a> implementation by <a target="_blank" href="https://github.com/lucidrains">Phil Wang</a> with a few tweaks to make it communication-efficient. See details about how to join and how it works on <a target="_blank" href="https://training-transformers-together.github.io/">our website</a>.