Awesome

Connect the World, Frame by Frame

Website | Documentation

English｜中文

🌟 Overview

VideoLingo is an all-in-one video translation, localization, and dubbing tool aimed at generating Netflix-quality subtitles. It eliminates stiff machine translations and multi-line subtitles while adding high-quality dubbing, enabling global knowledge sharing across language barriers.

Key features:

• 🎥 YouTube video download via yt-dlp

• 🎙️ Word-level subtitle recognition with WhisperX

• 📝 NLP and GPT-based subtitle segmentation

• 📚 GPT-generated terminology for coherent translation

• 🔄 3-step direct translation, reflection, and adaptation for professional-level quality

• ✅ Netflix-standard single-line subtitles only

• 🗣️ Dubbing alignment with GPT-SoVITS and other methods

• 🚀 One-click startup and output in Streamlit

• 📝 Detailed logging with progress resumption

• 🌐 Comprehensive multi-language support

Difference from similar projects: Single-line subtitles only, superior translation quality

🎥 Demo

Russian Translation

https://github.com/user-attachments/assets/25264b5b-6931-4d39-948c-5a1e4ce42fa7

GPT-SoVITS

https://github.com/user-attachments/assets/47d965b2-b4ab-4a0b-9d08-b49a7bf3508c

OAITTS

https://github.com/user-attachments/assets/85c64f8c-06cf-4af9-b153-ee9d2897b768

Language Support:

Current input language support and examples:

*Chinese requires separate configuration of the whisperX model, only applicable for local source code installation. See the installation documentation for the configuration process, and be sure to specify the transcription language as zh in the webpage sidebar

Translation language support depends on the capabilities of the large language model used, while dubbing language depends on the chosen TTS method.

🚀 Quick Start

Online Experience

Commercial version provides free 20min credits, visit videolingo.io

Colab (Currently not available)

Local Installation

VideoLingo supports all hardware platforms and operating systems, but performs best with GPU acceleration. For detailed installation instructions , refer to the documentation: English | 简体中文

Docker Installation

VideoLingo provides a Dockerfile. Refer to the installation documentation: English | 简体中文

🏭 Batch Mode

Usage instructions: English | 简体中文

⚠️ Current Limitations

1. WhisperX performance varies across different devices. Version 1.7 performs demucs voice separation first, but this may result in worse transcription after separation compared to before. This is because whisper itself was trained in environments with background music - before separation it won't transcribe BGM lyrics, but after separation it might transcribe them.

2. The dubbing feature quality may not be perfect as it's still in testing and development stage, with plans to integrate MascGCT. For best results currently, it's recommended to choose TTS with similar speech rates based on the original video's speed and content characteristics. See the demo for effects.

3. Multilingual video transcription recognition will only retain the main language. This is because whisperX uses a specialized model for a single language when forcibly aligning word-level subtitles, and will delete unrecognized languages.

4. Multi-character separate dubbing is under development. While whisperX has VAD potential, specific implementation work is needed, and this feature is not yet supported.

🚗 Roadmap

• SaaS service at videolingo.io
• VAD to distinguish speakers, multi-character dubbing
• Customizable translation styles
• Lip sync for dubbed videos

📄 License

This project is licensed under the Apache 2.0 License.The following open source projects provide important support for the development of VideoLingo:

whisperX | yt-dlp | json_repair | GPT-SoVITS | BELLE

📬 Contact Us

• Join our Discord: https://discord.gg/9F2G92CWPp
• Submit Issues or Pull Requests on GitHub
• Follow me on Twitter: @Huanshere
• Email me at: team@videolingo.io

⭐ Star History