Awesome
Anuvaad Parallel Corpus
This repository contains parallel language corpus links for popular Indian languages developed as part of the Anuvaad project.
Please reach out to nlp-nmt@tarento.com for any clarification/interpretation/usage of the linked datasets.
Shield: 
This work is licensed under a Creative Commons Attribution 4.0 International License.
Status
The current status of the parallel corpus built :
(Going forward, Anuvaad data contribution will be done via ULCA platform. Please refer https://bhashini.gov.in/ulca for further details)
| Language Pair | Parallel Corpus Count |
|---|---|
| English-Hindi | 5,627,366 |
| English-Bengali | 1,633,224 |
| English-Tamil | 1,522,854 |
| English-Malayalam | 1,505,760 |
| English-Telugu | 1,660,974 |
| English-Kannada | 1,352,017 |
| English-Marathi | 2,537,853 |
| English-Punjabi | 2,154,298 |
| English-Gujarati | 1,656,616 |
| English-Assamese | 47,174 |
| English-Urdu | 162,048 |
| English-Odia | 92,309 |
Goal
The goal is to build high quality parallel corpus for the Indian languages across various domains (General, Legal, Education, Healthcare, Automobile, News etc). This can be eventually used to train the ML models based on the use cases.
Read more about Anuvaad @ http://anuvaad.org/
The code for building the below mentioned datasets are available under https://github.com/project-anuvaad/anuvaad-corpus-tools
Links
English-Hindi
English-Bengali
* - General corpus from Wikipedia, Custom NER, Wikinews, PMIndia etc.
English-Tamil
English-Malayalam
| Domain | Source | Year | En-Ml pairs count | Corpus Download Link |
|---|---|---|---|---|
| Legal | Multiple Sources | 2010-2020 | 1,029,757 | Indian Judiciary |
| Legal | HC/SUVAS | NA | 8,222 | HC/SUVAS |
| Legal | Legal Terminologies | NA | 3,377 | Legal Terms |
| General | Nouns Dictionary | NA | 66,450 | Nouns Dict |
| News | PIB | 2017-2020 | 27,538 | PIB (2017-2020) |
| News | PIB Archives | 2014-2016 | 13,113 | PIB Archives (2014-2016) |
| General | Wikipedia | Until 2021-02 | 44,370 | Wikipedia (20210201) |
| News | OneIndia | Until 2021-03 | 48,891 | OneIndia (20210320) |
| Sports | MyKhel | Until 2021-03 | 26,526 | MyKhel (20210320) |
| Automobile | DriveSpark | Until 2021-03 | 49,663 | DriveSpark (20210303) |
| Tourism | Native Planet | Until 2021-03 | 27,024 | Native Planet (20210320) |
| Financial | GoodReturns | Until 2021-03 | 16,325 | GoodReturns(20210320) |
| Sports | Indian Express | Until 2021-03 | 13,305 | IE-Sports (20210320) |
| Technology | Indian Express | Until 2021-03 | 15,318 | IE-Technology (20210320) |
| News | Indian Express | Until 2021-03 | 36,337 | IE-News (20210320) |
| General | Indian Express | Until 2021-03 | 5,972 | IE-General (20210320) |
| Entertainment | Indian Express | Until 2021-03 | 12,430 | IE-Entertainment (20210320) |
| Financial | Indian Express | Until 2021-03 | 345 | IE-Financial (20210320) |
| Financial | MarketFeed | Until 2021-03 | 5,431 | MarketFeed (20210320) |
| News | AsianetNews | Until 2021-03 | 8,932 | AsianetNews (20210320) |
| News | Times of India | Until 2021-03 | 27,499 | TOI-News (20210320) |
| News | Zee News | Until 2021-04 | 18,935 | Zee-News (2011-2021) |
English-Telugu
| Domain | Source | Year | En-Te pairs count | Corpus Download Link |
|---|---|---|---|---|
| Legal | Multiple Sources | 2010-2020 | 1,010,463 | Indian Judiciary |
| Legal | HC/SUVAS | NA | 5,712 | HC/SUVAS |
| Legal | Legal Terminologies | NA | 3,595 | Legal Terms |
| General | Nouns Dictionary | NA | 63,802 | Nouns Dict |
| News | PIB | 2017-2020 | 65,842 | PIB (2017-2020) |
| News | PIB Archives | 2014-2016 | 15,734 | PIB Archives (2015-2016) |
| General | Wikipedia | Until 2021-02 | 82,257 | Wikipedia (20210201) |
| News | OneIndia | Until 2021-03 | 71,125 | OneIndia (20210320) |
| Sports | MyKhel | Until 2021-03 | 20,912 | MyKhel (20210320) |
| Automobile | DriveSpark | Until 2021-03 | 67,688 | DriveSpark (20210303) |
| Tourism | Native Planet | Until 2021-03 | 40,733 | Native Planet (20210320) |
| Financial | GoodReturns | Until 2021-03 | 17,159 | GoodReturns (20210320) |
| News | Sakshi | Until 2021-03 | 113,458 | Sakshi (20210320) |
| News | AsianetNews | Until 2021-03 | 10,054 | AsianetNews (20210320) |
| News | Times of India | Until 2021-03 | 49,233 | TOI-News (20210320) |
| News | Zee News | Until 2021-04 | 23,207 | Zee-News (2011-2021) |
English-Kannada
| Domain | Source | Year | En-Kn pairs count | Corpus Download Link |
|---|---|---|---|---|
| Legal | Multiple Sources | 2010-2020 | 859,455 | Indian Judiciary |
| Legal | Legal Terminologies | NA | 3,356 | Legal Terms |
| General | Nouns Dictionary | NA | 54,126 | Nouns Dict |
| News | PIB | 2017-2020 | 50,731 | PIB (2017-2020) |
| News | PIB Archives | 2015-2016 | 20,910 | PIB Archives (2015-2016) |
| News | OneIndia | Until 2021-03 | 49,622 | OneIndia (20210320) |
| Sports | MyKhel | Until 2021-03 | 30,416 | MyKhel (20210320) |
| Automobile | DriveSpark | Until 2021-03 | 57,151 | DriveSpark (20210303) |
| Tourism | Native Planet | Until 2021-03 | 31,654 | Native Planet (20210320) |
| Financial | GoodReturns | Until 2021-03 | 15,302 | GoodReturns (20210320) |
| News | AsianetNews | Until 2021-03 | 37,204 | AsianetNews (20210320) |
| News | Times of India | Until 2021-03 | 76,763 | TOI-News (20210320) |
| News | Zee News | Until 2021-04 | 65,327 | Zee-News (2011-2021) |
English-Marathi
English-Punjabi
| Domain | Source | Year | En-Pa pairs count | Corpus Download Link |
|---|---|---|---|---|
| Legal | Multiple Sources | 2010-2020 | 1,261,948 | Indian Judiciary |
| Legal | HC/SUVAS | NA | 10,978 | HC/SUVAS |
| Legal | Legal Terminologies | NA | 3,578 | Legal Terms |
| General | Nouns Dictionary | NA | 53,665 | Nouns Dict |
| News | PIB | 2017-2020 | 204,575 | PIB (2017-2020) |
| News | PIB Archives | 2015-2016 | 30,923 | PIB Archives (2015-2016) |
| News | Tribune | Until 2021-03 | 485,333 | Tribune (20210320) |
| Government Press Release | Punjab Govt | As of 2021-03 | 32,589 | Punjab Govt (20210320) |
| News | Jagran | Until 2021-03 | 34,162 | Jagran-News (20210320) |
| Technology | Jagran | Until 2021-03 | 6,524 | Jagran-Technology (20210320) |
| Education | Jagran | Until 2021-03 | 3,155 | Jagran-Education (20210320) |
| Entertainment | Jagran | Until 2021-03 | 7,565 | Jagran-Entertainment (20210320) |
| Financial | Jagran | Until 2021-03 | 7,741 | Jagran-Financial (20210320) |
| Sports | Jagran | Until 2021-03 | 9,034 | Jagran-Sports (20210320) |
| Lifestyle | Jagran | Until 2021-03 | 2,528 | Jagran-Lifestyle (20210320) |
English-Gujarati
| Domain | Source | Year | En-Gu pairs count | Corpus Download Link |
|---|---|---|---|---|
| Legal | Multiple Sources | 2010-2020 | 1,227,233 | Indian Judiciary |
| Legal | Legal Terminologies | NA | 3,575 | Legal Terms |
| General | Nouns Dictionary | NA | 55,037 | Nouns Dict |
| News | PIB | 2017-2020 | 104,591 | PIB (2017-2020) |
| News | PIB Archives | 2015-2016 | 29,443 | PIB Archives (2015-2016) |
| News | OneIndia | Until 2021-03 | 48,325 | OneIndia (20210320) |
| News | Times of India | Until 2021-03 | 85,777 | TOI-News (2008-2020) |
| Government Press Release | Gujarat Govt | As of 2021-03 | 5,260 | Gujarat Govt (20210320) |
| News | Zee News | Until 2021-04 | 97,375 | Zee-News (2011-2021) |
English-Assamese
| Domain | Source | Year | En-As pairs count | Corpus Download Link |
|---|---|---|---|---|
| News | PIB | 2017-2020 | 38,100 | PIB (2017-2020) |
| News | PIB Archives | 2015-2016 | 6,188 | PIB Archives (2015-2016) |
| News | The Sentinel | Until 2021-03 | 2,886 | The Sentinel (20210320) |
English-Urdu
| Domain | Source | Year | En-Ur pairs count | Corpus Download Link |
|---|---|---|---|---|
| News | PIB | 2017-2020 | 141,246 | PIB (2017-2020) |
| News | PIB Archives | 2015-2016 | 9,939 | PIB Archives (2015-2016) |
| News | The Wire | Until 2021-03 | 10,863 | The Wire (20210320) |
English-Odia
| Domain | Source | Year | En-Or pairs count | Corpus Download Link |
|---|---|---|---|---|
| News | PIB | 2017-2020 | 73,020 | PIB (2017-2020) |
| News | PIB Archives | 2015-2016 | 19,289 | PIB Archives (2015-2016) |