#Monolingual corpora mods#

To this end, we create datasets for the following tasks: Article Genre Classification, Headline Prediction, Wikipedia Section-Title Prediction, Cloze-style Multiple choice QA, Winograd NLI and COPA. Lastly, we compile the (IndicGLUE benchmark for Indian language NLU. The pre-trained language models are based on the compact ALBERT model. The word embeddings are based on FastText, hence suitable for handling morphological complexity of Indian languages. The monolingual corpora contains a total of 8.8 billion tokens across all 11 languages and Indian English, primarily sourced from news crawls. These resources include: (a) large-scale sentence-level monolingual corpora, (b) pre-trained word embeddings, (c) pre-trained language models, and (d) multiple NLU evaluation datasets (IndicGLUE benchmark). In this paper, we introduce NLP resources for 11 major Indian languages from two major language families. IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languagesįindings of the Association for Computational Linguistics: EMNLP 2020Īssociation for Computational Linguistics

#Monolingual corpora mods#

Cite (Informal): IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages (Kakwani et al., Findings 2020) Copy Citation: BibTeX Markdown MODS XML Endnote More options… PDF: Data IndicCorp, = '.', Association for Computational Linguistics. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4948–4961, Online.

IndicNLPSuite: Monolingual Corpora, Evaluation Benchmarks and Pre-trained Multilingual Language Models for Indian Languages. | Findings SIG: Publisher: Association for Computational Linguistics Note: Pages: 4948–4961 Language: URL: DOI: 10.18653/v1/2020.findings-emnlp.445 Bibkey: kakwani-etal-2020-indicnlpsuite Cite (ACL): Divyanshu Kakwani, Anoop Kunchukuttan, Satish Golla, Gokul N.C., Avik Bhattacharyya, Mitesh M. Anthology ID: 2020.findings-emnlp.445 Volume: Findings of the Association for Computational Linguistics: EMNLP 2020 Month: November Year: 2020 Address: Online Venues: EMNLP It can also help the community in evaluating advances in NLP over a more diverse pool of languages. We hope that the availability of the dataset will accelerate Indic NLP research which has the potential to impact more than a billion people. Our embeddings are competitive or better than existing pre-trained embeddings on multiple tasks. We also include publicly available datasets for some Indic languages for tasks like Named Entity Recognition, Cross-lingual Sentence Retrieval, Paraphrase detection, etc. Lastly, we compile the ( IndicGLUE benchmark for Indian language NLU. These resources include: (a) large-scale sentence-level monolingual corpora, (b) pre-trained word embeddings, (c) pre-trained language models, and (d) multiple NLU evaluation datasets ( IndicGLUE benchmark).

Abstract In this paper, we introduce NLP resources for 11 major Indian languages from two major language families.