NVIDIA and AI4Bharat have open-sourced Indic pretraining data comprising 2.8 trillion tokens, along with the associated data generation pipeline.
In collaboration with IIT Madras's AI4Bharat, NVIDIA has released Nemotron-pretraining-Indic-v1. This dataset contains approximately 2.8 trillion tokens and covers the 22 languages listed in the Indian Constitution. It includes 260.1 billion native text tokens and 22.46 billion romanized tokens (based on the Gemma-4 tokenizer).
Return to original text
A short introduction is retained here; the full text from the source website is not reposted. For specific wording and subsequent updates, please refer to the original article.
Open AIHOT original