What is the state of Indian languages ​​in the age of AI and language models?

“Sanskrit is a good language for computers and those who study artificial intelligence can learn Sanskrit,” said S. Somanath, president of the Indian Space Research Organization, at an event in Ujjain on May 25. rice field. His remarks are the latest in a series of statements praising Sanskrit and its value in computing. But there is no proof or explanation.

But beyond Sanskrit, what are other Indian languages ​​doing in the field of artificial intelligence (AI) as language-based applications take over the world?

Answers vary. Despite the destiny of language being pushed by research and innovation by the public spirit, passive discrimination exists.

in ChatGPT

Behind seemingly intelligent chatbots and art-making computers, algorithms and data manipulation techniques transform verbal and visual data into mathematical objects (such as vectors) that are combined in specific ways to produce the desired output. generate. This is how ChatGPT answers your question.

When dealing with a language, machines first have to split sentences or words into smaller parts in a process called tokenization. These are the bits that the machine’s data processing model operates on. For example, “there’s a star” can be tokenized into “there”, “is”, “a”, and “star”.

There are several tokenization techniques. The Treebank tokenizer splits words and sentences based on rules used by linguists in their research. A subword tokenizer allows the model to learn common words such as “dusty” and “dustier”/”dustiest” and modifiers of those words independently.

OpenAI, creators of ChatGPT and the GPT series of large language models, use a type of subword tokenizer called Byte Pair Encoding (BPE). Here is an example of Gayathri Chakravorty Spivak’s OpenAI API using this in his statement.

english vs hindi

In 2022, Amazon will release a parallel database of 1 million utterances in 52 languages ​​called MASSIVE. “Parallel” means that the same utterance is presented in multiple languages. The utterance can be a simple query or phrase. For example, no. 38 in the Tamil section reads: க”. “How is it?” – means “Does the brightness of the screen seem low?”.

On May 3rd of this year, AI researcher Yenny Jun combined the OpenAI API with MASSIVE to analyze how BPE tokenizes 2,033 phrases in 52 languages.

Jun found that Hindi phrases are tokenized into 4.8 times more tokens on average than their English counterparts. Similarly, tokens in Urdu phrases he was 4.4 times longer, and tokens in Bengali he was 5.8 times longer. Running a model with more tokens increases operational costs and resource consumption.

Both GPT and ChatGPT can also accept a fixed number of input tokens at a time. This means that its ability to parse English text is better than Hindi, Bengali, Tamil, etc. This is important, he wrote Jun, because ChatGPT is ubiquitous. Adopted “in a variety of applications[from language learning platforms like Duolingo to social media apps like Snapchat]” and “understanding the nuances of tokenization to ensure fair language processing across diverse language communities” focus on the importance of

ChatGPT is switchable

AI expert Viraj Kulkarni said that in general, it is “quite difficult” to adapt a model trained to work in English to work in a language with a different grammar like Hindi. .

“If I take [an English-based] If modeled using a Hindi corpus and fine-tuned, the model might be able to reuse some of its understanding of English grammar, but still learn individual Hindi expressions and the relationships between those words. We need to do that,” said lead Dr. Kulkarni. said a data scientist at Pune-based DeepTek AI.

Therefore, sentences in different languages ​​are tokenized differently even if they have the same meaning. Thanks to Google Translate, Dr. Spivak’s statement in German is:

However, ChatGPT doesn’t cause any problems with other languages ​​in particular. According to Dr Kulkarni, GPT-4, which powers his premium version of ChatGPT, not only speaks English, but “most of the world’s major languages, [so it] I have no preference for English or any other language. ”

“I might be better at English than Hindi, maybe because I see more English sentences than Hindi, but switching between languages ​​is very fluid,” he added.

However, there is still a cost difference. In January 2023, machine learning expert Denys Linkov reviewed his first 50 utterances in English on MASSIVE with the prompt, “Rewrite the following sentences in a more friendly tone.” sent to ChatGPT. He repeated this work with his first 50 Malayalam utterances. He found that the latter used 15.69 times more tokens than he did.

Malayalam was also 15.69 times more expensive as OpenAI charges a fixed fee for using ChatGPT based on the number of tokens.

more and more data

This means that for models designed to work in individual languages, adaptation can be a problem as well in languages ​​that don’t have much indexed material available to train the model. This is true for many Indian languages.

“Recent breakthroughs, the biggest gains are [model] Performance depends only on two factors: the size or number of parameters of the network and the size of the training data,” said Dr. Kulkarni.

Parameters are ways to make words different from each other. For example, the type of word (verb, adjective, noun, etc.) he could be one parameter and the tense another. GPT-4 has trillions of parameters. The more parameters a model has, the better its capabilities.

The amount of training data for English is much larger than that for Indian languages. ChatGPT was trained on texts collected from the internet where approximately 55% of the content was in English.the rest are other languages ​​around the world Combined. Is that enough?

I do not understand. According to Dr. Kulkarni, the “minimum size” of the training dataset is unknown. “The minimum size also depends on the size of the network,” he explained. “Large networks are more powerful and can represent more complex functions, so they require more data to achieve maximum performance.”

Data for “fine-tuning”

“Text availability in each language will be a long tail. It will affect the model that handles the Macarand Tapaswi which is a machine. Learning scientist at the non-profit Wadhwani AI and assistant professor in the computer vision group at IIIT Hyderabad said:

“For uncommon languages, the model could be to first translate the word into English, find the answer, and then translate it back into the original language, but this can also be a source of error,” he added. .

“In addition to the next-word prediction task, models like GPT still need customization to ensure that they can follow natural language instructions, conduct conversations, align with human values, etc. is needed,” said Microsoft researcher Anoop Kunchukttan. . “The data for this customization is called ‘fine-tuning’ and should be of high quality, but is still mostly available only in English. Some of this is also present in Indian languages, [while] I need to create data for most kinds of complex tasks. ”

Amazon’s MASSIVE is a step in this direction. Others include Google’s “Daksina” dataset, which contains scripts for 12 South Asian languages. The open-source “No Language Left Behind” program creates “datasets and models” that close the “performance gap between low- and high-resource languages.”

Introducing AI4Bharat

In India, AI4Bharat is an IIT Madras initiative that is “building an open-source language AI for Indian languages, including datasets, models and applications,” according to its website.

Dr Kunchukttan is a founding member and co-leader of AI4Bharat’s Nile Crab Center. To train the language model, AI4Bharat has his corpus called IndicCorp containing 22 Indian languages, whose website says his crawler CommonCrawl can support “10-15 Indian languages”. he said.

Part of natural language processing is natural language Understanding (NLU) – The model manipulates the meaning of the sentence. For example, if asked “What is the temperature in Chennai?”, the NLU model might perform 3 tasks: It identifies that it is (1) a question that requires an answer, (2) is about the city, and (3) is related to the weather.

In a December 2022 preprint paper (updated 24 May 2023), AI4Bharat researchers reported a new benchmark for Indian languages ​​called ‘IndicXTREME’. They wrote that IndicXTREME has 9 NLU tasks for 20 languages ​​in the Constitution 8th Schedule, 9 of which lack sufficient resources to train language models. This will allow researchers to evaluate the performance of models that “learned” these languages.

moderately sized model

Preparing such tools is time consuming. For example, May 25th, AI4Bharat It was published “Bharat Parallel Corpus Collection”, “The Largest Publicly Available Parallel Corpus of Indian Languages”. It contains 23 million text pairs. Of these, 6.44 million were “manually translated”.

These tools should also consider caste, religion, regional dialects, stereotypes, slang, and contextual references.

Computational techniques help facilitate development. In his May 9 preprint paper, the AI4Bharat group addressed “the task of machine translation from very low-resource languages ​​to English using inter-language transfers from closely related high-resource languages”. .

“I think we are at a point where we have the data to train modestly sized models for the Indian language and can start experimenting in the directions mentioned above,” said Dr. Khunchukuttan. rice field. “The growing activity of open source modestly sized models in English indicates the potential for building promising models for Indian languages ​​and provides a starting point for further innovation.”