Open source AI efforts aim to make technology more widely available to researchers.Credit: Philippe Lejeanvre/Alamy
The generative artificial intelligence (AI) frenzy that began with the release of OpenAI’s ChatGPT shows no signs of slowing down. But while big tech companies such as OpenAI and Google are gaining broader public attention and looking for ways to monetize their AI tools, researchers and software engineers in smaller organizations are making quieter calls. A revolution is taking place.
While most big tech companies are becoming increasingly secretive, these small businesses remain open-minded in the field. They range from small businesses and non-profit organizations to individual hobbyists, and some of their activities are motivated by social goals such as democratizing access to technology and reducing its harm.
Computer scientist Stella Biderman, director of research at EleutherAI, an AI research institute in New York City, says there’s been an “explosion” of such open source activity. This is especially true for large-scale language models (LLMs), data-hungry artificial neural networks that power a variety of text-oriented software, including chatbots and automatic translation. Hugging Face, a New York City-based company aimed at expanding access to AI, lists his over 100 open source LLMs on its website.
LLaMA leak
Last year, Hugging Face led BigScience, a coalition of volunteer researchers and academics, to develop and release one of the largest LLMs to date. This model, called BLOOM, is a multilingual open source system designed for researchers. It continues to be an important tool, and the paper that described it has since attracted over 300 citations, mostly in computer science research.
Open source language AI challenges the model of big tech
In February, Facebook’s parent company Meta made a model called LLaMA freely available to select external developers, further boosting the open source movement. Within a week he had the LLaMA code leaked and posted online for anyone to download.
The availability of LLaMA has been a game-changer for AI researchers. Since this is much smaller than other of his LLMs, it can be used at scale to host pre-trained models or to adapt models for specialized applications such as acting as math assistants or customer service chatbots. Doesn’t require computing facilities. The largest version of LLaMA consists of 65 billion parameters. This is a variable that is set during the initial generic training of the neural network. That’s less than half of BLOOM’s 176 billion parameters and a fraction of his 540 billion parameters for Google’s latest LLM, PaLM2.
“At LLaMA, some of the most interesting innovations are on the efficiency side,” says Joelle Pineau, vice president of AI research at Meta and a computer scientist at McGill University in Montreal, Canada.
A developer has created a leaked version of AI LLaMA that can run on a Raspberry Pi computer.Credit: Dominic Harrison/Alamy
Open source developers are experimenting with ways to shrink LLaMA even further. Some of these techniques reduce the precision of parameters while keeping the number of parameters the same. This is an approach that, surprisingly, does not cause unacceptable performance penalties. Other ways to downsize neural networks include reducing the number of parameters. For example, train another smaller neural network based on the responses of a larger pre-trained network rather than using the data directly.
Within weeks of the LLaMA leak, developers were able to develop versions that fit laptops and even the Raspberry Pi, a minimalist credit-card-sized computer that’s a favorite of the ‘maker’ community. Hugging Face currently primarily uses LLaMA and has no plans to promote BLOOM-2.
Vukosi Mariveto, a computer scientist at the University of Pretoria, said shrinking AI tools could make them more widely accessible. For example, he could help organizations such as Masakhane, a community of African researchers led by Marivate, trying to make LLMs work in languages that don’t have many existing descriptive texts that can be used to train models. But efforts to expand access still have a way to go, and even the finest laptops may be out of reach for some researchers in low-income countries. “It was great,” says Marivete. “But I would also like to know the definition of ‘cheap’.”
look under the hood
Over the years, AI researchers have routinely open-sourced their code and posted their results to repositories such as arXiv. “People had a common understanding that the field would advance faster if they agreed to share things with each other,” said Colin Raffel, a computer scientist at the University of North Carolina at Chapel Hill. . For example, the current state-of-the-art innovation underlying his LLM, called the Transformer architecture, was created at his Google and released as open source.
What ChatGPT and Generative AI Mean for Science
By open sourcing neural networks, researchers can look “under the hood” and see why the system sometimes answers questions in unpredictable ways and can carry over biases and harmful information from pre-trained data. You can try to understand, says Ellie Public. He is a computer scientist at Brown University in Providence, Rhode Island, has collaborated on the BigScience project, and has also worked on Google AI. “One of her advantages is that many people, especially those in academia, can work on mitigation strategies,” she says. “If you had a thousand eyes, you would come up with a better way.”
Public’s team analyzed open source systems such as BLOOM to find ways to identify and correct biases inherited from training data. A classic example of this is that language models tend to associate “nurse” with the female gender and “doctor” with the female gender. male gender.
pre-training bottleneck
Even if the open source boom continues, the push to make language AI more powerful will continue to come from big companies. Only a handful of companies can create language models from scratch that truly drive the state of the art. LLM pre-training is resource-intensive, with researchers estimating that OpenAI’s GPT-4 and Google’s PaLM 2 took tens of millions of dollars worth of computational time. It also requires a large amount of “secret sauce,” the researchers say.
Will ChatGPT and AlphaCode be replaced by programmers?
“There are some common recipes, but there are often small details that aren’t documented or written down,” says Public. “Somebody doesn’t give you a code, you press a button and you get a model.”
“There are very few organizations or people who can do pre-training,” says Luis Castricato, an AI researcher at Stability AI, an open-source software company in New York. “It’s still a big bottleneck.”
Some researchers warn that making powerful language models widely accessible makes them more likely to fall into the wrong hands. Connor Leahy, co-founder of EleutherAI and chief executive of his London AI company Conjecture, believes AI will soon be intelligent enough to put humanity in existential danger. . “I don’t believe they should be open sourced,” he says.
