Meta today debuted an updated version of its free Llama AI model. Although it is one of the most popular open source models on the market, it faces increasing competition from both other open source competitors and companies offering paid closed access models. This new set of models, called Llama 3, represents Meta's attempt to match some of the features that rivals like OpenAI, Anthropic, and Google currently offer in their latest models, but these features have so far been closed. It was only available as a paid proprietary service. .
Meta said it wants its most capable Llama 3 models to be multimodal, meaning they can take in text, images, and even video and produce output in all those different formats. We are also aiming to make the model multilingual. It also has a larger “context window”, meaning it can feed large amounts of data for analysis and summarization. (Increasing the context window has also been shown to reduce the model's hallucination rate, or how often it outputs inaccurate information in response to a prompt.) According to Mehta, they improve reasoning ability and coding ability. has also improved.
As expected by many industry observers, the company initially released two smaller versions of Llama 3, which the press release described as “the best open source model in its class,” and will soon be available. It will be available on platforms such as AWS. Google Cloud, Databricks, Microsoft Azure, and Hugface. However, these models are not as capable as some of the most capable proprietary models on the market.
a An even larger version of Llama 3, with over 400 billion parameters, is still in training, and the company is unsure if and how it will release it after safety testing in the coming months. He says he will decide whether to do so.
However, Ragavan Srinivasan, Meta's vice president of product for Llama, says: luck This larger version “tends to be on par with some of the best-in-class proprietary models currently on the market,” he said, adding that additional features “will be incorporated.” These features match or exceed those currently offered by models such as Claude 3, Gemini, and GPT-4.
The announcement of Llama 3 was accompanied by the release of a new version of the company's assistant, Meta AI. Powered by Llama 3, the assistant will integrate real-time knowledge into the search box at the top of WhatsApp, Instagram, Facebook, and Messenger. Google and Bing have the answers.
“We believe Meta AI is the most intelligent AI assistant at our disposal,” Meta CEO Mark Zuckerberg said in the announcement. Meta's AI assistant received mixed reviews after it was first released in September 2023, with its cast of celebrity AI characters sometimes reacting inappropriately, especially when it was released in September 2023. And just yesterday, people reported that Meta's AI unexpectedly commented in a Facebook group for parents.
The state of competitive AI models
Meta is putting Llama 3 into a much different generative AI environment than its predecessor, Llama 2, which debuted last summer. Since then, open source AI has exploded while debate swirls around the security and safety of AI models that give users open access to the source code and model weights.
Paris-based Mistral was co-founded by former meta-researchers and debuted in June 2023. The company has released a variety of well-received open source models, and just this week it was reported that it was seeking a $5 billion valuation. Two months ago, Google released his Gemma, an open model built from the same research and technology as its own Gemini.
Meanwhile, the capabilities of proprietary models, such as those developed by OpenAI, Google, and Anthropic, continue to evolve, but their training becomes more compute-intensive and increasingly costly. . In fact, Meta is one of the big tech leaders in spending on model training and execution. In January, Mark Zuckerberg suggested that Meta was spending billions of dollars on his NVIDIA AI chips and said that by the end of 2024, the company's computing infrastructure would be completely gone. Includes 350,000 of his H100s. But Meta is making its model freely available as an open source product in hopes of finding a way to control and eventually monetize its position on the platforms that other companies are building on. We are also working on making this possible. This is an expensive strategy and there is no surefire path to quick profits.
Finally, the competition for AI talent continues to intensify, with top researchers and many former Big Tech engineers jumping in to launch their own startups.as luck It was recently reported that Meta has recently experienced an AI brain drain within the company, with several high-level departures, including a senior director of generative AI. This has implications for Zuckerberg's generative AI race. If Meta wants to stay ahead of the curve, it needs to make sure it has access to the best AI talent to build these models. Conversely, building the best models can help attract the best talent typically drawn to the most ambitious AI labs.
AI is a top priority in the meta
AI has become Meta's top priority, replacing the company's previous focus on the Metaverse, so it's clear that it intends to do whatever it takes to stand out in a crowded field. Last October, Zuckerberg said, “AI will be the biggest investment area in 2024, both in engineering and computing resources.” As part of his Llama announcement today, he emphasized that theme, saying, “We are making huge investments to build cutting-edge AI.”
Meta is also a long-time advocate of open source research. Having built an open source ecosystem around the Pytorch framework, he recently celebrated his 10th anniversary at FAIR (Fundamental AI Research). FAIR (Fundamental AI Research) was founded “to advance the cutting edge of AI through open research for the benefit of all” and is led by Meta's Chief Scientist Yann LeCun.
It was LeCun who pushed for Llama 2 to be released under a commercial license along with model weights. “I advocated this internally,” he said at the AI Native conference in September 2023. “I thought this was inevitable, because large language models are going to be the basic infrastructure that everyone uses, so they have to be open.”
Mahohar Paruri, vice president of AI organization Meta, said: luck With today's fierce open source AI competition, the company feels that “our mission to accelerate innovation and do it openly is supported and validated. “We build a model with high performance and get better with each iteration.” The more models that build on each other, including Llama, “the faster we can move forward in enabling more use cases for end users.”
Explore Llama 3 data in Meta
Meta did not provide details about what data was used to train Rama 3, but said it used “a variety of public data” (which the company previously acknowledged included public posts on Facebook and Instagram). It emphasized that it was trained on “excluded and deleted data”. It comes from certain sources that are known to contain large amounts of personal information about individuals. ”
Mehta said the training dataset is seven times larger and contains four times more code than what was used to train the previous version of Llama 2. More than 5% of his pre-training dataset for Llama 3 consists of “high-quality non-English data covering over 30 languages.”
Additionally, Meta revealed that an earlier version of Llama was used to train Llama 3. “We found previous generations of Llama to be great at identifying high-quality data, so we're using Llama 2 and also leveraging synthetic data to help improve areas like coding, inference, and long context. For example, we created a long document using synthetic data.
Meta competition for AI superiority
But as Rama 3 kicks into the wild, the bottom line is that the meta will need to keep running fast and spending big to realize Mark Zuckerberg's AI superiority. Catching up with OpenAI while promoting open source is no small feat, but if any organization has the will and money to make it happen, it's Meta.
The bigger question, of course, is why? What does this mean for Meta? Strategically, Zuckerberg believes that if AI is the next big platform change, as many investors and analysts have argued, appears to be dedicated to being platform independent. Zuckerberg believes leadership in the AI race is needed if the meta is to remain master of its own destiny.
Correction on April 18th: A previous version of this article incorrectly stated that Meta has already decided to release the largest and most powerful version of Llama 3 as an open source AI model. This story has been updated to clarify that Meta has not yet made this decision and to add additional context regarding model size and future safety testing.
