Not everyone in AI is ready to declare the end of the era of scaling.
“I'm not convinced that AI is completely over,” Jeffrey Hinton, the “godfather of AI,” recently told Business Insider while discussing one of the hottest debates in AI this year.
Hinton is aware that one of his former students, OpenAI co-founder Ilya Satskeva, said last month that the pendulum of AI development is swinging back toward research and away from companies that make breakthroughs simply by scaling up or acquiring more compute and more chips.
“Do you really think, 'Oh, this is so big, but 100 times that would change everything so much?'” Certainly not. But is it true that increasing the scale by 100 times will change everything? I don't think that's true,” Sutskeva said on an episode of “The Dwarkesh Podcast.”
“So we're back to the era of research using only big computers,” added Sutskever, who now runs his own AI startup.
Hinton said more data will always be needed. (Another challenge facing scaling is that the amount of high-quality data is finite.) He predicted that large-scale chatbots will also start generating their own data, much like Google DeepMind's AlphaGo and AlphaZero do on a much smaller scale to master the board game Go.
“No one worries about running out of data, because it attacks itself and generates data that way,” Hinton said of the early programs. “And the equivalent of a language model is when a language model starts making inferences and says, 'Look, I believe these things and these things imply that, but I don't believe that, so I might as well change something somewhere.'” And by making inferences that check the consistency of his own beliefs, he can generate more data. ”
Scaling is at the very heart of Big Tech's capital spending, a bet based on the belief that AI models will continue to grow smarter and more sophisticated as they acquire more computing and training data.
A growing number of AI leaders are expressing concerns about making future bets based on confidence in scaling. Alexander Wang, currently head of Superintelligence at Meta, said scaling up will be the “biggest problem in the industry” in 2024.
Yann LeCun, who worked with Hinton on pioneering AI research, also disputes the scope of the scale doctrine.
“You can't simply think that more data and more computing means smarter AI,” LeCun said in April, when he was still Meta's chief AI scientist. Like Sutskever, LeCun has since launched his own startup.
Sutskever said scaling is attractive because it allows companies to bet on advances in AI in a “very low-risk way.”
In contrast, Google DeepMind CEO Demis Hassabis said scaling methods could ultimately unlock AI's biggest and most elusive prize: artificial general intelligence (AGI).
“We need to push the scale of our current system to the maximum extent possible, because at least that's going to be a key component of the final AGI system,” Hassabis said at Axios' AI+ Summit in December. “It could be the entire AGI system.”
