Ilya Sutskever, co-founder of OpenAI, believes that the AI industry needs to move back to the research stage.
In an episode of “The Dwarkesh Podcast” published on Tuesday, Sutskever, who is widely regarded as a pioneer of modern artificial intelligence, challenged conventional wisdom that scaling can be a key roadmap to AI advancement.
Technology companies have poured hundreds of billions of dollars into acquiring GPUs and building data centers to essentially improve their AI tools, whether it’s LLM or image generation models.
Conventional wisdom says that the more compute you have and the more training data you have, the smarter your AI tools will become.
Sutskever said in an interview that this “recipe” has been producing impactful results for the past five years or so. This method is also efficient for companies, as it allows them to invest resources in a simple and “very low-risk way” compared to spending money on research that is not expected to produce any results.
But Sutskever, who now runs Safe Superintelligence, believes that approach is reaching its limits. Data is finite and organizations already have access to massive amounts of computing, he said.
“Do you really think, ‘Oh, this is so big, but 100 times that would change everything so much?'” Certainly not. But is it true that if you scale it 100 times everything will change? I don’t think that’s true,” Sutskever said. “So we’re back to the days of research using only large computers.”
Sutskever did not discount the need for computing, saying that research still requires computing and that it can be one of the “big differentiators” in an industry where all major organizations operate under the same paradigm.
But he said the research will be critical to finding ways to effectively or productively use all the compute gained.
One area where more research is needed, Sutskever said, is to make models generalize the way humans do, essentially learning using small amounts of information and examples.
“I think the most fundamental thing is that these models just somehow generalize dramatically worse than humans,” he says. “It’s very obvious. It seems very basic.”
