NVIDIA dominates the AI ​​chip market, but competition is heating up

Rival chipmakers Intel and Advanced Micro Devices reported gross margins of 41% and 47%, respectively, in their most recent quarters.

Nvidia's position in the AI ​​chip market has been described by some experts as a “moat”: the combination of its AI graphics processing units (GPUs) such as its flagship H100 and CUDA software gives it a huge advantage over competitors, making switching to an alternative seem almost unthinkable.

Still, Nvidia CEO Jensen Huang, whose net worth has ballooned from $3 billion to about $90 billion in the past five years, said he is “worried and concerned” that the 31-year-old company is losing its edge. At a conference late last year, Mr. Huang acknowledged that many strong competitors have emerged.

“I don't think that someone is trying to destroy my business,” Huang said in November. “I know that they're probably trying to do that, so that's not it.”

Nvidia has committed to releasing new AI chip architectures every year, rather than every two years as in the past, along with new software that will allow chips to be more deeply integrated into AI software.

But Nvidia's GPUs aren't the only ones capable of carrying out the complex calculations that underpin generative AI, and given that less-powerful chips can do the same work, Huang is understandably skeptical.

The move from training AI models to what's known as inference — deploying the models — could also give companies an opportunity to replace Nvidia's GPUs, especially if they're cheaper to buy and operate. With Nvidia's flagship chips costing roughly $30,000 and up, there's plenty of incentive for customers to look for alternatives.

“NVIDIA wants 100% of the market share, but customers don't want NVIDIA to have 100% of the market share,” said Sid Sheth, co-founder of aspiring rival D-Matrix. “It's just too big an opportunity. It's just too unhealthy for one company to have everything.”

Founded in 2019, D-Matrix plans to release a server-oriented semiconductor card later this year that aims to reduce the cost and latency of running AI models. The company raised $110 million in September.

In addition to D-Matrix, a wide range of companies, from multinational corporations to startups, are vying for a share of the AI ​​chip market, which could reach $400 billion in annual sales over the next five years, according to market analysts and AMD. Nvidia has generated about $80 billion in revenue over the past four quarters, and Bank of America estimates it sold $34.5 billion of AI chips last year.

Many of the companies jumping on board with Nvidia's GPUs are convinced that different architectures and specific tradeoffs could produce better chips for specific tasks.Device makers are also developing technologies that could eventually do much of the AI ​​computing that's currently done on large GPU-based clusters in the cloud.

“Today, no one can deny that NVIDIA is the hardware of choice for training and running AI models,” Fernando Vidal, co-founder of 3Fourteen Research, told CNBC. “But there's been a gradual effort to level the playing field, from hyperscalers building their own chips to smaller startups designing their own silicon.”

AMD CEO Lisa Su wants investors to believe there's plenty of room for many companies to succeed in this space.

“The key is to have a lot of options,” Su told reporters in December as the company unveiled its latest AI chip. “I think we'll get to a point where there's not just one solution, but multiple solutions.”

AMD makes gaming GPUs and, like Nvidia, is adapting them for AI in the data center. Their flagship chip is the Instinct MI300X. Microsoft has already bought AMD processors and is making them accessible through its Azure cloud.

During the announcement, Su emphasized that the chip is not competing with Nvidia in training, but rather excels in inference.Last week, Microsoft said it was using AMD Instinct GPUs in its Copilot models.Morgan Stanley analysts took the news as a sign that AMD's AI chip sales this year could exceed the company's stated target of $4 billion.

Intel, which was overtaken by NVIDIA in terms of revenue last year, is also looking to establish a presence in the AI ​​space. The company recently announced the third version of its AI accelerator, Gaudi 3. This time, Intel compared it directly to the competition, describing it as a more cost-effective alternative, outperforming NVIDIA's H100 in terms of performing inference and training models faster.

Bank of America analysts recently predicted that Intel will capture less than 1% of the AI ​​chip market this year, and the company says it has a $2 billion backlog of orders for the chips.

The main obstacle to wider adoption may be software: AMD and Intel are both part of a larger industry consortium called the UXL Foundation, which also includes Google, that is working to develop a free alternative to Nvidia's CUDA for controlling hardware for AI applications.

A potential challenge for Nvidia is that the company is competing with its biggest customers: Cloud providers such as Google, Microsoft and Amazon are all developing processors for their own internal use. The big three tech companies and Oracle account for more than 40% of Nvidia's revenue.

Amazon introduced its own AI-focused chips in 2018, branded Inferentia, which is currently in version 2. In 2021, Amazon Web Services introduced Tranium, targeted at training. Customers can't buy the chips, but can rent the systems through AWS, which the company touts as more cost-effective than Nvidia's chips.

Google is perhaps the cloud provider most committed to its own silicon: The company has been using what it calls Tensor Processing Units (TPUs) to train and deploy AI models since 2015. In May, it unveiled Trillium, the sixth version of the chip that it says is being used to develop models such as Gemini and Imagen.

Google also uses Nvidia chips and provides them through its cloud.

Microsoft isn't quite there yet: The company announced last year that it was developing its own AI accelerators and processors, called Maia and Cobalt.

Meta is not a cloud provider, but it needs huge amounts of computing power to run software, websites and serve ads. Facebook's parent company has bought billions of dollars' worth of Nvidia processors, and said in April that some of its homegrown chips are already in data centers, offering “greater efficiency” compared to GPUs.

JPMorgan analysts estimated in May that the market for custom chip manufacturing for large cloud providers could be worth up to $30 billion and is expected to grow 20% annually.

Venture capitalists see an opportunity for startups to get into the space: They're investing $6 billion in AI semiconductor companies in 2023, up slightly from $5.7 billion the year before, according to Pitchbook data.

Semiconductors are a tough space for startups because they are expensive to design, develop and manufacture, but they also offer an opportunity for differentiation.

Silicon Valley AI chipmaker Cerebras Systems, which focuses on the fundamental operations and bottlenecks of AI rather than the more general-purpose properties of GPUs, was founded in 2015 and was valued at $4 billion in a recent round of funding, according to Bloomberg.

Cerebras' chip, the WSE-2, combines GPU capabilities as well as central processing and additional memory into a single device, making it suitable for training large models, CEO Andrew Feldman said.

“We have big chips and they have lots of little chips,” Feldman said. “They have a hard time moving data around, and we don't.”

Feldman said his company, whose clients include the Mayo Clinic, GlaxoSmithKline and the U.S. military, is competing with Nvidia for supercomputer system business.

“There's enough competition, and I think that's healthy for the ecosystem,” Feldman said.

D-Matrix's Sheth said the company plans to release cards later this year with chiplets that enable more computation in memory rather than on a chip like a GPU. D-Matrix's product can be inserted into AI servers alongside existing GPUs, reducing work for the Nvidia chips and helping to reduce the cost of generative AI.

Customers are “very open and eager to bring new solutions to the market,” Sheth said.

The biggest threat to Nvidia's data center business may be a shift in where processing happens.

Developers are becoming convinced that AI work will move from server farms to the laptops, PCs, and phones we own.

Large models like those developed by OpenAI require large clusters of powerful GPUs for inference, but companies like Apple and Microsoft are developing “mini models” that require less power and data and can run on battery-powered devices. They may not perform as well as the latest version of ChatGPT, but they can be used for other purposes, such as text summarization and visual search.

Apple and Qualcomm have been updating their chips to run AI more efficiently, adding a section dedicated to AI models called the neural processor, which offers privacy and speed benefits.

Qualcomm recently announced PC chips that will enable laptops to run Microsoft AI services, and the company is also investing in several chipmakers that make low-power processors that can run AI algorithms outside of smartphones and laptops.

Apple has touted that its latest laptops and tablets are optimized for AI thanks to the inclusion of a Neural Engine in the chips, and at its upcoming developers conference, Apple is set to show off a suite of new AI features that will likely run on the silicon in its iPhones.