Artificial Intelligence (AI) is now essential to creating business value, with many leading technology companies leveraging AI to satisfy the stock market and drive transformation initiatives. While much attention is focused on the chipsets and data centers that enable AI, what is the role of network connectivity?
AI faces many challenges to meet its massive data needs, increasing requirements at the network core and edge. With so much at stake, the underlying connectivity infrastructure that enables AI cannot be the weakest link. Network backbone providers will play a critical role in the future of AI by ensuring that the high-capacity bandwidth, low latency, and resiliency requirements of AI applications are not compromised. While hyperscalers are busy building for the next moonshot, enterprises must also start future-proofing their networks in anticipation of AI demands.
Bird's Eye View: The scenario from a career perspective
The future of AI is still being determined. But what's clear is that technology companies and their enterprise customers are investing heavily in AI. In 2024, hyperscalers (such as Alphabet, Amazon, and Meta) are projected to invest $200 billion (45% more than last year) in data centers, hardware, and other technologies needed to deploy generative AI models. And the data center industry is as busy as ever, with total capacity in North America and Europe expected to double over the next three years. These data centers need to train models on ever-more specific questions and ever-larger datasets, and they need the connectivity to link them and serve the demands of end users and applications.
However large these requirements may be, they are a significant change compared to today's networks. Let's consider two primary network use cases in AI: training models (“training”) and utilizing models (“inference”) to better understand how the network backbone enables both use cases. In each scenario, AI applications require low latency speeds, high bandwidth, and resiliency in the network underlay to function optimally, but the scale of the demands and the solutions needed to meet them may differ.
Scenario 1: Model learning drives explosive growth between core data centers and clouds
AI learning is computationally intensive, resulting in bursty workloads. It also relies on large datasets to enable meaningful training, making the collection and replication of this data critical. Meanwhile, power and data center colocation space are scarce, and companies struggle to build data centers quickly. Sustainable power is even more limited and rarely matches today's data center hubs. Thus, workloads must be distributed across clusters of data centers, necessitating higher capacity across the network underlay. Scaling to meet these massive capacity requirements can take many forms. But both approaches require operators to build resiliency into the underlying network, providing companies with reliable data transport over multiple protected paths between data centers.
For enterprises, traditional wide-area networks (WANs) are neither sufficient nor cost-effective. Instead, high-bandwidth Ethernet can accommodate most learning use cases. Major carriers have already been implementing 400GE for three years, with their backbone networks carrying hundreds of terabits per second at peak traffic, so Ethernet services can scale to enormous capacity to meet the bandwidth requirements of AI. The protected nature of Ethernet, combined with deterministic routing based on class of service (CoS) and fixed paths at high bandwidth, is also optimized for reliability and fault tolerance.
But this is not enough for hyperscalers and new AI technology companies. Nx400G, managed optical networks, or dark fiber, are the primary solutions to solve these challenges for hyperscaler use cases. Nx400G (multiple 400G Ethernet links connected in parallel) provides high-capacity links to reliably support large data flows across the network. Economies of scale and the need for control also drive the demand for dark fiber. Managed optical networks are a hybrid of these two, providing a means for network service providers to build in bespoke locations based on a single customer and extend their network to underserved locations built specifically for AI/ML workloads. This approach allows hyperscalers to focus on their core business operations by relying on specialized providers to ensure optimal performance and reliability.
Scenario 2: Inference takes the edge to new heights
Inference at the edge focuses on the end user, allowing businesses to leverage pre-trained AI models to process user requests. Inference is not as computationally intensive or power-hungry as AI learning; however, IT teams want to run inference closer to the edge of their networks to reduce latency and improve performance for end users. This is similar to how content delivery networks (CDNs) currently optimize access to videos, games, and websites globally by localizing content and always delivering the latest blockbuster content in sync.
This content is consumed by end users through the internet. Currently, CDNs rely primarily on Tier 1 internet providers to ensure access to the 70,000+ networks that make up the internet. Connectivity for AI applications needs to be optimized similar to CDNs, but AI is more demanding because it is less cacheable, business-critical, and sensitive to latency. It is clear that in the future, bandwidth requirements for AI could grow exponentially. Compared to today's single text string queries, users can send voice-based requests, images for editing, or sentences describing the code they need, and receive new video content or fully functional software. Additionally, edge nodes need to communicate with the core for a variety of purposes, including synchronization, retrieving the latest trained models, sharing their own learnings, or asking the core model to compute requests that have not yet been trained.
In most cases, internet connectivity will remain the obvious choice for AI application delivery to end users, highlighting the role of backbone networks in this scenario. Enterprise buyers can also take advantage of Ethernet backbone services on the same ports, mixing the public internet with performance-optimized private connections to their own data centers and cloud cores. Scalability is key here, so that networks can dynamically allocate bandwidth according to AI real-time peaks. It's an end-to-end game. Unless internet providers have peering in place with sufficient edge capacity and deep connections to local providers, critical traffic could get stuck. For cloud connectivity, network providers also need high-capacity network-to-network interfaces (NNIs), as upgrades to 10G connectivity are long overdue.
Enterprise AI: Reflecting Hyperscaler Requirements
In either scenario, enterprises must carefully evaluate how they collect, process, and protect the data they gather, as this is the “treasure” for getting the most out of AI. With the reliance on hybrid and multi-cloud environments, it is important to rethink the enterprise connectivity architecture. Currently, Internet and cloud security serves mostly hybrid working environments, and the next horizon is between the data center core and edge. Therefore, enterprises must prioritize high-quality Internet connectivity backed by high-capacity, deep, uncongested peering and scalable Ethernet or wavelength backbones that provide direct on-ramps to data center and cloud infrastructure. Finally, enterprises must remember that their WAN assets will face similar challenges as hyperscalers, but on a smaller scale. Still, if they want to reap the benefits of emerging technologies, they will need high-capacity network solutions that can handle distributed AI workloads.
Understand the role of the underlay in each scenario
The enablement of AI is a driver of many companies' current valuations and investment outlooks. As a result, innovation in the underlying network is essential to realizing the potential of AI at all levels of the technology industry. While the exact impact of AI on networks is unknown, its connectivity requirements are well known. These possible scenarios demonstrate that AI is not a “one size fits all” solution, meaning businesses must carefully evaluate their connectivity choices as they seek to leverage AI's unique advantages. For network backbone providers, AI creates a call to action to deliver scalable bandwidth, reliability, performance, and reach to address AI's network requirements across a range of industries and scenarios.
