Google pushes AI inference from cloud to browser — Virtualization Review

news

Google pushes AI inference from the cloud to the browser

Google is further pushing AI inference from cloud datacenters to users’ browsers with the release of LiteRT.js, a JavaScript binding for the cross-platform LiteRT on-device runtime.

“By bringing LiteRT, our trusted on-device inference library, to the web, web developers will be able to run ML and AI models completely locally and at maximum performance,” Google said in announcing LiteRT.js, LiteRT’s JavaScript binding for running AI directly within a web browser. “This means enhanced user privacy, zero server costs, and ultra-low latency for real-time experiences. For developers with existing .tflite models, LiteRT.js makes deployment to mobile and desktop web browsers smoother than ever and serves as a powerful evolution from TensorFlow.js for running .tflite models.”

Announced on July 9, open-source LiteRT.js allows JavaScript and TypeScript developers to run machine learning models entirely on client devices using local CPUs and graphics processing units (GPUs), rather than sending each inference request to a cloud-hosted service.

This changes the equation for Web AI adoption. Cloud inference is still necessary for workloads that require large models, centralized data, and large amounts of compute, but now models compact enough can be downloaded as part of a web application and run repeatedly on your own hardware.

Instead of paying to process all images, audio segments, and text input on cloud infrastructure, developers can move appropriate workloads to the browser and effectively use the client hardware as an inference layer.

Moving the inference layer
Traditional cloud AI architectures send application data to remote endpoints, where models process requests and return results. While this approach gives developers centralized control and access to powerful hardware, it also introduces network latency, server capacity requirements, API fees, and potential privacy concerns.

LiteRT.js provides another option for workloads that can fit within browser memory and run efficiently on consumer hardware. Once the model and required WebAssembly components are delivered to the browser, the client can perform subsequent inference locally without making a server request for each operation.

Potential use cases include:

• Perform object detection or image classification on webcam feeds without continuously uploading video.
• Process speech, audio, or images locally for real-time web applications.
• Generate embeddings and perform vector searches within the browser.
• Upscale or manipulate images without transferring the original files to a remote service.
• Add offline or intermittently connected AI functionality to your web applications.
• Keep sensitive user input on the device instead of sending it to a cloud endpoint.
• Reduce token or API consumption by processing simple classification, extraction, embedding, or compact generative AI tasks locally while reserving cloud models for more difficult requests.

The token cost scenario is most suitable for hybrid applications. Local models can perform preliminary work such as classifying input, extracting relevant information, generating embeddings, or determining whether a request requires a high-performance cloud model. Only requests that exceed the capabilities of the local model are escalated to paid cloud services.

LiteRT.js does not automatically reduce charges for existing cloud AI products or run cloud-hosted models locally. Developers should design their applications to route appropriate work to compatible local models instead of making corresponding remote API calls.

Browser hardware acceleration
LiteRT.js brings Google’s native LiteRT runtime to the browser through WebAssembly, rather than relying primarily on a JavaScript-based inference kernel. Google positions it as an evolution of TensorFlow.js for developers who deploy models in “.tflite” format.

The runtime can target three hardware paths:

• CPU: LiteRT.js uses XNNPACK, a Google-optimized library for multi-threaded CPU inference.
• GPU: Accelerate workloads on integrated or discrete graphics hardware using Google’s ML Drift technology and WebGPU browser API.
• NPU: Planned and experimental support uses the Web Neural Network API (WebNN) to access dedicated neural processing units.

WebGPU is already available in current versions of major browsers, but support and performance vary by browser, OS, and hardware. WebNN is experimental and requires specific browser settings and compatible system drivers. Google said its development roadmap includes deeper WebNN integration for native, power-efficient NPU inference.

The company claims that LiteRT.js performed up to 3x better than other web runtimes across CPU and GPU tests, including traditional computer vision and audio processing models. We also reported that GPU or NPU performance is 5 to 60 times faster than CPU execution for workloads such as object tracking, transcription, and image manipulation.

These benchmark results were obtained in a controlled browser environment on a 2024 Apple MacBook Pro with M4 silicon. Google cautioned that actual performance will vary depending on local GPU power, thermal throttling, and browser driver optimization.

From PyTorch to the Web
Developers can convert compatible PyTorch models to LiteRT models .tflite
Format it with LiteRT torch and then install the new file @litertjs/core
Use npm packages or load them from content delivery networks.

The runtime handles loading, compiling, and running the model, and the developer provides the surrounding pre- and post-processing logic. Applications can choose to run on WebGPU, WebNN, or WebAssembly-based CPUs, depending on the browser and hardware environment.

Google also offers AI Edge Quantizer, which reduces model size and improves execution performance with model-specific quantization settings. Smaller models are especially important for browser deployments because large downloads, memory limitations, and slow client hardware can make models with other features impractical.

The current implementation also has limitations. According to Google’s LiteRT.js documentation, some model operations may not be supported by all backends, input and output tensors are currently restricted to certain data types, partial delegation between CPU and GPU is not supported, and large models may exceed browser or WebAssembly memory limits.

In some cases, your application may fall back to CPU from GPU or WebNN execution. Other compatibility issues may require developers to modify and retranslate the original model.

Next up is generative AI.
Rather than presenting LiteRT.js as a browser replacement for cloud-scale, large-scale language models, the initial announcement focuses on established machine learning workloads such as object detection, depth estimation, audio processing, vector search, and image upscaling.

Still, Google has cited highly optimized on-device generated AI as a key development priority. Developers can also refer to LiteRT-LM.js, the browser-enabled component of the company’s open source LiteRT-LM framework for large-scale language model inference.

As such, the broader impact of the cloud will only become more significant over time. As smaller language models and multimodal models become more capable, web applications will be able to divide their work into three execution tiers. This means a local browser model for frequent, privacy-sensitive operations, edge or regional infrastructure for intermediate workloads, and a large-scale cloud model for requests that require the most functionality.

Such an architecture does not preclude cloud AI. However, it has the potential to reduce the number of requests that arrive, limit the amount of user data that needs to be sent, and allow developers to reserve expensive centralized computing for the tasks they actually need.

LiteRT.js is available through npm and includes demos covering vector search, object detection, depth estimation, and image upscaling. Google said future developments will expand model coverage, tools, WebNN integration, and support for generative AI.