Google wants developers to build on-device AI applications

Today's phones and PCs have new hardware to run AI directly on the device. And at this year's Google I/O, Google encouraged programmers to take advantage of it.

The idea is to run large language models on locally stored data without an internet connection. Your data remains private and never leaves your device, and this approach saves you money.

“Developers can reduce or eliminate the need to deal with server-side maintenance, capacity, constraints, or separate entry costs,” said Group Product Manager Sachin Kotwani in a session at Google I/O. said.

structure

The ability to develop on-device AI applications is a huge step forward from how AI is processed today.

Neural processors in new phones and PCs enable on-device AI.

AI is already present in your devices, even if you don’t realize it. Perform basic smartphone activities such as text message suggestions, image improvement, and power consumption analysis to save battery.

Neural processors in new phones and PCs enable on-device AI. However, LLMs with over 1 billion parameters, such as TinyLlama and Phi-2, run very slowly on a PC without an AI accelerator. LLM can only be run on CPUs with Jan.ai or GPT4All, but it is taxing on your computer.

Running LLM on a PC with a powerful GPU is very comfortable. However, setup is cumbersome. You need to download the model, load a neural network environment (such as Nvidia's CuDNN), install developer tools and compile.

A new wave of accelerators and GPUs that can perform matrix operations on devices will enable AI on mobile phones.

As a result, most of the AI ​​runs in the cloud on powerful GPUs. This is easily accomplished by simply loading the GPT-4 API into the chatbot interface, and the queries are offloaded to the GPUs of OpenAI's server infrastructure. However, these APIs are not free and you must pay to use OpenAI's infrastructure.

A new wave of accelerators and GPUs capable of matrix operations on devices will enable AI on mobile phones.

Google's new Pixel 8A smartphone has an Edge TPU (Tensor Processing Unit) for AI, and Intel and AMD have Neural Processing Units in their PCs. On-device AI can also be combined with cloud-based AI resources.

development tools

Development tools to run LLM on devices are becoming available from chip manufacturers such as AMD, Intel, and Nvidia.

Most recently, Google talked about development kits, APIs, and other tools that leverage its proprietary Gemini Nano LLM for mobile devices. This LLM is multimodal, allowing developers to build voice, image, video, or chatbot applications around it.

“Gemini Nano is Android's preferred path to production.”
– Thomas Ezan, Google

A Google representative said Gemini Nano is the most capable model for on-device AI and can also be integrated well into Android apps.

Thomas Ezan, senior developer relations engineer at Google, said at I/O:

For those who don't want to be locked into Google's proprietary AI development environment, Google supports an open source LLM with 2 billion to 3 billion parameters.

“Open large-scale language models have also grown in popularity over the past year if you want to run general inference on the device, but performance and memory issues make them unsuitable for production environments,” Ezan said. says.

These include Falcon 1B (1.3 billion parameters), Flan-T5 (2.7 billion parameters), StableLM 3B (2.8 billion parameters), and Llama 2B (2.5 billion parameters). Google will also support the open source Gemma LLM 7 billion parameter model.

Google's own tools

Developers can integrate Nano AI into their apps and development through the Edge AI SDK. The SDK provides high-level APIs, pipelines, model inference, and hardware hooks to run AI models efficiently.

Mobile devices have limited computing power, bandwidth, and memory. Developers can fine-tune their models by accessing a system service called AICore that's integrated into Android 14 running on targeted devices like the Pixel 8A and Samsung's S24.

Developers can use quantization to optimize models for mobile devices, reducing model size and processing requirements.

LoRA is considered a key building block for fine-tuning AI for devices and applications.

“The context window will probably also be smaller, and the model will be less generalized…which means fine-tuning will be important to get production quality,” says Terence, Developer Relations Engineer at Google. Zhang said.

AICore also includes a fine-tuning layer called Low-Rank Adaptation (LoRA) that app developers can use to customize models to perform specific tasks. LoRA is considered a key building block for fine-tuning AI for devices and applications.

“Apps can train their own specialized LoRA tweaking blocks to optimize the performance of Gemini Nano models,” said Miao Wang, a software engineer at Google.

Supports open source LLM

MediaPipe is a key API that enables developers to create on-device AI applications using multiple open source LLMs such as Falcon and Gemma.

Developers use the MediaPipe API to create AI web apps for Android and iOS devices.

The MediaPipe API provides pre-optimized models, so developers need to provide weights to run on-device applications. Supports vision, text, and audio applications. Some LLMs are better at specific tasks, and the API provides flexibility for developers to choose a model.

Developers use the MediaPipe API to create AI web apps for Android and iOS devices. Chrome 126 is in beta and integrates support for low-code APIs that connect web apps to Nano and open source LLMs.

“This all runs completely locally in your browser, and it's fast because it's accelerated on your computer's GPU via WebGPU. So you can build very attractive, completely local web applications. It's fast enough to do that,” Cormac Brick, principal software engineer for core machine learning, said at Google I/O.

TensorFlow Lite

Google also uses the TensorFlow Lite development environment, a lightweight version of the TensorFlow machine learning framework. TFLite also includes a kit to convert TensorFlow models into more compact versions that can run on your device.

“You can find a ready-made model or you can train a model in the framework of your choice,” says Brick. “In one step, you can convert your models to TensorFlow Lite and run them all in his bundle of app runtimes across Android, Web, and iOS.”

Chipmaker Qualcomm announced last week that developers will be able to port LLM to smartphones using its latest chips.

assignment

App developers are in a gold rush to leverage every last bit of processing they can to make their apps more efficient.

The new generation of devices will have more AI capabilities and the AI ​​brain on the device will be enhanced.

Another challenge is adapting the app to the right AI chip. The new generation of devices will have more AI capabilities and the AI ​​brain on the device will be enhanced.

As Dell introduces new PCs powered by Intel's NPUs, on-device AI will really take off as developers discover relevant apps, said Zach Noski, director of product management at Dell. states.

Developer participation in tools like Intel's OpenVino is critical to driving the industry forward. Vendors also need to work closely with developers who don't know where to start on preparing their applications.

For example, OpenVino provides an Intel NPU plugin for Gimp that supports stability diffusion image generation prompts.

“It's about continuing to enable it in the community. It's going to slow down a little bit, just as application CPU and GPU utilization has been in years past,” Noskey said.