Unless you're an accountant, filing your taxes at the end of the year can be a nightmare. You may be looking forward to filing your tax return, but you may not be keen on paying a tax professional or spending a few hours doing it yourself. But what if you could complete the entire process with a digital assistant on your smartphone? These productivity applications could save you a lot of time and money. That's the potential power of on-device artificial intelligence (AI).
The tax filing example is just one of the many ways that on-device AI can save consumers and businesses time and money. From optimizing smart appliances to automatically creating client contracts, on-device generative AI and the productivity apps it enables are key to ushering in an exciting new era for the smartphone and PC market.
Fewer AI workloads from the cloud
AI for personal and professional devices is not a new concept, but most applications run on the cloud. Although using the cloud offers advantages in terms of resource capacity and storage, cloud-centric AI models suffer from technical challenges such as high latency and network congestion. As a result, the user experience of many cloud-based AI apps does not meet customer expectations.
To address these technical challenges, smartphone manufacturers have begun to incorporate AI accelerators that support local AI inference into high-end devices. However, on-device AI applications are primarily limited to voice control, AI-enhanced imaging, and other “experience-centric” applications. Realizing the full value of on-device AI requires the development of a wide range of productivity AI applications tailored to specific use cases using compressed generative AI models.
The Value of On-Device AI
ChatGPT kicked off a massive hype cycle for generative AI among consumers and enterprises, leading to testing and deployment in various markets. Most of these AI models are deployed on the public cloud, causing users to experience network congestion, data privacy concerns, and increasing cloud fees as the user base grows. In contrast, local AI workloads enabled by on-device AI improve user experience by eliminating network latency, reducing various expenses, supporting future AI capabilities, and enhancing data security. These benefits are discussed in more detail below.
- Improved network latency: AI applications such as digital assistants and enterprise extended reality (XR) require low latency to deliver the most natural, personalized, and engaging interactions possible. Bringing AI inference to the device eliminates the risk of network latency and enables software developers to create a wider range of productivity applications for “mission-critical” applications that are not possible with cloud-centric AI architectures.
- Cost reduction: As AI adoption expands, it creates demands on networks and cloud hosting, further increasing costs for application developers and enterprises. Local AI processing reduces many of these costs and also reduces energy usage in data centers. Optimization tools such as compression and quantization play a key role in enabling generative AI on-device by developing accurate, low-power AI models with fewer than 15 billion parameters.
- Support for future AI capabilities: No one wants to invest in a device that will be outdated in a year or two. On-device AI accelerators can be optimized to support generative AI models and applications that are not yet on the market, thereby maximizing return on investment (ROI) for smartphone and PC owners.
- Enhanced data security: Public cloud service providers have security safeguards in place, but they are not foolproof, as recent cloud-based breaches at multiple organizations have proven. On-device AI keeps user and sensor data local, minimizing the risk of personal information and intellectual property (IP) being compromised. It is also worth noting that the low-latency capabilities of on-device AI models improve threat detection and other cybersecurity capabilities.
- Model personalization: AI models can be personalized in the public cloud, but this runs counter to end-user demands for data privacy and cost optimization. On-device processing allows AI models to be fine-tuned locally to end-user preferences, behavior, and applications. This is especially valuable as it allows for efficient personalization of AI models by leveraging various sensor/user data sources, including Wi-Fi, GPS, and sensor data. This has significant benefits, including increased AI productivity, improved accessibility, and more intuitive and automated interactions/experiences.
On-device AI improves consumer productivity
Consumers are upgrading their smartphones at a slower pace than in the past. The market may be reaching a stage of diminishing returns, where each new smartphone launch feels like it offers little to no added value over previous models. ABI Research believes that on-device AI combined with productivity-focused AI applications can stimulate consumer demand for smartphones and tablets.
If device manufacturers can demonstrate measurable ROI (cost and time savings) on these in-device AI apps, consumers will be motivated to upgrade their devices more frequently. Whether it's saving time by automatically scheduling family gatherings or saving on utility bills by optimizing energy usage, consumers will have another reason to buy a new smartphone model. Additionally, productivity AI apps can help artists and producers bring their creative ideas to life.
Symbolizing a market trend, Qualcomm and Samsung recently partnered to support mobile AI capabilities on the Galaxy S24 series. Not only will productivity AI applications enable devices to lower their refresh rates, but the new hardware will give device manufacturers such as Samsung the justification to increase the retail price of their products.
How businesses can leverage on-device AI
The enterprise market sees a similar picture, with PC and laptop shipment growth stagnating due to a lack of device innovation. Introducing AI natively to these devices will attract enterprises due to the value created by offline productivity, reduced latency, enhanced data privacy, improved user-device communication, and model personalization. On-device productivity AI saves enterprises time and money by automating administrative tasks (e.g., scheduling, drafting contracts, taking notes) and enabling users to remain productive even when devices are offline. Enterprises leveraging these new generative AI applications can save thousands of dollars per employee per year, as employees can leverage generative AI-powered apps such as Microsoft Copilot while on the move (e.g., traveling to client sites).
ABI Research has seen the earliest adoption of on-device AI within the enterprise occur in back-end operations, offices, and professional services, as early applications (such as Microsoft Copilot) offer a clear ROI, but as on-device AI matures with support for productivity AI applications and different form factors, we expect to see adoption in other industries, including manufacturing, healthcare, logistics and transportation, and communications.
While the discussion of on-device AI for the enterprise has centered on smartphones and PCs, the same benefits apply to automotive, XR, and Internet of Things (IoT)/wearables. Indeed, reduced latency will enhance in-car digital assistant capabilities, data privacy will protect sensitive data for healthcare patients and manufacturers, and reduce cloud computing costs. Additionally, mining and logistics companies will appreciate the increased reliability of on-device AI when using XR and IoT devices in remote locations where network interruptions are more likely. Similar to the consumer segment, on-device AI hardware with the right productivity AI apps is expected to drive less frequent device updates among businesses looking for the next “killer app.”
The future of on-device AI
A recent wave of trends is essential to support on-device AI. Heterogeneous chipsets such as Qualcomm's Snapdragon X Elite for PCs combine graphics processing units (GPUs), central processing units (CPUs), and neural processing units (NPUs) into a single system-on-chip (SoC). This allows AI workloads to run more efficiently, improving application performance. In addition to this, there is a significant move towards building highly optimized, device-ready, small generative AI models that can match the accuracy, performance, and knowledge of much larger models without the high power, memory, and compute requirements. This software innovation is complemented by increased collaboration between key players, combining low barriers to entry (through software development kits (SDKs) such as the Qualcomm AI Stack and no-code/low-code platforms) with accelerated development of productivity AI applications.
The fate of the on-device AI market rests on the shoulders of three key players.
- Independent software vendors (ISVs) leverage available AI models and tools to build AI applications that are optimized for the underlying hardware.
- By providing an SDK, chipset vendors ensure that their chipsets are capable of running AI on the device and make app development easier. It is also important that chipset vendors ensure the capabilities of their silicon to address the limitations of their devices.
- Original Equipment Manufacturers (OEMs) integrate various components into a single device and tailor the application to the consumer/business pain points and hardware.
Close collaboration between these companies will drive further innovation and ensure long-term revenue streams through on-device productivity AI. For example, the Ray-Ban Meta smart glasses collection uses Qualcomm chipsets to deliver on-glasses AI to reduce network latency and real-time translation capabilities. What was once considered an “entertainment” device will now be seen as a key “productivity” device that offers more than just enhanced photography and a typical voice assistant.
Finally, ABI Research predicts that the market will gradually adopt a “hybrid AI” approach. In a hybrid AI architecture, AI workloads reside on the edge, in the cloud, or on the device, depending on commercial and technical priorities. For example, in highly data-sensitive applications, model training may occur in the cloud, while inference and fine-tuning may occur on the device leveraging user data, ensuring maximum privacy. By adopting a hybrid AI approach, users can distribute power consumption, reduce memory bottlenecks, and maximize price/performance ratio.
Reece Hayden is a Principal Analyst at ABI Research, where he leads the firm's AI and Machine Learning Research services.
