June 4, 2024
blog
The year 2024 has seen a flurry of announcements from companies like OpenAI and Google about new Large Language Model (LLM)-based artificial intelligence (AI) products. It's no exaggeration to say that AI has not just entered the mainstream, but is now mainstream, in consumer and computing endpoints like smartphones and personal computers.
It is therefore surprising that embedded device manufacturers have not yet widely adopted AI and machine learning (ML) in endpoint devices for multiple reasons, not the least of which is the computing resource and power consumption constraints of traditional microcontrollers, application processors, and systems-on-chips (SoCs).
However, momentum for ML in IoT and other types of devices that are typically based on MCUs has been slowed by device designers' reluctance to incorporate ML algorithms. While this reluctance is understandable, given the strong appeal and novel capabilities of AI software in consumer devices, rapid adoption of ML techniques in IoT could bring significant potential value.
So what should MCU manufacturers do to help OEMs overcome the technical and operational barriers to embedded device designers embracing ML software?
The Uncertainty Principle: How ML Pushes Embedded Designers Out of Their Comfort Zones
Embedded device designers are hesitant to incorporate ML algorithms into MCU-based devices because their training and development methods are suited to an entirely different kind of software: deterministic and programmatic.
Traditional real-time control functions take inputs such as temperature readings from a sensor and take a specified action, such as powering off a device if the temperature exceeds a safe threshold. MCUs have evolved to become the preeminent hardware foundation for this type of real-time, deterministic control function. MCUs based on RISC CPUs such as Arm Cortex-M cores offer guaranteed latency and fast sequential execution of functions required for applications from motor control to sensor data processing to display control.
For application developers, writing traditional programmatic “if/then” software code has a logical thread and its behavior is constrained by known conditions that can be explicitly defined. Once debugged, the code results in completely predictable and reliable output in devices such as security cameras and power tools. These are classic applications for MCU-based control. Today, such devices add new value by adding ML capabilities to their existing control functions. And this ML software can run inside the MCU directly at the edge.
For example, security cameras can automate real-time monitoring for potential intruders. Powerful cloud-based ML software can accurately detect and analyze the behavior of people within their field of view. However, the cost and power consumption of such cameras can be significantly reduced if the video feed is pre-scanned by a local processor that can distinguish between human shapes and other objects in its field of view. This means that the camera would only trigger the system to upload frames to the cloud if the video feed contains potentially relevant information, instead of continuously uploading the entire feed.
ML brings similar transformative value to power drills: Drills equipped with ultra-wideband (UWB) radio can receive RF signals that change depending on the material being drilled, allowing the drill to verify that the correct type of bit is being used and detect hidden hazards such as water pipes buried in walls.
Creating traditional programmatic software to effectively perform this type of function may be impossible and would certainly require an unfeasible amount of development effort and time. However, training an appropriate neural network to recognize patterns in the reflection of UWB transmissions from, say, copper pipes embedded in concrete is a relatively straightforward task. Similarly, large open source training datasets and neural networks are readily available for ML systems to detect human movement in video streams.
However, this overall approach to training ML algorithms is alien to that of a traditionally trained developer of program software for MCUs or embedded processors. For ML algorithms, instead of structuring, writing and debugging code, embedded developers need to consider:
- How to Assemble a Good Training Dataset
- Evaluating and selecting an appropriate neural network
- Configuration and operation of deep learning frameworks such as TensorFlow Lite that are compatible with the target MCU and other hardware
While programmatic software can be improved through processes such as debugging and detailed code analysis, optimizing the operation of ML inference is an entirely different type of task. It involves analyzing factors such as the possibility of increasing the size of the training dataset or improving the quality of the examples in the training dataset to improve the accuracy and speed of inference.
The output of ML algorithms is also qualitatively different from traditional software: it is probabilistic, providing an answer to a question (“Is the copper pipe embedded in the wall?”) in the form of an inference drawn with some degree of confidence. It is not deterministic: the answer is probably right, but it could also be wrong.
Therefore, to get value from ML capabilities, designers of MCU-based devices must adopt new development methods and embrace new types of probabilistic output rather than deterministic. This is unfamiliar territory, and it is not surprising that the embedded industry has been a bit hesitant to implement ML.
So what can MCU manufacturers do to ease the transition to an AI-centric embedded environment and create a suitable operating environment for ML software?
Three features of ML-enabled MCUs
For Alif Semiconductor, this is an existential issue: since its founding in 2019, the company's mission has been to provide manufacturers of embedded and IoT devices with new MCUs and fusion processors that deliver the best AI/ML at the lowest power.
According to Alif's analysis, three key features of MCUs give manufacturers a better chance of success with new ML-based products:
- It provides a hardware environment that supports, rather than hinders, the operation of neural networking algorithms. RISC CPUs are the core of an MCU's control functions, but their sequential mode of operation makes them unsuitable for running the MAC cycles of neural networks. Neural Processing Units (NPUs), on the other hand, are optimized for running MAC and other neural networking operations. MCU architectures with one or more NPUs working in parallel with one or more CPUs provide an ideal foundation for the high-speed, low-power inference required for edge AI applications.
Testing against industry-standard benchmarks for ML functions such as voice activity detection, noise cancellation, echo cancellation and automatic speech recognition has shown that the combination of the Arm Cortex-M55 CPU and Ethos-U55 NPU delivers 50x faster inference speeds and 25x lower power consumption compared to a high-end Cortex-M7 CPU alone. - Allow device design teams to work on ML applications in a familiar development environment. For control functions performed by the CPU, the MCU market has been successful in consolidating options onto one architecture (Arm's Cortex-M). However, some MCU manufacturers are complementing their CPUs with their own NPUs, forcing users to leave the familiar Arm environment for the ML portion of their design.
Ultimately, the MCU market is likely to converge on Arm, not just for CPUs but also for NPUs. MCUs with Arm Ethos NPUs and Cortex-M CPUs will enable developers to share the same Arm tools and software ecosystem for both the control and ML parts of their applications. - Enables early-stage experimentation with common ML application types: The probabilistic nature of ML inference lends itself to a trial-and-error approach to proof-of-concept development, based on the use and refinement of open-source neural networking models and training datasets.
To this end, Alif Semiconductor offers AI/ML AppKit, development hardware that is pre-configured for vibration, audio and vision data collection, and comes with a range of demo systems for various AI use cases.
The kit includes a 4″ color LCD, an integrated camera module, PDM and I2S microphones, and an inertial sensor. Device drivers and operating system packs, as well as demo applications and samples, are available on the GitHub platform.
Today's Opportunities for Adding Value to Your Embedded Devices
The opportunity to bring the transformative capabilities of ML to embedded devices is now here. The technology is real and ready for mainstream adoption. Until now, adoption may have been slowed by the lack of availability of ML-native MCUs that work with familiar Arm development environments, but this is no longer a reason to hesitate to implement ML. With the introduction of products such as Alif Semiconductor's Ensemble family of MCUs and Fusion processors, with a choice of single-core and multi-core Arm Cortex-M55 CPU and Ethos-U55 NPU-based product options, embedded developers now have new hardware and development platforms well suited for ML.
With development tools such as the AI/ML AppKit in hand, it's time to dive into the world of machine learning at the edge.
