Listen to any discussion of tinyML and chances are Pete Warden’s name will come up in the conversation. Widely known as one of the founders of tinyML, Warden modestly avoids that title, but he joined Google when his deep learning startup Jetpac was acquired by his company. Did. Warden became one of the creators of the TensorFlow framework in the wake of his work at Google. In 2019, he is co-author of a groundbreaking book introducing tinyML concepts for running tiny machine learning neural networks on low-power microcontrollers. EE Times Europe interviewed Warden to find out how he’s developed tinyML over the past four years and get his thoughts on the future.
EE Times Europe: When I joined Google in 2014, I discovered a small wake word model running on the DSP. At what stage did it become clear that there was a concrete, real-world application of ML on microcontrollers?
Pete Warden: What really impressed me was seeing people running deep learning models on DSPs and microcontrollers. Something like wake word was used within Google, but it also refers to companies like Keurig, the US coffee machine maker that uses computer vision to recognize coffee machine pods. Therefore, I do not want to claim its inventor. [tinyML] But I was inspired by the seeds of innovation that were already born. I was confident that there was an application that the developers had painstakingly built from scratch.
A lot of what I’ve done is inspired by these existing examples and the confidence that there are enough compelling applications that people will want to crawl through barbed wire and develop. It was a case of, “This approach is useful, so let’s figure out how to make it a little easier and a little more well-known.”
EETE: The tinyML book published in 2019 seems to have inspired many embedded developers to take action. Were there any specific efforts to accelerate the commercial adoption of tinyML?
Observer: What’s interesting is that it was a strange partnership between a high-end manufacturer and many chip makers. Chip makers saw the potential of tinyML.They found another reason to sell more computing [devices] And it’s an opportunity for microcontrollers to do some of the heavy lifting when it comes to processing. What was lacking was providing enough applications for commercial manufacturers.
The first tipping point was when manufacturers like Qualcomm, Arm, Cadence, CEVA and Synopsys said, “Yes, we support ML in our embedded chips and we put a lot of engineering work behind it.” It was time The tinyML Foundation’s work has also been great. This is a forum where hardware makers and everyone else can come together and work together to accelerate work. This helps share best practices, which is a key factor in accelerating tinyML.
EETE: Toolchains and frameworks for tinyML such as Edge Impulse emerged early on. How important have these tools been in educating the embedded MCU market, and how do you see them evolving?
Observer: I think they are great when it comes to making ML easier to use on embedded systems. That’s one of the big barriers to adoption.It can be very difficult to approach something different like machine learning [if you’re] Use tools and terminology put together by data scientists and machine learning experts. [The platforms] We provide embedded engineers with tools that understand and abstract many of the complexities so they can get started quickly. It also covers many vendor tools. ST, NXP, Cadence, Arm, and Qualcomm all provide his SDKs with examples that facilitate running machine learning on chips.
EETE: What are the major MCU advances over the past four years that have driven tinyML adoption?
Observer: Emergence of specialized hardware focused on performing neural network operations such as NPU [has been key]And almost every hardware vendor has a version. They can run 10x to 100x more powerful models. [than previously possible]without these features, you’ll end up with the same kind of power usage and bill of materials costs you paid for the previous generation.
EETE: Similarly, what are the major developments in ML neural network models that extend the capabilities and use cases of tinyML?
Observer: I would like to highlight some important advances. First, Edge Impulse’s commitment to her FOMO. [faster objects, more objects]This is a great example of a model specifically designed to run on an embedded device and solve one or more problems. I would also consider OpenAI’s Whisper. A complete speech recognition model that fits in just 40 MB. That’s a lot of memory for most embedded systems, but it’s not out of the realm of possibility.
And after the current generation of NPUs, we expect the next generation to provide the processing power needed to do full speech recognition locally on low-cost hardware. This is a key driver for this feature as OpenAI has released its models as open source.
EETE: In the preface to the book, you highlight the changing state of tinyML. Did it unfold as you expected? What happened as expected, what didn’t happen, or is it still not adopted?
Observer: There’s a lot of enthusiasm coming from developers, engineers, and manufacturers in companies big and small. It exceeded my expectations. When I last checked, he had over 40,000 people who had taken our Harvard edX tinyML course a few years ago. This is great. Support from hardware companies exceeded my expectations. I was also impressed with the amount of engineering resources and all that went into speeding up neural networks.
One aspect I was overly optimistic about was how quickly end product makers would be able to get this product and use it. There were many obstacles to producing and shipping the product. [So] We are still waiting for a real showcase application to showcase. [that will let us say] tinyML is more than just an engineering fad. It actually makes commercial sense and will change the way we interact with everyday objects.
EETE: Last March, you co-founded Useful Sensors, according to their website. [was formed] Enable manufacturers to add ML capabilities to existing products. It specifically mentions privacy and keeping data local. Are growing privacy concerns likely to limit tinyML adoption?
Observer: I think the difficulty manufacturers have in trying to do machine learning projects internally is one of the key motivations behind starting Useful Sensors, and privacy is another. One reason companies are hesitant is that adding cameras and microphones to everyday objects can be creepy. If you have a TV that uses a camera to recognize and control your gestures, you might be thinking, “Oh, that’s great.” However, if you have a TV in your bedroom, you need to be 100% sure that no one can spy on you. Our sensors are modules, and we worked with Kudelski to create a report on module security. Only gesture commands can be obtained from the sensor.Camera data is not streamed, it is checked by a third party [to confirm this]. It makes a lot of sense to put everything inside the sensor and output what you want, not what the sensor is measuring.
EETE: How do you see tinyML progressing in the next four years? For example, will it become an integral part of embedded development toolchains? Will tinyML adoption become the norm?
Observer: I think the functionality that tinyML provides will allow you to use off-the-shelf stuff instead of having to collect training data yourself, train a model, choose an architecture, figure out how to deploy it, and I hope so. My guess is that most of the work on machine learning has been done by someone else, and there will be more black his boxes that can offer plug-and-play stuff for embedded developers.
Also read:
