Machine learning is inherently an experimental science. To drive true AI innovation, we must accept that commonly held wisdom and what has worked in the past may not be the best way to solve a new problem. Rethinking how we approach training data and how we evaluate performance metrics is key.
This isn't always what teams developing new products want to hear. But sometimes breakthroughs are worth the extra days on your timeline. It's a reminder of why many of us became data scientists, engineers, and innovators in the first place: We are curious, and we will do whatever it takes to solve even the most seemingly impossible problems.
I have applied this concept first-hand with the team at Ultraleap, developing a variety of machine learning models to meet the demanding hand tracking needs of enterprises and consumers, and have seen success driving the future of virtual interactions.
How machine learning (ML) experimentation can turn challenges into opportunities
Many companies and industries have unique challenges in ML adoption that cannot be addressed by generic, one-size-fits-all solutions currently on the market. This could be due to the complexity of the application domain, lack of budget and available resources, or being in a niche market that does not attract the attention of large technology companies. One such domain is developing ML models for defect inspection in automotive manufacturing. Finding small defects across a large surface area of a car on a moving assembly line requires dealing with the constraints of low frame rates but high resolution.
My team and I face the other side of the same constraints when applying ML to hand tracking software: the resolution must be low, but the frame rate must be high. Hand tracking uses ML to identify human gestures to enable a more natural and realistic user experience within a virtual environment. The AR/VR headsets for which we develop this software are typically at the edge with limited computing power, so we can't deploy large ML models, and they need to respond faster than the speed of human perception. Additionally, since it's a relatively new field, there isn't much industry data available for training.
These challenges force us to be as creative and curious as possible when developing our hand tracking models. Rethink how we train, examine our data sources, and try different model quantization approaches as well as compilation and optimization. Don't just look at how our models perform on specific datasets, but iterate on the data itself and experiment with how we deploy our models. This is, in most cases, how we learn, but do not have Solving “x” also means that our discoveries will be even more valuable; for example, creating a system that can operate with 1/100,000th the computational power of ChatGPT while maintaining extremely low latency that allows a virtual hand to accurately track a real hand. Solving these hard problems, while challenging, also brings commercial advantages: our tracking runs at 120 Hz compared to the standard 30 Hz, providing a better experience for the same power budget. This is not unique to our problem; many companies face specific challenges due to niche application areas that give them enticing prospects of turning ML experimentation into market advantage.
Machine learning is, by its very nature, constantly evolving. Just as pressure creates diamonds, enough experimentation can produce machine learning breakthroughs. But like all machine learning deployments, the backbone of this experimentation is data.
Data training ML model evaluation
AI innovation often revolves around the model architectures used and the annotation, labeling, and cleaning of data. However, this approach is not always sufficient when solving complex problems where previous data may be irrelevant or untrustworthy. In these cases, data teams must innovate on the very data they use for training. When training data, it is important to evaluate what makes the data “good” for a particular use case. If it does not adequately answer your question, you may need to approach the data set differently.
While surrogate metrics on data quality, accuracy, dataset size, model loss, and metrics are all useful, when it comes to training ML models, there are always unknowns that must be explored experimentally. At Ultraleap, we combine simulated and real data in different ways, iterating over datasets and sources and evaluating them based on the quality of the models they produce in the real world; literally testing them in the wild. This has broadened our knowledge of how to model hands for accurate tracking regardless of the type of image or device that is input. This is especially useful when creating software that is compatible across XR headsets. Many headsets work with different cameras and layouts, so ML models need to work with new data sources, which is why it's useful to have a diverse dataset.
To explore all parts of the problem and all the solutions, you need to accept that your metrics may also be imperfect and test your models in the real world. Our modern hand tracking platform, Hyperion, is built on an approach to data evaluation and experimentation, offering a range of hand tracking models that address specific needs and use cases, rather than a one-size-fits-all approach. Validating the data, models, metrics and execution without shying away from any part of the problem space leads to models that are not only responsive and efficient, but also offer new capabilities, such as tracking objects held in the hand or even very small microgestures. Again, the message is that broad and deep experimentation can deliver a unique product.
Experimentation (from every angle) is key
The best discoveries are hard-won. When it comes to true AI innovation, there's no substitute for experimentation. Don't rely on what you know, experiment in real application domains and answer questions by measuring your model's performance on the task. This is the most important way to ensure that ML tasks fit your specific business needs, widening the scope of your innovation and providing a competitive advantage to your organization.
About the Author
Iain Wallace is the Director of Machine Learning and Tracking Research at Ultraleap, a global leader in computer vision and machine learning. He is a computer scientist fascinated by the research and development of application-focused AI systems. At Ultraleap, Iain leads the hand tracking research team, enabling new interactions in AR, VR, MR, outside the home and wherever we interact with the digital world. He holds an MEng in Computer Systems and Software Engineering from the University of York and a PhD in Information Sciences (Artificial Intelligence) from the University of Edinburgh.
Sign up for the free insideBIGDATA newsletter.
Join us on Twitter: https://twitter.com/InsideBigData1
Join us on LinkedIn: https://www.linkedin.com/company/insidebigdata/
Join us on Facebook: https://www.facebook.com/insideBIGDATANOW
