For many of us innovating in the AI space, we are working in uncharted territory. Given how quickly AI companies are developing new technologies, the painstaking work behind the scenes may seem like a given. But in a field like XR, whose mission is to blur the boundaries between the real and digital worlds, we need to think outside the box because there isn't much past data or research to fall back on at this time.
While it is most convenient to rely on traditional machine learning wisdom and proven techniques, in emerging fields this is often not possible (or is not a complete solution): solving a problem that has never been solved before requires a new way of approaching it.
It's a challenge that forces you to remember why you got into engineering, data science, or product development in the first place: a passion for discovery. I experience this every day in my job at Ultraleap, where we develop software that can track and respond to the movements of the human hand in mixed reality environments. Much of what we thought we knew about training machine learning models is turned upside down in our work, because the human hand, and the objects and environments it encounters, are wildly unpredictable.
Here are some of the approaches my team and I have taken to rethink experimentation and data science to bring intuitive interactions to the digital world that are as accurate and feel as natural as they are in the real world.
Innovation within the line
When innovating in emerging fields, we are often faced with constraints that seem to contradict each other. My team is tasked with capturing the complexity of hand and finger movements and how they interact with the world around them. All of this is packaged into a hand tracking model that fits XR hardware on constrained computing. This means that our models, while sophisticated and complex, need to consume significantly less storage and significantly less energy (on the order of 100,000 times less) than the large-scale LLMs we see in the news. This is an exciting challenge that requires us to thoroughly experiment and evaluate our models in real-world applications.
But the countless tests and experiments are worth it. Creating powerful models while keeping inference costs, power consumption, and latency low is an incredible thing that can be applied to edge computing outside of the XR realm.
The constraints encountered during experimentation have implications for other industries as well: nuances in the application domain pose unique challenges for some businesses, while others are in niche markets not served by major tech companies and therefore have limited data to process.
While a one-size-fits-all solution may be sufficient for some tasks, many application domains require solving real, challenging problems that are task-specific. For example, automotive assembly lines implement ML models for defect inspection. These models must process very high-resolution images required to identify small defects across a car's large surface area. In this case, the application requires high performance, but the problem to be solved is how to achieve high-resolution models even with low frame rates.
Evaluating model architectures to drive innovation
Good datasets are the driving force behind the success of any AI breakthrough. But what does it mean for a dataset to be “good” for a particular purpose? And how can we trust that existing data is adequate when solving a previously unsolved problem? We cannot assume that metrics that work for some ML tasks will also apply to performance for another specific business task. This is where we need to counter commonly held ML “truths” and instead actively explore how to label, clean, and apply both simulated and real data.
By nature, our field is difficult to evaluate and requires manual quality assurance. We don't just look at data quality metrics; we iterate over datasets and data sources and evaluate them based on the quality of the models they produce in the real world. When we reevaluate how we rank and classify data, we often find datasets and trends we had overlooked. Now we know more about those datasets and what data is being used to model them. do not have Relying on this has allowed us to open up new avenues that we had previously overlooked.
Hyperion, Ultraleap's latest hand tracking platform, is a great example: advances in datasets have enabled the development of more sophisticated hand tracking that can accurately track microgestures and hand movements, even when the user is holding an object.
A small step back, a big step forward
The pace of innovation never seems to slow down, but it can. We are in the business of experimenting, learning, and developing, and taking the time to do exactly that often produces something much more valuable than following a textbook and rushing out the next technology innovation. There is no substitute for the breakthroughs that occur when we explore data annotations, question data sources, and redefine the quality metrics themselves. And the only way to achieve this is by experimenting in real application domains with model performance measured against the tasks. Rather than viewing unusual requirements and constraints as limitations, we can turn these challenges into opportunities for innovation and ultimately into competitive advantage.
