As advances in robotics, autonomous driving, and spatial computing continue to advance, an increasing number of computer vision and machine learning (CVML) algorithms are incorporating three-dimensional data into their frameworks. Debugging these 3D CVML models often requires going beyond traditional performance evaluation methods and requires a deeper understanding of the algorithm’s behavior within its spatiotemporal context. However, the lack of appropriate visualization tools makes it difficult to effectively explore 3D data and spatial features in relation to key performance indicators (KPIs). To address this challenge, we consider applying immersive analytics (IA) techniques to enhance the debugging process of 3D CVML models. Through in-depth interviews with eight CVML engineers, we identify common tasks and challenges faced during the development of spatial algorithms and establish a set of design principles for creating tools tailored for evaluating spatial models. Based on these insights, we propose a new immersive analysis system for debugging indoor localization algorithms. The system is built using web technologies and integrates WebXR to enable fluid transitions across the real-virtual continuum. We conducted qualitative research with six CVML engineers using the system on Apple Vision Pro to observe their analysis workflows as they debug indoor localization sequences. We discuss the benefits of employing immersive analysis in model evaluation workflows and highlight the role of seamlessly integrating 2D and 3D visualization across different immersion levels to facilitate more effective model evaluation. Finally, we review implementation trade-offs and discuss the generalizability of our findings for future efforts in debugging immersive 3D CVML models.
- † Harvard University, Cambridge
