Qualitative results
Our qualitative analysis reveals subtle landscapes of clinician attitudes, information needs, and preferences regarding AI-driven closed-loop medical neural technology. To construct the findings, we organized clinician responses across three dimensions of the algorithmic system. input (training data), Core Architecture (algorithm), and its output (Predictions and clinical decisions). Additionally, we also report their design, taking into account the user interface and XAI visualization. Given the centrality of the current discussion of explanability, we begin with the clinician's view of the algorithm itself. A complete overview of the concepts related to all explanations is displayed in the concept map below (see Figure 1).
Explains explanations related concept maps. The concept maps related to this explanation show new themes based on thematic analysis of expert interviews.
Algorithm specifications and perceived associations
A consistent theme throughout the interview was the limited interest of clinicians in the technical specifications of AI models embedded in closed-loop medical neural technology systems. Several participants (9/20) explicitly reported little value in understanding technical details such as algorithm type, number of layers, and parameter counts. These aspects were generally perceived as overly technical and out of the scope of clinical expertise and responsibility, providing minimal practical or clinical utility for decision-making in patient care.
One participant noted that many physicians lacked expertise in distinguishing different machine learning (ML) algorithms, and that several clinicians questioned the explanability and even whether the XAI method was necessary in the context of closed-loop medical neurological technology. Six of the 20 people recognized the inherent opacity of the AI models, but still expressed openness to use them. One participant stated that a detailed understanding of the underlying AI model is rarely essential for physicians, while another observed that patients usually show little concern about the transparency of the algorithm. This practical posture was reflected in comparison with non-AA-driven interventions such as traditional deep brain stimulation (DBS) in Parkinson's disease. Still, one clinician has repeatedly emphasized that algorithm transparency is important for AI developers and engineers, especially for system verification, safety assurance, and error detection. From this perspective, even if clinicians do not require direct algorithmic insights, they rely on technical and regulatory actors to ensure that such understandings are firmly embedded at the system level.
Input data information request
Our analysis reveals that clinicians' interest in input data is shaped by underlying concerns related to clinical representation, data accessibility, data quality, and real-world therapeutic decisions. Several interviewees highlighted the important role of knowledge about input data in shaping clinicians' trust and expectations for future AI-driven medical closed-loop neurotechnology.
Training data representativeness
Clinicians (4/20) expressed skepticism about whether the training dataset could properly capture the entire range of neurological and psychiatric symptoms. In disorders such as Parkinson's disease, where symptoms such as tremors vary widely among individuals, participants emphasized the need to understand whether AI models are trained in data representing their patient population.
Accessing input data
Some research-oriented clinicians (3/20) highlighted the need for transparent access to raw data making AI decisions, and highlighted it to better understand AI and determine its applicability. In particular, in high-stakes contexts such as critical care, they expressed their desire to independently inspect and interpret input signals. This demand for data transparency was closely related to the needs of clinicians to validate AI-driven interventions and maintain clinical surveillance, particularly when treatment mechanisms cannot be fully explained without AI.
Multimodal input and new biomarkers
Several interviewees agreed that neural activity data alone was insufficient to construct and implement high-performance closed-loop systems. Clinicians strongly advocated for the integration of complementary inputs, including wearable sensor data, video-based movement assessments, and subjective patient-reported outcomes, including measurements of quality of life. One clinician even stated that as long as the patient's well-being is improved, the therapeutic effect does not require complete mechanical understanding. As a result, clinicians also expressed their desire to know whether multimodal and subjective patient data were used to train a particular AI system. At the same time, four clinicians said that specific biomarkers of neurological and psychiatric conditions are already too complicated to fully understand without the use of AI, and that parameter settings for some neurostimulation devices are often determined through trials and errors. This highlights the shift towards result-oriented verification and supports the integration of multimodal data pipelines in AI system design.
High data quality and generalizability
Three of the 20 clinicians raised concerns about the noise and susceptibility of artifacts in brain data, especially when acquired under clinical conditions. They emphasized the need for robust pre-processing pipelines and artifact removal to ensure that the algorithm learns from signals rather than noise, and requested that appropriate procedures be reported. Additionally, one participant questioned the validity of the existing data set, citing small sample sizes and variability in patient status, device configuration, and electrode placement as potential limitations.
Important information about output
Clinicians expressed their strongest concerns (11/20) about the output of AI-driven systems, particularly in terms of safety, patient benefits and clinical relevance, and linked this to information needs and preferences. Many emphasized that their trust in AI is shaped by the real-world outcomes of their decisions and actions rather than insight into the algorithm itself.
Safety of output and operational transparency
Another concern raised by one clinician was the safety of the algorithm output, particularly in a scenario where the system could autonomously adjust neural stimulation parameters in real time. Four of the 20 clinicians emphasized the need to understand not only the accuracy of the system, but also how its output is operated and translated into clinical action. This concern was particularly severe in sensitive or unpredictable environments that include potential real-time interventions, such as seizure detection by a responsive nerve stimulation (RNS) system. Seven of the 20 clinicians pointed to the importance of well-defined safety boundaries to prevent output that could lead to harmful or socially dangerous behavior (e.g., driving or unsupervised movement). Several participants emphasized that autonomous adjustment of the tuned stimulus parameters in AI models must be constrained by stiff safety restrictions and that output decisions must be ensured not to exceed clinical safety thresholds.
Coordination of clinical reasoning with evidence of patient benefits
Clinicians are asked to consider whether the system's recommendations are clinical reasoning or “Intestinal sensation”. In their view, trust was not obtained through detailed algorithmic explanations, but through a consistent agreement between AI recommendations and clinician intuition. Most clinicians are optimistic about the future role of AI in closed-loop systems, but expressed frustration at the current gap between technical potential and clinical application. For example, he warned that adding adaptation capabilities to DBS treatment acknowledges the strengths of AI in data processing and predictive modeling. He emphasized that the real-world dataset required to rigorously assess the clinical utility of AI should only be deployed if AI leads to clear and demonstrable improvements in patient outcomes, and that these benefits should be assessed carefully through transparent, ethically sound clinical trials.
Meaningful Clinician-AI Interactions and Clinical Relationships
Eight participants emphasized that AI needs to support clinical judgment rather than replace it. One clinician warned that AI models could identify statistically significant patterns that lack medical validity, and strengthen the need for clinician monitoring in defining use cases and verifying model output. Several clinicians also emphasized that AI systems require clear hypotheses to address real clinical problems and cannot generate meaningful solutions on their own.
AI User Interface Design Requirements
Clinicians provided valuable insight into the design of user interfaces for AI-driven closed-loop neurotechnic technologies, when highlighting the importance of intuitive, context-specific visualizations that support understanding without requiring technical expertise in ML, and when prompted, highlighted the importance of intuitive, context-specific visualizations.
Transparency of data through visualization
There was broad consensus among participants that visual summaries such as descriptive statistics and charts enhanced understanding of training data. These tools were considered essential to assess the representativeness and relevance of the dataset to individual patient cases. One clinician proposed incorporating visualization of symptoms clusters to verify whether individual patients match a particular subgroup of the training set.
Links to scientific evidence
Clinicians welcomed the idea of embedding hyperlinks in peer-reviewed publications that underpin model decisions. Such links can strengthen trust by indicating that the output of the algorithm is based on established clinical knowledge and that the user can verify the underlying medical evidence.
Explanability Tool
Only three out of the 20 participants voluntarily expressed interest in formal XAI methods such as functional relevance, functional importance rankings, and counterfactual examples. Those who appreciated their ability to identify top predictors and explore counterfactuals. What if scenario. Additionally, one participant sought access to source data within the interface, particularly for research or verification purposes. Interestingly, the two clinicians prefer a paper-based format over a digital dashboard when reviewing complex patient profiles, suggesting that interface flexibility is important.
Limitations of visualization
Despite the general assessment of transparency tools, several participants warned that data visualization alone would not completely solve the challenge of algorithmic opacity. As a result, clinicians are not completely transparent; Practical and easy to understand: An interface that conveys enough information to safely and confidently integrate AI support into clinical workflows without overwhelming users with unnecessary complexity.
Descriptive theme-specific citations are shown in Table 1.
