The complexity of managing mental health AI

millions of americans is affected Even though mental illness occurs every year, treatment remains expensive. is out of reach for many people and shortage The proportion of licensed doctors means that even those with insurance often wait months for an appointment. Into this void has entered a rapidly expanding market of AI-powered tools, including chatbots that provide therapeutic counseling and apps that provide on-demand cognitive behavioral therapy exercises. In times of loneliness and emotional distress, both children and adults turn to general-purpose chatbots like ChatGPT and “companion” bots like those offered by Character.ai and Replika.

The use of AI in mental health care has promising potential benefits. These include tools that can improve access, reduce costs, and expand patient coverage. overextended systemand is a form of social and emotional support for people experiencing loneliness. But these promises also come with risks. In the absence of clear regulation and standardized third-party testing, these tools risk providing substandard care and burdening users. at risk. There are a lot of news headlines about the development of minors. unhealthy emotional attachment A chatbot receives a message from a user in crisis. Harmful or inappropriate responseand research showing that general-purpose AI chatbots are commonly used. miss the warning signs.

The role of AI in mental health care is rapidly increasing, and lawmakers are struggling to keep up. To date, most legislative activity has taken place in the states; 140 or more invoices Relates to the use of AI in the context of mental health. federal government law teeth pendingbut so far the focus has been on the protection of minors. this fragmented Policy prospects are further hampered by persistent policies Lagging evidence base: Many purpose-built AI mental health tools lack validated results or representative samples, and are rarely evaluated in rigorous research designs. On the other hand, the models behind general-purpose chatbots are updated rapidly, so safety research results can quickly become outdated.

Recognizing these governance challenges, the Stanford Human-Centered AI Institute (HAI) convened a select group of leading researchers, clinicians, policy makers, behavioral health experts, ethicists, AI developers, and patient advocates for a policy workshop on mental health and AI in June 2026. AI for Mental Health Symposium Held earlier in the day, the organizers HAI Healthcare AI Policy Steering Committee in cooperation with universities AI for Mental Health (AI4MH) Initiative.

Based on the Chatham House Rules, participants openly discussed new efforts to regulate the use of AI in mental health care. Evidence gaps that must be addressed to enable sound policymaking. and the technical feasibility of potential policy measures and guardrails. Below we summarize three key policy issues that the group identified for further study.

1. This field needs a clear definition

To achieve effective regulation, policy makers, mental health experts, and AI developers must agree on the boundaries of “mental health AI.” “Mental health AI” is a broad umbrella term that refers to a variety of tools and applications (see table). At this time, there is no widespread consensus on what specifically counts as an AI mental health tool, where the line is between clinical and wellness functions that require regulation, and what policies should apply to which products. Should chatbots that are clinically developed specifically for mental health-related matters, and not originally designed for that purpose, be treated the same as general-purpose chatbots that users turn to during a mental health crisis?

While all mental health AI tools must meet some basic safety expectations, different types of AI-powered mental health tools may have different expectations and risks, and therefore may require different regulation. However, many legislative approaches do not consider the differentiated impacts of different mental health AI products. for example, complete ban All the AI ​​used by human providers to deliver psychotherapy services does not address the reality that people may still turn to generic chatbots for treatments that have not yet been vetted or supervised by a clinician. In other words, laws aimed at moving AI away from therapeutic relationships may instead steer users toward less customized, less regulated, and common tools. Until definitions are clear, regulations will remain fragmented, evaluation criteria will be inconsistent, and companies will continue to operate in ambiguity.

Overview of mental health AI categories

2. Delay in evaluation method

While methods to systematically evaluate the performance of other healthcare AI tools are rapidly developing, methods to assess how effectively and safely mental health AI tools perform in the real world have lagged behind. of High-stakes interactions are rare. Simulation is difficult, model behavior is unpredictable in real-world situations, and single-session testing reveals little about long-term effects. On the other hand, the real-world chat data needed to study safety and efficacy at scale resides primarily within the industry, with no meaningful structure for sharing it with independent researchers or regulators.

This creates a policy bottleneck. Without large-scale interaction data, many reasonable safety requirements cannot be justified or enforced. Let’s think about people who are in a good mood. The trend is for AI tools to validate and affirm users. may be particularly harmful For people with OCD, seeking validation and long-term involvement can reinforce compulsive patterns. In order to develop and implement appropriate policy mechanisms to address a problem, we need to know how often it occurs, who it affects, and how it impacts it. We cannot answer those questions at this time.

Even when data exists, the assessment landscape is fragmented. Although researchers have proposed a variety of approaches to benchmarking, there is no consensus on what and how it is most important to measure or whether benchmarking is an appropriate tool for systems whose behavior changes from week to week. Of the many existing metrics, most reflect technical priorities set by developers, e.g. Message percentage It is not a goal or technique of mental health care (e.g., patient-tailored treatment), but rather a possible symptom of a mental health emergency, which is itself objectionable. Currently, most evaluation frameworks focus on single point-in-time analysis, which makes them poorly suited for understanding the long-term impact of chatbot usage on users. This is a big gap: early evidence suggest Long-term evaluation is essential, as long-term use of some types of chatbots can worsen health conditions. Policy makers, researchers, and industry need to collaborate further to standardize the assessment of mental health AI and move toward multidisciplinary, multidisciplinary assessment. Chatbot multiple interactions.

3. Aim for low-hanging fruit while tackling deeper problems.

Workshop participants agreed that more comprehensive mental health AI policies are urgently needed. The good news is that there are some low-hanging fruit that policymakers can and should act on immediately. Transparency requirements, crisis response mechanisms, data protection measures, and parental controls for minors have broad agreement and urgency, so these items are already subject to regulation. most commonly passed Provisions of state mental health AI laws. When states enact thoughtful regulations on these issues, they can set precedents for other states and lead to meaningful safety improvements.

But achieving long-term, meaningful change requires addressing deeper policy challenges and resolving tensions across today’s patchwork of state laws. Different state laws regarding therapeutic tools can be a complication for psychotherapists licensed in multiple states with different regulations. Perhaps the most under-considered and unresolved issue in this conversation is one of the most fundamental. Business models built around maximizing user engagement are structurally inconsistent with the goal of fostering healthy relationships with chatbots. We have seen this play out before – court already have tied Social media’s “keep it on the platform” logic leads to addiction and harm to minors. the study Heavy use can lead to poor mental health and suicidal behavior. Chatbots that are optimized to simulate intimate human relationships can also encourage overuse and overdependence. Without mechanisms to reward responsible behavior, there is little reason to expect industries to self-regulate differently.

Finally, the policy debate is currently too narrow in scope. The report is dominated by the perspectives of people with high incomes, commercial insurance, and professional qualifications, with underrepresentation of people with severe mental illness, young people, and individuals involved in social services and the criminal justice system. leave them alone risk Existing inequalities are being exacerbated on a large scale, especially at a time when policymakers are seeking immediate relief.