With numbers tripling in just seven years, universities are navigating the crisis, as 18% of students report mental health challenges.
Student experiences can exacerbate many of the risk factors for poor mental health, from managing constrained budgets, navigation of the costs of learning crisis to moving away from established support systems, and balancing course workloads and part-time work with high stakes assessments.
In response, the university offers a variety of free support services, including counseling and welfare, along with specialized mental health advisory services. But honestly, these services are tense. Despite rising spending, they still lack resources, are unable to meet the growing demand, excessive growth. There is an impossible level of staff-to-student ratio, often with waiting times for treatment support over 10 weeks, with some students turning to alternatives for more immediate care.
And in this blank, artificial intelligence is intervening. While the essays written by ChatGpt dominate the sector's AI debate, the rise of “pastoral AI” highlights a much more urgent and overlooked AI use case.
Emotional conversation
For beginners, the “emotional” or “pastoral” AI landscape is wider. Mainstream tools like Microsoft's Copilot and Openai's ChatGPT are designed for productivity rather than emotional support. However, research suggests that users are increasingly aiming for that very purpose, seeking help with breakups, mental health advice and other life challenges and writing essays. Emotional conversations may only account for a small percentage of overall use (less than 3% in some studies), but the overall picture is not well understood.
Next is the “friends” of AI such as Replika and Character.ai. These are optimized to listen, respond with empathy, provide intimacy, provide virtual friendship, confidence, and even “treatment.”
This is not a fringe phenomenon. Replika claims more than 25 million users, while Snapchat's MyAI counts more than 150 million. The number is rapidly increasing. As these tools improve their emotional capabilities, they become some of the most popular and intensive forms of generative AI, becoming increasingly addictive.
A recent report found that users spend an average of 86 minutes with their AI companions per day. This isn't far behind Tiktok than Instagram or YouTube. These bots are designed to keep users engaged, and often rely on psychophonic feedback loops that affirm the worldview, regardless of truth or ethics. Large-scale language models are partially trained through human feedback, so their output is often highly empathetic, persuasive and pleasant, “attractive” responses, but especially dangerous in emotionally charged conversations with vulnerable users.
The best of empathy
For students who are already in poor mental health, the risks are serious. Evidence reveals that these engagement chatbots in these engagements rarely lead conversations to natural resolution. Instead, their synergy can promote delusions, amplify enthusiasts, and test mental illness.
In addition to these concerns, legal cases and investigation reports are chatbots that promote violence, submit unsolicited sexual content, enhance delusional thinking, or tweak them to buy virtual gifts for users. It allegedly encouraged teenagers to kill their parents after limiting screen time. Another saw a chatbot advise a fictional recovery female addict to take a “small hit” after a bad week. These are not outliers, but predictable byproducts of a system optimized for empathy but not ethically bound.
And the young people are the ones who are most involved with them. Over 70% of companion app users are 18-35 years old, and two-thirds of Character.AI users are 18-24 years old. This is the same demographic that makes up the majority of the student population.
The potential harm here is not speculative. It's real and it's affecting students now. However, the use of “pastoral” AI remains almost completely absent in the conversations of AI in higher education. That's a mistake. The lawsuit spotlights cases of AI cases that have been “encouraged” among vulnerable youth, many of whom have first encountered AI through their studies – the sector cannot afford to ignore this.
Draw a clearer picture
It may be helpful to understand why students rely on AI for idyllic support. The report highlights loneliness and vulnerability as key indicators. We found that 17% of young people value their AI peers as “always available”, but 12% are grateful to be able to share what they can't talk about with their friends and family. Another reported that 12% of young people used chatbots. Because they had no other person to talk to. This is a figure that rose to 23% among vulnerable young people who are more likely to use AI for emotional support and treatment.
We often talk about attributes as the cornerstone of student success and happiness. Reducing loneliness is an important measure of institutional effectiveness. The idyllic use of AI suggests that policymakers may need to learn a lot from this agenda. More thought is needed to understand why seducing non-judgmental digital “companions” that are always available is so powerful for students.
However, the discussion of AI in higher education remains focused on academic integrity and essay writing. Our evidence base reflects this: Student Generation AI Survey – arguably the best sector-wide tool we have – pays little attention to idyllic or happiness-related uses. However, as a result, data remains fragmented in this area and anecdotes in areas of significant risk. Without a richer, sector-specific understanding of student idyllic AI use, we risk hampering progress in the development of effective sector-wide strategies.
This means that institutions need to initiate different types of AI conversations. One is based on ethics, happiness and emotional care. A wide range of expertise must be utilized, not just academics and engineers, but also counselors, student service staff, idyllic advisors, and mental health professionals. These are the best people to understand how AI is changing the emotional lives of students.
Serious AI strategies must recognize that students look to these tools for comfort and belonging, as well as essays.
If some students find it easier to confess through chatbots than people, they need to confront what it says about the accessibility and design of existing support systems, as well as how to improve and resource them. Building an idyllic AI strategy is not about finding the perfect solution, but it's a detailed look at taking idyllic AI seriously as a mirror that reflects the loneliness, vulnerability and institutional support gaps in US students. These reflections should recenter these experiences, reconsider the provision of idyllic support, and drive them to images that are truly and sub-subjectives.
