I am sometimes A university asked me to give a talk about how I became a writer. The easy thing to do is to offer a kind of guided tour through a forest of literary self-formation. A series of anecdotes designed to elicit a few laughs, a moment or two of reflection on the inevitable turns in the road, things that felt important but turned out to be unimportant, or things that seemed unimportant at the time but turned out to be most important in hindsight.
These tours usually end at the same location. The author found a path through the wilderness and along the way discovered a voice. Voices are what guide us out of the forest.
The problem, at least for me, is that this kind of speech is mostly fiction. In retrospect, that path is just a path. Telling the story this way ignores, smoothes over, and downplays the role of circumstance and bad luck. Most of what writers experience is failure. It takes years to develop a voice. The point is not to get out of the woods quickly or unscathed. Getting lost is not a big deal. That’s all.
Now there’s an AI that claims to be our GPS through the woods. It’s not just a guide. Tireless, fearless and knows all the shortcuts. AI eliminates the need to go into the forest in the first place. Why are we faced with a blank page and a blinking cursor? Why is it so difficult to understand what we want to say and how to express it? Why listen to your own hoarse, wayward voice when you can always press a button and speak smoothly, clearly, and elegantly on any subject? Audio on demand.
When I talk to high school and college students (including my own children), I worry that they are being told, “You don’t need to bother,” at a time when they need to develop their voices. AI writes for us, reads for us, and thinks for us. Replace our voice with your own.
Except the AI has no voice. It’s our lip sync. It’s an average remix. Initially, large-scale language models did not include anything other than human language. Without the natural voice, there could be no artificial voice. However, if we become satisfied with substituting our own language for the language generated by the AI, we end up in a closed loop where the same output is recycled as input.
What I fear is that we are losing the ability to distinguish between our own voices and the voices of machines. Worse, you may lose the will to claim that it exists.
And that’s a discussion. The most bullish on machine learning claim that artificial general intelligence (AGI), an artificial intelligence model that can match or exceed human cognitive ability at any task, is just around the corner in just a few years. Some say more than 10 years. It’s a rolling target, always just over the horizon. But regardless of the timeline, the idea is that all of our “cognitive work” will soon be automated. They believe this is possible because they believe that the language we produce is interchangeable with the language produced by LLM.
I’m not interested in predictions or timelines or who is right or wrong and how much. I’m neither an AI expert nor an AI amateur. I’m not a neuroscientist or a cognitive scientist or a scientist of any kind. I am a parent of teenagers, a human being, a reader, and a writer, in that order. What I struggle with, like many others, is about AI and what it means for work, school, and life, and how to talk about all of that with my kids (who are definitely far more insightful about AI than I am).
What I’m most interested in is the “I” in AGI. What does that actually mean? And why let a few wealthy businessmen define it?
OpenAI CEO Sam Altman promised that working on Chat GPT-5 is like talking to “legitimate PhD-level experts in any field.” I can’t help but think about how obvious and strange the definition of intelligence is.
Don’t get me wrong. It’s incredible that we’re even having this conversation. I don’t want to minimize how far technology has come, how fast it has gone, or how far it might go in the future. The questions I would like to ask are: How can we create intelligence when we don’t fully understand what intelligence is, and can’t even really define it?
Returning to Altman’s formula, general intelligence means being a PhD-level expert in any field. Such expertise is undoubtedly impressive and is definitely related to or part of intelligence, no matter how it is defined. But that’s only a small part of intelligence. My alma mater, the University of California, Berkeley, offers doctoral programs in 94 research areas. Perhaps AGI will cover them all.
However, getting a degree does not cover or even touch on emotional intelligence. What is a PhD? Are you reading the room? How to teach a child how to ride a bicycle? Were you moved to tears by the music? We think of elephants as wise because they mourn the death of a dead person. What is a PhD? Sadness, awe, wonder, curiosity?
Perhaps no one is surprised that some of the world’s best scientists and engineers define intelligence the way they do. Even if an AGI champion’s motivations were completely altruistic, they would still be biased by their own view of the world, their own experiences and successes. Researchers at the forefront of AI are some of the brightest and most skilled minds on the planet. And they constitute a very limited, self-selected group of people who are primed to understand certain types of knowledge better than others. Knowledge that forms the basis of our most extensive and highly accurate theory of the universe. With this knowledge, we were able to create technologies that changed the world. However, it is only a small part of all knowledge, a small part that can be expressed symbolically in language and mathematics.
The rest is what philosopher Michael Polanyi called “tacit knowledge,” which constitutes a much larger volume of data and interacts with it in many more ways. His philosophy of knowledge can be summarized as follows. “We know more than we can tell.”
Is it part of AGI? I don’t think so. I can’t believe it until ChatGPT texts me a link to a video that makes me laugh, cry, or makes me reconsider my opinion of something I said last time.
Until that happens, I would argue that the “I” that engineers are chasing is either a surrogate or a misnomer. It’s not like intelligence as we understand it.
Some might say that this argument, based on a human-centered view of intelligence, is flawed. Perhaps we need to let go of our preconceptions and embrace the idea that machine intelligence can, and perhaps should, be fundamentally different from human intelligence. Perhaps machine intelligence doesn’t need sentience, autonomy, curiosity, or emotion.
Please say I admitted it all. My point is that whatever a machine can do, no matter how incredibly useful or potentially economically valuable that capability may be, none of it is worth words. intelligence.
A few outliers aside, even the most ardent supporters of AGI do not believe that frontier AI models can: feeling. This means that we must assume that the intellect can be separated from the body and emotions. they say: We understand what intelligence is in a distilled and separated form.
I would like to say this. Please share that definition with others.
If they’re right, we’ll soon find out.
But if they are wrong, the relentless pursuit of AGI poses real risks to social policy, education, the power grid, the economy, and the environment. Already, it feels like generative AI is supplying demand. The need to scale, combined with the ever-present pressure for higher rates of return, creates a daunting convergence of capital and social resources into one industry. Generative AI is the technology equivalent of high fructose corn syrup. This is a potentially useful ingredient that is now incorporated into many of the things we consume without our consent.
But perhaps just as important is the potential harm to our own self-concepts as individuals and as a species.
AI will continue to evolve. It could change the world. Perhaps it already is. But it doesn’t replace the human voice for now, and probably never will.
Voice is what we use to communicate with each other. Voices are the sounds we make as we travel through the unknown, the sounds we make when we echolocation, trying to map the world and place ourselves within it. Voices encode experience, loss, pain, and joy. We don’t gain a voice in spite of our failures; we gain a voice through our failures. Because of that.
AI has no voice and cannot communicate with us. not much. it answers our question. That’s what it was made for. It’s an answering machine. But we are questioning machines. Questions are essential to intelligence. Without them, we become static and stagnant. Without them we would not evolve. We can learn the answers, but only by asking questions. The question is how do we recursively self-improve? We humans constantly inspire each other in creative ways. I urge you. I’ll answer. Our answers become new prompts. Our context window is our whole life. Our tokens are countless.
This is more than semantics. By calling what AI can do “intelligence,” we confuse technological capabilities with human characteristics. We degrade ourselves, not by talking to AI, but by measuring ourselves against AI. The danger is not that we overestimate AI. It’s that we underestimate ourselves.
This essay is adapted from Charles Yu’s 2026 Joel Connarrow Lecture, delivered at Davidson College on February 10th.
