For most of the history of science, prediction and understanding have been two sides of the same coin. Alan Hodgkin and Andrew Huxley captured action potentials using four variables and a small number of parameters. Their equation predicted the waveform, but explained It’s the quick intake of sodium and the slow excretion of potassium, and the interaction between them that produces the spike. The predictions were helpful. That understanding was great. And one cannot exist without the other.
Artificial intelligence (AI) is cracking them open.
Consider AlphaFold. Its predictive accuracy is amazing. However, there is no explanatory model that humans can internalize or reason about, and there is no “I see!” AlphaFold does not aid human understanding. that Replace the steps where understanding would occur.
Neuroscience is heading down the same path. The transformer architecture behind models such as Claude et al. Modeling population spike trains From Neuropixel recording. Foundation models are beginning to incorporate large-scale calcium imaging datasets. From startups like Edison Scientific to projects within Alphabet, Anthropic, and OpenAI, a growing roster of scientific AI companies are competing to automate scientific discoveries from big data, especially discoveries with commercial potential.
But the tools they are building may provide answers without insight or predictions without compression. They bypass the step of distilling complex phenomena into simpler general principles. And without that distillation, we are no better than we were before. We couldn’t understand the brain, and we still can’t understand models of the brain.
To address this, a whole subfield called mechanistic interpretability was born, just to unravel these models and find the principles inside. In a recent paper, we fit one type of machine learning model (sparse autoencoders) to another type (transformers trained on calcium imaging); Search for human-interpretable features Inside of the first model. This is a new age of irony. We need a scientific model of a scientific model now.
Why is “compression” necessary? Albert Einstein said (probably falsely) that a good theory should be as simple as possible, but no simpler. A theory is a short explanation that explains a much larger observation. The Hodgkin-Huxley model reduced spikes to four variables. The ring attractor model reduced head direction tuning to a single equation, which also accounted for path integration. These theories are compressed so they fit inside our heads and can be simulated in our heads. The ability to simulate a model in your head creates a sense of understanding. AI models have no such constraints. They can fit much more into their “head”. That means their internal model is much less compressed and can be much less readable for us. AI doesn’t need to be understood.
This raises uncomfortable questions. If AI provides accurate predictions that lead to drugs that cross the blood-brain barrier or stimulation patterns that suppress seizures, is there still room for understanding?Of course, many people, not just scientists, value understanding. So did the Medici family, who funded Galileo not because they needed a better navigation table but because they wanted a court philosopher. Science began as the pet dog of the wealthy and was funded for the same reason as art and music: because its patrons found it beautiful. But that’s not how we make a living anymore. The National Endowment for the Arts has an annual budget of approximately $200 million. The combined budgets of the National Institutes of Health and the National Science Foundation exceed $50 billion. Society doesn’t fund science at 250x the level of art because it thinks understanding is beautiful. The warm glow of understanding is the personal reward of scientists, not the social rationale for writing checks. Even if the forecast and its downstream benefits arrive without being understood, the public will likely accept it in stride.
Recent papers suggest that we should not surrender too soon.
Keyon Vafa and his colleagues Simulate tens of millions of planetary orbits We calculated it from Newton’s laws and trained the converter on the resulting sequence. This model predicted future positions with very high accuracy. But when they tweaked the model to infer the underlying gravity vector, the model produced nonsense. The implicit law of gravity varied depending on the subset of data the researchers examined. The converter had assembled a patchwork of accurate heuristics for every solar system in its training set, but it had not discovered the principle of universal gravitation. Without that principle, a model could predict the movement of a point of light in the sky, but it wouldn’t be able to send a rocket to the moon.
A transformer trained on motor cortex recordings may beautifully predict retained firing rates, but it cannot tell us what the circuit is actually calculating. Ptolemaic astronomers had a similar problem. Their geocentric model predicted the planet’s position over a millennium with amazing accuracy by stacking epicycles. Previous thinking has been theological. In other words, God’s universe must move in a perfect circle. When Newton eventually replaced it, there was little improvement in predictive accuracy. What changed was compression. All orbits, falling apples, and ocean tides were explained by a single law. In fact, the transformer is even less principled than Ptolemy. Ptolemy at least had a criminal record.
Understanding generalizes in ways that prediction alone cannot. David Hubel and Torsten Wiesel’s discovery of directional receptive fields in V1 did more than simply describe the activity of a set of neurons. It gave us feature detection hierarchies, a framework that generalized across the sensory cortex and inspired the convolutional neural networks that now power computer vision. Drift-diffusion models of decision-making originated in psychophysics and eventually came to explain the ramping activity of single neurons in temporoparietal regions. Compression provides a creative leap from one domain to another. A model that memorizes input-output relationships without learning the underlying structure will never be able to make that leap forward.
So where does this leave us? The compression step compresses a sprawling dataset into something portable and teachable, but remains a human task (at least for now). AI models can predict. They have not yet learned to explain. But Vafa’s results suggest that the drive for understanding is more than just scientists’ vanity. Even in the age of large-scale AI models, it may remain our most important work.
