Machine learning leads to the creation of modern AI systems, and when used to write code, it can create machine learning-like systems.
This is the implication behind a recent post by AI researcher and Keras creator Francois Chollet. He draws striking similarities between agent coding systems and machine learning. This should give every engineering leader pause.
Cholet says so clearly. “Sufficiently sophisticated agent coding is essentially machine learning. Engineers set an optimization goal and some constraints on the search space (specifications and their tests), and the optimization process (coding agents) is repeated until the goal is reached,” he said in a post on X.
This analogy extends beyond workflow. Chollet takes it to its logical conclusion at the artifact level. “The result is a black-box model (a generated codebase), an artifact that performs a task and is deployed without inspecting its internal logic, much like ignoring the individual weights of a neural network.”
Here, observation sharpens into warning. “This means that all the classic problems you encounter in ML quickly become problems in agent coding: overfitting to the specification, clever Hans shortcuts that don’t generalize outside of testing, data leakage, concept drift, etc.”
Chollet concludes with a forward-looking design question. “I also ask: What would Keras be for agent coding? What would be the optimal set of high-level abstractions that would allow humans to control the ‘training’ of the codebase with minimal cognitive overhead?”
Although the framework is deceptively simple, its implications are important. ML practitioners have learned over the years that models that meet benchmarks can actually break down, often through costly failures in production. Clever Hans shortcuts (learning how to exploit statistical artifacts in the test data rather than the underlying logic that the system should master) are notoriously difficult to detect until they are exposed by deployment. If agent coding systems become subject to the same failure modes, the software industry is about to inherit a set of problems for which it has no established tools or culture to address. A codebase that passes all tests but generalizes poorly is not a software bug in the traditional sense. This is a model failure. And most engineering teams don’t have the ability to diagnose that way.
Chollet’s final question on “Keras for agent coding” points to the urgent need to close the gap between organizations and tools. Keras was important because it allows practitioners to meaningfully control complex systems without requiring them to reason at the level of raw tensor operations. Current agent coding has few comparable abstraction layers. This allows engineers to specify their intentions, constraints, and generalization requirements in terms of ensuring that the optimization process can run reliably. Whoever builds that interface, whether it’s a startup, an established development tools company, or one of Frontier Labs, will define how the industry manages its AI-generated codebase for years to come. The questions Chollet is actually asking are not academic. It’s about who controls the handle when writing the code itself.
