Most people still use AI like the search box of 2015. Type, read, type again. The new pattern replaces manual back-and-forth operations with a loop. This guide will explain loop engineering Use two validated artifacts. Source is Andrej Karpathy. autoresearch repository and Bilevel Autoresearch paper. Frame configuration is as written by @0xCodila.
What is loop engineering?
First, let’s compare the two modes. A prompt is a single instruction that then determines the next step. In contrast, a loop is a goal that the model pursues until it is reached. The model repeats planning, execution, and checking the results. Once you define your objective, the loop handles the iterations. Importantly, loops capture costs only when the work is measurable.
Three parts that make loops work
So what differentiates real loops from chatbot iterations? Every reliable loop has three components.
- a verifier Score each attempt. That check could be a test passing, a metric moving, or a build. In the absence of verifiers, agents simply agree with themselves repeatedly.
- state Record what was tried, what failed, and what remained. If you use a small side file, the next run will resume without restarting.
- a Stop conditions Prevent runaway costs. The loop stops after either the goal is met or after N tries.
The Karpathy Loop: Inside the “Automated Investigation”
These three parts are not theoretical. On March 7, 2026, Karpathy was released autoresearchan open source repository under the MIT license. It ships with three core files and approximately 630 lines of code. The project quickly spread within days and currently has nearly 90,000 GitHub stars. It was later presented as a pattern called “Kalpathy Loop”.
The design is intentionally small and yet rigorous. Agent only edits train.pyGPT model, optimizers (Muon and AdamW), and training loop. The evaluation utility is untouchable. prepare.py. This separation prevents the agent from improving the model to facilitate testing. Meanwhile, humans write program.mdinstructions that agents must comply with.
One experiment is run in each cycle. The agent reads the code, suggests changes, trains for 5 minutes, then maintains or rolls back. The scoring indicators are val_bpbvalidation bits per byte. Lower is better. This budget allows for approximately 12 experiments per hour, resulting in approximately 100 experiments per night.
The reported results are tangible. Karpathy pointed it out to me already optimized nanochat GPT-2 training code. It ran for two days, completed approximately 700 experiments, and left 20 true improvements. Cumulatively, these fixes reduced GPT-2 quality training time by 11% from 2.02 hours to 1.80 hours. One discovery is that QK-Norm The implementation lacked a scalar multiplier, which was too distracting.
What’s interesting to note is that humans get tired after about 12 experiments, whereas loops don’t get tired. Separately, Shopify CEO Tobi Lütke said: autoresearch All night long in the internal model. He reported a 19% improvement after 37 trials. Karpathy’s lesson: If you have an objective metric, it becomes a bottleneck.
Prompt vs Loop vs Two-Level Loop
The difference becomes more obvious when you compare them side by side.
| side | one shot prompt | Karpathy loop (autoresearch) |
Bi-level automatic research |
|---|---|---|---|
| you define | each step | The goal is once | The goal is once |
| who iterates | you | internal agent | internal agent + external agent |
| verifier | you manually | prepare.py (val_bpb) |
Same metric, two levels |
| state | chat only | Experiment record | Logs and inserted code |
| role of humans | engine | author of program.md |
author of program.md |
| Report results | various | 700 runs → 20 fixes, 11% faster | val_bpb drop is 5x larger |
5 building blocks
As a result, the AI engineering team is currently assembling the next work loop. 5 reusable pieces:
- automation Start the loop on a schedule, event, or trigger.
- a skill Store your project’s knowledge in markdown files that are read every time you run it.
- subagent It separates writers and reviewers because one model rates itself too generously.
- connector Run loops inside real tools like issue trackers and Slack.
- lastly, verifier It remains a gate that rejects bad work. Claude Code and Codex are currently shipping all five.
Bi-level automated research: loop-on-top-of-the-loop
Next, the researchers asked even more pointed questions. If automated research is research, can automated research be automated research?Research paper Bilevel Autoresearch: Meta-Autoresearching Itself I say yes.
of inner loop Matches Karpathy’s original: suggest, train, evaluate, keep or discard. of outer loop Watch the inner loop, read and trace its code. Identify where the search itself keeps getting stuck. Next, create new Python mechanisms, inject them at runtime, and rerun the inner loop.
The results held true for Karpathy’s GPT pre-training benchmark. outer loop cut val_bpb 5 times the single loop (-0.045 vs. -0.009). In particular, both loops used the same LLM, which would have benefited from the architecture rather than a smarter model. The design is actually divided into three levels. Level 1 executes a basic loop. At level 1.5, we adjust the search parameters every 5th iteration. Level 2 involves generating mechanisms through four round sessions. The reported experiments used an RTX 5090 32GB and a budget of 300 seconds.
The reason is worth noting. The inner loop kept returning to the same previous state even after it stopped working. The outer loop broke these patterns by forcing unfamiliar exploration.
Usage and examples
These ideas apply beyond pre-training. For model work, the loop searches for hyperparameters. val_bpb dripping For software, it is refactored until the tests, types, and builds pass. Content will be rewritten until all rubric scores clear the threshold. For data, tune the pipeline until the schema check passes. Each case has one characteristic. Automatic gates that may fail to work.
Try it yourself: Loop with one prompt
Theory aside, you can feel the mechanics even without the Claude Code or Codex. Paste this into the corresponding model and watch it auto-fix.
You will work in a loop until the task meets the bar.
TASK:
[describe exactly what you want produced]
SUCCESS CRITERIA (be strict):
- [criterion 1]
- [criterion 2]
- [criterion 3]
LOOP PROTOCOL, repeat every turn:
1. PLAN - state the single next step.
2. DO - produce or improve the work.
3. VERIFY - score the result 1-10 on each criterion. Be honest.
4. DECIDE - if every criterion is 8+, print FINAL and stop.
Otherwise print ITERATING and fix the weakest point first.
RULES:
- Never call it done until every criterion is 8 or higher.
- Each pass must fix the weakest score from the last VERIFY.
- Do not ask questions. Make a sensible assumption and continue.
Begin.The control flow underneath is small. The skeleton below shows three parts of Python: a validator, a decision, and two stopping conditions.
current = baseline
best = evaluate(current) # verifier: lower val_bpb is better
for step in range(MAX_STEPS): # stop condition 1: experiment budget
candidate = propose_change(current) # agent edits train.py
score = train_and_eval(candidate) # train 5 min, then verify
if score < best: # keep only real improvements
current, best = candidate, score # commit
# else: discard candidate, restore baseline
if best <= TARGET: # stop condition 2: goal met
breakBoth versions are available in limited quantities. You are still the trigger and the state is cleared when you close the tab. Adding automation, state files, and actual gates turns this into an autonomous loop.
try running
The interactive demo below animates one complete loop: propose, train, validate, and retain or rollback. Adjust and monitor your goals and step limits val_bpb It will fall until the stop condition fires.
