Karpathy's Autoresearch: AI Agents Running ML Experiments While You Sleep

The Pitch

On March 7, 2026, Andrej Karpathy dropped a new open-source repo called autoresearch. The idea is simple: give an AI agent a small but real LLM training setup and let it experiment autonomously overnight. The agent modifies the training code, trains for five minutes, checks if the result improved, keeps or discards the change, and repeats. You wake up in the morning to a log of experiments and, hopefully, a better model.

Two days later, Karpathy shared results. He’d left autoresearch running on a depth-12 model for roughly 48 hours. The agent found about 20 changes that improved validation loss. Every single one was additive, and they all transferred to larger depth-24 models. That’s not the agent overfitting to a toy setup. It was finding real, generalizable improvements to neural network training.

The repo hit 15,000+ stars in its first weekend.

What Autoresearch Actually Is

Autoresearch is not a framework or a library. It’s three files totaling around 630 lines of Python, plus a Markdown file that tells the AI agent what to do. The whole thing is built on top of Karpathy’s nanochat LLM training code, stripped down to run on a single GPU.

Here’s Karpathy’s own framing from the README:

Give an AI agent a small but real LLM training setup and let it experiment autonomously overnight. It modifies the code, trains for 5 minutes, checks if the result improved, keeps or discards, and repeats. You wake up in the morning to a log of experiments and (hopefully) a better model.

The key twist: you, the human, don’t touch the Python code. Instead, you write and refine program.md, a Markdown file that instructs the AI agent on what to try, what constraints to respect, and how to evaluate results. As Karpathy puts it: “you’re not touching any of the Python files like you normally would as a researcher. Instead, you are programming the program.md Markdown files that provide context to the AI agents.”

He also included a fictional prologue in the README that captures the spirit of the project:

One day, frontier AI research used to be done by meat computers in between eating, sleeping, having other fun, and synchronizing once in a while using sound wave interconnect in the ritual of “group meeting”. That era is long gone. Research is now entirely the domain of autonomous swarms of AI agents running across compute cluster megastructures in the skies.

The Three Files

1. prepare.py (Immutable)

This file downloads and preprocesses the training data. The agent cannot modify it. It handles:

Downloading the climbmix-400b-shuffle dataset (6,543 parquet shards from HuggingFace)
Training a BPE tokenizer using rustbpe with a GPT-4-style split pattern
Vocabulary: 8,192 tokens plus 4 special tokens
Contains the evaluate_bpb() function, which is the ground-truth evaluation metric
Contains make_dataloader() with BOS-aligned sequence packing for 100% token utilization

Fixed constants that can’t be changed: MAX_SEQ_LEN = 2048, TIME_BUDGET = 300 seconds (five minutes per experiment), EVAL_TOKENS = ~20.97M tokens.

The immutability of this file is a design decision. It prevents the agent from gaming the evaluation metric or the data pipeline to get artificially better numbers.

2. train.py (~630 lines, Agent Modifies This)

This is where the action happens. It contains a complete GPT model implementation, a custom optimizer, and a training loop. The agent reads it, modifies it, trains, and evaluates.

The model architecture:

RMS normalization (not LayerNorm)
Rotary position embeddings (RoPE)
Flash Attention 3 support
Sliding window attention with configurable patterns (SSSL = 3 short windows + 1 long)
Value embeddings with input-dependent gating (ResFormer pattern)
ReLU squared activation (not GELU)
Logit soft-capping at 15
Model size controlled by DEPTH (default 8) with ASPECT_RATIO = 64, so model_dim = depth × 64

The optimizer (MuonAdamW):

Muon for 2D weight matrices (using “polar express” orthogonalization)
AdamW for embeddings, unembedding, and scalar parameters
Separate learning rates per parameter group
Learning rate schedule with configurable warmup and warmdown

The training loop:

Fixed five-minute wall-clock budget (first 10 warmup steps excluded for compilation)
Gradient accumulation with ~524K tokens per batch
bfloat16 autocast
Fast-fail at loss > 100
Aggressive GC management (collects, freezes, then disables Python garbage collection after step 0)

Default configuration at depth=8 produces a 50.3M parameter model using about 45 GB of VRAM.

3. program.md (Human Modifies This)

This is the instruction file for the AI agent. It’s the bridge between human intent and autonomous execution. Here’s how it structures the agent’s work:

Setup phase:

Create a branch (autoresearch/<tag>)
Read all in-scope files
Verify data exists
Initialize a results.tsv log
Run baseline (unmodified train.py) first

The experiment loop (runs forever):

Check git state
Make a single change to train.py
Git commit the change
Run: uv run train.py > run.log 2>&1
Extract val_bpb and peak_vram_mb from the log
If val_bpb improved (lower is better): keep the commit, advance the branch
If equal or worse: git reset back to previous state
Log everything to results.tsv
Repeat

One instruction in particular stands out: “Once the experiment loop has begun, do NOT pause to ask the human if you should continue. Do NOT ask ‘should I keep going?’. The human might be asleep… You are autonomous.”

The TSV log tracks every experiment:

commit    val_bpb     memory_gb   status    description
a1b2c3d   0.997900    44.0        keep      baseline
b2c3d4e   0.993200    44.2        keep      increase LR to 0.04
c3d4e5f   1.001300    44.1        discard   decrease batch size

The Git Ratchet

This is one of the cleverest parts of the design. The branch only advances on improvement. Failed experiments get reverted via git reset. The result is a monotonically improving commit history. You can read the git log like a research paper, with each commit representing a confirmed positive finding.

The TSV log, which isn’t tracked by git, keeps the complete record of all experiments: keeps, discards, and crashes. So you get both the clean record of what worked and the full messy history of what was tried.

Why val_bpb, Not Loss

Autoresearch uses bits-per-byte (BPB) as its metric instead of raw cross-entropy loss. This matters because BPB is vocabulary-size-independent. If the agent decides to change the tokenizer vocab size from 8,192 to 4,096, the BPB numbers are still directly comparable. Raw loss would not be.

Results

Karpathy’s Own Runs

On a single GPU at depth=8 (baseline val_bpb of 0.997900): 83 experiments run, 15 kept as improvements. The agent found wins in batch size tuning, warmup/warmdown schedules, depth adjustments, and RoPE base frequency changes.

On 8xH100 production-scale runs with nanochat: 276 experiments, 29 kept improvements.

The depth-12 run from March 9 was the most impressive. Over roughly two days, the agent found about 20 changes that all reduced validation loss. When Karpathy tested them on larger depth-24 models, every single change transferred. This is the kind of result that matters. The agent wasn’t just hill-climbing on a specific configuration. It was finding genuine architectural and hyperparameter insights.

Tobi Lutke’s Results

Shopify CEO Tobi Lutke adapted autoresearch for an internal query-expansion model. After an 8-hour overnight run, the agent produced 37 experiments and a 19% improvement in validation score. The agent-optimized 0.8B parameter model outperformed the previous manually-tuned 1.6B model.

Karpathy’s response: “Who knew early singularity could be this fun?”

How to Use It

Requirements

An NVIDIA GPU (tested on H100, community forks exist for other hardware)
Python 3.x with uv package manager
An AI coding agent (Claude Code, Codex, or similar)
About $50-200 in API fees per 100 experiments

Setup

# Install uv
curl -LsSf https://astral.sh/uv/install.sh | sh

# Clone and install
git clone https://github.com/karpathy/autoresearch.git
cd autoresearch
uv sync

# Prepare data (~2 minutes)
uv run prepare.py

# Run baseline (~5 minutes)
uv run train.py

Launch the Agent

Point your AI coding agent at the repo and say something like: “Have a look at program.md and let’s kick off a new experiment!”

The agent reads program.md, understands the experimental protocol, runs the baseline, and starts iterating. You can watch it work or go to bed. Each experiment takes about five minutes of GPU time, plus whatever time the agent needs to analyze results and plan the next change.

At roughly 12 experiments per hour, an overnight run of 8-10 hours gets you around 100 experiments.

Adapting for Smaller Hardware

Karpathy provided specific recommendations for people without H100s:

Switch to the TinyStories dataset (lower entropy, better results with small models)
Decrease vocab_size (down to 4096, 2048, 1024, or even 256 for byte-level)
Lower MAX_SEQ_LEN to 256, increase DEVICE_BATCH_SIZE to compensate
Decrease EVAL_TOKENS for faster evaluation
Lower DEPTH from 8 to 4
Use WINDOW_PATTERN = "L" (full attention) instead of "SSSL"
Lower TOTAL_BATCH_SIZE to 2^14

Community forks have already appeared for macOS, Apple Silicon via MLX, and Windows with RTX GPUs.

What the Agent Actually Tries

This is the part that makes autoresearch genuinely interesting. The agent doesn’t just do random hyperparameter sweeps. It reads the code, understands the architecture, and proposes targeted changes.

Examples of experiments that improved validation loss:

Adjusting learning rate schedules for different parameter groups
Changing warmdown ratios and final learning rate fractions
Modifying the RoPE base frequency
Tuning gradient accumulation batch sizes
Experimenting with attention window patterns
Adjusting weight decay values
Modifying the MuonAdamW momentum ramp schedule

The agent also tries things that don’t work, and that’s fine. Failed experiments get reverted. The constraint that only train.py can be modified prevents the agent from doing anything too creative, like changing the evaluation metric to make its results look better.

One known failure mode: agents sometimes change the random seed (e.g., from 42 to 137) and report an “improvement.” The seed change might slightly shift validation results without any real architectural insight. This is something to watch for when reviewing the commit log.

Design Philosophy

Several choices in autoresearch deserve attention:

Fixed five-minute budget. Every experiment gets exactly the same training time. This makes results directly comparable regardless of what the agent changes. It also means autoresearch finds the optimal model for your specific hardware within that time budget. The tradeoff: results don’t transfer across different GPU types.

Single mutable file. Keeping the agent to one file means diffs stay reviewable, scope stays manageable, and the agent can’t game the system by modifying the evaluation harness.

Self-contained. No distributed training, no configuration files, no dependency injection. One GPU, one file, one metric.

Git as experiment tracker. Instead of building a custom experiment tracking system (MLflow, Weights & Biases, etc.), autoresearch uses git commits and a TSV file. Simple, portable, and transparent.

Where This Goes Next

Karpathy has already outlined his vision for the next step. From a March 8 tweet:

The next step for autoresearch is that it has to be asynchronously massively collaborative for agents (think: SETI@home style). The goal is not to emulate a single PhD student, it’s to emulate a research community of them.

Right now, autoresearch runs a single thread of experiments on one machine. Karpathy wants to parallelize it: multiple agents, multiple GPUs, sharing findings and building on each other’s work. He compared it to SETI@home, the distributed computing project that once used idle home computers to search for alien signals.

He also reframed the meta-question: “The real benchmark of interest is: ‘what is the research org agent code that produces improvements on nanochat the fastest?’ This is the new meta.” In other words, the competition isn’t just about writing better training code. It’s about writing better instructions for agents that write better training code.

The Lineage

Autoresearch didn’t come out of nowhere. It fits into a pattern of increasingly autonomous AI coding systems:

The “Ralph Wiggum Technique” (Geoffrey Huntley, mid-2025): A bash one-liner that ran while :; do cat PROMPT.md | claude-code; done, feeding a prompt file into an AI agent in an infinite loop. Crude, but it demonstrated the core idea.

Gas Town (Steve Yegge, January 2026): A system running 30 AI agents in parallel for software development work. More sophisticated orchestration, but general-purpose rather than research-focused.

Autoresearch (Karpathy, March 2026): Specialized for ML research with built-in evaluation rigor. The five-minute budget, immutable evaluation harness, and git-based ratchet give it the scientific discipline that earlier approaches lacked.

The program.md file itself was “90% AI written,” according to Karpathy. There’s something fitting about an AI agent following instructions that were mostly written by an AI.

Limitations

Autoresearch is a prototype, and Karpathy is upfront about its constraints:

NVIDIA-only in the official repo. You need a serious GPU, ideally an H100, though community forks are expanding hardware support.
API costs add up. Expect $50-200 in API fees per 100 experiments for the AI agent’s thinking time.
Single-threaded. One experiment at a time, roughly 12 per hour. This caps how much ground you can cover in a night.
Compute-specific results. The five-minute budget means findings are optimized for whatever GPU you’re running on. They may or may not transfer to different hardware.
Not production-ready. This is a research tool for research, not something you’d point at your production model training pipeline.
Agent quality matters. The results you get depend heavily on which AI agent you use and how good it is at reasoning about ML code.

Why It Matters

The most interesting thing about autoresearch isn’t the specific improvements it finds. It’s the shift in what the human does.

Traditionally, ML research looks like this: you have an idea, you modify the training code, you run the experiment, you analyze the results, you have another idea. The human is in the loop at every step.

With autoresearch, the human writes instructions and then leaves. The loop runs without them. The human’s job shifts from “doing research” to “writing better instructions for an agent that does research.” The skill being tested isn’t knowledge of optimization algorithms or network architectures. It’s the ability to communicate intent, constraints, and evaluation criteria clearly enough that an autonomous agent can execute productively for hours unsupervised.

That’s a different skill. And if Karpathy’s SETI@home vision plays out, the next step is writing instructions that coordinate hundreds of agents doing research in parallel. At that point, the program.md isn’t just a prompt. It’s more like a research agenda.

The repo is MIT-licensed and available at github.com/karpathy/autoresearch.