What I Learned at Y Combinator's Harness Club About Why the Harness Matters More Than the Model

What I learned at Y Combinator's Harness Club about why the harness matters more than the model

I watched a recording of Y Combinator's Harness Club, a night dedicated entirely to agent harnesses, the scaffolding, tools, memory, and orchestration logic wrapped around a model, not the model itself. Harnesses get dismissed a lot as "just a wrapper" or "just prompt engineering." The talks made a strong case that this dismissal is wrong, and one statistic from the night made that case better than any argument could.

The stat that changed how I think about this

On ARC-AGI, a benchmark specifically designed to measure how quickly a system adapts to a genuinely new problem, Claude Opus scored 30 percent with an early, unoptimized harness. With better harness design alone, on the exact same model weights, that number went up to 95 percent. A separate system from Nvidia reached 100 percent. Same intelligence, completely different result, purely from how the surrounding system was built. That's not prompt engineering. That's a real lever.

A short sketch of how we got here

One of the speakers walked through the history, and it's worth condensing. Early language model harnesses in 2019 were barely harnesses at all: a system prompt, a loop, and a stop condition. Over the next few years, the field added capability layer by layer: few-shot examples, then chain-of-thought reasoning, then the ability to call external tools (Toolformer, WebGPT), then the ability for a model to read and write its own memory instead of just appending to a growing transcript (MemGPT), then the ability to chain tools into reusable skills and save them for later (the Voyager paper, built on Minecraft). After that came multi-agent patterns: a model reflecting on and correcting its own output, spawning persistent sub-agents it could message later, and eventually recursive setups where an agent can spin up more agents to solve pieces of a larger problem.

The newest and most interesting shift is harnesses that improve themselves. Instead of a fixed system prompt, some systems now search over candidate prompts and keep the better ones. The next step past that is a harness that can rewrite its own code, not just its prompt, effectively evolving its own architecture over time based on how well each version performs.

Treating context like a memory hierarchy

Seth, presenting Prime Agent (a self-improving harness built at Prime Intellect), framed this in a way I found genuinely useful: think of everything a model has access to like a memory hierarchy, similar to how a computer has caches. Model weights are the fastest and cheapest to access, but also the most expensive to change. Active context is next, and it's limited: once you run out of room, you need compaction to summarize history and keep going. Beyond that, a live code environment (a Python REPL the agent can read and write variables in) works like a layer below active context: it lets the agent hold and manipulate state without paying for every value in tokens. Sub-agents work similarly. You hand off a narrow task with just the context it needs, and it reports back a summary instead of flooding the parent's context with everything it did along the way.

The part I liked most: sub-agents in this system are persistent and addressable. A parent session can spin one up, let it finish and go idle, and message it again later with full context intact, rather than throwing away everything and starting a fresh session each time. And on evaluation: instead of just checking whether a model solved a task within some fixed budget, it's worth checking where performance actually plateaus as you give it more time and more tokens. Comparing two models at the same fixed budget can hide real differences in how each one uses extra time.

Running frontier-level agents fully on your own device

John presented Open Jarvis, a Stanford project built around a simple observation: local, on-device models are now only six to twelve months behind the frontier, not years. The pitch is a personal AI stack, model inference, agent logic, memory, and learning, that runs entirely on your own laptop or workstation instead of the cloud. The reasons that matters: no per-query API cost, no personal data leaving your machine, and far lower energy use.

The cleverest part of the design: rather than trying to make a small local model smarter at inference time, they use a large cloud model once, offline, to optimize the local harness itself, tuning the prompts, tools, and configuration for the specific model and task before anything runs locally. Cloud intelligence pays for the design work up front; the actual day-to-day inference stays fully local and cheap. Reported results: up to roughly 800x lower cost than running the equivalent workload through a cloud model, with lower latency too, and quality that's competitive with cloud stacks on everyday personal tasks, even if it still trails on the hardest ones.

What YC's own internal agent harness taught them

Josh and Reagan walked through QM, YC's internal agent harness, and the most useful part was the history of what didn't work before they landed on the current design. YC's first internal agent, built in January 2025, was a simple system prompt and a tool loop shared by everyone. It got noticeably better as the underlying models improved, with no other changes. Later, giving agents their own sandboxed computer (the same pattern used by tools like Claude Code or Codex) was a real capability jump, but running a fleet of dozens of these for every employee turned into constant manual firefighting, described as a "whack-a-mole" problem of individually managing instances.

QM's fix was to stop treating the sandbox as the agent's home. State now lives centrally, so a conversation and its context aren't trapped inside one machine, and a sandbox becomes a resource the agent reaches for on demand: a small one for a light task, a bigger one when the job needs more compute, freed up again once the task is done. The harness itself is kept deliberately thin: three core tools (running code in a remote sandbox, reading and writing object storage, and publishing small internal web apps), plus a handful of smaller ones for memory and scheduled jobs. The agent can also choose which model or provider handles a given task itself, which turns out to be a practical way to route around a specific model's refusals for a legitimate but sensitive task by switching providers.

Two problems they haven't fully solved yet are worth knowing before you build something similar. First, agents tend to give up too early on longer tasks, so they added an explicit budget, sometimes measured in wall-clock hours rather than token count, that the agent isn't allowed to quit before hitting. Second, and harder: agents don't naturally understand social or organizational context. A person intuitively knows which information is fine to share in which room. An agent doesn't, unless the permission system around it is built carefully enough to enforce that boundary for it.

Where I land

The common thread across all three talks: the model is not the product anymore, or at least not the whole product. What you build around it, how it manages its own memory and state, how it decides when to spin up help and when to stop, how cheaply and safely it can run, is where most of the real engineering work (and most of the real performance gains) actually happen now.