It's Not the Token Count: What Actually Makes or Breaks an AI Product

At the last AI Engineer conference, organizers expected roughly 50 companies to qualify as "token billionaires" - teams burning a billion tokens a week. The actual number, out of about 7,000 attendees, was closer to 300. That tells you something about how fast token spend is climbing. But in an interview at the same event, the founder of the conference and the Latent Space newsletter made a point that matters more than that headline number: sure, you're burning a billion tokens a day - but you'd better have real humans actually using what you built, because it's remarkably easy to generate a mountain of unusable slop and simply waste the money and the time.

Why burning tokens isn't the metric that matters

When tokens are practically free, it's easy to mistake activity for progress. Token volume says nothing about the quality of what gets produced. Most engineers default to hating waste and optimizing for efficiency - which is exactly why this shift feels counterintuitive: suddenly there's real value in letting an agent generate a lot of output, even knowing most of it will get thrown away, because every so often something genuinely useful shows up. The new skill isn't conserving tokens, it's filtering fast between what's worth keeping and what isn't. That's a reasonable bet to make - as long as you never lose sight of the question that actually matters: is anyone using this?

Loops that build loops

One idea that keeps resurfacing in conversations about working with AI agents is moving from writing a single prompt to writing a loop that generates prompts on its own. At the simplest level that looks like a goal that checks itself repeatedly until some condition is met. But the more interesting move is building loops that spawn further loops - working bottom-up (take anything you currently do manually, hand it to a single agent first, then turn it into a repeating process) and top-down (start from the highest-level loop you actually want running, and break it into sub-loops).

A few starting loops worth setting up on any new project:

• The "interview me" loop - flip the roles: ask the agent to interview you with questions instead of you writing it a prompt. People are bad at articulating exactly what they want - they don't think about the forks in the road until someone shows them the forks. An agent that behaves like a sharp employee, asking clarifying questions before starting work, is worth more than one that just executes blindly.
• The free research and brainstorm loop - have the agent scan competitors, propose new ideas weekly, and even prototype a few. Most of it gets discarded, but it's nearly free, and every so often a genuinely good idea surfaces.
• The close-the-loop feedback loop - wire production logs and errors back to the agent so it can read them and propose a fix before anyone files a bug report. That's the first real step toward a self-healing product.
• The goal-driven loop - instead of a one-off task, define a long-term objective (more sales, more time on site) and work toward it over months, almost like gradient descent: try something, measure it, take the next step.

What you never hand off

Two things stay firmly human, even once most of the code is being written by an agent.

Real user testing: models don't live inside your app every day the way you and your users do - they don't have the intuition for where a button should sit after a resize, or the unspoken expectation of an audit log, or a clear separation between orgs and teams. An agent can build those things, but only if someone specifies them explicitly, or keeps testing for them - they don't materialize on their own.

Understanding your own data structures: the biggest risk in a project built almost entirely by an agent is discovering, two months in, that you no longer actually understand what's being recorded, what gets reported, or what the interface can even show - and having to throw away a significant chunk of work. As long as you still deeply understand what data is collected, where it lives, and how often it's accessed, it's easy to keep building on top of it. Once you lose control of the data layer, it's very hard to claw back.

Taste - the edge you can't delegate

When anyone can generate 100 landing pages or 100 versions of a feature in minutes, judgment - deciding what's actually worth keeping - becomes the scarcest resource. It sounds strange, but one of the clearest signs of missing taste is simply lying: claiming your small model beats a frontier model when everyone in the room knows that's not true. People with taste tell the real story, even the less impressive version, instead of empty bragging.

Most important: check every feature yourself, by hand, before it ships - not just because automated tests passed, but to ask yourself whether this is what your users actually want, or just what the model felt like coding today. If you end up with 50 configuration options nobody asked for, that's a sign they were easy to write, not that they're worth having. Code has never been easier to generate - which is exactly why it's harder than ever to say no, and exactly why the people who still know how to say no stand out.

Mission, not just benchmarks

The same principle applies to how you talk about what you've built. Companies that only ever say "we're the best on the benchmarks" without sharing anything about the actual problem, or the insight that led to the solution, sound identical to every other competitor. The companies that manage to pull people in are the ones that connect what they're building to something bigger - not "buy our product," but "here's why this problem matters, and here's what we learned along the way." That's true for raising money, and it's just as true for any business building with AI that wants to earn trust: customers who understand that you actually care about their problem - not just that you can solve it fast - are the ones who stick around.

The takeaway

Inference speed will keep climbing - what looks blazingly fast today will be the baseline within a year. But that's exactly why what separates a good AI product from a bad one won't be how many tokens it burns. It'll be how much taste, human judgment, and real testing went into it: loops built with intent, a real understanding of the data underneath, and the willingness to say no to what's easy to build but not worth keeping.