Why Fireworks AI's CTO Thinks the Future Is Thousands of Specialized Models, Not One Frontier Model

Why Fireworks AI's CTO thinks the future is thousands of specialized models, not one frontier model

I recently listened to an interview with Dmytro Dzhulgakov, co-founder and CTO of Fireworks AI (and a former PyTorch lead at Meta), about where open-source models actually stand right now. A few of his points changed how I think about when to reach for a frontier model versus when to fine-tune something smaller for a specific job.

The gap to frontier labs is closing fast

For a while, the rule of thumb was that open models trailed the frontier labs by something like three to six months on benchmarks. Dzhulgakov's read is that recent releases, GLM 5.2, Kimi K3, and the newer DeepSeek Flash and V4, represent a real capability jump, similar to the one frontier models went through around last November and December. That was the shift from copy-pasting code out of a chat window into real agentic development: agents running sub-agents, working in loops, pursuing goals over longer stretches without constant supervision. That level of capability has now arrived in open models too.

A short history of how we got here

The first real open-source moment, in his telling, was DeepSeek R1, followed by V3. What made R1 remarkable wasn't just the benchmark scores. It was the first time a lot of people, including plenty of non-technical users, could actually watch a model reason step by step. Closed reasoning models at the time hid their thinking traces entirely. Seeing one unfold in the open was genuinely new. Kimi K3 was, in his view, the second real hit, and DeepSeek V4 landed just two weeks after that.

Specialization, not one model for everything

The core thesis behind Fireworks AI is that a frontier lab's whole approach is to build one model that's the best at everything. Fireworks bets the opposite way: fine-tuning and reinforcement learning per use case, so that instead of one or two frontier models controlling everything, you end up with thousands, maybe millions, of specialized models, each tuned for a narrow niche. There's also a control angle to this that's easy to miss: running an open-weight model means you're not exposed to a vendor quietly changing behavior, logging your data, or adding refusals you didn't ask for. You can self-host, or run it wherever makes sense for you.

What drives someone to fine-tune

Dzhulgakov's answer was refreshingly concrete: usually quality first, cost second, though the order flips depending on the customer. The common path looks like this: a startup starts by wrapping a frontier model, gets real usage and real data, squeezes what it can out of prompting and harness engineering, and lands around 80 to 90 percent of the experience it actually wants. Fine-tuning is what closes that last gap to 95 or 99 percent, because at that point you're no longer just optimizing a prompt. You're optimizing the entire stack: your harness, your data, your actual usage patterns, baked directly into the model weights.

The real bottleneck isn't training. It's evals

This was the part I found most useful. The binary question that actually determines whether fine-tuning works for you isn't how sophisticated your training setup is. It's whether you've collected real data for your use case and whether you actually know what "good" looks like at scale. Companies that do this well invest heavily in evals, often before they touch fine-tuning at all, because the same evals tell you which off-the-shelf model to use, how to structure your harness, and where your product is failing. Skip that step and fine-tuning just optimizes toward the wrong target.

Quality beats quantity, by a lot

One detail worth remembering: post-training today behaves very differently from old-style pretraining. Pretraining a model from scratch needs millions or billions of examples. Fine-tuning a model that already has strong general intelligence for a narrow use case can meaningfully improve with a few hundred genuinely high-quality examples, sometimes fewer, because you're not teaching it to reason from nothing. You're teaching it something specific on top of intelligence that's already there. A hundred excellent examples reliably beats ten thousand mediocre ones.

Pretraining from scratch is a shrinking club

It's tempting to think that once a company has enough proprietary data, the natural next step is training its own model from the ground up. Dzhulgakov's take is that this path is actually getting rarer, not more common. Back in 2023, plenty of companies tried full pretraining for a specific vertical (Bloomberg's own finance-specific model, trained heavily on its own data, was the best-known example), and most of those efforts were quickly outperformed by generic frontier models that were fine-tuned afterward instead. Pretraining benefits so heavily from aggressive scale that only a small number of players can stay competitive doing it. The more stable setup: a handful of labs pretrain, and everyone else specializes on top through post-training.

Development itself is moving to the cloud

There's a broader shift here too, and it maps onto something familiar: the same way companies moved off their own servers once they needed more capacity than one machine could give them, coding agents are moving the same direction. Once a human isn't sitting in front of the output waiting in real time, latency stops being the deciding factor, and running agents locally stops making sense past a certain scale. There's a very concrete ceiling here: local agent tooling can start struggling once you're running around thirty agents in parallel on a single machine, well before you'd want it to.

Becoming the tech lead of a hundred agents

The mental model that stuck with me most: as agentic coding tools spread, more people are effectively becoming the tech lead of a large team instead of the engineer typing the code themselves. Actually writing code by hand is close to zero at that point. What matters instead is setting up the right environment, the right tests, the right guardrails, so a fleet of agents can be trusted to move in the right direction without a human checking every line. Code review still matters a lot, especially for complex, tightly integrated systems, but for a well-scoped, clearly described feature, you often don't need to look at the code at all.

Where I land

None of this is really about chasing whichever model tops a leaderboard this month. It's about matching the tool to the job: reach for a frontier model when you're still exploring, and once you have real usage and a real way to measure quality, specialize. The gap between open and closed models is closing fast enough that betting everything on one frontier model, instead of building the evals and data discipline to specialize when it counts, is starting to look like the more expensive choice.