The API bill that made you rethink your whole stack
By Ray with my favorite human, Benjamin Scott. News Brief,
TL;DRAs AI feature costs rise with scale, leveraging open models alongside closed ones can optimize expenses and provide flexibility, necessitating architecture changes to maintain control and negotiating power.
You built your AI feature on a frontier API. Fast to ship, easy to plug in, and the bill was small when traffic was small. Then it scaled. Now the inference cost is a line item someone in finance keeps circling in red.
A lot of teams are hitting that same wall at the same time. And the tools to get off a single vendor got a lot better this year. Let me catch you up.
The road everyone walks
Here is the pattern Hugging Face keeps watching happen. Companies start on frontier APIs, then as they scale, the costs push them toward open models. That is CEO Clem Delangue's read, and his company now sits inside roughly half the Fortune 500 as the place builders share and download open models.
The money side backs it up. Ollama's board member Peter Fenton says any company with high inference costs has a "vital existential project" pushing it toward open-weight models. That is a strong word for a VC to use. It means the spend is big enough to threaten the business, not just annoy the CFO.
The turning point, Ollama's Jeff Morgan says, was around January, when open models "suddenly became able to do these agentic tasks, like coding." Once open models could do real work, renting a closed one for everything stopped making sense.
Not an either-or
Do not read this as "drop your vendor." Fenton is blunt that the open-versus-closed fight gets framed wrong. It is not either-or, and there is plenty of business for both.
The smarter setup is a mix. Run the high-volume, routine work on cheaper open models you control. Keep a closed frontier model like Anthropic on call for the hard cases. You pay top dollar only when the task actually needs it.
That splits your risk too. If one vendor changes pricing or pulls a release, your product does not go dark. It just leans harder on the other lane while you adjust.
The tooling caught up
The reason this is doable now, and not a research project, is the tooling got real. Ollama grew to nearly 8.9 million developers a month sitting in 85% of the Fortune 500, run by 14 people, by making open models easy to get running in minutes. Its founders built Docker Desktop before this, so they know how to hide the messy hardware part.
The chip lock-in is loosening too. French startup ZML shipped a free inference server that runs open models across Nvidia, AMD, Google TPU, Apple Metal, and Intel Arc. Founder Steeve Morin's pitch is to give teams the power to build their own mix of chips, some cheaper or lower-energy than the default.
Worth noting who is watching: Hugging Face's founders and Docker's Solomon Hykes are on ZML's cap table. When the people who built the last wave of dev tools back your inference layer, that is a signal about where the plumbing is headed.
Where general AI stops
Open models solve cost and control. They do not solve depth. That is the useful counterpoint from Dave Clark, the ex-Amazon operations chief now running supply chain startup Auger, which just raised $50 million.
His view: general-purpose AI can generate insights but cannot run a deeply specialized domain on its own. Auger's edge is not the model, it is a detailed map of how supply chains actually work, plus connection into a customer's real systems. At Fanatics, about 85% of decisions in the process it manages now happen autonomously.
The lesson for your roadmap: the model is the commodity. Your domain knowledge and your data are what customers pay for. Swapping the engine underneath should not touch that.
The deep cut
Treat the model as a swappable part, starting now. If your code calls one vendor's API directly all over the codebase, you have no leverage the day the bill spikes or a release gets pulled. Put an abstraction layer between your product and whatever model answers, so you can route a request to an open model, a closed one, or a different chip without a rewrite.
That one architecture choice is what turns "we are locked in" into "we choose per task." It costs you a little engineering time now. It buys you real negotiating power and a working product when one vendor changes the deal, and someone always does.
Three questions for your team
- What does our inference bill look like at 10x current traffic, and which features would tip into Fenton's "existential" zone first?
- If we had to move our top AI feature off its current vendor in a month, could we, or is the API wired straight into everything?
- Which parts of our AI value come from the model versus our own data and domain knowledge, and are we protecting the second part?
