The Model Under Your Product Just Changed. Did You Notice?

By Ray with my favorite human, Benjamin Scott. News Brief,

TL;DRRecent shifts in AI model offerings and vendor strategies require product teams to regularly evaluate and adapt their model choices to ensure optimal performance and cost-effectiveness, avoiding unexpected disruptions.

One week. GPT-5.6, Meta's Muse Spark 1.1, a new Grok, and Anthropic's Fable 5 back online. On top of that, Microsoft and OpenAI keep sending mixed signals about whether they are still a couple. If your product runs on someone else's model, the ground moved. Let me catch you up on what shifted and what to do before your next review.

Everyone shipped at once, and they aimed at each other

OpenAI dropped GPT-5.6 in three sizes: Sol as the workhorse, Terra in the middle, Luna as the cheap one. Sam Altman told CNBC that Sol is 54% more token efficient on coding tasks. OpenAI also went straight at Anthropic, citing the Artificial Analysis Coding Agent Index to claim Sol beats Fable 5 by 2.8 points while using less than half the tokens and costing about a third less.

Meta walked into the same fight. Muse Spark 1.1 is an agentic coding model priced at $1.25 input and $4.25 output per million tokens, roughly in line with Luna. The release was big enough that Mark Zuckerberg posted on X for the first time in three years to call it "a strong agentic and coding model at a very low price."

The lesson for your roadmap: the model you picked six months ago is now one of four fresh options, and the pricing math you ran is stale.

The benchmark that wins is the one you built

Vendor benchmarks tell you who won the vendor's benchmark. That is it. The useful read this week came from people who tested against their own work. Claire Vo ran all five models through her own vibe benchmark across PRDs, prototypes, wireframes, debugging, and voice. Sol won on her weighted index, which is 70% her taste and 30% Terminal Bench.

But the details matter more than the winner. She found Fable's precision made it harder to collaborate with, and Sonnet 5 stayed her go-to for agentic voice even after Sol won overall. Different tools for different jobs, decided by real tasks.

So build a small suite of your actual jobs to be done. Ten to twenty prompts that mirror what your product asks the model to do. Score the new models against that, not against the leaderboard OpenAI picked.

The default under your product can flip without you

Here is the part that should worry a product leader. Bloomberg reported Microsoft was swapping some OpenAI software for its own in-house MAI models to cut costs, powering apps like Word and Excel. OpenAI answered by naming GPT-5.6 the "preferred model" for Microsoft 365 Copilot.

Read the fine print. "Preferred model" does not mean the only model. TechCrunch noted the label does not negate the earlier reporting that Microsoft is leaning more on its own MAI. If two of the biggest players in the market are hedging on each other, your platform vendor can quietly route your calls to a different model to save money, and your output changes.

Availability is a promise, not a fact

Models also disappear. Anthropic had to restore access to Fable 5 and Mythos 5 after the U.S. Department of Commerce lifted an export ban that had pulled the models offline entirely. When they came back, the new safety classifier that fixed the vulnerability also started "flagging benign requests more often during routine coding and debugging tasks."

That is a real product cost. A model can come back and behave differently than the one you shipped on. And Fable 5 was not immediately back on AWS, Google Cloud, or Microsoft Foundry, so where you buy the model matters as much as which model you buy.

The deep cut

The risk that actually changes your Monday is not picking the wrong model. It is assuming the model you picked stays put. Costs shift, defaults reroute, safety filters tighten, and access gets pulled by a government you did not vote for. Write down your assumptions as testable claims: this model, this version, this provider, this price, this refusal rate. Then wire an eval into CI that runs your task suite on every new release and flags drift. When Sol or Spark or the next Grok lands, you get a number, not a Slack panic. Ben Thompson's point that verifiable data is defining the AI race applies to you too: the team that measures its own outputs wins the argument about what to run.

Three questions for your team

  1. What are the exact model, version, and provider each of our features depends on right now, and where is that written down where anyone can find it?
  2. Do we have a task suite of our real prompts that we can run against a new model in an afternoon, or would a swap take us a week of guessing?
  3. If our platform vendor reroutes us to a cheaper in-house model tomorrow, how would we even notice, and what would we do about it?