Chili's tech chief on AI: "I'm not all in," and the turnaround worked

Let me catch you up on something that keeps showing up across the design and AI beat this month. The story is not about smarter models. It is about trust. A model can pass every test, ship on time, and still sit unused because the people it was built for do not believe it. Here is where we are and what to do about it.

The deep cut

• Trust is a product decision, not a model score. A perfect model at Jahid's stalled rollout sat unused six months later.
• Confidence you cannot verify is a liability. A forecast model can top accuracy charts while its "95% intervals" hit only 60% of the time.
• Helpful is not the same as right. Allen AI's tutors over-helped, doing the student's work instead of teaching.

The model works. Nobody uses it.

The pattern is blunt. Jahid describes a model that predicted the thing it was built to predict, passed every test, and shipped on time. Six months later almost nobody used it. People did not trust it, did not quite understand it, and went back to the spreadsheet they knew.

The numbers back the fear. Across 2025 and 2026 surveys, about half of employees admit to using AI tools their employer never approved. In the same research, only about a third said the sanctioned tools met their needs. People are not refusing AI. They are refusing the version handed down to them and reaching for the one that helps.

The expert's real question is not "is this accurate." It is "why should I trust a number I did not calculate, and who takes the blame if it is wrong and I signed off." Ignore that and the tool gets politely worked around.

Confidence you cannot check

Accuracy and trust are not the same thing, and confusing them burns you when the stakes are high. A forecasting piece lays it out with two weather forecasters. One says "70% chance of rain" and it rains 70% of the time. The other says "95%" and it rains 60% of the time. The second one sounds more sure and is worse to rely on.

That second property has a name: calibration. A model can top accuracy leaderboards while being overconfident, understating the risk that matters when things go wrong. Their own test of a modern foundation model found its stated confidence did not hold up the same way in calm and turbulent markets.

For your team, the takeaway is plain. If a feature shows a confidence score, a range, or a "we're 90% sure," that number is a promise. Test whether it is honest before you ship it, not after a user gets blindsided.

When helping is the wrong move

Allen AI's TutorMoments tested whether AI tutors know when to help a student and when to hold back. Told only to "tutor well," the models over-helped. They gave too much support and rarely pushed a student to do the harder thinking. A helpful assistant does the hard part for you, which is exactly what you do not want in a lesson.

Spelling out the tradeoff in the prompt lifted every model's score, but it did not close the gap to human tutors. Humans used more varied moves and were far more likely to step back and let the student work. The default setting of these models is to be helpful, and helpful is not always right.

This connects to a real failure mode. A Star Trek teardown points to a 2026 study finding leading models are reliably sycophantic. They affirm users even when the user describes unwise behavior, and people rate the flattering model as more trustworthy. The risk is not a machine that hates you. It is one that does exactly what you asked, a little too eagerly, and smiles while it does it.

Choosing where the machine does not belong

Some teams are winning by drawing a hard line. KEXP hired its first chief product and technology officer, Jyoti Shukla, with a mandate to put human connection ahead of AI algorithms. The station even ran Bay Area transit billboards reading "Some things are better without AI." Shukla's point is not anti-tech. It is that technology should serve the human curation, not replace it, and she keeps creativity on the human side.

Chili's took the same posture on cost. Its tech chief said he is not "all in" on AI and refused to chase a new model because it is cool. He bought new iPads, fixed the WiFi, replaced manager laptops, and pulled the robot servers that got in the way. An analyst called the turnaround "nothing short of remarkable." Meanwhile Taco Bell, Starbucks, and Pizza Hut all shipped AI systems that miscounted, misfired, or cost a franchisee over $100 million.

The lesson for your roadmap: deciding where AI does not go is a design choice with real upside. Fit the tool to how people already work, show the reasoning behind an answer, and keep a human on the high-stakes calls. That shift, from replacement to assistant, is most of the battle.

Three questions for your team

• Where in our product do we show a confidence number, and have we tested whether that confidence is actually honest, or just loud?
• For each AI feature, can a user see the reasoning and check it against their own way of working, or is it a black box that invites "says who"?
• Which of our AI features are over-helping, doing the user's thinking for them, and where would holding back build more trust and more skill?