The demo is the easy part

Building an AI feature that demos well takes an afternoon now. You wire a model to a prompt, feed it a good question, and it answers beautifully. Everyone in the room nods. The hard part starts the moment a real person asks a real question you didn’t rehearse.

Because then it does one of the things that never happens in a demo. It answers confidently and wrongly. Or it surfaces information the person asking wasn’t supposed to see. Or it works fine and saves nobody any time, because it doesn’t actually fit the job it was dropped into. The gap between the demo and the daily tool is where most of these projects quietly die.

Grounding beats cleverness

The first thing we do is stop trusting the model’s memory. A model’s training data is not your data, and the confident paragraph it writes about your refund policy is a guess. So we ground it: retrieval against your actual documents and systems, with citations, so an answer points to where it came from. If it can’t find a source, it should say so rather than improvise.

Grounding also forces the permission question, which is the one people skip. The AI should see exactly what the person using it is allowed to see, through the same permission model as the rest of your product. An assistant that can read every customer’s record regardless of who’s asking isn’t a feature, it’s an incident waiting for a date.

”It feels good” is not a metric

The part teams resist most is evaluation, because it’s less fun than the demo. But “it feels good” is not something you can ship against. So we build a test set out of real cases, the messy ones and the edge ones, and we measure accuracy on it. Now “is it working” has a number, and “did that change make it better or worse” has an answer instead of an argument.

This is also how you know when to stop. Models don’t have to be perfect to be useful, but you need to know where the floor is and what the system does when it hits it. A wrong answer with a graceful fallback is fine. A wrong answer delivered with total confidence and no guardrail is the thing that ends up in a screenshot on social media.

Build for the second month

The test that matters isn’t the launch. It’s whether the feature is still saving someone time a quarter later, after the novelty has worn off and it’s just part of the furniture. Most AI demos can’t pass that test, because they were built to impress a room, not to survive a Tuesday.

The voice concierge on this site is a small version of how we think about it. It’s grounded in what the studio actually does, it knows what it doesn’t know, and it’s measured against the questions people really ask. That’s the bar. Everything before it is just a demo.