A shopper is using a brand’s AI agent to find wide-fit running shoes to train for a marathon. The agent recommends three pairs, answers every follow-up with confidence, and closes the conversation. Score that transcript on helpfulness, relevance, and clarifying-question quality and it comes back a fraction under perfect.
None of the three pairs came in a wide fit. The agent’s catalog search never passed the width parameter. The shopper didn’t complain. She opened a new tab, searched the site herself, and left without buying anything.
The transcript reads like a win. Nobody on the agent team can see the part that wasn’t.
That is the gap in how this industry evaluates agents, and it’s why we built a framework to close it.
Evals Are Necessary. They Are Not Proof.
Let me concede the obvious first, because it matters. Evals work. Tracing works. LLM-as-a-judge scoring is real engineering, it catches real regressions, and if you’re shipping an agent without it you’re being reckless. I’m not here to tell you to rip out your eval stack.
I’m here to tell you what it can’t do.
An eval score measures quality of response. It tells you whether a single reply was accurate, grounded, and on-topic against a rubric somebody wrote in advance. That’s a necessary bar. It is not the same question as whether the person on the other end got what they came for — and right now most of the industry is answering the first one while calling it the second.
You cannot close that gap by reading harder, either. There is nothing wrong with the shoe shopper’s transcript. The agent was polite, on-topic, and articulate the whole way through. The failure isn’t in what the agent said. It’s a dropped parameter on a tool call nobody was watching, and it only becomes visible when you line that conversation up against every other conversation like it and notice one segment of shoppers quietly leaving right after they ask about fit.
That’s not a trace problem. That’s a population problem. You will never find a population problem by improving your rubric.
EPIC: Four Parts, All Four Required
We named the methodology EPIC: Experience, Population, In Production, Contrastive. Don’t just score traces. Contrast experience in production. Miss any one of the four and you’re not measuring quality of experience. You’re measuring output and hoping.
Experience: not just the trace
Most evals grade the reply. They don’t grade what came before it, what happened between turns, or what the person did once the conversation ended, and they certainly don’t grade the sequence and timing of all of it strung together.
Two users can ask an agent the identical question, get the identical answer, and be having two completely different experiences. One adds the item to a cart. The other goes quiet, opens a new tab, and searches the site manually mid-conversation, a customer doing the agent’s job for it in real time. A trace-level score cannot tell those two people apart. It reads the same reply and gives the same grade.
Experience means reading the whole arc: before the chat opened, the gaps between turns, and the moment after it closed.
Population: not a sample
The shoe shopper isn’t a failure. She’s the visible edge of a pattern that only exists once you stop reading conversations one at a time.
On its own, her transcript looks fine. Line it up against every other shopper who asked about width, fit, or sizing and the pattern surfaces: a whole segment leaving after the same kind of question, for the same reason, every time.
We track this with named signals built for exactly this. Clarification Ratio: how often customers have to restate themselves. Scope-Narrowing Rate: how often they settle for less than they asked for. Post-Conversation Correction Rate: how often they go back and fix the record after the chat already closed. None of those numbers mean anything at the level of one conversation. They only mean something at the level of everyone.
In Production: not in a lab
A curated eval set is built to make an agent look good. Real customers are not.
Here’s the math that explains why demos lie. An agent that is 95% reliable at every single step is 77% reliable across five steps, 60% across ten, and 36% across twenty. Same agent. Same model. The demo was three steps. Production is twenty-five.
Reliability lives at the trajectory level, not the step level, and it only shows up when real people with actual money and actual patience start asking things you didn’t anticipate in an order you didn’t design for.
Contrastive: not an isolated score
Knowing a failure exists somewhere in your population is not the same as knowing where.
Picture a product documentation assistant scoring between 0.96 and 0.98 on factuality and groundedness, as strong as those numbers get, while a tool bug silently drops a tier parameter and hands every customer the wrong edition’s docs. The scores say excellent. The customers are getting answers that are wrong for them, every time, and no aggregate number will ever tell you that.
You find it by slicing the population against every dimension available; sentiment, intent, product, tool call, timing, the specific parameter that got dropped, until the cohort that’s failing separates itself from everyone else. That’s the difference between knowing something is wrong and knowing enough to fix it.
Put the four together and you get the actual test. A score tells you if your trace passed. Experience tells you if your agent works for the user, and how well.
The Two Bad Options, and the Third One
Today teams get offered a choice between two things that don’t work.
LLM-as-a-judge scales, but it isn’t grounded in whether anything worked for the person on the other end. A model grading another model’s homework is still a model’s opinion. Human review is grounded, and you cannot read your way through a million conversations a day.
There’s a third way, and it isn’t a compromise between those two. It’s letting the user do the grading, reading what every customer actually did, at full population scale, automatically. Your evals grade the agent. We let the user grade it.
And here’s the part that surprises people: doing this well saves money. Every clarification the agent didn’t need to ask, every loop it didn’t need to run, every conversation that took two turns instead of six, that’s a better experience and fewer tokens burned at the same time. Inefficiency and wasted spend are the same problem viewed from two angles. Better root cause work, lower token spend. You don’t have to trade one for the other.
We’ve catalogued the specific ways agents fail while looking successful: missing context the agent should have had, ambiguous tool instructions, no clarifying question asked when one was needed, a default assumed instead of confirmed, no check against the original ask, retrieval that came back stale or partial. Every one of them passes today’s controls. Every one of them is invisible in a single trace and obvious the moment you look at the population.
I’ve Watched This Movie Before
I’ve spent two decades measuring live digital experiences, 10 Olympic Games, 20 Super Bowls, 5 FIFA World Cup tournaments. Video had this exact blind spot fifteen years ago.
The industry measured quality of service. Did the stream start? Did the app load? Did it play? Every one of those metrics could come back green while a viewer sat there watching a frozen screen during a penalty shootout and never came back to the service again. It took the industry years to accept that “it played” and “it was a good experience” are different questions, and that only the second one predicts whether the customer stays.
Same insight. Same playbook. A market that is exponentially bigger.
What I’d Ask Every Team Building Agents Right Now
Are you evaluating at the session level or the population level?
Has your agent ever scored well on sentiment and still failed to convert, resolve, or retain the person it was talking to?
How much of your evaluation runs on synthetic test sets instead of real conversations?
And the one that matters most: do you have as much visibility into what the user experienced as you have into what your agent’s traces and tool calls say it did?
If the honest answer to that last one is no, you’re not measuring quality of experience yet. You’re measuring quality of service and calling it something else.
EPIC is the framework. Agent Experience Insights is what we built on top of it. If you’re running a customer-facing agent in production and you can’t currently prove it worked for the people using it, that’s the conversation I want to have.