.webp)
AI agents can pass every evaluation and still fail the customer. A new IDC Spotlight, The AI Agent Performance "Blind Spot": Customer Experience, examines why. Most teams measure AI agents with metrics like accuracy, latency, task completion, and cost. As the IDC Spotlight puts it, "What is lacking is any direct measure of the customer's actual experience during the interaction." Closing that gap means measuring agent experience from what customers do during and after the interaction, not from surveys or thumbs-up ratings.
The blind spot is the gap between what teams can observe about an AI agent and what customers actually experience. Evaluation scores show whether a response was correct. They don't show whether the customer got what they came for.
According to the IDC Spotlight, "Without a complete picture, when it comes to customer experience, the current state of agent performance evaluations is at best inferred, and at worst a guess."
The usual proxies fall short. Few customers leave a rating, and a "resolved" status only means the conversation ended.
Put together, the picture is clear: teams are spending more on agents than they planned, they're optimizing for efficiency, and they can't give agents more autonomy because they can't see how customers are being served.
The IDC Spotlight identifies six metrics:
None of these signals come from a survey or a thumbs-up. They come from what customers actually do, which is why they show where an agent is creating friction even when every response looks correct.
Poor agent experience and wasted token spend come from the same source. According to the IDC Spotlight, "Experience inefficiency and token waste are the same problem viewed two ways." Each turn a customer didn't need adds to the bill and takes them further from what they came for.
A spike in token spend can look like an agent working harder when it's actually failing. The IDC Spotlight concludes that "High token spend with poor customer experience is pure waste" and that "The fastest way to cut agent spend is to eliminate the turns customers never needed."
That's also where the opportunity is. When teams can see which turns customers didn't need, they can cut spend and improve the experience at the same time.
The IDC Spotlight outlines four steps for closing the blind spot:
For 20 years, Conviva has measured quality of experience for the moments millions of people share, from live sports streams to the agents customers now talk to every day.
Agent Experience Insights measures the human side of every agent conversation, including behavior on the surrounding website or app that never appears in the transcript, such as manual searches mid-conversation, purchases made outside the chat, downstream rework, and return visits about the same issue.
Conviva's approach is built on the EPIC framework:
IDC Spotlight, The AI Agent Performance "Blind Spot": Customer Experience, November 2026, IDC #US55007226.
Measure what customers do, not just what the agent says. Start with the six behavioral metrics the IDC Spotlight identifies, listed above. Track them across every session in production, then compare customer segments to find where the experience breaks down. Conviva's Agent Experience Insights computes these signals inside and outside the conversation.
Evaluations measure whether a response was correct, not whether the customer's problem was solved. Most evaluations, including LLM-as-judge, only see the transcript. A polite, accurate answer can still leave a customer repeating themselves or giving up, and none of that shows up in an evaluation score.
The IDC Spotlight defines it this way: "Agent experience is the customer's perceived outcome and efficiency of an agent interaction: whether they got what they came for, how efficiently, and whether they came away satisfied. It is derived from behavioral signals rather than self-reported feedback." Agent experience is the human side of AI agent analytics, and it's what Conviva measures.
AI agents waste tokens on turns the customer never needed: repeated clarifications, retries, corrections, and time spent gathering context. When the conversation still ends with the customer giving up, all of that spend produced nothing. Tying token spend to experience metrics shows which spending is waste.
AI agent evaluations test accuracy, task completion, latency, and cost, usually before an agent ships or on sampled traffic. Agent experience measurement looks at how real customers experienced every interaction in production, using behavior during and after the conversation. Teams need both: an inside-out view of performance and an outside-in view of the customer.