There's no better test of a product than using it yourself. That's why Conviva runs Agent Experience Insights (AEI) on Nexa, the AI analyst our customers rely on every day.
As adoption of Nexa has grown, it's become a core part of how our customers analyze the performance of their own digital experiences, which means data accuracy and response quality have to be incredibly high. To ensure that, we can't rely on direct feedback alone; we must consider the implicit signals Nexa users produce every day. And recently, those signals showed us that users were questioning the agent's answers.
At a glance
AEI found that Nexa was giving confident but wrong answers to questions about a specific country, customer, or device. Nexa's infrastructure, error logs, and LLM judge all passed, because the failure left no trace in any of them. The Nexa team caught it anyway, by using AEI to watch how users responded.
- Problem: When a question was scoped to a country, customer, or device, Nexa sometimes wrote the filter in a format the data service didn't accept. Instead of rejecting the request, the service ignored the filter and returned totals for all traffic. Nexa presented that total as the filtered answer.
- Why it was hidden: Infrastructure health, HTTP and tool status, and an LLM judge all rated the affected sessions as successful.
- How it was found: AEI showed users correcting Nexa, repeating questions, and asking for redos.
- Fix: A deterministic check now validates every filter before Nexa requests data. In its first week, it caught every malformed filter before it reached the data service, and filter-format warnings fell 91%.
- Result: All four AEI metrics improved, by 15% to 41%, in the first week after the fix.
The challenge: confident answers to the wrong question
In real sessions, users asked Nexa questions like how many viewers watched in Canada last week. Nexa returned a clean, specific number in a well-formatted answer, and nothing about it looked off. But the number wasn't just covering viewers in Canada. It included every viewer, in every location.
To answer a question like that, Nexa writes a query to Conviva's time-state data service, including a filter for the country, customer, or device the user asked about. Sometimes Nexa wrote that filter in a format the data service didn't recognize. Rather than rejecting the request, the service skipped the filter and returned totals for all traffic. Nexa, believing it had asked for Canada, presented the worldwide total as the Canadian one. The same thing happened with filters like iOS.
This is the failure mode AI agent teams worry about most. The agent didn't crash, stall, or produce gibberish. It fluently answered a different question than the one it was asked, and anyone who trusted the number would have made a decision on bad data.
Why didn't system checks or an LLM judge catch it?
Every standard check passed because each one measured whether Nexa ran, not whether it worked for the user. Infrastructure health showed 0% errors, with just 0.1% of data requests failing. HTTP and tool status reported success on every request, because the data service technically did succeed: it simply ignored the filter while doing so. Even an LLM judge, the automated evaluator many AI teams rely on to grade agent output, rated the affected sessions as successful.
That last point matters most. An LLM judge grades the reply, and this reply looked right. It was clear, confident, and full of believable numbers. The judge had no way of knowing the data underneath it ignored what the user asked for. Nothing in the trace was broken, so nothing that only reads the trace could see the problem.
The evidence lived somewhere else entirely: in what users did after they read the answer.
How AEI found it: user QoE as the judge
AEI flagged the problem through user QoE (quality of experience): the implicit signals in how people respond to an agent, read across the full population of sessions rather than a sample. No system alerted, and no one on the team spotted it first. AEI did, because users were pushing back in the conversation, telling Nexa that the number it gave was for everyone, not for the country or device they had asked about.
In the week before the fix shipped, four user QoE signals were elevated at the same time.
- The repeated-question rate, where a user asks the same thing again, was 10.3%.
- The redo request rate, where a user asks Nexa to try again, was 6.2%.
- The correction rate, where a user tells Nexa its answer is wrong, was 6.0%.
- The self-contradiction rate, where Nexa reverses itself after pushback with some version of "you're right, I was wrong," was 5.5%.
Any one of those signals on its own can be noise. All four rising together is a pattern, and the pattern said users weren't getting what they came for. That's the difference between LLM-as-a-judge and user-as-a-judge. The LLM judge scored the reply. The users scored the experience.
Diagnosing the root cause with a coding agent over MCP
Once AEI showed where users were struggling, the Nexa team needed to know why. The team connected Claude Code to AEI over MCP and pointed it at the flagged sessions. Working from the session traces and Nexa's codebase together, it found the common thread: malformed filters were reaching the data service, which ignored them and returned unfiltered totals as a normal success. The team then replayed the requests and confirmed the root cause.
This is how Conviva expects agent teams to work with AEI, which is why we're already doing it ourselves. A team's own coding agent brings the context of the codebase, AEI brings full-population evidence of what users actually experienced, and together they produce an answer neither could reach alone.
The fix: a small check in the harness
The fix wasn't a bigger model or a rewritten prompt. It was a deterministic check in the agent's harness. After Nexa writes a query, a validator now checks the format of every filter before any data is requested. If a filter is malformed, the request goes back to Nexa with instructions to rewrite it, and Nexa tries again. The check is simple, and that's the point. In its first week, it stopped 871 bad requests across 42 customer accounts, each of which would otherwise have produced a confident answer to the wrong question.
Results: proving the fix from the user's side
The team validated the fix against the same user QoE signals that surfaced the problem, comparing the week before the fix shipped with the first week after. All four moved in the right direction:
- Repeated-question rate fell 41%, from 10.3% to 6.1%.
- Self-contradiction rate fell 40%, from 5.5% to 3.3%.
- Redo request rate fell 24%, from 6.2% to 4.7%.
- Correction rate fell 15%, from 6.0% to 5.1%.
These signals don't drop to zero, and they shouldn't be expected to. Like any agent in production, Nexa still has other issues to work through. But this particular failure is fixed, and the team can show it in the measure that matters most: how users responded.
Frequently asked questions
What is agent analytics?
Agent analytics is the practice of measuring how AI agents behave and whether they work for the people using them, across the full user population in real time. It goes beyond whether an agent ran or passed an eval to what users did next. Conviva's Agent Experience Insights (AEI) applies agent analytics to consumer-facing and internal AI agents.
Why can't an LLM judge catch every AI agent failure?
An LLM judge grades the agent's reply, so it misses failures where the reply looks correct but rests on bad data or a misread request. In Conviva's case, an LLM judge rated sessions as successful even though Nexa had answered the wrong question. User behavior exposed the gap, which is why agent analytics pairs evals with a user-as-a-judge signal.
What are user QoE signals for AI agents?
User QoE (quality of experience) signals are implicit behaviors that show whether an agent worked for the user, such as correction rate, repeated-question rate, redo requests, and agent self-contradiction. Read across every session, they reveal failures no system metric shows. After Conviva fixed a filter issue in Nexa, its repeated-question rate fell 41%.
What is closed-loop agent improvement?
Closed-loop agent improvement is a cycle where the user signal that reveals an AI agent problem is also used to verify the fix. Teams detect the issue from production behavior, diagnose it, ship a change, and measure the same signals again. Conviva used AEI this way to find, fix, and prove an improvement in its Nexa analytics agent.
Can AEI connect to coding agents like Claude Code?
Yes. AEI is reachable over MCP, so a team's own coding agent can query agent experience data directly. Conviva's Nexa team connected Claude Code to AEI, had it read the traces behind flagged sessions, and found a root cause without opening a dashboard, combining codebase context with full-population evidence.
