.webp)
Most e-commerce agents fall into one of three buckets: customer service agents, shopping assistants, and third-party agents acting on shoppers' behalves. As customer interactions are increasingly passed to these agents, brands are grappling with something familiar but unsettling — they don't really know what shoppers are experiencing.
The three agents your shoppers are already talking to
Two of these live on your own site and app. The third belongs to the shopper.
Customer service agents
Customer service agents handle requests for tracking, payment processing, and returns and exchanges. They issue refunds and swap sizes, ideally without handing the customer to a person. Sierra, Fin, and Zendesk's AI agents are popular platforms brands build these on.
This is the most common first use case we're seeing with our e-commerce customers because volume is high and the requests repeat. It's also where a bad experience costs the most. A shopper who leaves a returns conversation frustrated may not come back, even with a refund in hand, and a ticket the agent marked resolved can turn into a second contact the next day.
Shopping assistants
Shopping assistants guide product discovery by answering pre-purchase questions, narrowing options, and personalizing recommendations.
They're the most visible type of agent, and the easiest to get wrong. Conviva tracks experience metrics like Clarification Ratio, how often shoppers restate what they want; Manual Search Between Turns, when they leave the chat to hunt for the product themselves; Scope-Narrowing Rate, when they settle for something other than what they asked for; and Post-Conversation Correction Rate, when they change or remove what the agent added to the cart. A recommendation only counts as a success if the shopper didn't have to work around the agent to get there.
Third-party shopping agents
Assistants like Muse, ChatGPT, and Gemini research products and sometimes check out on the shopper's behalf. For brands, the first conversation about a product may now happen in an agent they don't own, and it's already converting: agent-to-agent commerce converts at up to 4x the rate of traditional search-and-browse. Meta's Muse passed 10 million users in about six weeks, and more than a quarter of them shop through it, with an average purchase size reported to be over $200.
"Resolved" isn't the same as "it worked"
The problem is what gets measured. The agent marks the ticket resolved, CSAT comes back fine, and evals score every response as helpful. None of that tells you whether the shopper got what they came for. That's the gap agent analytics is meant to close.
The real evidence is behavioral, and it shows up during and after the conversation when:
- The shopper leaves the chat to search the site for what they just asked about.
- They add an item to the cart from the chat, then remove it and don't check out.
- They escalate to a person, or contact support again, after the agent marked the issue resolved.
- They log in afterward to check that a refund or exchange actually went through.
None of these show up in common agent observability tools, which is why the conversation and what happens around it it have to be measured together.
Measure agents by what shoppers experienced, not how the answers scored
At Conviva, we think an agent should be measured the way shoppers experience it. That takes three things:
- Every shopper, not a sample. Measure every conversation in production, not a test set or a slice of traffic. Samples miss the rare failures that cost the most.
- Signals inside and outside the conversation. Read what shoppers do on the site around the chat, like searches, carts, tickets, and purchases, so you see the full experience, not just the conversation transcript.
- Contrastive analysis. Slice conversations by dimensions so you can compare against what's normal under certain conditions to find where the agent is failing.
That's what Conviva Agent Experience Insights is built to do.
Frequently asked questions
Are e-commerce brands using AI agents?
AI agents are becoming a common way for brands to interact with their customers. Some of the most beloved retailers have announced their adoption of them, including Sephora's AI Beauty Chat, Walmart's Sparky Agent, and Amazon's Alexa for Shopping. Online retailers of this size need a comprehensive understanding of how their agents are working and the experience they create for shoppers.
What are common AI agents in e-commerce?
AI agents in e-commerce are software systems that act on a shopper's behalf, answering support questions, processing returns, recommending products, or completing a purchase without a person directing each step. Customer service agents, shopping assistants, and third-party agents like Muse and ChatGPT are the three main types used today.
How do you know if an e-commerce AI agent is working?
Look at what shoppers do during and after the conversation, not how the agent's answers scored. Restating a request, searching the site mid-chat, or contacting support again after a "resolved" chat all signal an agent that isn't working, even when CSAT looks fine.
