An AI shopping agent can complete a purchase and still fail the customer. Why? Because outcome and experience are not the same metric.
In this episode of Beyond The Cart, Keith Zubchevich, CEO of Conviva, discusses what e-commerce brands need to consider when deploying AI agents. Watch the full episode on YouTube and read the highlights below.
Why Outcome-Based Metrics Miss the Real Failure Point
Purchase completion is the easiest event to track, which is exactly why so many agent vendors sell against it — some even price their offerings purely on outcomes. But a completed purchase can mask a shopper who fought their way to the finish line but has no intention of coming back.
Zubchevich describes this as the gap between outcome and experience:
"The fact that the shopper checked out or successfully completed a return just means they had enough incentive or made enough progress to commit and get it done. It doesn’t necessarily mean that they had a good experience or will return to your site. Experience is where you build shopper loyalty."
That’s why Zubchevich says brands with consumer-facing AI agents need to measure more than just conversion. Here are four experience metrics he recommends starting with:
- Turns to intent measures how many exchanges it takes the agent to understand what the shopper actually wants (ideally one or two, not five or six)
- Sentiment trend measures whether emotional tone improves or degrades across the conversation
- Reclassification rate measures how often a shopper has to correct the agent (for example, saying, "No, I meant this")
- Agent-attributed cart lift measures whether the agent moved a shopper from their stated intent to a higher-value outcome, not just whether they checked out
None of these show up in a standard conversion dashboard.
Turning Conversation Into a Measurable Pattern
Tracking metrics like turns to intent and sentiment trend requires a different unit of measurement than most analytics tools can produce today.
Conviva's approach converts raw dialogue into semantic patterns: a hundred shoppers can phrase the same request a hundred different ways, but the underlying intent — a greeting, a price objection, a size question — is the same pattern repeated. That abstraction is what makes it possible to measure turns to intent, sentiment shifts, and context failures in real time, and to isolate exactly what caused a conversation to go sideways.
This same lens reframes personalization. Purchase history tells a brand what someone bought. Behavioral context — stateful, sequential, tied to how someone actually shops — tells a brand how they buy, which is the signal that lets an agent create truly personal experiences.
What's at Stake for Brands That Wait
Getting this right isn't just about experience quality — it's about who owns the customer relationship at all. When a third-party agent (a general-purpose assistant, not the brand's own) sits between a shopper and a purchase, the brand loses the behavioral signal that used to come from owning that conversation.
"[When customers land on your site from ChatGPT] You immediately go from being a retailer to a wholesaler,” says Zubchevich. “You literally get a purchase order from ChatGPT. You don't know what the shopper was looking for. You probably competed on price, which means it's a race to zero."
Brands that wait to build their own agent lose exactly the behavioral intelligence this piece has been building toward — the context that turns a completed transaction into a returning customer. Zubchevich compares agent deployment to a fitness routine: uncomfortable at first, but the discomfort doesn't go away by delaying it. By the time a poorly performing agent shows up in quarterly revenue, the shoppers it lost are already gone.
Key Takeaways
- Outcome and experience are different metrics — a completed purchase doesn't mean a shopper is satisfied or likely to return; measuring only conversion hides the churn risk building underneath it.
- Turns to intent is the leading indicator brands are missing — fewer turns to understand shopper intent correlates directly with satisfaction and repeat behavior; five or more turns signals wasted time and wasted spend, since every turn carries a token cost.
- Conversations need to be measured as patterns, not events — converting dialogue into semantic patterns is what makes it possible to track turns to intent, sentiment, and context failures in real time, rather than after the fact.
- Behavioral context (how people buy) matters more than purchase history (what they bought) — the same request from two different shoppers can require two entirely different agent responses.
- Brands that don't deploy competitive agents risk losing the customer relationship itself — not just a sale, but the behavioral intelligence, pricing control, and repeat-purchase loyalty that come with owning the shopping conversation.
