The Problem
Consider checkout flows on an e-commerce site. Thousands of users start purchases each day, and some fraction don’t finish. You want to know why: Is it a browser issue? A device? A failing API endpoint? In a real site there are thousands of potential explanatory variables. This is a discovery problem over high-dimensional observational data, which makes it fundamentally different from A/B testing. You can’t run thousands of A/B tests for every possible factor, and many factors (device type, network conditions) aren’t under your control anyway. You have to work with the natural variation in the data you already have. One common approach is to manually inspect individual sessions. This is expensive—a PM can easily spend hours chasing a symptom that turns out to be a red herring. It also struggles to tease out more subtle problems, like a UX friction that makes users only marginally less likely to succeed.Why not coding agent + DB?
Feeding billions of clickstream events to an LLM is not a good approach; LLMs are useful for summarizing individual sessions, but are not designed to discover statistically robust patterns across huge datasets. Since AI agents have become ubiquitous, a standard approach could be an off-the-shelf coding agent, with access to an off-the-shelf query engine on top of clickstream event data, that can wield a variety of statistical tools to discover patterns autonomously. In our experience, such systems are not yet capable of reliably solving difficult problems like this, and it is difficult for users to validate their results. However, AI agents provide an incredible orchestration capability, and ML systems like Predictive Intelligence fit nicely as tools to empower them.A Worked Example
We’ll illustrate with a scenario based on checkout flows at a real e-commerce site. There are two real causes of checkout failure:- Mobile Device: Mobile users complete checkout at lower rates—many are just browsing, not committed to buying
- API Failure: Some users hit backend API failures that block checkout
- US Daytime: More users browse on mobile during US business hours (work, commute)
- Social Media Referral: Users arriving via social media links (more common on mobile)
- Search Discovery: Users who found products via search rather than browsing
Figure 1. Mobile Device and API Failure are the true causes; the other three factors are correlated red herrings.
Naive Approach 1: Single-Factor Comparisons
The simplest thing to try is comparing outcomes one factor at a time: what’s the success rate for mobile vs. desktop? During US daytime vs. off-hours?
Figure 2. Univariate success rates by factor value. Red bars highlight factors that appear problematic.
All 5 factors appear problematic. This is the familiar confounding story: US Daytime, Social Media, and Search Discovery show lower success rates because they’re correlated with mobile usage, not because they independently hurt conversion. The result is a dashboard full of “findings,” most of which are echoes of the same underlying issue. API Failure, one of the two real problems, is lost in the noise.Naive Approach 2: Multi-Factor Combinations
A more careful approach is to stratify by combinations of factors: compare mobile users during US daytime vs. mobile users during off-hours, thereby isolating the time-of-day effect. This addresses confounding, but creates a combinatorial problem. When we apply this to a real system, we end up with billions of combinations. Even with just 5 binary factors there are 32 strata to examine:
Figure 3. Success rates for all 32 factor combinations. Most flagged as problematic simply because they include a true cause.
Many of these strata look problematic simply because they include a true cause (e.g., mobile). The API failure signal is buried among mobile-related findings, and it’s still unclear whether the mobile problem is really about devices, time of day, referral source, or search behavior.What’s Missing
The right intuition is to compare outcomes while controlling for the other observed factors—e.g., compare search vs. non-search users who are otherwise identical in device, time of day, etc. However, doing this manually is impractical: you’d need 16 controlled comparisons per factor, many with sparse data, and no principled way to reconcile conflicting results across strata. We need a method that estimates each factor’s effect while simultaneously adjusting for all the others. This is what multivariable modeling does.Regression-Based Attribution
Practitioners in epidemiology, economics, and clinical trials have long used regression models for exactly this problem: estimating effects from observational data with multiple confounders. For binary outcomes like conversion, the standard tool is logistic regression.Results
Here is the logistic regression output for our example:
Figure 4. Regression-based effect sizes relative to baseline. Only Mobile Device and API Failure show significant negative effects.
The model gets this right: It estimates large negative effects for Mobile Device and API Failure, and near-zero effects for the other three correlated but non-causal factors. This matches the ground truth.