Metrics & decisions
A Bayesian engine that recomputes from the raw event ledger on every read — and one metric class it will never let auto-decide.
Metrics
Every experiment has a primary metric — trial_started by default. The dashboard's Primary metric field is a fixed list. POST /api/v1/experiments (dashboard session) also takes any event name as primary_metric: the exact string you already pass as the first argument to logEvent() in your app. Named a milestone generation_success? Send generation_success as the primary metric and the engine counts it as a rate, with failures left out — no extra wiring, no separate event you have to declare for experiments specifically.
Metrics split into two kinds, and the split matters for what happens next:
| Kind | Examples | Can decide a winner |
|---|---|---|
| Rate | trial_started, purchase_completed, paywall_cta_tapped | Yes |
| Descriptive-only | revenue_per_user, trial_conversion_rate, active_subscribers, churned_subscribers | No — see below |
The decision engine
Bayesian, Beta-Binomial, recomputed from the event ledger on every read rather than stored, so changing which metric you're watching never needs a backfill. 10,000 Monte Carlo draws per evaluation give each arm a prob_best, a 95% credible interval, a lift against the control with its own 95% range, and a chance to beat the control (RevenueCat's "Chance to win"). Evaluation runs every 15 minutes.
You can look at the numbers any time, but checking a 95% bar every 15 minutes still finds noise on small samples, so a winner requires all of:
- the test has run at least 7 days (two weeks is safer for weekday swings);
- every arm has at least 100 users who saw it;
- every arm has at least 30 conversions, and 100 in total;
- the leading arm's
prob_bestclears your confidence threshold.
Until then the dashboard says who is ahead and why it's too early, and auto-ship waits. With zero conversions, or a single arm, there is no leader at all.
What counts
- Users who saw it. A user counts once they reach the surface (
paywall_shownoronboarding_started) after being assigned, and a conversion counts only for those users. A buyer who never saw the test doesn't move its rate. - Real money only. Purchases RevenueCat marks
SANDBOX, and events that carryenvironment: SANDBOX, are left out. The results page says how many test purchases it ignored. - Refunds come off. Refunds and chargebacks (and their reversals) are subtracted from revenue, and refunded customers are counted.
- Only the arm's own subscriptions. A renewal counts only when the subscription started inside the arm, so an existing subscriber's old renewals never count; a returning customer's new purchase does.
Health checks
Every results page lists plain-words checks: whether the split matches what you set (a chi-square sample-ratio test; p < 0.01 means assignment or tracking is broken), whether it has run long enough, whether there are enough conversions, whether anyone has reached it, and whether trial tracking looks broken (billing shows trials the events don't).
Revenue and descriptive-only metrics never auto-decide — by design
If your primary metric is revenue_per_user, or any metric flagged descriptive-only, the experiment can never become decisive, no matter how confident the numbers get. This isn't a bug to work around — heavy-tailed money through a win/loss posterior manufactures false confidence, and a metric like churn would happily declare the worst arm the winner if left to decide on its own.
You still get everything else: prob_best, credible intervals, lift, the full billing-outcomes table. What you don't get is auto-ship, and in notify mode you don't get an alert either — the leader-found notification is gated on the same decisiveness check. Concluding a revenue-metric experiment is a manual action: read the numbers yourself, then promote a variant or stop the test.
Alerts
Three notifications, all fanned out to whichever of Slack, email, and outgoing webhook you've connected:
experiment_winner_shipped— auto mode, decisive, and it actually promoted.experiment_weekly_summary— Mondays from 14:00 UTC, once per app per week while any experiment runs: users seen per arm, the leader and its chance, and whether each test is ready to call.experiment_leader_found— notify mode, decisive, and this is the first time it crossed the bar. It fires exactly once per experiment; if the leader later flips while the test keeps running, there's no second alert. Re-check the dashboard rather than waiting on a notification for anything past the first signal.
Promotion
Roll out (promote) works while a test runs or is paused, and after you stop it. Rolling out a variant sets its config live for that placement and demotes whatever was live before — the losing arms need no rollback, since not promoting them is enough. A winning holdout promotes nothing, because the point of a holdout is that the baseline already won.