Experiments Overview

This page zooms out so you can see the full lifecycle of an experiment at a glance: what each status means, what objects it depends on, and what actions you can take.

First time creating an experiment? Go straight to Quick Start. This page gives you the big picture; Quick Start walks you through every click.

What is an experiment

An experiment is a method for scientifically validating product decisions by comparing different approaches. On the ABC platform, you randomly split users into multiple variants, serve each variant different parameter values or feature versions, and then use metric comparisons to identify the best option.

Why run experiments

• Reduce decision risk — validate hypotheses with data rather than intuition
• Quantify impact — measure precisely how a change affects primary metrics (retention, revenue, engagement)
• Ship safely — validate with a small traffic slice first, then roll out fully once there are no negative signals
• Iterate continuously — find optimal parameter configurations through successive experiments

Experiment types

ABC supports three main experiment types, selectable from the dropdown when you click Create on the Experiments page:

Type	Description	Best use case
Layer Experiment	Isolates traffic using a layer-and-domain architecture so multiple experiments can run in parallel without interfering	Most A/B scenarios, especially when you need multiple experiments running simultaneously without contamination
Remote Config Experiment	Binds Remote Config parameters to experiments, with each audience owning 100% of the traffic independently	Multi-region or multi-segment parameter tuning, and staged parameter value migrations
MAB Experiment	Dynamically adjusts traffic across variants based on the primary metric, automatically shifting toward the better-performing variant	Multi-variant rapid selection, time-sensitive campaigns, and scenarios requiring fast convergence

Experiment lifecycle

Every experiment follows the same state machine:

Status	Meaning
Not Started	The experiment has been created but never started
In Progress	The experiment is running and accumulating exposures; data is visible on the Results tab
Shipped	Make Decision → Ship was confirmed; the winning variant's values are now live
Archived	Make Decision → Archive was confirmed; the experiment is closed and cannot be restarted

Make Decision is the only entry point from In Progress to a terminal state, and it offers two actions:

• Ship — select the winning variant; its parameter values are written back to the audiences covered by this experiment
• Archive — close the experiment without shipping any variant

Key milestones

Every experiment follows the same milestone sequence:

1. Create experiment — fill in basic information, hypothesis and metrics, traffic and variants

2. Start experiment — click Start when prerequisites are met; status changes to In Progress

3. Configure the stats engine — set significance level, power, FDR, and CUPED on the Results tab

4. Read results — review Suggestion, cumulative exposures, Basic Analysis, Explore, and HTE

5. Make a decision — click Make Decision to choose Ship or Archive

---