Skip to content

Create an Experiment

Before You Start

  1. Install the SDK and check the experiment in your code.
  2. Import user action data as metric source to measure experiment impact.

Where to Create?

Go to the Experiments section in the left sidebar, select the "Experiments", and click on the Create button. image.png

Fill in the Basic Information

● Experiment ID

ID can only contain letters, numbers, and underscores e.g., exp_image_optimization.

● Owner

The experiment creator is the default owner, and additional owners can be added as needed to facilitate experiment management through filtering.

● Advanced (optinal) -- Add Layers

By default, a dedicated layer with 100% traffic is automatically created for your current experiment and can be found under "Experiments -- Layers". By ticking the "Add Layer" option, you can avoid potential traffic starvation issues. The layers created under "Experiments -- Layers" can be selected here. You can also directly create one on this page. For more information about layers, please refer to ABetterChoice.ai. image.png

Set the Experiment Objectives

● Hypothesis

Describe the change and rationale for your experiment plan. For example, based on data analysis, changing the [xxx function] from [xxx] to [xxx] is expected to improve the xxx (effect). The [xxx metric] can quantify the improvement.

● Select Metrics

Only one Primary Metric is allowed, serving as the core indicator to validate the benefits. Multiple Secondary Metrics can be added to assess the benefits related to functionality. image.png

● How to add metrics?

Check desired metrics on the left side, and they will appear on the right side. Click the star icon on the right side to select the Primary Metric, which will be displayed above ( red box). To deselect a metric, click the "x" button or use the icon (blue box) to clear all selections. image.png

Allocation and Groups

● Traffic Allocation

  1. Total traffic percentage: Determines user participation in the experiment. If Layers are selected in Step 2, allocated traffic will not exceed the maximum defined within the Layer. By default, the total traffic of an experiment or layer is set to 100%.
  2. Add Group: Click the icon (blue box) on the right side to add groups. Up to a hundred groups can be added in one experiment, with at least one group required. Group names, traffic proportion, and parameter values are all editable. The sum of traffic percentages allocated to all experimental groups should be 100%. Actual traffic of the group = Total traffic percentage * traffic proportion of the group.
  3. Add Parameters: Click the icon (blue box) on the left side to add parameters. Parameters represent business or function strategies. Unlimited quantity for adding.

Values format limitation: 1. String: strings, letters, numbers, and underscore are accepted. 2. Number: positive/negative integer, 0, decimal, and scientific notation are accepted. 3. Boolean: FALSE or TRUE, control group defaults to be FALSE, treatment group defaults to be TRUE. 4. Json: object, array, numbers, strings. It also allows nested struct, for example{"test":{"service":{"ctr_model":0}}}. image.png

● WhiteList (optional)

Help to ensure each of the variants you're testing are working as expected. Select and add groups from the previous step by clicking "Add group" (blue box).

● Targeting Audience (optional)

Experiments will only target users within the specified audience. Additionally, you can click Add Condition (blue box) to add multiple conditions and select relationships (AND/OR) between multiple audiences. image.png

Finish Creation and Start

After clicking the "Create" button, you will receive a successful creation notification, and the experiment will be visible on the "Experiments -- Experiments" page. In the resulting page, you can start the experiment and perform other operations by clicking the icon in the upper right corner. image.png image.png