ROI Dashboard

What is the total benefit of the published experiments? Whether the increase and decrease of core indicators are affected by the experiments? The ROI dashboard can help you answer all of your questions! By evaluating the cumulative long-term benefits of multiple experiments, you can make better decisions for your business and increase growth. You can follow the steps below to use the ROI dashboard:

• Step1 Create a holdout and associate it with the layer (which the experiments are in) you want to evaluate
• Step2 Start the experiments on the associated layer, obtain experiment results, and launch the experiment
• Step3 View the holdout results and analyze the long-term benefits of the experiment.

How to use ROI dashboard

Implementation Principles of Holdout

"Holdout" refers to a portion of all experimental users randomly selected, who will not receive any experiment. These holdout users serve as the control group and are used to evaluate the cumulative comparative effects of multiple experimental strategies. The main purpose of the Holdout method is to provide a benchmark for more accurate evaluation of the performance of the experimental group. By comparing the differences between the experimental group and the holdout group, we can better understand the impact of experimental variables on user behavior. This helps ensure the reliability and accuracy of experimental results, providing strong support for implementing improvements. For example, in a website design A/B test, group A users see the original design, group B users see the new design, and holdout users do not participate in the experiment. By comparing the user behavior of the experimental group and the holdout group, we can more accurately evaluate the impact of the new design on user experience. In implementation, we first divide a Holdout layer and a virtual experimental layer from all experimental users. Users in the Holdout layer are not affected by the experimental strategy, while users in the virtual experimental layer are affected by all experimental strategies. Then, we automatically create an experiment on both layers, with equal traffic for both experiments. Finally, by comparing the control group users in the Holdout layer experiment with the experimental group users in the virtual experimental layer experiment, we can understand the benefits of multiple experiments.

Create holdout

You can access the page from the 'ROI Dashboard' in the left navigation, click the 'Create' button in the top right corner. and then fill in the necessary information to create a new holdout:

Name

Fill in the name of the holdout. Names can only contain letters, numbers, and underscores.

Size

Set the traffic required for holdout. If you fill in 1%, it means that 1% users will not be affected by any experiment, and these users will be compared with users affected by multiple experiments to evaluate the cumulative long-term benefits of multiple experiments.

Metric

Select the metrics you want to see in the holdout. The benefits of these indicators will show in the dashboard.

Layers

Select the layers to be associated with the holdout. When experiments are released on these layers, the increases or decreases on the indicators of these experiments will be counted as the ROI of this holdout experiment. Moreover, please note that:

1. One layer can only be associated with one holdout
2. Please aware that the results of holdout experiment will be impacted if the layer has been disassociated

Owner

The owner can modify and delete the holdout. You can add other people to manage the holdout with you.

Parameter

You can choose an existing parameter or create a new one to associate with holdout experiment.

Notes: In order to avoid differences between the users in the holdout and the users in the experiment group due to the lack of experiment strategies for a long period of time, we suggest that you use a new holdout to measure the experiment benefits periodically.

Analyzing Experimental Benefits

Click the "Name" or "ROI" button of the holdout, and you can enter the following ROI dashbaord of this holdout experiment:

How to analyze the cumulative benefit of multiple experiments?

Overview

First of all, the "Overview" module shows how many experiments you have done on this holdout within the selected time period, how many experiments have been released, and how many experiments are positive in selected indicators. This can help you analyze the effectiveness of your experiments over a period of time.

Analysis

Then, you can view the cumulative experiment benefits of the metrics you care about in the "Analysis" module. This means that you can compare the performance of the users who are affected by the experiment and the users who are not affected by the experiment on a certain metric. You can see whether the multiple experiment strategies you released really bring an increase in the data of a certain metric, or conversely, lead to a decrease in the data of a certain metric.

View The Trends

From here you can see the approximate daily changes of a certain indicator within the selected time period.

Indicator Details

Click the bar chart of each indicator, and you can see the details of the following indicator. In this dashboard you can analyze the specific performance and fluctuation of a certain indicator.

Long-Term Benefits

In the long-term benefit section of a certain indicator, you can see the differences of the indicator value between the users in the holdout and the users in the experimental group. You can also see the relative difference in a certain indicator between these two parts of user data. When you hover the mouse over a certain date, you can see which experiments were released on this date, helping you determine which experiments caused the data fluctuations. We display cumulative data by default, and you can switch to non-cumulative data in the upper right corner.

Short-Term Benefits

You can see the relative difference data accumulated on this indicator in the experiments you released in the short-term income module. Green pots represents that experiments have a positive impact on this metric. Red pots represent that experiments have a negative impact on this metric.