Layers and Parameters

A layer is a logical construct that represents the entirety of users against whom you will conduct experiments. Typically, each layer corresponds to a tangible module within your system. For instance, you might establish a layer for the user interface of your webpage and another for the backend server that supports your page.

Within each module, numerous features may require experimentation. Consider a button on your landing page: you might wish to experiment with three parameters - the button's size, the text color, and the background color. It's important to note that parameters within the same module are often interdependent. For example, blue text color may not be compatible with a blue background color.

This interdependency offers another perspective on the relationship between parameters and layers. We can divide all parameters into N subsets, with each subset corresponding to a layer. Parameters from different subsets are entirely independent of each other, while parameters from the same subset are typically related. Consequently, they should belong to the same layer.

Run Experiments concurrently within a Layer

Each layer allows for the allocation of mutually exclusive traffic to each experiment, thereby enabling the simultaneous execution of multiple experiments. Let's assume a layer named layer_on_landing_page has been created with three parameters: button_size, button_text_color, and button_background_color. The current values for these parameters are 12, white, and blue respectively, which we can set as default values. The objective is to identify the combination that optimizes the button's CTR (click-through-rate). To achieve this, three experiments are run concurrently. The first experiment tests the button size, the second tests the combination of text and background color, and the third tests all three parameters in combination. Each experiment utilizes 10% of the total traffic, leaving 70% of the traffic unallocated within the layer.

experiment_for_button_size

experiment_for_button_color

experiment_for_button_combination

For experiment_for_button_size, no explicit values were set for the parameters button_text_color and button_background_color. At runtime, they will default to the layer's default value, which will be explained subsequently.

Retrieving Experiment Assignments Utilizing Parameters

The use of layer parameters to run experiments can significantly enhance engineering efficiency and iteration speed. This is particularly crucial when deploying new code can take several days or weeks, such as when a build-release cycle is required for mobile apps. Without leveraging parameters to conduct your experiment, your code might resemble the following:

	buttonSize := DEFAULT_SIZE
	buttonTextColor := DEFAULT_TEXT_COLOR
	buttonBackgroundColor := DEFAULT_BACKGROUND_COLOR
	experiment, err := abcUserContext.GetExperiment(context.TODO(), projectID, "layer_on_landing_page")

	if experiment.ExperimentName == "experiment_for_button_size" && experiment.GroupName == "Control" {
		buttonSize = 12
	} else if experiment.ExperimentName == "experiment_for_button_size" && experiment.GroupName == "Treatment_A" {
		buttonSize = 16
	} else if experiment.ExperimentName == "experiment_for_button_color" && experiment.GroupName == "Control" {
		buttonTextColor = "white"
		buttonBackgroundColor = "blue"
	} else if experiment.ExperimentName == "experiment_for_button_color" && experiment.GroupName == "Treatment_A" {
		buttonTextColor = "yellow"
		buttonBackgroundColor = "green"
	} else if experiment.ExperimentName == "experiment_for_button_combination" && experiment.GroupName == "Control" {
		buttonSize = 12
		buttonTextColor = "white"
		buttonBackgroundColor = "blue"
	} else if experiment.ExperimentName == "experiment_for_button_combination" && experiment.GroupName == "Treatment_A" {
		buttonSize = 16
		buttonTextColor = "yellow"
		buttonBackgroundColor = "green"
	}

This code is verbose, prone to errors, and requires updating and deploying new code for each new experiment. However, by leveraging the layer key and strong-type APIs to retrieve the experiment assignment, the process becomes much simpler:

experiment, err := abcUserContext.GetExperiment(context.TODO(), projectID, "layer_on_landing_page")
buttonSize := experiment.GetInt64WithDefault("button_size", DEFAULT_SIZE)
buttonTextColor := experiment.GetStringWithDefault("button_text_color", DEFAULT_TEXT_COLOR)
buttonBackgroundColor := experiment.GetStringWithDefault("button_background_color", DEFAULT_BACKGROUND_COLOR)

The code is now more concise and cleaner. Adding a new experiment simply involves creating a new experiment under the same layer layer_on_landing_page via the UI and configuring the corresponding parameters. This allows for the execution of the new experiment without any code modification. If a parameter is not explicitly set under an experiment, the API will default to the layer's default value. The API default value, such as DEFAULT_TEXT_COLOR, is a fallback when a parameter is forgotten to be added to the layer, and its value is lower in priority than the layer's default value. When a user is not allocated to any experiment, such as the 70% remaining traffic under layer_on_landing_page, the APIs will return the layer's default values for the three parameters.

Parameter-Based Experiment Launch

After conducting the experiments over several days, let's assume we observe a significantly higher click-through-rate for the treatment group of experiment_for_button_size compared to the control group. This indicates that 16 is a superior value to 12 for the parameter button_size. If we wish to launch this winning value for the remaining users, we can simply undertake two steps:

1. Archive the experiment experiment_for_button_size as we have already reached a conclusion. This leaves 80% of the traffic unallocated within the layer.

2. On the layer page, alter the default value for the parameter button_size from 12 to 16. This modification will affect the button_size parameter value for the 80% unallocated traffic, as well as the traffic under experiment_for_button_color, since experiment_for_button_color did not establish a value explicitly for button_size. For the traffic under experiment_for_button_combination, the value for button_size should continue to adhere to the explicit setup (which holds a higher priority than the layer default value) within that experiment, as we do not wish to modify user behavior during an ongoing experiment.

The following diagram illustrates the complete workflow: