Test Objective

Define the main objective of the test.

Example: Increase the CTR of banners.

Formulating the Hypothesis

State the hypothesis.

• Null Hypothesis ($H_0$): There are no significant differences between the control group (A) and the experimental group (B). Mathematically: $µ_A = µ_B$
• Alternative Hypothesis ($H_1$): There are significant differences between the control group (A) and the experimental group (B). Mathematically: $µ_A ≠ µ_B$

Choosing Metrics

Select the metrics.

Primary Success Metric (OEC)

The primary metric is used to determine whether the solution successfully improves the strategic opportunity.

Example: If the goal is to increase the efficiency of an advertising campaign, the primary metric could be the click-through rate (CTR) of banners.

Secondary Success Metrics

These are additional metrics that may indirectly influence decision-making. They help understand the impact of changes on different aspects of the system.

Example: For the goal of increasing banner CTR, secondary metrics might include the total number of impressions and the number of clicks.

Tradeoff Metrics

Each positive improvement in one metric might lead to a decline in another important metric. We want to monitor this to ensure we are making positive improvements to our system as a whole.

Example: When increasing the CTR of banners, a tradeoff metric could be the bounce rate. It is important to ensure that the increase in CTR does not lead to a significant increase in bounce rate.

Leading Indicators

These metrics can serve as early indicators of final metrics, providing valuable information for early analysis and decision-making.