Learning Metrics that Maximise Power for Accelerated A/B-Tests (2024)

2.1. Statistical Significance Testing

A one-tailed $p$-value for the one-tailed null hypothesis $A\nsucc B$ is given by $p_{i}^{A\nsucc B}=1-\Phi(z_{i}^{A\succ B})$, and rejected when $<\frac{\alpha}{2}$.Throughout, we use two-tailed $p$-values unless mentioned otherwise.

2.2. Learning Metrics that Maximise Sensitivity

The observation that we can learn parameters to maximise statistical sensitivity is not new.Yue etal. apply such ideas specifically for interleaving experiments in web search(Yue etal., 2010).Kharitonov etal. extend this to A/B-testing in web search, aiming to learn combinations of metrics that maximise the average $z$-score(Kharitonov etal., 2017).Deng and Shi discuss “lessons learned” from applying similar techniques(Deng and Shi, 2016).We introduce the approach presented by Kharitonov etal.(Kharitonov etal., 2017), as our proposed improvements build on their foundations.

3.1. Learning Metrics that Maximise Power

When directly optimising $z$-scores, an implicit assumption is made that the utility we obtain from increased $z$-scores is linear.This is seldom a truthful characterisation of reality, considering how we wish to actually use these metrics downstream.We provide a toy example in Table1, reporting $z$-scores and one-tailed $p$-values for two experiments and three possible metrics $m_{1},m_{2},m_{3}$, inspired by real data.In this toy example, we know that $A\succ B$, based on a hypothetical North Star metric.Nevertheless, as we do not know this beforehand, we typically test for the null hypothesis $A\simeq B$ with two-tailed $p$-values.In the table, this means that the outcome is statistically significant if the reported one-tailed values are $p<\frac{\alpha}{2}$ or $p>1-\frac{\alpha}{2}$.We report the power of every metric at varying significance levels $\alpha\in\{0.05,0.01\}$, reporting whether(i) the null hypothesis is correctly rejected ($p<\frac{\alpha}{2}$, ✓ ),(ii) the outcome is inconclusive (?), i.e. a type-II error, or(iii) the null hypothesis is rejected, but for the wrong reason ($p>1-\frac{\alpha}{2}$, $\times$).

This latter case is deeply problematic, as it signifies disagreement with the North Star.Such errors have been described as type-III or type-S errors in the statistical literature(Mosteller, 1948; Kaiser, 1960; Gelman and Carlin, 2014; Urbano etal., 2019).Naturally, we would rather have a metric that fails to reject the null than one that confidently declares a faulty variant to be superior.Indeed, Deng and Shi argue that both directionality and sensitivity are desirable attributes for any metric(Deng and Shi, 2016).Nevertheless, considering candidate metrics in Table1, type-III errors are not sufficiently penalised by the average $z$-score: metric $m_{3}$ maximises this objective despite yielding statistical power that is on par with a coin flip.

Directly maximising power might prove cumbersome, as it is essentially a discrete step-function w.r.t. the $z$-score, dependent on the significance level $\alpha$.Instead, it comes natural to minimise the one-tailed $p$-value reported in Table1.Indeed, the $p$-value transformation models diminishing returns for high $z$-scores, which allows type-III errors to be sufficiently penalised.When considering this objective, $m_{3}$ is clearly suboptimal whilst $m_{2}$ is preferred.

Note that the change in objective would not affect the geometric heuristic as described in Section2.2.1.As we simply apply a monotonic transformation on the $z$-scores, the weight direction that maximises the $z$-score equivalently minimises its $p$-value.When learning via gradient descent, however, the $p$-value transformation affects how we aggregate and attribute gains over different input samples.This allows us to stop focusing on increasing sensitivity for experiments that are already “sensitive enough”, and more equitably consider all experiments in the training data.

Note that whilst direct optimisation of $p$-values is an improvement over myopic consideration of $z$-scores, there is another caveat: the “worst-case” loss of a type-III/S error is bounded at 1, which does not reflect our true utility function: metrics that disagree with the North Star are far less reliable than those that simply remain inconclusive.As such, we also consider another variant of the objective, where $\bar{p}=-p\log(1-p)$.Figure1 provides visual intuition to clarify how this monotonic transformation on the $p$-values more heavily penalises type-III/S errors, whilst retaining the optimum.From a theoretical perspective, this function provides a convex relaxation for minimising the number of type-III/S errors a metric produces.As a result, we expect this surrogate to exhibit strong generalisation.We refer to this objective as minimising the $\log p$-value.

Note that one could envision further extensions here where the significance level $\alpha$ is directly incorporated into the objective function to maximise statistical power at a given significance level $\alpha$.Nevertheless, we conjecture that their discrete nature might hamper effective optimisation and generalisation, compared to the strictly convex and smooth surrogate we obtain from the $\log p$-value.

3.2. Accelerated Convergence for Scale-Free Objectives via Spherical Regularisation

The objective functions we describe —either $z$-scores or $(\log)p$-values— are scale-free w.r.t. the weights that are being optimised.As a result, out-of-the-box gradient-based optimisation techniques are not well-equipped to handle this efficiently.

For this low-dimensional problem, we can visualise the $z$-score as a function of the metric weights in a contour plot, as shown in Figure2(a).

Here, it becomes visually clear that whilst the direction of the $w=[w_{1},w_{2}]$ vector matters, its scale does not.The consequence is that the gradient vectors w.r.t. the objective on the right-hand plot can lead to slow convergence, even in this concave objective.Indeed, for poor initialisation in the bottom left quadrant (e.g. $w=[-1,-2]$), the gradient direction is perpendicular w.r.t. the optima.

4.1. Effectiveness of Learnt Metrics (RQ1)

We learn and evaluate metrics through leave-one-out cross-validation: for every experiment, we train a model on all other experiments and evaluate the $z$-score (Eq.7) and $p$-value (Eq.3) the metric yields for the held-out experiment.We report the mean and median $z$-scores and $p$-values we obtain for all A/B-pairs with known outcomes (i.e. $\mathcal{E}^{+}$) in Table2.Best performers for every column (either maximising $z$-scores or minimising $p$-values) are highlighted in boldface.Empirical observations match our theoretical expectations: whilst the $z$-score objective does effectively maximise the average $z$-score, it is the worst performer for both mean and median $p$-values, and even the median $z$-score.Our proposed $\log p$-value objective effectively improves both the median $z$-score and $p$-value over alternatives.

4.2. Agreement with the North Star (RQ2)

From the obtained $z$-scores and $p$-values summarised in Table2, we can additionally derive (dis-)agreement with the North Star, for varying significance levels $\alpha$.We visualise these results in Figure3: if the obtained $p$-value under a learnt metric is lower than $\alpha$, that metric yields a statistically significant result (agreement).If the obtained $p$-value for the alternative hypothesis (i.e. $B\succ A$ when we know $A\succ B$) is lower than $\alpha$, we have statistically significant disagreement, or type-III error.This is a capital sin we wish to avoid at all costs, as it severely diminishes the trust we can put in the learnt metric.If the $p$-value reveals a statistically insignificant result, we say the result is inconclusive, implying a type-II error.We observe that both the $z$-score-maximising metric and the heuristic approach fail to steer clear from type-III error.Optimising ($\log$)$p$-values instead, alleviates this issue.For this reason, we only consider these metrics for further evaluation.Indeed: an analysis of type-II error is rendered meaningless when type-III errors are present.

Figure7(a) in AppendixA highlights that for the Moj platform as well, type-III errors are common in the case of $z$-score-maximising or heuristic metrics.As such, we only consider ($\log$)$p$-values to assess increases in statistical power and potential reductions to the cost of running online experiments.

4.3. Power Increase from Learnt Metrics (RQ3–4)

Until now, we have leveraged experiments with known outcomes to assess sensitivity and agreement with the North Star.Now, we additionally consider A/A-experiments ($\mathcal{E}^{\simeq}$) and experiments with unknown outcomes ($\mathcal{E}^{?}$) to measure type-I and type-II error respectively.We measure this for the North Star, for the best-performing proxy metric that serves as input to the learnt metrics, and for learnt metrics that exhibit no empirical disagreement with the North Star.We plot the type-I error (i.e. fraction of A/A-pairs in $\mathcal{E}^{\simeq}$ that are statistically significant at significance level $\alpha$) and the type-II error (i.e. fraction of A/B-pairs in $\{\mathcal{E}^{+}\cup\mathcal{E}^{?}\}$ that are statistically insignificant at significance level $\alpha$) for varying values of $\alpha$ in Figure4(a).We observe that we are able to significantly reduce type-II errors compared to the North Star (up to 78%), whilst keeping type-I errors at the required level (i.e. $\alpha$).However, we also observe that the type-II error we obtain when using learnt metrics does not significantly improve over the top proxy metric, when considered in isolation.

Nonetheless, this is not how evaluation metrics are used in practice.Indeed, we track several metrics and can draw conclusions if any of them are statistically significant.As such, metrics should be evaluated on their complementary sensitivity.That is, we compute $p$-values for a set of metrics, apply a Bonferroni correction, and assess statistical significance.The statistical power that we obtain through this procedure is visualised in Figure4(b).We consider either the North Star in isolation, the North Star in conjunction with the top-proxy, or a further combination with any learnt metric.Here, we observe that the learnt metric provides a substantial increase in statistical power: statistical power (i.e. $1-$ type-II error) is increased by up to a relative 210% compared to the North Star alone, and 25–30% over the North Star plus proxies.Furthermore, as the Bonferroni correction is slightly conservative, we observe lower than expected type-I error for higher significance levels $\alpha$.This implies that a more fine-grained multiple testing correction can further improve statistical power.We empirically observe that this works as expected, but its effects are negligible in practice.

4.4. Cost Reduction from Learnt Metrics (RQ5)

So far, we have shown that metrics learnt to minimise ($\log$)$p$-values are effective at improving sensitivity (Table2), whilst minimising type-III error (Figure3) and improving statistical power (Figure4).

On one hand, powerful learnt metrics can lead to more confident decisions from statistically significant A/B-test outcomes.Another view is that we could make the same amount of decisions based on fewer data points, as we reach statistical significance with smaller sample sizes.This implies a cost reduction, as we can run experiments either on smaller portions of user traffic or for shorter periods of time, directly impacting experimentation velocity.

This reduction in required sample size is equal to the square of the relative $z$-score(Chapelle etal., 2012; Kharitonov etal., 2017).We visualise this quantity in Figure5, for varying significance levels $\alpha$, for the same Bonferroni-corrected procedure as Figure4(b).To obtain a $z$-score for a set of metrics, we simply take the maximal score and apply a Bonferroni correction to it as laid out in Section2.1.1.Note that this procedure depends on $\alpha$, explaining the slope in Figure5.We observe that our learnt metrics can achieve the same level of statistical confidence as the North Star with up to 8 times fewer samples, i.e. a reduction down to $12.5\%$.This significantly reduces the cost of experimentation for the business, further strengthening the case for our learnt metrics.

4.5. Spherical Regularisation (RQ6)

Our goal is to assess and quantify the effects of the proposed spherical regularisation method in Section3.2.We train models on all available data with known outcomes $\mathcal{E}^{+}$, where we have a vetted preference over variants $A\succ B$.We consider three weight initialisation strategies to set $w_{\rm init}$, and normalise weights to ensure $\mathcal{L}_{\lVert w_{\rm init}\rVert}=0$:(i) goodinitialisation at $w_{\rm init}=\frac{1}{|\mathcal{E}^{+}|}\sum_{(A,B)\in\mathcal{E}^{+}}\bm{\mu}$,(ii) constantinitialisation at the all-one vector $w_{\rm init}=\vec{\bm{1}}$, and(iii) badinitialisation at $w_{\rm init}=\frac{1}{|\mathcal{E}^{+}|}\sum_{(A,B)\in\mathcal{E}^{+}}\bm{\mu}$.We train models for all learning objectives we deal with in this paper: $z$-scores, $p$-values, and $\log p$-values; whilst varying the strength of the spherical regularisation term $\delta$.As discussed, this term does not affect the optima, but simply transforms the loss to be more amenable to gradient-based optimisation methods.Thus, we expect convergence after fewer training iterations.All models are trained until convergence with the adam optimiser(Kingma and Ba, 2014), initialising the learning rate at $510-4$ and halving it every 1 000 steps where we do not observe improvements.We use the radam variant to avoid convergence issues(Reddi etal., 2018; Liu etal., 2020), and have validated that this choice does not significantly alter our obtained results and conclusions.We consider a model converged if there are no improvements to the learning objective after 10 000 steps.All methods are implemented using Python3.9 and PyTorch(Paszke etal., 2019).

Figure6 visualises the evolution of the learning objective over optimisation steps, for all mentioned learning objectives, initialisation strategies, and regularisation strengths.We observe that the method is robust, significantly improving convergence speed for all settings, requiring up to 40% fewer iterations until convergence is reached.This positively influences the practical utility of the learnt metric pipeline for researchers and practitioners.

Ratio metrics are easily fooled.

Often, important metrics can be framed as a ratio of the means (or sums) of two existing metrics(Baweja etal., 2024; Budylin etal., 2018).Examples include click-through rate (i.e. clicks / impressions), variants of user retention (i.e. retained users / active users), or general engagement ratios (e.g. likes / video-plays).We observe that, whilst these metrics can be important from a business perspective, they typically exhibit significant type-III/S errors w.r.t. the North Star.Indeed, in the examples above both the numerator and denominator represent positive signals we wish to increase.Suppose an online experiment increases the number of video-plays by $Y\%$, and the overall number of likes by $X\%<Y\%$.These two positive signals will lead to a decreasing ratio, whilst we are likely to still prefer the treatment w.r.t. the North Star if $X$ and $Y$ are substantially large.Similar observations cautioning the use of ratio metrics have been made by Dmitriev etal. (2017).

We believe this is connected to common offline ranking evaluation metrics prevalent in the recommender systems field(Steck, 2013; Jeunen, 2019).Indeed, such metrics are cumulative in nature, optimising overall value instead of some notion of value-per-item(Jeunen etal., 2024).

User-level aggregations conquer general counters.

In the previous example, we describe general count metrics for the number of likes and the number of video-play events.User behaviour on online platforms often follows a power-law distribution: a few “power users” generate the majority of such events(Chi, 2020).As a result, such metrics are easily skewed, and they are not guaranteed to accurately reflect improvements for the full population of users—empirically leading to type-III/S errors w.r.t. the North Star.Aggregating such counters per users (e.g. count the number of days a user has at least $X$ video-plays) instead of using raw event counters, provides strong and sensitive proxies to the North Star.

Interestingly, this framing is reminiscent of recall, as we effectively measure the coverage of users about whom we have positive signals.Recall metrics are again strongly connected to offline evaluation practices in recommender systems, especially in the first stage of two-stage systems common in industry(Covington etal., 2016; Ma etal., 2020).