a/b testing

A Collaborative Template for A/B Tests

Oct 12, 2023 · 9 min read · a/b testing product data science ·

As I've written about before, as a data scientist supporting a product or marketing team with A/B testing, the job is communication -- helping to translate between business requirements and what we can learn from statistics. I (and many, many others) have found that there is a lot of value in having a document, shared …

The Five Types of A/B Test Decisions

Jul 7, 2023 · 9 min read · a/b testing statistics data science ·

Share on:

Recently I wrote a blog post that mentioned “Superiority” as a type of A/B test decision. In this post I want to talk about all five types of A/B test decision that I think are relevant. This is an adaptation and extension of a talk I gave last year at the Quant UX conference (it’s a great event, you should check it …

p Values Are Useful for A/B Tests, Sometimes

May 2, 2023 · 9 min read · a/b testing statistics data science ·

Share on:

The "best practice", when evaluating the results of an online controlled experiment (A/B test), is to use classical statistical tests, proceeding with a change if (and only if) the result of the test includes a p value of less than 0.05. But, the American Statistical Association (ASA) said in a prominent 2016 …

Communicating A/B Test Results for Conversion Rates with Ratios and Uncertainty Intervals

Aug 20, 2022 · 8 min read · a/b testing data science statistics ·

Share on:

A/B testing is a tool for supporting decision-making in business, and so in addition to getting the statistics right, it’s really important to communicate well with the non-statisticians who will have the final say on the go/no-go decision. Most A/B tests in practice are testing ratios, conversion rates of various …

A/B Testing and Product Ratings, Part 2: Multiple Ratings

May 23, 2022 · 3 min read · data science analytics business statistics a/b testing ·

Share on:

Image not found

Web path: https://www.harlan.harris.name/images/path/thumbnail.png

Disk path: static/images/path/thumbnail.png

Using Page Bundles: false

Second, your A/B test split users, but you’re measuring ratings, which are per-item. Some customers will buy multiple pairs of shoes, and most likely will either rate all of them or none of them. Does this fact affect how confident you can be at the end of the experiment? Randomization Unit Mis-Match A/B tests are …

A/B Testing and Product Ratings, Part 1: Delays and Bias

May 22, 2022 · 7 min read · data science analytics business statistics a/b testing ·

Share on:

Suppose you’re a data scientist at an e-commerce web site that sells shoes, responsible for supporting A/B tests. Many A/B tests are easy, and there are a number of companies that sell tools that make the easy cases as simple as clicking a few buttons and looking at pretty graphs. But A/B tests can get statistically …