Harlan D. Harris
open-menu closeme
Publications
About
Medium
github linkedin stackoverflow
  • The Five Types of A/B Test Decisions

    calendar Jul 7, 2023 · 9 min read · a/b testing statistics data science  ·
    Share on: twitter facebook linkedin copy

    Recently I wrote a blog post that mentioned “Superiority” as a type of A/B test decision. In this post I want to talk about all five types of A/B test decision that I think are relevant. This is an adaptation and extension of a talk I gave last year at the Quant UX conference (it’s a great event, you should check it …


    Read More
  • p Values Are Useful for A/B Tests, Sometimes

    calendar May 2, 2023 · 9 min read · a/b testing statistics data science  ·
    Share on: twitter facebook linkedin copy

    The “best practice”, when evaluating the results of an online controlled experiment (A/B test), is to use classical statistical tests, proceeding with a change if (and only if) the result of the test includes a p value of less than 0.05. But, the American Statistical Association (ASA) said in a prominent …


    Read More
  • Communicating A/B Test Results for Conversion Rates with Ratios and Uncertainty Intervals

    calendar Aug 20, 2022 · 8 min read · a/b testing data science statistics  ·
    Share on: twitter facebook linkedin copy

    A/B testing is a tool for supporting decision-making in business, and so in addition to getting the statistics right, it’s really important to communicate well with the non-statisticians who will have the final say on the go/no-go decision. Most A/B tests in practice are testing ratios, conversion rates of various …


    Read More
  • A/B Testing and Product Ratings, Part 2: Multiple Ratings

    calendar May 23, 2022 · 3 min read · data science analytics business statistics a/b testing  ·
    Share on: twitter facebook linkedin copy
    Image not found

    Web path: https://www.harlan.harris.name/images/path/thumbnail.png

    Disk path: static/images/path/thumbnail.png

    Using Page Bundles: false

    Second, your A/B test split users, but you’re measuring ratings, which are per-item. Some customers will buy multiple pairs of shoes, and most likely will either rate all of them or none of them. Does this fact affect how confident you can be at the end of the experiment? Randomization Unit Mis-Match A/B tests are …


    Read More
  • A/B Testing and Product Ratings, Part 1: Delays and Bias

    calendar May 22, 2022 · 7 min read · data science analytics business statistics a/b testing  ·
    Share on: twitter facebook linkedin copy

    Suppose you’re a data scientist at an e-commerce web site that sells shoes, responsible for supporting A/B tests. Many A/B tests are easy, and there are a number of companies that sell tools that make the easy cases as simple as clicking a few buttons and looking at pretty graphs. But A/B tests can get statistically …


    Read More
  • intuitive visualizations of categorization for non-technical audiences

    calendar Apr 25, 2011 · 8 min read · analytics dataviz ggplot2 graphics predictive R visualization statistics  ·
    Share on: twitter facebook linkedin copy

    For a project I’m working on at work, I’m building a predictive model that categorizes something (I can’t tell you what) into two bins. There is a default bin that 95% of the things belong to and a bin that the business cares a lot about, containing 5% of the things. Some readers may be familiar with the use of …


    Read More
  • On "Analytics" and related fields

    calendar Apr 15, 2011 · 6 min read · conferences data science job titles operations research statistics  ·
    Share on: twitter facebook linkedin copy

    I recently attended the INFORMS Conference on Business Analytics and Operations Research, aka “INFORMS Analytics 2011”, conference in Chicago. This deserves a little bit of an explanation. INFORMS is the professional organization for Operations Research (OR) and Management Science (MS), which are terms describing …


    Read More
  • "Data Scientist" and other titles

    calendar Feb 13, 2011 · 3 min read · data science job titles statistics  ·
    Share on: twitter facebook linkedin copy

    Neil Saunders has an interesting (to me) blog post up this morning, with the title “Dumped on by data scientists.” He uses the use of “data scientist” in a Chronicle of Higher Ed article to rant a little bit about the term. For Neil, it’s redundant, as the act of doing science necessarily requires data; it’s insulting, …


    Read More
  • Prediction with Multilevel Regression Models, and Pizza

    calendar Oct 15, 2010 · 3 min read · meetup statistics R  ·
    Share on: twitter facebook linkedin copy

    The Meetup phenomenon, which is now substantial and longstanding enough to be more of a cultural change than a flash in the pan, continues to impress me. Even more so than tools like LinkedIn, Meetups have changed the nature of professional networking, making it more informal, diverse, and decentralized. Last night, …


    Read More

Featured Posts

  • A Collaborative Template for A/B Tests
  • Are LLMs a simple solution? And if so, for what problems?
  • LLMs and Theories of Consciousness
  • The Five Types of A/B Test Decisions
  • p Values Are Useful for A/B Tests, Sometimes
  • Communicating A/B Test Results for Conversion Rates with Ratios and Uncertainty Intervals
  • Staff Data Scientist: Comments on Will Larson's Staff Engineer Book

Recent Posts

  • A Collaborative Template for A/B Tests
  • Are LLMs a simple solution? And if so, for what problems?
  • LLMs and Theories of Consciousness
  • The Five Types of A/B Test Decisions
  • A Critique of The Markup’s Investigation into Predictive Models of Student Success
  • p Values Are Useful for A/B Tests, Sometimes
  • Communicating A/B Test Results for Conversion Rates with Ratios and Uncertainty Intervals
  • Staff Data Scientist: Comments on Will Larson's Staff Engineer Book

Categories

PROFESSIONAL 41 PERSONAL 13

Tags

DATA-SCIENCE 25 R 14 ANALYTICS 12 STATISTICS 9 MACHINE-LEARNING 8 CONFERENCES 7 A/B-TESTING 6 OPERATIONS-RESEARCH 6 PROGRAMMING 6 BUSINESS 5 MEETUP 5 JOB-TITLES 4 META 4 RECIPES 4
All Tags
A/B-TESTING6 ADVERTISING2 ALGORITHMS2 ANALYTICS12 BIKESHARE1 BLUETOOTH1 BUSINESS5 CIVIC-HACKING1 COGNITIVE-SCIENCE1 COMPUTER-SCIENCE1 CONFERENCES7 CONSCIOUSNESS1 COOKING1 COOKING-CLUB2 CSA3 DATA1 DATA-SCIENCE25 DATAVIZ2 DESSERT1 ETHICS2 FOOD2 GADGETS3 GGPLOT23 GRAPHICS3 HACKATHON2 HOUSING1 INTERNET2 JOB-TITLES4 JOURNALISM2 LLMS2 MACHINE-LEARNING8 MANAGEMENT3 MARKETING2 MEDIA1 MEETUP5 META4 MICROECONOMICS1 MONEY1 MP31 OPEN-DATA1 OPERATIONS-RESEARCH6 OPTIMIZATION1 PHONES1 PREDICTIVE2 PRODUCT2 PROGRAMMING6 R14 RECIPES4 REPORTING1 SCALING1 SHINY2 SIMPLICITY1 SMARTPHONES2 SMARTWATCHES1 SOFTWARE-ARCHITECTURE3 SOFTWARE-ENGINEERING2 STATISTICS9 SURVEY2 TEACHING1 TECHNOLOGY2 TELEVISIONS1 UI-INTEGRATION1 VIDEO1 VISUALIZATION2 WASHINGTON-DC1
[A~Z][0~9]
Harlan D. Harris

Copyright 2009- 2025 HARLAN D. HARRIS. This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License

to-top