A Big Win from a Small Search UI Change and a Predictive Model

One of my biggest commercial wins was a seemingly small UI tweak that led to a roughly 2% marketplace revenue bump. Teachers Pay Teachers (TPT, think "Etsy for Teachers" -- educational resources created by and sold to other educators) was a mature marketplace, where revenue wins were typically under 0.5%. How we got this win I think is instructive, covering the value of a search scientist on a search team, and a holistic view of search UI/UX and algorithms.

Ratings and reviews are important proof of value to users searching on an e-commerce site. In 2020, the resource cards on the search results page looked like this -- the average historical rating, shown as partial stars, followed by the number of ratings in bold text.

The Problem

We knew from user interviews that buyers anchored on the number of ratings as a critical signal of value, and that the partial star format made it hard to see differences. We also used the number of ratings as a component of search ranking -- everything else being equal, resources with more ratings were ranked higher.

At some level this is reasonable. Historically popular resources were generally good. But this pair of design features -- a UI design that focused on rating count, and an algorithm that reinforced that -- had serious down-sides. First, it clearly penalized new resources, which may have been excellent. Second, it meant that resources whose first few ratings were poor, for whatever reason, were basically doomed. Nobody will purchase an item with "⭐️⭐️ (2)" next to it.

The Approach

So, we set about trying to solve these problems. The goal was multi-part:

• Buyers should anchor on how good the resource likely is, not how historically popular it has been.
• Sellers should not be penalized for bad luck on their initial ratings.
• The UI should be clear and informative.

The solution we landed on had several components. Critically, all of those components had to be deployed together. Let me walk you through them:

• On the front end, resource ratings were shown numerically, with the rating in bold and the less-important count now shown in grey.
• On the back end, a statistical model estimated the likely long-term rating average for this resource, based on the data accumulated so far. (More on this below.)
• In the search engine, the likely long-term rating average replaced the rating count in the scoring algorithm.

Now, the page looks like this:

Note that, as a buyer, you're likely to focus on the highlighted number, which now reflects our best guess of where the ratings will eventually land. This is a better signal of quality. And, you'll now see more items with high ratings, even if they're newer and haven't had the opportunity to accumulate many ratings yet.

The Algorithm

Consider two resources. Resource 1 has 200 ratings, averaging 4.7 stars, and is from a seller whose resources average 4.6 stars. Resource 2 only has 2 ratings, of 3 and 5, averaging 4.0 stars. But this resource is from a seller whose many resources average 4.8 stars.

How should all of this information be put together to construct an estimate of the long-term rating average, assuming many people will eventually purchase and rate the item?

The solution is Bayesian statistics. The intuition is not too complicated -- if you don't have much (or any) data, "borrow" some data from other cases until additional data comes in. So, for Resource 2, you can supplement the existing 2 ratings with some number of ratings of other resources by the same seller, or if necessary from other sellers, to get a more conservative estimate of the eventual rating.

Here's a very simple simulation of this in R code. I'm papering over a number of technical details, but assume that we've established that we want to borrow 50 observations from other resources. This code computes the expected value of the rating for the two resources. You can see that Resource 2 has an expected value that's quite different from the 4.0, while Resource 1, with many more actual ratings, changes little.

 1# Each resource has an actual number of ratings and an actual average
 2# rating. Each is also linked to a seller whose *other* resources have
 3# their own average rating -- this is the "prior" belief about a
 4# typical rating for this seller, before we've seen much data on this
 5# particular resource.
 6resources <- data.frame(
 7  resource        = c("Resource 1", "Resource 2"),
 8  n_ratings       = c(200, 2),
 9  actual_rating   = c(4.7, 4.0),
10  seller_rating   = c(4.6, 4.8)
11)
12
13# We "borrow" this many imaginary ratings from the seller's other
14# resources. The fewer real ratings a resource has, the more its
15# estimate will be pulled toward the seller's average.
16# Tuned as part of the model-fitting process, but static in production.
17borrowed_n <- 50
18
19# The posterior (Bayesian) estimate is just a weighted average of the
20# resource's actual ratings and the borrowed, seller-level ratings,
21# weighted by how many ratings each group contributes.
22resources$posterior_rating <- with(resources,
23  (n_ratings * actual_rating + borrowed_n * seller_rating) /
24    (n_ratings + borrowed_n)
25)
26
27resources

1##     resource n_ratings actual_rating seller_rating posterior_rating
2## 1 Resource 1       200           4.7           4.6         4.680000
3## 2 Resource 2         2           4.0           4.8         4.769231

Tuning this algorithm to work properly with 1-5 star ratings, handling sellers with few other ratings, and other edge cases, requires careful analysis by a statistically-savvy data scientist. From there, computing these values nightly, and incorporating them into the search algorithm, requires some data pipelines to be set up but is not particularly difficult.

The Impact

We ran an A/B test of these changes, and saw a roughly 2% increase in sales during the rollout, with neutral or positive impact on other metrics. The rollout required careful communication from the product marketing team, as some sellers saw shifts in how often their historically popular resources appeared, but we were able to clarify the situation and complete the rollout.

Overall, this example shows how search design and algorithms are tightly coupled. Changing the algorithm without the UI would have confused users -- items with large bold rating counts would seem wrongly ranked for no visible reason. Changing the UI alone would have been equally problematic, highlighting a noisy metric that penalized bad luck. Together, the changes directed users to a useful signal, encouraging them to click through and make the qualitative evaluation critical for purchase.

Notes

• It's been a few years since these changes were made. I may have mis-recalled some details. Other aspects of the page design have subsequently evolved, and I have no information as to the current algorithm.
• Statistical note -- The algorithm was an empirical Bayes shrinkage model with a normal approximation to the ordinal data. Offline back-testing allowed computation of variances and the shrinkage parameter. Production code was only slightly more complex than what's shown here.
• This article was primarily authored by me, with secondary contributions from AI systems. I wrote almost all of the text. The AI generated the R code (which I reviewed) and the diagram, and made a number of suggestions and copy-edits.
• Are you looking for support with search and discovery on your company's website? Does this post make you think I could help your company make great design or technology decisions? I'm a freelance consultant with extensive experience -- please reach out!