I keep seeing AI skeptic takes like "AI is useless" or "LLMs are only good at Natural Language Processing", usually because of hallucinations and AI slop. Both are real problems. But that conclusion still misses the point: it misidentifies what LLMs are actually good at. My take is: LLMs are great …

Read More

Algolia is a tremendously powerful search platform, especially for e-commerce. But like most SaaS tools, the management dashboard doesn't do everything you need it to do. In particular, it's hard to compare search result rankings across variations in indexes -- a super common pattern when you're testing and iterating …

Read More

The rise of AI-assisted coding tools has made it dramatically easier and faster to build small side projects than ever before. What used to take weeks of evenings and weekends can now be done in a few focused sessions, with an AI pair programmer handling much of the boilerplate, debugging, and implementation details. …

Read More

As I've written about before, as a data scientist supporting a product or marketing team with A/B testing, the job is communication -- helping to translate between business requirements and what we can learn from statistics. I (and many, many others) have found that there is a lot of value in having a document, shared …

Read More

The other day, I was reading a post by Venkatash Rao (thousands of words of under-edited brilliance, as usual), and was struck by this note about the complexity of climate solutions: I tend to take as an article of faith the systems science rule of thumb that the complexity of solutions generally matches the complexity …

Read More

There’s been an immense amount of discussion about Large Language Models (LLMs) such as ChatGPT over the last year, of course. Some of that discussion has been whether they are intelligent, conscious, or on the path to Artificial General Intelligence. I’m particularly interested in the "consciousness" …

Read More

Recently I wrote a blog post that mentioned “Superiority” as a type of A/B test decision. In this post I want to talk about all five types of A/B test decision that I think are relevant. This is an adaptation and extension of a talk I gave last year at the Quant UX conference (it’s a great event, you should check it …

Read More

Recently, tech-journalism site The Markup ran a long, detailed, critical investigation of a predictive machine learning model used by the State of Wisconsin to identify public school students at risk of not graduating. I mostly agree with the conclusions of the piece -- the system appears not to be fit for purpose and …

Read More

The "best practice", when evaluating the results of an online controlled experiment (A/B test), is to use classical statistical tests, proceeding with a change if (and only if) the result of the test includes a p value of less than 0.05. But, the American Statistical Association (ASA) said in a prominent 2016 …

Read More

A/B testing is a tool for supporting decision-making in business, and so in addition to getting the statistics right, it’s really important to communicate well with the non-statisticians who will have the final say on the go/no-go decision. Most A/B tests in practice are testing ratios, conversion rates of various …

Read More