Say you're an Engineering Manager or a Product Manager, responsible for a business-critical search system on a website. Maybe e-commerce, maybe a marketplace, maybe something else. Search is sort of working, it doesn't crash, and it's fast, but there's a feeling that "search isn't great", and you've been tasked with finding someone to help. Your first instinct, or that of your VP, might be some version of: “we need to hire a search engineer.”

Sometimes that’s true. If your search system is broken at the infrastructure level, you need engineering help immediately. If it tends to crash during high-traffic sales, focus on that problem first. If you need new data pipelines to pull catalog information into your search engine, that's an engineering problem.

But for most established products, those aren't the main bottlenecks. Unless you're at a household-name company, the hard part is not getting search to run at your scale. The hard part is getting search to improve in a reliable, measurable way.

I think many teams are actually looking for a search scientist: someone who can research and diagnose failure patterns in user behavior, design meaningful experiments, build the right internal tools, and lead the application of insights to product and algorithmic improvements.

What Changes After Search Is "Working"

Early in a search product’s life, engineering dominates. You need indexing, retrieval, ranking, query understanding, scalable infrastructure, and all the reliability basics.

After that foundation is stable, most production changes become incremental:

• parameter changes to improve relevance and ranking
• ingestion of new data fields
• A/B tests of UI design changes
• tooling to support operational thumbs on the scale

Those are important, but they usually aren’t technically difficult to implement, and they don't take many lines of code. The hard part is figuring out what to measure, what needs to improve, what often-small changes might address the opportunity, and figuring out if the change actually worked, or caused unintended side effects.

That is search science. It's not just engineering or product work, it's some of each, plus a strong quantitative research core. This builds on Daniel Tunkelang's point that search leadership requires blended product and engineering judgment.

Where The Work Actually Is

In mature systems, much of the coding work moves away from user-facing production code and into internal tools and analysis code.

You need software that helps the team think:

• query replay and side-by-side result comparison
• segment-level metric drill-down dashboards
• annotation and judgment workflows -- making it easy for domain experts to flag issues for follow-up
• search failure databases and curation tooling
• experimentation readouts with guardrails
• offline evaluation and diagnostic notebooks

Without these, teams operate on anecdotes and intuition. With these, teams can move from “search feels off” to “high-intent accessory queries on mobile are converting less-well because synonym expansion is overfiring for brand modifiers.”

Why This Looks More Like Data Science Than Software Engineering

Search quality is fundamentally an inference problem under uncertainty and complex constraints. User intent is messy. Outcomes are sparse and ambiguous. Behavior changes in response to your product. Metrics are proxies. Everything is confounded.

That environment needs people who are strong at:

• hypothesis design
• causal reasoning
• metric and guardrail construction
• segmentation and bias checks
• statistical judgment under noisy data

These are core data science skills.

Senior engineers are still essential, especially for architecture, performance, reliability, and implementation quality. PMs are still essential for prioritization, alignment, and strategy. (See my earlier post on search teams and the product trio.) But if you are asking “who can represent the voice of the user in a search system?”, the answer is a person who can read the behavioral evidence and reason about algorithmic cause and effect.

In mature search products, leverage comes less from expanding production engineering capacity and more from strengthening the insight-to-experiment loop. That loop is usually owned best by a search scientist.

The Role of AI and Agentic Engineering

The LLM-aided revolution in software engineering since 2025 changes the equation even further. I've argued for years that a data scientist should be able to write backend software as well as a software engineer two standard levels below their title seniority, e.g., a Staff Data Scientist should code like a Senior Software Engineer. But what can those people do now, when the AI writes the code under human supervision? Your search scientist can draft front-end changes that work hand-in-hand with algorithmic changes. Or write internal tools in an afternoon, to amplify their own or the team's velocity and improve workflows.

You still need a software engineer on a team, to focus on code scalability, robustness, uptime, architecture, and integrations. Not to mention to be on-call if something breaks in production. But the long pole is now even more decision-making, research, and prioritization, not lines of code.

How to Hire a Search Scientist

If you’re a PM or EM staffing a search team, here is a starting point for the "What You'll Do" section of your job description (and check LinkedIn for current postings):

• Own search quality diagnosis: query segmentation, failure taxonomy, and root-cause analysis.
• Design and run experiments: offline eval plans, online tests, guardrails, and reports.
• Build internal relevance tooling: replay, side-by-side diffing, annotation workflows, diagnostics.
• Translate findings into shipped changes with PM + engineering partners. Contribute to production code.
• Maintain a search learning system: metric definitions, experiment history, and reusable playbooks.
• Talk to and learn from customers as well as internal domain experts.

You're looking for someone who can build this capability and run it reliably, again and again.

---

Notes:

• Are you looking for a fractional or interim search scientist? Want external help hiring a full-timer? I'm a freelance consultant with extensive experience in this area -- please reach out!
• The first draft of this post was written by an AI, based on a few notes about the argument to be made, but it was then extensively rewritten and refined by hand. The AI was a secondary author and most ideas and the final voice are mine.