When AI Evals Move to the PM, What Moves Out?

AI evals require product judgment, statistical intuition, and hours of trace review. If PMs take that on, leaders must decide what work comes off their desks.

8 minute read

A product manager's desk with a long task list on one side and a short list of tasks being handed off on the other

I had coffee recently with two product managers who were spending their afternoons learning to write and score evals. Not data scientists. Not machine learning researchers. Senior product managers.

They weren’t poking at chat prompts for fun. They were knee-deep in trace logs, annotating failure modes, and wrestling with whether a model’s summary hallucinated an edge case or merely summarized it aggressively. With product leaders getting deeper into the muck while working on generative AI products, evaluations have migrated from the research lab directly onto the product desk. What struck me as they described their weeks wasn’t that they were taking on the work—they were genuinely engaged by it. What struck me was that nobody had asked what they were putting down to make room for it. Earlier this week I wrote that a team can only absorb so much change at once (Capping In-Flight Change). This is the same constraint at the scale of a single job description.

What an AI Eval Actually Is

In traditional software, testing is deterministic and boolean. You pass input A, you assert output B, and the test suite turns green or red. If a user clicks a button, a record either lands in the database or it throws an exception. Product managers defined the acceptance criteria, handed them across the aisle to engineering and QA, and checked them off during acceptance testing.

AI evaluations are nothing like that. Because Large Language Models are non-deterministic probability distributions, there is no single “correct” string output.

An eval is an empirical measurement system designed to answer a much squishier question: Is this output acceptable, safe, and useful to a human being?

In practice, running rigorous evals means a PM has to take on three distinct technical chores:

  1. Curating Golden Datasets: Hand-crafting and maintaining hundreds of realistic, messy user inputs along with expected reference answers, edge cases, and adversarial prompt injections.
  2. Authoring Failure Taxonomies: Defining the exact boundaries of what constitutes a failure. When an agent answers a user query, did it fail because of retrieval noise, tool-call hallucination, bad tone, or poor formatting?
  3. Calibrating Judges and Error Analysis: Reading hundreds of execution traces step-by-step. If you use an LLM-as-a-judge to automate scoring, you must first validate that the judge’s scoring agrees with human intuition (measuring inter-annotator agreement like Cohen’s kappa, the statistic that separates real agreement from agreement you’d get by chance).

In their AI Evals FAQ, Hamel Husain and Shreya Shankar capture why this responsibility has landed squarely on the PM:

“Engineers catch technical issues like retrieval issues and tool errors. PMs identify product failures like unmet user expectations, confusing responses, or missing features users expect.”

Husain and Shankar suggest that teams should lean toward a “benevolent dictator” for error analysis, typically a domain expert or a PM who understands user needs, to serve as the final arbiter of quality.

The rationale makes total sense: a machine learning engineer can tell you whether a vector search retrieved the top-5 cosine matches, but only someone steeped in the customer’s problem can tell you whether the third match actually answers the user’s question.

60-80%
of development time goes to error analysis and evaluation
reported by the Husain/Shankar team, who build this tooling for a living
Husain & Shankar, AI Evals FAQ, 2026

That’s with engineers doing the work and purpose-built tooling behind them. A PM standing up the same practice without that background should expect the first pass to take meaningfully longer, and nobody should mistake either number for a quick afternoon task.

The Shift That Burns Out

Evals take time, but the harder shift is that they rewire how many product managers think about their job.

For years, product management taught people to eliminate ambiguity up front. You wrote PRDs, locked down user stories, and mapped out explicit user journeys. Success meant the software behaved predictably according to specification.

Generative AI destroys that predictability. You are no longer specifying deterministic behavior. You are managing statistical distributions of error. A model might be 91% accurate on contract extraction. Is 91% ready to ship? Which 9% of errors will users laugh off as minor glitches, and which 9% will cost your customer millions of dollars or land your company in legal discovery?

Answering that question forces the PM out of backlog administration and into the posture of an empirical scientist. Instead of writing specs, you inspect log traces. Instead of running bug scrubs, you evaluate confidence intervals.

And trace review is intellectually exhausting, high-cognitive-load work. You cannot multitask during an error-analysis session. You have to read multi-turn tool traces, analyze retrieved context chunks, and evaluate subtle semantic nuances. Husain and Shankar’s minimum viable setup is 30 minutes of manual review across 20 to 50 outputs after every significant change, and that’s the floor, not the deep error-analysis pass where the real failure modes surface. Doing that while juggling twenty Slack threads and a sprint review is a recipe for immediate burnout.

Scope Grows One Duty at a Time

Meanwhile, executive expectations around the PM role are compounding. At the Lenny & Friends Summit (Lenny’s Newsletter, September 2026), Geoff Charles, CPO at Ramp, stated the ambitious case: “PMs will become GMs, and they will own the actual business outcome and lead the entire function.” In the same summit, Tamar Yehoshua, Atlassian’s Chief Product and AI Officer, observed that “Roles are overlapping, but they are also expanding. . . . Everyone can do so much more than they could do before.”

Both observations are accurate descriptions of where the industry is heading. But neither one addresses the reality of a forty-hour workweek.

Yehoshua’s line deserves a second read. Everyone can do so much more than they could before. Whether they should absorb all of it is the cognitive-load question, and it’s the one neither quote answers.

PM Duty Reallocation MatrixDiagram showing incoming AI responsibilities displacing legacy administrative tasks.Inbound AI Duties- Authoring & Running Evals- Trace Inspection & Triage- Error Quality AnalysisFixed 40h CapacityZero-Sum BandwidthOutbound Off-Ramps- Status Decks & Syncs (Drop)- Ticket Grooming (Eng hand-off)- Manual Release QA (Automate)
 
Adding a duty takes thirty seconds in an executive offsite, but removing one takes an explicit leadership decision to prioritize the right type of work. A product manager cannot unilaterally drop responsibilities that the rest of the organization still expects to see completed.

When leadership hands a product manager ownership of evals without removing anything else, most PMs won’t push back, but will try to do both. They review traces early in the morning, answer Slack all day, and write status decks late into the night. Within two quarters, your best product thinker is exhausted, the quality of both the core roadmap and the AI evaluations starts to fray, and you may find that they no longer have time for product thinking that you valued from them.

Building the Off-Ramp

Your responsibility as a product leader goes beyond championing AI adoption. You have to deliberately construct the off-ramps that make adoption sustainable for the team.

If you expect your PMs to invest ten to fifteen hours a week in eval design, trace analysis, and failure triage, you have to decide what that time comes out of and give them the means to let the work go. That means sitting down and establishing a clear exchange. While not for all organizations, here are a few things that I’ve seen successful in the past few months:

  • De-escalate ticket administration: Hand sprint ceremony ownership, story point estimations, and routine ticket grooming entirely to engineering tech or QA leads.
  • Kill recurring status presentations: Replace slide decks and synchronous weekly alignment meetings with automated, async summaries of merged pull requests and customer telemetry.
  • Redefine the definition of PM output: Make trace evaluation and quality rubrics first-class deliverables in performance reviews, weighted equally with feature delivery.

One more question the exchange has to answer: where the eval work lands in the feature pipeline. It isn’t a phase you bolt on at the end. Some of it belongs before a feature is built, when you’re defining what “acceptable” means for the outputs. More of it lands during development, when error analysis tells you which failure modes are real versus imagined. And it keeps running after release, because production traffic will surface failures your golden dataset never imagined. Every one of those touchpoints draws from the same fixed weekly capacity, which is why the off-ramps have to cover the whole pipeline, not just the build.

Earlier this year, I wrote that The PM Role Isn’t Disappearing. Its Work Is Moving.. The work is moving toward deep product discernment, technical intuition, and direct engagement with model behavior. None of that settles into place while the old work is still sitting on the same desk.

 
If you cannot name what came off the list, your PM is carrying both jobs on personal stamina. That is not a strategy. It’s operational debt.

So before the evals land on a product manager’s desk, write both halves of the exchange: what they take on, and what comes off the list by your decision, not theirs. Product managers learning evals are stepping up to what modern software demands. The unfinished business belongs to the leaders above them.

When you hand your product managers AI quality, what work leaves their desk because you decided it should?