Evals Are the New Product Spec
In normal software, a product spec says what should happen.
In AI software, that is not enough.
When the product is probabilistic, agentic, or adaptive, the more important question becomes: how will we know if it did the right thing?
That is why I think evals are becoming the new product spec.
Not a replacement for product thinking. The opposite. Evals force product thinking to become more precise.
A Spec Describes the Feature. An Eval Describes the Taste.
A traditional spec might say:
The system should generate a clinical note from a visit transcript.
That sounds clear, but it is not enough.
What makes the note good? Is it concise? Is it complete? Does it preserve clinical reasoning? Does it avoid overclaiming? Does it match the specialty? Does it use the clinician's preferred structure? Does it leave out irrelevant conversation? Does it surface uncertainty? Does it avoid inventing things that were not said?
Those are not implementation details.
Those are the product.
The eval is where taste becomes operational. It turns "make it good" into dimensions the system can be measured against.
Evals Capture Judgment
The best evals are not generic.
They capture the judgment of the people and company building the product. They encode what the team believes quality means for a specific customer, in a specific workflow, with specific risks.
For an AI scribe, a generic eval might check grammar or section formatting.
A useful eval asks whether the note would actually save the clinician time, whether the assessment reflects the visit, whether the plan is faithful to what was discussed, and whether the output is safe to review quickly.
That difference matters.
The eval is not just QA. It is a compressed version of the product team's understanding of the customer.
Agents Change the Shape of Work
This becomes even more important as AI moves from chat to agents.
Chat is relatively easy to understand. You ask a question. The system answers. You judge the response.
Agents are different. You give intent, and the system goes off to do work. It may search, write, call tools, edit files, generate code, make decisions, retry, and produce an output after many steps.
That changes the role of the human.
The human is no longer only typing instructions. The human is steering a system.
This is already obvious in coding. At first AI completed lines. Then it answered questions. Then it started editing files. Now you can delegate a task and review the result. The same pattern is going to happen across product, operations, marketing, support, sales, finance, and healthcare administration.
The product question becomes: how do humans orchestrate many agents without losing understanding or control?
The Interface Becomes a Control Room
If agents are doing real work, the interface cannot just be a chat box.
A chat box is fine for one conversation. It breaks down when you have ten agents working on ten tasks, each with different state, risk, confidence, and dependencies.
People will need agent inboxes, review queues, task boards, traces, checkpoints, permissions, approvals, and ways to intervene. They will need to see what an agent did, what it assumed, where it got stuck, and where it needs help.
The interface becomes less like a search bar and more like a control room.
This is a product design problem as much as a technical problem.
What should the human see?
What should be hidden until needed?
Where should the system ask for confirmation?
How much autonomy is appropriate for this task?
What does failure look like, and how does the user recover?
These questions are the new workflow design.
Evals Make Delegation Safer
Delegation only works when trust has a structure.
If I ask an agent to write a draft, I can review it manually. If I ask it to coordinate a workflow, touch customer data, generate code, or make decisions across systems, I need more than vibes.
I need evals.
I need to know what success means before the agent starts. I need checkpoints that catch failure early. I need a way to compare outputs over time. I need to know whether a cheaper model is good enough, whether a new prompt made things worse, or whether a workflow performs well only on easy cases.
Without evals, AI products become demo machines.
With evals, they become learning systems.
The PM Role Moves Upstream
This changes product management.
The PM cannot only write user stories and acceptance criteria. The PM has to help define the evaluation system.
That means asking:
- What outcome matters?
- What does good look like?
- What mistakes are unacceptable?
- What should be optimized, and what should be protected?
- What examples represent the real world?
- What human judgment should be captured?
- What level of autonomy is appropriate?
These are product questions. They are also technical questions. In AI products, those categories start to blend.
The best PMs will not need to become full-time ML engineers, but they will need to understand evals well enough to shape the system.
Taste Becomes Infrastructure
I like the idea that taste can become infrastructure.
Usually taste sounds personal, soft, or hard to measure. But in AI products, if you can define taste clearly enough, the system can start optimizing toward it.
That is what makes evals so powerful.
They are how a team turns judgment into a reusable asset.
They are how a company improves without relying only on memory and meetings.
They are how agents become useful without becoming chaotic.
The product spec still matters. But in AI, the deeper spec is the eval.
Because the most important thing is not just what the system does.
It is whether the system learns to do the right thing, in the right way, for the right customer.