A buyer used to ask for a demo. The seller would open a laptop, walk a happy path, and everyone would nod. That ritual is ending. The same buyer now asks for an eval: a scored run of the agent against the buyer's own tasks, with a pass rate the seller cannot talk around.

The demo is a performance

A demo is a rehearsed path. The seller picks the inputs, so the agent looks sharp. Anything that might fail never gets invited on stage. The buyer watches a highlight reel and leaves with a feeling, not a measurement. Feelings do not survive the first live task.

The demo also burns the one thing a technical buyer has least of: attention. Thirty minutes of watching someone type into a chat box proves almost nothing about what happens when the model meets a messy, novel request at two in the morning. A demo answers a single question: can this ever work. A serious buyer stopped asking that months ago.

The eval is a number

An eval is the opposite shape. It starts with a task set: fifty or five hundred examples that look like the buyer's actual work. Each example has a known-good answer or a graded rubric. The agent runs against all of them, and a number comes out the other side. Pass rate, time to answer, cost per task, refusal rate. You can rerun each one, compare two versions, and audit the scoring. A new release either holds the score or drops it, and a drop shows up the next morning.

A demo shows the best path. An eval shows the failure rate.

The number is what makes the eval uncomfortable to sell. A demo can be polished until it glows. A pass rate cannot. Either the agent clears the threshold on Tuesday's data or it does not, and the seller does not get to pick the data.

Buyers changed first

The shift did not start with vendors. It started with a procurement team that got burned once. They bought an agent off a slick demo, watched it fall over on their own documents, and decided never again. Now the request for proposal lists the eval harness before it lists the feature set. They want the test, the data, and the score, in that order.

A founder can run the same held-out ticket set against three vendors and rank them by pass rate. The tooling is cheap now. Evals were an internal research practice at a handful of labs. They are now a thing a small team can stand up in an afternoon. The hard part is no longer building the eval. It is building the product that survives one.

What to ship instead

Sellers should stop treating evals as an objection to handle and start treating them as the front door to the product. The vendors who win the next round are the ones who ship the harness alongside the agent.

  • A public task set. The examples the agent is claimed to pass, in a form the buyer can run.
  • A pass threshold. A stated bar the agent clears, with the numbers to back it.
  • A rerunnable harness. The buyer runs it on their own data, not the seller's cherry-picked set.

None of this requires exposing the model weights or the secret sauce. It requires admitting what the agent does badly, which is the part most sellers would rather skip. Skipping it stops working the moment a buyer brings their own data to the call. The vendor who hides behind a demo is protecting the gap between what the demo shows and what the product does.

What this costs us

We sell the middle: the work between a raw model and a finished job. For us the demo was always a weak sales motion, because the work we automate is exactly the kind that demos flatter and live work punishes. Evals suit us. We can state a pass rate on a held-out set, hand over the set, and let the number do the talking.

The risk is the same as the opportunity. Once buyers can compare everyone on the same evals, the vendors with inflated demos and no numbers get filtered out fast. That is fine for us. It is also the point. A market that runs on measured pass rates rewards the people who do the work, not the people who stage the best afternoon.