The 96% Turned Red the Moment We Split It by Channel — Two Fatal Failures of Eval Sampling

Post #1 of the After Launch series. Agentic AI in Practice is about building an agent and shipping it; this new series is about what comes after — sampling, labeling, evaluation, and the feedback loop that keeps an agent right. It is the biggest blind spot in today's enterprise agent market, and the hardest thing to paper over with a demo. This first post starts with the least glamorous step, the one where the gap shows first: sampling. 中文版：随机抽样报的 96%，按渠道一拆就红了。

You think the stronger AI gets, the fewer people you need? The opposite — where agents are built with real money, labeling and evaluation teams are growing, not shrinking (China's AI foundational-data-service market grew ~47% CAGR 2019–2025, reaching 95,000 professional annotators by end of 2025). Top labs are growing thousand-person eval teams against the tide, because launch is where evaluation actually begins. And most projects ship and abandon.

Where does the abandonment start? With how you draw those few hundred evaluation rows.

This year, running a customer-service agent project for a footwear-and-apparel retailer, the vendor emailed a weekly "96% accuracy." I asked for the eval set and asked one thing: how were these 200 drawn? Answer: at random from live traffic.

I did the one thing they hadn't — split those 200 by channel. The private domain was the vast majority; the public-domain channel (a short-video marketplace) had only twenty-odd rows. The private channel scored 95%, fine; but in those twenty-odd public rows, a few "can I get the price difference refunded" cases were all wrong: the public channel has no price-protection policy, yet the agent answered with the private-channel script, "sure, I can file it for you."

The moment we split that 96% by channel, the public cell went from green to red — and the room realized the vendor's pretty number had never been broken out by channel. The private channel's large volume had weight-diluted a high-risk intent that was actively crashing in the public channel into a gate-clearing 96%.

That's the truth about random sampling: it isn't just imprecise — it uses the "fairest" method possible to hide the very risk you most need to see. And sampling is the first brick of the whole evaluation system.

Random sampling's first failure: it misses the rare high-risk tail

Random sampling measures the common case — where the agent actually fails is the rarest.

Production traffic is extremely long-tailed. A customer-service agent handles tens of thousands of conversations a day; ~90% are read-only lookups like "where's my order." The ones that get someone out of bed at 2 a.m. are the few money-moving writes — refund, cancel, address change, intercept. Those are often a fraction of one percent of traffic. In a random 200, their expected count is 0 to 1.

Random sampling splits evaluation evenly across traffic — so the high-risk writes that most deserve scrutiny get diluted into statistical noise.

So the dashboard reads 96%, but it measures almost entirely the part the agent is least likely to get wrong. The risk that keeps you up — one wrong refund, one mishandled compliance case — sits whole in the hundred-plus rows you never sampled. An eval metric's credibility is capped not by the model, but by the sampling frame.

Random sampling's second failure: a big channel drowns a small one

Random sampling makes a second, sneakier error — merging different channels into one number, letting a high-traffic channel bury a low-traffic channel's crash. That's exactly what the opening 96% fell into.

The channel is a real variable in accuracy: different channels have different KB sources, policy definitions (exchange / shipping insurance / small refund / price protection), and retrieval quality, so the same intent can have different correct answers across channels. Private-domain members have price protection, the public channel doesn't — the same line "sure, I can file the price difference" is right in private, wrong in public.

Merge two channels into one random sample and the high-traffic private channel weight-dilutes the low-traffic public one, the total looks past 90% while public is crashing. Worse, the public channel is new and small — a random draw can't even reach a 30-row floor for it, so that crashing channel simply doesn't exist in the evaluation.

The private channel's volume weights a crashing public channel up to 96% — each channel must be framed and gated on its own; ramp a channel only when it clears on its own.

So the eval set can't be drawn merged — it has to be framed with channel drill-down: each channel framed and judged on its own, ramped only when it clears — not sliced by one cross-channel total.

The fix is a layered frame: channel drill-down + risk-layer over-sampling

Put the conclusion on the table first: don't random-sample the eval set by traffic share — stratify by "channel × intent × cost of getting it wrong": a token slice of common intents, deliberate over-sampling of rare high-risk ones.

A workable sampling frame's logic looks like this:

for channel in [private, public]:            # channel drill-down: framed & gated separately
    for intent in intents(channel):
        if intent in asset_action_intents:   # refund/cancel/address/edit-return/nudge/intercept…
            census(intent, channel)          # no sampling, cap 150; not one money-moving row missed
        else:
            draw_until_CI_lb_over_90(intent, channel, floor=30)   # consult intents: stop when you can conclude

Two disciplines:

• Over-sample the risk layer. Pull the money-moving writes out and over-sample them past 30% of the eval set — one wrong refund does not cost the same as one wrong "where's my order," so why split evaluation evenly by traffic.
• Each "intent × channel" is its own cell. Counted and gated separately, cells under 30 shown as "accumulating," never a merged number papering over the gate.

For a week of ~3,000 conversations: risk ~20% (≈600, 7 intents), consult ~80% (≈2,400, ~15 intents); a first full round of ~1,500 labels is enough — and that 1,500 is mostly private; the public channel is new, most of its cells haven't reached the 30-row floor, so accumulate and judge it separately, not folded into the private denominator. This is why stratified sampling saves the most effort: labeling volume grows with "intents × channels," not with traffic.

One kind of sampling must stay hidden from the agent

Almost no one thinks of this layer, but it's what separates evaluation from gaming the score.

The moment the agent — or the vendor — knows which samples get audited and scored, evaluation gets optimized to please the audit set rather than to do the job right. Any measured system drifts toward what's measured. Fix the same intents and channels every time and the optimization presses exactly where it's watched, while the unwatched corners keep rotting. This is reward hacking in the evaluation stage: you think you're testing capability, you're testing exam performance against a known question bank.

So two operating disciplines: rotate the specific samples, keep randomness, don't let them be predictable; inject the risk layer unannounced into live traffic as a blind test. The eval set can be a public method, but not a public question bank.

Three things you can do this week

1. Re-compute the eval set split by channel, once. Take any merged accuracy figure and break it out by channel — a crash in one channel that a big channel diluted shows up the moment you split. No new tooling, just the rows you already have.
2. Add a risk layer and a channel frame. Over-sample those 7 money-moving write intents past 30%; make each "intent × channel" its own cell with a 30-row floor. Change "sample by traffic" to "sample by channel × cost."
3. Build a fixed baseline set, judged per channel. Freeze a risk-layer, channel-split eval set as baseline; re-test every change on the same set; ramp a channel only when it clears on its own.

Back to the opening gap. Why do top labs grow thousand-person eval teams as AI gets stronger? Because they figured out long ago: the model only sets the agent's ceiling; sampling, labeling, evaluation, and feedback are the mechanism that decides whether it slides downhill after launch. Launch isn't the finish line — it's where evaluation begins.

Sampling is the first brick. The next post covers the second — how much to label: given the traffic, how many rows is actually enough, and why the answer isn't "a percentage of traffic."

---

If this turned your eval set from "randomly mixed" into a channel × risk stratified frame, send me the keyword "SAMPLING KIT" and I'll share the stratified-frame template plus the 3-question review-meeting checklist.