50 Rows at 96%, Ship It? Size Labeling by the CI Lower Bound, Not the Pretty Number

Post #2 of the After Launch series. The previous post, The 96% Turned Red the Moment We Split It by Channel, covered which rows to draw — channel drill-down, risk-layer over-sampling. This one answers the other half: for each cell, how many rows is enough. 中文版：标了 50 条、96% 做对，能上线吗？。

This year, designing the labeling gate for a footwear-and-apparel customer-service agent project, the first rule I pinned was: read the confidence-interval lower bound, not the point estimate.

The rule was forced out by a recurring scene. The most common line in a review meeting is — this intent has 50 rows labeled, 96% correct, ready to ship? The number looks great; no one in the room objects.

But 96% out of 50 has a statistical lower bound of only about 86%. In other words, you have no confidence it actually hit 90% — it just looks like it did. A small sample's point estimate lies: the fewer the rows, the wider the gap between that pretty percentage and the true level behind it — and that gap sits exactly in the rows you didn't label.

So the labeling-volume question was never "how many did you label, what's the accuracy," but "have you labeled enough to draw a conclusion against the launch threshold." And that hides two things most teams haven't thought through: what the required volume depends on, and — when to stop.

50 rows at 96% can't ship — a small sample's point estimate lies

An accuracy number has two readings: the point estimate (that 96%) and the confidence-interval lower bound (the line you're confident you're not below). What you ship on is the lower bound, not the point estimate.

Why? Because the point estimate is a function of luck. 48 right out of 50 is 96%; but if two of those were right by chance, the true level might sit around 86%. The smaller the sample, the more room for that "by chance." Reading the lower bound strips the luck out — it answers "in the worst case, am I still at 90%," not "what score did this particular batch happen to get."

The point estimate is a function of luck — whether you clear the threshold depends not on your score, but on how many questions you sat.

Same 96% point estimate: at 50 rows the lower bound is ~86%; to push it past 90% you'd label ~140. Whether you're above the threshold depends not on your score, but on how many questions you sat.

That's where the "read the lower bound" rule came from, and it's a this-week move: whenever someone brings an accuracy number to request launch, ask one thing — is this a point estimate, or a confidence-interval lower bound? "It's just what we got from labeling N rows" means the number has no standing in a launch decision.

How much to label depends on how good the agent truly is — a table that "explodes"

Put the conclusion on the table first: how much to label isn't a fixed number you pick — it depends on how well the agent actually does on that intent, and the closer to the threshold, the more the required sample size explodes.

The table I kept reaching for on the project (threshold = "accuracy lower bound clears 90%"):

Agent's true accuracy on the intent	Rows to label / intent	Read
100%	~35	very cheap
98%	~53	cheap
96%	~84	acceptable
94%	~204	expensive
92% or below	700+, or never passes	stop labeling — fix the agent

Below the threshold, more labeling just burns effort on an intent that will never pass — the confidence interval is calling a stop for you.

The key insight is in the last row. At a true accuracy of 92%, 700 rows might not push the lower bound past 90%; at exactly 90%, it mathematically never passes. Here the confidence interval isn't being difficult — it's honestly telling you one thing: it's not that you lack data, it's that the agent isn't good enough; labeling more is wasted, go fix the agent first.

This line divides labor between "labeling" and "fixing the model": near the threshold (96% up), more labeling buys a shippable conclusion — worth it; below it (92% down), more labeling just burns effort on an intent that will never pass. A healthy admission process should call a stop right here, not let the labeling team grind forever.

Labeling volume is a dynamic process: a CI stopping rule

Don't pin a fixed number for each intent ("everyone gets 100 rows") — the volume should be decided dynamically by the confidence interval: label more where it's needed, don't waste it where it's done, call a timely stop on what can't pass.

The stopping logic I put in the gate looks like this:

after each batch → compute Wilson lower / upper bound:
    lower ≥ threshold(90%)   → cleared, stop, fall back to routine spot-checks
    upper < threshold(90%)   → will never clear, stop, report "agent needs work"
    interval straddles the bar → no conclusion yet, next batch labels a bit more
    sample < 30 (floor)      → "accumulating," wait a few weeks before judging

The value of this logic is routing finite labeling effort to the cells that need it most. An intent that clears easily (high true accuracy) hits the lower bound in tens of rows and stops on the spot; an intent stuck near the bar gets a few more batches to converge; and an intent whose true accuracy sits below the bar — the moment its upper bound can't clear 90%, the CI tells you outright to "stop labeling," pulling effort out of a pit that will never pass.

Detection move: ask "how many rows per intent — a fixed number, or labeled until the CI concludes?" An answer of "everyone gets 100" means you're wasting effort on easy intents and under-labeling the hard ones — losing on both ends.

Three things you can do this week

1. Kick "point estimate" out of launch decisions. Next time someone requests launch with an accuracy figure, ask "point estimate or confidence-interval lower bound?" — accept only the lower bound; ready means the lower bound clears the threshold.
2. Bring the "true accuracy → rows to label" table into the review meeting. Make "how much to label" a data-driven quantity, not a number pulled from the air; and agree: an intent still not clearing its lower bound after several hundred rows gets stopped and reported "agent needs work," no more effort poured in.
3. Wire a CI stopping rule onto labeling volume. Compute the lower/upper bound after each batch and, per the four branches above, decide stop, continue, or call it — routing labeling effort to the cells that need it most.

Back to the opening question. "How many rows is enough" stumps people because most treat it as a traffic problem, when it's really a "how confident are you" problem. Tie labeling volume to the CI and you get three things at once: an honest launch gate, saved effort, and a signal that proactively tells you "stop labeling — go fix the agent."

The next post covers the link in this mechanism most easily dismissed as grunt work, and yet is the ground truth — labeling itself: who labels, how, and when two people disagree on the same row 30% of the time, whether your "answer key" still counts at all.

---

If this turned "how much to label" from a gut call into a CI rule that calls a stop, send me the keyword "VOLUME KIT" and I'll share the "true accuracy → rows to label" table plus the CI stopping-rule template.