Tag

Evaluation

6 posts

Make the Agent Get Sharper With Use, Not Dumber — Spinning Up the Data Flywheel

In labeling we found back-to-school promo questions spiked, but within them "how do I get the gift-with-purchase" was never answered well — so we fixed that KB specifically, and next round that class climbed. That's the data flywheel. In 5 minutes you'll see the first five posts are five spokes of one wheel; in 10 you'll spot why a flywheel spins in place (data piles up, never ships); in 20 you'll have a feedback loop ranked by frequency × cost that also forces you to split intents.

After Launch · 12 min read · EN · 中

A Pretty Accuracy Number Hid Dozens of Money-Moving Errors — How to Read the Eval to Ship

On a money-moving project I ran, the overall accuracy looked great; but pull the money-moving intents out on their own and the wrong-action rate was alarming — dozens of money-touching errors sat there the whole time, hidden by one blended number. In 5 minutes you'll see through "one accuracy figure to request launch"; in 10 you'll put a separate wrong-action gate on money-moving errors; in 20 you'll have a launch-decision flow: CI lower bound + per-scenario version cut + per-channel ramp.

After Launch · 13 min read · EN · 中

Your Dashboards Are Green While the Agent Quietly Gets Dumber — Post-Launch Silent Drift

Running a customer-service agent at a consumer-tech company, I learned one counterintuitive thing from watching the metrics: they're never a flat line. Every product launch, every back-to-school season, the question distribution shifts and a wave of new phrasings pours in — the agent quietly gets dumber, and not a pixel of it shows on the green CPU / QPS / latency dashboards. In 5 minutes you'll see through "dashboards green = healthy"; in 10 you'll have 6 leading signals that fire weeks before complaints; in 20 you'll turn the eval set from "frozen at launch day" into one that re-samples current traffic.

After Launch · 13 min read · EN · 中

Your Labels Are Your Ceiling — One "Swap Half a Size Up" Gets Three Answers From Two Agents

A customer's one line — "I want to swap half a size up" — hides four calls: exchange or return, intercept the shipment or not, refund the price difference or not, and against which price. Two skilled agents label the same 50 rows back-to-back and agree on only 35 — your 96% accuracy was measured with a ruler that's only 70% self-consistent. In 20 minutes you'll have a flow for measuring agreement first, then writing rules like "which price the difference is refunded against" into a rubric.

After Launch · 13 min read · EN · 中

50 Rows at 96%, Ship It? Size Labeling by the CI Lower Bound, Not the Pretty Number

"We labeled 50, 96% correct — ship it?" No — the statistical lower bound is only 86%. When I designed the labeling gate for a customer-service agent, the first rule I pinned was: read the confidence-interval lower bound, not the point estimate. In 5 minutes you'll see through the small-sample 96% mirage; in 10 you'll have a "true accuracy → rows to label" table that explodes near the threshold; in 20, a CI stopping rule — label more where it's needed, don't waste it where it's done, and call a stop when the answer is fix-the-agent.

After Launch · 12 min read · EN · 中

The 96% Turned Red the Moment We Split It by Channel — Two Fatal Failures of Eval Sampling

The vendor drew 200 rows at random and reported "96% accuracy." I split those 200 by channel — in the small public-domain channel, the price-difference cases were nearly all wrong, and the high-traffic private channel had diluted it into a pretty number. In 5 minutes you'll see through both failures of random sampling (missing the tail + a big channel drowning a small one); in 20 you'll have a stratified frame with channel drill-down and risk-layer over-sampling.

After Launch · 13 min read · EN · 中
← All posts