Your Code Is Fixed. Production Isn't.

The 14th piece in the Agentic AI in Practice series. The earlier pieces are about designing and gating an agent before launch; this one is about the gap that opens after launch — the one everyone misses. Your code is fixed; production isn't. 中文版：你说的「修好了」，是在哪台机器上？.

Your boss stares at the screen: "Didn't we fix that last week?"

The bug fix you merged last week — tests all green, code review passed, merged, deployed. This week's review meeting, the business owner pulls up a fresh conversation: the user typed two words, "thank you," and the agent fired back a canned fallback line, "Your issue has been logged, a human will follow up shortly." Your boss stares at the screen and drops one sentence on the table: "Didn't we fix that last week?"

The room goes quiet. Because from an engineering standpoint, it was fixed — the commit that reworked the classifier landed on main three weeks ago, "thank you" routes fast and correct locally, that one pytest line is glowing green. Review done, merge done. But in production, it's as if nothing changed. Someone mutters "maybe it never deployed." Pull the deploy log — clear as day, it deployed three weeks ago. Time-to-first-token p95 spiked to 18.5s, and the intent span alone eats 42.7% of the whole request.

Peel down through Langfuse layer by layer, and the root cause isn't in the code at all — the .env on the box we deployed to still points at a model name the gateway retired long ago. Every round of intent classification hits that dead model, takes a 503, then fail-opens back to the slow, dumb legacy classifier. The code was upgraded. The one line of config on that machine never came along.

That's the gap this piece is about, in one sentence: fixing the code ≠ fixing production. Config drift survives redeploy — it does not get fixed just because you redeployed.

.env, relevance thresholds, feature flags, reindex — git push carries none of them. You assume one deploy hauls them all up with it; in reality they each sit on some machine, in some index, inside some script nobody re-ran. Below are four real traps, each one a case of "the code is right, production is wrong," closing with a checklist you can carry into your next review meeting.

A single git push carries code and tests but not the .env model name, relevance threshold, feature flag, or reindex — four independent delivery steps, and missing one means "code right, prod wrong."

Even "thank you" hits the fallback — the model isn't dumb, that machine's .env is wrong

Get one thing straight first: when the simplest inputs — "thank you," "hi" — hit the fallback, the problem is almost never the model. It's a line of config on the deploy box, and the router is fail-open the entire time.

Replaying that "thank you" conversation, the first instinct is "the intent classifier wasn't trained well." But reproduce the whole chain and the router itself is 12 / 12 correct — feed the same 12 inputs straight into the classification logic and every one routes right. So the model is clean. Where do the slowness and the errors come from?

The answer is in the latency distribution. p50 is only 2.1s, looks fine; but p95 is 18.5s and max hits 25.7s. That "median normal, tail exploding" shape is the signature of one path retrying over and over. Split each trace into per-span latency and the intent span alone hits p95 14.7s, 42.7% of the whole request — and intent=unknown correlates hard with "slow": nearly all the slow ones failed to classify.

Hit the model name directly against the internal model gateway: 503. It takes a few seconds of staring at that response to register how stupid the cause is: the router defaults to qwen-turbo, but the tier the gateway actually ships is called qwen-turbo-latest — one suffix off. Every request hits qwen-turbo, takes a 503, and that except: fallback line quietly drops back to the old HybridClassifier, doubling intent latency. Nowhere in the chain does anything error; the logs are spotless — because fail-open's whole job is to "degrade quietly, don't bother the user." Except this time what it quietly degraded away was the entire point of last month's fix. Same gateway, three tiers tested live: qwen-flash answers in 0.42s, qwen-turbo 503s, qwen-max in 3.3s — the working ones are working fine, nobody just matched the default name to one of them.

The second spot is nastier. The LLM_CRITIC_MODEL env var was never set, so the critic falls back to the max tier that chat uses. The max tier is slow — a single critic pass takes 8008ms ≈ 8s and slams straight into the timeout. And the critic is fail-closed by design — on timeout, escalate to a human. Sounds safe, right? The result was every single refund got misrouted to a human because the critic timed out. A safety net meant to fire only when the model is "uncertain" turned into mass collateral damage over one unset model name. The fail-closed design isn't wrong — what's wrong is that it's running on a model that times out reliably. The safety net was hung from a mis-tied rope.

First-token latency: p50 is just 2.1s, but p95 spikes to 18.5s and max 25.7s; per-span, the intent segment alone is 42.7% of the whole (p95 14.7s) — one path retrying a dead model that 503s.

Here's the line worth nailing to the wall: .env does not travel with a code release — a code upgrade ≠ a config upgrade. And there's a sneakier trap under it: in dev you get used to uvicorn --reload, but --reload only watches .py files — it does not watch .env. You change .env on the server, the process never re-reads it, and "I changed it" is not the same as "it took effect" — you have to pkill uvicorn and truly restart for it to count. I've been burned by this one myself: changed .env, sat watching the logs for ten minutes wondering why nothing moved.

The fix isn't changing code logic, it's locking down config. In config.py, change the default from turbo to the tested-working flash; in .env, write the router and critic model names explicitly, no relying on defaults; in .env.example, add 7 lines of a startup checklist spelling out which vars must be set. The most important one: add a model-availability assertion to the startup preflight — the first thing the service does on boot is ping the gateway with every model name in config, and if any 503s, fail loud and refuse to start, instead of silently degrading to the old classifier and pretending everything's fine. Better it can't start than it goes live sick.

Your safety threshold might be a prop: passed 94 tests, never once fired in prod

A merged safety threshold with 94 green tests can be completely dead in production — because "tests pass" proves it exists, not that it's alive.

This trap is scarier than the last. The system has an off-topic guard: a cosine relevance floor that keeps "questions unrelated to this shop" out of knowledge-base retrieval, so the agent doesn't confidently answer nonsense off irrelevant docs. The code's there, the logic's clear, 94 unit tests all green.

In production it never fired once. That's not a guess — pull the off-topic trigger logs since launch and count them: that floor's block count is 0. A guard written into the design doc, on the acceptance checklist, protected by 94 tests, has a production block count of 0.

Dig down: the ES index's content_vector mapping is there, the field is defined fine, first glance says "vectors are all set up." But actually count the vector dimensions per document — 221 docs, 0 with a real embedding computed. The mapping fooled the first glance. Here's how the chain broke: the embedding-model line in .env is empty → the code never builds the embedding client → the ES kNN branch throws and silently swallows it → falls back to pure BM25 keyword retrieval. And that offline index_to_es.py script never computed a vector at all, start to finish. The mapping is an empty house built for vectors, and nobody moved in.

Pure BM25 can't stop off-topic. One test input is "do you have a Nike size-1 shirt" — nothing to do with this shop's business — yet relevance scored it 0.614, higher than most legitimate queries (which mostly land in 0.49–0.71). BM25 matched high-frequency words like "shirt" and "size," so the score floated up. Only cosine separates them: a genuinely relevant campaign query is 0.595, unrelated ones sit below 0.39 — in semantic space they were far apart all along.

The fix is two steps: first dry-run embed all 221 docs and confirm 221 / 221 have vectors; then calibrate against the real distribution and set kb_min_relevance to 0.52 — right between the relevant 0.595 and the unrelated 0.39.

For a delivery owner, the takeaway of this section is one sentence: a merged, green-tested safety mechanism can be entirely dead in production, because the index has no vectors and the fallback is a silent except. Next time engineering says "the off-topic guard is done, tests all pass," don't buy "tests pass" — ask the falsifiable question: how many documents in the index actually carry a vector right now? How many times has this threshold blocked anything in production since launch? A guard that's blocked 0 times is the same as no guard.

The content_vector mapping exists, but 221 docs have 0 real embeddings → the kNN branch silently excepts → falls back to pure BM25; off-topic "Nike size-1 shirt" scores rel 0.614, higher than legit queries — only cosine separates them (floor calibrated to 0.52).

Changed the KB but it never takes effect? Don't rush to reindex — you're fixing the wrong thing

When the business changes the knowledge base and the answer stubbornly won't change, the reflex is "the index is stale, reindex it" — nine times out of ten that's a category error, because the source of that answer isn't in the KB at all.

Real scenario: a user asks "can I get a price adjustment on this," and the agent replies every time with a fixed transfer-to-human line. The business owner edits the "price protection" knowledge entries several versions over, production doesn't budge. Engineering gets anxious too, gears up to reindex — rebuild the index, recompute, reload, the whole ordeal — and still nothing changes.

Because the answer never went through the KB. This path has three layers, and any one of them alone can keep it out of the KB:

• Layer one: the routing enum has no "price protection" class at all. The POLICY sub_topic enum has no price-related intent, so the LLM can only go by name similarity and jam it into the closest match, ACTION:price_diff_refund — routed elsewhere from the very start.
• Layer two: the line in that workflow is hardcoded. The private-channel branch that hits returns a written-in constant, _PRIVATE_TRANSFER (the transfer-to-human line), and this code never reads the KB. Edit the knowledge base ten thousand times, it won't glance at it.
• Layer three: even if the first two were right, there's a whitelist in the way. price_inquiry belongs to SHOP_DOMAIN but isn't in the SHOP_KB_ANSWERABLE whitelist → intercepted by the guided-selling stub, never reaching the KB-retrieval step.

Three layers each blocking on their own, and reindex solves none of them — it fixes "stale index," but there's no stale index here, only a code path that never touches the index.

This section gives you two diagnostics you can use immediately, far more reliable than reindex:

One: grep the answer string to its source. Take a distinctive fragment of that fixed line and grep it straight across the codebase. A hit in app/workflows/*.py or some router constant = hardcoded, the KB will never be read, editing the KB is wasted effort. A hit in a KB data file is the only thing that proves the answer really comes from the knowledge base. This step takes 30 seconds and saves you a whole round of reindex flailing.

Two: to judge whether the index is actually stale, byte-compare — don't guess. Compare the document body in ES against the source file byte for byte. Did it once for real: 6 docs all EXACT MATCH=True — one shot proves the index is identical to the files, the index is innocent, the problem is elsewhere. A bonus trap: don't use match_phrase to search a whole sentence in Chinese — Chinese tokenization shreds your phrase and returns a false negative that fools you into thinking "it's not in the index." The right move to deploy a new answer is restart uvicorn so config takes effect, not reindex.

The "price adjustment" question never reaches the KB — the routing enum has no price class, the workflow line is hardcoded, and a guided-selling stub whitelist each block it; reindex fixes a stale index, but a byte-compare (6 docs EXACT MATCH=True) already proves the index innocent.

The thing you think isn't live is already running on someone else's machine

"That data isn't live yet" — that promise only holds on your own machine; in a system sharing one production index, someone else's seed data may be answering your real users right now.

Another real complaint: a user in a gated-rollout wxapp (private-channel mini-program) asks "do you offer free shipping," and the agent answers "free shipping over 99." The user tops up to 99, places the order, and is then told by support that the private-channel threshold is actually 399 — off by 300. A fabricated promise turns straight into a complaint, and it's one the agent gave confidently after "checking the knowledge base," which is harder to explain than not answering at all.

Where did "over 99" come from? It's the free-shipping threshold on the public Douyin channel. Two errors stacked:

• Private-channel retrieval never filters by channel. Private and public docs are loaded into the same ES index — 286 wxapp docs and 249 douyin docs mixed together. A private-channel user asks, but retrieval searches the whole index, so Douyin's "99" gets recalled naturally.
• That Douyin seed batch never should have reached production. The public-channel launch was pushed back, so that douyin data was supposed to stay in test. But index_to_es.py --recreate reads the entire xlsx by default, and the Douyin seed rode in that way.

Underneath sits a harder problem: no test / prod isolation. ES_URL in .env points straight at the production cluster, and the index names on both sides are hardcoded identical. So anyone running index_to_es.py --recreate once on their laptop is deleting and rebuilding the production index — and for those few seconds of rebuild, production is empty and every user hits the fallback. This dirty data took three passes to clean: 249 → 286, because the second pass forgot the --douyin-xlsx "" argument and the Douyin seed came back to life again; only the third pass wiped it clean.

The verdict is blunt: a partition field with no channel + a test/prod-shared index + one laptop whose .env points at production = a single slip of --recreate can delete production. A bare general fallback bucket lets data from different channels bleed into each other; a promise like "the seed isn't in production" simply doesn't hold on someone else's machine — because between "someone else's machine" and "production," there may be nothing but one identical line of ES_URL.

Detection toolkit: one release checklist + 5 questions to press in the review meeting

Flip those four traps over and you get a release checklist — its core is admitting that model names / thresholds / feature flags / reindex are all "non-code deliverables," and git push carries none of them.

When you ship a change, beyond the code, confirm item by item whether these things that live independent of the code actually came along:

Deliverable	Who carries it	Confirm before go-live
Code logic	git push	Tests green, review passed
Model names (router / critic)	Deploy box .env	Preflight-pinged, not a dead model
relevance / critic thresholds	Config + index	Index actually has vectors, threshold calibrated to distribution
feature flag / whitelist enum	Config + code constants	Target intent really is in the answerable whitelist
KB / reindex	Offline script + ES	Byte-compare matches, answer source is the KB not a workflow

Then five questions you can carry into the review meeting and press on the spot:

1. "Thank you" / "hi" also hits the fallback = the router is fail-open the whole time. Check the .env model names on the deploy box first, don't rush to change code — when the simplest inputs are wrong, the problem is config, not the model.

2. Make engineering prove the guard is "alive," not "present." Don't accept "tests pass"; ask two falsifiable numbers: how many documents in the index carry a vector right now? How many times has this threshold blocked anything in production since launch?

3. It changed but doesn't take effect — do two things before reindexing. Grep the answer string to its source (a hit in workflows/*.py = hardcoded, editing the KB is pointless); then byte-compare ES vs the source file to judge whether the index is really stale. Don't treat reindex as a cure-all.

4. The release checklist must include non-code items. Model names, thresholds, feature flags, reindex — git push carries none of them. Who is responsible for checking them off item by item after deploy? Write it into the process.

5. Three questions for the vendor. To change one KB answer, tune one threshold, or flip one feature flag — what steps does each go through, how long, and who owns it, before it actually takes effect in production? If they can't answer, they don't know where their own config drift is either.

What these five share is one move: stop looking at "is the code right," look at "what state is that machine / that index / that config in, in production." The code lives in your repo. The truth of production lives elsewhere.

Grab this piece's checklist

If you want to put this straight to work on your next release and review — without re-reading the whole piece each time — I've put together a kit for readers who made it this far. Send me the keyword "PROD-TRUTH KIT" and I'll send the pack: a one-page release checklist (model names / thresholds / flags / reindex, checked off item by item), a card version of the 5 questions to press, and a quick-reference table of the four "code is right but production is wrong" symptom classes. All of it judgment tooling ground out of a year on customer-service Agent projects.

(Channels in the footer — X or email both work.)