Deploy and Abandon — The Costliest Misconception in AI Agent Projects

Fourth piece in the Agentic AI in Practice methodology series. The first three pieces broke down "can the design ship" — L0-L3 grading, the five L2 architecture decisions, the fail-closed Critic deep dive. This one breaks down "after the design ships" — why 90% of projects that claim to have "shipped an AI Agent" are, structurally, a quiet conspiracy between vendor and buyer to skip ongoing operations. 中文版：上线即放养：AI Agent 项目最贵的认知陷阱.

"We're Not Apple, We Don't Need This Much Detail" — What a B Grade Actually Cost

My boss graded my customer-service Agent Critic design a B. The exact reasoning: "This is the kind of thing Apple-scale companies need. We're not at that level yet."

I had spent 40 minutes walking through Critic-on-failure escalation, the 2D threshold table, eval sets with a miss rate under 1%. Thirty PPT slides. The verdict was B, plus one sentence of industry judgment.

I've heard that sentence in several other project post-mortems. Wording varies — "that's a Big Tech move," "we don't have that scale," "later, later." Underneath it's always the same call: safety, backstops, continuous eval are luxuries, the company isn't ready for them yet.

That sentence is the costliest misconception in AI Agent adoption right now. Companies that swallow it gather in a post-mortem six months later and ask "why did nobody think about this earlier."

The truth runs the opposite direction — the less you are Apple, the more you need Critic, monitoring, backstops, eval. The logic takes a minute:

	An Apple-scale company	Your company (ordinary mid-size enterprise)
Who catches AI mistakes	1000-person review team + legal + PR + insurance	Nobody
One misfired $980 refund	Invisible in the quarterly	A week of these ≈ one employee's monthly salary
Post-mortem mechanism	Quarterly review committee	The CEO asks "what happened" next week
Error budget	Effectively infinite	First major incident = project gets killed
Regulatory exposure	Legal triages instantly	One social-media complaint hits the news

Apple can run a slightly weaker technical Critic because it has 1000 humans doing manual review. Your company has three people, which is exactly why the technical Critic has to be designed right — it's replacing the 1000 reviewers you can't afford to hire.

"We're not at that level yet" is logically inverted. The correct sentence is: "We're not at that level yet, which is precisely why we can't skip this."

Every "deploy and abandon" Agent project I've seen this year has the same line from the same kind of boss in its background. This piece is for the executors stuck under that line — to put the math on the table so you can push back next time it comes up.

Demo Passes Tests ≠ Product Ships: Six Skipped Environments

The vendor's demo runs the refund flow. The internal "AI Adoption Office" snaps a photo for the corporate WeChat. Many companies call that "we shipped an AI Agent."

But between a demo running and a product shipping, there are six environments. Skip one and that one is the hole the incident comes through.

#	What the demo phase does	What the product phase needs (90% of projects skip)
1	Runs the happy path (the demo conversation)	Covers failure paths (API timeout, LLM rate-limit, hostile user, adversarial prompts)
2	One-time demo	Continuous eval (data drift, model upgrades regressions, prompt-change verification)
3	No backstop, errors are errors	Critic + human queue + escalation SLA
4	Watches a demo video	Observability: error rate, latency, confidence distribution, token-spend dashboard
5	Frozen after launch	Prompt version control + canary rollout + one-click rollback
6	No accountability	Audit trail: every write op traceable to "who approved, why, who's responsible"

The single box on the left is the vendor's entire deliverable, the one that closes the contract; the six on the right are what production actually needs, dropped from the SOW by default. Click to view full-size.

Each of these six costs money — and not a one-time launch cost, ongoing monthly ops cost. A stably-running L2 customer-service Agent with all six in place runs about 3-5 FTE (1-2 prompt engineers, 1 business labeler, 0.5-1 SRE, 0.5 model iteration), which at a Tier-1 city headcount cost is roughly $12,000-$20,000 per month.

That sounds like a lot. Now compare incident cost. One misfired $980 refund is the money part — the bigger number is what follows: integration freeze for 1-2 weeks, customer churn, a CEO post-mortem, the credibility of the innovation team going to zero. Conservatively, a single incident's hidden cost starts at $60,000-$120,000. Six months of ops cost = $72,000-$120,000. One incident = $100,000+. The math never works out in favor of skipping.

Vendors don't bring up these six items when pitching, because bringing them up loses the deal. The customer's budget line is "implementation fee," not "monthly ops." Vendors quietly drop the six from the SOW, deliver a demo-runnable design, and the rest goes into the "we'll handle it when it comes up" pile.

The customer side doesn't bring them up either, because bringing them up loses the budget. The AI project was already pushed through by an under-pressure innovation team — adding a monthly ops line makes the CEO ask "is this project saving money or spending money."

Both sides understand, neither speaks, the design ships. That's the actual meaning of "deploy and abandon" — it's not that nobody wants to do it right, it's that both sides defaulted to not doing it.

"Deploy and Abandon" Is a Vendor-Buyer Conspiracy — Short-Term Win-Win, Long-Term Lose-Lose

Calling this a conspiracy isn't rhetoric, it's a structural fact. The short-term incentives on both sides align exactly:

Vendor side —

• Sales KPI is closed deals, not renewals; demo sells fast, ops doesn't enter the commission calc
• Pre-sales engineers write the proposal, post-sales catches the fallout; pre-sales has zero incentive to put ops in the SOW
• The industry is in land-grab phase; market share trumps delivery quality

Buyer side —

• The CEO wants "we shipped AI" — for the quarterly report, the roadshow deck, the industry conference, the faster the better
• The innovation team wants "project shipped on schedule"; post-launch ops failures are "the business team's problem"
• The business team wants its "automation rate KPI"; everyone looks at month-1 numbers, nobody at month-6

Five roles, five goals, not a single role's KPI is tied to "the system is still alive in six months." So nobody owns "six months from now" — which is the production mechanism for deploy-and-abandon.

Who pays the long-term bill? When the incident hits, everyone pays:

• Vendor loses the customer (no renewal next year) + reputation damage
• CEO gets asked at the board ("the AI project you said was live — six months later it's already broken?")
• Innovation team gets penalized ("how did you scope this without thinking about that")
• Business team rolls back to a BOT or human-only path; sunk cost in the original build
• Customer churn (a brand that misfired one $980 refund visibly takes a measurable hit on repeat purchase)

The long-term bill far exceeds the ops cost that was originally skipped — but because it's a bill that arrives six months later, no short-term decision mechanism prevents it.

There's exactly one way out: write ops terms into the contract before signing. The next section gives you four questions to ask in the vendor review, to put that bill on the table early.

Four Questions to Ask in the Vendor Review — Any Unanswered = You're Buying a Demo

At the next vendor review, after the deck has been flipped, ask these four questions. The point isn't to grill anyone — it's to lock the post-launch contract terms in advance. If the vendor can't answer, the design didn't account for these in the first place — don't sign.

Question 1: On day 30 after launch, how is the error rate computed? Who looks at it? How often?

A passing answer covers three things: the definition of error rate (per order? per conversation? per write op? include user-changed-their-mind cancellations?), who looks (customer business team? vendor SRE? jointly?), and how often (real-time dashboard? daily report? weekly?).

Failing answer: "We log everything — pull whatever you need." Translation: nobody is actively watching after launch.

Question 2: When the model or prompt ships a new version, how do you canary, how do you roll back?

A passing answer gives an operational flow: what percentage of traffic the new version sees first (10%? 1%?), the canary period (1 day? 1 week?), the rollback trigger thresholds (error rate up X%? escalation rate up Y%?), and the rollback execution time (seconds? minutes? hours?).

Failing answer: "We test thoroughly before any change." Translation: no canary mechanism. A new version going live = 100% of users hit it directly. This is the core reason a lot of LLM integration projects blew up last year.

Question 3: How are monthly eval samples drawn and labeled?

Eval is not "system self-check" — it's external human-labeled ground truth. A passing answer covers: sampling method (random? stratified by business type?), sample size (500/month? 1000?), labeler identity (business expert? outsource? joint team?), inter-labeler agreement protocol (double-label + adjudication?).

Failing answer: "Our model self-evaluates." LLM self-eval is this generation's most common fake metric — it doesn't even dodge the circular-reasoning trap.

Question 4: After launch, how many FTE keep this design alive? Who pays?

The most important question, saved for last — because the vendors who answered the first three well often can't get this one out.

A passing answer is a concrete number plus accountability split: total FTE (a typical L2 customer-service Agent: 3-5), role breakdown (prompt engineer, business labeler, SRE, model iteration), who pays (customer in-house? vendor on-site? hybrid? monthly fee?).

Failing answer: "After launch the design is self-driving, basically no human intervention needed." End the review. Any LLM system described as "self-driving" is a synonym for "abandoned."

Two out of four unanswered — that's a demo proposal, not a product proposal. Make the vendor redo it, or change vendors. If the budget really won't stretch, narrow the scope (start with one intent, one write op) — don't ship a full-scope design with six skipped environments. The first is slow. The second is a landmine.

Five Self-Check Items — Is Your Current Design "Deploy and Abandon"?

If a design is already running inside your company, hold it against these five. Any yes — the design is on the "deploy and abandon" path:

1. Nobody can say in 30 seconds how many cases the Agent handled yesterday, how many errored, which ones errored — no observability
2. The prompt has been changed N times post-launch, never once through canary, and nobody can recall "what the launch-day prompt looked like" — no version control
3. Eval hasn't run since launch, or it has run but the samples were written by the dev team — no continuous eval
4. The Critic passes through on failure, or there's no Critic at all — no backstop (this is what the third piece in the series is entirely about)
5. No role's KPI is tied to "the system is alive at six months" — every KPI is "we shipped" or "month-1 automation rate" — no long-horizon accountability

Zero yeses — at minimum, the design cleared the deploy-and-abandon threshold. Three or more yeses — there's an incident coming within six months. Put these five on the next post-mortem agenda and have everyone openly agree on where the design actually stands.

Closing: Put the Conspiracy on the Table

"Deploy and abandon" is not a technical problem — it's a decision-structure problem. Every operational mechanism (Critic, monitoring, eval, canary) is mature, the industry has been running them for decades. What's stuck is that nobody's incentives are tied to "six months from now."

The thing the executor layer can do is small but decisive: put this quiet conspiracy on the table. At the next vendor review, ask the four questions. At the next internal project kickoff, put the six environments into the SOW. The next time someone says "we're not Apple, we don't need this much detail" — hand them this piece.

"We're not at that level yet" is a sentence that pushes the responsibility into "later." In AI Agent projects there is no "later." The day you ship, the seed of the incident is in the ground. What gets dug up six months later is every dollar of ops cost you originally skipped, plus 10x interest.

If your design hits more than two yeses on the five-point self-check, start patching today. Patching one is easier than patching five, and patching five is cheaper than one incident.

---

The Toolkit

If you want these tools straight in your next vendor-proposal review — without re-reading this article every meeting — I packaged a PDF kit for readers who got this far. Send me the keyword "PRODUCTION" and I'll send the pack:

1. The 6-environment demo-vs-production comparison table (one-page A4, tick through it during a vendor pitch)
2. The four-question review script + sample passing/failing answers (ready to read verbatim; includes the most common failing answers and how to spot them)
3. The five-point self-check (print and pin above your desk, run it monthly)

A year of project debugging, distilled into judgment tools.

(Channels in the footer — X or email both work.)

What's Next in This Series

The next pieces in the series, scheduled:

• Three-tier intent cascade tuning (series 2 decision 2 deep dive): rules + embedding + LLM fallback confidence thresholds, cache strategy, rolling out new intents without disturbing existing ones
• Wiring an LLM into enterprise SaaS (series 2 decision 4 deep dive): the 25-API → 5-tool wrapping contract, idempotency, timeout retry, gradual rollout
• How to test an AI system (standalone deep dive): dual-track architecture + seven quality dimensions
• The first major-incident post-mortem SOP (continuation of this piece): what to do in the first 24h, how to write the post-mortem, how to change KPIs so the same incident doesn't repeat

If your team is building a customer-service Agent or another L2 write-operation Agent, these four are coming over the next 3-4 weeks, one at a time.

---

This is the fourth piece in the Agentic AI in Practice methodology series. Earlier pieces:

• Piece one: 28 "Agent" Projects, Only 5 Are Real — The L0-L3 Grading Framework
• Piece two: Five Architecture Decisions That Determine Whether Your Customer-Service Agent Can Ship
• Piece three: Why a 70% Critic Beats a 95% Critic — A Fail-Closed Design Deep Dive

The series follows the methodology of taking an Agent project from kickoff to launch. Subscribe at the bottom; new pieces get pushed.