Problem: You deployed a local LLM but nobody can actually use it — what separates "it runs" from "it works"? —— Trove problem library

Symptom

The most expensive "model tuition" I ever paid was an 8-GPU A100 rig rented for two weeks and wasted.

Back then we were building an intelligent customer-service assistant, and our thinking was brutally simple: the biggest model must be the best. So we spent two weeks setting up a machine with 8× A100 GPUs to run a 72B-parameter model. The result?

6–8 seconds of inference latency per reply — users were ready to curse.

The kicker: compared with a 7B model, the 72B was only ~2 percentage points more accurate in our customer-service scenario — but cost more than ten times as much to serve.

We later switched to 7B + LoRA fine-tuning: latency dropped under 2 seconds, and accuracy actually beat the un-fine-tuned 72B (88% → 93.7%).

The A100 money was gone up in smoke.

This is not just my story. There is another line of failure: "installed but useless". You follow the tutorial step by step, and you end up stuck at "can't connect" or "it runs but nobody uses it".

I've watched too many teams freeze in that state — the model runs, the demo looks cool, and then production humiliates them: it crashes under peak load, replies are too slow for anyone to use, business accuracy is too low to ship.

There is a saying that nails it: most teams' open-source model deployments get stuck somewhere between "it works" and "it works well".

And then there is the most insidious class of failure: not "can't install", but "it silently gets dumber after install".

Same GPU, identical weights — a tiny difference in the inference software stack can make the model emit completely different tokens at critical positions, even break tool calling entirely.

You think the model is bad. The environment is lying to you.

If any of this sounds familiar — you deployed locally but nobody uses it; or you're about to deploy and your head is full of "bigger is better"; or your demo works and production exposes you — this article is for you. It is not an installation tutorial. It is the consensus of people who actually put open-source models into production, paid with GPU bills and on-call incidents: the hard part of local LLM deployment was never "making it run". It is "making it accurate, fast, stable, and sustainable on limited hardware".

Cause

Let's break down what separates "it runs" from "it works". I've classified the ways teams fail into four categories, each with a mechanistic cause — not because someone didn't try hard enough:

Category 1: The "bigger parameters = better" superstition. The most common trap. People assume larger models are always better and blindly deploy tens or hundreds of billions of parameters, ignoring their own server specs and concurrency needs.

Result: laggy service, inference timeouts, OOM.

The truth: model selection is not "pick the strongest", it's "pick the most fitting" — for lightweight text classification, a 1B–3B model can beat a 72B, because the accuracy gap is under 2% while the cost gap is 10x.

Category 2: Treating "deploy" as "landing". Many teams think AI landing means download, deploy, call. They skip quantization, fine-tuning, prompting, inference-engine optimization, and business adaptation.

Open-source models are trained on public data — they are strangers to your internal business terms, ticket formats, and industry rules. Raw weights on real business data often score under 60% accuracy.

Between "it runs" and "it works" sits an entire tuning pipeline.

Category 3: Treating "demo works" as "production ready". Basic calling only, no batched inference optimization, no retries, no rate limiting, no hot model swapping. Then peak traffic arrives: timeouts first, outages next.

A demo with one person clicking twice is not production with 20 people asking at once.

Category 4: Treating "environment" as a given. The biggest silent killer in local deployment is not the model — it's the environment: dependency versions, inference engines, quantization backends, CUDA versions.

Any tiny difference can change model behavior drastically. Research has shown the same model can produce different outputs with a different inference software stack — which is why "mine is the same, but mine is dumber".

And one common thread under all these failure modes: no business-oriented acceptance. Selecting with generic benchmarks, accepting with "it produced output", judging with "the demo looked good" — without a business evaluation set, nobody can catch the failure before it ships.

Every experience below is how to fill these holes, one by one, from measured practice in real teams.

Solution

Each experience below follows the same shape: phenomenon → practice → why. None of it is one person's opinion — these are patterns that recur across many independent practices.

Experience 0 (L1 one-line takeaway): Set constraints first, then pick the model — drop the "bigger is better" obsession

Selection is not "bigger is better"; it's "set constraints first, then pick": decide the business scenario, hardware budget, concurrency needs, and latency ceiling first, then work backwards to model size. If you only read one thing: for lightweight text classification, content parsing, and simple Q&A, start with 1B–3B;

consider 7B+ only for high-concurrency complex reasoning; and benchmark with real business data. Experiences 1–5 are the why and how.

Experience 1: Quantization is mandatory — but it is not mindless compression

Phenomenon: native FP16 models eat VRAM and infer slowly, and can't handle concurrency. Everyone knows quantization exists, but they either skip it (VRAM explodes) or blindly 4-bit it (accuracy craters).

I've seen teams watch accuracy fall off a cliff after quantizing and blame the model — the real problem was the quantization config.

Practice: make 4-bit quantization (nf4 + double quant) the default starting point for lightweight business scenarios, then verify with the business evaluation set that the accuracy loss is acceptable.

A measured reference: native FP16 used 3.2G VRAM and ~2.1s per inference; 4-bit dropped VRAM to ~800M and latency under 400ms, with near-zero accuracy loss. Key principle: quantization is not "the more aggressive the better"; it's "compress as far as accuracy allows" — every quantization config change must be re-validated on business data.

Why: quantization is the highest-ROI step in local deployment — it decides whether you can "fit it and run it fast". VRAM down to a quarter, latency down to a fifth, at nearly zero accuracy cost — the math works out no matter how you slice it.

But "nearly zero" has preconditions: different bit widths, backends (GGUF/AWQ/GPTQ), and compute dtypes behave differently — which is exactly why re-validation is non-negotiable.

Experience 2: Deployment is only the beginning — fine-tuning + prompting is "business adaptation"

Phenomenon: "heavy on deployment, light on tuning" is the second most common trap: use raw open-source weights with zero scenario fine-tuning and zero prompt optimization. The raw output is a stranger to your business terms, and accuracy is too low to ship.

One team measured raw weights at 58% accuracy on ops-log analysis — barely better than a coin flip.

Practice: three steps, none skippable — ① small-sample fine-tuning (LoRA is enough): use a few hundred to a few thousand real business examples to teach the model your terms and output format; ② prompt engineering: use a system prompt to enforce output structure (fixed format, no fluff, low temperature);

③ combine quantization + fine-tuning + prompting. That same team went from 58% to 92% accuracy after these three steps. Reference line: get accuracy to an acceptable level with fine-tuning + prompting first; only then optimize latency and concurrency.

Why: a general model's knowledge is "public-world knowledge", not "your company's knowledge". Fine-tuning teaches the model your business; prompting teaches it to speak your language — the former solves "does it know", the latter "does it sound right".

Skip these two and no amount of quantization will help; you've just made a car that doesn't know the road go faster.

Experience 3: Evaluate with real business data — don't trust generic benchmarks

Phenomenon: the second reason model selection fails: comparing models with generic benchmarks. On benchmarks, 72B crushes 7B — but in your customer-service scenario, 72B's advantage may be 2%.

That gap never shows in benchmark reports; it only shows in a real business evaluation set.

Our wasted A100 was tuition paid to benchmark thinking.

Practice: when selecting, build a "business evaluation set": pull 50–200 representative inputs from real business (including edge cases and nasty ones), run every candidate model with the same prompt template, score by human review or keyword coverage, and record latency and cost.

Compare on the three dimensions of "business accuracy × latency × cost", not just benchmark scores. Freeze the evaluation set — rerun it every time you change model, quantization, or prompts. It is your regression test.

Why: benchmarks measure "the model's public capability"; you need "the model's performance on your business" — the correlation is far weaker than you'd hope.

A business evaluation set turns selection from "watching ads" into "seeing results", and once built it's a long-term asset: every future model upgrade and tuning pass has evidence to lean on.

Experience 4: Engineering optimization beats algorithm tuning — "it runs" ≠ "it can ship"

Phenomenon: "heavy on features, light on stability" is the third common trap. Basic calling only, no batched inference, no retries, no rate limiting, no hot swap — and peak load kills it.

The other extreme: teams obsess over pushing accuracy from 92% to 93% while a single GPU can only serve 2 concurrent requests.

In production, stable availability, low latency, and high concurrency matter far more than marginal accuracy.

Practice: before going to production, complete the engineering four-piece set — ① concurrency/batching: use an inference engine like vLLM with continuous batching, never serial one-by-one calls; ② rate limiting & degradation: set a concurrency cap and queue on the inference service;

over the limit, return a degraded result instead of cascading; ③ retries & timeouts: set timeouts and retries on the caller side, distinguishing "model is slow" from "model is down"; ④ hot model swapping: switch versions without downtime. Reference line: first guarantee "survives peak load, no cascading failures"; then go squeeze accuracy.

Why: model services fail differently from normal web services — it's either "timeout" or "OOM", and every request is expensive. The cost of the engineering four-piece set is far below the cost of one production incident.

Remember: demo means "it runs"; production means "it runs reliably" — the gap between them is engineering, not the model.

Experience 5: The environment is hell — "the error is lying", pin every version

Phenomenon: the sneakiest trap in local deployment is the environment.

Same model, same GPU, identical weights — a tiny difference in the inference software stack (quantization backend, attention implementation, KV-cache quantization, CUDA/driver versions) can produce entirely different outputs, even break tool calling.

Beginners assume "the model is broken" — but the error is lying; the real problem is the environment.

Practice: three things — ① pin versions: write exact versions of CUDA, drivers, inference engine, and Python dependencies into deployment scripts and docs; never "latest"; ② minimal reproduction: when something breaks, suspect the environment first — reproduce with the official Docker image or a minimal dependency set to rule out stack differences;

③ traceable changes: after ANY environment change (engine upgrade, backend switch), rerun the business evaluation set — environment changed, behavior may have changed.

Why: an LLM's output is a function of the software stack — the same weights on different stacks are not the same model. Environment issues are the hardest to debug because they masquerade as model issues.

Pinning versions + minimal reproduction + re-test on change is the only way to turn the environment from "Schrödinger's trap" into a controllable variable.

How to start (checklist, L1 execution layer)

1. Set constraints before selection: write down business scenario, hardware budget, concurrency needs, and latency ceiling, then work backwards (lightweight scenarios start at 1B–3B);
2. Build a business evaluation set: 50–200 real business inputs (with edge cases), frozen as your regression test;
3. The three-step path: 4-bit quantization as the base, LoRA fine-tuning for business fit, system prompt for output structure;
4. Engineering four-piece set: batched concurrency, rate limiting/degradation, retries/timeouts, hot swapping — none skippable;
5. Pin the environment: exact CUDA/driver/engine/dependency versions in deployment scripts; re-run the evaluation set after any change.

How to apply this in your situation (L3 deep layer)

• Personal tinkering (your own machine): no business pressure, so the risk is "bigger is better" bias — ask what you actually need: chat? code? doc Q&A? Lightweight tasks are fine with 1B–8B GGUF quantized models; make your evaluation set "a few fixed questions" and rerun it after every model swap — more useful than reading benchmarks.
• Small team pilot (1–3 business scenarios): pick the scenario with the highest business value and clearest boundaries; first run the minimal chain of "quantization + fine-tuning + prompting + evaluation set", then talk about scaling; the metric that matters during the pilot is "business accuracy meets the bar", not "model is big".
• Production systems / high concurrency: full engineering four-piece set, evaluation set in CI; model versions, environment versions, and prompt versions all under version control; every upgrade goes "evaluation set → canary → full rollout"; never swap models directly.
• The AI era (using AI to deploy AI): let AI write deployment scripts, tune quantization, and generate evaluation sets — but acceptance must be human. AI will tell you "it's installed"; it won't tell you "this install outputs differently from your last one". Environment version pinning and a business evaluation set are the last human guardrails in the AI era of local deployment.
• If you're stuck in "installed but useless": don't doubt yourself first — check in order: environment versions (pin them) → evaluation set (build it) → quantization/fine-tuning/prompting (adapt to business) → engineering four-piece set (survive concurrency). Most "can't use it" is not "you can't install" — it's that you only installed the "it runs" half.

(Final words) The sentence I have said to the most people: do not start with a 70B. The truth about local LLMs is that a quantized 8B covers nine out of ten use cases; the bottleneck is never parameters, it is constraints, data, and the environment.

My own worst night: I kept swapping models, only to discover the Python environment had been silently upgraded — the models were never the problem.

Lock the environment first, then talk models. That one sentence is worth a full night of sleep.