# Don't let the LLM do the math

> Keep pricing and numbers deterministic in AI products. Let the LLM write prose only. A hybrid architecture from a real deployment advisor.

Author: [Flavio Copes](https://flaviocopes.com/about/) | Published: 2026-07-20 | Topics: [Essays](https://flaviocopes.com/tags/essay/) | Canonical: https://flaviocopes.com/dont-let-the-llm-do-the-math/

When I built [StackPlan](https://stackplan.dev), I made one rule early: the LLM never picks a price.

It never ranks a stack. It never computes monthly cost at 10k DAU.

It writes the explanation. That's it.

## LLMs are bad at arithmetic

Ask a model what Railway charges for 50GB egress. You'll get a confident number. Often wrong. Out of date by six months.

Ask it to multiply overage rates across three services and sum the total. It will drift. Round wrong. Forget a tier boundary.

This isn't a prompting problem. Models optimize for plausible text, not ledger math.

Pricing pages change. Free tiers get resized. Your product cannot depend on whatever the model memorized last training run.

## The hybrid architecture

StackPlan's recommendation engine has three layers:

1. **Curated data in SQLite** — providers, services, plan tiers with `effective_from` dates. An admin can update pricing without redeploying code.

2. **Deterministic scoring in TypeScript** — filter by capability fit, assemble candidate stacks, run the cost model. Integer cents end to end. No floating-point surprises.

3. **LLM as presentation** — given the pre-computed candidates, write strengths, tradeoffs, and gotchas. Prose only.

The flow looks like this:

```
Questionnaire → rules engine → cost model → ranked stacks → LLM rationale → report page
```

The LLM sees exact numbers in the prompt. The prompt says: do NOT invent or alter any cost figures.

```ts
return `You are writing rationale prose for a stack recommendation tool.
The rules engine already computed candidate stacks with exact costs.
Your job is ONLY to explain — never invent or alter any numbers.`
```

If the model hallucinates a price anyway, it doesn't matter. The UI renders costs from the JSON the engine produced. The rationale is decoration.

## The cost model stays pure

`costModel.ts` has no network calls. No `fetch`. No env vars.

It takes tier definitions and a usage profile. It returns monthly cost in cents, or `null` when a hard limit is exceeded.

```ts
function computeTierCostCents(tier, usage) {
  let total = tier.basePriceCents

  for (const unit of ALL_USAGE_UNITS) {
    total += overageCostCents(tier, unit, usage)
  }

  return total
}
```

Same function runs on the server when building a report. Same bundled copy runs in the browser when you drag a tier knob on the canvas.

One source of truth. Testable with unit tests. No "the AI said $47" ambiguity.

## Graceful degradation

`getAdapter()` returns `null` when no API key is configured.

No LLM? Reports still generate. Stacks still rank. Costs still compute. The rationale section simply doesn't render.

Prefill from a free-text description also degrades. The form works manually.

This is a product decision, not an afterthought. The engine must stand alone. The LLM is a nice layer on top.

I run Cursor's SDK locally through a dev bridge. Production uses a plain `fetch` adapter to an OpenAI-compatible endpoint. Both implement the same `LlmAdapter` interface. Swap providers without touching the engine.

## Testability is the real win

When the LLM owned pricing, every test would be flaky. Different run, different number.

Now I test the engine with fixtures. Given this questionnaire and this KB snapshot, expect these three candidates in this order at these cent values.

The LLM output gets zod-validated JSON parsing. Malformed response? Retry once, then fail silently. The report is still useful without the prose block.

You can cache rationale by input hash in KV. Same questionnaire + same candidates = same text. Cheaper and consistent.

## What this means for your AI product

If your product outputs numbers users will act on — prices, scores, rankings, compliance flags — compute them in code.

Use the LLM for:

- summarizing structured data
- filling a form from free text
- writing user-facing explanations
- generating labels and copy

Don't use it for:

- arithmetic
- looking up live prices
- deciding who wins a comparison
- anything you'd put in a contract or invoice

The split feels like more work upfront. You maintain a data layer. You write a cost function.

You ship something users can trust. And when the model is down, your product still works.

That's the bar I want for anything that touches money.