Don't let the LLM do the math

When I built StackPlan, I made one rule early: the LLM never picks a price.

It also never ranks a stack or computes monthly cost at 10k DAU. It only writes the explanation.

LLMs are bad at arithmetic

Ask a model what Railway charges for 50GB egress. You’ll get a confident number. Often wrong. Out of date by six months.

Ask it to multiply overage rates across three services and sum the total. It will drift. Round wrong. Forget a tier boundary.

Prompting cannot fix this because models optimize for plausible text, not ledger math.

Pricing pages change. Free tiers get resized. Your product cannot depend on whatever the model memorized last training run.

The hybrid architecture

StackPlan’s recommendation engine has three layers:

1. Curated data in SQLite. This holds providers, services, and plan tiers with effective_from dates. An admin can update pricing without redeploying code.

2. Deterministic scoring in TypeScript. It filters by capability fit, assembles candidate stacks, and runs the cost model. It uses integer cents end to end, avoiding floating-point surprises.

3. LLM as presentation. Given the pre-computed candidates, it writes strengths, tradeoffs, and gotchas. It only produces prose.

For judgment-shaped branches — an angry support message, which path to take — a decision model like Jev can sit where a normal if fails. Pricing stays deterministic.

The flow looks like this:

Questionnaire → rules engine → cost model → ranked stacks → LLM rationale → report page

The LLM sees exact numbers in the prompt. The prompt says: do NOT invent or alter any cost figures.

return `You are writing rationale prose for a stack recommendation tool.
The rules engine already computed candidate stacks with exact costs.
Your job is ONLY to explain — never invent or alter any numbers.`

If the model hallucinates a price anyway, it doesn’t matter. The UI renders costs from the JSON the engine produced. The rationale is decoration.

The cost model stays pure

costModel.ts has no network calls, fetch, or environment variables.

It takes tier definitions and a usage profile. It returns monthly cost in cents, or null when a hard limit is exceeded.

function computeTierCostCents(tier, usage) {
  let total = tier.basePriceCents

  for (const unit of ALL_USAGE_UNITS) {
    total += overageCostCents(tier, unit, usage)
  }

  return total
}

The same function runs on the server when building a report. Its bundled copy runs in the browser when you drag a tier knob on the canvas.

This gives me one source of truth that I can unit test, without any “the AI said $47” ambiguity.

Graceful degradation

getAdapter() returns null when no API key is configured. Reports still generate, stacks still rank, and costs still compute. Only the rationale section disappears.

Prefill from a free-text description also degrades. The form works manually.

I designed the engine to stand alone, with the LLM as an optional layer on top.

I run Cursor’s SDK locally through a dev bridge. Production uses a plain fetch adapter to an OpenAI-compatible endpoint. Both implement the same LlmAdapter interface. Swap providers without touching the engine.

Testability is the real win

If the LLM owned pricing, a different run could produce a different number and make every test flaky.

Now I test the engine with fixtures. Given this questionnaire and this KB snapshot, expect these three candidates in this order at these cent values.

The LLM output gets zod-validated JSON parsing. Malformed response? Retry once, then fail silently. The report is still useful without the prose block.

You can cache rationale by input hash in KV. The same questionnaire and candidates then return the same text, reducing cost and keeping the result consistent.

What this means for your AI product

If your product outputs numbers users will act on, compute them in code. This includes prices, scores, rankings, and compliance flags.

Use the LLM for:

• summarizing structured data
• filling a form from free text
• writing user-facing explanations
• generating labels and copy

Don’t use it for:

• arithmetic
• looking up live prices
• deciding who wins a comparison
• anything you’d put in a contract or invoice

This split takes more work upfront because you maintain a data layer and write a cost function. It lets you ship numbers users can trust, and the product still works when the model is down.

That’s the bar I want for anything that touches money.

Want me to talk about your product? You can sponsor this site.