Understand the real cost

Local inference has no per-token invoice, but it is not free.

Count the hardware over its useful life, electricity while computing, idle power when the machine must stay available, storage, maintenance time, and replacement cost.

Then compare an equivalent service. A small local model should not be justified by comparing it only with the price of a much stronger frontier API. Compare like with like for the task you actually ship.

Local often makes sense when:

• you already own suitable hardware
• data must remain inside a controlled boundary
• usage is frequent and predictable
• a modest model passes the task evaluation
• offline operation matters

An API often makes sense when usage is low or spiky, the task needs frontier quality, you do not want to operate hardware, or scaling beyond one machine matters.

A hybrid design is normal. Route private routine work locally and difficult public-data work to a cloud model when policy allows it.

Notice utilization. A GPU that runs ten summaries a day is an expensive calculator. The same GPU running thousands of automated checks may pay for itself quickly.

Electricity matters on always-on servers. Laptops on battery behave differently from a desktop left awake for Ollama. Measure wall power if cost is part of the decision, not just sticker price.

Quality has cost too. If a local model fails often and humans rework the output, add that time to the local side of the ledger.

My advice is to write one paragraph for stakeholders: what stays local, what might still call an API, and why. Honest hybrid language prevents false “zero cost AI” claims.

Try this on your own project: estimate requests per day and minutes of human rework per failed summary. That pair tells you whether local hardware is a bargain or a science project.