Choose a local runtime

A runtime is the software that loads a model and performs inference. It is not the model itself. Swapping runtimes does not swap weights unless the new runtime supports the same artifact.

Ollama manages model downloads, local serving, and a simple API. It is a good starting point for this course. You get ollama pull, ollama run, and an HTTP server on localhost without wiring files by hand.

llama.cpp is a lightweight C and C++ inference project with broad hardware support. It runs GGUF models and includes an HTTP server with compatible API routes. Choose it when you want direct control over backends, batching, and server flags.

MLX LM targets Apple Silicon. It can generate, quantize, and fine-tune models using Apple’s MLX framework. It fits experiments on a Mac when you live inside that ecosystem.

Desktop applications such as LM Studio add model discovery and a graphical chat interface. They can be convenient, but the model, format, and runtime compatibility still matter underneath the UI. The chat window hides the same architecture and template questions we covered earlier.

Choose based on the boundary you need:

• quick local chat and API: Ollama
• direct GGUF control and broad backends: llama.cpp
• Apple Silicon experiments and fine-tuning: MLX LM
• graphical exploration: a desktop interface

The model is portable only when another runtime supports its architecture, format, template, and features. Moving from Ollama to llama.cpp is feasible for many GGUF models. Moving a Safetensors-only training repo into Ollama without conversion is not.

My advice is to pick one runtime for your application adapter, then document how you would migrate. “Swappable” starts with a clear boundary, not with three runtimes installed on every laptop.

Try this on your own project: write one sentence naming your runtime and the file format it consumes. If you cannot name both, pause before you build features on top.