How to run an LLM locally on macOS, Linux, and Windows
By Flavio
Updated
Tagged: AI
Explore the ways to run LLMs locally on macOS, Linux, and Windows: desktop apps, terminal tools, browser models, libraries, and self-hosted servers.
You can run an LLM on your own computer through a desktop app, a terminal tool, a local web interface, or a library inside your own program. Some models even run inside a browser tab.
Which one you pick depends on what you want to do. Chatting with a model on a laptop needs a different setup from serving several people on a Linux GPU machine. macOS, Windows and Linux all have options, including computers without a dedicated graphics card.
If you just want a model running right now, jump to Run your first model with Ollama. The rest of the post explains the other routes.
What running an LLM locally means
A language model is a collection of learned numbers, called weights, plus the configuration needed to use them. You download those files and use a program to generate answers. That generation process is called inference.
You don’t need to train a model to use it. Training and fine-tuning are separate jobs with different hardware requirements.
There are several pieces involved:
| Piece | What it does | Examples |
|---|---|---|
| Model | Supplies the learned weights and model architecture | Downloadable models from families such as Qwen, Gemma, and Llama |
| Runtime | Loads the weights and computes the answer | llama.cpp, MLX, ONNX Runtime |
| Model manager or server | Downloads models or makes them available to other programs | Ollama, LocalAI |
| Interface | Gives you somewhere to type and read replies | A desktop chat app, Open WebUI, your own application |
Some products combine several pieces: a desktop app can download models, run them, show a chat and expose an API. Others handle just one part. That’s why tools that look like competitors often work together, like a browser interface sending messages to a local server that uses an inference engine underneath.
Local describes where inference happens. A chat page open in your browser can still send every message to a cloud provider, and a web interface can talk only to a model on your laptop.
You also need downloadable weights. A hosted model’s API name doesn’t mean you can download that model, and permission to download weights is a different question from permission to use them commercially. I cover what “open” means for these models in open-weight models.
The main options at a glance
Start with how you want to interact with the model.
| Approach | Good for | What you manage |
|---|---|---|
| Desktop app | Personal chat and trying models visually | App installation and model downloads |
| Model manager with a local API | Scripts, coding tools, and applications | A background service and its model catalog |
| Direct command-line runtime | Control over model files and hardware settings | The executable, weights, and runtime options |
| Local web interface | Chat history, documents, or access from several devices | An interface plus its inference backend |
| Developer library | Embedding inference in your own software | Dependencies, model loading, and application behavior |
| Browser inference | Trying models without installing a native runner | Browser support, model downloads, and browser storage |
| Dedicated inference server | Concurrent requests and shared GPU hardware | Drivers, serving configuration, access, and capacity |
An app with a download button is the shortest route to a first conversation. A local API is handy when several programs need to share one loaded model. If your application must work without a background server, embed a runtime instead, which means more development work, including how users download and update the model.
Run your first model with Ollama
Let’s get a model running with Ollama, since it works the same way on all three systems.
On macOS and Linux, install it from the terminal:
curl -fsSL https://ollama.com/install.sh | sh
On macOS you can also download Ollama.dmg from the download page.
On Windows, run this in PowerShell, or download OllamaSetup.exe from the same page:
irm https://ollama.com/install.ps1 | iex
Now download and run a model. gemma4:e2b is the smallest Gemma 4 variant in the Ollama library, with 2.3 billion effective parameters and a download of 4.6 to 7.5 GB:
ollama run gemma4:e2b
The first run downloads the model, then you get an interactive prompt where you can chat with it. Everything runs on your computer.
To see which models you’ve downloaded:
ollama ls
And to see which ones are loaded in memory right now:
ollama ps
That’s a working local LLM. My Ollama tutorial goes further: model commands, the local API, and calling it from JavaScript.
Desktop apps
These give you a familiar chat window and usually help you find models, with no code required.
LM Studio
LM Studio combines model discovery, downloads, chat and a local server. It runs GGUF models through llama.cpp, and MLX models on Apple Silicon. You can compare models in the chat interface, then turn on the server when an application needs an API.
It has macOS, Windows and Linux versions, but read the system requirements: the current macOS app needs Apple Silicon, so an Intel Mac needs another route.
Jan
Jan is another desktop app, with a model hub and local chat. It also supports remote providers, so if you want offline work, download a local model and select it explicitly. Otherwise you may be using a desktop interface to remote inference.
Jan has Windows and Linux builds. On macOS, its current local inference engine requires Apple Silicon.
GPT4All
GPT4All has desktop apps for Windows, macOS and Linux. It runs models without a dedicated GPU, with speed depending on model size and your computer, and includes LocalDocs for bringing local files into conversations.
Document chat means the app also has to extract and search your documents, so look at those settings as well as the chat model.
Other packaged interfaces
KoboldCpp packages a GGUF runner with a browser interface, aimed at writing and story workflows. It has downloadable builds and hardware acceleration options.
The oobabooga project, long known as text-generation-webui, is now textgen. It provides a local interface and APIs, with lots of generation settings and several model loaders.
Model managers and command-line runners
These suit you if you’re comfortable in a terminal, or want one model that several applications can call.
Ollama
Ollama manages model downloads and runs models behind a local API on macOS, Linux and Windows, and it has an app interface too. A script, a chat client and a coding tool can all connect to the same service instead of each loading the model.
Ollama also offers cloud models, so pick a downloaded local model when you want inference to stay on your computer.
Magnitude
Magnitude is an open-source inference engine with a desktop app and a CLI for macOS, Linux, and Windows. When you download a model, Magnitude tests different ways to run it on your hardware and keeps the fastest settings it finds.
You can serve models from its catalog through a local OpenAI-compatible API or connect them to coding agents such as Codex and OpenCode.
llama.cpp
llama.cpp is an inference engine you use directly. It supports CPU inference, Apple Metal, NVIDIA CUDA, AMD HIP, Vulkan and other backends, and its model format is GGUF. Use it when you want direct control over the model file, context size and GPU offloading, either through its command-line programs or its HTTP server.
It can split work between CPU and GPU, so some models run even when they don’t fit in GPU memory, slower the more work stays on the CPU. A prebuilt binary saves you from compiling, as long as its backend matches your hardware.
llamafile
llamafile packages models as portable executables, so you can run or distribute a model without a model-management service. Executable handling and acceleration differ between operating systems, and some setups use a separate weights file, so read its platform notes.
MLX-LM
MLX-LM runs and fine-tunes language models with MLX on Apple Silicon, as command-line tools and a library. It needs models converted for MLX, because a GGUF download belongs to a different runtime. You can also get MLX through LM Studio without touching the developer tools.
Local web interfaces and document chat
A local API doesn’t give you saved conversations, file uploads, or an interface you can open from another device.
Open WebUI is a self-hosted interface that connects to Ollama and OpenAI-compatible APIs, local or remote. You could run the interface and the model on a desktop computer and use them from a browser on your laptop. The desktop does the computation, so that’s inference on your own network rather than on the laptop.
AnythingLLM focuses on document workspaces. Its configuration separates the language model, embeddings and other services, and each can be local or cloud.
Embeddings turn text into numbers used to search for related passages. A document assistant uses an embedding model to find relevant text, then gives that text to the chat model.
If the whole workflow has to stay local, check both models. A local chat model with a cloud embedding provider still sends your documents elsewhere, and the same goes for OCR, web search and external tools.
Developer libraries and embedded inference
You can put inference directly inside an application, so users don’t install a separate model server. Your app then takes care of loading models and managing memory.
Hugging Face Transformers is the common route for working with downloaded checkpoints, for experiments and apps that need direct access to the model. Acceleration depends on the underlying framework, operating system and hardware.
For JavaScript, Transformers.js runs supported models through ONNX Runtime in the browser or Node.js, for chat and other tasks like embeddings. It needs converted model files, an arbitrary checkpoint won’t load.
ONNX Runtime GenAI embeds generative models in applications, with execution providers deciding which hardware runs the computation.
mistral.rs is a Rust library and server for local inference that, despite the name, supports several model families.
llama.cpp and MLX are building blocks for native software too. The real decision is whether inference lives inside your app process or behind an API your app calls. A local server is easier to swap during development, while an embedded runtime gives you control over distribution and more platform details to handle.
Running an LLM inside a browser
You can run inference inside a browser tab without installing a native server.
WebLLM uses WebGPU: the browser downloads the model and runs it on the user’s device. Transformers.js also runs in the browser, through WebGPU or WebAssembly depending on configuration and model.
This works well for demos, browser apps and small models, since anyone can open a page and try the model. The first download can be large, browser storage can be cleared, GPU support varies by browser and device, and not every model has a browser build.
The computation being local doesn’t tell you what the page does with your inputs. Analytics, logging and other network requests are separate.
Dedicated inference servers
A GPU workstation serving several applications is a different kind of local setup from a model on a laptop.
vLLM is built for efficient serving, when you need concurrent requests and throughput rather than one chat window. SGLang is another serving framework, with scheduling and caching optimizations for language and multimodal models.
Both need more care around supported architectures, accelerator dependencies and deployment config, and Linux is the usual GPU target. vLLM’s GPU installation documentation covers its hardware paths, including vLLM-Metal for Apple Silicon, and each backend has its own platform support.
LocalAI puts compatible APIs in front of multiple inference backends, so text generation and other local AI workloads sit behind one service.
You can run these on hardware you own or on rented servers. A rented server is self-hosted remote inference, the computation isn’t on your computer anymore.
Running local LLMs on macOS
The first question is Apple Silicon or Intel.
On Apple Silicon, start with a desktop app like LM Studio, Jan or GPT4All. Ollama gives you a managed service, llama.cpp direct control through Metal, and MLX-LM the MLX route.
Apple Silicon uses unified memory shared by the CPU and GPU, which makes a Mac with lots of memory good for local models. macOS and your other apps need part of it, though, so a 16 GB Mac doesn’t have 16 GB for weights.
On an Intel Mac, current LM Studio and Jan local inference don’t work, since they require Apple Silicon. A llama.cpp CPU build works, at a speed that depends on the processor and model size.
On either Mac, a native runtime gets you hardware acceleration more directly than a Linux container. Docker is still fine for a web interface while a native runner does the inference.
Running local LLMs on Windows
LM Studio, Jan, GPT4All and Ollama all have native Windows apps, and llama.cpp and KoboldCpp can run model files directly.
Watch the processor architecture. x64 and ARM64 builds aren’t interchangeable, so on Windows on ARM look for a build for your machine.
For GPU inference, match the runner to your card. NVIDIA, AMD and Intel GPUs use different acceleration paths, and support varies by tool. Ollama publishes its own hardware compatibility list, so check your card there instead of assuming any gaming GPU works.
WSL2, the Windows Subsystem for Linux, is the way to use tools that target Linux, with GPU access depending on your hardware and drivers. Follow the project’s WSL instructions, not its native Windows ones. WSL is one more environment to manage, storage and networking included, so if a native app does the job, start there. For a Linux-oriented development or serving workflow, WSL makes more sense.
The NPU in your laptop only helps if the model format, the runtime and the device backend all support it.
Running local LLMs on Linux
Linux runs everything: desktop chat apps, command-line tools and dedicated servers, on a workstation or on a machine without a display.
LM Studio, Jan and GPT4All have Linux builds, and Ollama and llama.cpp cover a local API or a terminal workflow. MLX-LM is Apple Silicon only.
On NVIDIA hardware, CUDA is the common acceleration path. On AMD, check ROCm/HIP support for your exact GPU and distribution. Vulkan works in runtimes that support it, and Intel has its own paths, including SYCL in llama.cpp.
Driver and runtime versions matter. Your desktop rendering fine says nothing about whether the inference backend can use the card.
On a headless machine, decide where models are stored, how the service starts and which network interfaces it listens on. Containers make deployment repeatable, but they still need compatible host drivers and access to the GPU devices.
Do you need a GPU?
No. CPU inference is fine for small models, occasional use, and testing an application before buying hardware. A GPU makes generation faster when the runtime supports it and the model fits.
Memory is the first thing to check. A rough estimate for the weights is:
weights in GB ≈ parameters in billions × bits per parameter ÷ 8
An 8-billion-parameter model stored at 4 bits needs roughly 4 GB for the weights. The file includes other data, and the running model needs more memory on top.
Quantization stores the weights at lower precision so they take less space, which makes a model practical on a smaller machine, possibly at some cost in answer quality. A 4-bit variant is a good starting point when your runtime supports it.
The context window uses memory too. Long conversations, large documents and coding-agent requests use much more than a short chat, and concurrent requests add more.
On a discrete GPU, count VRAM separately from system RAM. 64 GB of RAM doesn’t turn an 8 GB graphics card into a 64 GB one. A runtime can offload some work to the CPU, at a cost in speed.
I explain the estimate in how much VRAM a local LLM needs, and there’s an LLM memory calculator for trying different sizes.
Model formats and compatibility
A model family can come in several formats, so download the files for your runtime.
| Format or packaging | Where you’ll encounter it | What to check |
|---|---|---|
| GGUF | llama.cpp and applications built around it | Model architecture support and quantization |
| Safetensors checkpoints | Transformers and many serving frameworks | Configuration, tokenizer, dependencies, and supported architecture |
| MLX model conversions | MLX-LM and MLX-enabled applications | A conversion supported by the MLX runtime |
| ONNX models | ONNX Runtime and Transformers.js | Required files, operators, and execution provider |
| Browser-specific model assets | WebLLM and other browser engines | Supported model build and browser GPU requirements |
| Managed model tags | Catalog-based runners such as Ollama | The exact variant, size, and whether inference is local |
The file extension isn’t enough. The runtime has to understand the model’s architecture, and newer architectures can need an updated runtime.
You’ll also see base and instruct variants. For normal conversation, use a chat or instruct model. A base model is trained to continue text and needs more work to behave like an assistant.
Image input, embeddings, tool calling and structured output are separate capabilities, and both the model and the runtime need to support the one you want. A chat model that fits in memory isn’t automatically a reliable coding agent.
Using Docker or another machine you own
Docker packages a runner or an interface. It doesn’t replace the inference engine, add memory, or enable GPU access by itself. It’s useful on Linux servers when you want a reproducible deployment. On macOS and Windows there’s a VM or WSL layer in between, so check the documented acceleration path.
You can also put the model on another machine you own and connect over your local network, so a laptop uses a desktop’s bigger GPU. Keep in mind that requests now cross the network to another machine, which matters for sensitive data even when you control both ends.
A server listening on localhost only accepts connections from its own machine, and opening it to other devices takes deliberate configuration. A local API often accepts requests without credentials, so add authentication and network protection before sharing it.
How to choose your first setup
I would start with the work I want the model to do, then pick the smallest setup that handles it.
| What you want | A starting point |
|---|---|
| Chat without using a terminal | Compare LM Studio, Jan, and GPT4All |
| A model that scripts and tools can share | Ollama, Magnitude, or a desktop app’s local server |
| Direct control over GGUF files | llama.cpp |
| Writing-oriented local chat | KoboldCpp |
| MLX experimentation on an Apple Silicon Mac | MLX-LM |
| Saved chats in a self-hosted web interface | Open WebUI with a local backend |
| Chat with a collection of documents | AnythingLLM or GPT4All LocalDocs, with local embeddings |
| Inference inside a web page | WebLLM or Transformers.js |
| Inference embedded in your application | A compatible library such as llama.cpp, MLX, or ONNX Runtime GenAI |
| Several applications or users sharing GPU capacity | Evaluate vLLM, SGLang, or LocalAI |
Then I would download a small instruct model and try it on a few real tasks. For a document assistant, that means questions whose answers I can verify in the documents. For code, a small change with tests I can run.
Watch the time to the first answer, memory use, and whether the computer stays usable while it generates. Try the context length you expect to need, because a short greeting tells you nothing about how a model handles a long source file.
If it’s too slow, shrink the model or the context before tweaking runtime settings. If the answers are poor, try another model on the same prompts, since a faster runner can’t add knowledge the model doesn’t have.
For offline use, download the models first, select local providers, and run the complete task with networking turned off. That tests the real workflow, document processing and tools included.
Local inference removes per-request charges, but hardware, electricity and maintenance still cost money. I worked through that comparison in running an LLM locally versus paying for an API.
Once you’ve picked a runner, my guide to open-weight models explains what you’re downloading and how to evaluate a model before you rely on it.
Want me to talk about your product? You can sponsor this site.
Related posts about ai: