# How to run an LLM locally on macOS, Linux, and Windows > Explore the ways to run LLMs locally on macOS, Linux, and Windows: desktop apps, terminal tools, browser models, libraries, and self-hosted servers. Author: [Flavio Copes](https://flaviocopes.com/about/) | Published: 2026-10-31 | Updated: 2026-10-31 | Topics: [AI](https://flaviocopes.com/tags/ai/) | Canonical: https://flaviocopes.com/run-llm-locally/ You can run an LLM on your own computer through a desktop app, a terminal tool, a local web interface, or a library inside your own program. Some models can even run inside a browser tab. The choice depends on what you want to do. Chatting with a model on a laptop needs a different setup from serving several people on a Linux GPU machine. macOS, Windows, and Linux all have options, including computers without a dedicated graphics card. This guide covers the main approaches and the tools available for each. It stays at the overview level, so you can choose a direction before following a tool-specific tutorial. There are too many projects and forks to list every executable, but the categories below cover the ways you can run local inference. ## What running an LLM locally means A language model is a collection of learned numbers, called **weights**, plus the configuration needed to use them. You download those files and use a program to generate answers. That generation process is called **inference**. You don't need to train a model to use it. Training and fine-tuning are separate jobs with different hardware requirements. There are several pieces involved: | Piece | What it does | Examples | | --- | --- | --- | | Model | Supplies the learned weights and model architecture | Downloadable models from families such as Qwen, Gemma, and Llama | | Runtime | Loads the weights and computes the answer | llama.cpp, MLX, ONNX Runtime | | Model manager or server | Downloads models or makes them available to other programs | Ollama, LocalAI | | Interface | Gives you somewhere to type and read replies | A desktop chat app, Open WebUI, your own application | Some products combine several pieces. A desktop app can download models, run them, display a chat, and expose an API. Other tools handle just one part. This explains why two apparently competing tools can work together. A browser interface can send messages to a local server, which uses an inference engine underneath. ```mermaid flowchart TB accTitle: Where local language model inference happens accDescr: A desktop app, terminal client, or local web interface sends prompts to a runtime on your computer, which loads downloaded weights. Browser inference instead runs the runtime inside the browser. A["Desktop, terminal, or web interface"] --> B["Runtime on your computer"] C["Downloaded model weights"] --> B B --> D["Generated answer"] E["Browser app"] --> F["Runtime inside the browser"] G["Downloaded browser model"] --> F F --> H["Answer inside the browser"] ``` **Local** describes where inference happens. A chat page open in your browser can still send every message to a cloud provider. Conversely, a web interface can talk only to a model on your laptop. You also need downloadable weights. A hosted model's API name doesn't mean you can download that same model. Read the model's license too: permission to download weights and permission to use them commercially are separate questions. ## The main options at a glance Start with how you want to interact with the model. | Approach | Good for | What you manage | | --- | --- | --- | | Desktop app | Personal chat and trying models visually | App installation and model downloads | | Model manager with a local API | Scripts, coding tools, and applications | A background service and its model catalog | | Direct command-line runtime | Control over model files and hardware settings | The executable, weights, and runtime options | | Local web interface | Chat history, documents, or access from several devices | An interface plus its inference backend | | Developer library | Embedding inference in your own software | Dependencies, model loading, and application behavior | | Browser inference | Trying models without installing a native runner | Browser support, model downloads, and browser storage | | Dedicated inference server | Concurrent requests and shared GPU hardware | Drivers, serving configuration, access, and capacity | An app with a download button is usually the shortest route to a first conversation. A local API is convenient when several programs need to share one loaded model. For an application that should work independently of a background server, an embedded runtime may be a better fit. That takes more development work, including deciding how users download and update the model. ## Desktop apps These give you a familiar chat window and usually help you find models. You don't need to write code to get started. ### LM Studio [LM Studio](https://lmstudio.ai/docs/app) combines model discovery, downloads, chat, and a local server. It supports GGUF models through llama.cpp and MLX models on Apple Silicon. It's useful when you want to compare models visually and later connect an application to the same setup. You can begin in the chat interface and enable the server when you need an API. It has macOS, Windows, and Linux versions, but check the [system requirements](https://lmstudio.ai/docs/app/system-requirements). In particular, the current macOS app requires Apple Silicon; an Intel Mac needs another route. ### Jan [Jan](https://docs.jan.ai/desktop/quickstart) is another desktop option, with a model hub and local chat. It also supports remote providers, so check which model and provider a conversation uses. The distinction matters if you're choosing Jan for offline work. Download a local model and select it explicitly. Connecting Jan to a hosted provider gives you a desktop interface to remote inference. Jan has Windows and Linux builds. On macOS, its [current local inference engine requires Apple Silicon](https://docs.jan.ai/desktop/install/mac). ### GPT4All [GPT4All](https://docs.gpt4all.io/) offers desktop applications for Windows, macOS, and Linux. It can run models without a dedicated GPU and includes LocalDocs for bringing local files into conversations. This makes it another option for someone who wants a chat application rather than a development environment. A CPU can run a model, although the speed you get still depends on the model size and your computer. Document chat adds work beyond generating an answer: the application needs to extract and search the document content. Check those settings as well as the chat model. ### Other packaged interfaces [KoboldCpp](https://github.com/LostRuins/koboldcpp) packages a GGUF runner with a browser interface and is worth considering for writing and story-oriented workflows. It offers downloadable builds and hardware acceleration options. The oobabooga project, long known as **text-generation-webui**, is now [textgen](https://github.com/oobabooga/textgen). It provides a local interface and APIs, with options for experimenting with generation and different model loaders. These are useful alternatives when their interface or supported loaders suit your workflow. Check the current releases for your operating system and GPU before choosing one. ## Model managers and command-line runners These are useful if you're comfortable in a terminal, or want a model that several applications can call. ### Ollama [Ollama](https://docs.ollama.com/quickstart) manages model downloads and runs models behind a local API. It is available for macOS, Linux, and Windows, and also has an app interface. Its main convenience is the shared service. A script, chat client, and coding tool can all connect to the same runner instead of each implementing model loading. Ollama also offers cloud models. Choose a downloaded local model when the goal is to keep inference on your computer. The product name alone doesn't tell you where a particular request runs. I have a separate [Ollama tutorial](https://flaviocopes.com/ollama-local-llm/) for installation, model commands, and calling it from JavaScript. ### llama.cpp [llama.cpp](https://github.com/ggml-org/llama.cpp) is an inference engine you can use directly. It supports CPU inference, Apple Metal, NVIDIA CUDA, AMD HIP, Vulkan, and other hardware backends. Its model format is GGUF. Choose this route when you want direct control over the model file, context size, and GPU offloading. You can use its command-line programs or run its HTTP server. It can split work between CPU and GPU. That allows some models to run beyond the GPU's own memory capacity, although the speed depends on how much work remains on the CPU. A suitable prebuilt binary can save you from compiling it. The build and backend still need to match your hardware. ### llamafile [llamafile](https://github.com/mozilla-ai/llamafile) focuses on distributing and running models through a portable executable approach. It is another way to use local inference without adopting a model-management service. It is worth looking at when portability matters, or when you want to distribute a runnable model package. Read its platform notes: executable handling and acceleration differ between operating systems, and some setups use a separate weights file. ### MLX-LM [MLX-LM](https://github.com/ml-explore/mlx-lm) runs and fine-tunes language models using MLX on Apple Silicon. It provides command-line tools as well as a library. This is a Mac-specific option for people who want to work directly with the MLX ecosystem. Choose a model conversion intended for MLX; a GGUF download belongs to a different runtime path. You can also use MLX through an application such as LM Studio. You don't have to use its developer tools just to benefit from the runtime. ## Local web interfaces and document chat A local API doesn't necessarily give you the chat experience you want. You may want saved conversations, file uploads, or an interface you can open from another device. [Open WebUI](https://docs.openwebui.com/) provides a self-hosted interface that connects to Ollama and OpenAI-compatible APIs. It can sit in front of a local backend or a remote provider. For example, you could run the interface and model on a desktop computer, then use a browser on your laptop. The desktop does the computation. This is inference on your own network, rather than inference on the laptop itself. [AnythingLLM](https://docs.anythingllm.com/) is another option, especially for document workspaces. Its configuration separates the language model, embeddings, and other services. It supports local providers as well as cloud ones. **Embeddings** turn text into numerical representations used for searching related passages. A document assistant may use an embedding model to find relevant text before giving that text to the chat model. If you want the whole workflow to stay local, check both models. Selecting a local chat model while using a cloud embedding provider can still send document content elsewhere. The same applies to OCR, web search, and external tools. ## Developer libraries and embedded inference You can put inference directly inside an application. This avoids requiring the user to install a separate model server, but your application takes responsibility for loading models and managing memory. [Hugging Face Transformers](https://huggingface.co/docs/transformers/en/installation) is a common route for working with downloaded model checkpoints. It is useful for experiments and applications that need direct access to model behavior. The available acceleration depends on the underlying framework, operating system, and hardware. For JavaScript applications, [Transformers.js](https://huggingface.co/docs/transformers.js/en/index) runs supported models through ONNX Runtime in the browser or Node.js. It covers more than chat, including tasks such as embeddings. Models need compatible converted files; downloading an arbitrary checkpoint isn't enough. [ONNX Runtime GenAI](https://github.com/microsoft/onnxruntime-genai) is another route for embedding generative models in applications. Its execution providers determine which hardware can run the computation. It's relevant when you're building an application around a supported ONNX model and deployment target. [mistral.rs](https://github.com/EricLBuehler/mistral.rs) offers local inference with a Rust library and server options. Despite its name, it supports multiple model families. It is worth considering for Rust applications or when its supported formats and hardware match your needs. For native software, llama.cpp and MLX also provide building blocks. The decision is whether you want inference inside your application process or behind an API that your app calls. A local server is usually easier to swap during development. Embedding a runtime gives you control over distribution, at the cost of handling more platform details yourself. ## Running an LLM inside a browser You can run inference inside a browser tab without installing Ollama or another native server. [WebLLM](https://webllm.mlc.ai/) uses WebGPU for browser inference. The browser downloads model assets and runs the computation on the user's device. Transformers.js also supports browser inference, including WebGPU and WebAssembly paths depending on the configuration and model. This approach is useful for demos, browser applications, and smaller models. Someone can open a page and try the model without installing a desktop application. There are limits. The initial download can be large, browser storage can be cleared, and GPU support varies by browser and device. A model that works in a native runtime won't necessarily have a compatible browser build. Also check what the page does with your inputs. Local computation is one property of the application; analytics, logging, and other network requests are separate properties. ## Dedicated inference servers A model on a personal laptop is one kind of local setup. A GPU workstation serving several applications is another. [vLLM](https://docs.vllm.ai/en/latest/getting_started/installation/) is designed for efficient inference and serving. It is worth considering when you need concurrent requests and throughput rather than a single chat window. [SGLang](https://github.com/sgl-project/sglang) is another serving framework, with scheduling and caching optimizations for language and multimodal models. Both require more attention to supported architectures, accelerator dependencies, and deployment configuration. Linux is a common target for GPU serving. Don't assume a desktop app's model file or a working GPU driver automatically gives you a compatible server setup. vLLM's [GPU installation documentation](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/) covers its hardware paths, including vLLM-Metal for Apple Silicon. Platform support is specific to the backend; there isn't one installation that behaves identically everywhere. [LocalAI](https://localai.io/) takes another approach: it provides compatible APIs in front of multiple inference backends. It can bring text generation and other local AI workloads behind one service. These servers can run on hardware you own. They can also run on rented infrastructure, but renting a server makes it self-hosted remote inference. It doesn't keep the computation on your computer. ## Running local LLMs on macOS The first distinction is **Apple Silicon versus Intel**. On Apple Silicon, desktop apps such as LM Studio, Jan, and GPT4All are options to check first. Ollama provides a managed service, llama.cpp gives you direct control through Metal, and MLX-LM offers the MLX route. Apple Silicon uses unified memory shared by the CPU and GPU. That can make a large memory configuration useful for local models, but macOS and your other applications need part of it too. A Mac with 16 GB of memory doesn't have 16 GB available exclusively for weights. For an Intel Mac, check support before downloading a large model. Current LM Studio and Jan local inference require Apple Silicon. A compatible llama.cpp CPU build is one possible route, with performance determined by the processor and model size. On either kind of Mac, a native runtime is a more direct route to hardware acceleration than putting everything inside a Linux container. Docker can still be useful for the web interface while a native runner handles inference. ## Running local LLMs on Windows You can start with a native Windows application. LM Studio, Jan, GPT4All, and Ollama all provide Windows options. llama.cpp and KoboldCpp also offer routes for running model files directly. Pay attention to the processor architecture. An x64 application and an ARM64 application aren't interchangeable native builds. Check the project's support for your machine, particularly on Windows on ARM. For GPU inference, match the runner to your card. NVIDIA, AMD, and Intel GPUs use different acceleration paths, and support varies by tool. Ollama publishes its own [hardware compatibility list](https://docs.ollama.com/gpu); use that rather than assuming any gaming GPU works. You can also use **WSL2**, the Windows Subsystem for Linux, when the tool you need targets Linux. GPU access depends on the supported hardware and driver setup. Follow the inference project's WSL instructions rather than its native Windows instructions. WSL adds another environment to manage, including storage and networking. If a native app already does the job, you can start there. For a Linux-oriented development or serving workflow, WSL may be the more useful route. An NPU advertised in a laptop doesn't guarantee acceleration for your chosen model. The model format, runtime, and device backend all need to support that path. ## Running local LLMs on Linux Linux supports desktop chat applications, command-line tools, and dedicated servers. You can use a graphical app on a workstation or run a model service on a machine without a display. LM Studio, Jan, and GPT4All have Linux distributions. Ollama and llama.cpp are options for a local API or terminal workflow. MLX-LM is the Apple Silicon route, so choose another runtime for an ordinary Linux PC. For NVIDIA hardware, CUDA is a common acceleration path. For AMD, check ROCm/HIP support for your exact GPU and Linux distribution. Vulkan is another option in runtimes that support it. Intel hardware has its own supported paths, including SYCL in llama.cpp. The driver and runtime versions matter. A working desktop display proves the graphics card can draw your desktop; it doesn't prove the inference backend can use it. On a headless machine, think about where models are stored, how the service starts, and which network interfaces it listens on. Containers can make deployment repeatable, but they still need access to compatible host drivers and GPU devices. ## Do you need a GPU? No. CPU inference is a valid option, particularly for small models, occasional use, and testing an application before buying hardware. A GPU generally makes generation faster when the runtime supports it and the model fits. That doesn't mean CPU inference is useless, or that every workload needs a GPU server. Memory is the first constraint to check. A rough estimate for weights is: ```text weights in GB ≈ parameters in billions × bits per parameter ÷ 8 ``` An 8-billion-parameter model stored at 4 bits needs roughly 4 GB for the weight values. The actual file includes other data, and the running model needs additional memory. **Quantization** reduces the precision of the weights so they use less space. It can make a model practical on a smaller machine, with a possible effect on answer quality. A 4-bit variant is a reasonable starting point when your runtime supports it. The **context window** also uses memory. Long conversations, large documents, and coding-agent requests can consume much more than a short chat. Concurrent requests add further pressure. On a discrete GPU, check VRAM separately from system RAM. Having 64 GB of RAM doesn't turn an 8 GB graphics card into a 64 GB graphics card. A runtime may offload some work to the CPU, but that changes performance. I explain the estimate in [how much VRAM a local LLM needs](https://flaviocopes.com/llm-vram-requirements/), and there's an [LLM memory calculator](https://flaviocopes.com/tools/llm-vram/) for trying different sizes. ## Model formats and compatibility A model family can appear in several formats. Choose the files for the runtime you've picked. | Format or packaging | Where you'll encounter it | What to check | | --- | --- | --- | | GGUF | llama.cpp and applications built around it | Model architecture support and quantization | | Safetensors checkpoints | Transformers and many serving frameworks | Configuration, tokenizer, dependencies, and supported architecture | | MLX model conversions | MLX-LM and MLX-enabled applications | A conversion supported by the MLX runtime | | ONNX models | ONNX Runtime and Transformers.js | Required files, operators, and execution provider | | Browser-specific model assets | WebLLM and other browser engines | Supported model build and browser GPU requirements | | Managed model tags | Catalog-based runners such as Ollama | The exact variant, size, and whether inference is local | The filename extension alone isn't a guarantee. A runtime must understand the model's architecture, and newer architectures may require an updated runtime. You may also see **base** and **instruct** variants. For ordinary conversation, start with a chat or instruct model. A base model is trained to continue text and may need more work to behave like an assistant. Image input, embeddings, tool calling, and structured output are separate capabilities. Check that both the model and runtime support the feature you need. A chat model doesn't automatically become a reliable coding agent because it fits in memory. ## Using Docker or another machine you own Docker is a way to package and operate a runner or interface. It doesn't replace the inference engine, supply extra memory, or automatically enable GPU access. It can be useful on Linux servers, especially when you want to reproduce a deployment. On macOS and Windows, account for the VM or WSL layer and check the documented acceleration path. Another option is to put the model on a machine you already own and connect over your local network. A laptop can use a desktop's larger GPU without loading the model itself. Keep the difference clear when handling sensitive data. The request now crosses the network and reaches another machine, even though you control both ends. A server listening only on `localhost` accepts connections from its own machine. Allowing other devices requires deliberate network configuration. Add authentication and appropriate network protection before sharing it; a local API may otherwise accept requests without credentials. ## How to choose your first setup I would start with the work I want the model to do, then choose the smallest setup that handles it. | What you want | A starting point | | --- | --- | | Chat without using a terminal | Compare LM Studio, Jan, and GPT4All | | A model that scripts and tools can share | Ollama or a desktop app's local server | | Direct control over GGUF files | llama.cpp | | Writing-oriented local chat | KoboldCpp | | MLX experimentation on an Apple Silicon Mac | MLX-LM | | Saved chats in a self-hosted web interface | Open WebUI with a local backend | | Chat with a collection of documents | AnythingLLM or GPT4All LocalDocs, with local embeddings | | Inference inside a web page | WebLLM or Transformers.js | | Inference embedded in your application | A compatible library such as llama.cpp, MLX, or ONNX Runtime GenAI | | Several applications or users sharing GPU capacity | Evaluate vLLM, SGLang, or LocalAI | Then I would download a small instruct model and test it on a few real tasks. For a document assistant, that means questions whose answers I can verify in the documents. For code, it means a small change with tests I can run. Check the time to the first answer, memory use, and whether the computer remains usable while generating. Try the context length you expect to need. A short greeting doesn't tell you how a model behaves with a long source file. If it's too slow, reduce the model size or context before changing every runtime setting. If the answers are poor, compare another model on the same prompts. A faster runner can't supply knowledge or abilities the model doesn't have. For offline use, download the required models first, select local providers, and repeat the complete task with networking disabled. That checks the actual workflow, including document processing and tools. Local inference can remove per-request provider charges, but hardware, electricity, and maintenance still cost money. I worked through that comparison in [running an LLM locally versus paying for an API](https://flaviocopes.com/local-llm-vs-api-cost/). Once you've chosen a runner, the free [Local AI Models course](https://flaviocopes.com/courses/local-ai-models/) goes into model selection, evaluating answers, and building a JavaScript application around a local model.