# How to build an LLM from scratch > A practical guide for software engineers to build and train a small GPT-style language model on a Mac with Language Model Builder. Author: [Flavio Copes](https://flaviocopes.com/about/) | Published: 2026-09-04 | Topics: [AI](https://flaviocopes.com/tags/ai/) | Canonical: https://flaviocopes.com/build-llm-from-scratch/ [Paul Graham recently said](https://twitter.com/paulg/status/2091544343589060625) that if he were 17 today, he would learn how to build LLMs from scratch and train the strongest one his hardware allowed. I think this is very good advice. In this guide we will do exactly that: train a small GPT-style language model on a Mac, starting from random weights. I'm a software engineer. I don't train language models for a living, and I don't talk to an audience of machine learning researchers. I talk to software engineers. So this is not a data science tutorial. We won't derive equations, implement matrix multiplication, or spend weeks learning a machine learning framework. We will approach a language model as a software system. We will identify its inputs, internal representation, processing pipeline, persisted state, build artifacts, runtime controls, logs, and failure modes. Then we will build one using [Language Model Builder](https://languagemodelbuilder.com/). We won't write Python. We won't import a finished model. We will train the model on our Mac, inspect its checkpoints, fine-tune its behavior, and test the result. This is the blog-post version of my free [Build a Language Model on Your Mac course](https://flaviocopes.com/courses/language-model-builder/). The course divides the same project into lessons, exercises, and quizzes. ## What we are building We are going to build a small GPT-style language model. It will receive a sequence of tokens and predict the next token. It will append that token and repeat the process to generate text. Here is the inference path: ```text text ↓ token IDs ↓ embeddings ↓ Transformer blocks ↓ next-token probabilities ↓ generated text ``` Training runs the same model, then adds a feedback loop: ```text prediction ↓ compare with the real next token ↓ measure the error ↓ update the weights ``` If you are a software engineer, you already know many of the ideas around this system. They just have different names. | Software engineering concept | Language model concept | | --- | --- | | input encoding | tokenizer | | internal data representation | embeddings | | processing pipeline | Transformer blocks | | persisted application state | weights | | versioned build artifact | checkpoint | | held-out test fixtures | validation data | | runtime options | sampling settings | The mapping is not perfect. It is still a useful place to start. ## What "from scratch" means "From scratch" does not mean writing every layer of the stack ourselves. Language Model Builder uses [MLX](https://opensource.apple.com/projects/mlx/), Apple's machine learning framework for Apple Silicon. MLX handles tensors, gradients, and the low-level work needed to run the model. This is similar to building a web application without implementing TCP, a database engine, and an operating system first. We will use the framework, but create the model state ourselves. Our project will have: - a tokenizer selected for our data - a new Transformer architecture - randomly initialized weights - our chosen training data - checkpoints produced on our Mac We will not import pretrained weights. That is the boundary that matters for this experiment. ## What you need Language Model Builder currently requires: - an Apple Silicon Mac - macOS 15 or later - free disk space for datasets and checkpoints - time for a training run The app is free. You don't need an account, API key, cloud GPU, or Python environment. Download it from [languagemodelbuilder.com](https://languagemodelbuilder.com/). ![The Language Model Builder download page](https://flaviocopes.com/images/build-llm-from-scratch/website.webp) Move it to your Applications folder and launch the app. You will see three choices: - **Learn about models** - **Build a model** - **Open an existing project** We will use the first two. ## Define the experiment before pressing Train Click **Build a model** and create a project. I used the default name, `My First Model`. Do not start training yet. We want a repeatable experiment, not one impressive screenshot. Write down this goal: ```text Observe how a small Transformer changes during pre-training, then test whether fine-tuning improves its responses. ``` This is the equivalent of defining acceptance criteria before implementation. Our model will not replace ChatGPT or Claude. We only need to prove that it moves from random output toward the patterns in its data. ### Create three test prompts Save these prompts: ```text Once upon a time, a small fox The robot opened the door and Mia looked at the sky because ``` We will run them against several checkpoints with the same runtime settings. Before training, the output should be random. That gives us a baseline. ### Create a run log Create a Markdown file named `model-lab.md`. Add these headings: ```markdown # My First Model ## Goal ## Mac and app version ## Model configuration ## Tokenizer and dataset ## Training log ## Checkpoint samples ## Fine-tuning ## Final comparison ## Limitations ## Next experiment ``` Record your Mac chip, unified memory, macOS version, and app version. Paste the goal and test prompts into the file. Think of this as a short runbook. It records the inputs, configuration, outputs, and decisions needed to reproduce the run. ## Do not import a base model The **Base models** section lets you import models such as SmolLM2 or Qwen2.5. That is useful when you want to fine-tune an existing [open-weight model](https://flaviocopes.com/open-weight-models/). It is not the path we want here. ![The Language Model Builder catalog of existing base models](https://flaviocopes.com/images/build-llm-from-scratch/base-models.webp) An imported model arrives with an architecture, tokenizer, and learned weights. We want to watch the weights develop from random values. ## Use the embedded guide as architecture documentation Open **Learn about models**. Language Model Builder includes an interactive guide that takes about 90 minutes. ![The introduction in the interactive Language Model Builder guide](https://flaviocopes.com/images/build-llm-from-scratch/guide-introduction.webp) My advice is to go through it once before starting the long training job. You do not need to become a machine learning researcher. Focus on the system boundaries: - how text becomes model input - what representation moves through the model - where context is combined - what the model returns - how training changes persisted state Let's translate the important parts into software-engineering terms. ## The tokenizer is the input layer A language model does not receive a string directly. A **tokenizer** splits text into reusable pieces and maps every piece to an integer ID. For example: ```text the little cat played ``` might become: ```text the | little | cat | play | ed ``` The model receives the IDs for those pieces. You can think of the tokenizer as a codec or parser at the system boundary. Both the producer and consumer must agree on the format. Language Model Builder includes a byte pair encoding playground. It starts with small pieces and repeatedly merges the most common adjacent pair. At zero merges, every character is a separate token. ![The tokenizer playground before learning any token merges](https://flaviocopes.com/images/build-llm-from-scratch/tokenizer-zero-merges.webp) Click **Run next merge** a few times. Watch repeated character sequences become single tokens. Then run 10 or 50 merges and compare the result. ![The tokenizer playground after learning 65 token merges](https://flaviocopes.com/images/build-llm-from-scratch/tokenizer-after-merges.webp) A larger vocabulary can represent familiar text with fewer tokens. But the model must also produce one output score for every vocabulary entry. This is a format and capacity tradeoff, not a question with one correct answer. The important compatibility rule is simple: > A checkpoint only works with the tokenizer and token IDs used to train it. Change the token mapping and the saved weights no longer mean the same thing. ## Embeddings are the internal representation Token IDs are identifiers. The numeric distance between two IDs has no useful meaning. An **embedding** converts each token into a vector: a list of learned numbers the model can process. This is the model's internal data representation. During training, tokens used in similar contexts can develop similar vectors. The app projects a 50-dimensional embedding space into two dimensions so we can see it. ![The Language Model Builder embedding map grouping related words](https://flaviocopes.com/images/build-llm-from-scratch/embeddings.webp) Animals, colors, places, and family words form visible groups. Nobody hardcoded those categories. The positions emerged from usage patterns in the data. Remember that the map is a debugging visualization. Flattening many dimensions into two distorts some distances. ## The Transformer is the processing pipeline The **Transformer** is the architecture that processes the token sequence. Each Transformer block contains attention and a small feed-forward network. Several blocks run one after another. Attention lets each position use information from other relevant positions in the current context. Consider this sentence: ```text The book did not fit in the bag because it was too large. ``` The token for `it` needs information from earlier tokens. Attention gives the model a way to combine that context. A model generating text from left to right cannot read future tokens. It can only use itself and earlier positions. This is called **causal attention**. The complete processing path is: ```text token IDs ↓ token and position embeddings ↓ repeated Transformer blocks ↓ one score for every possible next token ↓ probabilities ``` The model does not return a stored answer. It returns a distribution of possible next tokens. ## The weights are persisted application state The architecture defines the code path and data shapes. The **weights** are the numbers changed during training. They contain what the model learned from the data. At the beginning, those numbers are random. During training, the system repeatedly: 1. reads a batch of token sequences 2. predicts the next tokens 3. compares the predictions with the real tokens 4. measures the error 5. changes the weights slightly The error value is called the **loss**. We do not need to implement gradient calculation to understand the contract. The training framework receives a loss and updates the state in a direction that should reduce it. Training is an offline job that produces a new model artifact. Generation is the runtime that uses that artifact. Keeping those two phases separate makes the rest of the app much easier to understand. ## The model blueprint is a compatibility contract Return to the project and open **Project setup**. The blueprint I used for this run was: ```text tokenizer GPT-2 BPE vocabulary 50,261 tokens context 512 tokens embedding width 256 blocks 8 attention heads 4 parameters about 19.3 million weights on disk 36.8 MB ``` ![The project setup for my 19.3-million-parameter language model](https://flaviocopes.com/images/build-llm-from-scratch/project-setup.webp) These values define the shapes of the data moving through the model. The **vocabulary** controls the number of input embeddings and output scores. The **context length** controls how many recent tokens the model can process at once. The **embedding width** controls the size of the internal representation. The **Transformer blocks** control the depth of the processing pipeline. The **attention heads** let the model calculate several attention patterns in parallel. Together, these choices produce the parameter count. Once a checkpoint exists, you cannot change these shapes and load the old state into the new architecture. This is similar to changing an application's serialized data format without a migration. Record the complete blueprint in `model-lab.md`. ## Choose the tokenizer before training Language Model Builder recommends **Fast 10K BPE** for a smaller first model. I selected **GPT-2 BPE** for the run shown in these screenshots. Its vocabulary contains about 50,000 tokens, so it makes the input and output layers larger. That choice increased my model from about 9 million to 19.3 million parameters. It also made training take longer. You can choose Fast 10K BPE if you want a faster first experiment. The important part is to record the choice and keep it unchanged for the whole run. ## Treat the dataset as an input dependency Open **Pre-training data**. ![The Language Model Builder dataset catalog with TinyStories 2 selected](https://flaviocopes.com/images/build-llm-from-scratch/dataset-selection.webp) The catalog includes TinyStories, WikiText, Simple English Wikipedia, scientific abstracts, poetry, and other collections. Choose **TinyStories 2**. It contains short stories with a limited vocabulary. This gives a small model a narrow target it can start learning on one Mac. Inspect the dataset before downloading it. Check: - source - license - size - sample records - repeated boilerplate - broken text - private or unwanted content The model learns the patterns available in its data. A stories dataset produces a different system from scientific abstracts or documentation. The dataset is not an implementation detail. It is one of the main inputs to the build. If you later use your own data, keep the exact input or a versioned manifest of it. Otherwise, you cannot reproduce the result. ## Pre-training is a long-running background job Open **Pre-training**. Before starting, confirm the summary matches your notes: - dataset - tokenizer - parameter count - number of steps - estimated duration ![The pre-training screen before starting the 20,000-step run](https://flaviocopes.com/images/build-llm-from-scratch/pretraining-ready-download.webp) The recommended run uses 20,000 steps. On my M4 Pro, the app estimated between one and four hours. Your estimate will depend on the model, tokenizer, and Mac. Connect the Mac to power, then click **Start pre-training**. Training does not begin immediately. The app must first download the data the model needs. In my run, it downloaded about 2.1 GB of TinyStories 2 data from Hugging Face. ![Language Model Builder downloading the TinyStories 2 dataset before training](https://flaviocopes.com/images/build-llm-from-scratch/dataset-downloading.webp) This is the dataset download, not a pretrained model download. We are still starting with random model weights. After the download, the app prepares the training data and encodes the source files using the selected tokenizer. ![Language Model Builder encoding the downloaded source file before training](https://flaviocopes.com/images/build-llm-from-scratch/dataset-encoding.webp) The step counter starts moving only after this preparation finishes. Treat this like any long-running job. You want progress metrics, checkpoints, a way to pause, and enough information to resume after a failure. Record the start time, initial loss, validation loss, and tokens processed per second. ## Read the metrics like an engineer The training screen plots two loss curves. ![My model during pre-training with live training and validation loss](https://flaviocopes.com/images/build-llm-from-scratch/pretraining-live.webp) The blue line is **training loss**. It measures prediction error on the data currently used to update the weights. The orange line is **validation loss**. It measures error on held-out data that does not update the weights. If both improve, the model is learning patterns that also work on unseen examples. If training loss keeps falling while validation loss rises, the model is fitting the training data without improving on held-out data. This is **overfitting**. Do not react to one spike. Look at the trend across many measurements. Also watch: - current step - learning rate - tokens per second - elapsed time - estimated time remaining Tokens per second is a performance metric. It says how fast the job runs, not how good the model is. Loss is a learning metric. It does not prove the generated text is useful. We need both metrics and behavioral tests. At this point, we have to wait. I waited more than two hours for my run to complete. This is not a quick compile-and-test loop, so start it when you can leave the Mac working for a while. You do not have to wait for all 20,000 steps before exploring the rest of the app. Let pre-training run until checkpoints appear in the **Sample timeline**. Then click **Pause**. ![Pausing pre-training after the model has produced several checkpoints](https://flaviocopes.com/images/build-llm-from-scratch/pretraining-pause.webp) You can now open **Sampling** and try the latest checkpoint. This is a faster way to understand the complete workflow. Return to **Pre-training** and resume the run when you are ready. ## Checkpoints are versioned build artifacts A **checkpoint** is a saved copy of the model weights at one training step. Checkpoints let you: - pause and resume training - compare earlier and later model states - return to a better state if later training overfits - fine-tune a specific base state Record the step, training loss, validation loss, and sample outputs for each checkpoint you keep. The latest checkpoint is not automatically the best one. If the job takes several hours, let it reach a checkpoint before closing the app. Reopen the project and confirm the checkpoint history is present before assuming the run is recoverable. ## Sampling is runtime configuration Open **Sampling** and choose a checkpoint. Run the first baseline prompt: ```text Once upon a time, a small fox ``` ![Sampling text from a checkpoint after pausing pre-training](https://flaviocopes.com/images/build-llm-from-scratch/sampling.webp) The checkpoint determines the model's probabilities. The sampling settings determine how the runtime chooses from them. Changing these controls does not retrain the model. **Temperature** changes how strongly generation favors the most likely tokens. **Top-k** keeps only the `k` strongest candidates. **Top-p** keeps the smallest group of candidates whose combined probability reaches the selected value. **Min-p** removes candidates that are extremely weak relative to the strongest one. The random **seed** makes a sample reproducible when the checkpoint, prompt, and other settings stay the same. This is the model equivalent of testing the same build with different runtime flags. For checkpoint comparisons, keep every setting fixed. ## Decide when the run is good enough There is no universal loss number that means the model is finished. Use three signals together: 1. validation loss 2. the gap between training and validation loss 3. samples from fixed prompts and settings If validation loss and samples keep improving, more training may help. If validation loss has stopped improving, more steps may waste time or make the model worse on unseen data. Our 19.3-million-parameter model will remain limited. The goal is visible, explainable improvement. ## Pre-training does not define product behavior Pre-training teaches the model to continue text. It does not create a reliable chat interface. A pre-trained model may complete this pattern: ```text User: Tell me a short story. Assistant: ``` But it was not specifically trained to treat the first line as a request and the second as a response boundary. That behavior comes from another training stage. ## Supervised fine-tuning teaches the interaction contract **Supervised fine-tuning**, or SFT, trains the model on prompts and desired responses. Open **Fine-tuning data** and inspect a conversational dataset. Each example usually contains: - a system instruction - a user message - an assistant response Reserved chat tokens mark where each role starts and ends. From a software-engineering perspective, SFT teaches the model the interaction contract we expect at runtime. Open **Supervised fine-tuning**, select the checkpoint with the best validation result, and use the recommended settings. After SFT saves a checkpoint, repeat the three baseline prompts with the same sampling settings. Record what improved and what regressed. SFT changes behavior. It does not magically add all the missing knowledge of a larger model. ## DPO is an optional policy-tuning stage **Direct preference optimization**, or DPO, learns from pairs of preferred and rejected answers. Open **Direct preference optimization**. The app runs the same prompt twice. Choose the answer you would rather receive. The app stores: - the prompt - the chosen answer - the rejected answer Four pairs unlock a run. That is enough to observe the mechanism, not enough to define broad product behavior. Think of preference pairs as policy tests. If the criteria are inconsistent, the training signal will be inconsistent too. For a first experiment, DPO is optional. Pre-training plus SFT already teaches the main pipeline. ## Use chat as an integration test Open **Chat with your model**. Choose the checkpoint you want to test: pre-trained, SFT, or DPO. Start with prompts that match the training data: ```text Tell me a story about a brave little boat. ``` Then test inputs outside the happy path: - a malformed request - a request outside the dataset's domain - a long conversation near the context limit - a prompt with unusual names or symbols Record the exact system instruction and sampling settings. Chat makes the model look familiar, but it is still a runtime calling next-token prediction repeatedly. ## X-ray mode is the debugger Click **X-ray** in the chat toolbar. The response becomes a sequence of individual tokens. Stronger colors indicate higher sampling probabilities. Click one token to see the alternatives available at that position. This view is useful when a generation surprises you. It can show whether one token dominated or several choices had similar probability. It does not reveal a human-like thought process. It shows the state exposed by this stage of the generation pipeline. The transcript panel shows the raw role tokens and messages sent to the model. The context meter shows how much of the 512-token window is in use. When the conversation exceeds that window, the runtime cannot keep every old token in the active input. This is a real system limit, not an unlimited memory feature. ## Build an evaluation instead of judging by vibes Run the same three prompts against the pre-trained, SFT, and DPO checkpoints. Keep these values identical: - temperature - top-k - top-p - min-p - maximum tokens - random seed - system instruction Save every output, including the bad ones. Add a table to `model-lab.md`: ```markdown | Prompt | Pre-trained | SFT | DPO | | --- | --- | --- | --- | | Small fox | | | | | Robot door | | | | | Mia and the sky | | | | ``` Evaluate things you can observe: - complete sentence - repetition - stays on topic - follows the requested format - unsupported claims Keep the raw output below the table. A score without the generation is hard to audit. Choose the checkpoint you would keep and write one reason supported by the results. ## Exporting a model means exporting an artifact Language Model Builder can export weights using the `safetensors` format. The weights are the artifact produced by training. They are not the complete application. A compatible runtime also needs: - the architecture definition - the tokenizer - the vocabulary and token mapping - the expected chat format - the sampling configuration Keep the Language Model Builder project and `model-lab.md` beside the exported weights. The artifact is much easier to use when its configuration and provenance are clear. ## How I would use this as a software engineer I would build this small model once to remove the magic from the stack. I would not use it as the base of a customer-facing general assistant. It is too small, too narrow, and too expensive to improve into a competitive model by myself. The value is the mental model. When an AI feature behaves badly, I can now separate several possible causes: - the tokenizer handled the input poorly - important context did not fit in the window - the model never learned the required domain - the fine-tuning data taught the wrong interaction pattern - the prompt is ambiguous - the sampling settings are too aggressive - the application failed to validate the output Those are different bugs. They need different fixes. This experiment also makes model APIs easier to reason about. An API hides training, weights, and most of the runtime, but the same boundaries still exist. If you are deciding between running models yourself and paying for an API, I did the math in [Running LLMs locally vs paying for an API](https://flaviocopes.com/local-llm-vs-api-cost/). For production software, I would usually start with an existing model and build strong application boundaries around it: - validate inputs - control context - request structured output where possible - add deterministic code for business rules - test representative prompts - log failures without leaking private data - keep the model provider replaceable Building one tiny model helps explain why those practices matter. ## Try one controlled change next Once the default project works, create a second project. Change one variable. For example, keep the architecture, tokenizer, and step budget fixed. Replace TinyStories with text you own, such as public blog posts or product documentation. Clean the input first. Remove duplicated pages, navigation, broken encoding, secrets, and content you do not want the model to imitate. Then compare: - tokenization - training speed - validation loss - outputs from the same test prompts - repeated failure patterns Because only the dataset changed, you can explain the difference more confidently. Other useful experiments include changing the context length, Transformer depth, embedding width, tokenizer, or training duration. Change one thing per run. ## Common software-engineering mistakes in this experiment ### Starting with the largest model A larger model makes every feedback loop slower. Finish one complete small run before increasing capacity. ### Changing configuration without a new run ID Give every run a clear identifier, such as `starter-tinystories-20k`. Record the full configuration beside its checkpoints. ### Looking only at training loss Training loss can improve while held-out behavior gets worse. Use validation loss and fixed behavioral tests too. ### Comparing different runtime settings Two outputs are not comparable if temperature, seed, prompt, or system instruction changed. Keep the runtime fixed while comparing checkpoints. ### Keeping only the best output One good sample is a demo, not an evaluation. Keep weak outputs and recurring failures. ### Treating local data as automatically safe Local training keeps data away from a cloud training service. It does not grant permission to use private, copyrighted, or licensed material. Use data you are allowed to use. ## How this differs from a production LLM Our model uses the same central pipeline as larger GPT-style models: - tokenization - embeddings - causal attention - Transformer blocks - next-token prediction - gradient-based training - supervised fine-tuning - probabilistic sampling The scale is completely different. Production models can have billions of parameters. They train on enormous datasets using distributed accelerator clusters and extensive evaluation systems. Even just running one of those models locally has real hardware requirements. I covered them in [How much VRAM do you need to run an LLM locally?](https://flaviocopes.com/llm-vram-requirements/) Language Model Builder keeps the system small enough to run on one Mac. That is the reason it works as a learning tool. It removes the distributed infrastructure and huge budget while preserving the boundaries a software engineer needs to understand. ## What you built You started with random state. You defined the input encoding and architecture. You selected the source data, ran a long background job, watched metrics, saved versioned artifacts, changed the interaction behavior, and ran repeatable tests. That is a software project. The output happens to be a language model. You now know what sits behind an LLM API: ```text architecture + tokenizer + data + learned weights + behavior training + runtime sampling + application code ``` Download [Language Model Builder](https://languagemodelbuilder.com/) and complete the smallest recommended run first. If you prefer individual lessons and quizzes, follow the free [Build a Language Model on Your Mac course](https://flaviocopes.com/courses/language-model-builder/). You won't build ChatGPT on a laptop. But you will understand the system well enough to build better software around language models.