How to count tokens in a prompt

Free toolThis tutorial uses the LLM Tokenizer Visualizer. It runs in your browser, so open it in another tab and follow along.

To count the tokens in a prompt, paste it into a tokenizer that uses the same encoding as your model, and read the number it gives you. Characters and words don’t work, because LLM APIs bill and cut off text by tokens, and the ratio between characters and tokens changes with the language, the formatting and even the capitalization.

We’ll use the tokenizer visualizer on this site. It runs the real byte-pair encoding tables from the gpt-tokenizer library in your browser, so the count is exact for the encoding you pick, not a guess based on characters divided by four.

What a token is

A tokenizer splits text into chunks it has seen often, and gives each chunk a number. A common English word is often one token. A rare word, a long number or a word in a language the tokenizer saw less often is split into several.

The model receives that list of numbers, not your letters, and the list is what fills the context window and what you pay for. If you want the longer explanation, my free AI Fundamentals course has a lesson on tokens and the context window.

Count the tokens in your prompt

1. Open the tokenizer visualizer.
2. Pick an encoding. The default is o200k_base.
3. Paste your prompt into the Text box, or click Load sample to try it first.
4. Read the Stats line.

The sample is a tiny system prompt plus a question:

You are a helpful assistant.

Explain BPE tokenization in one sentence.

With o200k_base the stats line says 15 tokens and 71 characters. Switch to cl100k_base and the same text is 16 tokens.

The stats line also shows how many unique token IDs there are, the average characters per token, and a roundtrip check. Roundtrip means the tool decoded the tokens back into text and got exactly what you pasted.

Below the stats you see every token as a colored span. Hover a span to see its numeric ID. The Token IDs box lists the whole sequence, which is the list the model receives.

Your text stays in the browser. The page only puts the encoding in the URL (?enc=cl100k_base), never the prompt.

Pick the right encoding

The count is only exact for the model family that uses that encoding. The tool offers four, and labels each one with the models it belongs to:

All four are OpenAI encodings. Claude, Gemini and other model families use their own tokenizers, so for them the number is an estimate. It’s useful for comparing two versions of a prompt, but don’t use it to sit right at the edge of a limit.

Why the same text gives different counts

Paste a few variations and you’ll see the count move in ways a character count would never show. Here are real numbers from the tool, with o200k_base and cl100k_base:

The English and Italian sentences have the same length. With cl100k_base, the Italian one needs three more tokens. The Japanese sentence has about half the characters and still needs more tokens than both.

Spaces and capitalization matter too. Hello is one token with ID 13225. Hello, with a space in front, is a different single token with ID 32949. HELLO is two tokens.

Formatting is the one you can control. This JSON object is 25 tokens:

{"id": 42, "email": "[email protected]", "plan": "pro"}

The same object without the spaces after the colons and commas is 20 tokens:

{"id":42,"email":"[email protected]","plan":"pro"}

That’s 5 tokens saved on a 3-field object, and the saving repeats for every record you put in a prompt.

Count tokens in your code

The tool is good for checking a prompt by hand. In an app, count before you send the request. The same library the tool uses works in Node.js and in the browser:

npm install gpt-tokenizer

countTokens() returns the count with the default encoding, which gives the same numbers as o200k_base in the tool:

import { countTokens, isWithinTokenLimit } from 'gpt-tokenizer'

const prompt = 'You are a helpful assistant.\n\nExplain BPE tokenization in one sentence.'

countTokens(prompt) //15
isWithinTokenLimit(prompt, 10) //false
isWithinTokenLimit(prompt, 100) //15

isWithinTokenLimit() returns false when the text goes over the limit, and the token count when it fits.

To match another encoding, import it from its own path, for example gpt-tokenizer/encoding/cl100k_base.

What to watch out for

The chat format adds tokens

OpenAI’s chat format wraps each message in a few formatting tokens. Its cookbook counts 3 extra tokens per message, plus 3 to start the reply. So the number a chat request is billed for is a bit higher than the count of the text you pasted.

Other providers count on their side

Anthropic has a count_tokens endpoint (client.messages.countTokens() in their JavaScript SDK) that takes the same input as a message, including the system prompt, tools and images. Anthropic’s docs call the result an estimate that can differ from the real request by a small amount. The Gemini API has countTokens too. Use those when you need the exact number for those models.

The prompt isn’t the whole request

Tool definitions, images, earlier messages in the conversation and the model’s answer all count. Output tokens are billed separately from input tokens, so a short prompt with a long answer can still be expensive.

Once you have the count, the Token Counter & Prompt Cost tool turns it into a price across models, and the Context Window Fit Checker tells you whether your prompt, documents and history fit in a model’s window. If the bill is what you’re worried about, I wrote about prompt caching and compaction in coding agents, which is where long prompts get cheaper.

Try the LLM Tokenizer Visualizer

See the real BPE tokens, token IDs, and character ratio for pasted text.

Open the LLM Tokenizer Visualizer →

Related tools

• Token Counter & Prompt Cost: Paste a prompt, get an approximate token count and what it costs as input across every major LLM.
• Context Window Fit Checker: Paste your prompt stack and see which LLMs fit — with a bar showing how much of each model’s context window you use.
• Inference Cost Calculator: Estimate your monthly LLM inference bill and compare models across OpenAI, Anthropic, Google, and more.

More tool tutorials →

Want me to talk about your product? You can sponsor this site.