Your Token Count Is Not a Property of Your Text

Ask how many tokens a piece of text is, and the honest answer is a question back. Which tokenizer? A token count is not a measurement of your text. It is a measurement of your text and a tokenizer together.

Change the language and the number moves by up to 15 times. Change to a newer model from the same vendor and it moves about 30 percent. The characters never changed.

Key takeaways#

• The same content can need 15× more tokens in one language than another.
• Anthropic's own docs say a newer tokenizer yields about 30% more tokens for identical text.
• The cost is not only money. Fewer of your words fit in the same context window.
• Third-party estimators are wrong by construction, since they model a different tokenizer.
• Counting tokens against the exact model is free, and most people still estimate.

What does a token count actually measure?#

It measures how a specific tokenizer splits your string. Tokenizers learn a vocabulary of frequent character sequences from training data. Text resembling that data compresses into fewer, longer tokens. Text that does not gets chopped into more, shorter pieces. The number therefore describes a relationship, not a property.

That is why two vendors give different answers for the same paragraph, and neither is wrong.

The rule of thumb has no error bars#

Almost every guide offers the same shortcut: roughly four characters per token, or about three-quarters of a word. It is a reasonable average for ordinary English prose.

The trouble is how it gets used. People treat it as an approximation with a small error. The real spread depends on inputs the rule never mentions. Language is one. Model version is another. Content type is a third.

An average across English prose tells you nothing about a JSON payload, a Thai sentence, or a column of UUIDs. So the shortcut is not wrong, it is just answering a narrower question than the one being asked.

Language moves the number by up to 15×#

The strongest evidence comes from Petrov and colleagues at NeurIPS 2023, who measured this directly across languages.

The same text translated into different languages can have drastically different tokenization lengths, with differences up to 15 times in some cases. These disparities persist even for tokenizers that are intentionally trained for multilingual support.

Read the second sentence carefully. This is not a problem that multilingual training has already fixed, because those tokenizers show the disparity too.

Their finding extends further than subword models:

Character-level and byte-level models also exhibit over 4 times the difference in the encoding length for some language pairs.

So even at the byte level the gap does not close. Some writing systems simply need more bytes per unit of meaning, and the tokenizer inherits that.

The bill is only part of it#

Here is the consequence that most coverage drops, and it is in the same abstract.

This induces unfair treatment for some language communities in regard to the cost of accessing commercial language services, the processing time and latency, as well as the amount of content that can be provided as context to the models.

Three costs, not one. Money, latency, and how much fits.

That third one is structural. A context window is measured in tokens. So a user writing in a heavily fragmented language gets materially less usable window from the identical model.

Two people buy the same product and one receives a fraction of the working memory. That is a different kind of problem from a larger invoice. The mechanics of that limit are covered in why LLMs have context windows.

Your own vendor moves the number too#

You do not need two languages to see this. One vendor changing models is enough, and Anthropic documents it plainly.

Claude 4.7 and later models and Claude Mythos Preview use a newer tokenizer. The same input text produces approximately 30 percent more tokens than on earlier models. The exact increase depends on the content and workload shape.

Note the last sentence. Even the 30 percent is not a constant, because it varies with what you are sending.

The docs give a direct instruction about what to do.

Recount prompts against the model you plan to use rather than reusing counts measured against earlier models.

They repeat it for billing, warning against reusing old counts "to estimate costs or context window fit". So a number you measured six months ago describes a model you may no longer be using.

A worked example#

Take a service sending 10 million input tokens a month, measured on a pre-4.7 model.

Move the identical traffic to a newer tokenizer and the same text bills at roughly 13 million tokens. Nothing about your prompts changed. Assume five dollars per million input tokens. That is fifteen dollars extra a month, which nobody notices.

Now scale it. A service at a billion input tokens a month gains 300 million tokens. That is about fifteen hundred dollars, for identical content. Still not catastrophic, and now large enough that a finance team asks what shipped.

The context effect bites sooner than the bill. A prompt you tuned to 150,000 tokens against a 200,000 window becomes roughly 195,000 on the newer tokenizer. You have not exceeded the window. But your headroom fell from 50,000 tokens to 5,000, and the next long document tips it over.

A table for the decision#

Estimating is fine in some places and reckless in others, so here is where I draw the line.

Situation	Approach	Why
Rough capacity planning, English prose	Four-characters rule	Cheap, and close enough at that resolution
Any billing forecast	Count against the real model	The rule has no error bars for your mix
Fitting a prompt to the window	Count against the real model	Headroom errors truncate silently
Multilingual product	Count per language	A single ratio hides a 15× spread
Chunk sizing for retrieval	Count once, then use characters	Stable proxy after you calibrate it
Migrating models	Recount everything	The vendor says so explicitly

Row six deserves emphasis because it is cheap and skipped. Counting tokens through the endpoint is free, and rate limits run to thousands of requests per minute.

Code and JSON are their own case#

Language gets the attention, and content type does the same thing quietly. That matters more than it sounds on a developer-tools site, because most of what we send a model is not prose.

Think about what a tokenizer learned. Its vocabulary rewards sequences that appeared often in training, and ordinary English words appeared constantly. A camelCase identifier, a UUID, a base64 blob and a run of punctuation did not.

So those fragment. A word like configuration may be one or two tokens, while cfgMgrInstance_v2 can be several, and neither the character count nor the four-characters rule warns you.

Structured data is the sharpest case. JSON spends many of its characters on quotes, braces, colons and repeated key names. Minified JavaScript and base64 sit near the worst case, since they resemble nothing in the training distribution.

Anthropic's docs concede the point in one clause. They say the increase from a tokenizer change "depends on the content and workload shape", so content shape is a variable in their own accounting.

The practical move is to stop treating your corpus as one thing. Measure a token count separately for prose, source code, JSON payloads and any encoded blobs you send. A single ratio averaged over all four describes none of them.

The failure mode, and how to detect it#

The estimate never announces itself as wrong. Your code computes a number and it looks plausible. Then the request either costs more than forecast, or gets truncated at the edge.

Truncation is the one that hurts. Trim a prompt to fit using an estimate that undercounts, and the API sees more tokens than you intended. The tail of your context then disappears. The reply reads fine, because a model given 90 percent of a document still answers confidently.

So measure the gap directly rather than trusting either number. Send a representative sample through the counting endpoint. Compare it against whatever your code estimated, and record the ratio per content type. English prose, source code, JSON and non-English text each deserve their own number.

Then alert on drift. Re-run that comparison whenever you change models, because a vendor can change the tokenizer under a familiar-looking model name, and your estimator will not notice.

One more check is worth building in. Log actual input_tokens from responses alongside your predicted value, so the two diverge visibly in a dashboard instead of quietly in a bill.

What a character count gets right#

There is one number here that does not move, and it is worth naming.

A character count is a property of the text alone. Nobody's tokenizer changes it, no model version shifts it, and it is identical in every language for the same string. Our character counter and word counter give you a figure that will still be true next year.

That stability makes characters a good proxy, once calibrated. Measure the token-to-character ratio for your own content and model, then use characters day to day and re-measure when something changes.

Calibrating takes one afternoon. Pull fifty real samples per content type, count each one against your production model, and divide tokens by characters. Store the ratio per type, and treat any sample more than about 20 percent from its type average as a sign the type needs splitting further.

You then get a stable unit for your code and an accurate one when it matters.

The competing view deserves a hearing too. If your product is monolingual English prose on a fixed model, the four-characters rule is genuinely fine, and building measurement infrastructure to replace a good-enough heuristic is effort spent on the wrong problem.

Where this advice is wrong#

Most teams will never feel this. An English-only product on one model has a stable ratio, and treating the rule of thumb as reliable will serve it perfectly well for years. The variance matters at the margins: multilingual users, tight context budgets, and model migrations.

The 15× figure is a maximum, not a typical case. It describes the worst language pair the researchers found, and most pairs sit far closer together. Quoting it as though every non-English user pays fifteen times more would be dishonest.

Counting has its own costs. The endpoint is free but not instant, and calling it on every request adds a round trip to your latency budget. Count during development and on samples in production, rather than in the hot path of every user request.

The short version#

A token count describes your text and a particular tokenizer, so it changes when either one does. Across languages the same content can differ by up to 15 times, and Anthropic's own docs put a newer tokenizer at roughly 30 percent more tokens for identical input. The consequences are money, latency and how much of your document actually fits in the window. So calibrate a character-to-token ratio for your own content, count against the exact model before you forecast a bill or trim a prompt, and re-measure the day you change models.

Sources: Petrov et al., Language Model Tokenizers Introduce Unfairness Between Languages (arXiv:2305.15425), the same paper at NeurIPS 2023, Anthropic: token counting