Quickly estimate token counts for Claude, GPT and Gemini — all models side by side, with context window check.
| {{ __t('th_model') }} | {{ __t('th_tokens') }} | {{ __t('th_context') }} | {{ __t('th_usage') }} | {{ __t('th_status') }} |
|---|---|---|---|---|
| {{ row.name }} | {{ row.tokens.toLocaleString() }} | {{ row.context.toLocaleString() }} | {{ row.pct.toFixed(1) }}% | {{ __t('status_ok') }} {{ __t('status_tight') }} {{ __t('status_over') }} |
Tokens are the building blocks an LLM breaks text into before processing it. A token is usually a word fragment — for English text roughly 4 characters or ¾ of a word; for Chinese/Japanese often 1–2 tokens per character. Tokens drive both cost (billed per million tokens) and capacity (context window in tokens).
Each provider trains their own tokenizer on a specific corpus. GPT-4o uses o200k_base (~200k vocab), older GPT cl100k_base (100k). Claude and Gemini have their own tokenizers with different subword splits. The same "internationalization" can be split into 4, 6 or more tokens depending on the tokenizer. Differences are largest for non-Latin scripts (Arabic, Thai, CJK).
A tokenizer is the layer that splits text into the units a language model processes. OpenAI has used Byte-Pair Encoding (BPE) since GPT-2, documented in the tiktoken library. Anthropic uses its own BPE vocabulary for Claude. Google Gemini uses SentencePiece. All three share one core idea: frequent letter sequences merge into one token, rare strings get split into multiple tokens. The word tokenization becomes two tokens in cl100k_base (GPT-4) — token and ization — while strawberry becomes three: straw, berry and the leading-space marker.
Vocabulary size is a key parameter. cl100k_base (GPT-4, GPT-3.5) has 100,256 tokens. o200k_base (GPT-4o, GPT-4o-mini) has 200,019 tokens — nearly double, which makes code, multilingual text and CJK scripts (Chinese, Japanese, Korean) more efficient. Claude (Anthropic) sits between the two and is especially dense for English (shorter strings = fewer tokens). Gemini SentencePiece is tuned for multilinguality: a Japanese sentence often costs 30-40% fewer tokens in Gemini than in cl100k_base.
These differences are not a detail — they directly drive cost and latency. Anthropic, OpenAI and Google bill input and output tokens separately (as of mid-2026). A 50,000-word German document sent to Claude and to GPT-4o typically shows a 10-20% difference on the bill — and at production scale (millions of tokens per day) that matters. The estimator here approximates via chars-per-token factors per model with a CJK-ratio adjustment; for exact counts use the official SDK (tiktoken, anthropic.tokenize_count) or Anthropic's /v1/messages/count_tokens API.
The tool uses an empirical heuristic that lands within ±10% of the true token count for natural-language text (no code, no base64):
// Per model: two factors
// chars_per_token = chars per token in Latin text
// cjk_chars = chars per token in CJK text
cjkRatio = (count of CJK codepoints) / (count of all codepoints)
eff = chars_per_token * (1 - cjkRatio) + cjk_chars * cjkRatio
tokens = ceil(charLen / eff)
// Factors (as of mid-2026):
// GPT-4o (o200k_base) : chars_per_token = 4.0 , cjk_chars = 1.5
// GPT-4 (cl100k_base): chars_per_token = 3.8 , cjk_chars = 1.4
// Claude : chars_per_token = 3.5 , cjk_chars = 1.3
// Gemini SentencePiece: chars_per_token = 4.0 , cjk_chars = 1.4
// Rule of thumb for English prose: 1 token ≈ 0.75 words ≈ 4 chars.Five everyday texts and their typical token size in GPT-4o (o200k_base). Numbers fluctuate ±5% depending on the exact content:
{}, [], ",) get dedicated tokens in BPE, which is why JSON is denser than YAML but looser than plain prose. For function-calling workflows: don't dump the whole dataset, filter in code first.cl100k_base about 900-1,100 tokens, in GPT-4o o200k_base about 650-800 tokens. For Japanese/Chinese workloads, model choice is also a cost choice: 30% fewer tokens = 30% lower bill.The heuristic is an approximation, not an exact counter. Three cases where it is systematically off: (1) code with many rare tokens — minified JavaScript, long hash strings, base64 blobs often need 1.5x more tokens than the rule of thumb gives. (2) images, audio, PDF attachments — multimodal models (GPT-4o, Claude Sonnet, Gemini) bill images as fixed token sums per resolution (e.g. ~765-1100 tokens for a 1024x1024 image in GPT-4o depending on detail). This is a text-only tool. (3) tool calls and function schemas — the JSON schemas you pass in tools=[] count too, often 1,000+ tokens per call. For exact billing: the official counters (OpenAI tiktoken, Anthropic /v1/messages/count_tokens) are authoritative. This estimate is for "does it fit in the context?" and capacity planning.
tiktoken (Python/JS), Anthropic /v1/messages/count_tokens, Google Vertex countTokens.cl100k_base vocabulary. GPT-4o and GPT-4o-mini instead use o200k_base and typically need 5-15% fewer tokens for the same input. Migrating an old GPT-3.5 app to GPT-4o means you cannot reuse the token estimate 1:1.