What is a token? Its relationship to text length, usage, and context budgets
Understand token units, counts for Chinese and code, input-output-reasoning usage, and how to account for a complete request.
Understand the concept
A token is a discrete unit used to process text, potentially a character, part of a word, or punctuation depending on the tokenizer. It is not a fixed number of Chinese characters or English words. Budgets and usage often use tokens, but accounting differs by model and interface. Use the target model’s counting method and inspect reported usage.
The same text can have different counts across models
Chinese prose and code with the same character count need not use equal tokens. Spaces and capitalization can affect segmentation. An exact count needs a specified tokenizer; common length estimates are only rough planning tools.
Tokens map to vocabulary IDs, which are not universal across models. The same ID can mean something different under another encoding. Numerical IDs do not measure word importance or factual reliability.
Request usage includes more than visible answer text
A request can include role markers, tool definitions, and processed files. Counting only the user’s sentence can miss other content. OpenAI distinguishes plain-text and full-request counting; use the relevant method and compare it with reported usage.
Some reasoning models consume tokens not shown in the final answer, so short responses need not mean low usage. Cached input can be counted separately and is not necessarily cost-free. Check the service’s current accounting and prices.
Estimate first and compare actual usage for diagnosis
Count a representative input with the target model’s tool and reserve output space. Growing conversation history changes the count. This explains accounting without paid calls, cost measurements, or a universal language conversion ratio.
Separate context capacity, output limits, and per-minute rate limits. Reducing one input can address context without fixing high concurrency. Record complete-request counts and finish reasons instead of treating all three problems as simply too few tokens.
Checked sources
These sources support the concepts and methods described here. Examples and test scope are labeled separately; check the version of the tools you use.