Why Token Counts Surprise New Users: 7 Steps

You send a short question, receive a short answer, and discover that the request used more tokens than expected. The useful debugging question is not simply how long the conversation looks, but what the model actually received and generated. Use this numbered walkthrough when instrumenting an integration through CodeRelay Hub.

1. Stop treating tokens as words

A token can represent a word, part of a word, punctuation, or another text fragment. Spaces and capitalization can change how text is split. Tokenization also depends on the model's encoding, so the same string does not have one universal token count. Treat character-based estimates as rough planning aids, not accounting records. ([help.openai.com](https://help.openai.com/en/articles/4936856-understanding-and-counting-tokens?utm_source=openai))

Try this: Compare a prose explanation with a JSON payload containing similar information using the target model's tokenizer. Inspect the actual splits instead of guessing from appearance.

2. Count the request, not just the question

The text box is not necessarily the complete model input. Instructions, message roles, tool definitions, structured-output schemas, and attached content can contribute to the request. Formatting tokens used to represent message boundaries may not appear in a local count of the user's text. A complete-input counting endpoint, where supported, is a better preflight check than tokenizing one string. ([developers.openai.com](https://developers.openai.com/api/docs/guides/token-counting?utm_source=openai))

Try this: Inspect a redacted outgoing payload. Inventory every message, schema, and attachment before deciding that the provider's usage report is wrong.

3. Follow the conversation into the next turn

A follow-up such as “now add tests” can be short while its input context is substantial. In manually managed conversations, earlier messages are supplied again to preserve context. OpenAI also documents that using a previous response identifier does not make the preceding input tokens free. A smaller HTTP payload is not proof of smaller model context. ([developers.openai.com](https://developers.openai.com/api/docs/guides/conversation-state?utm_source=openai))

Try this: Log input usage by turn. If it keeps climbing, inspect retained history and tool results. Remove obsolete material deliberately, preserving requirements and decisions that remain relevant.

4. Separate visible answers from generated usage

Reported output usage can exceed the tokens visible in the final answer. Reasoning models may generate internal reasoning tokens, and some models also generate non-visible formatting tokens. Output limits can cover these tokens as well as the answer itself. A short response therefore does not establish that generation was small. ([developers.openai.com](https://developers.openai.com/api/docs/guides/token-counting?utm_source=openai))

Try this: Record reported output usage and any reasoning breakdown separately. Check the endpoint's definitions before adding those numbers together; a breakdown may already be included in the total.

5. Distinguish cache reuse from context removal

Prompt caching reuses eligible input; it does not mean that the reused context disappears. Cache matching depends on the provider's rules. OpenAI documents exact-prefix matching and compatibility requirements involving settings such as the model and tools. Repeating roughly equivalent instructions is not the same as repeating an identical prefix. ([developers.openai.com](https://developers.openai.com/api/docs/guides/prompt-caching?utm_source=openai))

Try this: Keep stable instructions together and place changing task content afterward. Verify reported cache usage rather than assuming that similar requests must hit the cache.

6. Read usage categories before estimating cost

Input, output, cached input, and reasoning describe different aspects of usage. Their accounting relationships matter: OpenAI, for example, bills reasoning tokens as output tokens, while cached input can have different pricing from uncached input. Multiplying one total by one assumed rate can obscure those distinctions. ([help.openai.com](https://help.openai.com/en/articles/4936856-understanding-and-counting-tokens?utm_source=openai))

Try this: Build cost calculations from documented usage categories and the applicable rate card. Keep the original usage object so you can audit your interpretation later.

7. Make surprises reproducible

Adopt a small request ledger: request identifier, selected model, turn number, reported input and output, available cache and reasoning details, and completion status. Keep sensitive prompt content out of routine logs. When investigating a discrepancy, replay a redacted example and change one component at a time: history, tools, attachments, or output instructions.

The goal is not the smallest possible prompt. It is an explainable request whose context earns its place and whose reported usage you can reconcile.

Open a free workspace in CodeRelay Hub.