Estimate Tokens Before a Whole-Repository Scan

Before asking an agent to inspect an entire repository, define what you intend to send and how many passes you expect. Build two estimates: the tokens in your selected repository snapshot and the cumulative tokens across the scan workflow. Use this walkthrough to make both estimates reproducible.

1. Define the scan boundary

Write a scope statement before counting: review application code, tests, configuration, and architecture documentation at a specific commit. Decide whether local modifications belong in the snapshot. Record submodules and external dependencies separately rather than silently treating them as covered.

Start your inventory with git ls-files -z for tracked paths. If untracked files belong in scope, collect them with git ls-files --others --exclude-standard -z. The latter applies standard Git exclusions; do not assume it filters already tracked files. Preserve null-delimited paths when processing filenames. ([git-scm.com](https://git-scm.com/docs/git-ls-files?utm_source=openai))

2. Build an auditable inclusion manifest

Inspect each candidate before reading its contents. Exclude secrets, binary assets, dependency directories, generated bundles, and build outputs unless the review explicitly requires them. Treat lockfiles deliberately: include them for dependency analysis, or record their exclusion for a source-focused review.

For each selected file, record its path, content hash, byte count, category, and inclusion reason. Log unreadable files and decoding failures instead of silently skipping them. Set explicit policies for symbolic links and unusually large files. Your manifest should explain exactly what the estimate covers.

3. Count with the intended model

Prefer a tokenizer explicitly matched to the target model or the provider's token-counting interface. Anthropic's counting endpoint uses the tokenizer associated with the supplied model, so repeat the measurement when changing model families rather than carrying one count forward. ([platform.claude.com](https://platform.claude.com/docs/en/build-with-claude/token-counting?q=count_tokens))

Google also provides a countTokens interface for Gemini requests. Check whether your chosen routing path exposes the relevant counting operation; a shared inference endpoint does not establish that every provider-specific utility is available. ([ai.google.dev](https://ai.google.dev/gemini-api/docs/tokens?utm_source=openai))

When working through CodeRelay Hub, keep the route and model identifier alongside the estimate. If exact counting is unavailable, label the result approximate. Sample files across source code, tests, documentation, and structured data; calculate tokens per byte within each category, then apply those measured ratios to the corresponding byte totals.

4. Measure the assembled request

Count the payload you will actually submit, not just concatenated file bodies. Include path labels, separators, line numbers, instructions, and any conversation history you plan to retain. Anthropic's counting interface accepts structured message inputs, including system prompts and tool definitions, making these part of the request-level measurement. ([platform.claude.com](https://platform.claude.com/docs/en/api/messages/count_tokens?utm_source=openai))

Use per-file counts to identify expensive files and plan batches. Then count each fully assembled batch separately. Keep a content-only subtotal and a request-level total so reviewers can distinguish repository material from orchestration overhead.

5. Budget the workflow, not one upload

List the planned stages: inventory, subsystem review, cross-cutting analysis, verification, and final synthesis. For each stage, specify which files, summaries, instructions, and previous messages you intend to send. Add those request estimates together rather than multiplying the repository total by an unexplained safety factor.

For each request, reserve an explicit output allowance and verify the selected model's current context and output limits. Provider documentation recommends token counting to fit prompts within context constraints; use the measured request size as your starting point, not file count. ([platform.claude.com](https://platform.claude.com/docs/en/build-with-claude/token-counting?q=count_tokens))

6. Run a pilot and reconcile

Start with one representative subsystem. Record the predicted input, reported usage, output allowance, actual output, files inspected, and additional tool calls. Investigate discrepancies before expanding the scan. Update the remaining estimate using the observed request structure, not an arbitrary universal multiplier.

Finish with a compact ledger: snapshot identifier, inclusion manifest, counting method, model, estimated input by stage, output allowances, and unresolved exclusions. Keep token estimates separate from monetary forecasts; calculate spending only after checking applicable rates and billing categories.

Open a free workspace in CodeRelay Hub.