Concurrency Limits: An Eight-Step Design Walkthrough

Design concurrency control around a clear question: how much unfinished work should your application allow? For applications using CodeRelay Hub, put that policy around the shared API client so interactive requests, background jobs, and retries all pass through the same admission rules. The walkthrough below turns that policy into an implementation plan.

1. Separate concurrency from request rate

A rate limit controls how quickly requests start; a concurrency limit controls how many remain active. These are separate controls: Google Cloud Tasks, for example, exposes dispatch rate and concurrent dispatch settings independently. Your client should model them separately too. ([docs.cloud.google.com](https://docs.cloud.google.com/tasks/docs/configuring-queues?hl=en))

Create distinct configuration fields for maximum active requests, request-start rate, and any documented token budget. Do not assume that staying below one limit satisfies the others. Record the scope of each documented limit: account, credential, model, or another resource.

2. Gate requests before opening connections

Use a semaphore or bounded worker pool around each network attempt. Acquire a permit immediately before dispatch, and release it through guaranteed cleanup. Include connection establishment and response consumption inside that protected lifecycle.

For streaming, keep the local permit until the stream ends or transport cleanup completes, not merely until response headers arrive. Treat this as your application's accounting rule; confirm upstream accounting separately. Make completion, cancellation, and failure converge on one cleanup path, then test that permits cannot leak or be released twice.

3. Bound the waiting room

Define both a maximum queue length and a maximum queue wait. Before accepting work, check whether its remaining deadline leaves enough time for useful execution. Reject expired jobs before acquiring a network permit.

Give interactive work and bulk processing separate queues. Reserve some dispatch capacity for interactive traffic, while using fair scheduling to prevent indefinite background starvation. When the queue fills, return an explicit overload result or defer durable work. Do not silently accept jobs your service cannot finish within its contract.

4. Make retries re-enter admission control

Retries add load to a dependency that may already be struggling. Use capped exponential backoff with jitter and a bounded attempt budget rather than immediate repetition. AWS describes both retry amplification and the value of spreading attempts over time. ([d1.awsstatic.com](https://d1.awsstatic.com/builderslibrary/pdfs/timeouts-retries-and-backoff-with-jitter.pdf))

Release the active-request permit before sleeping, then send the next attempt through the normal admission path. Preserve the original end-to-end deadline across attempts. When Retry-After is supplied, interpret its delay-seconds or HTTP-date form and do not retry earlier; abandon the retry if the wait exceeds the deadline. ([rfc-editor.org](https://www.rfc-editor.org/rfc/rfc9110.html))

5. Account for uncertain outcomes

A client timeout does not prove that the server stopped working or that a side effect never happened. AWS specifically cautions against assuming failed responses mean side effects did not occur. ([d1.awsstatic.com](https://d1.awsstatic.com/builderslibrary/pdfs/timeouts-retries-and-backoff-with-jitter.pdf))

Separate logical job identifiers from attempt identifiers. Persist completed results before acknowledging durable jobs. Use idempotency mechanisms only where documented, and never invent an idempotency header. For interrupted streams, make partial output and replay behavior explicit instead of appending a fresh generation to an incomplete answer.

6. Coordinate across replicas

Decide whether your configured concurrency budget applies per process or across the deployment. For a shared upstream allowance, choose a central dispatcher, explicitly partitioned budgets, or a distributed lease mechanism. Document how scaling changes each process's allocation.

For leases, define expiration, renewal, ownership, and crash recovery. Treat lease expiry as permission bookkeeping, not proof that remote execution ended. That distinction follows the same timeout uncertainty: local failure does not establish remote completion. ([d1.awsstatic.com](https://d1.awsstatic.com/builderslibrary/pdfs/timeouts-retries-and-backoff-with-jitter.pdf))

7. Make fallback an explicit policy

Do not use model switching as an unbounded escape hatch. For CodeRelay Hub workloads, require each fallback to pass capability, context, output-format, and budget checks before dispatch. Give fallback attempts their own route budget while retaining the shared application ceiling.

Prefer deferring optional batch work over silently weakening an interactive task's requirements. Record which route produced the final result.

8. Verify behavior under overload

Test bursts, slow responses, stalled streams, cancellations, replica crashes, and exhausted retry budgets. Track active requests, queue depth, queue age, admission rejections, retries, and deadline failures separately. Define success as bounded resource use, predictable rejection, and recovery without leaked permits or duplicate job completion.

Open a free workspace to put these patterns into practice.