Retries and Idempotency for Model API Calls

A model request times out, and your worker sends it again. Before doing that, ask what actually failed: delivery, generation, or receipt of the response? For applications using CodeRelay Hub’s pay-as-you-go endpoint, design retries around the logical operation, not just the HTTP request.

1. Treat a timeout as an unknown outcome

A timeout means the caller stopped waiting; it does not prove that the server stopped working. Repeating the request can therefore repeat work. Separate connection failures from failures after transmission, and treat ambiguous outcomes differently from explicit rejections. Cancellation is not evidence that remote side effects were prevented. ([aws.amazon.com](https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/))

Assign a logical operation ID before sending anything. Keep it stable across retries, but generate a separate attempt ID for each transmission. This gives your logs a useful distinction between one user action and several network attempts.

2. Classify errors before retrying

Do not apply one policy to every non-success response. Authentication, permission, malformed-request, and oversized-request errors need correction rather than repetition. Inspect the structured error as well as the status: a 429 can represent temporary throttling or a spending restriction that waiting will not resolve. ([platform.claude.com](https://platform.claude.com/docs/en/api/errors))

• Potentially transient: Retry documented overload responses, selected server errors, and eligible transport failures within a bounded budget.
• Action required: Stop on invalid credentials, unsupported parameters, or exhausted spending allowances.
• Ambiguous completion: Reconcile existing work before launching another attempt when the outcome matters.

3. Budget time and retries together

Use capped exponential backoff with jitter rather than fixed, synchronized delays. Set both an attempt limit and an overall deadline. Retries increase load, and aggressive retries can worsen an overloaded dependency. A delay that exceeds the remaining deadline should end the operation, not extend it silently. ([aws.amazon.com](https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/))

For an illustrative starting policy, allow three total attempts, including the original. Choose each randomized delay between zero and the smaller of your backoff cap and the current exponential interval. Honor a documented Retry-After value as a minimum wait, and stop if that wait exceeds the deadline. Tune these settings from your own workload.

4. Verify what an idempotency key guarantees

An idempotency header is not a universal contract. Inspect the endpoint’s documentation for supported operations, retention, concurrent-request behavior, parameter matching, and error replay. Stripe’s documented behavior illustrates why these details matter: its API versions differ in how failed work is handled. Do not transfer another service’s guarantees to a model endpoint. ([docs.stripe.com](https://docs.stripe.com/api/idempotent_requests))

If supported, reuse the same key only for retries of the same operation. Keep the model, messages, tools, and generation settings unchanged. Treat a deliberate model fallback as a distinct generation attempt under the parent operation, not as permission to reuse an incompatible request key.

5. Deduplicate within your application

Create a durable operation record containing the tenant, operation ID, request fingerprint, state, and result reference. Enforce uniqueness on the tenant and operation ID. Atomically claim pending work; return the stored result for completed work, and reject reuse with a different fingerprint.

Use a lease and ownership check for workers, but do not mistake lease expiry for proof that an upstream generation stopped. Mark uncertain outcomes explicitly. This design can deduplicate delivery to your application without guaranteeing one upstream generation or one charge. Require documented upstream guarantees before promising either.

6. Handle streaming and tools separately

A successful HTTP response does not guarantee a completed stream. Streaming APIs can report errors after response headers have already been sent, so inspect stream events and completion signals rather than relying on the initial status alone. ([platform.claude.com](https://platform.claude.com/docs/en/api/errors))

Keep partial output separate from committed output. On restart, replace or explicitly version the partial answer instead of concatenating another generation. Execute tools only after complete argument validation. Give each business action its own durable deduplication key, and pass that key downstream when supported. Do not use newly generated tool-call IDs as the sole protection against duplicate payments or messages.

7. Test the retry boundary

Choose one layer to own retries. Independent retry loops at several layers can multiply downstream attempts, so inspect SDK, worker, and gateway policies together. ([aws.amazon.com](https://aws.amazon.com/builders-library/timeouts-retries-and-backoff-with-jitter/))

Inject failures before transmission, after server acceptance, during streaming, and after tool execution but before acknowledgement. Track attempts per operation, unresolved outcomes, duplicate suppression, and deadline exhaustion. Verify that a worker restart returns an existing result or enters reconciliation rather than blindly repeating a business action.

Open a free workspace in CodeRelay Hub.