Monitor API Latency From Your Own Code

Start latency monitoring at the boundary your application controls: the code that sends requests and consumes responses. For an application using CodeRelay Hub, instrument that boundary once, then use the same measurement contract across model calls. This walkthrough builds a useful monitoring layer without turning request handling into a telemetry project.

1. Define what your timer measures

Define three application measurements before adding instrumentation: logical request duration, individual attempt duration, and time to first visible content. Start the logical timer immediately before calling the SDK. Stop it after consuming the response, or when the operation fails or is cancelled. Include retry delays in this measurement so it represents the wait experienced by your caller.

Use separate metric names for these boundaries. OpenTelemetry defines http.client.request.duration as a histogram measured in seconds; keep your broader application timer distinct rather than assuming an HTTP metric covers the entire SDK operation. ([opentelemetry.io](https://opentelemetry.io/docs/specs/semconv/http/http-metrics/?utm_source=openai))

2. Wrap the call with a monotonic timer

In Python, capture time.perf_counter() before the operation and subtract it from a second reading afterward. This clock is intended for elapsed-duration measurement and includes time spent sleeping. Keep a separate wall-clock timestamp for locating the event in logs. ([docs.python.org](https://docs.python.org/3/library/time.html?utm_source=openai))

Implement the wrapper with try, except, and finally. Initialize an outcome field, distinguish completion from errors and cancellation, and record elapsed time in finally. Preserve the original exception. Make telemetry emission best-effort: a failed metrics export should not replace the API result or hide the original failure.

3. Keep streaming measurements honest

For streaming calls, keep the wrapper active through stream consumption, not just stream creation. Define first visible content as the first nonempty text fragment your application would display. Record that milestone once. Treat metadata, empty deltas, and other event types separately rather than calling every received event a token.

Do not read the whole response inside a hook merely to finish a timer. HTTPX response hooks run before the response body is read, so a hook timestamp alone does not measure complete body consumption. Put your completion measurement around the consuming loop instead. ([python-httpx.org](https://www.python-httpx.org//advanced/event-hooks/?utm_source=openai))

4. Separate attempts from retries

Give each logical operation a correlation identifier and each attempt an ordinal. Record attempt duration where your SDK or transport exposes individual attempts. If that boundary is unavailable, report logical duration honestly instead of presenting it as a single network request.

Record timeout categories when available. HTTPX distinguishes connect, read, write, and pool timeouts; preserve those distinctions rather than merging them into one generic timeout label. Configure an application deadline separately if you need to bound the whole operation, including retries and stream consumption. ([python-httpx.org](https://www.python-httpx.org/advanced/timeouts/?utm_source=openai))

5. Export bounded, useful dimensions

Start with operation, requested model, streaming mode, deployment environment, and outcome. For example, a workload using Claude Haiku 4.5 can carry that requested model name without storing prompt content. Store any reported serving-model identity separately; do not silently substitute it for the requested value.

Keep request identifiers in logs or traces, not metric labels. Avoid user identifiers, raw prompts, and arbitrary error messages as dimensions. Each distinct attribute combination contributes to metric cardinality, so choose bounded values and monitor overflow if your metrics SDK enforces a limit. ([opentelemetry.io](https://opentelemetry.io/blog/2026/cardinality-limits-in-opentelemetry/?utm_source=openai))

6. Build a dashboard around distributions

Export durations as histograms, then display request volume, failures, cancellations, and latency percentiles together. Compute fleet-wide percentiles from aggregated histogram data rather than averaging percentiles calculated independently by each worker. Histogram configuration affects percentile accuracy, so choose boundaries around your actual latency objectives. ([prometheus.io](https://prometheus.io/docs/practices/histograms/?utm_source=openai))

Give first-content latency and full completion separate panels. Group comparisons by streaming mode and workload, and record input and output sizes as measurements rather than unbounded labels. Define alerts against your application’s latency objective, with a minimum traffic requirement and a sustained evaluation window.

7. Test the monitoring before trusting it

Exercise a normal response, delayed first content, an interrupted stream, a timeout, a retry, and caller cancellation. Confirm that every logical operation produces exactly one terminal measurement. Check that an incomplete stream is never labeled successful and that credentials and response bodies stay out of telemetry.

Finally, disable the exporter temporarily and verify that API calls still work. Your monitoring should explain slow requests without becoming another reason they fail.

Open a free CodeRelay Hub workspace to get started.