Comparing Reasoning Models on Concurrency Bugs

Concurrency bugs are a useful stress test for reasoning-heavy models because plausible explanations are cheap and correct schedules are not. A model must connect shared state, synchronization, and execution order, then propose a repair that survives more than one interleaving. Compare models as debugging partners, not as generators of confident prose.

1. Build a small case with an explicit contract

Start with a bounded example from your own stack. Consider a background worker that processes queued jobs and shuts down on request. Its contract says that every accepted job finishes before shutdown returns, and no job is accepted after shutdown begins. Those two statements give the model something precise to prove or disprove.

Include the queue implementation, submission path, worker loop, shutdown method, and relevant runtime details. State whether the program uses threads, asynchronous tasks, or multiple processes. A thread lock and an asynchronous lock do not provide interchangeable guarantees.

2. Plant a bug with a known witness

Suppose submission checks a closing flag, releases its lock, and then enqueues a job. Shutdown sets that flag and enqueues a stop marker. A submitter can pass the check, pause, and enqueue its job after the stop marker. The worker exits first, leaving an accepted job stranded.

Write down that schedule before querying any model. Your reference answer should identify the competing operations, the missing atomicity boundary, and the violated contract. Also prepare a correct control case. A model that finds a race in everything is not a reliable reviewer.

3. Give each model the same debugging task

You could compare Claude Opus 5.5 with GPT-6.1 Sol, without assuming either will win. Through CodeRelay Hub, the same OpenAI SDK client can route requests through one endpoint, keeping the surrounding evaluation harness consistent. Keep provider-specific reasoning controls explicit rather than assuming identical parameter names imply identical behavior.

Use one prompt template and request four deliverables:

• A concrete interleaving that demonstrates the failure.
• The invariant that the interleaving violates.
• A minimal repair with its synchronization assumptions.
• A deterministic regression test and one limitation of the repair.

Keep source, tool access, output limits, and time limits consistent. Record reasoning settings where available. Run multiple independent trials, and preserve unsuccessful answers instead of selecting the most persuasive response.

4. Demand a schedule, not a label

Calling something a race condition earns little credit. The answer should say which operation pauses, which operation overtakes it, and what observable failure follows. For the shutdown example, ask exactly when submission counts as accepted. If that point is undefined, the model should surface the ambiguity before declaring a fix correct.

Watch for unsupported runtime claims. An interpreter lock does not make a multi-step protocol atomic. Likewise, replacing a container with a thread-safe queue does not automatically protect a lifecycle flag coordinated with that queue.

5. Test the repair under hostile scheduling

A proposed fix might hold the lifecycle lock while checking the flag and enqueueing the job, with shutdown using that same lock to enqueue the stop marker. That can restore ordering for an unbounded queue, but a blocking enqueue into a bounded queue needs further analysis. Holding a lock while waiting for capacity can create a deadlock if progress requires another operation that needs the lock.

Use barriers, events, or test hooks to force the original failing order. Avoid relying only on sleeps or repeated stress runs. Verify that the repaired program drains accepted work and terminates. Test cancellation, worker exceptions, repeated shutdown, and submission during shutdown separately.

6. Score evidence and operational cost

Score diagnosis, executable witness, repair correctness, regression-test quality, and stated assumptions separately. Have reviewers assess anonymized answers against the reference case. Treat passing tests as evidence, not proof, especially when the tests cover only selected schedules.

Track latency, token usage, and actual billed cost alongside correctness. A longer answer is valuable only if it improves the diagnosis or repair. Choose the model that reliably produces verifiable work within your operating constraints, and retain human review for consequential changes.

Open a free workspace to try your own comparison.