Picking the Right Model for Generating Tests
The best model for generating tests is not necessarily the one that writes the most code or produces the longest explanation. It is the one that reliably turns your requirements into executable checks that catch meaningful failures. With CodeRelay Hub, you can compare candidates through one API endpoint without rebuilding your integration for each provider. Start with a small evaluation, not a permanent commitment.
1. Define the testing job before choosing a model
Separate routine test expansion from work that requires deeper reasoning. Adding parameterized cases to a well-tested parser is different from discovering missing invariants in a payment workflow. Write down the expected output: unit tests, integration tests, regression reproductions, or property-based checks.
Also define what the model may change. A useful default is tests only, with production changes proposed separately. Otherwise, a model can make its tests pass by altering the behavior you intended to verify.
2. Build a small, representative evaluation set
Select tasks from your own repository. Include a straightforward function, a dependency-heavy component, an ambiguous boundary condition, and a historical defect. Keep the task instructions and repository snapshot identical across candidates.
- Provide the relevant implementation, interfaces, and nearby test examples.
- Include the exact test command and supported runtime versions.
- State behavioral requirements independently of the current implementation.
- Keep some known defects or mutations hidden from the generation prompt.
For a regression task, ask the model to write the test against the corrected version, then run it against the defective version too. A useful regression test should pass on the fix and fail on the defect for the intended reason.
3. Compare candidates without assuming a winner
Start with two candidates, such as Claude Haiku 4.5 and Claude Sonnet 5.5, as hypotheses to test rather than predetermined winners. Run both on the same tasks with the same tool access, output budget, and retry policy.
Record the exact model identifier and generation settings. Repeat tasks when practical, because one successful output does not establish reliability. If your workflow allows the model to inspect files and run tests, evaluate that workflow rather than a single isolated completion.
4. Score behavior, not the appearance of coverage
A generated suite can execute every line and still assert very little. First check whether it installs, compiles, and runs. Then inspect whether its assertions would detect incorrect behavior rather than merely reproduce implementation details.
- Correctness: Do expected values follow the specification?
- Fault detection: Does the suite reject relevant defective variants?
- Isolation: Are clocks, randomness, network calls, and shared state controlled?
- Maintainability: Are fixtures readable and mocks limited to appropriate boundaries?
Use mutation testing where available, but inspect surviving and killed mutations. A mutation score is evidence about the mutations exercised, not proof that the system is correct.
5. Test context selection before buying more capability
Missing context often looks like weak reasoning. A model cannot infer an undocumented retry contract or your preferred fixture pattern reliably. Supply a compact context bundle: the target code, its public contract, representative tests, and relevant dependency signatures.
Run a controlled comparison with and without that bundle. If both candidates improve substantially, fix retrieval and prompting before escalating model capability. Reserve larger-context workflows for tasks that genuinely require tracing behavior across multiple modules.
6. Measure the cost of an accepted patch
Track API usage alongside elapsed time, failed runs, retries, and reviewer effort. The useful denominator is accepted test changes, not generated tokens. A seemingly inexpensive response can become costly if it invents fixtures or requires repeated repair.
CodeRelay Hub provides a pay-as-you-go route through https://api.coderelayhub.com, making it practical to run a bounded comparison through a shared integration. Verify current model availability and pricing before setting your evaluation budget.
7. Route by evidence and keep evaluating
Choose the candidate that meets your acceptance criteria for each task class. Routine additions and cross-module investigations need not use the same model. Allow a bounded repair attempt with actual test output, then escalate or request human review rather than retrying indefinitely.
Re-run your evaluation after model updates, major framework changes, or shifts in repository architecture. Keep human approval for assertions that define security, financial, or compatibility behavior.
Open a free workspace to try your first comparison.