A rerun is not automatically bad. It can verify a hypothesis, recover from a known environment incident, or show whether a failure repeats under controlled conditions.
The problem begins when the final green result replaces the history. Passed on retry
and passed first time are different quality facts.
First identify what is being rerun
Teams use “rerun” for several operations:
- rerun the failed test inside the same test task
- rerun a pipeline job or stage
- create a new run of the whole pipeline
- manually repeat a scenario outside the original run
These have different inputs, environments, and evidence. Record the operation precisely. A full pipeline rerun may use a newer dependency or mutable environment even when it uses the same commit. A test-task retry may run in a fresh worker while keeping the same job.
Azure DevOps can display summarized test results for rerun attempts, including multiple attempts under the logical test. Microsoft’s test-results documentation also describes result history, errors, stack traces, attachments, and the surrounding run timeline. Start there before inventing another record.
The evidence that must survive
For every operationally meaningful rerun, retain:
| Evidence | Why it matters |
|---|---|
| Original run and attempt | Establishes what first failed |
| Test identity and exact error | Prevents unrelated failures being grouped as one |
| Candidate, branch, and commit | Shows whether the subject stayed comparable |
| Environment and data state | Exposes changed conditions |
| Rerun actor and time | Creates ownership and sequence |
| Stated reason or hypothesis | Turns a click into an experiment |
| Rerun scope | Shows whether one test or the whole gate was repeated |
| Result and diagnostic artifacts | Supports the next conclusion |
A team does not need a long form before every click. A short reason such as rerun after identity provider recovery is already more useful than an unexplained green attempt.

Native Azure DevOps capture from synthetic PlaneFlow run 345 on 29 October 2026. The screen preserves the run and failure evidence; it does not explain why a later attempt might pass.
A green rerun has several possible meanings
The second attempt passed. That could mean:
- a confirmed external incident ended
- the test data was repaired
- a product race did not reproduce
- an unstable test took a different timing path
- a dependency returned a different response
- the rerun exercised a smaller or changed scope
Only the first two examples contain a stated change that can be verified. The others need more investigation. The green result is evidence, not a root cause.
Use a rerun to test one hypothesis
Before rerunning, ask: what do we expect to be different?
If the answer is nothing, the rerun may still measure reproducibility, but label it that
way. If the answer is the shared environment recovered, check the recovery signal and
run a bounded scope. If the answer is the fixture was corrupt, preserve the original
fixture identity and record the replacement.
Change one important condition at a time where practical. Restarting the environment, changing data, updating dependencies, and rerunning the entire suite together can produce a green result without telling you which change mattered.
When retrying hides a quality problem
In one practitioner’s experience, developers or QA engineers reran failures directly, and some tests failed repeatedly without a clear notification path. This is a personal account, not a claim about Azure DevOps teams in general. It highlights four risks:
- The first failure becomes invisible. A final summary shows green while time and uncertainty from the first attempt disappear.
- Recurring instability has no owner. Each rerun is treated as a local interruption rather than another occurrence of the same reliability problem.
- The release gate changes silently. Only failed tests are repeated even though setup or shared state may have affected the wider suite.
- Consumed feedback time is understated. The team measures the successful attempt instead of the route from requested validation to usable evidence.
Preserve attempts separately and group them only for navigation. The summary should still
say failed, then passed.
Choose the next action
Retry now
Use a bounded retry when a specific transient condition is known, the repeated scope is valid, and both attempts will remain visible. This is common after a documented service recovery or runner interruption.
Investigate first
Stop automatic retries when the failure is new, affects a critical journey, changes its signature, or repeats without an understood condition. Open the exact logs, artifacts, test history, and surrounding run timeline.
Check Azure DevOps configuration
Investigate when the result is missing, the test task did not publish a full report, the pipeline reran a different stage, or the branch and candidate do not match. A data-quality problem in the evidence path cannot be repaired by repeating the product test.
Quarantine with an owner and expiry
Quarantine is a risk decision, not a label for unknown failures. State the affected test, covered risk, owner, repair date, continued observation path, and what evidence replaces it in the release gate. An indefinite ignored list is not controlled quarantine.
Keep retries observable in the test runner
Modern runners can distinguish first-run passes, retry passes, and persistent failures.
For example, Playwright documents passed, flaky, and failed result categories when
retries are enabled. See the official
Playwright retries guide.
Do not flatten these categories when publishing JUnit or other structured results. Keep attempt identity or enough annotations to reconstruct the sequence. If the reporting format cannot express the full history, retain runner artifacts and pipeline attempts beside the summarized test result.
Measure the real reliability signal
Useful review measures include:
- tests that passed only after a retry
- first-attempt failure rate
- repeated failure signatures over time
- feedback time including retry delay
- reruns without a recorded reason
- quarantined tests past their expiry
These measures support investigation. They do not prove that a test, environment, or product is at fault.

Pilot screenshot using synthetic qa-soft data, captured 29 October 2026 from
extension 0.0.121. It illustrates preserved execution evidence; it does not classify
the root cause of a rerun.
QaCockpit can organize accessible failure and execution evidence and link it back to Azure DevOps. It does not infer a root cause from one green rerun or decide which failure a team may ignore. The durable practice is simpler: preserve the first failure, state the hypothesis, control the rerun, and make the resulting action explicit.
