A rerun is not automatically bad. It can verify a hypothesis, recover from a known environment incident, or show whether a failure repeats under controlled conditions.

The problem begins when the final green result replaces the history. Passed on retry and passed first time are different quality facts.

First identify what is being rerun

Teams use “rerun” for several operations:

  • rerun the failed test inside the same test task
  • rerun a pipeline job or stage
  • create a new run of the whole pipeline
  • manually repeat a scenario outside the original run

These have different inputs, environments, and evidence. Record the operation precisely. A full pipeline rerun may use a newer dependency or mutable environment even when it uses the same commit. A test-task retry may run in a fresh worker while keeping the same job.

Azure DevOps can display summarized test results for rerun attempts, including multiple attempts under the logical test. Microsoft’s test-results documentation also describes result history, errors, stack traces, attachments, and the surrounding run timeline. Start there before inventing another record.

The evidence that must survive

For every operationally meaningful rerun, retain:

EvidenceWhy it matters
Original run and attemptEstablishes what first failed
Test identity and exact errorPrevents unrelated failures being grouped as one
Candidate, branch, and commitShows whether the subject stayed comparable
Environment and data stateExposes changed conditions
Rerun actor and timeCreates ownership and sequence
Stated reason or hypothesisTurns a click into an experiment
Rerun scopeShows whether one test or the whole gate was repeated
Result and diagnostic artifactsSupports the next conclusion

A team does not need a long form before every click. A short reason such as rerun after identity provider recovery is already more useful than an unexplained green attempt.

Native Azure DevOps test results for a retained synthetic run, showing the exact run, failed-result history, pass percentage, duration, and failing tests.

Native Azure DevOps capture from synthetic PlaneFlow run 345 on 29 October 2026. The screen preserves the run and failure evidence; it does not explain why a later attempt might pass.

A green rerun has several possible meanings

The second attempt passed. That could mean:

  • a confirmed external incident ended
  • the test data was repaired
  • a product race did not reproduce
  • an unstable test took a different timing path
  • a dependency returned a different response
  • the rerun exercised a smaller or changed scope

Only the first two examples contain a stated change that can be verified. The others need more investigation. The green result is evidence, not a root cause.

Decision tree for preserving the first failure and choosing an action after a controlled rerun.

Use a rerun to test one hypothesis

Before rerunning, ask: what do we expect to be different?

If the answer is nothing, the rerun may still measure reproducibility, but label it that way. If the answer is the shared environment recovered, check the recovery signal and run a bounded scope. If the answer is the fixture was corrupt, preserve the original fixture identity and record the replacement.

Change one important condition at a time where practical. Restarting the environment, changing data, updating dependencies, and rerunning the entire suite together can produce a green result without telling you which change mattered.

When retrying hides a quality problem

In one practitioner’s experience, developers or QA engineers reran failures directly, and some tests failed repeatedly without a clear notification path. This is a personal account, not a claim about Azure DevOps teams in general. It highlights four risks:

  1. The first failure becomes invisible. A final summary shows green while time and uncertainty from the first attempt disappear.
  2. Recurring instability has no owner. Each rerun is treated as a local interruption rather than another occurrence of the same reliability problem.
  3. The release gate changes silently. Only failed tests are repeated even though setup or shared state may have affected the wider suite.
  4. Consumed feedback time is understated. The team measures the successful attempt instead of the route from requested validation to usable evidence.

Preserve attempts separately and group them only for navigation. The summary should still say failed, then passed.

Choose the next action

Retry now

Use a bounded retry when a specific transient condition is known, the repeated scope is valid, and both attempts will remain visible. This is common after a documented service recovery or runner interruption.

Investigate first

Stop automatic retries when the failure is new, affects a critical journey, changes its signature, or repeats without an understood condition. Open the exact logs, artifacts, test history, and surrounding run timeline.

Check Azure DevOps configuration

Investigate when the result is missing, the test task did not publish a full report, the pipeline reran a different stage, or the branch and candidate do not match. A data-quality problem in the evidence path cannot be repaired by repeating the product test.

Quarantine with an owner and expiry

Quarantine is a risk decision, not a label for unknown failures. State the affected test, covered risk, owner, repair date, continued observation path, and what evidence replaces it in the release gate. An indefinite ignored list is not controlled quarantine.

Keep retries observable in the test runner

Modern runners can distinguish first-run passes, retry passes, and persistent failures. For example, Playwright documents passed, flaky, and failed result categories when retries are enabled. See the official Playwright retries guide.

Do not flatten these categories when publishing JUnit or other structured results. Keep attempt identity or enough annotations to reconstruct the sequence. If the reporting format cannot express the full history, retain runner artifacts and pipeline attempts beside the summarized test result.

Measure the real reliability signal

Useful review measures include:

  • tests that passed only after a retry
  • first-attempt failure rate
  • repeated failure signatures over time
  • feedback time including retry delay
  • reruns without a recorded reason
  • quarantined tests past their expiry

These measures support investigation. They do not prove that a test, environment, or product is at fault.

QaCockpit Test Executions showing complete, failing, and missing evidence separately, with repeated failed runs and one exact failed execution kept visible for investigation.

Pilot screenshot using synthetic qa-soft data, captured 29 October 2026 from extension 0.0.121. It illustrates preserved execution evidence; it does not classify the root cause of a rerun.

QaCockpit can organize accessible failure and execution evidence and link it back to Azure DevOps. It does not infer a root cause from one green rerun or decide which failure a team may ignore. The durable practice is simpler: preserve the first failure, state the hypothesis, control the rerun, and make the resulting action explicit.