A test report answers what happened to the tests that appeared in that report. It does not automatically prove that every test the team expected was still present.

A changed tag, shard, path, filter, or test-discovery configuration can silently reduce the executed scope. The remaining tests may all pass. Without an explicit baseline, the result looks healthy because the missing tests are no longer available to fail.

This is test scope drift.

Counts are not enough to identify the change

Comparing only the number of reported tests creates ambiguous signals. A report with 500 tests followed by another report with 500 tests may still contain a different set:

  • ten tests were removed and ten new tests were added;
  • a suite was renamed;
  • parametrized cases changed;
  • one shard disappeared while another produced duplicates;
  • the workflow selected a different branch or configuration.

QaCockpit builds the comparison from exact test identities within a Test Source and its run configuration. JUnit testsuite.name, testcase name, class, file, parameters, and an optional stable ID provide the technical evidence needed to explain the difference.

The first complete report establishes a versioned baseline. Later reports are compared with that active revision rather than with a manually maintained expected count.

Keep missing tests separate from newly added tests

An added test is usually useful scope expansion. A removed test may be intentional, but it can also be the first sign of a configuration problem.

The comparison should therefore preserve separate sets:

  • Removed: present in the active baseline and absent from the current report;
  • Added: absent from the baseline and present in the current report;
  • Reported: exact identities included in this execution;
  • Missing: expected identities that did not report.

Automatically lowering the baseline after every smaller report would erase the evidence. The next incomplete run would appear complete relative to the already reduced scope.

Use repeated evidence as a review signal

Not every one-run difference deserves a baseline change. A temporary shard failure, a branch-specific selection, or an interrupted reporter may explain a single deviation.

QaCockpit keeps the active baseline and records the candidate set. When the same changed set appears in three consecutive reports, the product suggests a review. A permitted user can still inspect the difference earlier.

The available decisions remain explicit:

  • keep the active baseline;
  • accept the candidate as a new revision;
  • restore a historical revision later.

Each decision remains part of the audit history. Accepting a new revision does not turn off future drift detection.

Show scope completeness beside test outcomes

Scope is not a hidden administration setting. It affects whether the current report can support a release or regression decision.

QaCockpit Campaign Run Report showing complete source collection but only two of three imported reports with complete expected scope.

Collection completeness and Test Scope completeness answer different questions. Both remain visible in the report.

A campaign may receive reports from every required Test Source while one source still omits expected tests. The campaign is collected, but the evidence is partial. That is more precise than marking the whole workflow red or pretending the observed pass rate is complete.

Common causes worth checking

When a scope difference appears, useful checks include:

  1. Did the repository branch, workflow, or pipeline change?
  2. Did a test selector, marker, tag, or project configuration change?
  3. Did every matrix job or shard retain its JUnit output?
  4. Were reports overwritten because several jobs used the same path?
  5. Did a framework rename suites or parametrized cases?
  6. Was a test intentionally removed, replaced, or moved to another Test Source?

The scope comparison provides the identities. The team still owns the product decision about whether the change is expected.

When a baseline is unnecessary

A very small suite with deliberate manual review of every run may not need a formal scope baseline. The same is true for exploratory or branch-specific runs whose membership is expected to change continuously.

Baseline comparison is most useful for stable regression, smoke, contract, security, and release suites where disappearance should be visible rather than silently accepted.

For a practical setup and deterministic public runs, use the Test Scope guide. For the wider evidence model, read A green pipeline is not enough. If missing tests are currently discovered through manual count comparisons, tell us how your reports are produced.