Test Scope and missing results

01

Separate pass rate from completeness

A tag, path, shard, browser project, module, or discovery change can remove tests while every remaining assertion passes. QaCockpit therefore keeps several signals separate:

02

Keep test identity stable

The first complete report can establish a versioned baseline. Later reports are compared using the suite, test name, class or file, parameters, and optional stable identifiers available in the imported JUnit evidence.

• Use deterministic test and parameter names instead of timestamps or random values.
• Keep framework project or browser context explicit when it changes test meaning.
• Avoid renaming suites merely to change display formatting.
• Treat an intentional removal as a reviewed baseline change, not an automatic count reduction.

03

Retain every shard and module

Scope comparison is only reliable when the CI/CD run preserves all report files. Use unique artifact names for parallel GitHub jobs, keep every Maven module directory, and avoid a shared output path that lets the last shard overwrite earlier XML.

Review the provider steps for GitHub Actions andAzure DevOps.

04

Review repeated scope changes explicitly

One difference may come from a temporary shard failure or a branch-specific filter. Repeated matching evidence can become a candidate baseline, but a permitted user still decides whether to keep the active revision or accept the new scope.

1. Inspect added and removed exact identities.
2. Confirm the branch, workflow inputs, modules, projects, and shards.
3. Decide whether the change is expected product evolution or missing evidence.
4. Keep or accept the baseline with an auditable decision.

05

Inspect deterministic scope-drift runs

Both public demo repositories expose a scope-drift scenario. The workflow completes with valid JUnit XML while deliberately omitting one normally reported test, which demonstrates why result success and evidence completeness are different questions.