A green CI workflow feels definitive. The jobs completed and the checks passed, so the release must be safe. But the color of a pipeline only describes what the pipeline observed. It does not prove that the intended regression scope actually ran.
That distinction becomes important as automation grows.
Passing observed tests versus completing expected scope
Imagine a workflow expected to report 800 tests. A configuration error causes one suite to be excluded, and the remaining 620 tests pass. Most CI interfaces will show a successful job. The tests that ran are green, but the regression signal is incomplete.
A useful TestOps view therefore needs two sets of information:
- Expected scope: what the profile or campaign intended to execute.
- Observed results: what the test framework actually reported.
The difference between them is not noise. “Results missing” is a status worth investigating in its own right.
Context turns a result into evidence
Counts alone are not enough either. Every execution should retain the context that makes it reproducible and explainable:
- source and execution profile;
- branch and commit SHA;
- target environment;
- provider run identifier and attempt;
- start time and duration;
- result files, logs, and protected artifacts.
When a rerun happens, it should create a new attempt rather than overwrite the first one. The initial failure and the later pass tell a different story from a single green result.
Skipped, errored, and unknown are real states
Flattening every non-failure into “passed” destroys information. A skipped test may be intentional, but a sudden increase in skipped tests can reveal a setup issue. An error often means the test could not make an assertion at all. An unknown result may point to a dialect the importer does not yet understand.
Those states should remain visible and traceable.
Build confidence from complete evidence
The objective is not to make dashboards look green. It is to give a team enough reliable evidence to decide what to do next. That means comparing expectation with observation, preserving context, and making incomplete execution explicit.
QaCockpit treats that complete signal as a first-class product concept rather than an optional reporting detail.