Why Azure DevOps cannot natively calculate acceptance-criteria coverage

A User Story contains four acceptance criteria. The team has five Test Cases, a nightly automation pipeline, and a green result. What is the acceptance-criteria coverage?

Azure DevOps can report traceability and test quality at the requirement level. It cannot derive reliable coverage for the individual acceptance criteria from those facts alone.

The obstacle is not a missing chart. Acceptance criteria are usually written as bullets inside one rich-text field. Azure DevOps gives the work item a stable ID, but it does not give each criterion its own native ID, relation, lifecycle, or test link. Test Plans can connect the requirement work item to Test Cases. Automated results can connect to Test Cases or requirements. Neither connection proves which individual criterion a test covers.

The team must first define stable criteria and the meaning of “coverage.” Only then can a percentage have a defensible numerator and denominator.

Do not combine these measures into one readiness score. Each numerator and denominator answers a different question.

One work item is structured; its acceptance criteria are not

Azure Boards supports several default processes:

• Basic uses Issues
• Agile uses User Stories
• Scrum uses Product Backlog Items
• CMMI uses Requirements
• inherited processes can introduce custom work item types and fields

The product behavior is often described in an Acceptance Criteria field on the selected item. That field is rich text. It can contain paragraphs, bullets, tables, copied screenshots, Given/When/Then scenarios, or an informal note.

Azure DevOps stores and revisions the field as part of the work item. It does not turn the individual bullets into addressable objects. Microsoft also states that the Analytics Service does not support reporting on plain-text and HTML fields. See the official work item field reference.

As a result, Azure DevOps has no native object corresponding to AC-03 unless the team creates one through a convention or a separate work item.

Why splitting the HTML is not coverage

A parser could split the field at bullets or numbered lines. That may find candidate criteria, but it does not establish stable identity.

Consider this original version:

- A duplicate payment request is rejected.
- The shopper sees the existing payment result.
- Support can find the original transaction.

During development, the customer clarifies the behavior:

- A request with the same idempotency key returns the original successful result.
- A request reusing the key with different payment details is rejected.
- The shopper sees the existing payment result.
- Support can search by the idempotency key.

Did the first criterion become two? Was the second criterion merely reworded? Is the support criterion the same behavior after its wording changed? A bullet position cannot answer those questions.

Counting lines would make the denominator change from three to four without recording which test mapping survived. AI can suggest likely continuity, but suggestion is not a confirmed traceability edge.

What Test Plans adds — and what it still does not add

Azure Test Plans provides native requirement-based suites. Microsoft documents that a requirement-based suite links a backlog item to Test Cases and is the native path for requirement tracking in planned testing. The supported requirement item differs by process: Issue, User Story, Product Backlog Item, or Requirement.

That link is valuable:

Requirement work item → requirement-based suite → Test Cases

It establishes that the requirement has planned tests. It does not create this deeper mapping automatically:

Requirement → AC-01 → Test Case 842
Requirement → AC-02 → Test Cases 843 and 844

A Test Case can also be associated with an automated test method. The Azure DevOps web portal supports association from a published pipeline result, including Python and Java tests. Microsoft notes two important constraints: one Test Case cannot have more than one associated test method, and manual Test Case parameters do not drive the automated test. See requirement-based suites and Associated Automation.

These features help teams that actively maintain Test Plans. They do not solve teams where manual testing happens outside Test Plans, automation is created later, or tests run directly in CI without Test Case work items.

A verified example: the requirement link exists, the AC mapping is a convention

We verified this behavior in a synthetic NovaCart Commerce project. Azure Boards Issue #1 contains four labeled acceptance criteria in its Description field. Azure preserves the work item and its revisions, but AC-01 through AC-04 are still text inside that single field rather than four addressable Azure objects.

Stable AC labels make the text reviewable, but the labels are a team convention rather than native Azure entities.

We then created a requirement-based Test Suite and three manual Test Cases:

• [1:AC-01] Return the original payment result
• [1:AC-02] Reject conflicting key reuse
• [1:AC-03] Restore checkout result without resubmission

AC-04 intentionally has no designed test. The suite therefore demonstrates 3/4 test-design coverage, but Azure derives neither that fraction nor the individual mapping from the rich-text field. The mapping exists because the stable AC key is part of each Test Case title.

The native suite links the Test Cases to Issue 1. The AC key in each title supplies the criterion-level relation.

Opening Test Case #6 shows another important boundary. The project already has published automated results, but its Associated Automation section is empty. Azure does not decide that a similarly named or nearby automated result implements AC-02; a user or accepted deterministic process must establish that association.

Published automation is not the same as mapped automation.

Azure DevOps does have requirement-level traceability

The boundary is important. Azure DevOps can link a requirement work item to automated test results. Its Requirements quality widget can use a requirement query and a selected pipeline stage to show requirement-level pass rate, failed-test count, and requirements without associated tests. Microsoft documents that flow in Requirements traceability.

This is native and useful:

Requirement #1842 → linked tests → selected pipeline results

This deeper chain is not created automatically:

Requirement #1842 → AC-02 → mapped tests → exact execution → result

The article is therefore not claiming that Azure DevOps lacks requirement coverage. The specific gap is stable, criterion-level identity and mapping inside the rich-text Acceptance Criteria field.

The distinction is visible in an actual pipeline run. Azure run #75 contains 25 published results: 22 passed, 2 failed, and 1 other/not executed. Those rows are valid execution evidence, but their names do not contain the work item or AC key, and no Test Case association was recorded.

The execution denominator is known. The criterion-level mapping denominator is still unknown until the team records it.

“Coverage” is at least four separate questions

Treating coverage as one number hides the exact gap. A useful model separates four levels. In the formulas below, an eligible acceptance criterion is an active, accepted criterion from the selected work-item revision that requires verification in the scope under review. Retired, replaced, or explicitly out-of-scope criteria must be recorded separately rather than silently removed from the denominator.

1. Test design coverage

Question: Does each eligible acceptance criterion have at least one identified test?

The test can be manual or automated. At this level, it is a test definition, not an execution result.

AC test design coverage = eligible AC with at least one mapped test / eligible AC

2. Automation coverage

Question: Which eligible criteria have at least one mapped automated test?

This is not the same as test design coverage. A criterion can have a valid manual test and no automated implementation. That is an explicit state, not missing data.

AC automation coverage = eligible AC with mapped automation / eligible AC

For detailed engineering analysis, teams may also calculate automation at Test Case or logical-test level. The denominator must be named because those percentages answer a different question.

3. Execution coverage

Question: Did the required mapped tests run for the exact release candidate and environment under review?

An automated test can exist and still be absent from the selected run. It might be filtered out, skipped, removed from the suite, or executed against another deployed version.

execution coverage = required mapped tests executed in scope / required mapped tests

4. Outcome

Question: What happened to the tests that produced conclusive results?

Pass rate belongs here. It must not replace execution coverage.

pass rate = passed conclusive results / conclusive executed results

A 100% pass rate can coexist with 50% execution coverage. Both values are correct. The release conclusion changes only when the reader can see both.

A numerical example without a magic score

Suppose work item #1842 has four eligible acceptance criteria after review:

Criterion	Test designed	Automation mapped	Required automation executed for release	Result
AC-01 Accept a valid idempotency retry	Yes	Yes	Yes	Passed
AC-02 Reject key reuse with different details	Yes	Yes	No	No evidence
AC-03 Show the original payment result	Yes	No	Not applicable	Manual test not executed for this release
AC-04 Search support records by key	No	No	Not applicable	No test design

The separate facts are:

• test design coverage: 3/4 = 75%
• AC automation coverage: 2/4 = 50%
• execution coverage for the two required automated mappings: 1/2 = 50%
• automated pass rate among conclusive executed results: 1/1 = 100%

Reporting only 100% passed would hide an unexecuted automated check, an unexecuted manual check, and an acceptance criterion with no designed test. Combining the four facts into one weighted score would hide which action the team should take.

This is the acceptance-criteria equivalent of test scope drift: the denominator can change even when every reported test is green. Stable logical test identity also matters, as described in building a test catalog from CI results.

Recommended team convention: stable AC IDs

The following is QaCockpit guidance, not native Azure DevOps behavior.

Keep acceptance criteria in the existing work item, but give each durable criterion a small stable identifier:

AC-01 — Given a completed payment, when the same idempotency key is retried with the
same details, then the original successful result is returned.

AC-02 — Given an existing idempotency key, when it is reused with different payment
details, then the request is rejected and no second charge is created.

AC-03 — Given a recovered payment result, when the shopper returns to checkout, then the
existing result is shown without resubmitting the payment.

Use these rules:

1. The identifier remains stable when wording becomes clearer but the behavior stays the same
2. A materially new behavior receives a new identifier
3. Removed criteria are marked retired instead of renumbering the remaining list
4. The work item history explains a split, merge, or replacement
5. Test mappings use {workItemId}:AC-{number}, for example 1842:AC-02

This convention does not require one child work item per criterion. Teams that need formal approval, separate ownership, or an auditable criterion lifecycle can choose structured child items later. Most teams can start lighter.

Two supported working styles

The convention should fit teams that use Test Plans and teams that do not.

Test Plans route

For a team already maintaining Azure Test Plans:

1. Create a requirement-based suite for the work item
2. Name or tag Test Cases with the relevant key, such as [1842:AC-02]
3. Associate automation only where the Test Case and automated method represent the same behavior
4. Keep manual-only Test Cases explicit instead of marking them missing
5. Select the exact Test Run when reviewing execution coverage

The Test Case remains the primary test definition. The AC key adds the relation Azure does not create by itself.

Automation-first route

For a team where test code is the maintained source and Test Plans would add duplicate administration, keep a small reviewed mapping with the code:

requirement: 1842
criteria:
  AC-01:
    manual: [checkout-charter-2026-09-duplicate-payment-retry]
    automated: [checkout.api.returns-original-result]
  AC-02:
    manual: [TC-842]
    automated: [checkout.api.rejects-conflicting-retry]
  AC-03:
    manual: [TC-843]
    automated: []

The identifiers should refer to stable Test Cases, checklists, exploratory charters, or logical automated tests, not a line number or a display name that changes with every parameter. An exploratory session counts as a durable mapping only when its charter has a stable identity and retained outcome. The mapping can evolve in the same pull request as the test code.

Azure DevOps does not interpret this file natively, and QaCockpit does not read it today. It is a proposed portable convention for reducing manual linking while preserving a reviewable source of truth.

Handle criteria changes during development

Changing acceptance criteria is not automatically a process failure. Discovery can show that the first solution is unsafe, too expensive, or inconsistent with the product.

The dangerous part is silently changing the text while keeping old tests and old coverage claims.

When a criterion changes:

1. Decide whether it is clarification, replacement, split, merge, or removal
2. Preserve or replace its stable ID accordingly
3. Review existing manual and automated mappings
4. Mark mappings that need redesign instead of carrying them forward automatically
5. Recalculate each coverage layer against the accepted revision
6. Record which revision the release review used

This turns change into visible evidence instead of hidden denominator drift.

What not to infer

Do not infer that:

• four bullets in HTML are four stable criteria
• a Test Case linked to the requirement covers every acceptance criterion
• Automation Status = Automated means the test ran for this release
• a published automated result proves a current mapping to the work item
• a green test run means every eligible criterion had a test
• missing Test Plans data means the team performed no testing
• title similarity is a confirmed traceability relation

Use unknown, not mapped, manual only, not executed, and inaccessible as separate states. None of them should silently become zero or passed.

How QaCockpit uses this evidence

QaCockpit for Azure DevOps currently organizes accessible pipeline runs, test results, logical test history, missing reports, and selected campaign evidence. It does not currently read acceptance-criteria text, Test Plans requirement suites, or Test Case automation associations. It therefore does not claim acceptance-criteria coverage.

In the same NovaCart project, QaCockpit reports the current project evidence as two monitored pipelines, two pipelines with tests, a 95.1% project pass rate, and two failed tests. Those values describe observed pipeline evidence. They deliberately do not become AC coverage because the required AC identities and mappings are outside the current evidence model.

A useful quality picture can remain honest about what it does not measure.

Future requirements and acceptance-intelligence capabilities may discover process types, native requirement links, Test Cases, and accepted mappings. This is a product direction, not a currently shipped feature or a commitment to a final feature name. Any future percentage must show its denominator, provenance, access gaps, and drill-down. Text parsing can suggest candidate criteria, but only stable IDs or explicit mappings can confirm AC-level coverage.

A setup checklist for one team

1. Choose which work item types represent eligible requirements
2. Add stable AC-01 identifiers without changing the team’s process template
3. Define what clarification, replacement, split, and retirement mean
4. Select the Test Plans or automation-first mapping route
5. Map at least one small feature before attempting organization-wide reporting
6. Report test design, automation, execution, and pass rate separately
7. Tie execution to the exact release candidate and environment
8. Show missing, inaccessible, and changed evidence explicitly
9. Review the mapping when criteria or tests change
10. Publish a percentage only when the denominator can be listed

Acceptance-criteria coverage becomes useful only after the team can explain every row behind the number.

Frequently asked questions

Can Azure DevOps calculate acceptance-criteria coverage?

Not from ordinary acceptance-criteria text alone. Azure DevOps can report at the requirement level, but each criterion needs a stable identity and an explicit test mapping before criterion-level coverage can be calculated reliably.

Does the Requirements quality widget provide AC-level coverage?

No. It reports requirement-level test quality for the configured requirement query and pipeline stage. It does not split the Acceptance Criteria field into individually addressable criteria.

Do Azure Test Plans map tests to individual acceptance criteria?

Requirement-based suites link Test Cases to the requirement work item. Mapping a Test Case to AC-01 or AC-02 still needs a team convention or another structured object.

Can AI infer acceptance-criteria coverage?

AI can propose likely criteria and test matches. Treat those suggestions as candidates until a person or an accepted deterministic rule confirms the stable identities and mappings.

Documentation and the Azure DevOps example checked on October 1, 2026.