Traceability and Test Summary Reports

What a traceability matrix is

A traceability matrixTraceability matrixA table linking each requirement to the test cases that cover it.Open in glossary → (RTM, requirements traceability matrix) is a map of links. In its simplest form it’s two columns: requirement and test cases. Most teams add the latest run result and the bugs, because that’s what people actually ask about.

Here’s the FoodDash 2.0 matrix after the build 412 iPhone run:

Requirement	Test cases	Latest result	Bugs
FD-101 Sign in with email	C101, C102	1 passed, 1 failed	FD-241
FD-102 Sign in with Face ID	C103	Passed	–
FD-103 Add and remove dishes	C201, C203	Passed	FD-118 (closed)
FD-104 Coupon rules	C202	Failed	FD-242
FD-105 Tip the courier	none	–	–
FD-106 Pay with a saved card	C301	Blocked	–
FD-107 Choose a delivery slot	C302	Failed	FD-243
FD-108 Track the order	C401, C402	Passed	–
FD-109 Reorder a past order	none	–	–
FD-110 Edit profile details	C501	Passed	–

In ten seconds you can answer the product owner: no, the courier tip (FD-105) has no tests. Neither does reorder. And card payment has a test, but it was blocked, so it’s not verified either.

Forward and backward traceability

When a matrix supports both directions it’s called bidirectional. That’s the normal case in tools, where a link is visible from both ends.

Building an RTM, step by step

Using the matrix

Finding coverage gaps

Read the matrix forward and look for four kinds of gaps:

• No tests at all (FD-105, FD-109). Write cases, or state in the report that it’s untested.
• Tests exist but didn’t run or were blocked (FD-106). Covered on paper, not verified.
• Only failing tests (FD-104, FD-107). The requirement is not met yet.
• Thin coverage: one happy-path case for a complex rule. A link doesn’t tell you how well something is tested. FD-104 with a single case doesn’t check expired codes, minimum order value or what happens after removing an item.

Impact analysis when a requirement changes

Product decides that from 2.1, a coupon needs a minimum order of €15. The matrix answers what is directly affected? in seconds:

• Cases linked to FD-104: C202, which checks that the expired code SPRING5 is rejected, uses a €12 cart. After the change the code would be refused for the minimum order too, so the case no longer proves anything about expiry. Update its data in the repository and add a case for the new minimum before the next run.
• Bugs linked through those cases: FD-242 (“Expired coupon SPRING5 is still accepted”). Still a bug, but its steps use the same €12 cart. Update them so developers reproduce the expiry problem, not the new rule.

Then think beyond the links. FD-109 (reorder) copies an old cart together with its coupon. Old orders under €15 would now be refused, so reorder is affected too, even though nothing links it to FD-104. The matrix gives you a starting point; impact analysisImpact analysisWorking out which parts of a product a change could affect, to decide what to retest.Open in glossary → still needs your head.

Status updates during the cycle

Between “run created” and “summary report”, people want to know where you are. A short, regular update beats a long one nobody reads. Post it at the same time every day (or week) in the same format:

FoodDash 2.0 · build 412 · day 3 of 5
Progress: 84/120 executed (70%). 71 passed, 9 failed, 4 blocked.
New since yesterday: FD-243 (Major) delivery slots offered after closing time.
Blocked: 4 card-payment cases, PayCo sandbox down until 15:00.
Need: a decision on FD-241 ('Remember me' ignored) by Wednesday.
On track for Thursday's summary: yes, if the sandbox is back today.

Keep the detail one click away (a link to the run and to the bug filter). For non-technical readers, translate: “some users can’t sign in on their phone” rather than “case-sensitive email comparison”. How do you present testing status to stakeholders? goes deeper.

The test summary report

A test summary reportTest summary reportA report at the end of testing that summarizes what was tested, results, open bugs, risks and a recommendation.Open in glossary → (also called a test closure or test completion report) closes a test cycle. It helps the team decide whether to release, and it records what was known at that moment. For an agile team it’s one page. For a regulated project it can be a formal document, but the content is the same.

Section	What goes in	Watch out for
Scope	Release, build, environments and devices, dates	“Latest build” is not a build number.
Tested / not tested	Features and requirements covered; what wasn’t tested and why	The not-tested list is the most valuable part. Never leave it out.
Results	Executed, passed, failed, blocked, not run	Counts with meaning, not counts alone.
Open bugs	By severitySeverityHow much damage a bug does to the product or the user.Open in glossary →, with the user impact of the worst	A list of 40 bug titles is not a summary.
Risks	What could still go wrong, how likely, how bad	Include risks from untested areas.
Recommendation	Go, no-go, or go with conditions, and why	You recommend; the product owner decides.

Worked example: FoodDash 2.0

Notice what’s not in it: no “64% passed”, no “testing is complete”, no bug list copied from Jira. Every number is next to what it means.

A template you can copy

Test summary: <app> <version> (build <n>)
Scope: <environments / devices>, <dates>, <stories or features in scope>
Tested: <what was covered>, <executed / passed / failed / blocked>
Not tested: <what> – <why> – <plan or accepted risk>
Open bugs: <n Critical, n Major, n Minor>; worst: <key> – <user impact>
Risks: <what could still go wrong>
Recommendation: <go / no-go / go if ...> – decision by <who>
Links: <run>, <bug filter>, <traceability matrix>

Metrics that help, and metrics that mislead

Numbers belong in a report, but only with context. A useful rule: if a number could make a reader feel safer than they should, put the explanation right next to it.

Metric	Helps when…	Misleads when…
Pass rate	Shown per area, next to what failed	Used alone as “92% ready”
Requirement coverage	It points at untested stories	Read as “tested well”
Open bugs by severity	It comes with the user impact	Bugs are simply counted
Burndown of remaining cases	Used to spot blockers early	Testers rush to make it look good
Bugs found per tester	Almost never	Always: it rewards noise and punishes testers in stable areas

Teams that turn a number into a target get the number, not the quality. Read Which quality metrics do you find useful, and which can do more harm than good? for a deeper take, and entry and exit criteria for how numbers become agreed release rules.

Common mistakes