Test Runs and Test Management Tools

The test case repository

A test case repositoryTest case repositoryThe central library where a team keeps all its test cases, organized into sections and suites.Open in glossary → is where the team keeps the master copy of every test case. Whether it’s a tool or a spreadsheet, a healthy repository has three things: a clear structure, useful fields and a way to change cases safely.

Structure: sections and suites

Group cases the way people think about the product, usually by feature or screen:

• FoodDash 2.0 → Login · Cart · Checkout · Tracking · Profile
• inside Checkout → Payment · Delivery time · Address

Tools call these groups sections, folders, suites or test sets. A test suiteTest suiteA group of related test cases that are run together.Open in glossary → can also be a cross-cutting selection, like “Smoke” or “Accessibility”, built from cases in many sections. You’ll meet those in Regression and Smoke Suites.

The fields that make cases easy to pick

Beyond the fields you already write (steps, data, expected result), a few metadata fields let you build runs in seconds:

Field	Example values	Why it matters
Priority	P1, P2, P3	“Run all P1 first” is the most common release filter.
Type	Functional, UI, Accessibility, Security	Lets you build a focused run, e.g. an accessibility pass.
Automation status	Manual, Automated, To automate	Shows what CI already covers and what’s a candidate for automation.
Platform	iOS, Android, Web, All	Keeps iOS-only cases out of Android runs.
Requirement link	FD-104	Powers the traceability matrixTraceability matrixA table linking each requirement to the test cases that cover it.Open in glossary →.

Keep the values few and agreed. A priority field where half the cases are “High” helps nobody.

Versioning and reuse

The big win of a repository is reuse: the case “An expired coupon code is rejected” is written for 2.0 and runs again in 2.1, 2.2 and 3.0. That only works if cases stay up to date.

• When a requirement changes, update the case in the repository, not just in your head. Mention the change in the case history or in a comment (“Expected result updated for FD-104 v2: minimum order €15”).
• Many tools keep a version history per case, and a run remembers which version of the case it used. That way an old run still shows what was actually checked back then.
• Archive, don’t delete, cases for removed features. You may need them for the old app version that’s still in the stores.

Creating a test run

A test runTest runOne execution of a chosen set of test cases on a specific build and environment, with a result for each case.Open in glossary → answers: which cases, on which build, where, by whom?

Execution statuses, and when to use each

During test executionTest executionThe activity of running test cases, comparing actual with expected results and recording the outcome.Open in glossary →, every case in the run gets an execution statusExecution statusThe result recorded for a test case in a run: Passed, Failed, Blocked, Retest, Skipped or Untested.Open in glossary →. These are the common ones. TestRail’s defaults are close to this list, other tools use different names (Xray, for example, uses TODO and EXECUTING), and many teams add their own.

Status	Meaning	FoodDash example
Untested	Not run yet. Every case starts here.	The run was created this morning.
Passed	Actual result = expected result.	Home opens with “Hi, Mia” after sign-in.
Failed	Actual ≠ expected, because of a product defect.	The expired code SPRING5 is accepted: −€0.60 instead of “This code has expired”.
Blocked	You can’t run the case to the end, for a reason outside it.	The payment sandbox returns 503 errors until 15:00.
Retest	A linked bug is fixed; run the case again.	FD-118 (“Remove” deleted the wrong dish) is fixed in build 412.
Skipped / N/A	Deliberately not run, or doesn’t apply here.	A Face ID case in an Android run.

Blocked vs Failed: the classic confusion

Why it matters: a Failed case sends a developer looking for a bug in FoodDash. If the real cause was a sandbox outage, that’s an afternoon wasted, and your next failures are trusted a bit less. In the other direction, marking a real defect as Blocked hides it from the bug count.

Skipped, N/A and Untested

• Skipped / N/A is a decision: “doesn’t apply” or “we chose not to run it this time”. Say why in a comment. Better still, don’t put platform-specific cases into the wrong run in the first place.
• Untested at the end of a run is not a status you choose, it’s what’s left. That’s fine if you say so in the report (“12 P3 cases not run, low risk”). It’s not fine if it’s hidden.

Recording actual results and evidence

A status alone is not enough for anything other than a clean pass. Write the actual resultActual resultWhat really happened when you performed the test.Open in glossary → and attach proof.

Good evidence, depending on the problem:

• a screenshot of the wrong state (with the important part visible, not just the top of the screen);
• a short screen recording for anything that happens in steps or over time;
• logs: a HAR file from DevTools, device logs, a network trace from a proxy toolProxy toolA program that sits between the app and the server so you can watch and change network traffic.Open in glossary → (DevTools for Testers, ADB Basics);
• the build, device and OS (the run usually records these for you).

For passed cases, most teams don’t attach anything. Some regulated projects (finance, health) do want evidence for passes too; your test plan should say.

Linking a failure to a bug

One bug can fail several cases (a broken API can fail three checkout cases), and one case can have several bugs over its life. The links keep that history.

Retests and re-runs

When a developer fixes a bug, the fix lands in a new build. Two things happen:

• RetestingRetestingRunning the same test again to confirm that a reported bug is really fixed.Open in glossary →: the case that found the bug is run again, with the same steps and data, on the new build. Depending on the tool and setup, the case is set to Retest when its linked bug is resolved, or you do it yourself. If it passes, the bug can be closed. If it fails again, the bug is reopened.
• Regression testingRegression testingRe-running tests to check that new changes didn't break things that used to work.Open in glossary →: checking that the fix didn’t break anything else. That’s a different job with a different selection of cases. Don’t call it “retesting”.

For a new build, most teams create a new run containing the failed, blocked and untested cases from the previous run, plus a regression subset around the fixes. The old run stays as it was: it’s the record of build 412.

Tracking progress

A run is also a live dashboard. Three numbers matter most:

• Executed: how many cases have a final status. “64 of 120 executed (53%)”.
• Results: passed, failed, blocked. Watch failed and blocked P1 cases closely; they decide the release.
• Remaining: untested and retest. This is what a burndown tracks.

A burndown is a simple chart: remaining cases per day, going down towards zero by the deadline. Here’s the FoodDash 2.0 regression run:

Progress numbers are for steering, not for judging people. “Ana executed 40 cases, Ben 15” says nothing if Ben’s were the hard payment cases. More on numbers that help and numbers that mislead in Traceability and Test Summary Reports.

TestRail, Xray and Zephyr

Job ads name these three most often. They share the ideas from this lesson but organize them differently. This is a high-level picture; editions, names and screens change often, so check the vendor’s current documentation when you start using one.

	Where it lives	How tests and runs are stored
TestRail	A standalone web app (cloud or self-hosted) with a Jira integration.	Projects contain suites and sections of cases. You execute in test runs; a test plan groups several runs, e.g. one per device.
Xray	An app inside Jira.	Tests are Jira issues of type Test. A Test Execution issue holds the results for a build; a Test Plan issue groups executions for a release.
Zephyr	A family of Jira apps from SmartBear (you may see the names Zephyr Scale and Zephyr Squad).	Cases are organized in a test library or as Jira issues, depending on the edition. Executions are grouped into test cycles.

Other tools exist (qTest, PractiTest, Azure Test Plans and more). Once you understand repository → run → status → bug link, switching tools takes a day, not a month. In interviews, say what you did with a tool (“I built runs per device and linked failures to Jira”), not just its name.

Spreadsheets are a legitimate option

A small team with one app and one tester doesn’t need a paid tool. A shared spreadsheet works well if you keep it disciplined:

• One tab for the repository: ID, Section, Title, Priority, Type, Platform, Steps, Expected result, Requirement.
• One tab per run, named after the build and device: ID, Title, Status, Actual result, Evidence link, Bug key, Tester, Date.
• Use a dropdown (data validation) for Status so nobody types “pass”, “OK” and ”✓” in the same column.
• Add a small summary at the top with COUNTIF per status.

You’ll outgrow it when you have several testers editing at once, many devices per release, or need history and reports across releases. That’s when a tool pays for itself.

Good practices

• One run per build and environment. Results from different builds don’t mix.
• Don’t edit cases mid-run without noting it. If you discover the expected result is outdated, add a comment on the result, finish the run with what you know, and update the case in the repository afterwards (or tell the lead right away if it changes the verdict). Silent edits make old results meaningless.
• Keep cases atomic. One case checks one thing, so one status means one thing. A case with five checks that “half passed” can’t be marked honestly.
• Never pass a case you didn’t run. If you ran out of time, it’s Untested. That’s honest; a fake pass is the worst thing a tester can do.
• Report unrelated bugs separately. If you notice a stretched logo in the Orders list while running a tracking case, file it (and suggest a new case), but don’t fail the tracking case.
• Update the run as you go, not at the end of the day. The lead is looking at it.

Common mistakes