Regression and Smoke Suites

The smoke suite: is this build testable?

Smoke testingSmoke testingA quick check of the most important features to decide if a build is stable enough to test further.Open in glossary → gets its name from hardware: plug in a new circuit board, and if smoke comes out, there’s no point testing further. In software, a smoke suite is a short list of checks covering the critical user journeys of the app, run on every new build before anything else.

A good smoke suite is:

• Small: around 10-20 checks. If it grows to 60, it’s a mini regression suite.
• Fast: 10-30 minutes manually, a few minutes automated.
• Wide and shallow: it touches every critical journey once, on the happy path. No negative cases, no edge cases.
• Stable: the same checks every time, so a failure means something.
• Decisive: it ends with a clear yes or no. “Build 409 smoke: passed, testing can start” or “Build 409 rejected: sign-in fails with error 500.”

Smoke checks are also a common entry criterionEntry and exit criteriaConditions that must be met before a test activity can start (entry) or before it counts as done (exit).Open in glossary →: “testing of a build starts only after the smoke suite passes.” That one rule saves teams huge amounts of wasted effort.

The regression suite: did we break what used to work?

A regression is a feature that worked before and stops working after a change. It happens all the time, because code is connected in ways nobody fully sees: a developer changes how prices are rounded for a new currency, and suddenly promo codes give the wrong discount.

Regression testingRegression testingRe-running tests to check that new changes didn't break things that used to work.Open in glossary → means re-running existing tests after a change to catch these side effects. The regression suite is the collection of those tests, usually organized by feature area: Sign in, Address, Search, Cart, Checkout, Tracking, Profile and so on.

Where the tests come from

Don’t build a regression suite by writing a test for every button. Build it from four sources, in this order:

Prioritizing: P1, P2, P3

A regression suite grows over time. Without tiers, you end up with “run everything” (impossible) or “run whatever we feel like” (risky). Most teams use three tiers:

Tier	What belongs here	QuickBite examples
P1: critical	Critical journeys, money, data, escaped serious bugs	Pay with saved card · Declined card creates no order · Order appears in tracking
P2: important	Main features and their main negative cases	Promo codes · Edit address · Search filters · Push notifications
P3: nice to have	Rare paths, cosmetic and low-impact checks	Dark mode on Settings · Rate the courier · About screen

Building a suite, step by step

Worked example: a smoke suite for QuickBite 3.8.0

Here’s a complete smoke suite for a food delivery app release. Nine checks, about 16 minutes, run on one Android and one iOS device on staging.

#	Check	Passes when	Min
S1	Cold start	App opens to Home, no crash	1
S2	Sign in	Email + password lands on Home, signed in	2
S3	Choose saved address	Address shows in the header	2
S4	Restaurant list	Restaurants near the address appear	1
S5	Search	“pizza” returns pizza places	2
S6	Menu	A restaurant opens with items and prices	1
S7	Add to cart	Cart badge shows 1, cart shows the item	2
S8	Pay	Order placed with saved test cardTest cardA card number published by a payment provider to simulate payments in test mode.Open in glossary →	3
S9	Tracking	New order shows status “Preparing”	2

The result goes in the team chat in one line:

Smoke 3.8.0 (409), Pixel 8 / Android 15 + iPhone 15 / iOS 18, staging: 9/9 passed + Google Pay OK. Ready for testing.

or, on a bad day:

Smoke 3.8.0 (409): FAILED at S8, Pay returns "Payment unavailable" on both devices (SHOP-161). Build rejected, regression on hold.

Choosing a regression subset: impact analysis

Before each release, you’ll rarely run the full suite. Instead you choose a subset with impact analysis: working out which areas a change could affect. Ask:

Change	Test fully	Quick pass
Payment provider SDK updated	Checkout and payment (all methods, declined, 3-D Secure)	Promo codes, order creation
Translation strings changed	Languages, text truncationTruncationText that gets cut off because it doesn't fit its space.Open in glossary → on every screen	Checkout buttons, push texts
New address autocompleteAutocompleteSuggestions shown under a field while typing, which the user can pick to finish the input.Open in glossary →	Address screen	Restaurant list, order address, courier map

Manual or automated regression?

Regression suites are the classic candidate for automation: the same checks, over and over, on every release. But not everything is worth automating.

Worth automating	Better kept manual
Stable, repeated checks (sign in, add to cart, pay)	Features that change every sprint
Many data combinations (prices, currencies, tax)	Look and feel, layout, animations
API-level checks of the backend	One-off checks for a rare release
Smoke suite on every build	Exploratory testing of new features
Checks that are slow or boring for humans	Things that need real devices, cameras, real payments

How often to run each

What	When	Typical owner
Smoke suite	Every new build that reaches QA	Automated in CI, or the tester on duty
P1 regression	Nightly (automated) and on every release candidateRelease candidateA build believed ready to ship, pending final testing.Open in glossary →	Automation + QA
Change-based subset	When a feature or fix lands	The tester of that area
Full regression (P1-P3)	Before major releases, after big refactors	Whole QA team
Sanity check	After a hotfixHotfixAn urgent fix released outside the normal release schedule.Open in glossary → or small fix	Tester who verifies the fix

Keeping suites healthy

Suites rot. Features change, screens get redesigned, tests get copied. A rotten suite wastes time and, worse, gives false confidence. Keep it healthy with a few habits:

• Remove obsolete tests. If the “Pay by invoice” option was removed, delete its tests (or archive them). Running tests for features that don’t exist teaches people to ignore failures.
• Update tests right after a change. When a story changes a flow, updating the affected regression tests is part of the story’s work, not “later”.
• Avoid duplication. If three people wrote “sign in with email” three times, merge them into one. Duplicates cost time on every run and drift apart.
• Handle flaky tests. A flaky testFlaky testA test that sometimes passes and sometimes fails without any change to the code.Open in glossary → passes and fails without any code change (timing, test data, network). Never just re-run until green. Investigate, fix the cause, and if that takes time, move it out of the suite into a “quarantine” list so it doesn’t hide real failures.
• Review the suite every few releases. Ask: which tests never fail and cost a lot? Which areas had escaped bugs but few tests? Rebalance.

The bad one is vague, enormous and can’t be finished in a smoke run. It’s at least ten regression tests hiding in one line.

Tracking suites in a test management tool

Small teams can live with a spreadsheet. As suites grow, most teams move to a test management tool, which stores test cases, groups them into suites, and records each test run (which tests ran on which build, by whom, with what result).

• TestRail is a standalone tool that integrates with JiraJiraA popular tool for tracking bugs, user stories and tasks.Open in glossary →.
• Xray and Zephyr are apps that live inside Jira, so test cases and runs can be linked directly to stories and bugs (in Xray, each test is itself a Jira issue).

Whatever the tool, the ideas are the same: tag tests with tier and area, create a run for each build or release, mark each test Passed / Failed / Blocked, link failures to bug tickets, and share the run report. That report is the backbone of a test summary reportTest summary reportA report at the end of testing that summarizes what was tested, results, open bugs, risks and a recommendation.Open in glossary → and lets you show test coverageTest coverageHow much of the product, requirements or risks your tests actually check.Open in glossary →: which requirements were tested and which weren’t. See Working with Jira for the Jira side.

Common mistakes

A template you can copy

Suite: QuickBite Smoke           Version/build: 3.8.0 (409)
Environment: Staging             Devices: Pixel 8 / Android 15, iPhone 15 / iOS 18
Tester: ______                   Date: ______

| ID | Area     | Tier | Check                         | Passes when                    | Min | Result |
|----|----------|------|-------------------------------|--------------------------------|-----|--------|
| S1 | Launch   | P1   | Cold start                    | Home opens, no crash           | 1   |        |
| S2 | Sign in  | P1   | Sign in with email + password | Home shows "Hi, <name>"        | 2   |        |
| …  |          |      |                               |                                |     |        |

Verdict: PASS / FAIL (build rejected)     Bugs: ______
Extra checks for this build (risky changes): ______