Pairwise Testing

The combinatorial explosion, in numbers

When parameters are independent, the number of full combinations is the product of their value counts. That’s the combinatorial explosionCombinatorial explosionThe rapid growth in the number of combinations as conditions are added.Open in glossary →:

Parameters	Values	All combinations
Browser, OS	4 × 4	16
+ Language	× 3	48
+ Payment	× 3	144
+ Account (guest / signed in)	× 2	288
+ Currency (3)	× 3	864

Meanwhile, the number of pairs grows gently, because you only add the pairs of the new parameter with each existing one. And a pairwise set can’t be smaller than the two largest parameters multiplied (every Browser + OS pair needs its own row), so with 4 browsers and 4 systems you need at least 16 rows however many 2- or 3-value parameters you add.

How to count the pairs

To know what “all pairs” means, count them: for every two parameters, multiply their value counts, then add everything up. For Browser (4), OS (4), Language (3), Payment (3):

Parameter pair	Pairs
Browser + OS	4 × 4 = 16
Browser + Language	4 × 3 = 12
Browser + Payment	4 × 3 = 12
OS + Language	4 × 3 = 12
OS + Payment	4 × 3 = 12
Language + Payment	3 × 3 = 9
Total	73 pairs

Each test row covers 6 pairs at once (one per parameter pair). So 73 pairs need at least 73 ÷ 6 ≈ 13 rows in theory, and the Browser + OS minimum pushes that to 16. Good tools land at or close to 16; the simple greedy generator in the lab below finds 18.

A worked example: checkout across browsers, OS, languages and payments

Try exactly this in the lab below: turn the constraints on and off and watch the numbers change.

Constraints: impossible and unwanted combinations

Without constraints, a generator happily produces “Safari on Windows” rows. You can’t run them, and the pairs they were supposed to cover (say, Safari + German) end up untested if you just delete the row. Constraints tell the tool what’s impossible up front, so it covers every valid pair with valid rows.

Be careful in the other direction too: every constraint removes pairs. Constraining “Apple Pay only with Safari” would hide a real bug where Apple Pay fails in Chrome on a Mac. Only exclude what truly can’t happen.

What pairwise doesn’t catch

• Higher-order interactions. A bug that needs German and PayPal and Firefox together appears only if that triple happens to land in one row. Pairwise doesn’t promise it. For risky features, tools can generate 3-way (or higher, often called t-way) coverage at the cost of more rows.
• Bad value choices. If you list “Chrome, Firefox” but forget mobile Safari, no algorithm will test mobile Safari.
• Rule-based behavior. When the result depends on specific combinations (members on rainy days pay €1), you need every rule tested: use a decision table.
• Order and timing. Pairwise picks values, not sequences. For flows over time, see state transition testing.
• The expected results. It tells you what to run, not what’s right.

Tools: PICT and friends

PICTPICTPairwise Independent Combinatorial Testing: Microsoft's free command-line tool that generates pairwise test sets.Open in glossary → (Pairwise Independent Combinatorial Testing) is a free, open-source command-line tool from Microsoft, available on GitHub. You write a plain text model file:

Browser:  Chrome, Safari, Firefox, Edge
OS:       Windows, macOS, Android, iOS
Language: English, German, Spanish
Payment:  Card, PayPal, Apple Pay

IF [Browser] = "Safari"    THEN [OS] IN {"macOS", "iOS"};
IF [Payment] = "Apple Pay" THEN [OS] IN {"macOS", "iOS"};

Then run pict checkout.txt and it prints one test per line, tab-separated, ready to paste into a spreadsheet or test management tool. A few useful options:

• /o:3 generates 3-way coverage instead of pairs (the default order is 2).
• /r randomizes the generation, so different runs give different sets with the same pair coverage.
• Values marked with ~ are treated as invalid (negative) values, and PICT puts at most one of them in each row, so a failing value doesn’t hide another.

Other options include NIST’s ACTS tool and many online pairwise generators. Whatever you use, the input is the same thinking: parameters, values, constraints.

When to use pairwise testing

Good fits:

• Configuration and device matrices: browser × OS × screen size, or device × OS version × language. See cross-browser testing and device fragmentation.
• Settings screens with many independent toggles and choices.
• Forms with many optional fields or filter panels: see search, filters and sorting.
• Feature flags that can be on or off in any mix.

Poor fits:

• Two parameters: pairwise = all combinations, there’s nothing to save.
• Business rules about specific combinations: decision tables.
• Safety- or money-critical combinations you can list: test them explicitly, and consider 3-way coverage.