A Passing Test Is Not a Pass

A passing test told me the system worked. It did not. I walk through what it took to close the gap.

Several years ago, I was testing the search function on an electronic program guide. An IPTV product. I wrote a test based on a simple set theory property: “Narrowing a search query should narrow the result set”.

I wrote what I called a relational test. I did not know the term “metamorphic testing” at the time. I was applying what my math training taught me, if one set is a subset of another, the relationship between them should hold under transformation.

I searched for the term “Old” and received many results. I searched for “Old House” and received fewer results. I checked that the result count from the narrower search was less than the broader result set. The subset relation held, except for Goldeneye. The James Bond film. Our search program was matching substrings too, and the word “old” sits inside “golden” like a splinter.

That test had two parts. The change I made to the input, narrowing “Old” to “Old House,” is the transformation. The relation is the hypothesis: narrow the query, and the results should narrow too.

A relational test does not check the answer. It checks whether the hypothesis holds, did the outputs change the way you expected, or stay the same? You are not looking for one right answer. You are looking at the relationship between the runs. You are measuring the consistency of that relationship. That is metamorphic testing¹. The term comes from Chen et al. (1998). The test is defined not by a specific expected output but by a relation between outputs.

The common introduction to metamorphic testing frames it around the oracle problem, systems where the expected output is unavailable. For example, ML models, compilers, nondeterministic systems. The implication is that normal software does not need it. Chen et al. themselves have noted that the technique applies even when an oracle exists. In my experience, that broader applicability is where the real value sits.

There are two different reasons you might lack an expected output to check against. The first: the output is genuinely unknowable. A compiler’s output may require a second compiler to verify. The second: the output is knowable but too expensive to have ready. You could, in principle, write out the correct results for every search query by hand. But the cost makes it impractical.

The search function I just described is the second kind. Fully deterministic. Fully specified. The expected output is not impossible to compute. It is too expensive to have ready. Metamorphic testing applies to both kinds, and that is the point most introductions miss. The relation does not care why you lack the answer. It only asks whether the outputs relate as expected.

Now try this one. Search for “car” in your favourite search engine. Then search for “red car.” You expect more results from the short query and fewer from the long one. The smaller set should fit inside the larger one. That’s a predictable change in the output based on a specific change in the input. Call this a variant (to change).

Sort a list. Sort it again. If sorting it twice gives you the same result as sorting it once, something held constant. The operation didn’t change the state beyond what the first sort already did. Call this an invariant (stay the same).

Hold those labels. They’ll come back.

A Held Relation Is Not a Pass

Let me walk through why that Goldeneye test could pass while the system was broken.

The search was matching substrings, not words. Goldeneye’s title contains the substring “old” buried inside “golden.” If the search matched substrings, Goldeneye would appear in both result sets, the “Old” search AND the “Old House” search. The subset relation would still hold. The test would still pass. The system would still be broken.

A held relation isn’t a pass. It’s a hypothesis that wasn’t violated.

Karl Popper² built a philosophy of science around this observation. You can never prove a theory is correct. You can only fail to prove it wrong. You can look at every experiment as a falsification attempt, and every time it survives, you haven’t confirmed the theory, you’ve failed to kill it this time.

The way I see it, a metamorphic test attempts to falsify a hypothesis about the system by transforming the input and checking whether the outputs still relate as expected. When the test passes, you have not proven the system is correct. You have failed to prove it wrong under these specific conditions.

Stacking Relations: Breadth Across Failure Classes

Writing three different tests on one behavior means each check constrains a different dimension, i.e. each one checks something the others cannot see.

Three tests emerged where I’d written one. Two are metamorphic relations. The third is something else, and the difference turns out to matter.

Note that three is a heuristic, not a quota. Two relations can leave a blind spot; three may close it; a high-risk behavior may earn five or six. The number follows the risk, not the other way around.

The subset relation (variant). Narrowing the query narrows the result set. This is what I already had. It held. The system was broken.

Case invariance (invariant). “OLD,” “Old,” and “old” must return the same result set. Case should not matter. Run it against the broken system and it passes, all three searches return Goldeneye, so the three sets are identical and the relation holds. Goldeneye is sitting in every one of them. Case invariance never notices. It only checks that the sets match each other, not what belongs in them. A relation sees only the dimension it constrains.

The exclusion check (a direct assertion, not a relation). A search for “old” must not return a title that matches only on a substring buried inside a longer word. Goldeneye violates that directly. Notice what this is and isn’t: it doesn’t compare two runs, and nothing is transformed. It is a property I assert on a single search. That makes it a plain oracle, not a metamorphic relation, and it is the only one of the three that catches Goldeneye

That last point is worth sitting with. Subset and case invariance are both metamorphic relations, and both are blind to the substring bug. They check how the outputs of different runs relate to each other. A wrong result that sits in every run equally never disturbs that.

The exclusion check is a different kind of oracle. It does not compare runs; it asserts one thing about a single output: a search for “old” must not return a title that matches only on a buried substring. I could never write out the full correct result for every query.

But I could always state one thing that must be true of it and checks that. A relation is a useful oracle. A direct assertion is an oracle. They catch different things, and a strong test uses both.

Each check constrains a different dimension. Subset constrains containment. Case invariance constrains case handling. The exclusion oracle constrains substring matching. Each catches a class of failure the others miss.

Triangulation: When Three Relations Actually Converge

Stacking three different checks gave me breadth. Subset, case invariance, and the exclusion oracle point in three different directions. They don’t intersect. They cover different failure classes.

But there’s a second pattern. Three relations that all converge on the same dimension. Three different relations constraining the same output property from different angles.

Take the output property: result set membership. Which items belong in the result set and which don’t?

Now, here is an example of three different relations that all constrain membership.

Relation 1: Subset (variant)

Narrowing the query must narrow the results. Constrains membership by checking containment.

Relation 2: Synonym equivalence (invariant)

Two queries that mean the same thing should return the same results. If you search “car” and “automobile,” the result sets should be identical. This constrains membership by checking semantic consistency. If the synonym dictionary is wrong, this relation catches it. The subset relation would not.

Relation 3: AND intersection (variant)

A compound query joined by AND must return only results that appear in both individual queries. Search “drama AND old”, every result must appear in both the “drama” result set and the “old” result set. This constrains membership by checking boolean logic. If the AND operator is broken, neither the subset relation nor the synonym relation would catch it.

Three different metamorphic patterns. Three different relational hypotheses. All three constrain the same output property: membership of a set.

When all three hold, you have convergence (triangulation) on membership correctness.

This doesn’t prove the system is correct. But it ruled out three different ways it could be wrong along the same axis. When one fails, you know more than “something is wrong.” You know the dimension (membership) and the specific constraint that was violated (synonyms, or booleans, or containment).

Therefore, in this example, breadth when you need coverage across failure classes, three relations on three different dimensions. Depth when one specific dimension is critical, three relations on the same dimension. Both share the same discipline: each relation tested across multiple transformations, not just one.

Three is a heuristic, not a quota.

Transformation multiplicity

For me, one passing input pair is not evidence. The discipline I apply against that is what I call transformation multiplicity: hold the relation constant and test it with every kind of change the variable allows, and within each kind, try different values.

The metamorphic testing literature typically treats each specific transformation as a separate relation. The tests are the same. The thinking that generates them is not, holding the relation constant and varying the transformations tells me whether it held because of the system’s logic or because I got lucky with one kind of input change. The question is simple: how many ways can I transform this variable while preserving the same relation?

A free-text query allows substitution, adding, removing, reordering. The space is wide. A single Boolean allows almost nothing, true or false, and you are done. The more room the variable gives me, the more transformations I can generate. When the variable is rich, I use that richness.

Take the subset relation from the Goldeneye test: narrowing the query narrows the results. That is one relation. But “narrowing” can be performed in more than one way.

Add a specifier: “car” to “red car.”

Replace with a narrower category: “car” to “sedan.”

Add a structural constraint: “car” to an exact-phrase match on “car.”

Three narrowing operations.

Three different things the search must do.

Same relation tested three ways.

If the subset relation holds across all three, I have stronger evidence than if it held on one pair alone. If it breaks on one, I know which kind of narrowing exposed the fault.

For an invariant relation, the same discipline applies. Take “sorting twice should not change the result.” Add an element to the list. Remove one. Replace one with a duplicate. Sort twice after each. The invariant holds or it does not. But I have checked it against three different starting conditions, not one.

Within each transformation, I also vary the concrete values. “Car” to “red car” is one trial. “Car” to “blue car” is another. “Car” to “electric car” is a third. Same operation (add a specifier), different values, each a separate trial. One trial is an anecdote. Five trials with diverse values across multiple transformations is confidence.

Three dimensions of multiplicity. Different operations on the same variable: substitute, add, remove. Within each operation, different concrete values: “red car,” “blue car,” “electric car.” And different variables entirely: if the subset relation should hold when narrowing by keyword, does it also hold when narrowing by category filter, or by price range? I find each dimension multiplies the evidence. My goal is to cover as much of the space the variable gives me as the risk warrants.

Transformation multiplicity applies within stacking

When three relations converge on one behavior, each one still deserves its own transformation multiplicity. The subset relation gets multiple narrowing operations. The synonym equivalence relation gets multiple synonym pairs. The AND intersection relation gets multiple compound queries. Stacking tells me which relations to write. Transformation multiplicity tells me how thoroughly to test each one. They are not competing moves. They are orthogonal.

What Does a Passing Test Actually Mean?

This is the question underneath everything I’ve said. If a passing relation isn’t proof the system is correct, what have we learned?

A passing relation means one thing: this specific hypothesis was not violated, with these specific inputs, at this specific moment.

That’s it.

When the subset relation held for “Old” and “Old House,” the test proved exactly that: the subset property was satisfied for those two queries, at that time, in that build. It proved nothing about substring matching. Nothing about case handling. Nothing about any query I did not test.

A green result is a single data point. A single constraint that held.

Testing never finishes. It stops when the evidence is enough for the risk.

Two Questions Are All You Need

The taxonomy of metamorphic testing patterns is overwhelming. Idempotency, monotonicity, conservation, symmetry, inclusion, exclusion. It reads like an academic paper, not something you can use on a Tuesday morning, before your coffee.

I have come up with one classification question applied to every variable in every test.

Should changing this variable matter to the output?

If no, the relation is invariant. Whitespace around a search term should not change search results. The user’s display name should not affect their permissions. The order of items in a shopping cart should not change the total. Two LLM outputs for the same prompt will produce different text, but the meaning should stay the same, invariant does not always mean identical, it means equivalent in the dimension that matters.

If yes, in a specific predictable way, the relation is variant. Adding an item to a cart should increase the total by the price of that item. Sorting a list descending should reverse the ascending order.

A thinking tool I use: “this shouldn’t matter” and “this should cause a specific change.”

Now look at the Goldeneye relations through this lens.

The subset relation was variant. Changing the query from “Old” to “Old House” should cause a specific predictable narrowing of the result set. It held. This relation couldn’t see the substring bug.

Case invariance is invariant. The case of the query should not matter. It would not have caught Goldeneye either. It rules out a completely different class of failure.

The exclusion check sits outside both buckets. The two-question lens classifies relations, a transformation and what it does to the output. The exclusion check isn’t a transformation; it’s a property asserted on a single run. That is exactly why it catches what the relations miss.

One variant relation, one invariant relation, and one direct oracle on the same behavior. The relations give you breadth across transformations. The oracle gives you the property they both leave uncovered. That is the shape of a fully constrained test.

This is not a script. It’s a thinking scaffold. Get the essence and adapt it. The steps are here so you don’t start from zero. They are not here so you can stop thinking.

Step 1: Identify your Variables

Step 2: Define and classify the relation.

Variant or invariant?

Step 3: Write a relation from the other bucket.

If a variant relation is the primary test? See if an invariant relation on the same behavior can be had. And vice-versa if the relation is invariant. The question: what class of failure can my primary relation never catch? The answer points you to the missing relation.

For Goldeneye, the primary was a variant (subset). The other bucket gives you case invariance: the case of the query should not matter. Writing it is good discipline but notice it still does not catch the substring bug. Two metamorphic relations, same blind spot.

That is the signal to step outside the relations and assert the property directly: a search must not return a title that matches only on a buried substring. That direct check is what catches Goldeneye.

When I get stuck, I look at every variable in the behavior and ask, “should changing this matter?” The answer, yes or no, points me toward a relation I have not written yet. A variable I overlooked is usually where the missing relation lives.

Step 4: Find a third independent angle.

The third check often lives in a category I have not examined yet, and it is not always another relation. Sometimes it is a property you assert directly.

Two relations can leave a blind spot. Three relations can cover more gaps. It’s not a hard rule, but it’s a helpful one.

The third relation serves a different purpose depending on whether you’re stacking for breadth or converging for depth.

Breadth. Coverage across failure classes, three checks on three different dimensions. Subset caught containment violations. Case invariance caught case-handling violations. The exclusion oracle caught the substring match. Two of the three are metamorphic relations; the third is a direct assertion. Breadth does not mean stacking only relations, it means covering different failure dimensions with whatever check fits each one.

Depth. One critical dimension, three relations on the same dimension. Subset checked containment. Synonym equivalence checked semantic consistency. AND intersection checked Boolean logic. Three different angles on the same question: which results belong in the set? I rule out three different ways the same property could be wrong.

Either way, the third relation isn’t a bonus.

Step 5: Apply transformation multiplicity to each relation you wrote.

For each relation, I ask: how many ways can I transform this variable? If I can apply more than one, then I identify other useful operations. Vary the values within each. One relation with one transformation is an anecdote. The same relation across multiple transformations is evidence.

Independent methods to assess correctness

One discipline applies across every relation and every transformation in this article. Verify the result through three independent paths. For example:

· an API call

· A web UI action

· A database query

· A log file

· A report

· Etc…

Same behavior, different routes to the answer.

In my practice, I was doing this long before I had a name for it, checking the same result from the API, then from the UI, then from the database, because I did not trust any single path to tell me the whole truth.

I later learned that Robert Sabourin³ teaches this as the principle of independent methods to assess correctness. This is not a metamorphic relation. It is not a transformation. It is a verification step. The result you observed, is it the result the system produced, or an artifact of the path you tested through? It applies to everything above.

Testing is Experimentation

Ask the question “should this variable matter?” of every input. Test every relation across multiple transformations where the variable allows it. Write the relation from the other bucket. Find the third independent angle.

Pick one test you wrote last week. Apply transformation multiplicity (if applicable) to its relation. Classify it as invariant or variant. Write the relation from the other bucket. Find a third variable neither covers and write a relation for it. Run all three.

You are not a checker. You are running experiments.

1.

Chen, T. Y., S. C. Cheung, and S. M. Yiu. “Metamorphic Testing: A New Approach for Generating Next Test Cases.” Technical Report HKUST-CS98-01, Department of Computer Science, The Hong Kong University of Science and Technology, 1998.

2. Popper, Karl R. The Logic of Scientific Discovery. Routledge, 2002. (Originally published as Logik der Forschung, 1934.)

3. Sabourin, Robert. 2024. Charting the Course: Coming Up with Great Test Ideas; Just in Time. Notion Press.