dmitriiev.dev

Passing tests that never talk to the server

· 8 min read

Passing tests that never talk to the server

Last time I took a production Playwright suite and inverted every primary assertion to find the greens that could not fail. Near the end I admitted where that gate goes blind: it proves a test can fail, not that it fails for the right reason. Assert on a heading that renders on every page, invert it, and the mutation dutifully dies. The test proves nothing about your feature and clears the gate anyway.

Here is the version of that problem that actually bites. A test asserts on something the user sees, and the thing the user sees is not the thing the server sent.

Take an order confirmation page that greets the customer by name:

await expect(page.getByTestId('receipt-greeting')).toHaveText('Thanks, Ada!'); // @primary-assert

Reads fine. It even fails honestly if you delete the greeting element. But look at where the name on screen actually comes from. To make the flow feel instant, the checkout page caches the shopper’s first name from their own input, and the confirmation page greets them from that cache instead of waiting on the server:

<!-- checkout page: remember what the shopper typed -->
<script>
  form.addEventListener('submit', () => {
    localStorage.setItem('shopperFirstName', nameInput.value.trim().split(/\s+/)[0]);
  });
</script>

<!-- confirmation page: greet from the cache, not from this response -->
<p data-testid="receipt-greeting">Thanks, Ada!</p>
<script>
  const el = document.querySelector('[data-testid="receipt-greeting"]');
  el.textContent = 'Thanks, ' + localStorage.getItem('shopperFirstName') + '!';
</script>

So the greeting the test reads comes from the browser’s cache, seeded by the shopper before the order ever went in, not from the confirmation the server returned. The one thing the test exists to verify, that the confirmation reflects the real order, is the one thing it no longer touches. If the server processed the order under the wrong name, or handed back someone else’s receipt, the browser would still redraw “Thanks, Ada!” from its cache and the test would stay green. The trap is narrow but common. Had the greeting used a name the server sent, corrupting the response would still catch a bad backend, because that name rode over the wire. It uses the name the shopper typed, so the assertion checks a copy the server never touched.

Inversion cannot see this. Flip toHaveText('Thanks, Ada!') to not.toHaveText(...) and the test goes red, because the greeting really does say “Thanks, Ada!” on screen. The mutation is killed, the gate is satisfied, and the weak test sails through. Inversion asks whether the assertion can fail. The question that catches this one is different: does the assertion fail when the data behind it is wrong.

Mutating the data instead of the assertion

So I stopped touching the assertion and started corrupting the thing it names. Keep the test exactly as written, but rewrite the value in the server’s response before it reaches the browser, then expect the test to go red. If it does, the assertion is genuinely wired to the backend. If it stays green, the value it checks is coming from somewhere other than the response, and the test is weak at the layer that matters.

The name matters here. This mutates the source of truth, not the assertion the way inversion does, and not the application code the way Stryker does. Corrupt the data the test leans on, then watch whether a green test finally goes red. Filed under a flat “mutation testing” the distinction disappears, and the distinction is the point: only this layer tells you the assertion is wired to the backend rather than to a copy the browser kept.

One dependency worth stating now, before you get attached: this only works when the asserted value crosses the network as text. Where the browser builds the string itself, there is nothing on the wire to corrupt, and the last section is about exactly that boundary.

Playwright already ships the hook for this, page.route. The gate injects it as a preamble at the very top of the test body, ahead of the test’s own page.goto, so the interception is already live for the first request the page makes:

await page.route('**/*', async (route) => {
  const type = route.request().resourceType();
  if (type === 'image' || type === 'font' || type === 'media') return route.continue();
  const resp = await route.fetch();
  let body = await resp.text();
  body = body.split('Thanks, Ada!').join('PMG_MUTATED_VALUE'); // break the value in transit
  await route.fulfill({ response: resp, body });
});

It fetches the real response, replaces every occurrence of the asserted string with a sentinel, and serves that. The pattern catches every request, but only text responses are rewritten: images, fonts, and media pass through untouched, since decoding them to search for a string would only corrupt them and the asserted value is never in there. The application does not change, and neither does the test. The gate reads the value straight out of the @primary-assert line, writes this preamble into a throwaway copy of the spec, runs it, records the verdict, and deletes the copy. You write none of it. It also means you need no access to the app’s source: the corruption happens in the browser at the network layer, against whatever URL the test already drives.

Three outcomes:

Run it on the confirmation test and the second pass says out loud what inversion could not:

tests/behavior-demo/receipt.spec.ts
  behavior:
  weak  Greets the customer by name on the confirmation page  :23
        the test stayed green while the value it names was broken

Where this works, and where it goes quiet

The catch has a hard requirement: the value your assertion names has to travel over the wire as a string the gate can find. That makes the second pass a good fit for server-rendered apps, Rails, Django, Laravel, Express templates, Next.js and Nuxt on the server, and for any client that prints an API string verbatim. Corrupt “Thanks, Ada!” in the HTML and, on a normal server-rendered page, the greeting on screen changes with it. Killed.

It goes quiet where values are assembled in the browser. I pointed it at my own suite first, which drives a client-rendered app, and got cannot-locate almost everywhere. That was the tool being honest rather than broken: on that app a total arrives as { "cents": 3400 } and becomes $34.00 in JavaScript, dates are formatted client-side, half the visible strings are stitched together from i18n keys. None of those literals are ever in a response, so there is nothing to corrupt and the gate says so instead of inventing a verdict. If your front end formats and composes its text in the browser, expect the behavior pass to shrug at most of it. Inversion still covers those tests; only this second pass is picky about your stack. A cannot-locate is a pointer as much as a limit: it marks the values that never reach the browser as text, the layer a unit test or an app-level mutation tool like Stryker should be covering instead.

The two verdicts are not equally strong, and it is worth being honest about which. killed is the softer one. Corrupting a value mid-flow can turn a test red for reasons that have nothing to do with the assertion: a downstream step that needed the real value, or a swap that left the payload malformed enough to blank the page. That inflates killed, not weak. The gate corrupts the exact string the assertion names, usually something specific like Thanks, Ada! rather than a bare 1 or true that would collide all over the page, which keeps the collateral down but never to zero. The point is the direction of the error: breakage can only ever turn a test red, so the worst it does is hand you a killed you did not earn. It cannot do the reverse and paint a weak test green.

So weak is the verdict I read without an asterisk. A test that stays green while the value it names is actively broken is simply not wired to that value, and no amount of structural damage could have faked that result. What weak does not tell you is whether you wanted the wiring. A test written to prove the UI renders an optimistic value from local state, before the server answers, is weak by construction and right to be. The verdict says the assertion does not depend on the response; whether that is a bug is your call, not the gate’s.

Try it

It is the same MIT-licensed gate I built for the last post, playwright-mutation-gate, with the second pass behind a flag:

npx playwright-mutation-gate tests/*.spec.ts --behavior

It stays behind a flag rather than running by default for two reasons: it doubles the work, one extra run per marked assertion on top of inversion, and, as the last section showed, it only earns its keep on some stacks. Point it at the specs a pull request touches, or run the full sweep nightly.

The greeting is real and runnable. It ships in the ai-qa-pipeline demo repo under tests/behavior-demo/, so you can watch a green, inversion-proof test get flagged weak with nothing to set up. Inversion tells you a test can fail. This tells you it fails for the reason you wrote it. Run both against a suite that has met neither and expect a backlog.