Your green tests are lying
CI goes green, the PR merges, everyone moves on. The whole ritual rests on one assumption nobody checks: that the test would have gone red if the feature were broken. For a surprising number of tests, that assumption is false. They pass because they cannot fail.
I maintain a production Playwright suite, and I wanted an actual answer instead of a feeling. So I took every test’s main assertion, inverted it, and re-ran the suite. If the original assertion is doing real work, the inverted version must fail. If the test stays green with its own assertion flipped, the assertion is hollow and the green checkmark has been decorative the whole time.
Before the results, here’s what hollow looks like in practice. None of these are exotic. All of them pass in CI every day.
The floating assertion. Playwright’s web-first assertions are async. Drop the await and the expectation becomes a promise nobody looks at:
test('shows confirmation toast', async ({ page }) => {
await page.getByRole('button', { name: 'Save' }).click();
expect(page.locator('.toast')).toBeVisible(); // no await
});The test finishes before the assertion resolves. Delete the toast from the app and this still passes. Recent Playwright versions flag some of these with no-floating-promises-style lint rules, but only if the lint rule is on, and in most suites I’ve seen it isn’t.
The negative assertion on a selector that never matched. Asserting absence is a footgun, because absence is also what you get when your selector is wrong:
await expect(page.locator('#eror-banner')).not.toBeVisible();The typo means this element has never existed on any page, so the assertion is trivially true forever. The error banner can be up in the user’s face and the test agrees everything is fine.
The guarded assertion. Someone got tired of a flaky check and wrapped it:
if (await banner.isVisible()) {
await expect(banner).toContainText('Payment failed');
}Now the assertion runs only when it would pass. When the banner doesn’t render at all, which is the actual bug, the whole block is skipped and the test is green.
The test that isn’t running. test.skip(true, 'flaky, revisit later') from eight months ago. The test file exists, the test count in the report includes it, and it has executed zero times since the commit.
Notice what these four have in common: only the first is a lint problem. A dropped await is visible in the syntax, and no-floating-promises will catch it if someone turned it on. The other three are perfectly well-formed code. No static tool can know that #eror-banner has never matched anything, that a guard condition never fires, or that a skip from last winter was supposed to be temporary. The only way to expose them is to run the test and demand that it fail.
Why this is getting worse right now
Hollow tests aren’t new, but AI test generation industrialized them. An LLM will happily produce a spec that compiles, follows your fixtures, uses idiomatic locators, and asserts something trivially true, or asserts against a field that doesn’t exist behind a guard that never fires. It looks like coverage. The industry has started calling it coverage theater, and the numbers back the name: in Mabl’s 2026 survey (n=996), teams report spending around a fifth of the week manually auditing AI-written tests, while 35% of production bugs are still found by customers first.
Manual audit doesn’t scale against a generator. If the machine writes tests faster than you can read them, the review has to be a machine too.
Inverting assertions as a gate
The check I described is mutation testing, but aimed at a different layer than usual. Stryker and friends mutate your application code and ask whether the tests notice. That’s the right tool for unit suites. For E2E it’s usually impractical: you’d be rebuilding and redeploying the app per mutant. Flipping it around is cheap: mutate the assertion and ask whether the test notices. One inversion per test, no app rebuild, and the runtime cost is a re-run of each test.
I packaged the workflow as playwright-mutation-gate. You mark the one assertion each test exists for:
await expect(page.getByRole('heading', { name: 'Order confirmed' })).toBeVisible(); // @primary-assertThe gate copies the spec, inverts the marked assertion (toBeVisible() becomes not.toBeVisible() and vice versa), and runs each mutation through npx playwright test. Every marked assertion gets a verdict. Killed means the mutated test failed: the assertion is alive. Survived means it stayed green with its own assertion flipped: hollow, and the gate exits 1. Cannot-mutate means the line resisted inversion, and the gate reports why instead of silently passing. Tests with no marker also fail the gate by default, because an unmarked test is a test whose main claim nobody has stated.
The cost is easy to reason about: one Playwright run per marked assertion, so the gate takes roughly as long as your suite times one. My 64 mutations ran against a live staging environment as a background job: noticeable, and clearly not something to bolt onto every commit. Treat it like any expensive suite: feed it only the spec files a PR touched (the CLI takes file paths, so git diff slots in naturally), shard it through --playwright-args, or run the full sweep nightly. A hollow assertion doesn’t get less hollow between merges; catching it a day later is fine.
The “one primary assertion per test” constraint is doing double duty here. It’s what makes the mutation tractable, and it’s a discipline worth having anyway: if you can’t point at the single line a test exists to check, the test is probably asserting incidentals.
What the suite confessed
Running the gate across the full suite: 16 spec files, 64 marked assertions. Verdict: 63 killed, 0 survived, 1 cannot-mutate.
The 63 are the good news, and genuinely useful news: I can now say the suite’s greens are load-bearing, instead of assuming it. A flaky test even got a fair trial along the way. The gate counts a mutation as killed if the inverted assertion failed at least once, so retries can’t launder a hollow assertion into a survivor.
The one cannot-mutate was the interesting find. It pointed at a spec covering location-dependent search results, whose entire describe block sat under test.skip(true, ...). The skip was legitimate when it was written: the staging proxy rewrites the forwarded-IP header the feature reads, so the behavior can’t be exercised in that environment. The comment promised a revisit. The revisit never came, and for months nothing in the CI output distinguished that file from a healthy one.
The gate flagged it through simple mechanics. A skipped test never executes, so there is no way to observe its inverted assertion fail. Rather than guess, the gate refuses to issue a verdict and says why. That refusal is the point. A verdict of “I could not check this” is information; a green checkmark over a test that never ran is misinformation.
This was a suite in good shape: 63 of 64 assertions just proved they can go red, about as clean a bill of health as E2E coverage gets. And it still contained an entire feature with zero automated coverage, invisible in every green run for months. The blind spot didn’t survive because anyone was careless; it survived because no human rereads skipped specs, and nothing in the toolchain is responsible for them. If a well-kept suite hides one of these, I don’t want to guess what an average one hides.
What this does not catch
The gate answers one question: can this assertion fail at all. It does not judge whether the assertion checks the right thing. Assert on a page heading that appears on every page of your app, and the inversion will dutifully fail, the mutation is killed, and the test still proves nothing about your feature. Weak assertions are a different disease from hollow ones. Catching them takes code review, or Stryker-style mutation of the application itself where you can afford it.
A kill is also slightly generous by construction. Playwright actions carry implicit assertions (click() fails on its own if the button never appears), so a mutated test can die before it ever reaches the flipped assertion, or die of ordinary flakiness on that run. A kill proves the test is capable of failing, not that the marked line is what failed. The asymmetry works in your favor, though: survived has no such excuse. A test that stays green with its own primary assertion inverted is hollow, full stop, and that’s the verdict the gate is built around.
There are mechanical limits too: one marked assertion per test, and multi-line expect chains report as cannot-mutate rather than getting clever. The README’s Limitations section is honest about all of this, because a gate you can’t trust to report its own blind spots would be recreating the exact problem it exists to solve.
Try it on your suite
The gate is MIT-licensed, on GitHub as playwright-mutation-gate, with the npm release close behind. Mark your primary assertions, run
npx playwright-mutation-gate tests/*.spec.tsand read the table. My suite was in good shape and still had a dead spec hiding in the green. On a suite that has never been through a check like this, expect the first run to surface a percent or two of assertions that can’t fail, and, if your codebase is anything like mine, at least one spec that quietly stopped running. The only way to know which checkmarks you can trust is to make every test prove, once, that it knows how to go red.