Treat flaky tests as defects

A flaky test passes and fails on the same commit without code changes. One retry might green the build. That teaches the team that red means nothing.

I treat flakes as production bugs in the test suite, not as CI noise.

When a required check flakes once a week, teams stop investigating failures. They click rerun until green. That hides real regressions next to fake ones.

Track flake rate per test over two weeks. If checkout-flow fails three times on unchanged commits, it owns a ticket.

Automatic retries can help diagnosis, but unlimited retries on required checks hide the problem:

- uses: nick-fields/retry@v3
  with:
    timeout_minutes: 10
    max_attempts: 2
    command: npm run test:e2e

Two attempts with logged first failure beats five silent successes.

Fix the root cause

Common flake sources:

• fixed sleep(2000) instead of waiting for a selector
• shared test data between parallel jobs
• clock or timezone assumptions
• race on port binding

Replace time guessing with a deterministic wait:

await page.waitForSelector('[data-testid="done"]')

Quarantine only as a temporary narrow skip with an owner and deadline:

test.skip('checkout-flow', () => { /* issue #482, fix by Friday */ })

Preserve first-failure evidence

Upload Playwright traces or screenshots on the first failed attempt. Fix the test, then remove the retry wrapper if you added one only as a bandage.

Flaky required checks are worse than flaky optional jobs because they block merge. Either fix the test or downgrade the check until it is stable.

Record flake fixes in the pull request that changes the test, not only in chat. The history helps the next person trust green again.

Try this on your own project: pick the noisiest test, run it twenty times locally, and replace one timing guess with an explicit condition.