A QA Strategy That Survives Weekly Releases

Not all bugs are equally expensive, and testing effort should follow that curve rather than spreading evenly.

We sort functionality into three tiers:

• Catastrophic — data loss, payment errors, security or privacy breaches, anything that lets one user see another's data. These need automated tests, and they should block a release.
• Serious — a core journey is broken. Users cannot sign in, complete the main task, or receive a critical notification. Automated coverage of the happy path, plus alerting so you find out quickly if something slips through.
• Annoying — layout issues, a slow screen, an unclear message. Worth fixing, not worth blocking a release, and usually not worth an automated test.

Most struggling suites over-test the third tier. Those tests are numerous, brittle, tied to markup, and generate the failures that teach people to re-run rather than investigate — which is how a suite loses its authority.

Proportions matter more than totals. What works in practice:

Many unit tests, on logic worth testing

Pure functions, calculations, validation, state transitions. Fast, stable, and precise about what broke. Testing framework glue or trivial getters adds maintenance without signal.

A solid layer of integration tests

This is where the best return usually sits and where teams under-invest. Testing a real API endpoint against a real database catches the mismatches that unit tests mock away and end-to-end tests find too slowly.

A deliberately small set of end-to-end tests

Five to fifteen, covering only the journeys whose failure would be catastrophic or seriously damaging. They are slow and the most likely to be flaky, so every one must earn its place.

A useful discipline: keep a hard cap on end-to-end tests. Adding one requires removing one or making an explicit case. Without a cap they accumulate until the suite is unusable.

A test that fails intermittently is worse than no test, because it trains the team to ignore red. The moment "just re-run it" becomes normal, the suite has stopped protecting you.

Treat a flaky test as a bug with an owner and a deadline. If it cannot be fixed within a short window, quarantine it — move it out of the blocking suite and track it — rather than leaving it to erode trust.

The common causes are consistent and fixable:

• Timing assumptions. Waiting a fixed duration rather than for a condition.
• Shared state between tests. Order-dependent tests that pass alone and fail together.
• Real network calls to services outside your control.
• Time and timezone dependence. Tests that fail overnight or at month end.

Automation is poor at judgement, and that is exactly where manual effort should concentrate. Not repeating scripted regression passes a machine could run, but:

• Exploratory testing of new features. A person trying to break something new finds issues no pre-written script would have anticipated.
• Does this feel right? Pacing, tone, clarity, and whether the flow makes sense. Especially important in sensitive contexts, where a technically correct screen can still be wrong.
• Real-device and real-condition checks. Poor connectivity, older hardware, interruptions, accessibility tools.
• First-time-user perspective. Something the team loses permanently after a few weeks on a product.

A short, focused exploratory session on what changed will usually find more real problems than re-running a hundred scripted cases.

Teams that release comfortably have a written answer to "how do we know this is safe to ship". Usually something like:

1. The blocking suite is green — no re-runs, no known-flaky exceptions.
2. Exploratory testing on what changed is done, with findings triaged.
3. Migrations are backward compatible with the running version.
4. Rollback has been confirmed possible for this release.
5. Someone is available to watch error rates for the first period after deploy.

The value is in it being explicit. An implicit standard drifts under deadline pressure, and it drifts silently — which is why the release that goes wrong is so often the one shipped on a Friday afternoon by someone who assumed the checks had happened.

• Re-running a failed build is the normal first response.
• The suite takes long enough that people push and stop watching.
• Bugs reaching production are mostly in areas with high test coverage — a sign you are testing implementation rather than behaviour.
• Nobody can say what the suite does not cover.
• Test maintenance regularly consumes more time than the features it protects.

Any of these means the suite has drifted from managing risk to producing coverage. The fix is usually deletion rather than addition — and teams find that much harder than writing more tests.