Software testing strategy: what a good QA plan covers

A software testing strategy sets out what is tested at each level, what is automated, where tests run and with what data, what must pass before a release, and who owns quality. It should fit on a few pages. A QA plan that is mostly a list of testing types hasn't answered the questions that decide whether bugs reach your users.

Strategy, plan and test cases are different things

The three terms get mixed up, which is how teams end up with a thick document and thin testing.

• The test strategy sets the approach for the whole product: test levels, tools, environments, release criteria and ownership. It changes rarely.
• The QA plan applies the strategy to one project or release. It says what is in scope, where the risk is, and who tests what and when.
• Test cases and automated tests are the checks themselves. They live with the code and in your test management tool, not in the strategy document.

Test levels and what each one catches

Each level catches a different kind of bug at a different cost. A balanced strategy uses all of them, with most of the checks at the fast, cheap end.

• Unit tests check one function or class on its own. They run in seconds and catch mistakes in business rules such as pricing, tax or permissions.
• Integration tests check that your code works with a real database, message queue or another service. Many of the expensive bugs live here.
• API and contract tests check that your services and third-party integrations still agree on the shape of requests and responses.
• End-to-end tests drive the real application through a browser or device for the journeys that matter most, such as sign-up, checkout or the core workflow. Keep them few, because they are slow and break easily.
• Non-functional tests cover performance, security and accessibility. Plans often leave them out, and users find the gaps.
• Exploratory and user acceptance testing puts a person in front of the product to find what scripts miss: confusing flows, odd data and missing error messages.

What to automate and what to leave to people

Automate the checks you will run again and again. Leave the checks that need judgement to people.

• Automate now: unit and integration tests for business rules, API contracts, the main user journeys, dependency and security scans, and everything in the regression suite.
• Keep manual: exploratory testing of new features, usability, visual polish, and one-off checks on a feature that is about to change anyway.
• Automate later: end-to-end tests for screens that are still being redesigned. Written too early, they need rewriting every sprint.

Automated tests should run on every pull request and block the merge when they fail. A suite that runs once a week, or that people are allowed to ignore, reports problems too late to matter. Fix or remove flaky tests quickly, because one unreliable test teaches a team to ignore a failed build. Our post on DevSecOps in the delivery pipeline covers which security checks belong at each stage.

Test environments and test data

Many bugs that only appear in production come from environments that don't match production, or from test data that is too tidy. The strategy should say:

• Which environments exist, such as development, staging and production, who can deploy to each, and how closely staging matches production.
• Where test data comes from: generated data, seed scripts or masked copies of real records. Unmasked personal data should never sit in a test environment, least of all for health or financial products.
• How data is reset between runs, so a test doesn't pass or fail depending on what ran before it.
• How third-party services are handled: their sandboxes, stubs, or a shared test account.

Regression: protecting what already works

Every release can break something that worked last month. The regression suite is the set of automated checks that guards existing behaviour. Add a test whenever you fix a bug, so the same bug can't come back unnoticed. Review the suite from time to time and remove tests that no longer earn their running time. Otherwise it slowly becomes too slow to run on every change.

Release criteria: what must be true before you ship

Release criteria turn 'is it ready?' from an opinion into a checklist. Agree them at the start of the project, not on release day. A typical set:

1. All automated tests pass on the release build.
2. No critical or high-severity defects are open, and the product owner has reviewed and accepted every known lower-severity defect.
3. QA has checked the acceptance criteria for every story in the release.
4. Performance on the main journeys is within the agreed limits.
5. Security scans show no unresolved high or critical findings.
6. Rollback has been tested, and someone on duty knows how to do it.
7. Release notes are written, and the support team knows what is changing.

For some products, a correct answer that arrives too late is still a failure. The streaming fraud scoring we built for a card issuer had to fit inside sub-second card authorisations. The model was rolled out in shadow mode, and a monitoring platform tracked scoring times and model health in production. Scoring takes 12ms at p95, and false positives fell by 35%.

Who owns quality

Saying that quality belongs to everyone doesn't help unless the jobs are split. A split that works:

• Developers write unit and integration tests alongside their code and don't hand over untested work.
• QA engineers design the test approach, build and maintain the automated suites, test each story as it finishes inside the sprint and do exploratory testing.
• The delivery lead holds the team to the release criteria and reports quality honestly, including when it is slipping.
• The product owner writes acceptance criteria, takes part in acceptance testing and decides which known defects can ship.

Each of our managed engineering squads includes QA from the first sprint. Automated tests and reviews are part of the squad's work in every sprint, not a separate phase at the end.

How to check that a vendor's QA is real

A QA plan is easy to write. These checks show whether it is being followed:

• Ask to see the pipeline. Tests should run on every pull request, and you should be able to see the results.
• Look at the defect log and note when each bug was found. Most should be caught inside the sprint, not by your users.
• Ask what was added to the regression suite after the last production bug.
• Pick one story from the last sprint and ask how it was tested.
• Watch the demo for edge cases: empty screens, error messages, slow connections and the wrong user role.
• Check that the release criteria were applied on the last release, not just written down.

For the numbers that show whether delivery stays healthy over time, see software delivery metrics for business leaders.

Where to start

If you have no strategy today, start with three habits: tests on every pull request, a written list of release criteria and a regression test for every bug you fix. Add rules for environments, test data and non-functional checks once those habits hold.

If you want a team that works this way from its first sprint, book a free 30-minute call and tell us about the product.