Defining a Release Candidate Test Cutline
Near release time, teams always face the same pressure: should we validate more, or should we ship now? Running every test is often too slow, but validating too little is dangerous. What teams need is not maximum test volume, but a clear release cutline.
A cutline is a risk policy, not just a test list
The best release cutlines define the risks that must be blocked:
- payment regression
- authentication failure
- data loss risk
- breakage on major browsers or devices
Once those risks are clear, the required test set becomes easier to define.
Not every test belongs at the same decision level
A practical release-candidate structure often has three layers:
- fast must-pass confidence set
- critical user-journey validation set
- extended reference-only validation set
The first set is the real cutline. The second strengthens confidence. The third is useful context, but should not always block shipment.
Flaky tests corrupt the cutline
If unreliable tests are allowed into the cutline, teams start ignoring failure. At that point, the cutline stops functioning.
Good cutline candidates should be:
- reproducible
- diagnosable
- predictable in execution time
Track untested risk, not only passed tests
If some scenarios were skipped because of time or environment issues, that is also decision information. Release reporting should include not just what passed, but also what risk remains unseen.
Conclusion
A release-candidate cutline is not a QA artifact. It is a decision device that makes clear what the team trusts and what it still does not know at ship time. Strong teams are not the ones that run the most tests. They are the ones that know exactly where the cutline is.
Continue Reading
Related posts
Flaky Test Triage Playbook
A practical way to classify, contain, and fix flaky tests before they erode trust in the entire pipeline.
🧪 TestDesigning Smoke Tests for Release Gates
You cannot run everything before deployment. A strong release gate depends on a short, reliable smoke test set with clear purpose.
📱 MobileRunning a Mobile Crash Budget
Mobile stability is not only about reducing crashes. It is also about deciding which level is acceptable and when release should stop.
📱 MobileA Mobile Feature Flag Expiry Playbook
Feature flags accelerate releases, but if they are never retired they quickly increase code and operational complexity.
Next Path