Why Should You Get Rid Of Flaky Tests? Unmasking the Hidden Nuisance in your CI/CD Pipeline
Amidst the quest for automation and speed, a common obstacle emerges: flaky tests. These seemingly innocent but troublesome tests can undermine the entire CI/CD optimization process.
To understand the nuisance of flaky test, it's like having a roommate you can more or less count on.
When you ask him for a hand, you never know how he'll handle your request.
He may or may not pay the rent, and the same goes for chores. When you get home from work, your last yoghurt may be waiting for you, or it may be digesting in his belly.
That's what we call an unreliable roommate. Despite identical rules, conditions and environment, he can make random decisions.
In fact, a flaky test is just the same. Under identical conditions, a flaky test will either pass or fail. A sort of Russian roulette.
In this article, we will dive into the concept of flaky tests, understand their implications, and explore why they are a nuisance.
What Are Flaky Tests?
Flaky tests, also known as unreliable or intermittent tests, are automated tests that exhibit inconsistent behavior.
These tests are characterized by their ability to produce both passing and failing results for the same codebase without any modifications. In simpler terms, running a flaky test multiple times can yield different outcomes, leading to inconsistent feedback regarding the stability and correctness of the system.
If you want to know more about what are flaky tests and their classification, you should read Fabien's article about flaky tests definition and classification.
The Nuisance of Flaky Tests
Flaky tests can be detrimental to CI/CD optimization in several ways:
✅ False Positives
Flaky tests can generate false positives, reporting failures when no actual defects exist. This can create unnecessary delays and frustration for developers, who have to investigate and fix non-existent issues, leading to wasted time and effort.
❌ False Negatives
Conversely, flaky tests can also produce false negatives, where actual defects go undetected. When tests sporadically pass, potential issues can slip through the cracks, jeopardizing the quality of the software being delivered.
🤨 Unreliable Feedback
The primary purpose of automated tests is to provide reliable feedback on the stability and quality of the codebase. Flaky tests undermine this objective, making it difficult to distinguish between genuine failures and transient issues, eroding developers' trust in the test suite.
📉 Decreased Productivity
Flaky tests hinder the efficiency of the development process. Developers spend precious time investigating and re-running tests to identify and reproduce failures, detracting them from focusing on crucial tasks like implementing new features or fixing genuine bugs.
🤔 Impaired Confidence
Flaky tests erode the confidence of both developers and stakeholders. Developers become skeptical of the test results, and stakeholders may lose faith in the reliability of the software being produced. This lack of confidence can have far-reaching consequences, affecting team morale and the overall success of the project.
How Can I Get Rid of Them?
Flaky tests are a real nuisance, and their impact is often underestimated. This problem, if not properly dealt with, will be dragged along like a ball and chain by the developers on your team. And the consequences can be dire:
- velocity at half-mast,
- vulnerable code base,
- loss of trust and consideration among developers...
Suffice it to say, it doesn't look good! Faced with this situation, it's time to adopt a proactive stance and look for concrete solutions. Fabien has already dealt with this subject in his article Flaky Tests: How to Fix Them?
You can also take advantage of various tools, such as those from CircleCI, Jetbrains, or CI Monitoring solution from Mergify.