Flaky Tests Automation: How to Make Them Frictionless

Flaky tests are automated tests that sometimes pass and sometimes fail without an obvious reason. This can be a major problem for software development teams, as it can make it difficult to determine whether a failure is due to a genuine bug or a flaky test.

Most of the time, one solution stands out for the software teams. A team member launches and reruns the test, by hand, until it passes.

Yes, it works, but it's a bit of a grind. So why not automate the retry of flaky tests?

How to Automate Flaky Tests Retries

One way to make flaky tests less frustrating is to automate their retries. This means that the test will automatically be re-run a certain number of times before it is considered to have failed. This can help to ensure that genuine bugs are not missed while also reducing the number of false positives caused by flaky tests.

There are a few different ways to automate test retries. One common approach is to use a test runner that supports this feature. Another option is to use a third-party tool that can be integrated with your test suite.

As we explained in the article How to Get Rid of Flaky Tests: Lethal Tools, there are a few tools like CI Monitoring by Mergify that can automate the retry of flaky tests.

Test Retries - Essential Factors

When setting up test retries, it is important to consider the following factors:

• The number of retries: This should be set high enough to ensure that genuine bugs are not missed but not so high that it causes the test suite to take too long to run.
• The wait time between retries: This should be long enough to give the test a chance to succeed but not so long that it causes the test suite to become unresponsive.
• The conditions for retrying: This should be set only to retry tests that fail for a specific reason, such as a timeout or an error message.

By automating test retries, software development teams can make flaky tests less frustrating and improve the reliability of their test suites.

Rather Than Getting Rid of Them, Avoid Them

Here are some additional tips for reducing flakiness in automated tests:

• Write clear and concise test cases.
• Use synchronization points to allow for asynchronous operations.
• Avoid using external dependencies in your tests.
• Use a test runner that supports test isolation.
• Monitor your test suite for flaky tests and take steps to fix them.

By following these tips, you can help reduce the number of flaky tests in your automated test suite and improve the reliability of your software.

To go one step further and increase your efficiency tenfold, you can also rely on tools that detect your flaky tests and automate their retry until they pass. This is what Mergify's CI Monitoring feature offers.