In recent years, Continuous Integration has become an essential part of the software development process.
It enables you and other developers to merge code changes into a shared repository, detect errors early on, and deliver software faster with better quality.
However, as you scale your software development efforts and the complexity of your projects increases, the CI costs can grow exponentially. This can result in slower build times, increased infrastructure costs, and decreased developer productivity.
In this article, you will explore the issues and causes of growing CI costs and discover how you can keep your CI budget under control while maintaining high-quality software delivery.
I. Continuous Integration, a Fast-Growing Budget
Growing Team, Growing Project, Growing Budget
As software companies grow, they tend to take on more projects and expand their teams to meet demand. Well, it was absolutely true before the big layoff wave, but still mostly true.
While you can see this growth as positive, it can also lead to a significant increase in CI costs. And that's not something you really want. Except for CI providers, of course.
That's pretty logical; the larger the team, the more changes are being introduced into the codebase, which means more builds and tests are needed to maintain the integrity of the code. What about economies of scale? Unfortunately, it does not apply very well to software development.
Additionally, larger projects often require more complex infrastructure to support the increased demand, leading to higher costs. The combination of more significant projects and more extensive teams can quickly result in a growing CI budget, making it essential for you to identify and address the causes of this increase to avoid unnecessary expenses.
But, after all, this could be the price of success for growing teams and projects, right?
Automate and Test Everything
Automating and testing everything is essential to ensure software quality and improve development efficiency.
It could not be more true for us. Indeed, we are strong advocates of automation and testing at Mergify. However, we must recognize that this can lead to some complications.
While automation and testing can bring you significant benefits, they can also result in a significant increase in CI costs. The more you automate and run tests, the more infrastructure is required. And this is especially true when testing across multiple environments and platforms.
It appears essential to monitor and manage the associated costs to avoid the CI budget from exploding. Yup, it sounds boring, but it's a blessing in disguise.
Test, Again, and Again
The last underlying problem is the need to re-test.
After a pull request has been created, a whole battery of tests is executed. Rather reassuring.
But this can lead to new problems:
- What happens if there is an error? You need to fix the error and then run all the tests again.
- What happens if the PR is out of date before being merged? You have to update it and run the tests again.
Running the same tests over and over again increases the CI cost for each pull request and, therefore, your budget.
II. Alternative Solutions, Not Such Good Ideas
While looking for a nice trade-off between CI needs and budget management, some solutions may seem attractive, but few are really. Let's check them to avoid you implementing them.
1. Stop Testing
Believe in yourself and in your teammates. Why would you test something if you are confident enough? Isn't it a sign of weakness?
Joke apart, how could it be a suitable solution? Of course, you would crunch your CI costs and even make them disappear. But you would spend even more time and money to repair broken changes you integrate.
Definitely not such a good idea.
2. Reduce Test Coverage
It's never a good idea to go to extremes, so why not opt for a sensible trade-off? Instead of deleting all your tests, why not target some of them?
To choose them, you have two options: remove the most expensive tests or the most painful ones, the least permissive ones that do not let you integrate your changes.
If you put jokes aside, you know as well as we do that this is not an option. On the other hand, you can try to understand why some tests take longer than they should; it can come from the test as well as from the pull requests you propose.
Instead of reducing your test coverage, try to optimize it. You will reduce your CI budget while securing your development workflow.
3. Choose a CI Tool Based on Pricing, not Performances
Another idea: if your CI budget is costing you more, why not switch to the least expensive solution? You should be able to save a lot of money.
Here again, the idea may seem attractive, but it is not necessarily good. Indeed, if some tools seem to cost less than others, they can be less efficient and therefore take more time to run your tests. In the end, this could cost you much more by creating new bottlenecks. The longer the feedback, the longer the pull request will be fixed. Developers don't like to wait. It's harder to dive again into a task.
Moreover, cheap CI tools come with many limitations:
- Limited range of features. They may not have the ability to integrate with other tools or provide advanced reporting and analytics capabilities. This can limit your ability to customize your workflow and optimize your development process.
- Limited scalability: Cheaper CI tools may not be able to handle large and complex projects. As your project grows, you may find that the cheaper tool cannot keep up, leading to slow build times and increased downtime.
- Limited support: Cheaper CI tools may not provide adequate support, leaving you without assistance when you encounter issues. This can lead to extended downtime and delays in your development process.
- Security risks: Cheaper CI tools may not have the same level of security as more expensive ones. This can lead to vulnerabilities in your application, putting your data and your users at risk.
4. Track Developers' CI Spendings
Tracking your developers to save on Continuous Integration costs is like trying to lose weight by weighing yourself every hour.
While it may seem like a good idea, it doesn't address the underlying issue. CI costs are primarily driven by infrastructure and tooling, not developer time.
Plus, micromanaging your developers can lead to decreased productivity and a culture of mistrust. Instead, focus on optimizing your tools and building a culture of trust to improve productivity and reduce costs.
Just like you can't out-exercise a bad diet, you can't out-track a bad CI process!
III. The Right Solution: a Merge Queue
One of the most effective strategies to mitigate the growing costs of CI is implementing a merge queue.
A merge queue enables organizations to prioritize and manage the flow of code changes into the mainline branch.
By regulating the number of concurrent builds and limiting the number of changes being merged at once, a merge queue can help reduce infrastructure costs and optimize build times.
1. A Smart and Prioritized Testing Strategy
With a merge queue, you can establish the most intelligent testing strategy.
Instead of running all your tests on your newly created pull request, you can split them into two parts:
- On the pull request, you can run your unit tests and linters to make sure the PR is valid and error-free. Thus, the PR is embedded in the merge queue.
- Once it arrives at the end of the queue, it is updated, and at this precise moment, you can launch your most expensive tests, such as the end-to-end tests.
This allows you to avoid launching all your tests on a PR with the risk of having to restart them at the slightest error or if your PR is out of date.
2. Safe Integration = Less Reverts = Less Tests
One of the main advantages of using a merge queue is to secure the integration of changes in the code base.
By updating and testing each PR sequentially and automatically, you ensure that only valid changes are merged. You approach zero risk in your merge actions, thus protecting your branch hand.
You reduce the number of necessary reverts and all the consequences they entail. In particular, the need to create a new PR and to revert the CI on it.
By extension, you directly reduce your CI costs.
3. Exclusive Must-Have Features
You understood by using a a merge queue, you will take control of your CI budget.
Indeed, you'll be able to custom, automate and manage your test strategy.
In addition, Mergify's Merge Queue offers many features that allow you to decrease your CI Costs. Indeed, if a classic Merge Queue could increase your CI time by testing PR twice, the features offered by Mergify counterbalance this undesirable effect.
With Mergify, you can set PR batches and define their sizes using the
If you set to
5, Mergify create a draft pull request with the latest version of the main branch, and then merge the commits of the 3 next pull requests in the queue.
This way, you to build many pull requests and run your CI once.
Priority Management and Multiple Queues
Let's say there is a bug or an error. At this point, you don't want to test pull requests for new features but focus on the bug fixes.
These two features allow you to concentrate the CI on the urgent pull requests embarked in the specific queue while suspending it on the others.
Indeed, what is the point of merging new PRs if the main branch is corrupted? Since all CI tests can lapse, they would have to be rerun a second time, doubling your costs.
Implementing a merge queue in your development workflow can significantly improve the stability of your main branch.
This not only reduces the risk of conflicts and bugs but also controls CI spending.
Interesting in trying the most advanced merge queue for free? Check Mergify Merge Queue and ask for a demo.