Keep Control of your CI Budget with a Merge Queue

In recent years, Continuous Integration has become an essential part of the software development process.

It enables you and other developers to merge code changes into a shared repository, detect errors early on, and deliver software faster with better quality.

However, as you scale your software development efforts and the complexity of your projects increases, the CI costs can grow exponentially. This can result in slower build times, increased infrastructure costs, and decreased developer productivity.

In this article, you will explore the issues and causes of growing CI costs and discover how you can keep your CI budget under control while maintaining high-quality software delivery.

I. Continuous Integration, a Fast-Growing Budget

Growing Team, Growing Project, Growing Budget

As software companies grow, they tend to take on more projects and expand their teams to meet demand. Well, it was absolutely true before the big layoff wave, but still mostly true.

While you can see this growth as positive, it can also lead to a significant increase in CI costs. And that's not something you really want. Except for CI providers, of course.

That's pretty logical; the larger the team, the more changes are being introduced into the codebase, which means more builds and tests are needed to maintain the integrity of the code. What about economies of scale? Unfortunately, it does not apply very well to software development.

Additionally, larger projects often require more complex infrastructure to support the increased demand, leading to higher costs. The combination of more significant projects and more extensive teams can quickly result in a growing CI budget, making it essential for you to identify and address the causes of this increase to avoid unnecessary expenses.

But, after all, this could be the price of success for growing teams and projects, right?

Automate and Test Everything

Automating and testing everything is essential to ensure software quality and improve development efficiency.

It could not be more true for us. Indeed, we are strong advocates of automation and testing at Mergify. However, we must recognize that this can lead to some complications.

While automation and testing can bring you significant benefits, they can also result in a significant increase in CI costs. The more you automate and run tests, the more infrastructure is required. And this is especially true when testing across multiple environments and platforms.

It appears essential to monitor and manage the associated costs to avoid the CI budget from exploding. Yup, it sounds boring, but it's a blessing in disguise.

Test, Again, and Again

The last underlying problem is the need to re-test.

After a pull request has been created, a whole battery of tests is executed. Rather reassuring.

But this can lead to new problems:

• What happens if there is an error? You need to fix the error and then run all the tests again.
• What happens if the PR is out of date before being merged? You have to update it and run the tests again.

Running the same tests over and over again increases the CI cost for each pull request and, therefore, your budget.

II. Alternative Solutions, Not Such Good Ideas

While looking for a nice trade-off between CI needs and budget management, some solutions may seem attractive, but few are really. Let's check them to avoid you implementing them.

1. Stop Testing

Believe in yourself and in your teammates. Why would you test something if you are confident enough? Isn't it a sign of weakness?

Joke apart, how could it be a suitable solution? Of course, you would crunch your CI costs and even make them disappear. But you would spend even more time and money to repair broken changes you integrate.

Definitely not such a good idea.

2. Reduce Test Coverage

It's never a good idea to go to extremes, so why not opt for a sensible trade-off? Instead of deleting all your tests, why not target some of them?

To choose them, you have two options: remove the most expensive tests or the most painful ones, the least permissive ones that do not let you integrate your changes.

If you put jokes aside, you know as well as we do that this is not an option. On the other hand, you can try to understand why some tests take longer than they should; it can come from the test as well as from the pull requests you propose.

Instead of reducing your test coverage, try to optimize it. You will reduce your CI budget while securing your development workflow.

3. Choose a CI Tool Based on Pricing, not Performances

Another idea: if your CI budget is costing you more, why not switch to the least expensive solution? You should be able to save a lot of money.

Here again, the idea may seem attractive, but it is not necessarily good. Indeed, if some tools seem to cost less than others, they can be less efficient and therefore take more time to run your tests. In the end, this could cost you much more by creating new bottlenecks. The longer the feedback, the longer the pull request will be fixed. Developers don't like to wait. It's harder to dive again into a task.

Moreover, cheap CI tools come with many limitations:

1. Limited range of features. They may not have the ability to integrate with other tools or provide advanced reporting and analytics capabilities. This can limit your ability to customize your workflow and optimize your development process.
2. Limited scalability: Cheaper CI tools may not be able to handle large and complex projects. As your project grows, you may find that the cheaper tool cannot keep up, leading to slow build times and increased downtime.
3. Limited support: Cheaper CI tools may not provide adequate support, leaving you without assistance when you encounter issues. This can lead to extended downtime and delays in your development process.
4. Security risks: Cheaper CI tools may not have the same level of security as more expensive ones. This can lead to vulnerabilities in your application, putting your data and your users at risk.

4. Track Developers' CI Spendings

Tracking your developers to save on Continuous Integration costs is like trying to lose weight by weighing yourself every hour.

While it may seem like a good idea, it doesn't address the underlying issue. CI costs are primarily driven by infrastructure and tooling, not developer time.

Plus, micromanaging your developers can lead to decreased productivity and a culture of mistrust. Instead, focus on optimizing your tools and building a culture of trust to improve productivity and reduce costs.

Just like you can't out-exercise a bad diet, you can't out-track a bad CI process!

III. The Right Solution: a Merge Queue

One of the most effective strategies to mitigate the growing costs of CI is implementing a merge queue.

A merge queue enables organizations to prioritize and manage the flow of code changes into the mainline branch.

By regulating the number of concurrent builds and limiting the number of changes being merged at once, a merge queue can help reduce infrastructure costs and optimize build times.

1. A Smart and Prioritized Testing Strategy

With a merge queue, you can establish the most intelligent testing strategy.

Instead of running all your tests on your newly created pull request, you can split them into two parts:

• On the pull request, you can run your unit tests and linters to make sure the PR is valid and error-free. Thus, the PR is embedded in the merge queue.
• Once it arrives at the end of the queue, it is updated, and at this precise moment, you can launch your most expensive tests, such as the end-to-end tests.

This allows you to avoid launching all your tests on a PR with the risk of having to restart them at the slightest error or if your PR is out of date.

2. Safe Integration = Less Reverts = Less Tests

One of the main advantages of using a merge queue is to secure the integration of changes in the code base.

By updating and testing each PR sequentially and automatically, you ensure that only valid changes are merged. You approach zero risk in your merge actions, thus protecting your branch hand.

You reduce the number of necessary reverts and all the consequences they entail. In particular, the need to create a new PR and to revert the CI on it.

By extension, you directly reduce your CI costs.

3. Exclusive Must-Have Features

You understood by using a a merge queue, you will take control of your CI budget.

Indeed, you'll be able to custom, automate and manage your test strategy.

In addition, Mergify's Merge Queue offers many features that allow you to decrease your CI Costs. Indeed, if a classic Merge Queue could increase your CI time by testing PR twice, the features offered by Mergify counterbalance this undesirable effect.

Batches

With Mergify, you can set PR batches and define their sizes using the batch_size option.

If you set to 5, Mergify create a draft pull request with the latest version of the main branch, and then merge the commits of the 3 next pull requests in the queue.

This way, you to build many pull requests and run your CI once.

Priority Management and Multiple Queues

Thanks to Priority Management and Multiple Queues you choose which pull request should be tested and then merged first.

Let's say there is a bug or an error. At this point, you don't want to test pull requests for new features but focus on the bug fixes.

These two features allow you to concentrate the CI on the urgent pull requests embarked in the specific queue while suspending it on the others.

Indeed, what is the point of merging new PRs if the main branch is corrupted? Since all CI tests can lapse, they would have to be rerun a second time, doubling your costs.

queue_rules:
  - name: urgent
    merge_conditions:
      - "#approved-reviews-by>=2"
      - check-success=Travis CI - Pull Request

  - name: default
    merge_conditions:
      - "#approved-reviews-by>=2"
      - check-success=Travis CI - Pull Request

pull_request_rules:
  - name: move to urgent queue when 2 reviews and label urgent
    conditions:
      # A PR can be queued without the CI passing, but needs it to get merged
      - base=main
      - "#approved-reviews-by>=2"
      - label=urgent
    actions:
      queue:
        name: urgent

  - name: merge using the merge queue
    conditions:
      - base=main
      - "#approved-reviews-by>=2"
      - check-success=Travis CI - Pull Request
      - label!=urgent
    actions:
      queue:
        name: default

Conclusion

Implementing a merge queue in your development workflow can significantly improve the stability of your main branch.

This not only reduces the risk of conflicts and bugs but also controls CI spending.

Interesting in trying the most advanced merge queue for free? Check Mergify Merge Queue and ask for a demo.