Learn about the most used flow: GitOps

Meet Bob, a software developer from France. Being a senior developer and working in a startup, Bob develops, deploys, and maintains his application on his own.

Until the point when he came to know about GitOps, he maintained deployment pipelines which also had a lot of manual processes and requests to various internal service providers. Every 8 weeks, new changes were deployed, concluding a process that spanned months when the software development cycle was included. Something as simple as a few application code changes or modifications to configuration files was a large production. At the first sign of trouble, it was a requirement to roll back and conduct another 8 weeks test cycle.

The 8 weeks run-up consisted of deploying, testing, and retesting in each pre-production environment. If Bob changed the configuration of the firewall or required additional server resources, that required a minimum of 3 weeks leadup for each environment or at times longer due to problematic PR merge queues. No change could be initiated for production prior to testing in every lower environment and infrastructure changes across multiple environments could not be completed in a single 8 weeks cycle.

The pace changes were slowed to a crawl because of the fragile, unrepeatable, non-scalable approach that he followed. Bob was frustrated and tried because of the slow and erroneous development cycle. One day he met Jenny, who taught him about GitOps. She explained how companies are implementing  GitOps to get rid of all the problems listed above.

After speaking to Jenny for an hour, Bob was certain that GitOps was the solution to all his problems. If you don’t know “What is GitOps?” and “How GitOps act as a single source of truth?” you should read the conversation between them, where Bob asks and Jenny answers accordingly.

What is GitOps?

Jenny: GitOps is the methodology to manage infrastructure through code and automation, around a single Git repository. Broadly it is used to manage Kubernetes clusters and deliver applications, GitOps was introduced in 2017 by Weaveworks. Software agents can notify of any divergence between Git and what's running in a cluster with GitOps. If there's a difference, Kubernetes reconcilers will update or roll back the cluster based on the situation. Developers utilize familiar tools to create pull requests to accelerate and simplify application deployments and operations duties to Kubernetes when Git is at the heart of the delivery pipelines.

We only need to update the repository to deploy a new application or update an old one; the automated process will take care of the rest. It's like having cruise control when it comes to managing our production applications.

What are the underlying principles of GitOps?

Jenny: Broadly we will classify the principles into 4 segments.

1. Declarative Configuration:
   By leveraging a declarative format, you'll describe how your infrastructure works as well as the applications that are running on top of it. Doing this permits traceability of changes made to any environment and enables rollback, recoverability, and self-healing attributes using any source system. Move far away from imperative ad-hoc scripts to the declarative configuration in all levels (app and infra).
2. Version Controlled, Immutable Storage:
   Declarative descriptions stored within the repository support immutability, versioning, and versioning history, for instance, using Git for the declarations mentioned above, allows you one place from which everything for your application is derived and driven this enables you to easily pinpoint any changes made at any time. Don’t attempt to understand what is different between two environments by hand. Just examine all changes found in version control history, having the guarantee that the platform always matches what's described there.
3. Automatic Pull Operations:
   Using GitOps means you employ software agents that are always running in the cluster, automatically pulling the Git state at regular intervals and checking it against the live cluster state this manner you always know if the version in Git is the same as the live state or not.
4. Continuous Reconciliation:
   Operating during a closed loop ensures the desired state of a system matches the declared state this is often one of the most important features because it provides feedback allowing you and your team to better control your operations and workflow.
   When used together with your repository, GitOps software agents can perform various functions, ensuring self-healing. Agents automate fixes just in case there’s an outage, perform QA processes for your workflow, and protect against human error or manual intervention.

How is Git the Single Source of Truth?

Jenny: Do you remember the time you joined the startup? How was your onboarding experience?

Bob: Yes! The onboarding experience was so much of a headache. Dealing with multiple authoritative systems starting from firewalls, getting DNS access, a version control system, and finally, CI/CD tool not only took a lot of time but also consumed a lot of my energy. Moreover, the organization didn’t have great documentation to navigate across to know where to find the answer and praise it accordingly. But how GitOps play a role in enhancing the onboarding experience?

Jenny: Here comes the need for GitOps. GitOps allows developers to have their version control system as the authoritative to look for answers about every configuration they care about. The benefits of this start with streamlining onboarding, providing an enhanced developer experience, all the way up to avoiding the dreaded “context switching” that cuts into productive time.

The changes made to every aspect of the application live in one place and are versioned as each change is proposed by pull requests and implemented. With everything in code, any change can be reviewed and there is a full paper trail, and the ability to roll back changes.

What are the benefits of GitOps?

Jenny: GitOps extends DevOps by taking its best practices like version control, collaboration, and continuous deployment, and applying these to environment automation through application deployment, configuration, and infrastructure. GitOps allows developers and DevOps teams to version control the infrastructure and ship applications faster. The key benefits of GitOps are:

• Productivity improvements:  mean time to deployment is sped up using continuous deployment automation and an integrated feedback control loop. The team can ship 30-100 times more changes per day,  leading to a 2-3 fold increase in overall development production.
• Developer Experience Enhancement: Without knowing the internals of Kubernetes, developers may utilize familiar technologies like Git to manage upgrades and features to Kubernetes more quickly. Newly hired developers can come up and start working within days rather than months.
• Increased Reliability: You get stable and repeatable rollbacks with Git’s revert or rollback and fork capabilities. Because your entire system is defined in Git,  you'll recover from a meltdown using a single source of truth, cutting your mean time to recovery (MTTR) from hours to minutes.
• Standardization and Consistency:  you've got consistent end-to-end processes throughout your business because GitOps provides a single model for making infrastructure, app, and Kubernetes add-on modifications. Pull requests entirely drive your continuous integration and continuous deployment pipelines, and your operations chores are reproducible because of  Git.
• Easier Compliance and Auditing: Compliance and auditing are simplified because modifications are tracked and logged securely.  You'll also compare a trustworthy definition of the state of the cluster with the actual running cluster using comparison tools.
• Enhanced Security: Shifting security left to the GitOps operator. Security is additionally maintained as code. Traceability of who made/approved a change is automatically provided. Changes to the runtime state can only be applied by automated agents.  the necessity for external access to the system through tools like SSH, RDP, etc, are eliminated. Controls applied on the Git repository control who can make changes to which parts of the system.

Bob: That’s a lot of advantages for using GitOps. I am a bit confused about how is GitOps different from DevOps. Can you tell me more about it?

Difference between GitOps and DevOps?

Jenny: DevOps is certainly a well-established pattern for delivering cloud-native applications. DevOps generally takes a push approach where infrastructure updates are pushed into the runtime environment this needs good knowledge and understanding of the various environments and the current state of these environments during a GitOps model, we use the pull approach where the GitOps operator pulls changes supported any updates to the desired state. The GitOps operator is liable for resolving the state changes and applying them to the runtime environment.

In a DevOps model, the appliance pipelines and the deployment pipelines are disconnected and exclusive. The pipeline to create the runtime environment generally consists of IaC (Infrastructure as Code) scripts which are run once for a static one-time build of the environment the appliance pipelines are run multiple times to build and deploy the application to this static runtime environment within the GitOps model,  the specified runtime environment is described in the Git repository. Teams make changes to the specified state by creating Git commits. The updates to the specified state are then rolled into the runtime environment by agents which pull in these changes and deploy them automatically.

In the DevOps methodology, we use Terraform, Ansible, Helm, Kubectl, and other tools to script out the environment. In GitOps we use an autonomous agent, Operator to perform tasks like create, delete or update the environment supported by the declarative manifest in Git. The operator is liable for reconciling the environment definition in Git with the current state of the environment. We don't run Kubectl apply or Terraform apply directly on a runtime environment in GitOps.

What are GitOps Operators?

Jenny: There are many GitOps operators out there, but the two most discussed in the town are Flux and Argo CD. The Flux and Argo CD GitOps operators are designed to work with Kubernetes. However, GitOps is not limited to Kubernetes operators alone. Kubestack is a Terraform-based GitOps framework.

Why is the GitOps shift from Push to Pull Model Significant?

Jenny: In the still dominant push approach incumbent to DevOps, an external system (usually a CD pipeline tool) initiates deployment of changes to the cluster after a Git repository has been committed to or earlier CI pipeline successfully executed. The pipeline system must have access to the cluster.

In GitOps, CI/CD pipelines enact changes to a system when changes are made by pull request to Git, this is often automated by the observer agent within the cluster comparing its live state to that of the repository. When it diverges, an alert is issued, and if confirmed the repository phase change is pulled into the cluster.

Why is that important? A push pipeline means code starts within the  CI system and a flow of encoded scripts pushes changes to the K8s cluster. It’s possible to secure these CI scripts but that also means working outside of the cluster’s trust domain.

A pull pipeline involves a Kubernetes Operator,  like  Flux, deploying new images from inside the cluster. It spots any new images in Git and pulls them into the cluster state. This avoids cluster credentials being exposed outside of the assembly environment.

GitOps doesn’t mean not employing a  CI tool or automating testing or even deploying straight from development to production you're simply informed when the observable state of the cluster differs from that of the source of truth Git repository. How the pull deployments are then actioned is totally up to your policy:  deployed to development, staging, or QA.

Bob: I am convinced about why to use GitOps. But, Jenny when I was working in my previous job with a big team, I faced a huge problem which made me change my job. Can you help me with that?

Jenny: Sure thing! Tell me.

Bob: Combining the individual work of an enormous team of engineers into one codebase was tough. And the difficulty lies deeper than our standard merge conflicts, which version control is fairly good at handling. There’s a way more troublesome type of conflict we ran into when simultaneous changes are made to multiple areas of code that depend upon one another. Is there any way we can avoid Merge Queues with the help of GitOps?

Jenny: Absolutely! Organizations such as Uber, AWS, Mozilla, Airtable, Netflix, and a lot more have found a tool and using it extensively lately. Mergify provides a merge queue feature that is easy to set up and configure to our needs. It's free-forever for open-source projects and comes with a 14-day free trial for private projects.

Mergify not only helps solves problems with merge queues but has many fascinating features which can nowhere be found on GitHub. I think you should give it a try! Know about handling 780k GitHub events per day from here

Bob: Thanks a lot Jenny for telling me about GitOps and mentioning Mergify!