A software engineering team needs to maintain velocity to deliver continuously. They need a faster feedback loop to iterate quickly. Continuous Integration (CI), and Continuous Deployment (CD) is the mechanism to achieve.
If we think about the traditional CD pipeline and it’s challenges, some of them are:
- Sharing credentials to the build tool. Example – Jenkins needs access to Kubeconfig for a Kubernetes cluster.
- Effort to integrate post deployment monitoring. Example – After deploying the pods on the kubernetes cluster, test if service is reachable.
- Cross environment access. Example – Jenkins infrastructure needs connectivity to the deployment environment.
- Treating configuration as code. Example – Managing runtime configuration for different environments (dev/staging/production).
Gitops is an alternative to the traditional CD pipeline and solves the above problems. This blog will start with idea of Gitops, its principles, discuss the challenges with CD pipeline, the need, why or why not an engineering team should adopt Gitops practise. We will also discuss how we adopted Gitops for one of our client’s project.
What is Gitops?
Gitops is the process to deploy, configure, monitor, update and manage infrastructure as a code. Git in Gitops refers to a deployment repository, which is a single source of truth.
Weaveworks (the company which coined the term Gitops) describes it as:
“GitOps is a way to do Kubernetes cluster management and application delivery. GitOps works by using Git as a single source of truth for declarative infrastructure and applications. With GitOps, the use of software agents can alert on any divergence between Git with what’s running in a cluster, and if there’s a difference, Kubernetes automatically updates or rollback the cluster depending on the case. With Git at the center of your delivery pipelines, developers use familiar tools to make pull requests to accelerate and simplify both application deployments and operations tasks to Kubernetes.”
In other words, Gitops is the ability to sync the repository state and reflect the changes in deployment environment automatically. It is derivative of the operation model which focuses on Pull Requests/Merge Requests. Approved changes that are desired get deployed automatically. All these changes are observable in the git history.
To understand better what Gitops is, let’s ask a question and try to answer them.
Q: What do we have in the deployment repository?
A: Deployment repository contains infrastructure code and runtime configurations to perform deployments on an environment. It can have kubernetes manifests and helm-charts.
Gitops as concept was introduced with Kubernetes ecosystem.But other tools like ansible_tower and terraform with gitlab have adopted. We will limit ourselves to Kubernetes and Weaveworks Flux
The principles of Gitops:
- Use declarative configuration as code. Example – Kubernetes manifests as YAML files.
- Desired state of a deployment environment is versioned in the Git repository. Applying rollbacks becomes easier.
- Software agents reconcile the state and send alerts for any divergence.
Challenges and Solutions
We discussed earlier, about the challenges of the traditional CD pipeline. Let us discuss in details, how Gitops solve these problems. This should also answer “Why Gitops is useful”.
Security credentials shared across the boundaries.
Any CI/CD tools need access to the Kubernetes cluster API to deploy, which implies sharing credentials across boundaries. If we see in the diagram, the credentials boundaries are crossed where the CI system has access to Kubernetes Cluster.
We worked with a fintech client, where the security team did not encourage sharing credentials across environments. Hence, a pull-based pipeline was useful. Devops team need to make changes to the configuration repository. Kubernetes operators (discussed below) look for changes and apply on the cluster. If you look at the diagram, the Kubernetes cluster has access to the Github repo and docker registry. No need to share kubeconfig outside of the environment.
Monitoring systems after the deployment.
With a CD pipeline, we can deploy the changes but monitoring the deployment remains a challenge. Example: Jenkins does not have first class support for Kubernetes deployment. We need to write custom scripts to validate if a deployment was successful on the K8S cluster.
Gitops would solve the problem. It also provides retries and alert mechanisms to take actions.
Cross environment access.
Generally, CI/CD systems reside in different infrastructure to deployment environments, and the challenge is network reachability. One has to peer VPCs or establish site to site VPNs. This can become overwhelming at times and require expertise in the team. With Gitops, the software agents requires a read-only Git token, which pull the code from git repository and apply the changes.
Treating Configuration as Code
From our experience, managing configuration across different environments is a complicated problem. In the past, we have used Consul as key/value(KV) store for storing configuration. But running a consul cluster for KV store can be overkill. Periodic backups of Consul data is another overhead.
Gitops as a solution fulfils the following characteristic for configurations:
- Unique to an environment – Maintain all the configurations for different environments in one repository.
- Reproducible – Everything is part of source control. Single source of truth allows reproducing from scratch in case required.
- Versioning – Using VCS allows going back in time, looking for change, restoring and reverting when required.
Other benefits of Gitops
Hope, by now we understand why Gitops is useful. Some other benefits are:
- Treating infrastructure as code provides a repeatable process. It now becomes easier to replicate the changes across multiple environments. Rolling back to previous versions is easier.
- Reduce human errors because no manual steps are involved.
- Operate on git knowledge, no need for deep understanding of underneath tools to deploy. This is easier for developers to deploy by themselves.
FluxCD
Gitops is the process, and we used Flux CD to implement the strategy. Other tools to implement Gitops are Tekton, ArgoCD and Jenkins X, to name a few. Here, we will discuss FluxCD only.
Flux installs certain Controllers. Listing down these controllers with their functionality:
- Kustomize Controller – It is used to reconcile the state from the Git repository and reports any drift in the state to the notification controller. More details.
- Helm Controller – It is used to watch HelmRelease and manage HelmChats. More details.
- Notification Controller – It is responsible for sending notifications to Slack channels. More details.
How we implemented GitOps?
- Earlier, for all the deployments someone from the devops team had to deploy the application. This caused dependency and delay in testing the code changes done by developers. We could extend our Jenkins pipeline towards CD, but updating secrets and configmaps were still a challenge. Gitops made more sense for this use-case.
- One use case was to update the docker image tag and deploy the changes. After the CI pipeline run, developers had to update the docker image tag in deployment repository and push the changes. Open a PR and on merge to appropriate branch, changes to repository would reflect the changes to the K8S cluster. Developers can manage the whole process by themselves, without knowing the details of Kubernetes.
- We used helm charts. With Helm Controller, reusability of the charts was possible and avoided writing K8S manifests again.
- We maintained configuration for different environments in a single repository. Flux supports path-based lookup. Hence, we created a folder structure for dev, QA and production.
k8s/gitops
└── k8s/gitops/dev /* 1. Gitops configuration */
├── k8s/gitops/dev/kustomization.yaml
├── k8s/gitops/dev/kustomizeconfig.yaml
├── k8s/gitops/dev/service1 /* 2. Config map, secrets for service1 */
│ ├── k8s/gitops/dev/service1/configmap.yaml
│ ├── k8s/gitops/dev/service1/kustomization.yaml
│ ├── k8s/gitops/dev/service1/release.yaml
│ └── k8s/gitops/dev/service1/secrets.yaml
├── k8s/gitops/dev/service2
├── k8s/gitops/dev/service2/configmap.yaml
├── k8s/gitops/dev/service2/kustomization.yaml
├── k8s/gitops/dev/service2/release.yaml
└── k8s/gitops/dev/service2/secrets.yaml
--- similar directory structure for QA environment ---
--- similar directory structure for Production environment ---
k8s/helm-charts
└── k8s/helm-charts
├── k8s/helm-charts/service1 /* 3. Helm chart for service1 */
│ ├── k8s/helm-charts/service1/Chart.yaml
│ ├── k8s/helm-charts/service1/templates
│ └── k8s/helm-charts/service1/values.yaml
└── k8s/helm-charts/service2
├── k8s/helm-charts/service2/Chart.yaml
└── k8s/helm-charts/service2/templates
└── k8s/helm-charts/service2/values.yaml
k8s/helm-values
├── k8s/helm-values/service1 /* 4. Helm values for service1 */
│ ├── k8s/helm-values/service1/dev-config.yaml
│ ├── k8s/helm-values/service1/prod-config.yaml
│ └── k8s/helm-values/service1/qa-config.yaml
└── k8s/helm-values/service2
├── k8s/helm-values/service2/dev-config.yaml
├── k8s/helm-values/service2/prod-config.yaml
└── k8s/helm-values/service2/qa-config.yam- Storing sensitive information in plain-text is always discouraged. SOPS (secure operations) helps to keep secrets in an encrypted manner. Flux supports SOPS (Secure Operations). The configmap and secrets for particular service are under
k8s/gitops/<environment>/<service>directory. We use SOPS with asymmetric key pair. Use a public key for encrypting configmaps and secrets. Configure Flux with a private key to decrypt.
GitOps Workflow
Challenges applying Gitops
We talked about all the Jazz of Gitops. Not necessary, it fits the bill for engineering teams and are ready to adopt it. The reason could be the lack of essential devops practises like a test-release-deploy pipeline. Other challenges to adopt Gitops are:
- Git by default is not designed for programmatic updates; multiple CI pipelines can cause conflicts. This can be problem for updating docker tags in deployment repositories via CI pipeline.
- Maintaining multiple repositories for different clusters can have maintenance overheads. One can keep fewer git repositories, but it poses the challenge of programmatic updates to cause conflicts.
- SOPS solves the problem of keep secrets in plain text but causes inconvenience for PR reviews. A reviewer has to pull the branch and check on their local system.
Summary
- Gitops is a process to store the desired state of infrastructure committed in a Git repo.
- Gitops provides benefits of Infrastructure code versioning, rollback, security backups.
- Gitops solves the configuration problem by treating it as code.
- Challenges in gitops like programmatic updates, multiple repository management, PR reviews with SOPS.
