Create GitOps CI/CD pipeline to Kubernetes(Amazon EKS) using ArgoCD and Kustomize

What is GitOps?

These days, there are various tools and methods for continuous delivery to Kubernetes.

You can create a consistent CI/CD pipeline to Kubernetes using CircleCI.

On the other hand, we are seeing more and more cases of CircleCI being combined with OSS tools such as FluxCD and ArgoCD.

These tools are also called GitOps tools.

GitOps is a CD methodology introduced by Weaveworks in 2017 in their blog.

GitOps uses Git to declaratively describe and manage Kubernetes resource configuration information.

The benefits of GitOps are as follows:

Improving productivity

In GitOps, you can manage Kubernetes resource configuration by Git operations(commit, approve, merge, etc)

Therefore, the same change process can be applied for common application development.

You can also git revert, create a Pull Request, and merge it.

It also eliminates the need to manually run kubectl commands, etc.

Improving visibility

Because in GitOps, Git is the "Single Source of Truth".

Therefore, you can easily determine who did what based on the commit history.

Push GitOps / Pull GitOps

GitOps can be achieved, for example, by putting manifests in Git and applying changes to Kubernetes via CircleCI.

This type of GitOps is called "Push" GitOps.

For "Push" GitOps, you need to pass credentials to the tool to manipulate the Kubernetes API.

On the other hand, FluxCD and ArgoCD are called "Pull" GitOps.

"Pull" GitOps tools periodically poll Git to detect differences and apply changes.

The advantage of this "Pull" GitOps tool is that you do not need to pass credentials to manipulate the Kubernetes API.

Therefore, if you do not want to pass the credentials to manipulate the Kubernetes API to CI/CD tools, we recommend using "Pull" GitOps tools.

In such a case, to realize a CI/CD pipeline, CI and CD should be divided as follows:

CI(CircleCI)

• Run automated tests(unit tests, integration tests, E2E/UI tests), static analysis, vulnerability diagnosis, etc.
• Build docker image, push to container registry(Amazon ECR, etc)
• Create a Pull Request to the manifest repository to change the Docker image tag

CD(FluxCD, ArgoCD)

• (Approve & merge the above Pull Request)
• Detect Git diff by Git polling
• Apply these changes to the Kubernetes cluster

This time will use CircleCI as a CI tool, and ArgoCD as a CD tool, to achieve CI/CD pipeline to Kubernetes.

Install ArgoCD and apply manifest files

This time, we will use ArgoCD as a GitOps tool.

ArgoCD is unique compared to other GitOps tools in that it has a GUI.

The GUI allows users to configure settings, check status, and perform a manual synchronization.

The demo site is also available.

To achieve continuous delivery with ArgoCD, first, install ArgoCD on a Kubernetes cluster.

Please refer to the official documentation for installation procedures.

Here is the repository that will be deployed(it uses Kustomize).

GitHub - tadashi0713/circleci-demo-eks-argocd-manifest

Once the Application is registered and Sync is executed, it will be deployed and updated based on the manifest in the repository.

You can sync manually, or you can set the Sync Policy to Automatic to automatically deploy and update changes in Git.

Build a CI/CD pipeline in combination with CircleCI

The previous step alone only deploys the Kubernetes manifest but does not complete the CI/CD pipeline.

The next step is to create the CI/CD pipeline from application build to deployment to Kubernetes.

The application code repository and configuration file can be found here.

• GitHub - tadashi0713/circleci-demo-eks-argocd-app
• Config file

Build Docker image & push the image to Amazon ECR

The build_and_push_image job builds the application (Next.js) in Docker and pushes it to Amazon ECR.

build_push_ecr:
  machine:
    docker_layer_caching: true
    resource_class: large
    image: ubuntu-2004:current
  steps:
    - checkout
    - aws-ecr/build-and-push-image:
        assume-web-identity: true
        role-arn: arn:aws:iam::660990364978:role/tadashi_oidc_fargate
        repo: tadashi-eks-argocd-demo
        tag: '${CIRCLE_SHA1:0:7}'

There are CircleCI's features to optimize this pipeline.

• Customizeable compute built(Resource Classes) to build Docker images faster.
• Docker Layer Caching to build Docker images faster.
• CircleCI Orb(Amazon ECR) to simplify config file / setup pipeline easily.
• OpenID Connect to create CI/CD pipeline to Cloud Providers more securely

CitcleCI provides Orbs not just Amazon ECR, but also GCR(Google Container Registry) and ACR(Azure Container Registry).

Create a Pull Request to apply changes

After the build_and_push_image job completes, the create_release_pr job runs next.

The create_release_pr job changes the image tag of the Kubernetes manifest to the new tag for this release and creates a Pull Request on GitHub.

First, it is needed to clone the manifest repository.

Therefore, instead of the usual checkout command, we will use the GitHub CLI Orb to clone the repository.

steps:
  - github-cli/setup
  - github-cli/clone:
      repo: git@github.com:tadashi0713/circleci-demo-eks-argocd-manifest.git

Beforehand, you need to set the GitHub Token ($GITHUB_TOKEN) as an environment variable in CircleCI.

Then, change the target image tag.

Since we use Kustomize this time, we can use kustomize edit set image command.

- run:
    name: Change image tag
    working_directory: base
    command: |
      curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash
      ./kustomize edit set image circleci-demo-eks-argocd-app=${ECR_REPO}:${CIRCLE_SHA1:0:7}

Next, we will create a new branch, and git push.

- add_ssh_keys:
    fingerprints:
      - $SSH_KEY_FINGERPRINT
- run:
    name: Commit file
    command: |
      git config --global user.email "gitops@example.com"
      git config --global user.name "GitOps Bot"
      git remote set-url --push origin git@github.com:tadashi0713/circleci-demo-eks-argocd-manifest.git
      git checkout -b release-${CIRCLE_SHA1:0:7}
      git add .
      git commit -m"Release circleci-demo-eks-argocd-app ${CIRCLE_SHA1:0:7}"
      git push origin HEAD

In CircleCI, the SSH key generated and registered by default is read-only for the target repository.

Therefore, the following two settings must be made to git push to the manifest repository.

• Use GitHub user key instead to check out additional private repositories

• Add add_ssh_key step

Finally, use the gh(github-cli) command to create the Pull Request to the repository where the manifest is located.

Since github-cli authentication has already been done in the previous step, there is no need to perform additional authentication.

- run:
    name: Create a Pull Request
    command: gh pr create -t "Release circleci-demo-eks-argocd-app ${CIRCLE_SHA1:0:7}" -b ""

If the job succeeds, you will see the Pull Request created in the manifest repository.

After you merge this Pull Request, ArgoCD will apply these changes.

Summary

This time, I will introduce how to create GitOps CI/CD pipeline to Kubernetes(Amazon EKS) using ArgoCD, Kustomize, and CircleCI.

I hope this article will be helpful for those who have been considering how to implement CI/CD pipeline to Kubernetes.

In this article, I introduced several CircleCI features to improve CI(Continuous Integration)

• Customizeable compute built(Resource Classes)
• Docker Layer Caching
• CircleCI Orb(Amazon ECR)
• OpenID Connect

Therefore, I believe that combined with CircleCI and ArgoCD, you can develop quality Kubernetes applications continuously.