Automate Continuous Integration From Code to Deployment with Argo Events + Argo Workflows + Argo CD

One of the main benefits of CI is that it helps to identify and fix errors quickly. When code changes are regularly merged into the main branch, this triggers a build and deployment to your cluster, which it is easier to detect errors early on, rather than having to sift through a large change to locate and fix the problem.

In this article, I will walk through how to set up a basic code-to-deployment automation with Argo Events, Argo Workflows, and Argo CD.

Note: This article assumes that you have a Kubernetes cluster running. Since setting up an automated CI/CD is a complex setup depending on various factors, I will only be giving a high-level example walkthrough of the setup and it is not meant to be a quick copy-and-paste guide. You should refer to the official documentation.

Setting up Argo Workflows + Argo Events on your Kubernetes cluster

In order to automate deployment builds, we will be using Argo Workflows to carry out container image build jobs on Kubernetes. The good thing about Argo Workflows is that it works natively with Argo Events, which allows us to trigger workflows based on events, aka. when a new commit is pushed to a Git repository.

1. Install Argo Workflows & Argo Events in your Kubernetes cluster

To install Argo Workflows on your Kubernetes cluster, you can use the following command:

kubectl create namespace argo
kubectl apply -n argo -f https://github.com/argoproj/argo-workflows/releases/download/v3.4.3/install.yaml

Next, install Argo Events:

kubectl create namespace argo-events
kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/stable/manifests/install.yaml
# Install with a validating admission controller
kubectl apply -f https://raw.githubusercontent.com/argoproj/argo-events/stable/manifests/install-validating-webhook.yaml

Next, we will need to deploy the eventbus for Argo Events:

kubectl apply -n argo-events -f https://raw.githubusercontent.com/argoproj/argo-events/stable/examples/eventbus/native.yaml

You should now have both Argo Workflows and Argo Events installed on your Kubernetes cluster.

2. Create a Cluster Workflow Template for Container Image Build

Next, we will need to create a Cluster Workflow Template for building our container image. For this, we will be using Kaniko, which is a tool for building container images inside a Kubernetes cluster.

This example template will be used by Argo Workflows to build your container image.

apiVersion: argoproj.io/v1alpha1
kind: ClusterWorkflowTemplate
metadata:
  name: build-container-image
spec:
  templates:
    - name: build-kaniko-git
      inputs:
        parameters:
          - name: app_repo
          - name: ref
          - name: dockerfile
          - name: container_image
          - name: container_tag
      container:
        image: gcr.io/kaniko-project/executor:debug
        args:
          - --context={{inputs.parameters.app_repo}}#{{inputs.parameters.ref}}
          - --dockerfile={{inputs.parameters.dockerfile}}
          - --destination={{inputs.parameters.container_image}}:{{inputs.parameters.container_tag}}
          - --cache=true

        # If you are using a private container registry, you can specify the volume mount here for your repository secret
        # https://github.com/GoogleContainerTools/kaniko#pushing-to-different-registries
        # volumeMounts:
        #   - name: aws-secret
        #     mountPath: /root/.aws/

        # If you are using a private git repository, you can pass in the repository secret as environment variables
        # https://github.com/GoogleContainerTools/kaniko#using-private-git-repository
        # env:
        #   - name: GIT_TOKEN
        #     valueFrom:
        #       secretKeyRef:
        #         name: github-access
        #         key: token

        resources:
          requests:
            cpu: 1.8
            memory: 3Gi
          limits:
            cpu: 2
            memory: 6Gi
      retryStrategy:
        limit: '3'
        retryPolicy: Always

Go ahead and configure the template above and deploy it to your Kubernetes cluster:

kubectl apply -f build-container-image.yaml

3. Create a sensor to trigger the workflow

Next, we will need to create an event source for Argo Events to listen for events from our Git repository.

Set up a webhook on your git repository to send events to Argo Events, instructions here.

In this example, we will be using the GitHub event source.

# Info on GitHub Webhook: https://developer.github.com/v3/repos/hooks/#create-a-hook
apiVersion: argoproj.io/v1alpha1
kind: EventSource
metadata:
  name: github
spec:
  replicas: 1
  service:
    ports:
      - name: github-event
        port: 12000
        targetPort: 12000
  github:
    main:
      repositories:
        - owner: your_github_username_or_org
          names:
            - app_repo_name
      # Github will send events to following port and endpoint
      webhook:
        # endpoint to listen to events on
        endpoint: /push
        # port to run internal HTTP server on
        port: '12000'
        # HTTP request method to allow. In this case, only POST requests are accepted
        method: POST
        # url the event-source will use to register at Github.
        # This url must be reachable from outside the cluster.
        # The name for the service is in `<event-source-name>-eventsource-svc` format.
        # You will need to create an Ingress or Openshift Route for the event-source service so that it can be reached from GitHub.
        url: https://domain-name.com
      # type of events to listen to.
      # following listens to everything, hence *
      # You can find more info on https://developer.github.com/v3/activity/events/types/
      events:
        - '*'

      # apiToken refers to K8s secret that stores the github api token
      # if apiToken is provided controller will create webhook on GitHub repo
      # +optional
      apiToken:
        # Name of the K8s secret that contains the access token
        name: github-access
        # Key within the K8s secret whose corresponding value (must be base64 encoded) is access token
        key: token

      # webhookSecret refers to K8s secret that stores the github hook secret
      # +optional
      webhookSecret:
        # Name of the K8s secret that contains the hook secret
        name: github-access
        # Key within the K8s secret whose corresponding value (must be base64 encoded) is hook secret
        key: secret

      # type of the connection between event-source and Github.
      # You should set it to false to avoid man-in-the-middle and other attacks.
      insecure: false
      # Determines if notifications are sent when the webhook is triggered
      active: true
      # The media type used to serialize the payloads
      contentType: json

Next, we will need to create a sensor to trigger the workflow when a new commit is pushed to a Git repository. The workflow will push the newly built image with the tag 'latest' which will be used for the automated deployment later – you may change this tag to whatever you want.

apiVersion: argoproj.io/v1alpha1
kind: Sensor
metadata:
  name: trigger-build-container-image
spec:
  replicas: 1
  template:
    serviceAccountName: operate-workflow-sa
  dependencies:
    - name: github-eventsource
      eventSourceName: github
      eventName: main
      filters:
        data:
          # Name of the event that triggered the delivery: [pull_request, push, yadayadayada]
          # https://docs.github.com/en/developers/webhooks-and-events/webhook-events-and-payloads
          - path: headers.X-Github-Event
            type: string
            value:
              - push
          - path: body.repository.name
            type: string
            comparator: '='
            value:
              # specify the respository name, i.e.
              - (^myapp$)
          - path: body.ref
            type: string
            comparator: '='
            value:
              # specify the branch you want to trigger the workflow, i.e.
              - (^refs\/heads\/branch_name$)
  triggers:
    - template:
        name: build-my-app
        k8s:
          operation: create
          source:
            resource:
              apiVersion: argoproj.io/v1alpha1
              kind: Workflow
              metadata:
                generateName: ci-build-my-app-
                labels:
                  workflows.argoproj.io/archive-strategy: 'false'
              spec:
                entrypoint: build
                podGC:
                  strategy: OnPodCompletion
                volumes:
                  - name: github-access
                    secret:
                      secretName: github-access
                # specify the appropriate secret required to push image
                #   - name: aws-secret
                #     secret:
                #       secretName: aws-test-secret
                #       items:
                #         - key: credentials
                #           path: credentials
                arguments:
                  parameters:
                    - name: repo_name
                    - name: ref
                    - name: tag
                      value: latest
                templates:
                  - name: build
                    inputs:
                      parameters:
                        - name: repo_name
                        - name: ref
                        - name: tag
                    dag:
                      tasks:
                        - name: build-my-app
                          templateRef:
                            name: build-container-image
                            template: build-kaniko-git
                            clusterScope: true
                          arguments:
                            parameters:
                              - name: app_repo
                                value: 'git://github.com/<username_or_org>/{{inputs.parameters.repo_name}}'
                              - name: dockerfile
                                value: Dockerfile
                              - name: container_image
                                value: 'link.com/to/your/container/registry/my-app'
                              - name: ref
                                value: '{{inputs.parameters.ref}}'
                              - name: container_tag
                                value: '{{inputs.parameters.tag}}'
          parameters:
            - src:
                dependencyName: github-eventsource
                dataKey: body.repository.name
              dest: spec.arguments.parameters.0.value
            - src:
                dependencyName: github-eventsource
                dataKey: body.ref
              dest: spec.arguments.parameters.1.value
      retryStrategy:
        steps: 3

Once you have configured your event source and sensor, go ahead and deploy to your Kubernetes cluster:

kubectl apply -f event-source.yaml
kubectl apply -f sensor.yaml

4. Test the workflow

If everything is set up correctly, you should be able to see the container image build workflow triggered when you push a new commit to your repository.

If you have Argo CLI installed on your machine, you can inspect the workflow by running:

argo list -n argo-events

To check the status of the workflow, run:

argo get @latest -n argo-events

Automate Deployment with Argo CD

Now that we have a triggered workflow to build our container image, we can use Argo CD to automate the deployment of our application. For this, we will also be using Argo CD Image Updater to simplify the deployment of newly built container image.

1. Install Argo CD

kubectl create namespace argocd
kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml

You may also want to configure Argo CD acces to your private git repository, instruction here.

2. Install Argo CD Image Updater

kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj-labs/argocd-image-updater/stable/manifests/install.yaml

3. Create Argo CD Application

To create an Argo CD application, we will need to create a Kubernetes manifest file that contains the application definition. In this application manifest, we will also specify annotations that will be discovered by Argo CD Image Updater to detect and update the newly built container image.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: my-app
  annotations:
    argocd-image-updater.argoproj.io/write-back-method: git
    argocd-image-updater.argoproj.io/image-list: >-
      link.com/to/your/registry/image:latest
    argocd-image-updater.argoproj.io/update-strategy: digest
spec:
  destination:
    namespace: default
    server: 'https://kubernetes.default.svc'
  source:
    path: app/to/deployment/manifest.yaml
    repoURL: 'git@github.com:owner/repository_name.git'
    targetRevision: main
  project: default
  syncPolicy:
    # Automatically deploy application when a new image is available
    automated:
      selfheal: true

Once you have configured your application manifest, go ahead and deploy it to your Kubernetes cluster:

kubectl apply -f app.yaml

Test it out

Now that we have everything set up, we can test it out by pushing a new commit to your repository.

Once the commit is pushed, you should be able to see the container image build workflow triggered. Once the workflow is completed, you should be able to see Argo CD Image Updater was able to pick up the latest change and new container image deployed to your cluster automatically.

Tags: GitOpsDevOpsKubernetes