↓Skip to main content
Stonegarden
  1. Articles/

Argo CD Kustomize with Helm

··2155 words·11 mins·
Vegard S. Hagen
Author
Vegard S. Hagen
Pondering post-physicists
Table of Contents

I use Argo CD to maintain my Homelab as I find it intuitive. The nice GUI also helps me to quickly inspect problems which might occur when I try something fancy. Another widely used alternative is Flux CD which solves the same problem of GitOps-ing your cluster, but I have little experience with it yet.

In this article I’m going to try and explain how I use ArgoCD with Kustomized Helm to maintain my Homelab using GitOps-practices.

Kustomize + Helm = ❤️
#

Update 2024-02-13: I’ve switched to using the community maintained Helm chart for Argo CD which solves the Kustomize+Helm plugin in an imho much more elegant way. I’ve posted the updated config in the Helm chart section.

Sometimes a Helm chart doesn’t have everything you need nicely templated, or you want to reference a Helm chart in your kustomization.yaml file to have everything nice and neat together.

Out of the box ArgoCD comes with support for both Kustomize and Helm, but not both at the same time. You could fully render the Helm template and start manually editing it before checking the tailored manifest into Git, but I find that to be a not-so-elegant solution which imho would also be harder to maintain.

A better way would be to combine the power of both Kustomize and Helm to get your way.

Enable Kustomize Helm Integration Globally
#

If you have full control over the cluster — and know that you want to always enable Helm support in Kustomize, you can simply enable Helm support in Kustomize by default by adding

kustomize.buildOptions: "--enable-helm"

to the argocd-cm ConfigMap This can be done by either using a kustomize patch

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  kustomize.buildOptions: "--enable-helm"

or adding

configs:
  cm:
    kustomize.buildOptions: "--enable-helm"

to the Argo CD Helm values.

You might also consider adding --load-restrictor=LoadRestrictionsNone to kustomize.buildOptions to allow adding Helm-values outside the kustomize root directory depending on your setup.

Enabling Helm integration for every Argo CD Application comes with some overhead, if you instead want to only selectively enable Helm support, you have to create an Argo CD ConfigurationManagementPlugin.

The Old Ways
#

This is deprecated. Read on for a better solution that works with Argo CD v2.4+

Pre Argo CD v2.8, a way to achieve Kustomized Helm was to patch the argocd-cm ConfigMap

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  configManagementPlugins: |
    - name: kustomize-build-with-helm
      generate:
        command: [ "sh", "-c" ]
        args: [ "kustomize build --enable-helm" ]

using Kustomize

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

patches:
  - path: patches/argocd-cm-kustomize-helm-patch.yaml

which I was previously doing (commit) since I didn’t get the memo that this has been deprecated since Argo CD v2.4.

As a meager excuse on my part, a similar solution is explained in the official Argo CD example apps repo which led me to believe this was the way to do it.

Config Management Plugin
#

My first attempt (commit) was simply following the Argo CD documentation on Config Management Plugins (CMP) which boiled down to a ConfigMap with the CMP-config (CMP follows the Kubernetes-style spec convention, but is not a CRD)

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm-cmp-kustomize-build-with-helm
data:
  plugin.yaml: |
    apiVersion: argoproj.io/v1alpha1
    kind: ConfigManagementPlugin
    metadata:
      name: kustomize-build-with-helm
    spec:
      generate:
        command: [ "sh", "-c" ]
        args: [ "kustomize build --enable-helm" ]

and a patch to the argocd-repo-server Deployment to start a CMP-sidecar with the following configuration

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-repo-server
spec:
  template:
    spec:
      containers:
        - name: kustomize-build-with-helm
          command: [ /var/run/argocd/argocd-cmp-server ]
          image: quay.io/argoproj/argocd:v2.7.11
          securityContext:
            runAsNonRoot: true
            runAsUser: 999
          volumeMounts:
            - name: var-files
              mountPath: /var/run/argocd
            - name: plugins
              mountPath: /home/argocd/cmp-server/plugins
            - name: argocd-plugin-config
              mountPath: /home/argocd/cmp-server/config/plugin.yaml
              subPath: plugin.yaml
            - mountPath: /tmp
              name: cmp-tmp
      volumes:
        - name: argocd-plugin-config
          configMap:
            name: argocd-cm-cmp-kustomize-build-with-helm
        - name: cmp-tmp
          emptyDir: { }

which works, and made it possible to upgrade to Argo CD v2.8 without anything failing.

But we can do better!

Digging deeper 🐰🕳️
#

While the previous quick fix works, it has the annoying downside that we need to keep the patched Argo CD image tag in sync with the main image. The goal of this section is to create a solution that requires less hands-on maintenance while also being fairly minimal.

This is going to be kind of a ramble where I try to explain my process and reasoning, so if you want to jump to the conclusion take a look at the resulting commit.

You might’ve noticed that in the patch we’re referencing volumes in the containers-section that doesn’t seem to be defined in the volumes-section, so what’s going on? Taking a step back the answer is already in the formulation of the question in that this is a patch, and we need the full manifest in order to figure out what’s really going on.

This article is written for Argo CD v2.8.2, so we’ll be basing it on that manifest available here1

Looking at the argocd-repo-server Deployment we see an initContainer which copies the argocd-binary to the mystery var-files volume. Here edited for brevity

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-repo-server
spec:
  ...
  template:
    ...
    spec:
      ...
      initContainers:
          - name: copyutil
            image: quay.io/argoproj/argocd:v2.8.2
            command:
              - /bin/cp
              - -n
              - /usr/local/bin/argocd
              - /var/run/argocd/argocd-cmp-server
            volumeMounts:
              - mountPath: /var/run/argocd
                name: var-files
            ...
      volumes:
        - emptyDir: { }
          name: var-files
        - emptyDir: { }
          name: plugins
        - ...

By itself this seems pointless, and I’d argue that it is if you’re running vanilla Argo CD. Without digging too deep I assume it’s done to make running sidecar plugins easier.

Digging even deeper and looking inside the argocd-repo-server container we see the argocd-binary being soft linked to a several times, all with names hinting at different parts of Argo CD.

argocd@argocd-repo-server-6589ffb4b6-4rdvp:/usr/local/bin$ ls -lah
total 205M
drwxr-xr-x 1 root root 4.0K Aug 24 20:34 .
drwxr-xr-x 1 root root 4.0K Jun 24 02:02 ..
-rwxr-xr-x 1 root root 142M Aug 24 20:34 argocd
lrwxrwxrwx 1 root root   21 Aug 24 20:34 argocd-application-controller -> /usr/local/bin/argocd
lrwxrwxrwx 1 root root   21 Aug 24 20:34 argocd-applicationset-controller -> /usr/local/bin/argocd
lrwxrwxrwx 1 root root   21 Aug 24 20:34 argocd-cmp-server -> /usr/local/bin/argocd
lrwxrwxrwx 1 root root   21 Aug 24 20:34 argocd-dex -> /usr/local/bin/argocd
lrwxrwxrwx 1 root root   21 Aug 24 20:34 argocd-k8s-auth -> /usr/local/bin/argocd
lrwxrwxrwx 1 root root   21 Aug 24 20:34 argocd-notifications -> /usr/local/bin/argocd
lrwxrwxrwx 1 root root   21 Aug 24 20:34 argocd-repo-server -> /usr/local/bin/argocd
lrwxrwxrwx 1 root root   21 Aug 24 20:34 argocd-server -> /usr/local/bin/argocd
-rwxr-xr-x 1 root root  203 Aug 24 20:05 entrypoint.sh
-rwxr-xr-x 1 root root  934 Aug 24 20:05 git-verify-wrapper.sh
-rwxr-xr-x 1 root root  215 Aug 24 20:05 gpg-wrapper.sh
-rwxr-xr-x 1 root root  49M Aug 24 20:08 helm
-rwxr-xr-x 1 root root  15M Aug 24 20:08 kustomize
lrwxrwxrwx 1 root root   28 Aug 24 20:08 uid_entrypoint.sh -> /usr/local/bin/entrypoint.sh

Although intriguing I feel we’re digressing, and I’ll put this side quest on hold. (Though please contact me if you want to enlighten me why it’s done this way!)

Argo suggests that you create your own image with all the tools you need, and that would be certainly be a better solution than the upcoming hack for an enterprise environment, but this is my Homelab!

Knowing that we need both the helm and kustomize binaries and assuming that they’re both standalone, why not try to copy them to a minimal base image! Next I hijacked the copyutil-initContainer (and implicitly the var-files volume) with a patch to do just this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-repo-server
spec:
  template:
    spec:
      initContainers:
        - name: copyutil
          command: [ /bin/bash ]
          args:
            - -c
            - >-
              /bin/cp -n /usr/local/bin/argocd /var/run/argocd/argocd-cmp-server &&
              /bin/cp -n /usr/local/bin/kustomize /var/run/argocd/kustomize &&
              /bin/cp -n /usr/local/bin/helm /var/run/argocd/helm

I then tried using a busybox-image inspired by the Argo CD CMP example, but quickly ran into an issue with missing git. My first idea was then to try and just copy the git binary as well, but that yielded an error with a missing libpcre2-8.so.0 library.

Ok, so that only kinda halfway worked. We could try and hunt down every library git might need, but at this point I’d argue it’s easier to just install git (along with helm and kustomize) in a separate image, but that would mean more stuff to maintain.

I ultimately ended on copying the helm and kustomize binaries from the Argo CD image as in the previous patch, and hunting down a minimal image with git already installed. The minimal-image-with-git-search ended with alpine/git due to it being seemingly well maintained and independent.

The only remaining task was then to tie everything together by adding the location of the var-files volume to the PATH variable. I ineptly first tried /var/run/argocd:$(PATH) thinking $PATH would resolve to the one present in the image, but taking a step back I realize this would never work seeing as there’s no way to peek inside the container beforehand, and that you can only resolve already defined env-variable as stated in the Kubernetes documentation.

Realising my mistake I decided to just explicitly define the whole PATH-variable without any entrypoint shenanigans This then resulted in the following atomic patch for an Argo CD CMP-sidecar

apiVersion: apps/v1
kind: Deployment
metadata:
  name: argocd-repo-server
spec:
  template:
    spec:
      containers:
        - name: kustomize-build-with-helm
          env:
            - name: PATH
              value: "/var/run/argocd:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
          command:
            - argocd-cmp-server
          image: alpine/git:2.40.1
          securityContext:
            runAsNonRoot: true
            runAsUser: 999
          volumeMounts:
            - name: var-files
              mountPath: /var/run/argocd
            - name: plugins
              mountPath: /home/argocd/cmp-server/plugins
            - name: cmp-kustomize-build-with-helm
              mountPath: /home/argocd/cmp-server/config/plugin.yaml
              subPath: kustomize-build-with-helm.yaml
            - mountPath: /tmp
              name: cmp-tmp
      volumes:
        - name: cmp-kustomize-build-with-helm
          configMap:
            name: argocd-cm-cmp-kustomize-build-with-helm
        - name: cmp-tmp
          emptyDir: { }

which together with the previously defined ConfigMap and initContainer patch was my lösung for running Kustomized Helm with Argo CD until I switched over to using the community contained Argo Helm charts.

Helm chart
#

Updated Argo CD config. See this commit for the full context. The files below can be found in the GitLab repository for this blog.

With a clever reading of the Argo CD Helm chart template we can reuse the Argo CD repo-server image and skip the init container configuration altogether. Although this might go against the recommendation of only including the necessary binaries, I don’t see much issue in reusing an already running image.

To bootstrap the config run

kubectl kustomize --enable-helm . | kubectl apply -f -

in the containing directory.

# argocd/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

resources:
  - ns.yaml
  - http-route.yaml

helmCharts:
  - name: argo-cd
    repo: https://argoproj.github.io/argo-helm
    version: 7.3.4
    releaseName: "argocd"
    namespace: argocd
    valuesFile: values.yaml
# ns.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: argocd

The HTTPRoute assumes a Gateway as detailed in my Gateway API article.

This can easily be replaced by an Ingress (documentation), or IngressRoute (example) if you’re using Traefik. Note that this HTTPRoute only exposes the GUI and not the gRPC endpoint for the Argo CD CLI.

# argocd/http-route.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: http-route
spec:
  parentRefs:
    - name: gateway
      namespace: gateway
  hostnames:
    - "<URL>"
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: argocd-server
          port: 80

The clever bit is to reuse the Helm chart values in the values.yaml on the highlighted line due to the tpl-function used in the Deployment template here. Thanks to Olav for pointing this out!

# argocd/values.yaml
configs:
  cm:
    create: true
    application.resourceTrackingMethod: "annotation+label"
  cmp:
    create: true
    plugins:
      kustomize-build-with-helm:
        generate:
          command: [ "sh", "-c" ]
          args: [ "kustomize build --enable-helm" ]
  params:
    server.insecure: true

repoServer:
  extraContainers:
    - name: kustomize-build-with-helm
      command:
        - argocd-cmp-server
      image: '{{ default .Values.global.image.repository .Values.repoServer.image.repository }}:{{ default (include "argo-cd.defaultTag" .) .Values.repoServer.image.tag }}'
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        seccompProfile:
          type: RuntimeDefault
        capabilities:
          drop: [ "ALL" ]
      volumeMounts:
        - name: plugins
          mountPath: /home/argocd/cmp-server/plugins
        - name: cmp-kustomize-build-with-helm
          mountPath: /home/argocd/cmp-server/config/plugin.yaml
          subPath: kustomize-build-with-helm.yaml
        - mountPath: /tmp
          name: cmp-tmp
  volumes:
    - name: cmp-kustomize-build-with-helm
      configMap:
        name: argocd-cmp-cm
    - name: cmp-tmp
      emptyDir: { }

Credentials
#

If you’re using private Helm or OCI registries and need to add credentials, you can add the following values inspired by this comment from GitHub user chancez.

# argocd/values-credentials.yaml
repoServer:
  initContainers:
    - name: ghcr-auth
      image: '{{ default .Values.global.image.repository .Values.repoServer.image.repository }}:{{ default (include "argo-cd.defaultTag" .) .Values.repoServer.image.tag }}'
      securityContext:
        runAsNonRoot: true
        runAsUser: 999
        allowPrivilegeEscalation: false
        readOnlyRootFilesystem: true
        seccompProfile:
          type: RuntimeDefault
        capabilities:
          drop: [ "ALL" ]
      env:
        - name: HELM_CONFIG_HOME
          value: /helm-auth
        - name: HELM_REGISTRY_CONFIG
          value: /helm-auth/config.json
        - name: GHCR_USERNAME
          valueFrom:
            secretKeyRef:
              name: argocd-ghcr-helm-credentials
              key: username
        - name: GHCR_PASSWORD
          valueFrom:
            secretKeyRef:
              name: argocd-ghcr-helm-credentials
              key: password
      volumeMounts:
        - name: registry-auth-dir
          mountPath: /helm-auth
      command:
        - /bin/bash
        - -exc
        - 'echo -n $GHCR_PASSWORD | helm registry login ghcr.io --username $GHCR_USERNAME --password-stdin -'
  env:
    - name: HELM_REGISTRY_CONFIG
      value: /helm-auth/config.json
  volumes:
    - name: registry-auth-dir
      emptyDir: { }
  volumeMounts:
    - name: registry-auth-dir
      mountPath: /helm-auth
  containerSecurityContext:
    readOnlyRootFilesystem: true

with the following secret

apiVersion: v1
kind: Secret
metadata:
  name: argocd-ghcr-helm-credentials
  namespace: argocd
type: Opaque
stringData:
  username: "<USERNAME>"
  password: "github_pat_<TOKEN>"

This will create an extra Init Container that performs helm registry login and saves the auth token in /helm-auth/config.json which is then passed to the main Argo CD container.

  1. If it’s years later, and they’ve changed stuff you can find the latest manifest here if you want to do a similar analysis. ↩︎

← → Cloudflared SSH tunneling Browser rendered terminal → ←