I use Argo CD to maintain my Homelab as I find it intuitive. The nice GUI also helps me to quickly inspect problems which might occur when I try something fancy. Another widely used alternative is Flux CD which solves the same problem of GitOps-ing your cluster, but I have little experience with it yet.
In this article I’m going to try and explain how I use ArgoCD with Kustomized Helm to maintain my Homelab using GitOps-practices.
Kustomize + Helm = ❤️#
Sometimes a Helm chart doesn’t have everything you need nicely templated,
or you want to reference a Helm chart in your kustomization.yaml file to have everything nice and neat together.
Out of the box ArgoCD comes with support for both Kustomize and Helm, but not both at the same time. You could fully render the Helm template and start manually editing it before checking the tailored manifest into Git, but I find that to be a not-so-elegant solution which imho would also be harder to maintain.
A better way would be to combine the power of both Kustomize and Helm to get your way.
The Old Ways#
Pre Argo CD v2.8, a way to achieve Kustomized Helm was to patch the argocd-cm ConfigMap
apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-cm
data:
configManagementPlugins: |
- name: kustomize-build-with-helm
generate:
command: [ "sh", "-c" ]
args: [ "kustomize build --enable-helm" ]
using Kustomize
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: argocd
...
patches:
- path: patches/argocd-cm-kustomize-helm-patch.yaml
which I was previously doing ( commit) since I didn’t get the memo that this has been deprecated since Argo CD v2.4.
As a meager excuse on my part a similar solution is explained in the official Argo CD example apps repo which led me to believe this was the way to do it.
Config Management Plugin#
My first attempt ( commit) was simply following the Argo CD documentation on Config Management Plugins (CMP) which boiled down to a ConfigMap with the CMP-config (CMP follows the Kubernetes-style spec convention, but is not a CRD)
apiVersion: v1
kind: ConfigMap
metadata:
name: argocd-cm-cmp-kustomize-build-with-helm
data:
plugin.yaml: |
apiVersion: argoproj.io/v1alpha1
kind: ConfigManagementPlugin
metadata:
name: kustomize-build-with-helm
spec:
generate:
command: [ "sh", "-c" ]
args: [ "kustomize build --enable-helm" ]
and a patch to the argocd-repo-server Deployment to start a CMP-sidecar with the following configuration
apiVersion: apps/v1
kind: Deployment
metadata:
name: argocd-repo-server
spec:
template:
spec:
containers:
- name: kustomize-build-with-helm
command: [ /var/run/argocd/argocd-cmp-server ]
image: quay.io/argoproj/argocd:v2.7.11
securityContext:
runAsNonRoot: true
runAsUser: 999
volumeMounts:
- name: var-files
mountPath: /var/run/argocd
- name: plugins
mountPath: /home/argocd/cmp-server/plugins
- name: argocd-plugin-config
mountPath: /home/argocd/cmp-server/config/plugin.yaml
subPath: plugin.yaml
- mountPath: /tmp
name: cmp-tmp
volumes:
- name: argocd-plugin-config
configMap:
name: argocd-cm-cmp-kustomize-build-with-helm
- name: cmp-tmp
emptyDir: { }
which works, and made it possible to upgrade to Argo CD v2.8 without anything failing.
But we can do better!
Digging deeper 🐰🕳️#
While the previous quick fix works, it has the annoying downside that we need to keep the patched Argo CD image tag in sync with the main image. The goal of this section is to create a solution that requires less hands-on maintenance while also being fairly minimal.
This is going to be kind of a ramble where I try to explain my process and reasoning, so if you want to jump to the conclusion take a look at the resulting commit.
You might’ve noticed that in the patch we’re referencing volumes in the containers-section that doesn’t seem to be
defined in the volumes-section, so what’s going on?
Taking a step back the answer is already in the formulation of the question in that this is a patch,
and we need the full manifest in order to figure out what’s really going on.
This article is written for Argo CD v2.8.2, so we’ll be basing it on that manifest available here1
Looking at the argocd-repo-server Deployment we see an initContainer which copies the argocd-binary to the mystery
var-files volume.
Here edited for brevity
apiVersion: apps/v1
kind: Deployment
metadata:
name: argocd-repo-server
spec:
...
template:
...
spec:
...
initContainers:
- command:
- /bin/cp
- -n
- /usr/local/bin/argocd
- /var/run/argocd/argocd-cmp-server
image: quay.io/argoproj/argocd:v2.8.2
name: copyutil
volumeMounts:
- mountPath: /var/run/argocd
name: var-files
...
volumes:
- emptyDir: { }
name: var-files
- emptyDir: { }
name: plugins
- ...
By itself this seems pointless, and I’d argue that it is if you’re running vanilla Argo CD. Without digging too deep I assume it’s done to make running sidecar plugins easier.
Digging even deeper and looking inside the argocd-repo-server container we see the argocd-binary being soft linked
to a several times,
all with names hinting at different parts of Argo CD.
argocd@argocd-repo-server-6589ffb4b6-4rdvp:/usr/local/bin$ ls -lah
total 205M
drwxr-xr-x 1 root root 4.0K Aug 24 20:34 .
drwxr-xr-x 1 root root 4.0K Jun 24 02:02 ..
-rwxr-xr-x 1 root root 142M Aug 24 20:34 argocd
lrwxrwxrwx 1 root root 21 Aug 24 20:34 argocd-application-controller -> /usr/local/bin/argocd
lrwxrwxrwx 1 root root 21 Aug 24 20:34 argocd-applicationset-controller -> /usr/local/bin/argocd
lrwxrwxrwx 1 root root 21 Aug 24 20:34 argocd-cmp-server -> /usr/local/bin/argocd
lrwxrwxrwx 1 root root 21 Aug 24 20:34 argocd-dex -> /usr/local/bin/argocd
lrwxrwxrwx 1 root root 21 Aug 24 20:34 argocd-k8s-auth -> /usr/local/bin/argocd
lrwxrwxrwx 1 root root 21 Aug 24 20:34 argocd-notifications -> /usr/local/bin/argocd
lrwxrwxrwx 1 root root 21 Aug 24 20:34 argocd-repo-server -> /usr/local/bin/argocd
lrwxrwxrwx 1 root root 21 Aug 24 20:34 argocd-server -> /usr/local/bin/argocd
-rwxr-xr-x 1 root root 203 Aug 24 20:05 entrypoint.sh
-rwxr-xr-x 1 root root 934 Aug 24 20:05 git-verify-wrapper.sh
-rwxr-xr-x 1 root root 215 Aug 24 20:05 gpg-wrapper.sh
-rwxr-xr-x 1 root root 49M Aug 24 20:08 helm
-rwxr-xr-x 1 root root 15M Aug 24 20:08 kustomize
lrwxrwxrwx 1 root root 28 Aug 24 20:08 uid_entrypoint.sh -> /usr/local/bin/entrypoint.sh
Although intriguing I feel we’re digressing, and I’ll put this side quest on hold. (Though please contact me if you want to enlighten me why it’s done this way!)
Argo suggests that you create your own image with all the tools you need, and that would be certainly be a better solution than the upcoming hack for an enterprise environment, but this is my Homelab!
Knowing that we need both the helm and kustomize binaries and assuming that they’re both standalone,
why not try to copy them to a minimal base image!
Next I hijacked the copyutil-initContainer (and implicitly the var-files volume) with a patch to do just this:
apiVersion: apps/v1
kind: Deployment
metadata:
name: argocd-repo-server
spec:
template:
spec:
initContainers:
- name: copyutil
command: [ /bin/bash ]
args:
- -c
- >-
/bin/cp -n /usr/local/bin/argocd /var/run/argocd/argocd-cmp-server &&
/bin/cp -n /usr/local/bin/kustomize /var/run/argocd/kustomize &&
/bin/cp -n /usr/local/bin/helm /var/run/argocd/helm
I then tried using a busybox-image inspired by
the Argo CD CMP example,
but quickly ran into an issue with missing git.
My first idea was then to try and just copy the git binary as well,
but that yielded an error with a missing libpcre2-8.so.0 library.
Ok, so that only kinda halfway worked.
We could try and hunt down every library git might need,
but at this point I’d argue it’s easier to just install git
(along with helm and kustomize) in a separate image,
but that would mean more stuff to maintain.
I ultimately ended on copying the helm and kustomize binaries from the Argo CD image as in the previous patch,
and hunting down a minimal image with git already installed.
The minimal-image-with-git-search ended with alpine/git due to it being
seemingly well maintained and independent.
The only remaining task was then to tie everything together by adding the location of the var-files volume to the
PATH variable.
I ineptly first tried /var/run/argocd:$(PATH) thinking $PATH would resolve to the one present in the image,
but taking a step back I realize this would never work seeing as there’s no way to peek inside the container beforehand,
and that you can only resolve already defined env-variable as stated in
the Kubernetes documentation.
Realising my mistake I decided to just explicitly define the whole PATH-variable without
any entrypoint shenanigans
This then resulted in the following atomic patch for an Argo CD CMP-sidecar
apiVersion: apps/v1
kind: Deployment
metadata:
name: argocd-repo-server
spec:
template:
spec:
containers:
- name: kustomize-build-with-helm
env:
- name: PATH
value: "/var/run/argocd:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
command:
- argocd-cmp-server
image: alpine/git:2.40.1
securityContext:
runAsNonRoot: true
runAsUser: 999
volumeMounts:
- name: var-files
mountPath: /var/run/argocd
- name: plugins
mountPath: /home/argocd/cmp-server/plugins
- name: cmp-kustomize-build-with-helm
mountPath: /home/argocd/cmp-server/config/plugin.yaml
subPath: kustomize-build-with-helm.yaml
- mountPath: /tmp
name: cmp-tmp
volumes:
- name: cmp-kustomize-build-with-helm
configMap:
name: argocd-cm-cmp-kustomize-build-with-helm
- name: cmp-tmp
emptyDir: { }
which together with the previously defined ConfigMap and initContainer patch is my current lösung for running Kustomized Helm with Argo CD 🎉
Helm chart#
Updated Argo CD config. See this commit for the full context.
With a clever reading of the Argo CD Helm chart template we can reuse the Argo CD repo-server image and skip the init container configuration altogether. Although this might go against the recommendation of only including the necessary binaries, I don’t see much issue in reusing an already running image.
To bootstrap the config run
kubectl kustomize --enable-helm . | kubectl apply -f -
in the containing directory.
# kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- ns.yaml
- http-route.yaml
helmCharts:
- name: argo-cd
repo: https://argoproj.github.io/argo-helm
version: 6.0.6
releaseName: "argocd"
namespace: argocd
valuesFile: values.yaml
# ns.yaml
apiVersion: v1
kind: Namespace
metadata:
name: argocd
The HTTPRoute assumes a Gateway as detailed in
my Gateway API article.
This can easily be replaced by an
Ingress (
documentation), or
IngressRoute (
example)
if you’re using Traefik.
Note that this HTTPRoute only exposes the GUI and not the gRPC endpoint for the Argo CD CLI.
# http-route.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: http-route
spec:
parentRefs:
- name: gateway
namespace: gateway
hostnames:
- "<URL>"
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: argocd-server
port: 80
The clever bit is to reuse the Helm chart values in the values.yaml on the highlighted line due to
the tpl-function
used in the Deployment
template here.
Thanks to Olav for pointing this out!
# values.yaml
configs:
cm:
application.resourceTrackingMethod: "annotation+label"
cmp:
create: true
plugins:
kustomize-build-with-helm:
generate:
command: [ "sh", "-c" ]
args: [ "kustomize build --enable-helm" ]
params:
server.insecure: true
repoServer:
extraContainers:
- name: kustomize-build-with-helm
command:
- argocd-cmp-server
image: '{{ default .Values.global.image.repository .Values.repoServer.image.repository }}:{{ default (include "argo-cd.defaultTag" .) .Values.repoServer.image.tag }}'
securityContext:
runAsNonRoot: true
runAsUser: 999
volumeMounts:
- name: plugins
mountPath: /home/argocd/cmp-server/plugins
- name: cmp-kustomize-build-with-helm
mountPath: /home/argocd/cmp-server/config/plugin.yaml
subPath: kustomize-build-with-helm.yaml
- mountPath: /tmp
name: cmp-tmp
volumes:
- name: cmp-kustomize-build-with-helm
configMap:
name: argocd-cmp-cm
- name: cmp-tmp
emptyDir: { }