Skip to main content

Cluster Add-ons with ApplicationSets

1. Overview

Managing clusters at scale requires standardization and automation. Tasks that looked trivial with a handful of clusters become a burden when managing tens or even hundreds of them.

How did we get here? Read the backstory here.

Backstory by from the perspective of a cluster administrator named Nick.

While scrolling through Slack one morning, Nick got a ping that one of the product teams wanted to do performance testing on their application because of a new feature launch that ties together many different services. They are worried about how the influx of traffic might effect the environment. Given this fear, Nick decided to provision an environment dedicated to performance testing, base the stage environment.

Right now, they are using a cluster for each environment (dev, stage, and prod) with folders for each one that contains the Argo CD Applications for the “add-ons” (a standard set of Kubernetes resources used by applications, expected to be on every cluster).

diagram of cluster folders

Adding this new environment with a new cluster requires lots of copying and pasting of Argo CD Applications from an existing cluster configuration to a folder for the new one, and changing any details specific to the cluster (e.g. cluster name). While not a huge burden at this scale, it’s easy to forget to change something.

diagram of copying Applications between cluster folders

Nick updated the Terraform configuration to set up the new perf environment and create the cluster. Nick added the cluster to the central Argo CD instance and created the App-of-Apps to deploy the Applications from the cluster’s folder in their GitOps repo. Then, Nick confirmed that all of the Applications were synced and healthy.

diagram of terraform creating a cluster and argo cd pull from git to the cluster

After handing over the reins to the product team, Nick got a ping on Slack from a developer who deployed their application to the perf environment, but it’s not working the way it did in stage. After some investigation, Nick found that, in the transition to the new cluster, the Application for cert-manager pointed to the values file of the stage cluster. The values file contained the wrong subdomain, leading to the DNS challenge failing when the dev’s application requested a new cert—a minor oversight when copying over the Argo CD Applications for the new cluster’s add-ons.

diagram of mistake copying Applications

Nick thinks to himself, there must be a better way! A life without toil. 💡

In this tutorial presented by Akuity, attendees will learn how to leverage ApplicationSets, a powerful Argo CD feature, to streamline the deployment of a standard set of "add-ons" (like Sealed Secrets, cert-manager, and Kyverno) to each cluster. We will address the dynamic nature of clusters by demonstrating how they can "opt-in" to specific tooling and configurations, allowing for flexibility without sacrificing standardization.

Manually creating Applications is error-prone and tedious. Especially if the Applications are nearly identical aside from a few values that can be determined automatically. ApplicationSets template Applications and populate them using generators.

diagram of an ApplicationSet creating

In the control-plane repo, the bootstrap folder contains two ApplicationSets, one for the cluster add-ons (cluster-addons-appset.yaml) and one for the cluster apps (cluster-apps-appset). The distinction between "add-ons" and "apps" is delineation between cluster admin and developer. The cluster admin is primarily responsible for the "add-ons" as they are central to cluster and used by many other teams. The developers are responsible for their apps.

diagram with two ApplicationSets with one for add-ons and one for apps with dev/admin split

1.1. Prerequisites

The tutorial assumes you have experience working with Kubernetes, GitOps, and Argo CD. Given such, some underlying concepts won't be explained during this tutorial.

The tutorial requires that you have the following:

2. Creating the Workshop Environment

2.1. Kubernetes Clusters using devcontainer

To demonstrate multi-cluster management, you will create an environment for this workshop using a devcontainer. The devcontainer will be built based on the specification in the devcontainer.json in the workshop repo. Then, it will start up two minikube clusters named dev and prod.

The easiest way to get started is with GitHub Codespaces (especially if you are on conference wifi).

Open workshop in GitHub Codespaces

2.2. Creating an Argo CD Instance

The workshop demonstrates deploying resources to clusters connected to Argo CD. You will create an Argo CD instance using the Akuity Platform to simplify the installation and connecting clusters.

tip

Sign up for a free 14-day trial of the Akuity Platform!

  1. Log into the Akuity Platform with your GitHub account.

  2. Create an organization, naming it after your GitHub username.

  3. Create an Argo CD instance with the name cluster-addons (or any permitted string).

2.3. Connecting Clusters to Argo CD

  1. Provision an agent with the name dev.

    caution

    Using the name dev is crucial because there are corresponding folders in the control-plane workshop repo.

  2. Ensure your current kubeconfig context is set to the dev cluster in the devcontainer by running the cluster-dev command.

info

You can switch between the kubeconfig contexts for these two clusters by running the cluster-dev or cluster-prod commands (technically, bash aliases).

  1. Copy and paste the agent install command displayed after provisioning the agent.

Repeat this process for the prod cluster.

The same steps again.

  1. Provision an agent with the name prod.

    caution

    Using the name prod is crucial because there are corresponding folders in the control-plane workshop repo.

  2. Ensure your current kubeconfig context is set to the prod cluster in the devcontainer by running the cluster-prod command.

  3. Copy and paste the agent install command displayed after provisioning the agent.

2.4. Accessing Argo CD

  1. Enable the admin user and generate the password.

  2. Access your Argo CD instance.

3. Bootstrapping Argo CD

To deploy the ApplicationSets that will automate the creation of Applications for your clusters, you will manually create an initial bootstrap Application.

  1. Navigate to the UI of your Argo CD instance.

  2. Create an Application to manage the Argo CD configuration using the bootstrap-app.yaml manifest at the root of the workshop control-plane repo.

    • Click + NEW APP.

    • Click EDIT AS YAML.

    • Paste the contents of bootstrap-app.yaml.

    • Click SAVE.

    • Click CREATE.

After the Application automatically syncs, it will create two ApplicationSets: cluster-addons and cluster-apps. Notice that they do not create any Applications. Each cluster can opt-in to add-ons, meaning that nothing is deployed unless the cluster indicates that it should be.

4. Enabling Cluster Add-ons

4.1. cluster-addons ApplicationSet

The addons are located in the charts/addons folder, which contain Helm Umbrella charts for the various add-ons deployed to clusters. These Umbrella charts contain the default values for any cluster that opts into the add-on. Each cluster can specify overrides to the defaults values with a file with the same name as the add-on in the clusters/<cluster-name>/addons/ folder (e.g. for the dev cluster and cert-manager add-on, the values file would be clusters/dev/addons/cert-manager.yaml).

charts/
addons/
cert-manager/
Chart.yaml
values.yaml

clusters/
dev/
addons/
cert-manager.yaml

The cluster-addons ApplicationSet is a complex example of generators. It contains two different generators, git and clusters, and a meta-generator matrix to bring it all together.

  generators:
- matrix:
generators:
- git:
repoURL: https://github.com/akuity-cluster-addons-workshop/control-plane
revision: HEAD
directories:
- path: charts/addons/*
- clusters:
selector:
matchExpressions:
- {key: 'akuity.io/argo-cd-cluster-name', operator: NotIn, values: [in-cluster]}
- key: enable_{{ .path.basename | snakecase }}
operator: In
values: ['true']

The git generator is fairly straight-forward.

generators:
- git:
repoURL: https://github.com/akuity-cluster-addons-workshop/control-plane
revision: HEAD
directories:
- path: charts/addons/*

It takes a repoURL, a revision, and a directories value which determine where to get generator attributes from. In this case the git generator will produce an item for each folder in charts/addons of the control-plane repo on the main branch (HEAD points to the tip of the main branch). The result will be:

- path.basename: cert-manager
- path.basename: external-secrets
- path.basename: kyverno

The cluster generator is more complex.

generators:
- clusters:
selector:
matchExpressions:
- {key: 'akuity.io/argo-cd-cluster-name', operator: NotIn, values: [in-cluster]}
- key: enable_{{ .path.basename | snakecase }}
operator: In
values: ['true']

For each cluster registered with Argo CD, the cluster generator produces the name, server, labels, and annotations attributes based on the cluster's configuration. For example:

- name: dev
server: https://....
metadata:
labels:
enable_cert_manager: true
annotations: {}

The results returned from the cluster generator are filtered based on the metadata.labels, with two conditions:

  • the akuity.io/argo-cd-cluster-name label (which is automatically added to each cluster configuration by the Akuity Platform), excluding any that contain the value in-cluster. This prevents the ApplicationSet from attempting to deploy add-ons to the cluster hosting Argo CD, which on the Akuity Platform will only accept Argo CD manifests.

  • the label enable_{{ .path.basename | snakecase }} with the value true. This expression takes the attributes returned from the git generator to determine what labels to filter on. So if the git generator returned .path.basename: cert-manager, the cluster generator will only include clusters with the enable_cert_manager: true label.

    This label is what enables clusters to opt-in to add-ons based on their metadata. If a cluster does not have an enable_ label corresponding to the add-on folder name, the add-on will not be deployed to the cluster.

Then, to bring it all together, the matrix generator will combine the results from both the git and cluster generators. Each combination of cluster (matching the selector expressions) and add-on found in the charts/addons folder will produce an Application

Take the example of a cluster named dev with the label enable_cert_manager: true and the charts/addons folder that contains the folders cert-manager and external-dns.

The git generator will return:

- path.basename: cert-manager
- path.basename: external-secrets

The cluster generator will return:

- name: dev

The matrix generator will combine the results and return:

- name: dev
path.basename: cert-manager

The combination of the dev cluster and the external-secrets add-on is omitted due to the selector on the cluster generator.

Once these conditions are met (the cluster contains the enable_xxx label and the add-on folder exists), the ApplicationSet will produce an Application sourcing to the add-on Helm Umbrella chart using the path attribute supplied by the git generator.

source:
repoURL: 'https://github.com/akuity-cluster-addons-workshop/control-plane'
path: '{{.path.path}}'
targetRevision: HEAD

The Application uses the add-on name (the path.basename attribute supplied by the git generator) as the release name when templating the Helm chart, and uses the cluster name and add-on name in determining what values file to supply to the chart. The use of ignoreMissingValueFiles here allows the Application to work even if no cluster-specific overrides has been defined in the repo.

source:
...
helm:
releaseName: '{{.path.basename}}'
ignoreMissingValueFiles: true
valueFiles:
- '../../../clusters/{{.name}}/addons/{{.path.basename}}.yaml'

And finally, the cluster name is used for the destination cluster and the add-on name is used for the destination namespace.

destination:
namespace: '{{.path.basename}}'
name: '{{.name}}'

4.2. Enable the cert-manager Add-on for dev

You'll start by deploying the cert-manager add-on for the dev cluster by adding the enable_cert_manager: true label to the cluster configuration.

  1. Navigate to the Clusters dashboard for your Argo CD instance on the Akuity Platform.

  2. Click the settings cog on the right side of the dev cluster.

  3. Under the "Labels" section click Add Label.

  4. Set the key to enable_cert_manager and the value to true

  5. Click Update Cluster.

Return to Argo CD UI to see the cluster-addons ApplicationSet immediately create the addons-dev-cert-manager Application, which will deploy cert-manager to the dev cluster.

It's using the Helm Umbrella chart from the charts/add-ons/cert-manager folder of the control-plane repo.

5. Deploying Applications

5.1. cluster-apps ApplicationSet

The apps are unique to each cluster, so they are located in cluster/<cluster-name/apps folder.

Similar to the cluster-addons ApplicationSet, the cluster-apps ApplicationSet uses the cluster generator to create an Application for each cluster.

generators:
- clusters:
selector:
matchExpressions:
- {key: 'akuity.io/argo-cd-cluster-name', operator: NotIn, values: [in-cluster]}
- key: ready
operator: In
values: ['true']

Using the same selector technique, the creation of the Application is gated using the ready: true label in the cluster configuration. Once the cluster contains this label, the ApplicationSet generates an app-<cluster-name> Application which points to the cluster/<cluster-name>/apps folder (using the name attribute supplied by the cluster generator).

metadata:
name: 'apps-{{name}}'
...
source:
repoURL: https://github.com/akuity-cluster-addons-workshop/control-plane
targetRevision: HEAD
path: 'clusters/{{name}}/apps'

The Application is configured to take any plain Kubernetes manifest in the folder and deploy it to the Argo CD control-plane (i.e. the in-cluster destination in the argocd namespace). Therefore, it is expected to contain Argo CD manifests (e.g. Application, ApplicationSet, and AppProject).

source:
...
directory:
recurse: true
destination:
name: 'in-cluster'
namespace: 'argocd'

5.2. Mark the Cluster as Ready

Now that the add-ons have been deployed to the cluster, you'll add the ready: true label to the cluster configuration mark the cluster as ready for Applications.

  1. Navigate to the Clusters dashboard for your Argo CD instance on the Akuity Platform.

  2. Click the settings cog on the right side of the dev cluster.

  3. Under the "Labels" section click Add Label.

  4. Set the key to ready and the value to true

  5. Click Update Cluster.

Return to Argo CD UI to see the cluster-apps ApplicationSet immediately create the guestbook-dev Application from the clusters/dev/apps/ folder in the control-plane repo.