Advanced GitOps Tutorial

1. Overview
- 1.1. Prerequisites
- 1.2. Repositories
2. Control Plane Setup
3. Building Images
- 3.1. Creating the demo-app Repo
4. Manifest Management
- 4.1. Creating the demo-app-deploy Repo
5. Self-service Teams
- 5.1. Creating a Team Environment
6. Let's deploy something!
- 6.1. Update The Image Tag for the dev Environment
7. Automating Image Tag Updates
- 7.1. Automating the dev Environment
- 7.2. Automating Promotion to stage and prod
8. Rendered Manifests
- 8.1. Adding Render Manifests to the demo-app-deploy Repo
9. Notifications
- 9.1. Setting Up the GitHub Notification Service
- 9.2. Adding the Notification Subscription
10. Secrets
- 10.1. Add an ExternalSecret to the demo-app
11. Summary
12. Clean Up

1. Overview

The Advanced GitOps tutorial, presented by Akuity, will take you through creating a self-service multi-tenant Kubernetes environment using Argo CD and GitHub Actions. The tutorial will cover the following topics:

Automating Application generation using ApplicationSets.
Using GitHub for Single Sign-on (SSO).
Managing manifests for promotion between environments.
Enabling self-service environment creation for teams.
Integrating notifications and secrets with Argo CD.

Naturally, the Argo CD configuration is managed by the cluster administrators. However, developers will not need administrators to create their environments. With GitOps, teams can self-onboard, leveraging Helm Charts provided by the administrators to abstract the resources that comprise an environment and pull requests to propose the creation of and changes to them.

1.1. Prerequisites

The tutorial assumes you have experience working with Kubernetes, GitOps, GitHub Actions (or a similar CI engine), and Argo CD. Given such, some underlying concepts won't be explained during this tutorial.

The tutorial requires that you have the following:

a GitHub Account - you will use this to:
- host repositories for the control plane, demo application, and deployment configuration.
- run GitHub Actions. (The repositories are public, so there is no cost.)
- create a GitHub App for integrating SSO and notifications.
- create an Argo CD instance on the Akuity Platform.
a dedicated Kubernetes cluster with Cluster Admin access.
- This tutorial does not require a publicly accessible cluster, only egress traffic (i.e., internet access). Consider using Docker Desktop and kind your local machine.
- Admin access is required to create namespaces and cluster roles.
the Kubernetes command-line tool, kubectl.
a local IDE with access set up to your GitHub.
tip
Throughout this tutorial, all changes to repositories can be done from your browser by changing github.com to github.dev in the URL for the repository. See the GitHub docs for more details.
a browser with internet access.

This tutorial shows placeholder text between less-than and greater-than symbols (i.e., <...>), indicating that you must substitute it with the value relevant to your scenario.

<username> - Your GitHub username.

1.2. Repositories

The tutorial will use the following repositories:

The control-plane repository defines the desired state of the Kubernetes platform and enables self-service onboarding for teams. It contains the following directories:
- argocd - configuration for Argo CD (e.g., Applications, AppProjects)
- charts - local and Umbrella Helm Charts.
- clusters - cluster-specific configurations (e.g., ClusterSecretStore).
- teams - a folder for each team.
The demo-app repository contains the source code for a simple Golang application packaged into a container image used to demonstrate Argo CD driven GitOps processes and CI.
The demo-app-deploy repository uses Kustomize to define the manifests to deploy the demo-app and contains the following directories:
- base/ - the application manifests agnostic to any environment.
  - deployment.yaml - defines the Deployment.
  - service.yaml - defines the Service.
- env/ - the overlays specific to environments.
  - dev/
  - stage/
  - prod/

2. Control Plane Setup

The control plane setup assumes that you have already prepared a Kubernetes cluster with internet access.

2.1. Automating `Application` Creation with `ApplicationSet`s

Each cluster will get two addons (kyverno and external-secrets), the cluster-specific configurations, and Namespaces for the teams. Each requires an Application in Argo CD to install them into each cluster.

Manually creating Applications is error-prone and tedious. Scripting config management tools can automate the process, but there is a better way. ApplicationSets template Applications and populate them using generators.

These are the ApplicationSets in the control-plane repo:

argocd/clusters-appset.yaml will generate an Application for each cluster registered to Argo CD (excluding the in-cluster) and point it to the clusters/{{name}} folder in the control-plane repo.

argocd/addons-appset.yaml will generate an Application for each combination of cluster registered to Argo CD (excluding the in-cluster) and Helm Chart folder in the list generator.

# generators
- list:
    elements:  # The value in the pairs below are the folder name from `charts/`.
    - addonChart: external-secrets
    - addonChart: kyverno
- clusters:
    selector:
      matchExpressions:
      - {key: 'akuity.io/argo-cd-cluster-name', operator: NotIn, values: [in-cluster]}

The addons-values.yaml file from the cluster/ folder is passed to each Helm Chart. Using Umbrella charts (setting sub-charts as dependencies in the local chart), the same values file can be passed to each chart, without fear of conflicts.

# cluster/{{name}}/addons-values.yaml
external-secrets: {}
kyverno: {}

# values file from cluster folder
source:
  helm:
    valueFiles:
    - '../../clusters/{{name}}/addons-values.yaml'

argocd/teams-appset.yaml will generate an Application for each folder in teams/. The resulting Application will use the charts/team Helm Chart with the values.yaml from the team's folder.

# File structure of the `teams` folder.
teams/
  <username>/
    values.yaml

# ApplicationSet `git` generator.
- git:
  repoURL: https://github.com/<username>/control-plane
  revision: HEAD
  directories:
  - path: "teams/*"

# Application spec template using the values file.
  source:
    helm:
      releaseName: '{{path.basename}}'  # The team name.
      valueFiles:
      - '../../{{path}}/values.yaml'    # The team's folder.

In the team Helm Chart, the repo-appset.yaml template will create an ApplicationSet for each item in the repos value. Each one will generate an Application for the repository name and all the folders found under env/.

{{- range .Values.repos }}
---
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: '{{ $.Release.Name }}-{{ . | trimSuffix "-deploy" }}'
spec:
  generators:
  - git:
      repoURL: 'https://github.com/{{ $.Release.Name }}/{{ . }}'
      revision: HEAD
      directories:
        - path: env/*
  template:
    metadata:
      name: '{{ $.Release.Name }}-{{ . | trimSuffix "-deploy" }}-{{ "{{" }}path.basename{{ "}}" }}'
    spec:
      project: '{{ $.Release.Name }}'
      source:
        repoURL: 'https://github.com/{{ $.Release.Name }}/{{ . }}'
        targetRevision: 'HEAD'
        path: '{{ "{{" }}path{{ "}}" }}'
      destination:
        name: '{{ $.Values.cluster }}'
        namespace: '{{ $.Release.Name }}-{{ "{{" }}path.basename{{ "}}" }}'
{{- end }}

Notice the use of {{ "{{" }} in the path of the template; this is to prevent Helm from interpreting the ApplicationSet template ({{path}}) syntax during templating of the chart.

2.2. Creating an Argo CD Instance

The tutorial demonstrates deploying resources to clusters connected to Argo CD. You will create an Argo CD instance using the Akuity Platform to simplify the installation and connecting clusters.

Log into the Akuity Platform with your GitHub account.
- If you don't already have an organization (i.e., it's your first time logging in), create one naming it using your GitHub username.
Create an Argo CD instance on the Akuity Platform with the name adv-gitops (or any permitted string).
Enable Declarative Management for the Argo CD instance.
info
Declarative Management enables using the Akuity Platform with the App of Apps pattern and ApplicationSets by exposing the in-cluster destination. Using the in-cluster destination means an Application can create other Applications in the cluster hosting Argo CD.
Provision an agent for a cluster named workshop.
caution
Using the name workshop is crucial because the manifests provided for the tutorial assume this is the destination.name available for Applications in Argo CD.
Deploy the agent to the cluster.

2.3. Integrating GitHub SSO

Argo CD provides a basic implementation for user management with the expectation that most organizations will set up Single Sign-On (SSO). You will use GitHub as the SSO provider by creating a GitHub App.

In it's default configuration, this will allow anyone with a GitHub account to log into your Argo CD instance and assume the readonly role from the default policy.

Create a new GitHub App.
- Set GitHub App name to adv-gitops-<username>, replacing <username> with your GitHub username.
  caution
  GitHub App names are globally unique, so including the <username> is essential.
- Set Homepage URL to https://<instance-id>.cd.akuity.cloud/, replacing <instance-id> with the ID for the Argo CD instance on the Akuity Platform.
```
https://<instance-id>.cd.akuity.cloud/
```
- Set Authorization callback URL to https://<instance-id>.cd.akuity.cloud/api/dex/callback, replacing <instance-id> with the ID for the Argo CD instance on the Akuity Platform.
```
https://<instance-id>.cd.akuity.cloud/api/dex/callback
```
- Under the "Webhook" section, deselect the Active option.
- Under the "Permissions" section, expand the Account permissions then set Email addresses to Read-only access level.
  info
  The Read-only access for Email addresses is required for GitHub users that have their email address set to private.
- Click Create GitHub App.
In the settings for the GitHub App, under "Client secrets", click Generate a new client secret.
Back on the Akuity Platform, in the Settings for the Argo CD instance, go to SSO Configuration > Configuration and click Add new connector
- Set Type to github
- Click Add
Go to SSO Configuration > Secrets.
- Add key GITHUB_CLIENT_SECRET and set the value to the client secret generated on the GitHub App.
- Add key GITHUB_CLIENT_ID and set the value to the "Client ID" from the GitHub App.
- Click Save.
Configure the RBAC policies.
- In the Settings for the Argo CD instance, go to RBAC.
- Under OIDC Scopes, click Add Scope and enter email.
- Under Policy, add this line to make yourself an admin. Replace <email> with your GitHub primary email address.
```
g, <email>, role:admin
```
- Click Save.
After the instance has finished progressing, log into Argo CD using GitHub SSO.
- From the Akuity Platform dashboard for the Argo CD instance, click the <instance-id>.cd.akuity.cloud link.
- Click LOG IN VIA GITHUB.
- Click Authorize akuity-adv-gitops-workshop-<event>.

You are now logged into Argo CD by authenticating with your GitHub account. By setting the OICD scope to email and adding a policy to assign your email to the admin role, you can perform any function within Argo CD.

2.4. Creating the `control-plane` Repository

The control-plane repository is provided as a template on GitHub. For the tutorial, you will create a repository on your account using the template and update the placeholder text within it.

Create a new public repoistory named control-plane from akuity-adv-gitops-workshop/control-plane-template.
- Set the Owner to your account.
- Set the Repository name to control-plane.
- Select Public for the visibility of the repository.
- Click Create repository from template.
Open the repository, then find and replace <username>/control-plane with the new repo URL.
```
-- repoURL: https://github.com/<username>/control-plane
++ repoURL: https://github.com/morey-tech/control-plane
```
tip
Changing github.com to github.dev in the URL will open up VSCode in your browser. From there, you can use the search and replace feature.
Throughout this tutorial, all repository changes can be done from your browser.

2.5. Bootstrapping with App of Apps

To enable the GitOps process, a top-level Application is used to manage the Argo CD configuration and propagate repository changes to the argocd namespace on the in-cluster destination (i.e., the Akuity Platform control plane).

Navigate to the Argo CD UI.
Create an Application to manage the Argo CD configuration using the argocd-app.yaml manifest at the root of the control-plane repo.
- Click + NEW APP.
- Click EDIT AS YAML.
- Paste the contents of argocd-app.yaml.
- Click SAVE.
- Click CREATE.

`argocd` Applcation resource tree.

After creating the argocd Application, the automated sync policy deployed the addons, clusters, teams ApplicationSets, along with the default AppProject (assuming ownership of the existing project).

The clusters ApplicationSet generated an Application for the workshop cluster, which created a couple of Kyverno ClusterPolicy resources and an External Secrets SecretStore.

The addons ApplicationSet generated a Kyverno and External Secrets Application for the workshop cluster.

The addons create the CRDs on which resources in the cluster Application depend. This setup relies on the sync retry mechanism to make the resources eventually consistent.

3. Building Images

Now that infrastructure configuring is done, it's time to set up the demo-app.

3.1. Creating the `demo-app` Repo

To deploy an application using Argo CD, you need a container image. It would be boring if you deployed a pre-built image, so you will build one specific to you!

You are going to leverage GitHub Actions to build an image and push it to the GitHub Container Registry (GHCR). The workflow is already defined in the repository. You only need to create a repository from the template on your account and make a small change customizing it to you, which will trigger the workflow.

Create a new public repoistory named demo-app from akuity-adv-gitops-workshop/demo-app-template.
- Set the Owner to your GitHub account.
- Set the Repository name to demo-app.
- Select Public for the visibility of the repository.
- Click Create repository from template.
Create a new classic (not fine-grained) personal access token (PAT) with write access for repositories and packages.
Yes, a personal access token can be less safe, but you will use it during the tutorial for simplicity.
- Navigate to: https://github.com/settings/tokens/new
- Set the Note to demo-app.
- Select the repo and write:package scopes.
- Click Generate token.
Add the PAT to your demo-app repository secrets as DEPLOY_PAT:
- Navigate to https://github.com/<username>/demo-app/settings/secrets/actions/new
- Click New repository secret.
- Set the Name to DEPLOY_PAT.
- Paste the PAT as the Secret.
- Click Add secret.
Edit the main.go file and replace the <username> placeholder with your GitHub username.
Push the changes to trigger the build workflow and wait for it to complete.
note
Creating the repo will have triggered a run of the workflow too. That run will have failed due to the missing secret containing the PAT.
View the image created in the GitHub Packages page and update the visibility of the demo-app package to Public.
- Navigate to https://github.com/users/<username>/packages/container/demo-app/settings
- Danger Zone > Change package visibility, set to `Public`.

The demo-app image is avaiable at ghcr.io/<username>/demo-app with the latest and the commit id (SHA) tags.

4. Manifest Management

The image is ready, but you need the Kubernetes manifests to deploy the application. To define the deployment manifests, you will leverage a config management tool, Kustomize.

4.1. Creating the `demo-app-deploy` Repo

The demo-app-deploy repository is provided as a template on GitHub. For the tutorial, you will create a repository on your account using the template and update the placeholder text within it.

Create a new public repository named demo-app-deploy from akuity-adv-gitops-workshop/demo-app-deploy-template.
- Set the Owner to your GitHub account.
- Set the Repository name to demo-app-deploy.
- Select Public for the visibility of the repository.
- Click Create repository from template.
Create (another) new classic (not fine-grained) PAT with write access for repositories.
- Set the Note to demo-app-deploy.
- Select the repo scopes.
- Click Generate token.
Add the PAT to your demo-app-deploy repository secrets as DEPLOY_PAT:
- Navigate to https://github.com/<username>/demo-app-deploy/settings/secrets/actions/new
- Click New repository secret.
- Set the Name to DEPLOY_PAT.
- Paste the PAT as the Secret.
- Click Add secret.
Open the repository, then find and replace <username>/demo-app, updating the placeholder to your GitHub username.
```
-- image: ghcr.io/<username>/demo-app:latest
++ image: ghcr.io/morey-tech/demo-app:latest
```

5. Self-service Teams

Each team needs an AppProject, a Namespace for each environment, and Applications. A team can request to create these resources, but they are ultimately managed by the administrators who approve the Pull Request.

The teams are provided with a Helm Chart, charts/teams to abstract the concepts of Namespaces, AppProjects, and ApplicationSets. Instead, they must create a new folder under teams/ and add the example values.yaml from teams/USERNAME.

5.1. Creating a Team Environment

You will create a team for the tutorial using your GitHub username.

Open the control-plane repository.
Copy the teams/USERNAME folder and create a new folder with your GitHub username (i.e., teams/<username>).
Update the values.yaml file with your information.
- Set the email value (replacing <email>) to the email displayed on the User Info page in Argo CD (which should be your primary GitHub email address).
Commit changes and create a pull request to the main branch.

On the Applications page of Argo CD, add your project to the filter to experience what a team would see.

Once the teams ApplicationSet has detected the change in Git, it will create an Application for the Namespaces on the cluster, the AppProject for the team, and an ApplicationSet for each repo in the values.yaml.

The demo-app is deployed using the latest tag to each namespace. Check out the Deployment logs to confirm that your <username> is being printed.

6. Let's deploy something!

It's time to demonstrate deploying a new release of the demo-app.

6.1. Update The Image Tag for the `dev` Environment

Let's start by replacing the latest tag with the commit ID tag in the dev environment. You will use the tag built after setting up the demo-app repo.

Navigate to the demo-app packages page: https://github.com/users/<username>/packages/container/package/demo-app
Copy the most recent image tag using the commit ID (SHA) format (e.g., 923d0b0f341ebf0a9aa7f7597a18e4db690fe1b2).
Configure the dev environment to use the image.
- Open your demo-app-deploy repo.
- In the env/dev/kustomization.yaml file, replace latest with the commit ID (SHA) tag.
```
images:
- name: ghcr.io/<username>/demo-app
  newTag: latest
```
  info
  This is using the Kustomize edit image feature.
- Commit the changes and push them to the repository.

7. Automating Image Tag Updates

Updating the image tag in the kustomization.yaml is boring, and no one likes to make trivial tag changes manually. To make the process more efficient, you are going to automate this process with GitHub Actions!

7.1. Automating the `dev` Environment

The dev environment is intended to track the trunk of the repository (i.e., the main branch), and there is minimal impact if a change breaks the application. Given this, you will automate the deployment to dev each time a new build is created in the demo-app repo.

Navigate to your demo-app repository.
Uncomment the deploy-dev job in .github/workflows/ci.yaml workflow file:

  deploy-dev:
    runs-on: ubuntu-latest
    needs: build-image
    steps:
    - uses: imranismail/setup-kustomize@v1

    - name: Update env/dev image with Kustomize
      run: |
        git config --global user.name "Deploy Bot"
        git config --global user.email "no-reply@akuity.io"
        git clone https://bot:${{ secrets.DEPLOY_PAT }}@github.com/${{ github.repository_owner }}/demo-app-deploy.git
        cd demo-app-deploy/env/dev
        kustomize edit set image ghcr.io/${{ github.repository_owner }}/demo-app:${{ github.sha }}
        git commit -a -m "chore(dev): deploy demo-app:${{ github.sha }}"
        git notes append -m "image: ghcr.io/${{ github.repository_owner }}/demo-app:${{ github.sha }}"
        git push origin "refs/notes/*" --force && git push --force

Commit and push the changes, then watch the action run!

When the workflow completes (successfully), check out the latest commit on your demo-app-deploy repo.

demo-app-deploy-auto-dev-commit

Once changes are pushed, the CI workflow will build a new image and update the dev environment with the corresponding image tag. Developers no longer need to manually change deployment manifests to update the dev environment.

7.2. Automating Promotion to `stage` and `prod`

Updating the manifests for the stage and prod deployments requires a more careful process, but it can also be automated.

Allow GitHub Actions to create and approve pull requests on the demo-app repo.
- Navigate to: https://github.com/<username>/demo-app-deploy/settings/actions
- Under Workflow permissions, check the box for Allow GitHub Actions to create and approve pull requests.
- Click Save.
In your demo-app-deploy repo, open .github/workflows/promote.yaml:
- Uncomment the promote-image-change job.
- Delete the placeholder-job.
- Commit and push the changes.
While the workflow is running, read on to learn more about the logic.
Engineers might make arbitrary changes in the deployment repository, but you must automate only the image change. To distinguish between the image change and other changes, it uses git notes to explicitly annotate the image promotion commit. The workflow checks the commit note, and if it contains the note that matches theimage: <image> pattern, then propagates the image change to the staging and production environments.
Remember the approval step requirement. The approval step is implemented using GitHub Pull Request. Instead of updating the production environment directly, workflow pushes the changes to the auto-promotion-prod branch and creates a pull request, so to approve the change approver needs to merge the pull request.
To demonstrate the promotion workflow, make a change in your demo-app repo. Some suggestions:
- Add another exclamation mark to the output.
- Change the colour of the text.
- Break it somehow, promote that bug into prod at 17:00 on a Friday, then see how quickly you can get the fix to prod.
  - The real challenge is creating the bug since it fails fast (can't just be a syntax error because the build will fail.)
  - Use the colour brown as it is invalid. This fix (and the initial feature depending on the fix) will be out to prod in ~2:30. Plus, there was no actual downtime since the pod never became healthy, and the original replicaSet stayed up.

8. Rendered Manifests

Using config management tools, like Kustomize, makes it much easier to maintain the environment-specific configuration. However, it also makes it much more challenging to determine what exactly changed in the resulting manifests. This is a critical requirement for production environments. Luckily there is a solution for this, the Rendered Manifests pattern!

The idea is to create a branch with plain YAML manifests generated by the config management tool (i.e., Kustomize). The process of maintaining rendered manifests branches, of course, should be automated. A Continuous Integration system, like GitHub Actions, is a perfect tool for this task.

8.1. Adding Render Manifests to the `demo-app-deploy` Repo

The demo-app-deploy repo contains the application's manifests, so you'll add a workflow to generate the rendered manifests.

Navigate to your demo-app-deploy repo.
Open .github/workflows/rendered-manifests.yaml:
- Uncomment the render-manifests job.
- Delete the placeholder-job.
- Commit and push the changes.
- Once the workflow is created, it will create a branch for each environment and push the rendered manifests to the branch.
Navigate to your control-plane repository to update the teams Helm Chart to use the rendered manifests from the branches.

Update the spec.template.spec.source.targetRevision to {{ "{{" }}path{{ "}}" }}

Update the spec.template.spec.source.path to ./

# charts/team/templates/repo-appset.yaml

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: '{{ $.Release.Name }}-{{ . | trimSuffix "-deploy" }}'
spec:
  template:
    spec:
      source:
--      targetRevision: 'HEAD'
--      path: '{{ "{{" }}path{{ "}}" }}'
++      targetRevision: '{{ "{{" }}path{{ "}}" }}'
++      path: './'

Wait for the ApplictionSet to reconcile and see that nothing changes in the Applications since the rendered manifests are the same as what was already deployed.

9. Notifications

Notifications can be self-implemented by Argo CD users, for example, by using Resource Hooks. The hook is a pod that runs before or after the Application is synced, making it convenient for sending a notification. However, this requires every engineer to find answers to questions like:

How to get the notification service credentials?
Where to store the credentials?
What should be a notifiction message?
Where to send it?

Most answers are the same for every team except notification destinations. That is why it is better to leverage the Argo CD Notifications controller that enables configuring notifications in a centralized way and for end users to specify where and when to send notifications.

9.1. Setting Up the GitHub Notification Service

Argo CD Notifications supports integration with various notification services like Slack, PagerDuty, OpsGenie and more.

Typical examples are sending Slack or Email notifications; however, this tutorial will use the GitHub service to add Argo CD to commit statuses. For the sake of simplicity, the GitHub App created for SSO can be repurposed.

Update the permissions on the GitHub App to have write access on commit statuses.
- Navigate to: https://github.com/settings/apps/adv-gitops-<username>/permissions, replacing <username> with your GitHub username.
- Under Repository permissions > Commit statuses, select the Read and write access level.
- Click Save changes at the bottom of the page.
Generate a private key for the GitHub App.
- Navigate to: https://github.com/settings/apps/adv-gitops-<username>, replacing <username> with your GitHub username.
- Under Private keys, click Generate a private key.
- The .pem file containing the private key will download to your machine.
Install the GitHub App in your demo-app-deploy repo.
- Navigate to https://github.com/settings/apps/adv-gitops-<username>/installations, replacing <username> with your GitHub username.
- Click Install.
- Select Only select repositories.
- Select your demo-app-deploy repo.
- Click Install.
- Copy the installation ID from the URL
```
https://github.com/settings/installations/<install-id>
```

Configure the notification service in the settings for the Argo CD instance on the Akuity Platform.

From the Argo CD instance on the Akuity Platform, navigate to Settings > Notifications.
Go to the YAML Editor tab.

Paste the following snippet replacing:

<app-id> with the App ID in the settings of the GitHub App.
<install-id> with the value copied from the installation URL in the previous step.

service.github: |
  appID: <app-id>
  installationID: <install-id>
  privateKey: $github-privateKey
trigger.on-deployed: |
  - description: Application is synced and healthy. Triggered once per commit.
    oncePer: app.status.operationState.syncResult.revision
    send:
      - app-deployed
    when: app.status.operationState.phase in ['Succeeded'] and
      app.status.health.status == 'Healthy'
template.app-deployed: >
  message: |
    Application {{.app.metadata.name}} is now running new version of deployments manifests.
  github:
    status:
      state: success
      label: "argo-cd/{{.app.metadata.name}}"
      targetURL: "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}?operation=true"

Go to the Secrets tab.
Add the key github-privateKey and paste the contents of the .pem file containing the Private Key for the GitHub App as the value.
Click Save.

You can think about triggers as a function that continuously checks the Argo CD application state and returns true if the notification should be sent. The notification is sent when trigger's return value changes from false to true so that the user gets notified about an event only once when it happens.

trigger.on-deployed: |
  - description: Application is synced and healthy. Triggered once per commit.
    oncePer: app.status.operationState.syncResult.revision
    send:
      - app-deployed
    when: app.status.operationState.phase in ['Succeeded'] and
      app.status.health.status == 'Healthy'

In the triggers configuration, you might notice the send field. This field specifies the notification template that should be used to generate the notification content.

template.app-deployed: >
  message: |
    Application {{.app.metadata.name}} is now running new version of deployments manifests.
  github:
    status:
      state: success
      label: "argo-cd/{{.app.metadata.name}}"
      targetURL: "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}?operation=true"

Templates are using Go templates syntax. The template configuration includes the message field that is common for all notification services. You might configure service-specific fields like email.subject or slack.attachements for customized notifications for each notification service.

9.2. Adding the Notification Subscription

Once templates and notifications are configured in Argo CD, the end users must create a subscription. To subscribe to notifications, add the following annotation to the ApplicationSet template in the team chart:

Add the following annotation to the spec.template.metadata of the ApplicationSet using the demo-app-deploy repo that the GitHub App has access to

    template:
      metadata:
        name: '<username>-demo-app-{{path.basename}}'
++      annotations:
++        notifications.argoproj.io/subscribe.on-deployed.github: ""

Commit and push the changes to the repo.
Sync the argocd Application to pick up the changes.

Go ahead and trigger a new deployment of the demo-app. Once the new image has been built, the rendered manifests are updated on the demo-app-deploy repo, and the Application has been synced and healthy, the commit will show a status check with a link to the operation on in Argo CD.

Commit Status Check using GitHub Notification Service

10. Secrets

Secrets management is another task that could be solved by teams independently. However, in this case, every team would have to go through the same journey, and it is better to provide a centralized solution. More importantly, the solution must work and be secure and future-proof. Some available options are:

Of all of the above solutions, the best fit for Argo CD is External Secrets. You will go ahead and try using it.

Thanks to the simplicity of the SecretStore provided by External Secrets (the external-secrets.io/v1beta1 API), integration with a backend, like HashiCorp Vault, is nothing special:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-backend
spec:
  provider:
    vault:
      server: "http://my.vault.server:8200"
      path: "secret"
      # Version is the Vault KV secret engine version.
      # This can be either "v1" or "v2", defaults to "v2"
      version: "v2"
      auth:
        # points to a secret that contains a vault token
        # https://www.vaultproject.io/docs/auth/token
        tokenSecretRef:
          name: "vault-token"
          key: "token"
---
apiVersion: v1
kind: Secret
metadata:
  name: vault-token
data:
  token: cm9vdA== # For the sake of simplicity this uses a static token `root`.

For the tutorial, you'll use a fake Secret Store that will "simulate" using Hashicorp Vault as the Secret Store.

Using the addons ApplicaitonSet, the cluster ApplicationSet, and the configurations from the control-plane repository, the workshop cluster has already been configured with External Secrets.

10.1. Add an `ExternalSecret` to the `demo-app`

Add a secret to the demo-app using GitOps!

Open the file base/secret.yaml in your demo-app-deploy repo.
Replace <replace> with a value between /value1 through to /value10
- The options are available in the clusters/workshop/secret-storage.yaml file in the control-plane repository.
Next, open the file base/kustomization.yaml.

Uncomment the line for secret.yaml file under resources:.

   resources:
   - deployment.yaml
-- # - secret.yaml
++ - secret.yaml
   - service.yaml

To configure the Deployment to use the secret, uncomment the env section in the base/deployment.yaml manifest.

env:
- name: SECRET
  valueFrom:
    secretKeyRef:
      name: my-secret
      key: secret-key

Commit and push the changes to the repo.

Trigger a restart of your demo-app Deployment, then check the logs!

11. Summary

To summarize what you've built so far, you have a multi-tenant Argo CD instance. The instance is managed using a GitOps-based process where engineers can self-onboard their team by creating a pull request. Each application development team independently manages their application by maintaining a separate deployment repository. Deployment manifests are generated using Kustomize, allowing engineers to avoid duplications and efficiently introduce environment-specific changes. You improved the developer's quality of life by providing features like notifications and secret management.

The simple image tag changes are fully automated using the CI pipelines. On top of this, teams leverage the "rendered manifests" pattern that turns to get Git history into a traceable audit log of every infrastructure change. Believe it or not, you've achieved more than many companies in several years.

12. Clean Up

Complete this clean-up checklist after completing the tutorial. After doing so, you can follow the tutorial from the start again.

Delete the Argo CD instance on the Akuity Platform.
Delete the adv-gitops-<username> GitHub App for SSO and notifications.
Delete the control-plane, demo-app, and demo-app-deploy repos.
Delete the workshop cluster.
Delete the demo-app package from your GitHub account.
Delete GitHub PATs generated.

Advanced GitOps Tutorial

1. Overview​

1.1. Prerequisites​

1.2. Repositories​

2. Control Plane Setup​

2.1. Automating Application Creation with ApplicationSets​

2.2. Creating an Argo CD Instance​

2.3. Integrating GitHub SSO​

2.4. Creating the control-plane Repository​

2.5. Bootstrapping with App of Apps​

3. Building Images​

3.1. Creating the demo-app Repo​

4. Manifest Management​

4.1. Creating the demo-app-deploy Repo​

5. Self-service Teams​

5.1. Creating a Team Environment​

6. Let's deploy something!​

6.1. Update The Image Tag for the dev Environment​

7. Automating Image Tag Updates​

7.1. Automating the dev Environment​

7.2. Automating Promotion to stage and prod​

8. Rendered Manifests​

8.1. Adding Render Manifests to the demo-app-deploy Repo​

9. Notifications​

9.1. Setting Up the GitHub Notification Service​

9.2. Adding the Notification Subscription​

10. Secrets​

10.1. Add an ExternalSecret to the demo-app​

11. Summary​

12. Clean Up​