Advanced GitOps Tutorial
- 1. Overview
- 2. Control Plane Setup
- 3. Building Images
- 4. Manifest Management
- 5. Self-service Teams
- 6. Let's deploy something!
- 7. Automating Image Tag Updates
- 8. Rendered Manifests
- 9. Notifications
- 10. Secrets
- 11. Summary
- 12. Clean Up
1. Overview
The Advanced GitOps tutorial, presented by Akuity, will take you through creating a self-service multi-tenant Kubernetes environment using Argo CD and GitHub Actions. The tutorial will cover the following topics:
Automating
Application
generation usingApplicationSet
s.Using GitHub for Single Sign-on (SSO).
Managing manifests for promotion between environments.
Enabling self-service environment creation for teams.
Integrating notifications and secrets with Argo CD.
Naturally, the Argo CD configuration is managed by the cluster administrators. However, developers will not need administrators to create their environments. With GitOps, teams can self-onboard, leveraging Helm Charts provided by the administrators to abstract the resources that comprise an environment and pull requests to propose the creation of and changes to them.
1.1. Prerequisites
The tutorial assumes you have experience working with Kubernetes, GitOps, GitHub Actions (or a similar CI engine), and Argo CD. Given such, some underlying concepts won't be explained during this tutorial.
The tutorial requires that you have the following:
a GitHub Account - you will use this to:
host repositories for the control plane, demo application, and deployment configuration.
run GitHub Actions. (The repositories are public, so there is no cost.)
create a GitHub App for integrating SSO and notifications.
create an Argo CD instance on the Akuity Platform.
a dedicated Kubernetes cluster with Cluster Admin access.
This tutorial does not require a publicly accessible cluster, only egress traffic (i.e., internet access). Consider using Docker Desktop and kind your local machine.
Admin access is required to create namespaces and cluster roles.
the Kubernetes command-line tool, kubectl.
a local IDE with access set up to your GitHub.
tipThroughout this tutorial, all changes to repositories can be done from your browser by changing
github.com
togithub.dev
in the URL for the repository. See the GitHub docs for more details.a browser with internet access.
This tutorial shows placeholder text between less-than and greater-than symbols (i.e., <...>
), indicating that you must substitute it with the value relevant to your scenario.
<username>
- Your GitHub username.
1.2. Repositories
The tutorial will use the following repositories:
The
control-plane
repository defines the desired state of the Kubernetes platform and enables self-service onboarding for teams. It contains the following directories:argocd
- configuration for Argo CD (e.g.,Application
s,AppProject
s)charts
- local and Umbrella Helm Charts.clusters
- cluster-specific configurations (e.g.,ClusterSecretStore
).teams
- a folder for each team.
The
demo-app
repository contains the source code for a simple Golang application packaged into a container image used to demonstrate Argo CD driven GitOps processes and CI.The
demo-app-deploy
repository uses Kustomize to define the manifests to deploy thedemo-app
and contains the following directories:base/
- the application manifests agnostic to any environment.deployment.yaml
- defines theDeployment
.service.yaml
- defines theService
.
env/
- the overlays specific to environments.dev/
stage/
prod/
2. Control Plane Setup
The control plane setup assumes that you have already prepared a Kubernetes cluster with internet access.
2.1. Automating Application
Creation with ApplicationSet
s
Each cluster will get two addons (kyverno
and external-secrets
), the cluster-specific configurations, and Namespace
s for the teams. Each requires an Application
in Argo CD to install them into each cluster.
Manually creating Application
s is error-prone and tedious. Scripting config management tools can automate the process, but there is a better way. ApplicationSet
s template Application
s and populate them using generators.
These are the ApplicationSet
s in the control-plane
repo:
argocd/clusters-appset.yaml
will generate anApplication
for each cluster registered to Argo CD (excluding thein-cluster
) and point it to theclusters/{{name}}
folder in thecontrol-plane
repo.argocd/addons-appset.yaml
will generate anApplication
for each combination of cluster registered to Argo CD (excluding thein-cluster
) and Helm Chart folder in thelist
generator.# generators
- list:
elements: # The value in the pairs below are the folder name from `charts/`.
- addonChart: external-secrets
- addonChart: kyverno
- clusters:
selector:
matchExpressions:
- {key: 'akuity.io/argo-cd-cluster-name', operator: NotIn, values: [in-cluster]}The
addons-values.yaml
file from thecluster/
folder is passed to each Helm Chart. Using Umbrella charts (setting sub-charts as dependencies in the local chart), the same values file can be passed to each chart, without fear of conflicts.# cluster/{{name}}/addons-values.yaml
external-secrets: {}
kyverno: {}
# values file from cluster folder
source:
helm:
valueFiles:
- '../../clusters/{{name}}/addons-values.yaml'argocd/teams-appset.yaml
will generate anApplication
for each folder inteams/
. The resultingApplication
will use thecharts/team
Helm Chart with thevalues.yaml
from the team's folder.
# File structure of the `teams` folder.
teams/
<username>/
values.yaml
# ApplicationSet `git` generator.
- git:
repoURL: https://github.com/<username>/control-plane
revision: HEAD
directories:
- path: "teams/*"
# Application spec template using the values file.
source:
helm:
releaseName: '{{path.basename}}' # The team name.
valueFiles:
- '../../{{path}}/values.yaml' # The team's folder.In the
team
Helm Chart, therepo-appset.yaml
template will create anApplicationSet
for each item in therepos
value. Each one will generate anApplication
for the repository name and all the folders found underenv/
.{{- range .Values.repos }}
---
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: '{{ $.Release.Name }}-{{ . | trimSuffix "-deploy" }}'
spec:
generators:
- git:
repoURL: 'https://github.com/{{ $.Release.Name }}/{{ . }}'
revision: HEAD
directories:
- path: env/*
template:
metadata:
name: '{{ $.Release.Name }}-{{ . | trimSuffix "-deploy" }}-{{ "{{" }}path.basename{{ "}}" }}'
spec:
project: '{{ $.Release.Name }}'
source:
repoURL: 'https://github.com/{{ $.Release.Name }}/{{ . }}'
targetRevision: 'HEAD'
path: '{{ "{{" }}path{{ "}}" }}'
destination:
name: '{{ $.Values.cluster }}'
namespace: '{{ $.Release.Name }}-{{ "{{" }}path.basename{{ "}}" }}'
{{- end }}- Notice the use of
{{ "{{" }}
in thepath
of the template; this is to prevent Helm from interpreting theApplicationSet
template ({{path}}
) syntax during templating of the chart.
- Notice the use of
2.2. Creating an Argo CD Instance
The tutorial demonstrates deploying resources to clusters connected to Argo CD. You will create an Argo CD instance using the Akuity Platform to simplify the installation and connecting clusters.
Log into the Akuity Platform with your GitHub account.
- If you don't already have an organization (i.e., it's your first time logging in), create one naming it using your GitHub username.
Create an Argo CD instance on the Akuity Platform with the name
adv-gitops
(or any permitted string).Enable Declarative Management for the Argo CD instance.
infoDeclarative Management enables using the Akuity Platform with the App of Apps pattern and
ApplicationSet
s by exposing thein-cluster
destination. Using thein-cluster
destination means an Application can create other Applications in the cluster hosting Argo CD.Provision an agent for a cluster named
workshop
.cautionUsing the name
workshop
is crucial because the manifests provided for the tutorial assume this is thedestination.name
available forApplication
s in Argo CD.Deploy the agent to the cluster.
2.3. Integrating GitHub SSO
Argo CD provides a basic implementation for user management with the expectation that most organizations will set up Single Sign-On (SSO). You will use GitHub as the SSO provider by creating a GitHub App.
In it's default configuration, this will allow anyone with a GitHub account to log into your Argo CD instance and assume the readonly
role from the default policy.
Set GitHub App name to
adv-gitops-<username>
, replacing<username>
with your GitHub username.cautionGitHub App names are globally unique, so including the
<username>
is essential.Set Homepage URL to
https://<instance-id>.cd.akuity.cloud/
, replacing<instance-id>
with the ID for the Argo CD instance on the Akuity Platform.https://<instance-id>.cd.akuity.cloud/
Set Authorization callback URL to
https://<instance-id>.cd.akuity.cloud/api/dex/callback
, replacing<instance-id>
with the ID for the Argo CD instance on the Akuity Platform.https://<instance-id>.cd.akuity.cloud/api/dex/callback
Under the "Webhook" section, deselect the Active option.
Under the "Permissions" section, expand the Account permissions then set Email addresses to
Read-only
access level.infoThe
Read-only
access for Email addresses is required for GitHub users that have their email address set to private.Click Create GitHub App.
In the settings for the GitHub App, under "Client secrets", click Generate a new client secret.
Back on the Akuity Platform, in the Settings for the Argo CD instance, go to SSO Configuration > Configuration and click Add new connector
Set Type to
github
Click Add
Go to SSO Configuration > Secrets.
Add key
GITHUB_CLIENT_SECRET
and set the value to the client secret generated on the GitHub App.Add key
GITHUB_CLIENT_ID
and set the value to the "Client ID" from the GitHub App.Click Save.
Configure the RBAC policies.
In the Settings for the Argo CD instance, go to RBAC.
Under OIDC Scopes, click Add Scope and enter
email
.Under Policy, add this line to make yourself an
admin
. Replace<email>
with your GitHub primary email address.g, <email>, role:admin
Click Save.
After the instance has finished progressing, log into Argo CD using GitHub SSO.
From the Akuity Platform dashboard for the Argo CD instance, click the
<instance-id>.cd.akuity.cloud
link.Click LOG IN VIA GITHUB.
Click Authorize akuity-adv-gitops-workshop-<event>.
You are now logged into Argo CD by authenticating with your GitHub account. By setting the OICD scope to email
and adding a policy to assign your email to the admin
role, you can perform any function within Argo CD.
2.4. Creating the control-plane
Repository
The control-plane
repository is provided as a template on GitHub. For the tutorial, you will create a repository on your account using the template and update the placeholder text within it.
Create a new public repoistory named
control-plane
fromakuity-adv-gitops-workshop/control-plane-template
.Set the Owner to your account.
Set the Repository name to
control-plane
.Select Public for the visibility of the repository.
Click Create repository from template.
Open the repository, then find and replace
<username>/control-plane
with the new repo URL.-- repoURL: https://github.com/<username>/control-plane
++ repoURL: https://github.com/morey-tech/control-planetipChanging
github.com
togithub.dev
in the URL will open up VSCode in your browser. From there, you can use the search and replace feature.Throughout this tutorial, all repository changes can be done from your browser.
2.5. Bootstrapping with App of Apps
To enable the GitOps process, a top-level Application
is used to manage the Argo CD configuration and propagate repository changes to the argocd
namespace on the in-cluster
destination (i.e., the Akuity Platform control plane).
Navigate to the Argo CD UI.
Create an Application to manage the Argo CD configuration using the
argocd-app.yaml
manifest at the root of thecontrol-plane
repo.Click + NEW APP.
Click EDIT AS YAML.
Paste the contents of
argocd-app.yaml
.Click SAVE.
Click CREATE.
After creating the argocd
Application
, the automated sync policy deployed the addons
, clusters
, teams
ApplicationSet
s, along with the default
AppProject
(assuming ownership of the existing project).
The clusters
ApplicationSet
generated an Application
for the workshop
cluster, which created a couple of Kyverno ClusterPolicy
resources and an External Secrets SecretStore
.
The addons
ApplicationSet
generated a Kyverno and External Secrets Application
for the workshop
cluster.
The addons
create the CRDs on which resources in the cluster
Application
depend. This setup relies on the sync retry mechanism to make the resources eventually consistent.
3. Building Images
Now that infrastructure configuring is done, it's time to set up the demo-app
.
3.1. Creating the demo-app
Repo
To deploy an application using Argo CD, you need a container image. It would be boring if you deployed a pre-built image, so you will build one specific to you!
You are going to leverage GitHub Actions to build an image and push it to the GitHub Container Registry (GHCR). The workflow is already defined in the repository. You only need to create a repository from the template on your account and make a small change customizing it to you, which will trigger the workflow.
Create a new public repoistory named
demo-app
fromakuity-adv-gitops-workshop/demo-app-template
.Set the Owner to your GitHub account.
Set the Repository name to
demo-app
.Select Public for the visibility of the repository.
Click Create repository from template.
Create a new classic (not fine-grained) personal access token (PAT) with write access for repositories and packages.
Yes, a personal access token can be less safe, but you will use it during the tutorial for simplicity.
Navigate to:
https://github.com/settings/tokens/new
Set the Note to
demo-app
.Select the
repo
andwrite:package
scopes.Click Generate token.
Add the PAT to your
demo-app
repository secrets asDEPLOY_PAT
:Navigate to
https://github.com/<username>/demo-app/settings/secrets/actions/new
Click New repository secret.
Set the Name to
DEPLOY_PAT
.Paste the PAT as the Secret.
Click Add secret.
Edit the
main.go
file and replace the<username>
placeholder with your GitHub username.Push the changes to trigger the build workflow and wait for it to complete.
noteCreating the repo will have triggered a run of the workflow too. That run will have failed due to the missing secret containing the PAT.
View the image created in the GitHub Packages page and update the visibility of the
demo-app
package toPublic
.Navigate to
https://github.com/users/<username>/packages/container/demo-app/settings
- Danger Zone > Change package visibility, set to `Public`.
The demo-app
image is avaiable at ghcr.io/<username>/demo-app
with the latest
and the commit id (SHA) tags.
4. Manifest Management
The image is ready, but you need the Kubernetes manifests to deploy the application. To define the deployment manifests, you will leverage a config management tool, Kustomize.
4.1. Creating the demo-app-deploy
Repo
The demo-app-deploy
repository is provided as a template on GitHub. For the tutorial, you will create a repository on your account using the template and update the placeholder text within it.
Create a new public repository named
demo-app-deploy
fromakuity-adv-gitops-workshop/demo-app-deploy-template
.Set the Owner to your GitHub account.
Set the Repository name to
demo-app-deploy
.Select Public for the visibility of the repository.
Click Create repository from template.
Create (another) new classic (not fine-grained) PAT with write access for repositories.
Set the Note to
demo-app-deploy
.Select the
repo
scopes.Click Generate token.
Add the PAT to your
demo-app-deploy
repository secrets asDEPLOY_PAT
:Navigate to
https://github.com/<username>/demo-app-deploy/settings/secrets/actions/new
Click New repository secret.
Set the Name to
DEPLOY_PAT
.Paste the PAT as the Secret.
Click Add secret.
Open the repository, then find and replace
<username>/demo-app
, updating the placeholder to your GitHub username.-- image: ghcr.io/<username>/demo-app:latest
++ image: ghcr.io/morey-tech/demo-app:latest
5. Self-service Teams
Each team needs an AppProject
, a Namespace
for each environment, and Application
s. A team can request to create these resources, but they are ultimately managed by the administrators who approve the Pull Request.
The teams are provided with a Helm Chart, charts/teams
to abstract the concepts of Namespaces, AppProject
s, and ApplicationSet
s. Instead, they must create a new folder under teams/
and add the example values.yaml
from teams/USERNAME
.
5.1. Creating a Team Environment
You will create a team for the tutorial using your GitHub username.
Open the
control-plane
repository.Copy the
teams/USERNAME
folder and create a new folder with your GitHub username (i.e.,teams/<username>
).Update the
values.yaml
file with your information.- Set the
email
value (replacing<email>
) to the email displayed on the User Info page in Argo CD (which should be your primary GitHub email address).
- Set the
Commit changes and create a pull request to the
main
branch.
On the Applications page of Argo CD, add your project to the filter to experience what a team would see.
Once the teams
ApplicationSet
has detected the change in Git, it will create an Application
for the Namespace
s on the cluster, the AppProject
for the team, and an ApplicationSet
for each repo in the values.yaml
.
The demo-app
is deployed using the latest
tag to each namespace. Check out the Deployment
logs to confirm that your <username>
is being printed.
6. Let's deploy something!
It's time to demonstrate deploying a new release of the demo-app
.
6.1. Update The Image Tag for the dev
Environment
Let's start by replacing the latest
tag with the commit ID tag in the dev
environment. You will use the tag built after setting up the demo-app
repo.
Navigate to the
demo-app
packages page:https://github.com/users/<username>/packages/container/package/demo-app
Copy the most recent image tag using the commit ID (SHA) format (e.g.,
923d0b0f341ebf0a9aa7f7597a18e4db690fe1b2
).Configure the
dev
environment to use the image.Open your
demo-app-deploy
repo.In the
env/dev/kustomization.yaml
file, replacelatest
with the commit ID (SHA) tag.images:
- name: ghcr.io/<username>/demo-app
newTag: latestinfoThis is using the Kustomize edit image feature.
Commit the changes and push them to the repository.
7. Automating Image Tag Updates
Updating the image tag in the kustomization.yaml
is boring, and no one likes to make trivial tag changes manually. To make the process more efficient, you are going to automate this process with GitHub Actions!
7.1. Automating the dev
Environment
The dev
environment is intended to track the trunk of the repository (i.e., the main branch), and there is minimal impact if a change breaks the application. Given this, you will automate the deployment to dev
each time a new build is created in the demo-app
repo.
Navigate to your
demo-app
repository.Uncomment the
deploy-dev
job in.github/workflows/ci.yaml
workflow file:
deploy-dev:
runs-on: ubuntu-latest
needs: build-image
steps:
- uses: imranismail/setup-kustomize@v1
- name: Update env/dev image with Kustomize
run: |
git config --global user.name "Deploy Bot"
git config --global user.email "no-reply@akuity.io"
git clone https://bot:${{ secrets.DEPLOY_PAT }}@github.com/${{ github.repository_owner }}/demo-app-deploy.git
cd demo-app-deploy/env/dev
kustomize edit set image ghcr.io/${{ github.repository_owner }}/demo-app:${{ github.sha }}
git commit -a -m "chore(dev): deploy demo-app:${{ github.sha }}"
git notes append -m "image: ghcr.io/${{ github.repository_owner }}/demo-app:${{ github.sha }}"
git push origin "refs/notes/*" --force && git push --force
Commit and push the changes, then watch the action run!
When the workflow completes (successfully), check out the latest commit on your demo-app-deploy
repo.
Once changes are pushed, the CI workflow will build a new image and update the dev
environment with the corresponding image tag. Developers no longer need to manually change deployment manifests to update the dev
environment.
7.2. Automating Promotion to stage
and prod
Updating the manifests for the stage
and prod
deployments requires a more careful process, but it can also be automated.
Allow GitHub Actions to create and approve pull requests on the
demo-app
repo.Navigate to:
https://github.com/<username>/demo-app-deploy/settings/actions
Under Workflow permissions, check the box for Allow GitHub Actions to create and approve pull requests.
Click Save.
In your
demo-app-deploy
repo, open.github/workflows/promote.yaml
:Uncomment the
promote-image-change
job.Delete the
placeholder-job
.Commit and push the changes.
While the workflow is running, read on to learn more about the logic.
Engineers might make arbitrary changes in the deployment repository, but you must automate only the image change. To distinguish between the image change and other changes, it uses git notes to explicitly annotate the image promotion commit. The workflow checks the commit note, and if it contains the note that matches the
image: <image>
pattern, then propagates the image change to the staging and production environments.Remember the approval step requirement. The approval step is implemented using GitHub Pull Request. Instead of updating the production environment directly, workflow pushes the changes to the
auto-promotion-prod
branch and creates a pull request, so to approve the change approver needs to merge the pull request.To demonstrate the
promotion
workflow, make a change in yourdemo-app
repo. Some suggestions:- Add another exclamation mark to the output.
- Change the colour of the text.
- Break it somehow, promote that bug into
prod
at17:00
on a Friday, then see how quickly you can get the fix to prod.- The real challenge is creating the bug since it fails fast (can't just be a syntax error because the build will fail.)
- Use the colour
brown
as it is invalid. This fix (and the initial feature depending on the fix) will be out to prod in~2:30
. Plus, there was no actual downtime since the pod never became healthy, and the original replicaSet stayed up.
8. Rendered Manifests
Using config management tools, like Kustomize, makes it much easier to maintain the environment-specific configuration. However, it also makes it much more challenging to determine what exactly changed in the resulting manifests. This is a critical requirement for production environments. Luckily there is a solution for this, the Rendered Manifests pattern!
The idea is to create a branch with plain YAML manifests generated by the config management tool (i.e., Kustomize). The process of maintaining rendered manifests branches, of course, should be automated. A Continuous Integration system, like GitHub Actions, is a perfect tool for this task.
8.1. Adding Render Manifests to the demo-app-deploy
Repo
The demo-app-deploy
repo contains the application's manifests, so you'll add a workflow to generate the rendered manifests.
Navigate to your
demo-app-deploy
repo.Open
.github/workflows/rendered-manifests.yaml
:Uncomment the
render-manifests
job.Delete the
placeholder-job
.Commit and push the changes.
Once the workflow is created, it will create a branch for each environment and push the rendered manifests to the branch.
Navigate to your
control-plane
repository to update theteams
Helm Chart to use the rendered manifests from the branches.
Update the
spec.template.spec.source.targetRevision
to{{ "{{" }}path{{ "}}" }}
Update the
spec.template.spec.source.path
to./
# charts/team/templates/repo-appset.yaml
apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: '{{ $.Release.Name }}-{{ . | trimSuffix "-deploy" }}'
spec:
template:
spec:
source:
-- targetRevision: 'HEAD'
-- path: '{{ "{{" }}path{{ "}}" }}'
++ targetRevision: '{{ "{{" }}path{{ "}}" }}'
++ path: './'
Wait for the ApplictionSet to reconcile and see that nothing changes in the Application
s since the rendered manifests are the same as what was already deployed.
9. Notifications
Notifications can be self-implemented by Argo CD users, for example, by using Resource Hooks. The hook is a pod that runs before or after the Application
is synced, making it convenient for sending a notification. However, this requires every engineer to find answers to questions like:
- How to get the notification service credentials?
- Where to store the credentials?
- What should be a notifiction message?
- Where to send it?
Most answers are the same for every team except notification destinations. That is why it is better to leverage the Argo CD Notifications controller that enables configuring notifications in a centralized way and for end users to specify where and when to send notifications.
9.1. Setting Up the GitHub Notification Service
Argo CD Notifications supports integration with various notification services like Slack, PagerDuty, OpsGenie and more.
Typical examples are sending Slack or Email notifications; however, this tutorial will use the GitHub service to add Argo CD to commit statuses. For the sake of simplicity, the GitHub App created for SSO can be repurposed.
Update the permissions on the GitHub App to have write access on commit statuses.
Navigate to:
https://github.com/settings/apps/adv-gitops-<username>/permissions
, replacing<username>
with your GitHub username.Under Repository permissions > Commit statuses, select the
Read and write
access level.Click Save changes at the bottom of the page.
Generate a private key for the GitHub App.
Navigate to:
https://github.com/settings/apps/adv-gitops-<username>
, replacing<username>
with your GitHub username.Under Private keys, click Generate a private key.
The
.pem
file containing the private key will download to your machine.
Install the GitHub App in your
demo-app-deploy
repo.Navigate to
https://github.com/settings/apps/adv-gitops-<username>/installations
, replacing<username>
with your GitHub username.Click Install.
Select Only select repositories.
Select your
demo-app-deploy
repo.Click Install.
Copy the installation ID from the URL
https://github.com/settings/installations/<install-id>
Configure the notification service in the settings for the Argo CD instance on the Akuity Platform.
From the Argo CD instance on the Akuity Platform, navigate to Settings > Notifications.
Go to the YAML Editor tab.
Paste the following snippet replacing:
<app-id>
with the App ID in the settings of the GitHub App.<install-id>
with the value copied from the installation URL in the previous step.
service.github: |
appID: <app-id>
installationID: <install-id>
privateKey: $github-privateKey
trigger.on-deployed: |
- description: Application is synced and healthy. Triggered once per commit.
oncePer: app.status.operationState.syncResult.revision
send:
- app-deployed
when: app.status.operationState.phase in ['Succeeded'] and
app.status.health.status == 'Healthy'
template.app-deployed: >
message: |
Application {{.app.metadata.name}} is now running new version of deployments manifests.
github:
status:
state: success
label: "argo-cd/{{.app.metadata.name}}"
targetURL: "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}?operation=true"Go to the Secrets tab.
Add the key
github-privateKey
and paste the contents of the.pem
file containing the Private Key for the GitHub App as the value.Click Save.
You can think about triggers as a function that continuously checks the Argo CD application state and returns true
if the notification should be sent. The notification is sent when trigger's return value changes from false
to true
so that the user gets notified about an event only once when it happens.
trigger.on-deployed: |
- description: Application is synced and healthy. Triggered once per commit.
oncePer: app.status.operationState.syncResult.revision
send:
- app-deployed
when: app.status.operationState.phase in ['Succeeded'] and
app.status.health.status == 'Healthy'
In the triggers configuration, you might notice the send
field. This field specifies the notification template that should be used to generate the notification content.
template.app-deployed: >
message: |
Application {{.app.metadata.name}} is now running new version of deployments manifests.
github:
status:
state: success
label: "argo-cd/{{.app.metadata.name}}"
targetURL: "{{.context.argocdUrl}}/applications/{{.app.metadata.name}}?operation=true"
Templates are using Go templates syntax. The template configuration includes the message
field that is common for all notification services. You might configure service-specific fields like email.subject
or slack.attachements
for customized notifications for each notification service.
9.2. Adding the Notification Subscription
Once templates and notifications are configured in Argo CD, the end users must create a subscription. To subscribe to notifications, add the following annotation to the ApplicationSet
template in the team
chart:
Add the following annotation to the
spec.template.metadata
of theApplicationSet
using thedemo-app-deploy
repo that the GitHub App has access totemplate:
metadata:
name: '<username>-demo-app-{{path.basename}}'
++ annotations:
++ notifications.argoproj.io/subscribe.on-deployed.github: ""Commit and push the changes to the repo.
Sync the
argocd
Application
to pick up the changes.
Go ahead and trigger a new deployment of the demo-app
. Once the new image has been built, the rendered manifests are updated on the demo-app-deploy
repo, and the Application
has been synced and healthy, the commit will show a status check with a link to the operation on in Argo CD.
10. Secrets
Secrets management is another task that could be solved by teams independently. However, in this case, every team would have to go through the same journey, and it is better to provide a centralized solution. More importantly, the solution must work and be secure and future-proof. Some available options are:
- Sealed Secrets
- Argo CD Vault Plugin
- SOPS (Secrets OPerationS)
- Vault Agent
- Secrets Store CSI Driver
- External Secrets
Of all of the above solutions, the best fit for Argo CD is External Secrets. You will go ahead and try using it.
Thanks to the simplicity of the SecretStore
provided by External Secrets (the external-secrets.io/v1beta1
API), integration with a backend, like HashiCorp Vault, is nothing special:
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: vault-backend
spec:
provider:
vault:
server: "http://my.vault.server:8200"
path: "secret"
# Version is the Vault KV secret engine version.
# This can be either "v1" or "v2", defaults to "v2"
version: "v2"
auth:
# points to a secret that contains a vault token
# https://www.vaultproject.io/docs/auth/token
tokenSecretRef:
name: "vault-token"
key: "token"
---
apiVersion: v1
kind: Secret
metadata:
name: vault-token
data:
token: cm9vdA== # For the sake of simplicity this uses a static token `root`.
For the tutorial, you'll use a fake
Secret Store that will "simulate" using Hashicorp Vault as the Secret Store.
Using the addons
ApplicaitonSet
, the cluster
ApplicationSet
, and the configurations from the control-plane
repository, the workshop
cluster has already been configured with External Secrets.
10.1. Add an ExternalSecret
to the demo-app
Add a secret to the demo-app
using GitOps!
Open the file
base/secret.yaml
in yourdemo-app-deploy
repo.Replace
<replace>
with a value between/value1
through to/value10
- The options are available in the
clusters/workshop/secret-storage.yaml
file in thecontrol-plane
repository.
- The options are available in the
Next, open the file
base/kustomization.yaml
.Uncomment the line for
secret.yaml
file underresources:
.resources:
- deployment.yaml
-- # - secret.yaml
++ - secret.yaml
- service.yamlTo configure the
Deployment
to use the secret, uncomment theenv
section in thebase/deployment.yaml
manifest.env:
- name: SECRET
valueFrom:
secretKeyRef:
name: my-secret
key: secret-keyCommit and push the changes to the repo.
Trigger a restart of your demo-app
Deployment
, then check the logs!
11. Summary
To summarize what you've built so far, you have a multi-tenant Argo CD instance. The instance is managed using a GitOps-based process where engineers can self-onboard their team by creating a pull request. Each application development team independently manages their application by maintaining a separate deployment repository. Deployment manifests are generated using Kustomize, allowing engineers to avoid duplications and efficiently introduce environment-specific changes. You improved the developer's quality of life by providing features like notifications and secret management.
The simple image tag changes are fully automated using the CI pipelines. On top of this, teams leverage the "rendered manifests" pattern that turns to get Git history into a traceable audit log of every infrastructure change. Believe it or not, you've achieved more than many companies in several years.
12. Clean Up
Complete this clean-up checklist after completing the tutorial. After doing so, you can follow the tutorial from the start again.
- Delete the Argo CD instance on the Akuity Platform.
- Delete the
adv-gitops-<username>
GitHub App for SSO and notifications. - Delete the
control-plane
,demo-app
, anddemo-app-deploy
repos. - Delete the
workshop
cluster. - Delete the
demo-app
package from your GitHub account. - Delete GitHub PATs generated.