🔀GitOps & Platform Engineering
Git is the source of truth. Declarative. Versioned. Automated. Reconciled. Learn ArgoCD, Flux, Crossplane, and how to build self-service Internal Developer Platforms.
Architecture & How It Works 12.1
GitOps: desired state lives in Git. Operators continuously reconcile actual state to desired state. Everything is auditable, reviewable, and reversible.
Core Components 12.2
| Tool | Key Resources | Primary Strength |
|---|---|---|
| ArgoCD | Application, AppProject, ApplicationSet | Great UI, RBAC, sync waves |
| Flux | GitRepository, Kustomization, HelmRelease | Lightweight, native GitOps |
| Crossplane | XRD, Composition, Claim | Self-service infrastructure |
| Backstage | Software catalog + templates | Internal Developer Portal |
WSL Hands-On Lab 12.3
ArgoCD + ApplicationSet + Flux + Crossplane Composition on kind
$ kubectl create ns argocd $ kubectl apply -n argocd -f https://raw.githubusercontent.com/argoproj/argo-cd/stable/manifests/install.yaml $ flux bootstrap github --owner=$GITHUB_USER --repository=platform-gitops
- Install ArgoCD on kind cluster and access the UI
- Create an Application CR and successfully sync a sample app
- Create an ApplicationSet that generates apps for multiple environments
- Bootstrap Flux and reconcile a HelmRelease
- Define a simple Crossplane Composition and claim a database
- Complete Killercoda ArgoCD and Flux labs
Real-World Project 12.4
Single ArgoCD Application for one microservice
Multi-environment deployment with ApplicationSets and sync waves
Self-service IDP: Backstage + Crossplane + ArgoCD + team self-service namespaces
Troubleshooting 12.5
Inspect events and resource health. Use server-side apply. Check for webhook or CRD upgrade issues.
Check GitRepository status, image policy, and secret access for private repos.
Review Composition readiness, provider credentials, and XRD status conditions.
30-Day Roadmap 12.6
- Application CR
- Sync policies
- RBAC & projects
- GitRepository + Kustomize
- HelmRelease
- Image automation
- ApplicationSets
- Generators
- Progressive delivery
- Crossplane XRD/Composition
- Claims for self-service
- Backstage + templates
Deep Dive: Building Real Internal Developer Platforms 12.7
Platform Components
Backstage catalog + software templates. Crossplane for self-service claims. ArgoCD ApplicationSets for environments. Golden paths documented and enforced through templates.
Measuring Platform Success
Track cognitive load reduction, deployment frequency, MTTR, and developer satisfaction surveys. A good platform makes correct things easy and wrong things hard.
Real Incident: ArgoCD Self-Managed App Deleted Itself
What happened: The ArgoCD application CR that managed ArgoCD itself was stored in the cluster (App of Apps pattern). A developer ran kubectl delete application argocd-self -n argocd believing it was a test app. ArgoCD began deleting its own resources including the application controller.
# Prevention: use RBAC to restrict argocd namespace access $ kubectl create clusterrole argocd-readonly \ --verb=get,list,watch \ --resource=applications,appprojects \ -n argocd # App of Apps: manage ArgoCD app CRs in separate protected repo # Set finalizers to prevent accidental deletion $ kubectl patch application argocd-self -n argocd \ --type=merge \ -p '{"metadata":{"finalizers":["resources-finalizer.argocd.argoproj.io"]}}' Finalizer prevents kubectl delete from completing without --cascade=false
Recovery: Re-apply ArgoCD manifests from Git. Applications reconcile automatically once controller comes back. Self-healing GitOps means recovery is just kubectl apply -k away.
Enterprise GitOps: FluxCD, ESO & Multi-Cluster 12.8
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
name: payments-api
namespace: flux-system
spec:
interval: 5m0s # Reconcile every 5 minutes
path: ./kubernetes/apps/payments-api/overlays/prod
prune: true # Delete resources removed from Git
sourceRef:
kind: GitRepository
name: fleet-infra
healthChecks: # Wait for resources to be healthy
- apiVersion: apps/v1
kind: Deployment
name: payments-api
namespace: production
timeout: 5m0s
retryInterval: 2m0s
# Dependencies: ensure namespace exists before app
dependsOn:
- name: namespaces
postBuild:
substitute:
cluster_name: prod-eks-us-east-1
environment: production
substituteFrom:
- kind: ConfigMap
name: cluster-vars# FluxCD Image Automation: auto-update image tags in Git
# New image pushed to ECR -> Flux detects -> updates values.yaml -> commits -> reconciles
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageRepository
metadata:
name: payments-api
namespace: flux-system
spec:
image: 123456789.dkr.ecr.us-east-1.amazonaws.com/payments-api
interval: 1m
secretRef:
name: ecr-credentials
---
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
metadata:
name: payments-api
namespace: flux-system
spec:
imageRepositoryRef:
name: payments-api
policy:
semver:
range: '>=1.0.0' # Only stable releases in prod
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
name: flux-system
namespace: flux-system
spec:
interval: 30m
sourceRef:
kind: GitRepository
name: fleet-infra
git:
checkout:
ref:
branch: main
commit:
author:
email: fluxbot@acme.com
name: Flux Bot
messageTemplate: |
chore(image): update {{range .Updated.Images}}{{.}}{{end}}
push:
branch: main
update:
path: ./clusters/prod
strategy: Setters # Uses # {"$imagepolicy": "flux-system:payments-api"} markersapiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
name: aws-secrets-manager
spec:
provider:
aws:
service: SecretsManager
region: us-east-1
auth:
jwt:
serviceAccountRef:
name: external-secrets
namespace: external-secrets
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
name: payments-db-credentials
namespace: production
spec:
refreshInterval: 1h
secretStoreRef:
name: aws-secrets-manager
kind: ClusterSecretStore
target:
name: payments-db-creds # K8s Secret created here
creationPolicy: Owner
template:
type: Opaque
data:
DB_HOST: "{{ .host }}"
DB_PASSWORD: "{{ .password }}"
DB_USER: "{{ .username }}"
data:
- secretKey: host
remoteRef:
key: prod/payments/database
property: host
- secretKey: password
remoteRef:
key: prod/payments/database
property: passwordapiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
name: payments-api-all-clusters
namespace: argocd
spec:
generators:
# Cluster generator: creates one App per registered cluster
- clusters:
selector:
matchLabels:
env: production
values:
revision: main
template:
metadata:
name: 'payments-api-{{name}}'
spec:
project: default
source:
repoURL: https://github.com/acme-corp/fleet-infra
targetRevision: '{{values.revision}}'
path: 'apps/payments-api/overlays/{{metadata.labels.region}}'
destination:
server: '{{server}}'
namespace: production
syncPolicy:
automated:
prune: true
selfHeal: true
syncOptions:
- CreateNamespace=true
- ServerSideApply=true- Bootstrap FluxCD with flux CLI, create GitRepository and Kustomization for a sample app
- Configure FluxCD Image Automation — push a new image tag, watch Flux auto-commit to Git
- Install External Secrets Operator, create ExternalSecret that syncs from AWS Secrets Manager
- Create ArgoCD ApplicationSet with cluster generator — deploy to 2 kind clusters simultaneously
- Add finalizer to ArgoCD app, verify kubectl delete is blocked without --cascade=false flag