Skip to content
DevOps Architect

    Syllabus / Infrastructure / 03

    Module 03
    // helm

    Helm — Kubernetes Package Engineering

    Master Helm chart anatomy, Go templating with 100+ Sprig functions, values schema validation, release hooks, OCI registries, and production packaging patterns. Zero to chart publisher.

    🌱 Startup 🏢 SME 🏛 Enterprise ⚡ Killercoda 🎓 Udemy

    ⚙️
    Under-The-Hood: How Helm Works 01

    Helm v3 Architecture (Client-Only, No Tiller)

    Unlike Helm v2, Helm v3 is a client-only binary. There is no in-cluster Tiller server. All template rendering happens locally on your CLI.

    1. Load chart files: Helm reads Chart.yaml, values.yaml, and all files in templates/.
    2. Merge values hierarchy: Default values ← -f custom.yaml overrides ← --set key=val overrides (highest priority).
    3. Go template rendering: The text/template library processes all {{ }} expressions. 100+ Sprig helper functions are available (upper, trunc, include, tpl, toYaml, nindent, etc.).
    4. Schema validation: If values.schema.json exists, Helm validates user-supplied values against it before rendering.
    5. Hook separation: Helm separates manifests annotated with helm.sh/hook from regular resources.
    6. Deploy to API server: Validated manifests are pushed to the Kubernetes API server via your local kubeconfig context.
    7. Release state storage: Helm stores the complete release state (manifest + values + metadata) as a gzip-compressed, base64-encoded Kubernetes Secret in the release namespace.

    Every helm install or helm upgrade creates a versioned Secret:

    Release secrets
    $ kubectl get secret -n production -l "owner=helm,name=my-app"
    NAME                              TYPE                  DATA   AGE
    sh.helm.release.v1.my-app.v1      helm.sh/release.v1    1      5d
    sh.helm.release.v1.my-app.v2      helm.sh/release.v1    1      2d
    sh.helm.release.v1.my-app.v3      helm.sh/release.v1    1      1h  ← current
    
    # The release payload is base64 → gzip → JSON
    $ kubectl get secret sh.helm.release.v1.my-app.v3 \
        -o jsonpath='{.data.release}' | base64 -d | gzip -d | jq '.info.status'
    "deployed"

    Hooks intercept release operations. Execution order:

    1. Templates are rendered; hook manifests are extracted and sorted by helm.sh/hook-weight (ascending).
    2. Pre-install/pre-upgrade hooks run sequentially (lower weight first). Helm waits for each Job to complete.
    3. If a hook Job fails → release is aborted, marked FAILED.
    4. Standard resources are deployed to the cluster.
    5. Post-install/post-upgrade hooks run.
    6. helm.sh/hook-delete-policy: hook-succeeded cleans up completed hook pods.

    Helm Install Execution Flow

    graph TD A["helm install my-app ./chart\n-f values-prod.yaml"] --> B["Load Chart files\nChart.yaml + templates/ + values.yaml"] B --> C["Merge Values Hierarchy\ndefault ← -f file ← --set flags"] C --> D["Go text/template Engine\n+ 100+ Sprig Functions"] D --> E["values.schema.json\nValidation"] E --> F{Schema Valid?} F -- "No" --> G["Error: values validation failed\nRelease aborted"] F -- "Yes" --> H["Separate Hook manifests\nSort by hook-weight"] H --> I["Execute pre-install Hooks\ne.g. DB migration Job"] I --> J{Hook succeeded?} J -- "No" --> K["Release marked FAILED\nSecrets stored for debugging"] J -- "Yes" --> L["Apply standard manifests\nto K8s API server"] L --> M["Create Release Secret\nsh.helm.release.v1.my-app.vN\ngzip + base64 encoded"] M --> N["Execute post-install Hooks\ne.g. Slack notification"] N --> O["Print NOTES.txt\n& Release Summary"] style A fill:#8b5cf6,stroke:#7c3aed,color:#fff style G fill:#ef4444,stroke:#dc2626,color:#fff style K fill:#ef4444,stroke:#dc2626,color:#fff style M fill:#00f5ff,stroke:#00dce5,color:#000 style O fill:#10b981,stroke:#059669,color:#fff

    📦
    Chart Anatomy & Components 02

    File / DirectoryRoleKey DirectivesBest Practice
    Chart.yamlChart metadata, API version, dependencies listapiVersion: v2, name, version, appVersion, dependenciesKeep version (chart) and appVersion (app) semantically separate. Use SemVer.
    values.yamlDefault configuration values. Users override with -f or --setFlat or nested YAML. Supports global: scope for subchartsGroup logically, add YAML comments above every block to self-document defaults.
    templates/Go template files rendered into K8s manifests{{ .Values.* }}, {{ .Release.* }}, {{ .Chart.* }}, conditionals, loopsOne resource type per file: deployment.yaml, service.yaml, ingress.yaml.
    _helpers.tplReusable named templates (partials). Not rendered directly.{{- define "chart.name" -}}, {{ include }}Prefix helpers with chart name to avoid collision in subcharts.
    charts/Dependency chart packages (.tgz) fetched by helm dependency updateAuto-populated from dependencies: in Chart.yamlAdd to .gitignore. Commit only Chart.lock.
    values.schema.jsonJSON Schema to validate user-supplied values BEFORE installtype, required, properties, minimum, enum, patternRequired for all production charts. Catches misconfigurations before they reach the cluster.
    NOTES.txtInstructions printed to terminal after successful installCan use Go template expressions: {{ .Release.Name }}Include service URL, next steps, and relevant kubectl commands.
    Chart.lockExact dependency versions pinned after helm dependency updateDigest hashes for reproducibilityAlways commit this file. Ensures reproducible builds across environments.

    🖥
    WSL Hands-On Lab — Build & Publish a Chart 03

    ℹ️
    Prerequisites
    Requires: helm, kubectl, kind (from Module 02 lab). Estimated time: 90 minutes.

    Step 1 — Scaffold & Explore Your First Chart

    Create chart scaffold
    # Create chart scaffold
    $ helm create my-api-chart
    
    my-api-chart/
    ├── Chart.yaml            ← Chart metadata
    ├── values.yaml           ← Default values
    ├── charts/               ← Dependency charts (empty for now)
    └── templates/
        ├── _helpers.tpl       ← Reusable template partials
        ├── deployment.yaml
        ├── service.yaml
        ├── serviceaccount.yaml
        ├── hpa.yaml
        ├── ingress.yaml
        └── NOTES.txt
    
    # Validate the chart without installing
    $ helm lint ./my-api-chart --strict
    [INFO] Chart.yaml: icon is recommended
    [INFO] values.yaml: file linted
    1 chart(s) linted, 0 chart(s) failed
    
    # Preview rendered manifests (dry run)
    $ helm template my-release ./my-api-chart \
        --set replicaCount=3 \
        --debug
    
    # Install with dry-run to check against live cluster
    $ helm install my-release ./my-api-chart \
        --dry-run --debug -n default

    Step 2 — Write a Production Chart with Schema Validation

    📄 Chart.yamlyaml
    apiVersion: v2
    name: my-api-chart
    description: "A production-grade Node.js API Helm chart"
    type: application
    version: 1.0.0        # Chart release version (SemVer)
    appVersion: "2.3.1"    # Application version inside the image
    keywords:
      - nodejs
      - api
      - microservice
    maintainers:
      - name: DevOps Architect
        email: devops@company.com
    dependencies:
      - name: postgresql
        version: 12.1.0
        repository: https://charts.bitnami.com/bitnami
        condition: postgresql.enabled   # Only install if flag is true
    📄 values.yamlyaml
    # ── Replica Settings ──────────────────────────────
    replicaCount: 2
    
    # ── Image Configuration ───────────────────────────
    image:
      repository: my-registry/my-api
      pullPolicy: IfNotPresent
      tag: ""    # Defaults to Chart appVersion if empty
    
    # ── Service Account ────────────────────────────────
    serviceAccount:
      create: true
      annotations: {}
      name: ""
    
    # ── Service ────────────────────────────────────────
    service:
      type: ClusterIP
      port: 3000
    
    # ── Ingress ────────────────────────────────────────
    ingress:
      enabled: false
      className: "nginx"
      annotations: {}
      hosts:
        - host: api.example.com
          paths: [{ path: /, pathType: Prefix }]
    
    # ── Resource Limits ────────────────────────────────
    resources:
      requests: { cpu: 100m, memory: 128Mi }
      limits:   { cpu: 500m, memory: 256Mi }
    
    # ── Autoscaling ────────────────────────────────────
    autoscaling:
      enabled: false
      minReplicas: 2
      maxReplicas: 10
      targetCPUUtilizationPercentage: 80
    
    # ── Database Connection ────────────────────────────
    dbConnection:
      host: "postgres-svc"
      port: 5432
      name: "myapp_db"
      username: "app_user"
      passwordSecretName: "db-credentials"
      passwordSecretKey: "password"
    
    # ── PostgreSQL Subchart ─────────────────────────────
    postgresql:
      enabled: false   # Set true for local dev
    📄 values.schema.jsonjson
    {
      "$schema": "http://json-schema.org/draft-07/schema#",
      "type": "object",
      "required": ["replicaCount", "image", "service", "dbConnection"],
      "properties": {
        "replicaCount": {
          "type": "integer",
          "minimum": 1,
          "maximum": 50
        },
        "image": {
          "type": "object",
          "required": ["repository", "pullPolicy"],
          "properties": {
            "repository": { "type": "string" },
            "pullPolicy": {
              "type": "string",
              "enum": ["Always", "IfNotPresent", "Never"]
            },
            "tag": { "type": "string" }
          }
        },
        "service": {
          "type": "object",
          "properties": {
            "type": {
              "type": "string",
              "enum": ["ClusterIP", "NodePort", "LoadBalancer"]
            },
            "port": { "type": "integer", "minimum": 1, "maximum": 65535 }
          }
        },
        "dbConnection": {
          "type": "object",
          "required": ["host", "port", "name", "username"],
          "properties": {
            "host":     { "type": "string" },
            "port":     { "type": "integer", "minimum": 1 },
            "name":     { "type": "string" },
            "username": { "type": "string" }
          }
        }
      }
    }
    📄 templates/_helpers.tplgotmpl
    {{/* ── Chart Name ─────────────────────────────── */}}
    {{- define "my-api-chart.name" -}}
    {{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}}
    {{- end }}
    
    {{/* ── Fully Qualified App Name (max 63 chars) ─── */}}
    {{- define "my-api-chart.fullname" -}}
    {{- if .Values.fullnameOverride -}}
    {{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}}
    {{- else -}}
    {{- $name := default .Chart.Name .Values.nameOverride -}}
    {{- if contains $name .Release.Name -}}
    {{- .Release.Name | trunc 63 | trimSuffix "-" -}}
    {{- else -}}
    {{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}}
    {{- end -}}
    {{- end -}}
    {{- end }}
    
    {{/* ── Standard Labels ─────────────────────────── */}}
    {{- define "my-api-chart.labels" -}}
    helm.sh/chart: {{ printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 }}
    {{ include "my-api-chart.selectorLabels" . }}
    {{- if .Chart.AppVersion }}
    app.kubernetes.io/version: {{ .Chart.AppVersion | quote }}
    {{- end }}
    app.kubernetes.io/managed-by: {{ .Release.Service }}
    {{- end }}
    
    {{/* ── Selector Labels ─────────────────────────── */}}
    {{- define "my-api-chart.selectorLabels" -}}
    app.kubernetes.io/name: {{ include "my-api-chart.name" . }}
    app.kubernetes.io/instance: {{ .Release.Name }}
    {{- end }}
    
    {{/* ── Service Account Name ───────────────────── */}}
    {{- define "my-api-chart.serviceAccountName" -}}
    {{- if .Values.serviceAccount.create -}}
    {{- default (include "my-api-chart.fullname" .) .Values.serviceAccount.name -}}
    {{- else -}}
    {{- default "default" .Values.serviceAccount.name -}}
    {{- end -}}
    {{- end }}
    📄 templates/deployment.yamlyaml
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: {{ include "my-api-chart.fullname" . }}
      labels:
        {{- include "my-api-chart.labels" . | nindent 4 }}
    spec:
      {{- if not .Values.autoscaling.enabled }}
      replicas: {{ .Values.replicaCount }}
      {{- end }}
      selector:
        matchLabels:
          {{- include "my-api-chart.selectorLabels" . | nindent 6 }}
      template:
        metadata:
          labels:
            {{- include "my-api-chart.selectorLabels" . | nindent 8 }}
        spec:
          serviceAccountName: {{ include "my-api-chart.serviceAccountName" . }}
          securityContext:
            runAsNonRoot: true
            runAsUser: 1000
            fsGroup: 2000
          containers:
            - name: {{ .Chart.Name }}
              image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
              imagePullPolicy: {{ .Values.image.pullPolicy }}
              ports:
                - name: http
                  containerPort: {{ .Values.service.port }}
              env:
                - name: DB_HOST
                  value: {{ .Values.dbConnection.host | quote }}
                - name: DB_PORT
                  value: {{ .Values.dbConnection.port | quote }}
                - name: DB_PASSWORD
                  valueFrom:
                    secretKeyRef:
                      name: {{ .Values.dbConnection.passwordSecretName | quote }}
                      key:  {{ .Values.dbConnection.passwordSecretKey | quote }}
              resources:
                {{- toYaml .Values.resources | nindent 12 }}
              readinessProbe:
                httpGet: { path: /health, port: http }
                initialDelaySeconds: 5
              livenessProbe:
                httpGet: { path: /health, port: http }
                initialDelaySeconds: 15
    📄 templates/pre-install-hook.yamlyaml
    apiVersion: batch/v1
    kind: Job
    metadata:
      name: "{{ include "my-api-chart.fullname" . }}-db-migrate"
      annotations:
        # This runs BEFORE install AND upgrade
        "helm.sh/hook":               pre-install,pre-upgrade
        "helm.sh/hook-weight":        "5"
        "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded
    spec:
      backoffLimit: 3
      template:
        spec:
          restartPolicy: Never
          containers:
            - name: db-migrator
              image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}"
              command: ["node", "scripts/migrate.js"]
              env:
                - name: DB_HOST
                  value: {{ .Values.dbConnection.host | quote }}
                - name: DB_NAME
                  value: {{ .Values.dbConnection.name | quote }}
                - name: DB_PASSWORD
                  valueFrom:
                    secretKeyRef:
                      name: {{ .Values.dbConnection.passwordSecretName | quote }}
                      key:  {{ .Values.dbConnection.passwordSecretKey | quote }}

    Step 3 — Install, Upgrade & Package

    Full Helm workflow
    # Install to Kind cluster
    $ kubectl create secret generic db-credentials \
        --from-literal=password=supersecret -n default
    
    $ helm install my-api ./my-api-chart -n default
    
    # View release details
    $ helm list
    NAME     NAMESPACE  REVISION  STATUS    CHART               APP VERSION
    my-api   default    1         deployed  my-api-chart-1.0.0  2.3.1
    
    # Upgrade with new values
    $ helm upgrade my-api ./my-api-chart --set replicaCount=3
    
    # View history
    $ helm history my-api
    REVISION  STATUS    CHART               DESCRIPTION
    1         superseded my-api-chart-1.0.0  Install complete
    2         deployed   my-api-chart-1.0.0  Upgrade complete
    
    # Rollback to revision 1
    $ helm rollback my-api 1
    
    # Package chart as .tgz for distribution
    $ helm package ./my-api-chart
    Successfully packaged chart: ./my-api-chart-1.0.0.tgz
    
    # Push to OCI registry (GitHub Container Registry)
    $ helm push my-api-chart-1.0.0.tgz oci://ghcr.io/your-username/charts
    Pushed: ghcr.io/your-username/charts/my-api-chart:1.0.0
    Digest: sha256:abc123...
    
    # Install directly from OCI registry
    $ helm install my-api oci://ghcr.io/your-username/charts/my-api-chart \
        --version 1.0.0

    🏗
    Real-World Project: Package Your App 04

    🌱 Startup: Your First Helm Chart

    Package a Node.js Express API into a reusable Helm chart that you can deploy to Kind locally and push to GHCR for free.

    • Run helm create my-api-chart and explore the structure
    • Update values.yaml with your app's image name and port
    • Run helm lint --strict and fix any warnings
    • Install to Kind with helm install and verify the Pod is running

    🏢 SME: Multi-Environment Values Override

    You have dev, staging, and production environments. Each gets its own values file overriding defaults.

    Multi-env deployment
    # Development: 1 replica, internal image, local DB
    $ helm install my-api ./my-api-chart -f values-dev.yaml -n development
    
    # Staging: 2 replicas, ingress enabled, staging DB
    $ helm install my-api ./my-api-chart -f values-staging.yaml -n staging
    
    # Production: 3+ replicas, HPA enabled, Aurora RDS
    $ helm install my-api ./my-api-chart -f values-production.yaml -n production \
        --set image.tag=v2.3.1-prod \
        --atomic \
        --timeout 5m

    🏛 Enterprise: Private Chart Museum + CI/CD Integration

    Centralized Chartmuseum (or OCI registry on ECR) with automated chart publishing via GitHub Actions. Semantic versioning enforced by CI.

    🏛
    Enterprise Chart Pipeline
    PR merged → GitHub Actions: helm lint → helm test → helm package → chart version bumped → pushed to ECR OCI registry → ArgoCD ApplicationSet picks up new chart version → canary deploy to 5% traffic → full rollout.

    🔧
    Troubleshooting & Disaster Recovery 05

    When a CI job is killed mid-deploy, the Helm release Secret is locked. Fix:

    Unlock stuck release
    # 1. Find the stuck release Secret
    $ kubectl get secret -n production -l "owner=helm,name=my-app"
    
    # 2. Decode the release payload
    $ kubectl get secret sh.helm.release.v1.my-app.v5 \
        -o jsonpath='{.data.release}' | base64 -d | gzip -d > /tmp/release.json
    
    # 3. Change "status" from "pending-upgrade" to "failed" in /tmp/release.json
    $ jq '.info.status = "failed"' /tmp/release.json > /tmp/release-fixed.json
    
    # 4. Re-encode and patch the Secret
    $ ENCODED=$(gzip -c /tmp/release-fixed.json | base64 | tr -d '\n')
    $ kubectl patch secret sh.helm.release.v1.my-app.v5 -n production \
        --type='json' \
        -p="[{\"op\":\"replace\",\"path\":\"/data/release\",\"value\":\"$ENCODED\"}]"
    secret/sh.helm.release.v1.my-app.v5 patched
    
    # 5. Now helm list should show "failed" — you can upgrade normally
    $ helm list -n production
    $ helm upgrade my-app ./chart -n production
    Debug template errors
    # Use helm template to see rendering errors without deploying
    $ helm template my-release ./my-chart --debug 2>&1 | head -50
    
    # Common fix: use default function to handle nil values
    # BAD:  {{ .Values.ingress.annotations }} — panics if ingress is nil
    # GOOD: {{ .Values.ingress.annotations | default dict | toYaml }}
    
    # Test schema validation errors separately
    $ helm install test ./my-chart --set replicaCount="not-a-number" --dry-run
    Error: values don't meet the specifications of the schema(s)
      - replicaCount: Invalid type. Expected: integer, given: string
    Fix dependency issues
    # Add the repository first, then update
    $ helm repo add bitnami https://charts.bitnami.com/bitnami
    $ helm repo update
    $ helm dependency update ./my-api-chart
    Saving 1 charts
    Downloading postgresql from repo https://charts.bitnami.com/bitnami
    Deleting outdated charts
    Update Complete. ⎈Happy Helming!⎈
    
    # Verify the lock file is correct
    $ cat my-api-chart/Chart.lock

    🧪
    Interactive Lab Checklist 06

    ⛵

    Helm Hands-On Tasks

    • Create a chart with helm create, inspect every generated file, understand its purpose
    • Run helm lint --strict and helm template --debug on your chartKillercoda ↗
    • Add values.schema.json with required fields. Test that invalid values cause install to fail.
    • Add a pre-install hook Job that prints a database migration message, verify it runs before Deployment
    • Perform helm upgrade and helm rollback, inspect helm history
    • Simulate a stuck release by manually setting the Secret status to pending-upgrade, then fix it
    • Package your chart and push to GHCR OCI registry. Install it back with helm install oci://...

    🗓
    30-Day Learning Roadmap 07

    Week 1
    Go Templating & Sprig
    • Values hierarchy, context objects
    • if/else, range, with blocks
    • Sprig: trunc, upper, quote, nindent
    • Pipeline expressions
    Week 2
    Schema & Validation
    • values.schema.json: type, required, enum
    • Custom error messages
    • helm-unittest for template testing
    • Strict linting in CI
    Week 3
    Hooks & Registries
    • All hook types and weight ordering
    • Delete policies for cleanup
    • OCI registry push/pull
    • Chartmuseum setup
    Week 4
    GitOps & Hardening
    • ArgoCD + Helm integration
    • SOPS encrypted values
    • Chart dependency management
    • Release state recovery procedures

    Run: kubectl get secret -n production -l "owner=helm,name=my-app"

    Extra commands from this lesson (41) are kept out of this page. Quizzes were not in the source HTML.