Helm — Kubernetes Package Engineering
Master Helm chart anatomy, Go templating with 100+ Sprig functions, values schema validation, release hooks, OCI registries, and production packaging patterns. Zero to chart publisher.
Under-The-Hood: How Helm Works 01
Helm v3 Architecture (Client-Only, No Tiller)
Unlike Helm v2, Helm v3 is a client-only binary. There is no in-cluster Tiller server. All template rendering happens locally on your CLI.
- Load chart files: Helm reads
Chart.yaml,values.yaml, and all files intemplates/. - Merge values hierarchy: Default values ←
-f custom.yamloverrides ←--set key=valoverrides (highest priority). - Go template rendering: The
text/templatelibrary processes all{{ }}expressions. 100+ Sprig helper functions are available (upper,trunc,include,tpl,toYaml,nindent, etc.). - Schema validation: If
values.schema.jsonexists, Helm validates user-supplied values against it before rendering. - Hook separation: Helm separates manifests annotated with
helm.sh/hookfrom regular resources. - Deploy to API server: Validated manifests are pushed to the Kubernetes API server via your local kubeconfig context.
- Release state storage: Helm stores the complete release state (manifest + values + metadata) as a gzip-compressed, base64-encoded Kubernetes Secret in the release namespace.
Every helm install or helm upgrade creates a versioned Secret:
$ kubectl get secret -n production -l "owner=helm,name=my-app" NAME TYPE DATA AGE sh.helm.release.v1.my-app.v1 helm.sh/release.v1 1 5d sh.helm.release.v1.my-app.v2 helm.sh/release.v1 1 2d sh.helm.release.v1.my-app.v3 helm.sh/release.v1 1 1h ← current # The release payload is base64 → gzip → JSON $ kubectl get secret sh.helm.release.v1.my-app.v3 \ -o jsonpath='{.data.release}' | base64 -d | gzip -d | jq '.info.status' "deployed"
Hooks intercept release operations. Execution order:
- Templates are rendered; hook manifests are extracted and sorted by
helm.sh/hook-weight(ascending). - Pre-install/pre-upgrade hooks run sequentially (lower weight first). Helm waits for each Job to complete.
- If a hook Job fails → release is aborted, marked
FAILED. - Standard resources are deployed to the cluster.
- Post-install/post-upgrade hooks run.
helm.sh/hook-delete-policy: hook-succeededcleans up completed hook pods.
Helm Install Execution Flow
Chart Anatomy & Components 02
| File / Directory | Role | Key Directives | Best Practice |
|---|---|---|---|
Chart.yaml | Chart metadata, API version, dependencies list | apiVersion: v2, name, version, appVersion, dependencies | Keep version (chart) and appVersion (app) semantically separate. Use SemVer. |
values.yaml | Default configuration values. Users override with -f or --set | Flat or nested YAML. Supports global: scope for subcharts | Group logically, add YAML comments above every block to self-document defaults. |
templates/ | Go template files rendered into K8s manifests | {{ .Values.* }}, {{ .Release.* }}, {{ .Chart.* }}, conditionals, loops | One resource type per file: deployment.yaml, service.yaml, ingress.yaml. |
_helpers.tpl | Reusable named templates (partials). Not rendered directly. | {{- define "chart.name" -}}, {{ include }} | Prefix helpers with chart name to avoid collision in subcharts. |
charts/ | Dependency chart packages (.tgz) fetched by helm dependency update | Auto-populated from dependencies: in Chart.yaml | Add to .gitignore. Commit only Chart.lock. |
values.schema.json | JSON Schema to validate user-supplied values BEFORE install | type, required, properties, minimum, enum, pattern | Required for all production charts. Catches misconfigurations before they reach the cluster. |
NOTES.txt | Instructions printed to terminal after successful install | Can use Go template expressions: {{ .Release.Name }} | Include service URL, next steps, and relevant kubectl commands. |
Chart.lock | Exact dependency versions pinned after helm dependency update | Digest hashes for reproducibility | Always commit this file. Ensures reproducible builds across environments. |
WSL Hands-On Lab — Build & Publish a Chart 03
helm, kubectl, kind (from Module 02 lab). Estimated time: 90 minutes.Step 1 — Scaffold & Explore Your First Chart
# Create chart scaffold $ helm create my-api-chart my-api-chart/ ├── Chart.yaml ← Chart metadata ├── values.yaml ← Default values ├── charts/ ← Dependency charts (empty for now) └── templates/ ├── _helpers.tpl ← Reusable template partials ├── deployment.yaml ├── service.yaml ├── serviceaccount.yaml ├── hpa.yaml ├── ingress.yaml └── NOTES.txt # Validate the chart without installing $ helm lint ./my-api-chart --strict [INFO] Chart.yaml: icon is recommended [INFO] values.yaml: file linted 1 chart(s) linted, 0 chart(s) failed # Preview rendered manifests (dry run) $ helm template my-release ./my-api-chart \ --set replicaCount=3 \ --debug # Install with dry-run to check against live cluster $ helm install my-release ./my-api-chart \ --dry-run --debug -n default
Step 2 — Write a Production Chart with Schema Validation
apiVersion: v2 name: my-api-chart description: "A production-grade Node.js API Helm chart" type: application version: 1.0.0 # Chart release version (SemVer) appVersion: "2.3.1" # Application version inside the image keywords: - nodejs - api - microservice maintainers: - name: DevOps Architect email: devops@company.com dependencies: - name: postgresql version: 12.1.0 repository: https://charts.bitnami.com/bitnami condition: postgresql.enabled # Only install if flag is true
# ── Replica Settings ────────────────────────────── replicaCount: 2 # ── Image Configuration ─────────────────────────── image: repository: my-registry/my-api pullPolicy: IfNotPresent tag: "" # Defaults to Chart appVersion if empty # ── Service Account ──────────────────────────────── serviceAccount: create: true annotations: {} name: "" # ── Service ──────────────────────────────────────── service: type: ClusterIP port: 3000 # ── Ingress ──────────────────────────────────────── ingress: enabled: false className: "nginx" annotations: {} hosts: - host: api.example.com paths: [{ path: /, pathType: Prefix }] # ── Resource Limits ──────────────────────────────── resources: requests: { cpu: 100m, memory: 128Mi } limits: { cpu: 500m, memory: 256Mi } # ── Autoscaling ──────────────────────────────────── autoscaling: enabled: false minReplicas: 2 maxReplicas: 10 targetCPUUtilizationPercentage: 80 # ── Database Connection ──────────────────────────── dbConnection: host: "postgres-svc" port: 5432 name: "myapp_db" username: "app_user" passwordSecretName: "db-credentials" passwordSecretKey: "password" # ── PostgreSQL Subchart ───────────────────────────── postgresql: enabled: false # Set true for local dev
{
"$schema": "http://json-schema.org/draft-07/schema#",
"type": "object",
"required": ["replicaCount", "image", "service", "dbConnection"],
"properties": {
"replicaCount": {
"type": "integer",
"minimum": 1,
"maximum": 50
},
"image": {
"type": "object",
"required": ["repository", "pullPolicy"],
"properties": {
"repository": { "type": "string" },
"pullPolicy": {
"type": "string",
"enum": ["Always", "IfNotPresent", "Never"]
},
"tag": { "type": "string" }
}
},
"service": {
"type": "object",
"properties": {
"type": {
"type": "string",
"enum": ["ClusterIP", "NodePort", "LoadBalancer"]
},
"port": { "type": "integer", "minimum": 1, "maximum": 65535 }
}
},
"dbConnection": {
"type": "object",
"required": ["host", "port", "name", "username"],
"properties": {
"host": { "type": "string" },
"port": { "type": "integer", "minimum": 1 },
"name": { "type": "string" },
"username": { "type": "string" }
}
}
}
}{{/* ── Chart Name ─────────────────────────────── */}} {{- define "my-api-chart.name" -}} {{- default .Chart.Name .Values.nameOverride | trunc 63 | trimSuffix "-" -}} {{- end }} {{/* ── Fully Qualified App Name (max 63 chars) ─── */}} {{- define "my-api-chart.fullname" -}} {{- if .Values.fullnameOverride -}} {{- .Values.fullnameOverride | trunc 63 | trimSuffix "-" -}} {{- else -}} {{- $name := default .Chart.Name .Values.nameOverride -}} {{- if contains $name .Release.Name -}} {{- .Release.Name | trunc 63 | trimSuffix "-" -}} {{- else -}} {{- printf "%s-%s" .Release.Name $name | trunc 63 | trimSuffix "-" -}} {{- end -}} {{- end -}} {{- end }} {{/* ── Standard Labels ─────────────────────────── */}} {{- define "my-api-chart.labels" -}} helm.sh/chart: {{ printf "%s-%s" .Chart.Name .Chart.Version | replace "+" "_" | trunc 63 }} {{ include "my-api-chart.selectorLabels" . }} {{- if .Chart.AppVersion }} app.kubernetes.io/version: {{ .Chart.AppVersion | quote }} {{- end }} app.kubernetes.io/managed-by: {{ .Release.Service }} {{- end }} {{/* ── Selector Labels ─────────────────────────── */}} {{- define "my-api-chart.selectorLabels" -}} app.kubernetes.io/name: {{ include "my-api-chart.name" . }} app.kubernetes.io/instance: {{ .Release.Name }} {{- end }} {{/* ── Service Account Name ───────────────────── */}} {{- define "my-api-chart.serviceAccountName" -}} {{- if .Values.serviceAccount.create -}} {{- default (include "my-api-chart.fullname" .) .Values.serviceAccount.name -}} {{- else -}} {{- default "default" .Values.serviceAccount.name -}} {{- end -}} {{- end }}
apiVersion: apps/v1 kind: Deployment metadata: name: {{ include "my-api-chart.fullname" . }} labels: {{- include "my-api-chart.labels" . | nindent 4 }} spec: {{- if not .Values.autoscaling.enabled }} replicas: {{ .Values.replicaCount }} {{- end }} selector: matchLabels: {{- include "my-api-chart.selectorLabels" . | nindent 6 }} template: metadata: labels: {{- include "my-api-chart.selectorLabels" . | nindent 8 }} spec: serviceAccountName: {{ include "my-api-chart.serviceAccountName" . }} securityContext: runAsNonRoot: true runAsUser: 1000 fsGroup: 2000 containers: - name: {{ .Chart.Name }} image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}" imagePullPolicy: {{ .Values.image.pullPolicy }} ports: - name: http containerPort: {{ .Values.service.port }} env: - name: DB_HOST value: {{ .Values.dbConnection.host | quote }} - name: DB_PORT value: {{ .Values.dbConnection.port | quote }} - name: DB_PASSWORD valueFrom: secretKeyRef: name: {{ .Values.dbConnection.passwordSecretName | quote }} key: {{ .Values.dbConnection.passwordSecretKey | quote }} resources: {{- toYaml .Values.resources | nindent 12 }} readinessProbe: httpGet: { path: /health, port: http } initialDelaySeconds: 5 livenessProbe: httpGet: { path: /health, port: http } initialDelaySeconds: 15
apiVersion: batch/v1 kind: Job metadata: name: "{{ include "my-api-chart.fullname" . }}-db-migrate" annotations: # This runs BEFORE install AND upgrade "helm.sh/hook": pre-install,pre-upgrade "helm.sh/hook-weight": "5" "helm.sh/hook-delete-policy": before-hook-creation,hook-succeeded spec: backoffLimit: 3 template: spec: restartPolicy: Never containers: - name: db-migrator image: "{{ .Values.image.repository }}:{{ .Values.image.tag | default .Chart.AppVersion }}" command: ["node", "scripts/migrate.js"] env: - name: DB_HOST value: {{ .Values.dbConnection.host | quote }} - name: DB_NAME value: {{ .Values.dbConnection.name | quote }} - name: DB_PASSWORD valueFrom: secretKeyRef: name: {{ .Values.dbConnection.passwordSecretName | quote }} key: {{ .Values.dbConnection.passwordSecretKey | quote }}
Step 3 — Install, Upgrade & Package
# Install to Kind cluster $ kubectl create secret generic db-credentials \ --from-literal=password=supersecret -n default $ helm install my-api ./my-api-chart -n default # View release details $ helm list NAME NAMESPACE REVISION STATUS CHART APP VERSION my-api default 1 deployed my-api-chart-1.0.0 2.3.1 # Upgrade with new values $ helm upgrade my-api ./my-api-chart --set replicaCount=3 # View history $ helm history my-api REVISION STATUS CHART DESCRIPTION 1 superseded my-api-chart-1.0.0 Install complete 2 deployed my-api-chart-1.0.0 Upgrade complete # Rollback to revision 1 $ helm rollback my-api 1 # Package chart as .tgz for distribution $ helm package ./my-api-chart Successfully packaged chart: ./my-api-chart-1.0.0.tgz # Push to OCI registry (GitHub Container Registry) $ helm push my-api-chart-1.0.0.tgz oci://ghcr.io/your-username/charts Pushed: ghcr.io/your-username/charts/my-api-chart:1.0.0 Digest: sha256:abc123... # Install directly from OCI registry $ helm install my-api oci://ghcr.io/your-username/charts/my-api-chart \ --version 1.0.0
Real-World Project: Package Your App 04
🌱 Startup: Your First Helm Chart
Package a Node.js Express API into a reusable Helm chart that you can deploy to Kind locally and push to GHCR for free.
- Run
helm create my-api-chartand explore the structure - Update
values.yamlwith your app's image name and port - Run
helm lint --strictand fix any warnings - Install to Kind with
helm installand verify the Pod is running
🏢 SME: Multi-Environment Values Override
You have dev, staging, and production environments. Each gets its own values file overriding defaults.
# Development: 1 replica, internal image, local DB $ helm install my-api ./my-api-chart -f values-dev.yaml -n development # Staging: 2 replicas, ingress enabled, staging DB $ helm install my-api ./my-api-chart -f values-staging.yaml -n staging # Production: 3+ replicas, HPA enabled, Aurora RDS $ helm install my-api ./my-api-chart -f values-production.yaml -n production \ --set image.tag=v2.3.1-prod \ --atomic \ --timeout 5m
🏛 Enterprise: Private Chart Museum + CI/CD Integration
Centralized Chartmuseum (or OCI registry on ECR) with automated chart publishing via GitHub Actions. Semantic versioning enforced by CI.
Troubleshooting & Disaster Recovery 05
When a CI job is killed mid-deploy, the Helm release Secret is locked. Fix:
# 1. Find the stuck release Secret $ kubectl get secret -n production -l "owner=helm,name=my-app" # 2. Decode the release payload $ kubectl get secret sh.helm.release.v1.my-app.v5 \ -o jsonpath='{.data.release}' | base64 -d | gzip -d > /tmp/release.json # 3. Change "status" from "pending-upgrade" to "failed" in /tmp/release.json $ jq '.info.status = "failed"' /tmp/release.json > /tmp/release-fixed.json # 4. Re-encode and patch the Secret $ ENCODED=$(gzip -c /tmp/release-fixed.json | base64 | tr -d '\n') $ kubectl patch secret sh.helm.release.v1.my-app.v5 -n production \ --type='json' \ -p="[{\"op\":\"replace\",\"path\":\"/data/release\",\"value\":\"$ENCODED\"}]" secret/sh.helm.release.v1.my-app.v5 patched # 5. Now helm list should show "failed" — you can upgrade normally $ helm list -n production $ helm upgrade my-app ./chart -n production
# Use helm template to see rendering errors without deploying $ helm template my-release ./my-chart --debug 2>&1 | head -50 # Common fix: use default function to handle nil values # BAD: {{ .Values.ingress.annotations }} — panics if ingress is nil # GOOD: {{ .Values.ingress.annotations | default dict | toYaml }} # Test schema validation errors separately $ helm install test ./my-chart --set replicaCount="not-a-number" --dry-run Error: values don't meet the specifications of the schema(s) - replicaCount: Invalid type. Expected: integer, given: string
# Add the repository first, then update $ helm repo add bitnami https://charts.bitnami.com/bitnami $ helm repo update $ helm dependency update ./my-api-chart Saving 1 charts Downloading postgresql from repo https://charts.bitnami.com/bitnami Deleting outdated charts Update Complete. ⎈Happy Helming!⎈ # Verify the lock file is correct $ cat my-api-chart/Chart.lock
Interactive Lab Checklist 06
Helm Hands-On Tasks
- Create a chart with
helm create, inspect every generated file, understand its purpose - Run
helm lint --strictandhelm template --debugon your chartKillercoda ↗ - Add
values.schema.jsonwith required fields. Test that invalid values cause install to fail. - Add a pre-install hook Job that prints a database migration message, verify it runs before Deployment
- Perform
helm upgradeandhelm rollback, inspecthelm history - Simulate a stuck release by manually setting the Secret status to
pending-upgrade, then fix it - Package your chart and push to GHCR OCI registry. Install it back with
helm install oci://...
30-Day Learning Roadmap 07
- Values hierarchy, context objects
- if/else, range, with blocks
- Sprig: trunc, upper, quote, nindent
- Pipeline expressions
- values.schema.json: type, required, enum
- Custom error messages
- helm-unittest for template testing
- Strict linting in CI
- All hook types and weight ordering
- Delete policies for cleanup
- OCI registry push/pull
- Chartmuseum setup
- ArgoCD + Helm integration
- SOPS encrypted values
- Chart dependency management
- Release state recovery procedures