🕸Istio & Service Mesh
Secure, observe, and control every byte of east-west traffic in your Kubernetes cluster. Master Istio mTLS enforcement, progressive delivery via VirtualService weight-shifting, circuit breakers with DestinationRule, Ambient Mesh sidecar-less architecture, and Kiali for real-time traffic topology during incidents.
Real Incident: Lateral Movement Contained by mTLS
Scenario: A compromised CI/CD runner had AWS credentials and a valid Kubernetes service account token. Without mTLS, it could call any service in the cluster over HTTP.
With Istio PeerAuthentication STRICT mode:
# Attacker tries to call payments-api from compromised pod attacker$ curl http://payments-api.production.svc.cluster.local/v1/transfer \ -H "Authorization: Bearer stolen-token" \ -d '{"amount": 99999, "to": "attacker-account"}' upstream connect error or disconnect/reset before headers. reset reason: connection failure transport failure reason: TLS error: 268435581... # The connection was REJECTED because the compromised pod's # certificate was not in the payments-api's SPIFFE trust bundle # Istio enforced mTLS — no valid cert, no connection. # Check Istio access logs (what was attempted) $ kubectl logs -n production deploy/payments-api -c istio-proxy | \ grep "PEER_CERT_VERIFY_FAILED"
Lesson: Network policies alone are not enough. mTLS with SPIFFE certificate identity provides cryptographic proof of service identity. Enable STRICT mode across all production namespaces.
Istio Architecture 18.1
Pilot + Citadel + Galley] end subgraph "Data Plane (Sidecar Mode)" Pod1["Service A\napp container\nEnvoy sidecar"] Pod2["Service B\napp container\nEnvoy sidecar"] end subgraph "Data Plane (Ambient Mode)" ZTunnel[ztunnel DaemonSet
L4 mTLS per node] Waypoint[Waypoint Proxy
L7 per namespace] end subgraph "Observability" Prometheus2[Prometheus
Istio metrics] Kiali[Kiali
Traffic topology] Jaeger[Jaeger/Tempo
Distributed traces] end Istiod -->|certs + config| Pod1 Istiod -->|certs + config| Pod2 Istiod -->|certs + config| ZTunnel Pod1 -->|mTLS| Pod2 ZTunnel --> Waypoint Pod1 -->|metrics| Prometheus2 Prometheus2 --> Kiali Pod1 -->|traces| Jaeger
Sidecar Mode
Envoy proxy injected as sidecar into every Pod. Full L7 visibility. High resource overhead (~50MB/pod). Best for: existing clusters, full HTTP/gRPC observability, request-level policies.
Ambient Mode (Istio 1.22+)
No sidecars. ztunnel DaemonSet handles L4 mTLS. Optional Waypoint proxy for L7. 90% less memory overhead. Best for: new clusters, cost-sensitive environments, clusters with thousands of pods.
mTLS Zero-Trust Enforcement 18.2
# Step 1: Enable PERMISSIVE mesh-wide (allows both mTLS and plaintext)
# Use this first to verify all traffic switches to mTLS
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: default
namespace: istio-system # mesh-wide scope
spec:
mtls:
mode: PERMISSIVE
---
# Step 2: After verifying no plaintext traffic in Kiali,
# switch to STRICT (breaks plaintext connections)
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: default
namespace: istio-system
spec:
mtls:
mode: STRICT
---
# Namespace-level override (more granular)
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: production-strict
namespace: production
spec:
mtls:
mode: STRICT
# Port-specific exception for health probes
portLevelMtls:
8080:
mode: PERMISSIVE # liveness probe from kubelet (no cert)# Only payments-api service account can call fraud-service
# All other callers get 403 Forbidden
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: fraud-service-policy
namespace: production
spec:
selector:
matchLabels:
app: fraud-service
action: ALLOW
rules:
- from:
- source:
# SPIFFE identity — cryptographic, not IP-based
principals:
- cluster.local/ns/production/sa/payments-api
to:
- operation:
methods: ["POST"]
paths: ["/v1/fraud/check"]
---
# Deny all ingress except from ingress gateway
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
name: deny-all-except-gateway
namespace: production
spec:
action: DENY
rules:
- from:
- source:
notPrincipals:
- cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account# Check mTLS status between two services $ istioctl x check-inject -n production NAMESPACE POD INJECTED production payments-api-7d9f8b true production fraud-service-5c4d3a true # Verify mTLS is working (connection is encrypted) $ istioctl authn tls-check payments-api.production.svc.cluster.local HOST STATUS SERVER CLIENT payments-api.production.svc.cluster.local mTLS STRICT mTLS # Check proxy config for specific pod $ istioctl proxy-config listeners payments-api-7d9f8b -n production # Real-time access log from Envoy sidecar $ kubectl logs payments-api-7d9f8b -n production -c istio-proxy --tail=50
Traffic Management: Canary & Circuit Breaker 18.3
# Progressive canary: send 10% traffic to v2
# No DNS changes, no load balancer rules — pure Envoy routing
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: payments-api
namespace: production
spec:
hosts:
- payments-api
http:
# Header-based testing: QA team always gets v2
- match:
- headers:
x-canary-user:
exact: "true"
route:
- destination:
host: payments-api
subset: v2
# Weight-based: 90/10 split
- route:
- destination:
host: payments-api
subset: v1
weight: 90
- destination:
host: payments-api
subset: v2
weight: 10
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: payments-api
namespace: production
spec:
host: payments-api
subsets:
- name: v1
labels:
version: v1
- name: v2
labels:
version: v2
trafficPolicy:
connectionPool:
http:
h2UpgradePolicy: UPGRADE# Circuit breaker: eject unhealthy endpoints
# If >50% requests fail, eject host for 5 minutes
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: inventory-service
namespace: production
spec:
host: inventory-service
trafficPolicy:
connectionPool:
tcp:
maxConnections: 100
http:
http1MaxPendingRequests: 50
http2MaxRequests: 100
maxRetries: 3
outlierDetection:
# Eject if >5 errors in 30s consecutive interval
consecutiveGatewayErrors: 5
interval: 30s
# Keep ejected for 5 minutes
baseEjectionTime: 5m
# Never eject more than 50% of endpoints
maxEjectionPercent: 50
# Also detect 5xx errors (not just gateway errors)
consecutive5xxErrors: 10# Chaos engineering: inject 500ms delay for 20% of requests
# Use this to test timeout handling before incidents happen
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: inventory-service-chaos
namespace: production
spec:
hosts:
- inventory-service
http:
- fault:
delay:
percentage:
value: 20.0
fixedDelay: 500ms
abort:
percentage:
value: 5.0 # 5% of requests return HTTP 503
httpStatus: 503
route:
- destination:
host: inventory-service# Local rate limiting via EnvoyFilter
# Limits: 100 req/s per source IP on the /v1/search endpoint
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
name: search-rate-limit
namespace: production
spec:
workloadSelector:
labels:
app: search-api
configPatches:
- applyTo: HTTP_FILTER
match:
context: SIDECAR_INBOUND
listener:
filterChain:
filter:
name: envoy.filters.network.http_connection_manager
patch:
operation: INSERT_BEFORE
value:
name: envoy.filters.http.local_ratelimit
typed_config:
'@type': type.googleapis.com/udpa.type.v1.TypedStruct
type_url: type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit
value:
stat_prefix: local_rate_limiter
token_bucket:
max_tokens: 100
tokens_per_fill: 100
fill_interval: 1s
filter_enabled:
runtime_key: local_rate_limit_enabled
default_value:
numerator: 100
denominator: HUNDREDHands-On Labs 18.4
Service Mesh Labs
- Install Istio on kind with istioctl, enable auto-injection on production namespace
- Deploy bookinfo app, verify all pods have istio-proxy sidecar running
- Enable STRICT mTLS mesh-wide, verify with istioctl authn tls-check
- Create canary VirtualService — 90/10 split between v1/v2, verify with curl loop
- Apply outlier detection DestinationRule, manually kill pods, watch circuit breaker trip
- Install Kiali, observe real-time service graph during load test with k6
- Write AuthorizationPolicy that only allows payments-api → fraud-service, deny all others
Troubleshooting 18.5
Non-injected pods (e.g., monitoring tools, jobs without sidecar) cannot connect. Add istio-injection: enabled label to their namespace, or create a PeerAuthentication exception for that port. Also check external services that call into the cluster — they need TLS termination at the ingress gateway.
Check: (1) DestinationRule subsets match the actual pod labels exactly. (2) VirtualService hosts field uses the correct service name. (3) VirtualService is in the same namespace as the service. Debug with istioctl proxy-config routes <pod> to see what Envoy sees.
Consider migrating to Ambient Mesh (Istio 1.22+) which eliminates sidecars. Short-term: set resource limits on the sidecar via ProxyConfig CR or meshConfig.defaultConfig.proxyMetadata. Reduce logging verbosity with istioctl proxy-config log <pod> --level warning.