Skip to content
DevOps Architect

    Syllabus / Operations / 18

    ZERO-TRUST EAST-WEST TRAFFIC
    18 / 19 • ISTIO & SERVICE MESH

    🕸Istio & Service Mesh

    Secure, observe, and control every byte of east-west traffic in your Kubernetes cluster. Master Istio mTLS enforcement, progressive delivery via VirtualService weight-shifting, circuit breakers with DestinationRule, Ambient Mesh sidecar-less architecture, and Kiali for real-time traffic topology during incidents.

    Istio Install + Kiali mTLS + Canary Ambient Mesh + Multi-cluster

     Real Incident: Lateral Movement Contained by mTLS

    Scenario: A compromised CI/CD runner had AWS credentials and a valid Kubernetes service account token. Without mTLS, it could call any service in the cluster over HTTP.

    With Istio PeerAuthentication STRICT mode:

    # Attacker tries to call payments-api from compromised pod
    attacker$ curl http://payments-api.production.svc.cluster.local/v1/transfer \
      -H "Authorization: Bearer stolen-token" \
      -d '{"amount": 99999, "to": "attacker-account"}'
    upstream connect error or disconnect/reset before headers. 
    reset reason: connection failure
    transport failure reason: TLS error: 268435581...
    
    # The connection was REJECTED because the compromised pod's
    # certificate was not in the payments-api's SPIFFE trust bundle
    # Istio enforced mTLS — no valid cert, no connection.
    
    # Check Istio access logs (what was attempted)
    $ kubectl logs -n production deploy/payments-api -c istio-proxy | \
      grep "PEER_CERT_VERIFY_FAILED"

    Lesson: Network policies alone are not enough. mTLS with SPIFFE certificate identity provides cryptographic proof of service identity. Enable STRICT mode across all production namespaces.

    Istio Architecture 18.1

    graph TD subgraph "Control Plane" Istiod[istiod
    Pilot + Citadel + Galley] end subgraph "Data Plane (Sidecar Mode)" Pod1["Service A\napp container\nEnvoy sidecar"] Pod2["Service B\napp container\nEnvoy sidecar"] end subgraph "Data Plane (Ambient Mode)" ZTunnel[ztunnel DaemonSet
    L4 mTLS per node] Waypoint[Waypoint Proxy
    L7 per namespace] end subgraph "Observability" Prometheus2[Prometheus
    Istio metrics] Kiali[Kiali
    Traffic topology] Jaeger[Jaeger/Tempo
    Distributed traces] end Istiod -->|certs + config| Pod1 Istiod -->|certs + config| Pod2 Istiod -->|certs + config| ZTunnel Pod1 -->|mTLS| Pod2 ZTunnel --> Waypoint Pod1 -->|metrics| Prometheus2 Prometheus2 --> Kiali Pod1 -->|traces| Jaeger

    Sidecar Mode

    Envoy proxy injected as sidecar into every Pod. Full L7 visibility. High resource overhead (~50MB/pod). Best for: existing clusters, full HTTP/gRPC observability, request-level policies.

    Ambient Mode (Istio 1.22+)

    No sidecars. ztunnel DaemonSet handles L4 mTLS. Optional Waypoint proxy for L7. 90% less memory overhead. Best for: new clusters, cost-sensitive environments, clusters with thousands of pods.

    mTLS Zero-Trust Enforcement 18.2

    peer-authentication-strict.yaml
    # Step 1: Enable PERMISSIVE mesh-wide (allows both mTLS and plaintext)
    # Use this first to verify all traffic switches to mTLS
    apiVersion: security.istio.io/v1beta1
    kind: PeerAuthentication
    metadata:
      name: default
      namespace: istio-system   # mesh-wide scope
    spec:
      mtls:
        mode: PERMISSIVE
    
    ---
    # Step 2: After verifying no plaintext traffic in Kiali,
    # switch to STRICT (breaks plaintext connections)
    apiVersion: security.istio.io/v1beta1
    kind: PeerAuthentication
    metadata:
      name: default
      namespace: istio-system
    spec:
      mtls:
        mode: STRICT
    
    ---
    # Namespace-level override (more granular)
    apiVersion: security.istio.io/v1beta1
    kind: PeerAuthentication
    metadata:
      name: production-strict
      namespace: production
    spec:
      mtls:
        mode: STRICT
      # Port-specific exception for health probes
      portLevelMtls:
        8080:
          mode: PERMISSIVE  # liveness probe from kubelet (no cert)
    authorization-policy.yaml
    # Only payments-api service account can call fraud-service
    # All other callers get 403 Forbidden
    apiVersion: security.istio.io/v1beta1
    kind: AuthorizationPolicy
    metadata:
      name: fraud-service-policy
      namespace: production
    spec:
      selector:
        matchLabels:
          app: fraud-service
      action: ALLOW
      rules:
        - from:
            - source:
                # SPIFFE identity — cryptographic, not IP-based
                principals:
                  - cluster.local/ns/production/sa/payments-api
          to:
            - operation:
                methods: ["POST"]
                paths: ["/v1/fraud/check"]
    
    ---
    # Deny all ingress except from ingress gateway
    apiVersion: security.istio.io/v1beta1
    kind: AuthorizationPolicy
    metadata:
      name: deny-all-except-gateway
      namespace: production
    spec:
      action: DENY
      rules:
        - from:
            - source:
                notPrincipals:
                  - cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account
    # Check mTLS status between two services
    $ istioctl x check-inject -n production
    NAMESPACE    POD                    INJECTED
    production   payments-api-7d9f8b    true
    production   fraud-service-5c4d3a   true
    
    # Verify mTLS is working (connection is encrypted)
    $ istioctl authn tls-check payments-api.production.svc.cluster.local
    HOST                                    STATUS     SERVER     CLIENT
    payments-api.production.svc.cluster.local mTLS       STRICT     mTLS
    
    # Check proxy config for specific pod
    $ istioctl proxy-config listeners payments-api-7d9f8b -n production
    
    # Real-time access log from Envoy sidecar
    $ kubectl logs payments-api-7d9f8b -n production -c istio-proxy --tail=50

    Traffic Management: Canary & Circuit Breaker 18.3

    canary-virtual-service.yaml
    # Progressive canary: send 10% traffic to v2
    # No DNS changes, no load balancer rules — pure Envoy routing
    apiVersion: networking.istio.io/v1beta1
    kind: VirtualService
    metadata:
      name: payments-api
      namespace: production
    spec:
      hosts:
        - payments-api
      http:
        # Header-based testing: QA team always gets v2
        - match:
            - headers:
                x-canary-user:
                  exact: "true"
          route:
            - destination:
                host: payments-api
                subset: v2
        # Weight-based: 90/10 split
        - route:
            - destination:
                host: payments-api
                subset: v1
              weight: 90
            - destination:
                host: payments-api
                subset: v2
              weight: 10
    ---
    apiVersion: networking.istio.io/v1beta1
    kind: DestinationRule
    metadata:
      name: payments-api
      namespace: production
    spec:
      host: payments-api
      subsets:
        - name: v1
          labels:
            version: v1
        - name: v2
          labels:
            version: v2
      trafficPolicy:
        connectionPool:
          http:
            h2UpgradePolicy: UPGRADE
    circuit-breaker-destination-rule.yaml
    # Circuit breaker: eject unhealthy endpoints
    # If >50% requests fail, eject host for 5 minutes
    apiVersion: networking.istio.io/v1beta1
    kind: DestinationRule
    metadata:
      name: inventory-service
      namespace: production
    spec:
      host: inventory-service
      trafficPolicy:
        connectionPool:
          tcp:
            maxConnections: 100
          http:
            http1MaxPendingRequests: 50
            http2MaxRequests: 100
            maxRetries: 3
        outlierDetection:
          # Eject if >5 errors in 30s consecutive interval
          consecutiveGatewayErrors: 5
          interval: 30s
          # Keep ejected for 5 minutes
          baseEjectionTime: 5m
          # Never eject more than 50% of endpoints
          maxEjectionPercent: 50
          # Also detect 5xx errors (not just gateway errors)
          consecutive5xxErrors: 10
    fault-injection-chaos.yaml
    # Chaos engineering: inject 500ms delay for 20% of requests
    # Use this to test timeout handling before incidents happen
    apiVersion: networking.istio.io/v1beta1
    kind: VirtualService
    metadata:
      name: inventory-service-chaos
      namespace: production
    spec:
      hosts:
        - inventory-service
      http:
        - fault:
            delay:
              percentage:
                value: 20.0
              fixedDelay: 500ms
            abort:
              percentage:
                value: 5.0   # 5% of requests return HTTP 503
              httpStatus: 503
          route:
            - destination:
                host: inventory-service
    rate-limit-envoy-filter.yaml
    # Local rate limiting via EnvoyFilter
    # Limits: 100 req/s per source IP on the /v1/search endpoint
    apiVersion: networking.istio.io/v1alpha3
    kind: EnvoyFilter
    metadata:
      name: search-rate-limit
      namespace: production
    spec:
      workloadSelector:
        labels:
          app: search-api
      configPatches:
        - applyTo: HTTP_FILTER
          match:
            context: SIDECAR_INBOUND
            listener:
              filterChain:
                filter:
                  name: envoy.filters.network.http_connection_manager
          patch:
            operation: INSERT_BEFORE
            value:
              name: envoy.filters.http.local_ratelimit
              typed_config:
                '@type': type.googleapis.com/udpa.type.v1.TypedStruct
                type_url: type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit
                value:
                  stat_prefix: local_rate_limiter
                  token_bucket:
                    max_tokens: 100
                    tokens_per_fill: 100
                    fill_interval: 1s
                  filter_enabled:
                    runtime_key: local_rate_limit_enabled
                    default_value:
                      numerator: 100
                      denominator: HUNDRED

    Hands-On Labs 18.4

    Service Mesh Labs

    • Install Istio on kind with istioctl, enable auto-injection on production namespace
    • Deploy bookinfo app, verify all pods have istio-proxy sidecar running
    • Enable STRICT mTLS mesh-wide, verify with istioctl authn tls-check
    • Create canary VirtualService — 90/10 split between v1/v2, verify with curl loop
    • Apply outlier detection DestinationRule, manually kill pods, watch circuit breaker trip
    • Install Kiali, observe real-time service graph during load test with k6
    • Write AuthorizationPolicy that only allows payments-api → fraud-service, deny all others

    Troubleshooting 18.5

    Non-injected pods (e.g., monitoring tools, jobs without sidecar) cannot connect. Add istio-injection: enabled label to their namespace, or create a PeerAuthentication exception for that port. Also check external services that call into the cluster — they need TLS termination at the ingress gateway.

    Check: (1) DestinationRule subsets match the actual pod labels exactly. (2) VirtualService hosts field uses the correct service name. (3) VirtualService is in the same namespace as the service. Debug with istioctl proxy-config routes <pod> to see what Envoy sees.

    Consider migrating to Ambient Mesh (Istio 1.22+) which eliminates sidecars. Short-term: set resource limits on the sidecar via ProxyConfig CR or meshConfig.defaultConfig.proxyMetadata. Reduce logging verbosity with istioctl proxy-config log <pod> --level warning.

    Run: attacker$ curl http://payments-api.production.svc.cluster.local/v1/transfer \

    Extra commands from this lesson (8) are kept out of this page. Quizzes were not in the source HTML.