Kubernetes in a Day — Senior Level

01Architecture — Control Plane Deep

etcd ← all cluster state (Raft consensus, only source of truth)
↕
kube-apiserver ← REST endpoint, auth/authz, admission webhooks, watches
↕
kube-scheduler kube-controller-manager cloud-controller
↓ (watch API, assign node) ↓ (reconcile loops) ↓ (cloud LBs, volumes)
kubelet (each node) ← watch API, runs pods via CRI (containerd)
↕
kube-proxy ← iptables/IPVS rules for Service routing

02Key Concepts

etcd + Raft

All K8s state: pod specs, secrets, configmaps in etcd
Raft quorum: 3 or 5 members (n/2+1 write quorum)
etcd is the only stateful component — backup critical
Watch API: long-poll on key prefix → event streaming
Leader election: controllers use etcd distributed locks

Pod Lifecycle

Pending → ContainerCreating → Running → Terminating
Init containers: run before main containers
Sidecar pattern: shared namespace, lifecycle coupling
Graceful shutdown: SIGTERM → preStop hook → 30s grace → SIGKILL
readinessProbe: removes from Service endpoints (critical!)

Networking: CNI

Every pod gets unique IP, flat network (no NAT intra-cluster)
CNI plugins: Calico (BGP), Flannel (VXLAN), Cilium (eBPF)
Cilium: replaces kube-proxy with eBPF, no iptables
Network Policy: L3/L4 firewall (Calico extends to L7)
Pod-to-pod: route via node veth pair + bridge

Services + kube-proxy

ClusterIP: virtual IP, iptables DNAT to pod IPs
NodePort: exposes on every node (30000-32767)
LoadBalancer: cloud LB integration
Headless: no VIP, DNS returns pod IPs (StatefulSets)
IPVS mode: O(1) routing vs iptables O(n) — use for large clusters

Resource Management

Requests: scheduling guarantee, cgroup reserved
Limits: cgroup enforcement (memory: OOMKill, CPU: throttle)
QoS: Guaranteed (req==lim), Burstable (req
HPA: scale on CPU/memory/custom metrics (via Prometheus adapter)
VPA: adjust requests based on actual usage

RBAC

ServiceAccount: identity for pods (JWT projected into pod)
Role/ClusterRole: what verbs on what resources
RoleBinding: binds subject to role in namespace
Principle of least privilege: no default service account power
IRSA (AWS): ServiceAccount → IAM role via OIDC

03Must-Know Deep Dives

🔥 Operators + CRDs — K8s Extensibility Model

CRD (Custom Resource Definition): extend K8s API with your own resources. kubectl apply -f my-cbdc-node.yaml just like a Deployment. An Operator = CRD + Controller that reconciles desired state.

Reconcile Loop: Watch CRD events → compare actual state vs desired → apply changes → requeue on error. Idempotent by design. Your Hyperledger Fabric multi-environment Helm deployments could have been an Operator — think about this.

// Operator reconcile loop (controller-runtime framework)
func (r *CBDCNodeReconciler) Reconcile(ctx context.Context, req ctrl.Request) (ctrl.Result, error) {
    cbdcNode := &cbdcv1.CBDCNode{}
    r.Get(ctx, req.NamespacedName, cbdcNode)  // Fetch current state
    
    // Check if Deployment exists, create if not
    deployment := &appsv1.Deployment{}
    err := r.Get(ctx, req.NamespacedName, deployment)
    if errors.IsNotFound(err) {
        r.Create(ctx, newDeployment(cbdcNode))  // Create desired state
    }
    return ctrl.Result{}, nil  // Done, requeue on next event
}

🔥 Admission Webhooks — Intercept All API Calls

Mutating admission webhooks: modify resources before persistence. Validating: reject invalid resources. Every kubectl apply goes through these. Istio, OPA/Gatekeeper, Kyverno all use admission webhooks.

Your use case: Automatically inject Kafka sidecar config into pods with annotation kafka.io/inject: "true". Set default resource limits if not specified. Block pods without security context. This is production-grade K8s engineering.

🔥 StatefulSets — When Pods Need Identity

StatefulSet guarantees: stable network ID (pod-0, pod-1, pod-2), stable storage (PVCs not deleted on reschedule), ordered deployment. Each pod DNS: pod-0.service.namespace.svc.cluster.local.

When to use: Kafka (brokers need stable broker.id), databases (replication requires leader election by identity), Hyperledger Fabric peers (your use case — each org peer needs persistent identity + TLS certs).

Headless Service + StatefulSet: headless service creates DNS A records per pod — clients can discover individual replicas. Used by Cassandra for peer discovery, ZooKeeper for quorum, your Fabric orderers.

🔥 Production Debugging Playbook

# Pod stuck in Pending?
kubectl describe pod  | grep -A 5 Events
# Look for: Insufficient cpu/memory, Unschedulable, PVC not bound

# CrashLoopBackOff?
kubectl logs  --previous   # Logs from crashed container
kubectl describe pod | grep -A 3 "Last State"

# Service not routing?
kubectl get endpoints    # Should have pod IPs
# If empty: readinessProbe failing or label selector mismatch

# Debug network (ephemeral container)
kubectl debug -it  --image=nicolaka/netshoot --target=
curl http://service.namespace.svc.cluster.local

# High CPU/Memory?
kubectl top pods --sort-by=cpu
kubectl top nodes

04Resources

VIDEO

TechWorld with Nana — Kubernetes Architecture Explained (45 min)

Best architecture walkthrough. Dense, no fluff. Watch at 1.5x.

BLOG

Ivan Velichko — K8s Operator Development (iximiuz.com)

Production-grade operator patterns. The best operator resource available.

DOCS

Kubernetes Official — Operator Pattern

Canonical definition. Read alongside controller-runtime examples.

BLOG

Cilium Blog — CNI Performance Benchmark

eBPF vs iptables in real numbers. Critical for networking interview questions.

BLOG

learnk8s.io — Production Best Practices Checklist

200+ item checklist. Skip what you know. Hits resource limits, probes, security, anti-affinity.

DOCS

Helm Chart Best Practices — Official

Given your Helm experience — this will surface gaps you didn't know you had.

05Quick Revision Notes

K8S INTERVIEW FLASH CARDS

What does etcd store?

ALL cluster state. Pod specs, secrets, configmaps, service accounts. Single source of truth. Raft consensus, backup critical.

iptables vs IPVS?

iptables: O(n) rule chain per service. IPVS: O(1) hash table. Use IPVS for >1000 services. Cilium bypasses both with eBPF.

Readiness vs Liveness probe?

Readiness: remove from Service endpoints (not ready → no traffic). Liveness: restart container (not alive → kill). Both critical.

How does HPA work?

Metrics server → apiserver aggregation layer. HPA controller polls every 15s, compares metric to target, adjusts replica count.

StatefulSet vs Deployment?

StatefulSet: stable DNS, stable PVCs, ordered rollout. Use for DB, Kafka, Fabric. Deployment: stateless, all pods identical.

PodDisruptionBudget?

Minimum available (or max unavailable) during voluntary disruptions (node drain). Critical for zero-downtime node upgrades.

Pod security admission?

Replaced PSP in K8s 1.25. Enforces: privileged/baseline/restricted. Set at namespace level. Blocks root containers, host namespaces.

Image pull backoff cause?

Wrong image name, missing imagePullSecrets for private registry, rate limit (Docker Hub). kubectl describe pod → Events for exact error.