Kubernetes: DaemonSets & StatefulSets

Lesson 1 of 4

Advanced Workload Controllers

Beyond Deployments

While Deployments are perfect for stateless applications, Kubernetes provides specialized controllers for different use cases:

Workload Controller Types

Deployment: Stateless applications, replicated Pods
DaemonSet: One Pod per node (system services)
StatefulSet: Stateful applications with stable identity
Job: One-off tasks
CronJob: Scheduled tasks

When to Use Each Controller

Controller	Use Case	Examples
Deployment	Stateless, replicated apps	Web servers, APIs, microservices
DaemonSet	One Pod per node	Monitoring agents, log collectors, network plugins
StatefulSet	Stateful apps with identity	Databases, message queues, distributed systems
Job	Run-to-completion tasks	Batch processing, data migration
CronJob	Scheduled tasks	Backups, reports, cleanup jobs

This Course Focus

This lesson covers two critical controllers for production Kubernetes:

DaemonSets

Solve the per-node problem:

Automatically launch one Pod on every node
Centrally managed and configured
Perfect for node-level system services
Examples: monitoring agents, log collectors, network overlay components

StatefulSets

Manage stateful applications:

Stable network identity: Predictable hostnames
Ordered deployment: Sequential Pod creation/deletion
Persistent storage: Stable volume attachments
Examples: databases (MongoDB, PostgreSQL), message queues (Kafka), distributed systems

Course Outline

DaemonSets: One Pod per node for system services
StatefulSets: Managing databases and stateful applications
Kubernetes Federation: Multi-cluster management (bonus topic)

Prerequisites

Before starting this course, you should understand:

Pods and containers
Deployments and ReplicaSets
Services and networking
Volumes and storage

Lesson 2 of 4

DaemonSets: One Pod Per Node

The Per-Node Problem

Many system-level tasks require that a single instance of a Pod runs on every node in the cluster.

Common Scenarios

Monitoring agents: Collect metrics from each node and send to central location
Log collectors: Gather logs from all containers on a node
Network overlay: CNI plugins that need to run on every node
Storage daemons: Distributed storage agents
Security agents: Intrusion detection, vulnerability scanning

Requirements

The solution must ensure the agent:

Launches automatically on every node
Centrally managed and configured from one point
Scales with cluster: New nodes automatically get the Pod
Updates easily: Change image/config in one place

Why Alternative Solutions Fail

1. Static Pods (Not Suitable)

Static Pods Limitations

Static Pods are launched by kubelet based on manifest files in /etc/kubernetes/manifests/

✓ Run automatically on each node
✗ Cannot be centrally managed through Kubernetes API
✗ Must manually place manifests on each node
✗ No central configuration updates
✗ Typically used for Control Plane components only

2. Pod Anti-Affinity (Not Ideal)

Pod Anti-Affinity Approach

Pod anti-affinity can prevent two Pods from running on the same node, but:

✗ Complex to configure for every node
✗ Doesn't guarantee one Pod per node
✗ Requires manual scaling to match node count
✗ Not automatic when nodes are added/removed

DaemonSet: The Dedicated Solution

DaemonSet Guarantees

A DaemonSet ensures that exactly one copy of a Pod runs on every node (or selected nodes):

✓ Automatically launches on all nodes
✓ Centrally managed via Kubernetes API
✓ Automatically deploys to new nodes
✓ Removes Pods when nodes are deleted
✓ Single point of configuration
✓ Easy to update across all nodes

DaemonSet Manifest

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: monitoring-agent
  namespace: kube-system
  labels:
    app: monitoring
spec:
  selector:
    matchLabels:
      app: monitoring
  template:
    metadata:
      labels:
        app: monitoring
    spec:
      containers:
      - name: node-exporter
        image: prom/node-exporter:v1.5.0
        ports:
        - containerPort: 9100
          name: metrics
        volumeMounts:
        - name: proc
          mountPath: /host/proc
          readOnly: true
        - name: sys
          mountPath: /host/sys
          readOnly: true
      volumes:
      - name: proc
        hostPath:
          path: /proc
      - name: sys
        hostPath:
          path: /sys
      # Run on all nodes including masters
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/control-plane

How DaemonSets Work

1. DaemonSet Created
kubectl apply -f daemonset.yaml

↓

2. DaemonSet Controller Watches
Controller identifies all nodes in cluster

↓

3. Pods Created
One Pod scheduled to each node automatically

↓

4. New Node Joins
DaemonSet controller detects new node

↓

5. Pod Auto-Deployed
Pod automatically created on new node

↓

6. Node Removed
Pod automatically deleted when node leaves

Visual Example

DaemonSet Deployment Across Cluster

Node 1 (master)

📊 monitoring-agent-abc12

Node 2 (worker)

📊 monitoring-agent-def34

Node 3 (worker)

📊 monitoring-agent-ghi56

Node 4 (worker)

📊 monitoring-agent-jkl78

Every node has exactly one monitoring-agent Pod

DaemonSet Operations

Create DaemonSet

# Apply DaemonSet manifest
kubectl apply -f daemonset.yaml

# View DaemonSets
kubectl get daemonsets -n kube-system

NAME               DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE
monitoring-agent   4         4         4       4            4

# DESIRED: Number of nodes that should have the Pod
# CURRENT: Number of nodes that have the Pod
# READY: Number of Pods that are ready

View DaemonSet Pods

# List Pods created by DaemonSet
kubectl get pods -n kube-system -l app=monitoring

NAME                     READY   STATUS    RESTARTS   AGE   NODE
monitoring-agent-abc12   1/1     Running   0          5m    node1
monitoring-agent-def34   1/1     Running   0          5m    node2
monitoring-agent-ghi56   1/1     Running   0          5m    node3
monitoring-agent-jkl78   1/1     Running   0          5m    node4

# Notice one Pod per node

Update DaemonSet

# Update image version
kubectl set image daemonset/monitoring-agent \
  node-exporter=prom/node-exporter:v1.6.0 \
  -n kube-system

# DaemonSet performs rolling update across all nodes
# Old Pods replaced one by one with new version

Node Selectors and Tolerations

Run on Specific Nodes Only

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: gpu-monitoring
spec:
  selector:
    matchLabels:
      app: gpu-monitor
  template:
    metadata:
      labels:
        app: gpu-monitor
    spec:
      # Only run on nodes with GPU
      nodeSelector:
        hardware: gpu
      containers:
      - name: gpu-monitor
        image: nvidia/gpu-monitor:latest

Run on Master Nodes

spec:
  template:
    spec:
      # Tolerate master node taint
      tolerations:
      - key: node-role.kubernetes.io/control-plane
        effect: NoSchedule
      # Or tolerate all taints
      - operator: Exists

Common DaemonSet Examples

1. Log Collector (Fluentd)

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-system
spec:
  selector:
    matchLabels:
      app: fluentd
  template:
    metadata:
      labels:
        app: fluentd
    spec:
      containers:
      - name: fluentd
        image: fluent/fluentd-kubernetes-daemonset:v1
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers

2. Network Plugin (Calico)

# Calico CNI runs as DaemonSet
kubectl get daemonset -n kube-system calico-node

NAME          DESIRED   CURRENT   READY
calico-node   4         4         4

# Ensures network plugin runs on every node

3. Storage Plugin (Ceph)

# Distributed storage agents as DaemonSet
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: ceph-osd
spec:
  selector:
    matchLabels:
      app: ceph-osd
  template:
    metadata:
      labels:
        app: ceph-osd
    spec:
      containers:
      - name: osd
        image: ceph/daemon:latest
        volumeMounts:
        - name: devices
          mountPath: /dev
      volumes:
      - name: devices
        hostPath:
          path: /dev

DaemonSet Best Practices

Use for node-level system services only
Keep DaemonSet Pods lightweight (low resource usage)
Configure proper resource limits
Use readiness probes to ensure Pod is ready
Set appropriate tolerations for master nodes
Use node selectors to target specific node types
Monitor DaemonSet Pods across all nodes

Lesson 3 of 4

StatefulSets: Managing Stateful Applications

The Stateful Application Challenge

Some applications require more than what Deployments provide:

Deployment Limitations for Stateful Apps

Deployments treat all Pods as identical and interchangeable:

✗ Pods get random names (webapp-abc123)
✗ No stable network identity
✗ No guaranteed order of creation/deletion
✗ No persistent storage association
✗ Pods can be replaced on any node

This doesn't work for databases, message queues, or distributed systems!

What StatefulSets Provide

StatefulSet Guarantees

StatefulSets provide three critical features for stateful applications:

Stable Network Identity: Predictable, persistent hostnames
Ordered Deployment & Scaling: Sequential Pod creation/deletion
Persistent Storage: Stable volume attachments that persist across rescheduling

1. Stable Network Identity

Each Pod in a StatefulSet gets a stable, unique network identity that persists across rescheduling.

Naming Pattern

# StatefulSet name: mongodb
# Pods are named with ordinal index:

mongodb-0  ← First Pod (ordinal 0)
mongodb-1  ← Second Pod (ordinal 1)
mongodb-2  ← Third Pod (ordinal 2)

# Pod names are STABLE:
# - If mongodb-1 is deleted, the replacement is also named mongodb-1
# - Same hostname, same identity
# - This is critical for stateful systems

Stable Hostnames via Headless Service

# StatefulSet requires a Headless Service
apiVersion: v1
kind: Service
metadata:
  name: mongodb
spec:
  clusterIP: None  # Headless service
  selector:
    app: mongodb
  ports:
  - port: 27017

# Each Pod gets DNS name:
mongodb-0.mongodb.default.svc.cluster.local
mongodb-1.mongodb.default.svc.cluster.local
mongodb-2.mongodb.default.svc.cluster.local

# These DNS names are STABLE and persist across Pod restarts!

Why Stable Identity Matters

For databases and distributed systems:

Nodes need to discover each other by name
Configuration files reference specific members
Replication requires consistent identities
Example: MongoDB replica set members are identified by hostname

2. Ordered Deployment and Scaling

StatefulSets maintain a strict, ordinal index and deploy/scale/update Pods in a predictable, ordered manner.

Deployment Order

Create StatefulSet (replicas: 3)
kubectl apply -f statefulset.yaml

↓

Step 1: Create mongodb-0
Wait until Running and Ready

↓

Step 2: Create mongodb-1
Only after mongodb-0 is Ready
Wait until Running and Ready

↓

Step 3: Create mongodb-2
Only after mongodb-1 is Ready
All Pods now running

Termination Order

Delete StatefulSet (or scale down)
Pods deleted in REVERSE order

↓

Step 1: Delete mongodb-2
Wait until fully terminated

↓

Step 2: Delete mongodb-1
Only after mongodb-2 is terminated
Wait until fully terminated

↓

Step 3: Delete mongodb-0
Only after mongodb-1 is terminated

Why Order Matters

For databases and distributed systems:

Initialization: Primary node must start before replicas
Configuration: Earlier nodes configure later nodes
Graceful shutdown: Proper cleanup order prevents data loss
Example: MongoDB primary (mongodb-0) must be ready before secondaries join

3. Persistent Storage

StatefulSets manage PersistentVolumeClaims (PVCs) for each ordinal, ensuring stable storage.

Volume Claim Templates

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mongodb
spec:
  serviceName: mongodb
  replicas: 3
  selector:
    matchLabels:
      app: mongodb
  template:
    metadata:
      labels:
        app: mongodb
    spec:
      containers:
      - name: mongodb
        image: mongo:6.0
        volumeMounts:
        - name: data
          mountPath: /data/db
  # Volume Claim Template
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      resources:
        requests:
          storage: 10Gi

How Storage Works

# StatefulSet creates PVCs automatically:

data-mongodb-0  ← PVC for mongodb-0 (10Gi)
data-mongodb-1  ← PVC for mongodb-1 (10Gi)
data-mongodb-2  ← PVC for mongodb-2 (10Gi)

# Critical behavior:
# 1. Each Pod gets its own PVC
# 2. PVC is bound to PersistentVolume (PV)
# 3. If Pod is deleted and recreated:
#    - New Pod gets SAME PVC
#    - Data persists across Pod restarts
# 4. PVCs are NOT deleted when StatefulSet is deleted
#    - Must manually delete to reclaim storage

Storage Persistence Example

# Initial state
mongodb-0 → data-mongodb-0 → PV (10Gi with database files)

# Pod mongodb-0 crashes and is deleted
# StatefulSet recreates it

# New mongodb-0 Pod
mongodb-0 → data-mongodb-0 → SAME PV (database files intact!)

# Data survives Pod restart

Complete StatefulSet Example

# Headless Service
apiVersion: v1
kind: Service
metadata:
  name: mongodb
spec:
  clusterIP: None
  selector:
    app: mongodb
  ports:
  - port: 27017
    name: mongodb

---
# StatefulSet
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: mongodb
spec:
  serviceName: mongodb
  replicas: 3
  selector:
    matchLabels:
      app: mongodb
  template:
    metadata:
      labels:
        app: mongodb
    spec:
      containers:
      - name: mongodb
        image: mongo:6.0
        command:
        - mongod
        - "--replSet"
        - rs0
        - "--bind_ip_all"
        ports:
        - containerPort: 27017
          name: mongodb
        volumeMounts:
        - name: data
          mountPath: /data/db
        resources:
          requests:
            memory: "1Gi"
            cpu: "500m"
          limits:
            memory: "2Gi"
            cpu: "1000m"
        livenessProbe:
          exec:
            command:
            - mongo
            - --eval
            - "db.adminCommand('ping')"
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          exec:
            command:
            - mongo
            - --eval
            - "db.adminCommand('ping')"
          initialDelaySeconds: 5
          periodSeconds: 5
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: "fast-ssd"
      resources:
        requests:
          storage: 10Gi

StatefulSet Operations

Create StatefulSet

# Apply manifests
kubectl apply -f service.yaml
kubectl apply -f statefulset.yaml

# Watch Pods being created in order
kubectl get pods -w

NAME        READY   STATUS              RESTARTS   AGE
mongodb-0   0/1     ContainerCreating   0          1s
mongodb-0   1/1     Running             0          30s
mongodb-1   0/1     ContainerCreating   0          31s
mongodb-1   1/1     Running             0          60s
mongodb-2   0/1     ContainerCreating   0          61s
mongodb-2   1/1     Running             0          90s

View StatefulSet Status

# Get StatefulSet
kubectl get statefulset mongodb

NAME      READY   AGE
mongodb   3/3     5m

# Describe StatefulSet
kubectl describe statefulset mongodb

# View Pods
kubectl get pods -l app=mongodb

# View PVCs
kubectl get pvc

NAME              STATUS   VOLUME    CAPACITY   STORAGECLASS
data-mongodb-0    Bound    pv-001    10Gi       fast-ssd
data-mongodb-1    Bound    pv-002    10Gi       fast-ssd
data-mongodb-2    Bound    pv-003    10Gi       fast-ssd

Scale StatefulSet

# Scale up to 5 replicas
kubectl scale statefulset mongodb --replicas=5

# Pods created in order: mongodb-3, then mongodb-4
# Each gets its own PVC

# Scale down to 2 replicas
kubectl scale statefulset mongodb --replicas=2

# Pods deleted in reverse order: mongodb-4, mongodb-3, mongodb-2
# PVCs remain for future use

Update StatefulSet

# Update image version
kubectl set image statefulset/mongodb mongodb=mongo:6.1

# Rolling update strategy (default):
# - Updates in reverse order: mongodb-2, mongodb-1, mongodb-0
# - Waits for each Pod to be Ready before updating next

StatefulSet vs Deployment

Feature	Deployment	StatefulSet
Pod Names	Random (webapp-abc123)	Ordered (mongodb-0, mongodb-1)
Network Identity	None (pods interchangeable)	Stable DNS names
Creation Order	All at once (parallel)	Sequential (one by one)
Deletion Order	Random	Reverse sequential
Storage	Shared or ephemeral	Per-Pod persistent volumes
Use Case	Stateless applications	Stateful applications

Common StatefulSet Use Cases

1. Databases

MongoDB replica sets
PostgreSQL with replication
MySQL/MariaDB clusters
Cassandra
Redis with persistence

2. Message Queues

Kafka (requires stable broker IDs)
RabbitMQ clusters
Pulsar

3. Distributed Systems

Elasticsearch clusters
ZooKeeper ensembles
etcd clusters
Consul

StatefulSet Best Practices

Always create Headless Service first
Use PodDisruptionBudgets to prevent disruptions
Configure appropriate resource requests/limits
Implement readiness probes (critical for ordered startup)
Use init containers for initialization tasks
Consider using Operators for complex stateful apps
Test disaster recovery procedures
Back up persistent volumes regularly

StatefulSet Considerations

PVCs are NOT deleted with StatefulSet (manual cleanup needed)
Slower deployments due to sequential ordering
Requires more careful management than Deployments
Storage class must support dynamic provisioning
Deleting a Pod doesn't delete its PVC (prevents data loss)

Lesson 4 of 4

Kubernetes Federation (Bonus Topic)

Multi-Cluster Management

As organizations scale, they often need to run applications across multiple Kubernetes clusters in different data centers or regions.

Multi-Cluster Scenarios

Geographic distribution: Clusters in different regions for low latency
High availability: Survive datacenter failures
Compliance: Data residency requirements
Environment separation: Dev, staging, prod in separate clusters
Cloud diversity: Avoid vendor lock-in

What is Kubernetes Federation?

Federation Purpose

Kubernetes Federation is designed to unite multiple, separate Kubernetes clusters into a single, cohesive entity.

It provides a single point of control for managing resources across all member clusters.

Federation Capabilities

1. Unified Resource Management

# Without Federation:
# Deploy to Cluster 1
kubectl --context=cluster1 apply -f deployment.yaml
# Deploy to Cluster 2
kubectl --context=cluster2 apply -f deployment.yaml
# Deploy to Cluster 3
kubectl --context=cluster3 apply -f deployment.yaml

# With Federation:
# Deploy to ALL clusters with one command
kubefedctl create federateddeployment my-app --from-file deployment.yaml

# Deployment simultaneously applied to all member clusters!

2. Cross-Cluster View

View combined status of Pods running across all member clusters:

# View Pods across all federated clusters
kubectl get federateddeployments

NAME     CLUSTERS   REPLICAS   AGE
my-app   3          9/9        5m

# Shows:
# - 3 clusters in federation
# - Total 9 Pods across all clusters (3 per cluster)
# - All replicas running

3. Intelligent Scheduling

Federation can distribute workloads based on:

Cluster capacity and resources
Geographic preferences
Cluster policies
Cost optimization

Federation Architecture

Federation Setup

Federation Control Plane
Manages federated resources across all clusters

⬇ Manages ⬇

Cluster 1 (US-East)
Full Kubernetes cluster

Cluster 2 (US-West)
Full Kubernetes cluster

Cluster 3 (EU)
Full Kubernetes cluster

Federation Use Cases

1. Multi-Datacenter Deployment

# Deploy application to multiple DCs
apiVersion: types.kubefed.io/v1beta1
kind: FederatedDeployment
metadata:
  name: webapp
  namespace: default
spec:
  placement:
    clusters:
    - name: us-east
    - name: us-west
    - name: europe
  template:
    spec:
      replicas: 3
      selector:
        matchLabels:
          app: webapp
      template:
        metadata:
          labels:
            app: webapp
        spec:
          containers:
          - name: webapp
            image: myapp:1.0

# Creates 3 replicas in EACH cluster (9 total Pods)

2. Disaster Recovery

If one cluster fails, traffic automatically routes to healthy clusters:

Applications continue running in other DCs
Automatic failover
Minimal downtime

3. Compliance and Data Residency

# Deploy to specific regions based on data residency
apiVersion: types.kubefed.io/v1beta1
kind: FederatedDeployment
metadata:
  name: eu-data-app
spec:
  placement:
    clusters:
    - name: eu-west  # Only deploy to EU cluster
  template:
    # ... deployment spec

Federation Project Status

Federation v1: Deprecated

The first version of Kubernetes Federation was ultimately unsuccessful due to design issues:

Complex to set up and maintain
Limited adoption
Architectural limitations
Officially deprecated and removed

Federation v2 (KubeFed)

A second version is under development with improved design:

Project name: KubeFed (Kubernetes Cluster Federation)
Status: Beta/experimental
Improved architecture: More flexible and extensible
Production readiness: Still uncertain for critical workloads
Active development: Ongoing improvements

Alternatives to Federation

While Federation v2 matures, consider these alternatives for multi-cluster management:

1. Multi-Cluster Service Mesh

Istio multi-cluster: Connect multiple clusters at the service mesh level
Linkerd: Multi-cluster service discovery
Provides cross-cluster service communication

2. GitOps Approaches

Flux/ArgoCD: Deploy same manifests to multiple clusters
Infrastructure as code for all clusters
Centralized configuration management

3. Third-Party Solutions

Rancher: Multi-cluster Kubernetes management platform
Google Anthos: Unified management across clouds
Red Hat Advanced Cluster Management: Enterprise multi-cluster

When to Consider Federation

Good Use Cases

Large organizations with multiple datacenters
Global applications requiring regional deployment
High availability across geographic regions
Compliance with data residency requirements
Disaster recovery strategies

Consider Carefully If:

Single datacenter deployment
Small team with limited expertise
Simple application architecture
Just starting with Kubernetes
Need production-stable solution today

Summary: Advanced Workloads

Course Summary

DaemonSets

One Pod per node automatically
Perfect for system services (monitoring, logging)
Centrally managed and configured

StatefulSets

Stable network identity with predictable names
Ordered deployment and scaling
Persistent storage per Pod
Essential for databases and stateful apps

Kubernetes Federation

Multi-cluster management
Single control point for multiple clusters
v1 deprecated, v2 under development
Consider alternatives for production

Advanced Kubernetes Workloads