DaemonSet in Kubernetes
A DaemonSet, a key Kubernetes workload API item, ensures that a Pod runs on all nodes in a cluster. Unlike Deployments and ReplicaSets, which focus on replica scalability and redundancy, the DaemonSet is specific to the cluster architecture topology. Adding new nodes to the cluster instantly alerts the DaemonSet controller to schedule a new Pod instance. On the other hand, the corresponding Pods are automatically trash collected when nodes are removed from the cluster. DaemonSets are essential for managing dynamic, immutable infrastructure, where virtual machines may be regularly erased and regenerated during scaling events or upgrades, due to their “self-driving” behavior.
You can also read What is a Kubernetes ReplicaSet & Working with ReplicaSets
How Daemon Pods are scheduled
Run a DaemonSet on every eligible node to ensure Pod replication. The DaemonSet controller creates a Pod and adjusts its spec.affinity.nodeAffinity field to match the destination host for each eligible node. After Pod formation, the default scheduler attaches it to the target host using the .spec.nodeName option. If the new Pod cannot fit on the node, the default scheduler may preempt (evict) some Pods based on priority.
The user can specify a different scheduler for the Pods of the DaemonSet, by setting the .spec.template.spec.schedulerName field of the DaemonSet.
The DaemonSet controller evaluates eligible nodes using the original node affinity at the .spec.template.spec.affinity.nodeAffinity field, but substitutes it on the generated Pod with the node’s name.
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchFields:
- key: metadata.name
operator: In
values:
- target-host-nameBuilding a Basic DaemonSet
Define the DaemonSet
A simple DaemonSet is made in this task to guarantee that a copy of a Pod is scheduled on each node. To read and log the contents of /etc/machine-id from the host, the Pod will use an init container. The main container will be a pause container, which keeps the Pod operating.
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: example-daemonset
spec:
selector:
matchLabels:
app.kubernetes.io/name: example
template:
metadata:
labels:
app.kubernetes.io/name: example
spec:
containers:
- name: pause
image: registry.k8s.io/pause
initContainers:
- name: log-machine-id
image: busybox:1.37
command: ['sh', '-c', 'cat /etc/machine-id > /var/log/machine-id.log']
volumeMounts:
- name: machine-id
mountPath: /etc/machine-id
readOnly: true
- name: log-dir
mountPath: /var/log
volumes:
- name: machine-id
hostPath:
path: /etc/machine-id
type: File
- name: log-dir
hostPath:
path: /var/log
Create a DaemonSet based on the (YAML) manifest:
kubectl apply -f https://k8s.io/examples/application/basic-daemonset.yaml
Once applied, you can verify that the DaemonSet is running a Pod on every node in the cluster:
kubectl get pods -o wide
The output will list one Pod per node, similar to:
NAME READY STATUS RESTARTS AGE IP NODE
example-daemonset-xxxxx 1/1 Running 0 5m x.x.x.x node-1
example-daemonset-yyyyy 1/1 Running 0 5m x.x.x.x node-2You can inspect the contents of the logged /etc/machine-id file by checking the log directory mounted from the host:
kubectl exec <pod-name> -- cat /var/log/machine-id.log
Where <pod-name> is the name of one of your Pods.
Cleaning up
To delete the DaemonSet, run this command:
kubectl delete --cascade=foreground --ignore-not-found --now daemonsets/example-daemonsetImportant elements like host path volumes and init containers are introduced in this basic DaemonSet example, which may be extended for more complex use cases. To learn more, see DaemonSet.
Primary Use Cases
DaemonSets are usually used for cluster-level functions and infrastructure elements that enhance the cluster’s functionality.
- Log Collection Agents: Logstash and Fluentd send node logs to GCP or Elasticsearch.
- Monitoring Agents: CPU, memory, and disk utilization statistics must be collected by the Prometheus node exporter, Datadog agents, or other monitoring daemons on each machine
- Networking Infrastructure: Calico and Flannel use DaemonSets for cluster and virtual network overlays.
- Cluster Storage Daemons: Ceph or GlusterFS on each node provides scalable, accessible storage.
- Security and Compliance: Falco or Sysdig Secure DaemonSets detect malicious and policy breaches on nodes.
You can also read What is Kubernetes Cloud Controller Manager?
Writing a DaemonSet Specification
YAML manifests for DaemonSets must provide apiVersion, kind, and metadata. The item must have a DNS subdomain name. Manifest’s heart, the .spec section, has two crucial parts:
- Pod Template: The
.spec.templatefield specifies the Pod to execute. Nested, it has a standard Pod schema. A Pod template in a DaemonSet must have aRestartPolicyset to Always (or left unspecified to default to Always) to restart with the node. - Pod Selector: To match the labels of the Pod template, utilize the
.spec.selector. This selector cannot be changed once a DaemonSet has been built since doing so can inadvertently cause existing Pods to become orphaned.
Running Pods on select Nodes
A .spec.template.spec.nodeSelector tells DaemonSet to create Pods on nodes. Pods will be generated on nodes with the specified affinity by the DaemonSet controller if you specify a .spec.template.spec.affinity. The DaemonSet controller creates pods for all nodes if neither is provided.
Scoping and Node Targeting
DaemonSets target all nodes by default, although node selectors and labels can limit them. This is essential for heterogeneous clusters with GPUs or SSDs. The controller will only deploy Pods to nodes with matching labels if the Pod spec has a nodeSelector. If a needed label is deleted, the DaemonSet controller will automatically remove the Pod from a node to preserve the defined state.
The DaemonSet controller automatically adds tolerances to Pods, allowing them to run on unschedulable or resource-pressure (e.g., disk-pressure, memory-pressure, or pid-pressure). This avoids deadlocks in case a crucial network plugin (running as a DaemonSet) cannot be scheduled on an unready node, preventing it from being certified “Ready.”
You can also read What is a Kubernetes Controller Manager?
Communicating with Daemon Pods
In a DaemonSet, some potential communication patterns for Pods are as follows:
- Service: Make a service using the same Pod selector, then use it to connect to a daemon on any node. To restrict to pods on the same node, apply the Service Internal Traffic Policy.
- Push: DaemonSet pods are set up to transmit updates to other services, such a statistics database. They are not in business.
- NodeIP and Known Port: To make the pods accessible through the node IPs, DaemonSet pods can use a
hostPort. Customers are aware of the port by custom and the list of node IPs in some way. - DNS: Use the
endpointsresource to find DaemonSets or obtain multiple A records from DNS after creating a headless service with the same pod selector.
Updating a DaemonSet
The DaemonSet will immediately add Pods to newly matching nodes and remove Pods from newly non-matching nodes if node labels are modified.
The Pods that a DaemonSet generates can be changed. Nevertheless, not all fields can be modified using Pods. Additionally, even if a node has the same name, the DaemonSet controller will utilize the original template the next time one is formed.
Deleting a DaemonSet is possible. The Pods will remain on the nodes if you use Kubectl with --cascade=orphan. The new DaemonSet adopts the current Pods if you later build a new DaemonSet with the same selector. If any Pods require replacement, DaemonSet replaces them in accordance with its updateStrategy.
Best Practices and Challenges
Important best practices include making sure that restart policies are right, utilizing rollbacks to swiftly undo changes, and utilizing DaemonSets only when Pod scaling is linked to node count. To prevent orphaning, administrators should refrain from manually managing individual DaemonSet Pods.
Resource overhead is a problem since executing copies of a Pod on each node can use a large amount of CPU and memory throughout the cluster. Additionally, there may be node variances where certain nodes may not have the RAM or specific hardware needed by the Pod, as well as node affinity problems brought on by mislabeling or network connectivity issues. Some observability platforms leverage eBPF (running in kernel space) via DaemonSets to reduce performance issues. This offers deep visibility with significantly less overhead than conventional user-space monitoring agents.
You can also read What is Kube-Proxy in Kubernetes and it’s Lifecycle