CKS: Monitoring, Logging and Runtime Security
20% of the CKS exam, 5 competencies. Back to the CKS overview.
Each competency lists what to know and the official documentation to read, then a demo you can run. Demos run on the demo lab, a throwaway kind cluster on your own machine; set it up once before you start.
Perform behavioral analytics to detect malicious activities
Section titled “Perform behavioral analytics to detect malicious activities”Know
- Runtime tools such as Falco watch syscalls and alert on behaviour like a shell spawning in a container.
Read
Demo: Install Falco and catch a shell in a container
-
Create a namespace for this demo and make it the default for your context.
Terminal window kubectl create namespace cks-falcokubectl config set-context --current --namespace=cks-falco -
Install Falco from its chart (9.2.2, Falco 0.45.2) on the lab cluster.
Terminal window helm repo add falcosecurity https://falcosecurity.github.io/chartshelm install falco falcosecurity/falco --version 9.2.2 -n falco --create-namespace --set tty=truekubectl -n falco rollout status daemonset/falco --timeout=300s -
Do what an attacker would: open an interactive shell and read a sensitive file.
Terminal window kubectl run victim --image=registry.k8s.io/e2e-test-images/busybox:1.37.0-2 -- sleep 3600kubectl wait --for=condition=Ready pod/victim --timeout=60skubectl exec -it victim -- sh -c 'cat /etc/shadow > /dev/null; exit' -
Read Falco’s alerts.
Terminal window kubectl -n falco logs -l app.kubernetes.io/name=falco -c falco --tail=50 | grep -E 'Terminal shell in container|Read sensitive file'
What you should see: Falco reports the Terminal shell in container rule and the Read sensitive file untrusted rule, each with the pod, namespace, command, and user. It watches system calls through eBPF, so it sees what happens inside containers regardless of what the application logs.
Note: Falco needs a Linux kernel with eBPF support on the host. Leave it installed for the next demo, or remove it with helm uninstall falco -n falco.
Clean up:
kubectl config set-context --current --namespace=defaultkubectl delete namespace cks-falcoPractise
- Trigger a Falco rule by running a shell inside a container and read the alert.
Detect threats within physical infrastructure, apps, networks, data, users and workloads
Section titled “Detect threats within physical infrastructure, apps, networks, data, users and workloads”Know
- Combine audit logs, runtime alerts, and network flow logs to see an attack across layers.
Read
Demo: Write your own Falco rule
-
Create a namespace for this demo and make it the default for your context.
Terminal window kubectl create namespace cks-falco-rulekubectl config set-context --current --namespace=cks-falco-rule -
A rule that fires when a container reads its service account token.
Terminal window cat > falco-custom.yaml <<'EOF'customRules:rules-demo.yaml: |-- rule: Service account token read in containerdesc: A process inside a container opened its Kubernetes service account tokencondition: open_read and container and fd.name startswith /var/run/secrets/kubernetes.io/serviceaccount/output: "SA token read (proc=%proc.name file=%fd.name pod=%k8s.pod.name ns=%k8s.ns.name)"priority: WARNINGEOFhelm upgrade falco falcosecurity/falco --version 9.2.2 -n falco --reuse-values -f falco-custom.yamlkubectl -n falco rollout status daemonset/falco --timeout=300s -
Trigger it and read the alert.
Terminal window kubectl run reader --image=registry.k8s.io/e2e-test-images/busybox:1.37.0-2 -- sleep 3600kubectl wait --for=condition=Ready pod/reader --timeout=60skubectl exec reader -- cat /var/run/secrets/kubernetes.io/serviceaccount/token > /dev/nullsleep 5; kubectl -n falco logs -l app.kubernetes.io/name=falco -c falco --tail=50 | grep 'SA token read'
What you should see: Your rule fires with the process, file, pod, and namespace. Real applications that talk to the API read this token legitimately, so a rule like this needs exceptions for known workloads before it is useful; tuning out noise is most of the work in runtime detection.
Clean up:
rm -f falco-custom.yamlhelm uninstall falco -n falco && kubectl delete namespace falcokubectl config set-context --current --namespace=defaultkubectl delete namespace cks-falco-ruleInvestigate and identify phases of attack and bad actors within the environment
Section titled “Investigate and identify phases of attack and bad actors within the environment”Know
- Map what you find to attack phases (initial access, execution, persistence, privilege escalation) to decide what to contain first.
Read
Demo: Reconstruct an attack from the audit log
-
Run this on the audit cluster from the “Use Kubernetes audit logs to monitor access” demo below. Play the attacker: a stolen token is used to look around, gain access, and persist.
Terminal window kubectl config use-context kind-auditkubectl create namespace victimkubectl create serviceaccount intruder -n victimkubectl create rolebinding intruder-view -n victim --clusterrole=view --serviceaccount=victim:intruderkubectl config view --minify --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d > audit-ca.crtSERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')kubectl config set-cluster audit --server="$SERVER" --certificate-authority=audit-ca.crt --embed-certs=true --kubeconfig=intruder.kubeconfigkubectl config set-credentials intruder --token="$(kubectl create token intruder -n victim)" --kubeconfig=intruder.kubeconfigkubectl config set-context intruder --cluster=audit --user=intruder --kubeconfig=intruder.kubeconfigK="kubectl --kubeconfig=intruder.kubeconfig --context=intruder"$K get pods -n victim$K get secrets -n kube-system$K create clusterrolebinding pwned --clusterrole=cluster-admin --serviceaccount=victim:intruder -
Now investigate: everything that identity did, in order, with the result.
Terminal window docker exec audit-control-plane cat /var/log/kubernetes/kube-apiserver-audit.log \| jq -r 'select(.user.username=="system:serviceaccount:victim:intruder") | [.requestReceivedTimestamp, .verb, (.objectRef.resource // .requestURI), (.objectRef.namespace // "-"), (.responseStatus.code|tostring), .sourceIPs[0]] | @tsv'
What you should see: The timeline shows discovery (listing pods), an attempt on Secrets in kube-system that got 403, and an attempt to bind cluster-admin that got 403. Failed attempts are often the earliest sign; the source IP tells you where the token is being used from. The attacker’s kubeconfig holds only the token, so nothing else authenticates the requests.
Clean up:
kubectl delete namespace victim && rm -f audit-ca.crt intruder.kubeconfigkubectl config use-context kind-kindEnsure immutability of containers at runtime
Section titled “Ensure immutability of containers at runtime”Know
readOnlyRootFilesystem: true, no package managers in images, and writable paths only through explicit volumes.
Read
Demo: Containers that can’t be changed, and a policy that requires it
-
Create a namespace for this demo and make it the default for your context.
Terminal window kubectl create namespace cks-immutablekubectl config set-context --current --namespace=cks-immutable -
Require
readOnlyRootFilesystemfor every container in a namespace.Terminal window kubectl label namespace cks-immutable immutability=requiredkubectl apply -f - <<'EOF'apiVersion: admissionregistration.k8s.io/v1kind: ValidatingAdmissionPolicymetadata:name: read-only-rootspec:failurePolicy: FailmatchConstraints:resourceRules:- apiGroups: [""]apiVersions: ["v1"]operations: ["CREATE", "UPDATE"]resources: ["pods"]validations:- expression: "object.spec.containers.all(c, has(c.securityContext) && has(c.securityContext.readOnlyRootFilesystem) && c.securityContext.readOnlyRootFilesystem == true)"message: "every container needs securityContext.readOnlyRootFilesystem: true"---apiVersion: admissionregistration.k8s.io/v1kind: ValidatingAdmissionPolicyBindingmetadata:name: read-only-rootspec:policyName: read-only-rootvalidationActions: ["Deny"]matchResources:namespaceSelector:matchLabels:immutability: requiredEOFsleep 3kubectl run mutable --image=registry.k8s.io/e2e-test-images/busybox:1.37.0-2 -- sleep 3600kubectl run immutable --image=registry.k8s.io/e2e-test-images/busybox:1.37.0-2 --overrides='{"spec":{"containers":[{"name":"immutable","image":"registry.k8s.io/e2e-test-images/busybox:1.37.0-2","command":["sleep","3600"],"securityContext":{"readOnlyRootFilesystem":true}}]}}' -
Try to change the running container.
Terminal window kubectl wait --for=condition=Ready pod/immutable --timeout=60skubectl exec immutable -- sh -c 'echo evil > /bin/evil' || truekubectl exec immutable -- sh -c 'echo "* * * * * evil" > /etc/crontabs/root' || true
What you should see: mutable is rejected by the policy and immutable is admitted. Inside it, every write fails with Read-only file system. Combine this with no shell in the image, a pinned digest, and a runtime detector like Falco, and a change to a running container becomes both hard and visible.
Clean up:
kubectl delete validatingadmissionpolicybinding read-only-rootkubectl delete validatingadmissionpolicy read-only-rootkubectl config set-context --current --namespace=defaultkubectl delete namespace cks-immutablePractise
- Make a pod read-only except for
/tmpthrough anemptyDir.
Use Kubernetes audit logs to monitor access
Section titled “Use Kubernetes audit logs to monitor access”Know
- Write an audit policy, enable it on the API server with
--audit-policy-fileand a log backend, and search the log for Secret reads.
Read
Demo: An audit policy and the questions it answers
-
Write a policy: no noise from health checks, metadata only for Secrets, full bodies for RBAC changes.
Terminal window mkdir -p audit && cat > audit/policy.yaml <<'EOF'apiVersion: audit.k8s.io/v1kind: PolicyomitStages: ["RequestReceived"]rules:- level: NonenonResourceURLs: ["/healthz*", "/livez*", "/readyz*", "/version"]- level: Noneusers: ["system:kube-proxy"]verbs: ["watch"]- level: Metadataresources:- group: ""resources: ["secrets", "configmaps"]- level: RequestResponseresources:- group: rbac.authorization.k8s.ioresources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]- level: Requestresources:- group: ""resources: ["pods/exec", "pods/attach", "pods/portforward"]- level: MetadataEOF -
Create a cluster that uses it (this follows the kind auditing guide).
Terminal window cat > audit/kind.yaml <<'EOF'kind: ClusterapiVersion: kind.x-k8s.io/v1alpha4nodes:- role: control-planekubeadmConfigPatches:- |kind: ClusterConfigurationapiServer:extraArgs:audit-log-path: /var/log/kubernetes/kube-apiserver-audit.logaudit-policy-file: /etc/kubernetes/policies/audit-policy.yamlextraVolumes:- name: audit-policieshostPath: /etc/kubernetes/policiesmountPath: /etc/kubernetes/policiesreadOnly: truepathType: DirectoryOrCreate- name: audit-logshostPath: /var/log/kubernetesmountPath: /var/log/kubernetesreadOnly: falsepathType: DirectoryOrCreateextraMounts:- hostPath: ./audit/policy.yamlcontainerPath: /etc/kubernetes/policies/audit-policy.yamlreadOnly: trueEOFkind create cluster --name audit --config audit/kind.yaml --image kindest/node:v1.37.0@sha256:a1ed56cfb0e7b93589bdf97c8cd566405a265939e3620fc4f5de89adff580ae5 -
Generate activity and ask the log questions.
Terminal window kubectl --context kind-audit create secret generic watched --from-literal=k=vkubectl --context kind-audit get secret watched -o yaml > /dev/nullkubectl --context kind-audit create clusterrolebinding suspicious --clusterrole=view --user=malloryLOG="docker exec audit-control-plane cat /var/log/kubernetes/kube-apiserver-audit.log"$LOG | jq -c 'select(.objectRef.resource=="secrets" and .objectRef.name=="watched") | {user: .user.username, verb, code: .responseStatus.code}'$LOG | jq -c 'select(.objectRef.resource=="clusterrolebindings" and .verb=="create") | {user: .user.username, name: .objectRef.name, subjects: .requestObject.subjects}'$LOG | jq -r '.level' | sort | uniq -c
What you should see: The Secret events show who created and read it, with no Secret data in the log (Metadata level). The RBAC event includes the request body, so you can see that mallory was the subject. The level counts show the policy at work. Keep this cluster for the attack investigation demo, then delete it.
Clean up:
kind delete cluster --name audit && rm -rf auditkubectl config use-context kind-kindPractise
- Log all Secret access at Metadata level and find who read a given Secret.
Domain and competency names are from the CNCF CKS curriculum (v1.37), cncf/curriculum, licensed CC BY 4.0. Summaries, demos, and exercises are original to this site.