Skip to content

CKS: Monitoring, Logging and Runtime Security

20% of the CKS exam, 5 competencies. Back to the CKS overview.

Each competency lists what to know and the official documentation to read, then a demo you can run. Demos run on the demo lab, a throwaway kind cluster on your own machine; set it up once before you start.

Perform behavioral analytics to detect malicious activities

Section titled “Perform behavioral analytics to detect malicious activities”

Know

  • Runtime tools such as Falco watch syscalls and alert on behaviour like a shell spawning in a container.

Read

Demo: Install Falco and catch a shell in a container

  1. Create a namespace for this demo and make it the default for your context.

    Terminal window
    kubectl create namespace cks-falco
    kubectl config set-context --current --namespace=cks-falco
  2. Install Falco from its chart (9.2.2, Falco 0.45.2) on the lab cluster.

    Terminal window
    helm repo add falcosecurity https://falcosecurity.github.io/charts
    helm install falco falcosecurity/falco --version 9.2.2 -n falco --create-namespace --set tty=true
    kubectl -n falco rollout status daemonset/falco --timeout=300s
  3. Do what an attacker would: open an interactive shell and read a sensitive file.

    Terminal window
    kubectl run victim --image=registry.k8s.io/e2e-test-images/busybox:1.37.0-2 -- sleep 3600
    kubectl wait --for=condition=Ready pod/victim --timeout=60s
    kubectl exec -it victim -- sh -c 'cat /etc/shadow > /dev/null; exit'
  4. Read Falco’s alerts.

    Terminal window
    kubectl -n falco logs -l app.kubernetes.io/name=falco -c falco --tail=50 | grep -E 'Terminal shell in container|Read sensitive file'

What you should see: Falco reports the Terminal shell in container rule and the Read sensitive file untrusted rule, each with the pod, namespace, command, and user. It watches system calls through eBPF, so it sees what happens inside containers regardless of what the application logs.

Note: Falco needs a Linux kernel with eBPF support on the host. Leave it installed for the next demo, or remove it with helm uninstall falco -n falco.

Clean up:

Terminal window
kubectl config set-context --current --namespace=default
kubectl delete namespace cks-falco

Practise

  • Trigger a Falco rule by running a shell inside a container and read the alert.

Detect threats within physical infrastructure, apps, networks, data, users and workloads

Section titled “Detect threats within physical infrastructure, apps, networks, data, users and workloads”

Know

  • Combine audit logs, runtime alerts, and network flow logs to see an attack across layers.

Read

Demo: Write your own Falco rule

  1. Create a namespace for this demo and make it the default for your context.

    Terminal window
    kubectl create namespace cks-falco-rule
    kubectl config set-context --current --namespace=cks-falco-rule
  2. A rule that fires when a container reads its service account token.

    Terminal window
    cat > falco-custom.yaml <<'EOF'
    customRules:
    rules-demo.yaml: |-
    - rule: Service account token read in container
    desc: A process inside a container opened its Kubernetes service account token
    condition: open_read and container and fd.name startswith /var/run/secrets/kubernetes.io/serviceaccount/
    output: "SA token read (proc=%proc.name file=%fd.name pod=%k8s.pod.name ns=%k8s.ns.name)"
    priority: WARNING
    EOF
    helm upgrade falco falcosecurity/falco --version 9.2.2 -n falco --reuse-values -f falco-custom.yaml
    kubectl -n falco rollout status daemonset/falco --timeout=300s
  3. Trigger it and read the alert.

    Terminal window
    kubectl run reader --image=registry.k8s.io/e2e-test-images/busybox:1.37.0-2 -- sleep 3600
    kubectl wait --for=condition=Ready pod/reader --timeout=60s
    kubectl exec reader -- cat /var/run/secrets/kubernetes.io/serviceaccount/token > /dev/null
    sleep 5; kubectl -n falco logs -l app.kubernetes.io/name=falco -c falco --tail=50 | grep 'SA token read'

What you should see: Your rule fires with the process, file, pod, and namespace. Real applications that talk to the API read this token legitimately, so a rule like this needs exceptions for known workloads before it is useful; tuning out noise is most of the work in runtime detection.

Clean up:

Terminal window
rm -f falco-custom.yaml
helm uninstall falco -n falco && kubectl delete namespace falco
kubectl config set-context --current --namespace=default
kubectl delete namespace cks-falco-rule

Investigate and identify phases of attack and bad actors within the environment

Section titled “Investigate and identify phases of attack and bad actors within the environment”

Know

  • Map what you find to attack phases (initial access, execution, persistence, privilege escalation) to decide what to contain first.

Read

Demo: Reconstruct an attack from the audit log

  1. Run this on the audit cluster from the “Use Kubernetes audit logs to monitor access” demo below. Play the attacker: a stolen token is used to look around, gain access, and persist.

    Terminal window
    kubectl config use-context kind-audit
    kubectl create namespace victim
    kubectl create serviceaccount intruder -n victim
    kubectl create rolebinding intruder-view -n victim --clusterrole=view --serviceaccount=victim:intruder
    kubectl config view --minify --raw -o jsonpath='{.clusters[0].cluster.certificate-authority-data}' | base64 -d > audit-ca.crt
    SERVER=$(kubectl config view --minify -o jsonpath='{.clusters[0].cluster.server}')
    kubectl config set-cluster audit --server="$SERVER" --certificate-authority=audit-ca.crt --embed-certs=true --kubeconfig=intruder.kubeconfig
    kubectl config set-credentials intruder --token="$(kubectl create token intruder -n victim)" --kubeconfig=intruder.kubeconfig
    kubectl config set-context intruder --cluster=audit --user=intruder --kubeconfig=intruder.kubeconfig
    K="kubectl --kubeconfig=intruder.kubeconfig --context=intruder"
    $K get pods -n victim
    $K get secrets -n kube-system
    $K create clusterrolebinding pwned --clusterrole=cluster-admin --serviceaccount=victim:intruder
  2. Now investigate: everything that identity did, in order, with the result.

    Terminal window
    docker exec audit-control-plane cat /var/log/kubernetes/kube-apiserver-audit.log \
    | jq -r 'select(.user.username=="system:serviceaccount:victim:intruder") | [.requestReceivedTimestamp, .verb, (.objectRef.resource // .requestURI), (.objectRef.namespace // "-"), (.responseStatus.code|tostring), .sourceIPs[0]] | @tsv'

What you should see: The timeline shows discovery (listing pods), an attempt on Secrets in kube-system that got 403, and an attempt to bind cluster-admin that got 403. Failed attempts are often the earliest sign; the source IP tells you where the token is being used from. The attacker’s kubeconfig holds only the token, so nothing else authenticates the requests.

Clean up:

Terminal window
kubectl delete namespace victim && rm -f audit-ca.crt intruder.kubeconfig
kubectl config use-context kind-kind

Ensure immutability of containers at runtime

Section titled “Ensure immutability of containers at runtime”

Know

  • readOnlyRootFilesystem: true, no package managers in images, and writable paths only through explicit volumes.

Read

Demo: Containers that can’t be changed, and a policy that requires it

  1. Create a namespace for this demo and make it the default for your context.

    Terminal window
    kubectl create namespace cks-immutable
    kubectl config set-context --current --namespace=cks-immutable
  2. Require readOnlyRootFilesystem for every container in a namespace.

    Terminal window
    kubectl label namespace cks-immutable immutability=required
    kubectl apply -f - <<'EOF'
    apiVersion: admissionregistration.k8s.io/v1
    kind: ValidatingAdmissionPolicy
    metadata:
    name: read-only-root
    spec:
    failurePolicy: Fail
    matchConstraints:
    resourceRules:
    - apiGroups: [""]
    apiVersions: ["v1"]
    operations: ["CREATE", "UPDATE"]
    resources: ["pods"]
    validations:
    - expression: "object.spec.containers.all(c, has(c.securityContext) && has(c.securityContext.readOnlyRootFilesystem) && c.securityContext.readOnlyRootFilesystem == true)"
    message: "every container needs securityContext.readOnlyRootFilesystem: true"
    ---
    apiVersion: admissionregistration.k8s.io/v1
    kind: ValidatingAdmissionPolicyBinding
    metadata:
    name: read-only-root
    spec:
    policyName: read-only-root
    validationActions: ["Deny"]
    matchResources:
    namespaceSelector:
    matchLabels:
    immutability: required
    EOF
    sleep 3
    kubectl run mutable --image=registry.k8s.io/e2e-test-images/busybox:1.37.0-2 -- sleep 3600
    kubectl run immutable --image=registry.k8s.io/e2e-test-images/busybox:1.37.0-2 --overrides='{"spec":{"containers":[{"name":"immutable","image":"registry.k8s.io/e2e-test-images/busybox:1.37.0-2","command":["sleep","3600"],"securityContext":{"readOnlyRootFilesystem":true}}]}}'
  3. Try to change the running container.

    Terminal window
    kubectl wait --for=condition=Ready pod/immutable --timeout=60s
    kubectl exec immutable -- sh -c 'echo evil > /bin/evil' || true
    kubectl exec immutable -- sh -c 'echo "* * * * * evil" > /etc/crontabs/root' || true

What you should see: mutable is rejected by the policy and immutable is admitted. Inside it, every write fails with Read-only file system. Combine this with no shell in the image, a pinned digest, and a runtime detector like Falco, and a change to a running container becomes both hard and visible.

Clean up:

Terminal window
kubectl delete validatingadmissionpolicybinding read-only-root
kubectl delete validatingadmissionpolicy read-only-root
kubectl config set-context --current --namespace=default
kubectl delete namespace cks-immutable

Practise

  • Make a pod read-only except for /tmp through an emptyDir.

Use Kubernetes audit logs to monitor access

Section titled “Use Kubernetes audit logs to monitor access”

Know

  • Write an audit policy, enable it on the API server with --audit-policy-file and a log backend, and search the log for Secret reads.

Read

Demo: An audit policy and the questions it answers

  1. Write a policy: no noise from health checks, metadata only for Secrets, full bodies for RBAC changes.

    Terminal window
    mkdir -p audit && cat > audit/policy.yaml <<'EOF'
    apiVersion: audit.k8s.io/v1
    kind: Policy
    omitStages: ["RequestReceived"]
    rules:
    - level: None
    nonResourceURLs: ["/healthz*", "/livez*", "/readyz*", "/version"]
    - level: None
    users: ["system:kube-proxy"]
    verbs: ["watch"]
    - level: Metadata
    resources:
    - group: ""
    resources: ["secrets", "configmaps"]
    - level: RequestResponse
    resources:
    - group: rbac.authorization.k8s.io
    resources: ["roles", "rolebindings", "clusterroles", "clusterrolebindings"]
    - level: Request
    resources:
    - group: ""
    resources: ["pods/exec", "pods/attach", "pods/portforward"]
    - level: Metadata
    EOF
  2. Create a cluster that uses it (this follows the kind auditing guide).

    Terminal window
    cat > audit/kind.yaml <<'EOF'
    kind: Cluster
    apiVersion: kind.x-k8s.io/v1alpha4
    nodes:
    - role: control-plane
    kubeadmConfigPatches:
    - |
    kind: ClusterConfiguration
    apiServer:
    extraArgs:
    audit-log-path: /var/log/kubernetes/kube-apiserver-audit.log
    audit-policy-file: /etc/kubernetes/policies/audit-policy.yaml
    extraVolumes:
    - name: audit-policies
    hostPath: /etc/kubernetes/policies
    mountPath: /etc/kubernetes/policies
    readOnly: true
    pathType: DirectoryOrCreate
    - name: audit-logs
    hostPath: /var/log/kubernetes
    mountPath: /var/log/kubernetes
    readOnly: false
    pathType: DirectoryOrCreate
    extraMounts:
    - hostPath: ./audit/policy.yaml
    containerPath: /etc/kubernetes/policies/audit-policy.yaml
    readOnly: true
    EOF
    kind create cluster --name audit --config audit/kind.yaml --image kindest/node:v1.37.0@sha256:a1ed56cfb0e7b93589bdf97c8cd566405a265939e3620fc4f5de89adff580ae5
  3. Generate activity and ask the log questions.

    Terminal window
    kubectl --context kind-audit create secret generic watched --from-literal=k=v
    kubectl --context kind-audit get secret watched -o yaml > /dev/null
    kubectl --context kind-audit create clusterrolebinding suspicious --clusterrole=view --user=mallory
    LOG="docker exec audit-control-plane cat /var/log/kubernetes/kube-apiserver-audit.log"
    $LOG | jq -c 'select(.objectRef.resource=="secrets" and .objectRef.name=="watched") | {user: .user.username, verb, code: .responseStatus.code}'
    $LOG | jq -c 'select(.objectRef.resource=="clusterrolebindings" and .verb=="create") | {user: .user.username, name: .objectRef.name, subjects: .requestObject.subjects}'
    $LOG | jq -r '.level' | sort | uniq -c

What you should see: The Secret events show who created and read it, with no Secret data in the log (Metadata level). The RBAC event includes the request body, so you can see that mallory was the subject. The level counts show the policy at work. Keep this cluster for the attack investigation demo, then delete it.

Clean up:

Terminal window
kind delete cluster --name audit && rm -rf audit
kubectl config use-context kind-kind

Practise

  • Log all Secret access at Metadata level and find who read a given Secret.

Domain and competency names are from the CNCF CKS curriculum (v1.37), cncf/curriculum, licensed CC BY 4.0. Summaries, demos, and exercises are original to this site.