Kubernetes threat detection with Falco Sidekick

Let me quickly walk you through how to set up a basic Kubernetes threat detection homelab on Kali Purple. I'll start from a freshly installed of Kali Purple VM - so-called SOC-in-a-box, then install minikube - to easily manage local k8s cluster, and finally install containerized Falco with it's Sidekick user interface - for k8s threat detection. That will be the base of the lab. To test, I'll use one of Kubernetes Goat scenarios and Atomic Red Team tests. After that, you are ready for more advanced experimentations.

Intro work - install the latest Kali Purple in a hypervisor and enabled NAT for machine's internet connection. I used VirtualBox with guest additions, and for the VM - 60GB dynamic storage, 16GB memory and (at least) 2vCPU.
For convenient copy/pasting to and from host ensure that shared clipboard is bidirectional. In case you bump on issues there, check your host's settings for Secure Boot and Core Integrity, and additional instructions from Kali for the hypervisor.

Let's proceed with installing Docker:

sudo apt update -y && sudo apt upgrade -y

sudo apt install docker.io -y

sudo usermod -aG docker $USER && newgrp docker

Next, let's switch to Documents directory, install minikube environment and dedicate (at least) 6GB memory to it:

cd Documents

curl -LO https://storage.googleapis.com/kubernetes-release/release/`curl -s https://storage.googleapis.com/kubernetes-release/release/stable.txt`/bin/linux/amd64/kubectl

chmod +x ./kubectl

sudo mv ./kubectl /usr/local/bin/kubectl

curl -LO https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64

sudo install minikube-linux-amd64 /usr/local/bin/minikube

minikube start --memory=6G

kubectl get pods -A

If all pods are in the running state, proceed with installing Helm package manager for k8s:

curl -fsSL -o get_helm.sh https://raw.githubusercontent.com/helm/helm/main/scripts/get-helm-3

chmod 700 get_helm.sh

./get_helm.sh

If there are no errors, let's proceed to install falco, falcosidekick and falcosidekick-ui. Basically, falco detects k8s threat events, falcosidekick can take these events and forward them to different outputs, and falcosidekick-ui works as a web UI output for falcosidekick to display the latest events from falco.
Falco blog shows that [currently] the latest version of falco is 0.38.1, and you can check with [currently] the latest falco helm chart content.
Reading the instructions, you'll see several driver kinds mentioned, but for Kali Purple I had the least amount of errors and CrashLoopBackOffs using modern-bpf, so I'll proceed with that here:

apt search linux-headers-$(uname -r)     #check if installed, should show [installed]

sudo apt-get -y install linux-headers-$(uname -r)

apt search linux-headers-$(uname -r)

helm repo add falcosecurity https://falcosecurity.github.io/charts

helm repo update

kubectl create ns falco

helm install falco \
    --namespace falco \
    --set tty=true \
    --set falcosidekick.enabled=true \
    --set falcosidekick.webui.enabled=true \
    --set driver.kind=modern-bpf \
falcosecurity/falco

Now, this will take some time to get all pods and other workloads in falco namespace to a running state, expect up to 5 min. You can run to check:

kubectl get all -n falco  #or, to continuously watch pods state change until they finally come up and running: kubectl get pods -n falco -w, then Ctrl+C

To mention, when all pods in falco namespace get to a running state, you can continuously monitor for detected k8s events from a terminal, in a new terminal tab:

kubectl get pods -n falco --show-labels     #we expect to see label such as app.kubernetes.io/name=falco here

kubectl logs -f -l app.kubernetes.io/name=falco -n falco

And when you are done with continuous monitoring from the terminal, press Ctrl+C

With all falco relevant workloads in a running state, we can proceed with accessing Sidekick web user interface with default credentials admin/admin:

kubectl port-forward svc/falco-falcosidekick-ui \
  -n falco 2802:2802 &> /dev/null &

In your browser, go to http://127.0.0.1:2802 and login. There will be a pretty much empty Dashboard, but we'll fill it in with the help of Kubernetes Goat - intentionally vulnerable by design project to learn k8s security by Madhu Akula. Let's install the project as instructed:

git clone https://github.com/madhuakula/kubernetes-goat

cd kubernetes-goat

chmod +x setup-kubernetes-goat.sh

./setup-kubernetes-goat.sh

kubectl get pods -n default -w

Ensure these pods are running before running the access script:

chmod +x access-kubernetes-goat.sh

./access-kubernetes-goat.sh

Then navigate to http://127.0.0.1:1234 in your browser to verify Kubernetes Goat scenarios are live.

We can use adjusted Falco scenario, and check falco logs from a new terminal tab and if the Sidekick Dashboard is populating:

kubectl run hacker-container --rm -it --restart=Never --image=madhuakula/hacker-container   #[Tab 1]

kubectl logs -f -l app.kubernetes.io/name=falco -n falco | grep "Sensitive file opened"   #[Tab 2]

cat /etc/shadow     #[Tab 1]

You can use Falco Rules Explorer work-in-progress dashboard by Thomas Labarussias to quickly browse through publicly available rules, and get ideas for new rules or adjustments in your environment.

Exit the malicious pod now, and stop with terminal monitoring.

There's one last main thing for today left, testing with Atomic Red Team libraries. Essentially, you can follow along this post on Falco blog:

cd ..     #going back to Documents directory

kubectl create ns atomic-red

Now, create a file called atomicreddeploy.yaml with the following content, then apply:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: atomicred
  namespace: atomic-red
  labels:
    app: atomicred
spec:
  replicas: 1
  selector:
    matchLabels:
      app: atomicred
  template:
    metadata:
      labels:
        app: atomicred
    spec:
      containers:
      - name: atomicred
        image: issif/atomic-red:latest
        imagePullPolicy: "IfNotPresent"
        command: ["sleep", "3560d"]
        securityContext:
          privileged: true
      nodeSelector:
        kubernetes.io/os: linux

kubectl apply -f atomicreddeploy.yaml

kubectl get pods -n atomic-red -w    #watch until the pod comes up, then Ctrl+C

kubectl exec -it -n atomic-red deploy/atomicred -- bash

pwsh

Import-Module "~/AtomicRedTeam/invoke-atomicredteam/Invoke-AtomicRedTeam.psd1" -Force

Invoke-AtomicTest T1070.004 -ShowDetails

Invoke-AtomicTest T1070.004 -GetPreReqs

Invoke-AtomicTest T1070.004

Invoke-AtomicTest T1556.003

Let's see what tests are available, and kickstart a couple:

ls -l /root/AtomicRedTeam/atomics/

Invoke-AtomicTest T1036.005

Invoke-AtomicTest T1070.002

Invoke-AtomicTest T1070.003

Invoke-AtomicTest T1014

When you are finished - exit the powershell, exit the atomicred pod. Falcosidekick-ui Dashboard and Events tab should now be populated:

Here you can see some details of my Kali Purple VM OS, falco version, and running pods at the end of this homelab:

That's all! You can check out Falco training, especially the free one courtesy of Sysdig that goes deeper into Falco configuration and rules - Falco 101, or continue digging on your own.

Kubernetes threat detection with Falco Sidekick

Let's proceed with installing Docker:

Next, let's switch to Documents directory, install minikube environment and dedicate (at least) 6GB memory to it:

If all pods are in the running state, proceed with installing Helm package manager for k8s:

Now, this will take some time to get all pods and other workloads in falco namespace to a running state, expect up to 5 min. You can run to check:

To mention, when all pods in falco namespace get to a running state, you can continuously monitor for detected k8s events from a terminal, in a new terminal tab:

And when you are done with continuous monitoring from the terminal, press Ctrl+C

With all falco relevant workloads in a running state, we can proceed with accessing Sidekick web user interface with default credentials admin/admin:

In your browser, go to http://127.0.0.1:2802 and login. There will be a pretty much empty Dashboard, but we'll fill it in with the help of Kubernetes Goat - intentionally vulnerable by design project to learn k8s security by Madhu Akula. Let's install the project as instructed:

Ensure these pods are running before running the access script:

Then navigate to http://127.0.0.1:1234 in your browser to verify Kubernetes Goat scenarios are live.

We can use adjusted Falco scenario, and check falco logs from a new terminal tab and if the Sidekick Dashboard is populating:

You can use Falco Rules Explorer work-in-progress dashboard by Thomas Labarussias to quickly browse through publicly available rules, and get ideas for new rules or adjustments in your environment.

Exit the malicious pod now, and stop with terminal monitoring.

There's one last main thing for today left, testing with Atomic Red Team libraries. Essentially, you can follow along this post on Falco blog:

Now, create a file called atomicreddeploy.yaml with the following content, then apply:

Let's see what tests are available, and kickstart a couple:

When you are finished - exit the powershell, exit the atomicred pod. Falcosidekick-ui Dashboard and Events tab should now be populated:

Here you can see some details of my Kali Purple VM OS, falco version, and running pods at the end of this homelab:

That's all! You can check out Falco training, especially the free one courtesy of Sysdig that goes deeper into Falco configuration and rules - Falco 101, or continue digging on your own.

Note, throughout the post I mentioned a couple of open source projects, don't forget that you can get involved and contribute! :)

Hope you found the post useful, thanks for reading!

Update: Here you can check out my ~30 min talk "Runtime Kubernetes Security: Hands-On Threat Detection with Falcosidekick" @ Conf42 Kube Native 2024.