Auditd (Linux Audit Daemon) is the userspace component of the Linux Auditing System. It is responsible for collecting, filtering, and writing system event logs to disk based on pre-defined rules. It hooks into the Linux kernel to intercept system calls (syscalls) such as file access, network activity, and process execution. Records crucial metadata including timestamps, the User ID (UID), the Audit ID (AUID) which persists even after privilege escalation (e.g., via sudo), and the success or failure of the event.
On my servers, I usually increase the default value of ‘max_log_file‘ on ‘/etc/audit/auditd.conf‘ from 8 MB to 100 MB and keep the value of ‘num_logs‘ to 5 log files in order to keep enough history for the auditd events. You can increase these values, but keep in mind that the ausearch utility will consume more server resources and take longer times when you query auditd logs using this utility.
# /etc/audit/auditd.conf
...
max_log_file = 100
num_logs = 5
...
root@vps-2153e875:~# du -ks /var/log/audit/*
4528 /var/log/audit/audit.log
102408 /var/log/audit/audit.log.1
102408 /var/log/audit/audit.log.2
102408 /var/log/audit/audit.log.3
102408 /var/log/audit/audit.log.4You can search on the Internet for which auditd rules you should have to comply PCI DSS compliance. In addition to those rules, I have the following rules to monitor the file changes (creation, deletion, modification) on the indicated directories (configurations directory, binaries/libraries directories, temporary directories, logs, and user’s home directories) in order to be able to trace changes made by users, daemons, applications, and investigate any security breach.
# /etc/audit/rules.d/audit.rules
...
-w /etc -p wa -k etcmods
-w /usr -p wa -k usrmods
-w /boot -p wa -k bootmods
-w /tmp -p wa -k tmpmods
-w /var/tmp -p wa -k tmpmods
-w /var/log -p wa -k varlogmods
-w /home -p wa -k homemods
-w /root -p wa -k roothomemods
...I also have the following rules to trace which applications initiated network communications from my server.
-a always,exit -F arch=b64 -S connect -F success=1 -F key=CONNECT
-a always,exit -F arch=b64 -S connect -F exit=-EINPROGRESS -F key=CONNECTAnd finally, the following ones which are specific to Kubernetes environments.
-w /var/lib/kubelet -p wa -k kubeletmods
-w /var/lib/containerd -p wa -k containerdmods
-w /run/containerd -p wa -k runcontainerdmodsThe ‘connect‘ auditd rule we mentioned earlier will generate a large number of events for the ‘kubelet’ daemon. This is because the ‘kubelet’ daemon generates connections every second to the ‘kube-apiserver’ server, as well as other connections every few seconds doing Liveness/Readiness probes on running containers. To avoid auditd logs getting flooded by network connection events generated by kubelet, you can add the following exception rule at the top of the auditd rules to exclude logging events related to the kubelet executable.
-a never,exit -F exe=/usr/bin/kubelet -k ignoredaemonsAs an example, below I show an auditd event of the syscall ‘connect’. Auditd logs provide us with information like which process or command produced the event, the destination IP/Port of the connection, PID and UID of that process, and other information like ‘tty’ which tells us if the command or process is being run or not from an interactive shell.
root@vps-2153e875:~# ausearch -i -a 3109632
----
type=PROCTITLE msg=audit(01/12/26 00:36:32.572:3109632) : proctitle=/usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --confi
type=SOCKADDR msg=audit(01/12/26 00:36:32.572:3109632) : saddr={ saddr_fam=inet laddr=135.125.174.14 lport=6443 }
type=SYSCALL msg=audit(01/12/26 00:36:32.572:3109632) : arch=x86_64 syscall=connect success=no exit=EINPROGRESS(Operation now in progress) a0=0x18 a1=0xc001232bcc a2=0x10 a3=0x0 items=0 ppid=1 pid=1451 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=(none) ses=unset comm=kubelet exe=/usr/bin/kubelet subj=unconfined key=CONNECTGiven we already have an auditd rule monitoring all changes to the ‘/etc’ directory tree (key=etcmods), changes to relevant Kubernetes configuration directories like ‘/etc/kubernetes/manifests/’ (static Pod manifests) and ‘/etc/kubernetes/pki/’ (certificates) will be audited.
In case you have a specific dedicated directory for applications you want to monitor for changes, then you can add it. In my case, I have the directory tree ‘/shared‘ to store WordPress Web and Database files (used as Local Persistent Volumes by WordPress Pods for my site). For example, the WordPress Web Pod (Apache+WordPress) directory ‘/var/www/html‘ maps to host directory ‘/shared/kubernetes/web‘.
-w /shared -p wa -k sharedmodsLet’s see an example of these auditd logs. The following JPG file was generated by WordPress on January 12, 2026, at 18:28h UTC when I generated thumbnails from my WordPress site for a previously uploaded picture.
root@vps-2153e875:~# stat /shared/kubernetes/web/wp-content/uploads/2026/01/adib_profile_pic_reduced-1536x2048.jpg
File: /shared/kubernetes/web/wp-content/uploads/2026/01/adib_profile_pic_reduced-1536x2048.jpg
Size: 584798 Blocks: 1144 IO Block: 4096 regular file
Device: 8,1 Inode: 3153461 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 33/www-data) Gid: ( 33/www-data)
Access: 2026-01-14 00:00:02.019579550 +0000
Modify: 2026-01-12 18:28:46.922447740 +0000
Change: 2026-01-12 18:28:46.922447740 +0000
Birth: 2026-01-12 18:28:46.864446851 +0000Looking for changes to the ‘/shared‘ directory tree from auditd logs, it doesn’t show me any event related to that change, even all the audit rules I previously mentioned were set before that change, and even auditd logs contain events for that day. Why?
root@vps-2153e875:~# ausearch -i -k sharedmods
<no matches>The answer to the previous questions is because Kubernetes Local Persistent Volumes are mounted on the path ‘/var/lib/kubelet/pods/<POD-ID>/volumes/kubernetes.io~local-volume/<VOLUME-NAME>‘, and that is the host path from where the files are modified by Kubernetes Pods.
root@vps-2153e875:~# cat /proc/self/mountinfo | grep "\/shared"
---
966 28 8:1 /shared/kubernetes/db /var/lib/kubelet/pods/7027e0ab-88bd-4bf4-b19d-db8ce60fcf3d/volumes/kubernetes.io~local-volume/pv5gdb rw,relatime shared:1 - ext4 /dev/sda1 rw,discard,errors=remount-ro,commit=30
---
978 28 8:1 /shared/kubernetes/web /var/lib/kubelet/pods/e48914d1-3f2a-43c2-b27c-950bc2b8cab1/volumes/kubernetes.io~local-volume/pv5gweb rw,relatime shared:1 - ext4 /dev/sda1 rw,discard,errors=remount-ro,commit=30
---Searching on auditd logs by the name of the JPG file, it shows me the results, which match the auditd key ‘kubeletmods‘ (directory tree /var/lib/kubelet).
root@vps-2153e875:~# find /var/lib/kubelet/pods/ | grep 'adib_profile_pic_reduced-1536x2048.jpg'
/var/lib/kubelet/pods/e48914d1-3f2a-43c2-b27c-950bc2b8cab1/volumes/kubernetes.io~local-volume/pv5gweb/wp-content/uploads/2026/01/adib_profile_pic_reduced-1536x2048.jpg
root@vps-2153e875:~# ausearch -i -f adib_profile_pic_reduced-1536x2048.jpg
----
type=PROCTITLE msg=audit(01/12/26 18:28:46.864:2085030) : proctitle=apache2 -DFOREGROUND
type=PATH msg=audit(01/12/26 18:28:46.864:2085030) : item=1 name=/var/www/html/wp-content/uploads/2026/01/adib_profile_pic_reduced-1536x2048.jpg inode=3153461 dev=08:01 mode=file,644 ouid=www-data ogid=www-data rdev=00:00 nametype=CREATE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(01/12/26 18:28:46.864:2085030) : item=0 name=/var/www/html/wp-content/uploads/2026/01/ inode=3149630 dev=08:01 mode=dir,755 ouid=www-data ogid=www-data rdev=00:00 nametype=PARENT cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=SYSCALL msg=audit(01/12/26 18:28:46.864:2085030) : arch=x86_64 syscall=openat success=yes exit=13 a0=AT_FDCWD a1=0x7ffe029bb570 a2=O_RDWR|O_CREAT|O_TRUNC a3=0x1b6 items=2 ppid=3881 pid=3473882 auid=unset uid=www-data gid=www-data euid=www-data suid=www-data fsuid=www-data egid=www-data sgid=www-data fsgid=www-data tty=(none) ses=unset comm=apache2 exe=/usr/sbin/apache2 subj=cri-containerd.apparmor.d key=kubeletmods
----
type=PROCTITLE msg=audit(01/12/26 18:28:46.922:2085031) : proctitle=apache2 -DFOREGROUND
type=PATH msg=audit(01/12/26 18:28:46.922:2085031) : item=0 name=/var/www/html/wp-content/uploads/2026/01/adib_profile_pic_reduced-1536x2048.jpg inode=3153461 dev=08:01 mode=file,644 ouid=www-data ogid=www-data rdev=00:00 nametype=NORMAL cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=SYSCALL msg=audit(01/12/26 18:28:46.922:2085031) : arch=x86_64 syscall=chmod success=yes exit=0 a0=0x78748ffc4248 a1=0644 a2=0x7 a3=0x8 items=1 ppid=3881 pid=3473882 auid=unset uid=www-data gid=www-data euid=www-data suid=www-data fsuid=www-data egid=www-data sgid=www-data fsgid=www-data tty=(none) ses=unset comm=apache2 exe=/usr/sbin/apache2 subj=cri-containerd.apparmor.d key=kubeletmodsFrom the previous ‘ausearch‘ output there are two events, the first one related to the file creation (‘syscall=openat‘), and the second one to a file attribute change (syscall=chmod). The ‘subj=cri-containerd.apparmor.d‘ indicates that these changes were done by a process from a Pod (container) context. The ‘uid=www-data‘ (uid=33 in raw format, without the -i human-readable format flag) is the UID of the process inside the Pod. The ‘comm=apache2‘ is the process name inside the Pod. The ‘exe=/usr/sbin/apache2‘ is the full path of the process executable inside the Pod. The ‘pid=3473882‘ is the PID of the process on the Kubernetes node (the process inside the Pod has a different PID). The ‘type=PATH‘ indicates the affected files/directories by the change, and it contains the paths referenced from inside the Pod (/var/www/html/).
From the auditd logs, we can’t know directly the Pod which made the file changes because they don’t natively contain Kubernetes-specific metadata like namespace or Pod name. We will have to do some research for that. For example, given the host ‘pid=3473882‘ we mentioned earlier, using the below-indicated commands, I was able to find the Pod from which the file change was made.
root@vps-2153e875:~# pstree -p | grep -B9 3473882
|-containerd-shim(3776)-+-apache2(3881)-+-apache2(3907)
| | |-apache2(185329)
| | |-apache2(737302)
| | |-apache2(3473871)
| | |-apache2(3473873)
| | |-apache2(3473874)
| | |-apache2(3473879)
| | |-apache2(3473880)
| | |-apache2(3473881)
| | `-apache2(3473882)
root@vps-2153e875:~# lsof -nP -p 3776 | grep containers | head -1
container 3776 root 14u FIFO 0,24 0t0 2207 /run/containerd/io.containerd.grpc.v1.cri/containers/f8acd502d84a30099c06c1ce48e83b345561f2633ce238f7e4ad6c9ee55108fd/io/2434796930/f8acd502d84a30099c06c1ce48e83b345561f2633ce238f7e4ad6c9ee55108fd-stdout
root@vps-2153e875:~# crictl ps | egrep "f8ac|CONTAINER"
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD NAMESPACE
f8acd502d84a3 31ed2f3d73321 11 days ago Running wordpress 1 2087b420958f0 frontend-q8llk ns-adibexpress
root@vps-2153e875:~# kubectl get pods -A -o wide | egrep "frontend|NAMESPACE"
NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
ns-adibexpress frontend-q8llk 1/1 Running 1 (11d ago) 11d 10.244.0.36 vps-2153e875 <none> <none>
root@vps-2153e875:~# kubectl get pod frontend-q8llk -n ns-adibexpress -o jsonpath='{.metadata.uid}'
e48914d1-3f2a-43c2-b27c-950bc2b8cab1Another way is to do the research starting from the ‘inode=3153461‘, which is reported by the auditd logs for the JPG file. For a Local Persistent Volume mount, the inode of a file is the same inside as well outside the Pod. Using the following find command by the inode number on the host, we can find the path ‘/var/lib/kubelet’ where the file is located, and that path also contains the Pod ID.
root@vps-2153e875:~# find / -inum 3153461
---
/shared/kubernetes/web/wp-content/uploads/2026/01/adib_profile_pic_reduced-1536x2048.jpg
---
/var/lib/kubelet/pods/e48914d1-3f2a-43c2-b27c-950bc2b8cab1/volumes/kubernetes.io~local-volume/pv5gweb/wp-content/uploads/2026/01/adib_profile_pic_reduced-1536x2048.jpg
---
root@vps-2153e875:~# kubectl get pods -A -o json | jq -r '.items[] | select(.metadata.uid=="e48914d1-3f2a-43c2-b27c-950bc2b8cab1") | .metadata.name'
frontend-q8llkFor file changes in the Pod filesystem other than on Persistent Volumes, Kubernetes Pods primarily use OverlayFS for their container filesystems. This storage mechanism is handled by the container runtime to manage how data is stored and modified within a Pod. In this context, the interesting directory to monitor with auditd is ‘/var/lib/containerd‘ and the corresponding path ‘/run/containerd‘.
Here is an example of which path on the host is located a file that is created or changed inside a Pod container file system.
root@vps-2153e875:~# kubectl exec -it frontend-q8llk -n ns-adibexpress -- /bin/bash
root@frontend-q8llk:/var/www/html# cd /tmp/
root@frontend-q8llk:/tmp# touch tmpadib.txt
root@vps-2153e875:~# find /var/ | grep tmpadib.txt
/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/351/fs/tmp/tmpadib.txt
root@vps-2153e875:~# mount | grep -w 351
overlay on /run/containerd/io.containerd.runtime.v2.task/k8s.io/f8acd502d84a30099c06c1ce48e83b345561f2633ce238f7e4ad6c9ee55108fd/rootfs type overlay (rw,relatime,lowerdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/308/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/307/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/306/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/305/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/304/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/303/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/302/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/301/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/300/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/299/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/298/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/297/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/296/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/295/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/294/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/293/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/292/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/291/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/290/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/289/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/288/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/287/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/286/fs:/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/285/fs,upperdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/351/fs,workdir=/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/snapshots/351/work,uuid=on,nouserxattr)
root@vps-2153e875:~# crictl ps | egrep "CONTAINER|f8ac"
CONTAINER IMAGE CREATED STATE NAME ATTEMPT POD ID POD NAMESPACE
f8acd502d84a3 31ed2f3d73321 2 weeks ago Running wordpress 1 2087b420958f0 frontend-q8llk ns-adibexpressAnd now let’s check the auditd event that is generated when we manually remove the file mentioned earlier, by getting a shell inside the Pod and using the rm command.
root@vps-2153e875:~# kubectl exec -it frontend-q8llk -n ns-adibexpress -- /bin/bash
root@frontend-q8llk:/var/www/html# rm -f /tmp/tmpadib.txt
root@vps-2153e875:~# ausearch -i -f tmpadib.txt
...
----
type=PROCTITLE msg=audit(01/15/26 05:09:03.950:2568364) : proctitle=rm -f /tmp/tmpadib.txt
type=PATH msg=audit(01/15/26 05:09:03.950:2568364) : item=2 name=(null) inode=811065 dev=08:01 mode=dir,sticky,777 ouid=root ogid=root rdev=00:00 nametype=PARENT cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(01/15/26 05:09:03.950:2568364) : item=1 name=/tmp/tmpadib.txt inode=797772 dev=08:01 mode=file,644 ouid=root ogid=root rdev=00:00 nametype=DELETE cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=PATH msg=audit(01/15/26 05:09:03.950:2568364) : item=0 name=/tmp/ inode=1580708 dev=00:2c mode=dir,sticky,777 ouid=root ogid=root rdev=00:00 nametype=PARENT cap_fp=none cap_fi=none cap_fe=0 cap_fver=0 cap_frootid=0
type=SYSCALL msg=audit(01/15/26 05:09:03.950:2568364) : arch=x86_64 syscall=unlinkat success=yes exit=0 a0=AT_FDCWD a1=0x5e8a1c53c380 a2=0x0 a3=0x100 items=3 ppid=822013 pid=822045 auid=unset uid=root gid=root euid=root suid=root fsuid=root egid=root sgid=root fsgid=root tty=pts0 ses=unset comm=rm exe=/usr/bin/rm subj=cri-containerd.apparmor.d key=runcontainerdmodsThe interesting parts are: The ‘key=runcontainerdmods‘ indicates a change in the host directory tree ‘/run/containerd‘. The ‘subj=cri-containerd.apparmor.d‘ tells us that the change was from a container context. The ‘syscall=unlinkat‘ says that it’s a file deletion, and ‘comm=rm‘ tells the command used inside the Pod to remove the file ‘name=/tmp/tmpadib.txt‘ using the container ‘uid=root‘ user. The ‘pid=822045‘ is the PID of the rm command, and ‘ppid=822013‘ is the PID of the shell. Both PIDs are in the host level context. If the shell from where the rm command was executed is already closed, then having only this auditd event, is not possible to determine in which Pod or container was taken this action.
Regarding the previous limitations, we can consider installing and using Falco. In Kubernetes, Falco is the leading open-source runtime security tool used to detect and alert on anomalous behavior in real-time. It captures the same syscalls as auditd but automatically maps them to Kubernetes metadata (Pod Name, Namespace, Image) in real-time. Falco enriches system calls with metadata from container runtime and the Kubernetes API. This allows rules like “alert if a shell is opened in a Pod with the label app=wordpress”.
Adib Ahmed Akhtar