Auditd & Audit Logging: Know Exactly Who Touched What on Your Server

The File Changed and Nobody Knows Who Did It

The production config was different on Monday than it was on Friday. The SSH authorized_keys file has an entry nobody recognizes. Someone ran sudo and you don’t know what they did with it. Your compliance auditor is asking for six months of privileged access logs and your answer is “we have the syslog from last week?”

Linux has a full-featured audit framework built right into the kernel. It can record every file access, every syscall that requires privilege, every time someone uses sudo, every failed authentication attempt, and more. The tool is auditd and most people have never configured it beyond what their distro installs by default.

That’s a shame, because it’s genuinely powerful and the learning curve isn’t that steep.

What Auditd Actually Captures

The Linux audit framework operates at the kernel level. It intercepts system calls before they execute and records detailed information: who called it, from which process, what arguments were passed, what the result was, and when it happened.

This is categorically different from application-level logging. An application can lie, omit, or fail to log things. The kernel audit framework runs underneath the application — it sees everything regardless of what the application decides to record.

Auditd captures:

File system access: Which files were read, written, or had attributes changed
System calls: Any syscall can be audited, with full argument capture
User and group changes: Account creation, deletion, modification
Authentication events: Logins, sudo usage, PAM events
Network connections: Can be configured to log socket operations
Process execution: What commands were run, by whom
Privilege escalation: setuid/setgid usage

Installing and Starting Auditd

On most systems, auditd is either already installed or available in your package manager:

# Debian/Ubuntu
sudo apt install auditd

# RHEL/CentOS/Rocky
sudo dnf install audit

# Start and enable
sudo systemctl enable --now auditd

# Verify it's running
sudo systemctl status auditd

Logs land in /var/log/audit/audit.log by default. Fair warning: raw audit logs are not human-friendly. The format is intentionally terse and comprehensive:

type=SYSCALL msg=audit(1712000000.123:456): arch=c000003e syscall=2 success=yes exit=3 a0=7f1234 a1=0 a2=1b6 a3=24 items=1 ppid=1234 pid=5678 auid=1000 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts0 ses=1 comm="vim" exe="/usr/bin/vim" key="etc_passwd_watch"

This is why we have ausearch and aureport.

Writing Auditctl Rules

Rules are the heart of auditd configuration. Two primary rule types:

File watches (-w): Monitor a specific file or directory for access.

# Watch /etc/passwd for any access
sudo auditctl -w /etc/passwd -p rwxa -k passwd_changes

# Watch /etc/sudoers for writes and attribute changes
sudo auditctl -w /etc/sudoers -p wa -k sudoers_changes

# Watch SSH authorized_keys directory
sudo auditctl -w /root/.ssh -p rwxa -k ssh_root_keys
sudo auditctl -w /home -p rwxa -k ssh_home_keys

# Watch /etc/shadow (password hashes)
sudo auditctl -w /etc/shadow -p rwxa -k shadow_access

Permissions flags: r (read), w (write), x (execute), a (attribute change).

Syscall rules (-a): Audit specific system calls.

# Audit all privileged command executions
sudo auditctl -a always,exit -F arch=b64 -S execve -F euid=0 -k root_commands

# Audit use of setuid/setgid
sudo auditctl -a always,exit -F arch=b64 -S setuid -S setgid -k privilege_escalation

# Audit deletion of files
sudo auditctl -a always,exit -F arch=b64 -S unlink -S unlinkat -S rename -S renameat -k file_deletion

# Audit failed access attempts
sudo auditctl -a always,exit -F arch=b64 -S open -F exit=-EACCES -k access_denied
sudo auditctl -a always,exit -F arch=b64 -S open -F exit=-EPERM -k access_denied

Check currently loaded rules:

sudo auditctl -l

Making Rules Persistent

Rules set with auditctl are lost on reboot. For persistence, put them in /etc/audit/rules.d/:

# /etc/audit/rules.d/99-custom.rules

# Delete all existing rules first
-D

# Buffer size (increase if you see "kauditd hold queue overflow")
-b 8192

# Failure mode: 0=silent, 1=printk, 2=panic
-f 1

# Watch critical config files
-w /etc/passwd -p wa -k identity_changes
-w /etc/group -p wa -k identity_changes
-w /etc/shadow -p wa -k identity_changes
-w /etc/gshadow -p wa -k identity_changes
-w /etc/sudoers -p wa -k sudoers_changes
-w /etc/sudoers.d/ -p wa -k sudoers_changes

# Watch SSH configuration
-w /etc/ssh/sshd_config -p wa -k ssh_config
-w /root/.ssh -p rwxa -k ssh_root_keys

# Watch authentication config
-w /etc/pam.d/ -p wa -k pam_changes
-w /etc/security/ -p wa -k security_config

# Audit privileged commands
-a always,exit -F arch=b64 -S execve -F euid=0 -F auid!=4294967295 -k root_commands

# Audit network config changes
-a always,exit -F arch=b64 -S sethostname -S setdomainname -k network_changes

# Audit mount operations
-a always,exit -F arch=b64 -S mount -k mount_ops

# Audit file deletions by users
-a always,exit -F arch=b64 -S unlink -S unlinkat -S rename -S renameat -F auid>=1000 -F auid!=4294967295 -k user_file_deletion

# Make config immutable (must reboot to change rules after this)
# -e 2

Load the new rules:

sudo augenrules --load
# or
sudo service auditd restart

Searching and Analyzing Logs

ausearch — Find Specific Events

# Find events related to a specific key (rule tag)
sudo ausearch -k passwd_changes

# Find events for a specific user
sudo ausearch -ua 1000

# Find events for a specific time range
sudo ausearch --start yesterday --end now -k sudoers_changes

# Find failed authentication
sudo ausearch -m USER_AUTH -sv no

# Find events by file path
sudo ausearch -f /etc/passwd

# Interpret results in human-readable format
sudo ausearch -k root_commands -i

The -i flag (interpret) converts UIDs to usernames, times to readable format, and syscall numbers to names. Always use it for human consumption.

aureport — Summary Reports

# Overall summary
sudo aureport

# Authentication report
sudo aureport -au

# Failed events report  
sudo aureport --failed

# Executable report (what commands were run)
sudo aureport -x

# Account modification report
sudo aureport -m

# Summary of specific key
sudo aureport -k --summary

# Time-bounded report
sudo aureport --start 04/01/2026 00:00:00 --end 04/02/2026 23:59:59

The output of aureport with no flags gives a high-level view of audit activity — total events, failed logins, anomalies. Useful for daily review.

A Practical Rule Set for Compliance

If you’re trying to meet something like PCI-DSS, HIPAA, or just want sensible defaults:

# /etc/audit/rules.d/50-compliance.rules

# Capture all sudo usage
-w /usr/bin/sudo -p x -k sudo_usage
-w /usr/bin/su -p x -k su_usage

# Watch cron for persistence mechanisms
-w /etc/cron.d/ -p wa -k cron_changes
-w /etc/crontab -p wa -k cron_changes
-w /var/spool/cron/ -p wa -k cron_changes

# Watch for new setuid binaries
-a always,exit -F arch=b64 -S chmod -S fchmod -S fchmodat -F exit=0 -F perm=6000 -k setuid_created

# Kernel module loading (rootkit indicator)
-w /sbin/insmod -p x -k kernel_modules
-w /sbin/rmmod -p x -k kernel_modules
-w /sbin/modprobe -p x -k kernel_modules
-a always,exit -F arch=b64 -S init_module -S delete_module -k kernel_modules

# Watch for changes to audit configuration itself
-w /etc/audit/ -p wa -k audit_config
-w /sbin/auditctl -p x -k audit_tools
-w /sbin/auditd -p x -k audit_tools

Shipping Logs to a Central Store

Raw audit logs on the server they came from are better than nothing, but not much better. An attacker who compromises a system can delete local logs. Central log shipping is how you get logs that survive the incident.

Option 1: rsyslog Forwarding

# /etc/rsyslog.d/60-audit.conf
module(load="imfile")

input(type="imfile"
      File="/var/log/audit/audit.log"
      Tag="audit"
      Severity="info"
      Facility="local6")

# Forward to central syslog (replace with your server)
local6.* @syslog.internal:514

Option 2: Filebeat to Elasticsearch/Loki

# /etc/filebeat/filebeat.yml
filebeat.inputs:
  - type: log
    enabled: true
    paths:
      - /var/log/audit/audit.log
    fields:
      log_type: audit
      host: ${HOSTNAME}
    fields_under_root: true

# For Elasticsearch
output.elasticsearch:
  hosts: ["https://elasticsearch:9200"]
  index: "audit-logs-%{+yyyy.MM.dd}"

# For Loki (use logstash output or Loki plugin)
# output.logstash:
#   hosts: ["logstash:5044"]

Option 3: audisp-remote Plugin

The audit framework has a native remote logging plugin:

sudo apt install audispd-plugins

# /etc/audit/audisp-remote.conf
remote_server = logserver.internal
port = 60
transport = tcp

# /etc/audit/plugins.d/au-remote.conf
active = yes
direction = out
path = /sbin/audisp-remote
type = always

Auditd and Docker

Docker adds complexity. Container processes appear in the host audit log, but the container context isn’t always obvious.

On the host, audit logs show container activity:

# Find which container a process belongs to
sudo ausearch -p 12345 -i | grep exe

# The container ID appears in the cgroup path
sudo ausearch -k file_deletion | grep docker

For container-specific auditing, consider:

# Watch Docker socket for API calls
-w /var/run/docker.sock -p rwxa -k docker_api

# Watch Docker config
-w /etc/docker/ -p wa -k docker_config

# Watch container runtime
-w /usr/bin/docker -p x -k docker_commands

For container-internal audit logging at scale, look at Falco — it understands container context natively and can apply rules per container image or namespace. It’s the right tool when you have many containers and need per-container visibility.

Tuning to Avoid Log Floods

Naive audit configurations generate enormous log volumes. On a busy system, unfiltered syscall auditing can produce gigabytes per day.

# Exclude high-frequency, low-value events
-a never,exit -F arch=b64 -S getuid -S getgid -S geteuid -S getegid

# Exclude specific programs from auditing
-a never,exit -F exe=/usr/lib/systemd/systemd

# Exclude specific users from file watch auditing
-a never,exit -F auid=998  # Exclude service accounts

Start with file watches on critical files — they’re high-value and low volume. Add syscall auditing carefully, focused on privilege escalation and execution rather than every read/write.

The goal is signal, not noise. An audit system that produces so many logs nobody reads them has failed just as completely as one that records nothing.