Logrotate & Compression

Your /var/log directory fills up faster than you’d expect. Nginx access logs, Docker container logs, application errors — they all pile up and before you know it, you’re out of disk space. Here’s the thing: you don’t need to keep raw, uncompressed logs forever. That’s where logrotate comes in.

logrotate is a system utility that runs daily (via cron) and automatically archives old logs, compresses them, and deletes them after a set retention period. Pair it with a compression algorithm like xz (which uses LZMA2), and you can store way more logs in less space while diagnosing issues and maintaining audit trails.

How logrotate Works

logrotate runs once daily via a cron job (typically /etc/cron.daily/logrotate). It reads its configuration from /etc/logrotate.conf and from any files in /etc/logrotate.d/. The daemon walks through each log file, checks if rotation conditions are met (size, age, or time-based), and performs the rotation if needed.

The config hierarchy works like this:

1. Global defaults in /etc/logrotate.conf apply to all logs
2. Application-specific overrides in /etc/logrotate.d/<app> override the globals for that service

This lets you set a sensible default (rotate weekly, keep 4 copies) while giving Nginx stricter rules (rotate daily, keep 14 copies) or your app looser ones (rotate only when > 1GB).

Installing logrotate and Compression

First, install logrotate and a compression utility. Most systems have logrotate by default, but you may need to install the compressor:

# Debian/Ubuntu
sudo apt-get install logrotate xz-utils

# CentOS/RHEL
sudo yum install -y logrotate xz-utils

# Fedora
sudo dnf install logrotate xz-utils

xz is a lossless compressor using LZMA2 algorithms. It offers better compression than gzip (often 30–50% smaller) at the cost of slightly more CPU during compression. Perfect for logs you won’t access often.

Anatomy of a logrotate Config Block

Here’s what each directive does:

rotate 14                      # Keep 14 rotated log files
compress                       # Compress rotated logs
delaycompress                  # Don't compress the most-recently rotated log
missingok                      # Don't error if a log file doesn't exist
notifempty                     # Don't rotate empty log files
postrotate                     # Run a command after rotation (e.g., reload service)
  systemctl reload nginx
endscript

• rotate N — Keep N old log files before deletion (older ones are discarded)
• compress — Enable compression
• delaycompress — Compress on the next rotation, not immediately (safer for apps still writing to the log)
• missingok — Don’t fail if the log file is missing (useful for optional logs)
• notifempty — Skip rotation if the log is empty (saves compression cycles)
• postrotate — Run commands after rotation (reload Nginx, restart Docker, etc.)

Setting Up XZ Compression

Override the default gzip compression in your global config or app-specific config:

compress
compresscmd /usr/bin/xz
uncompresscmd /usr/bin/unxz
compressext .xz
compressoptions -9e

• compresscmd — Path to compression binary
• uncompresscmd — Path to decompression binary (for manual inspection)
• compressext — File extension for compressed logs (.xz instead of .gz)
• compressoptions — Compression level (-9 for max compression, -1 for fastest). Use -9e for ultra-high compression if you have CPU headroom

Lower compression levels (like -6 or -7) are faster but produce slightly larger files. -9e is overkill for most logs; -6 is usually the sweet spot.

Example: Nginx Logs

Here’s a real-world logrotate config for Nginx:

/var/log/nginx/*.log {
  daily
  rotate 14
  compress
  delaycompress
  compresscmd /usr/bin/xz
  compressext .xz
  compressoptions -6
  missingok
  notifempty
  postrotate
    systemctl reload nginx > /dev/null 2>&1 || true
  endscript
}

This rotates Nginx access/error logs daily, keeps 14 copies, compresses with xz at level 6, and reloads Nginx after rotation (so it switches to the new log file).

Testing Rotation

Don’t wait for the cron job. Test manually with:

# Dry run (show what would happen, don't do it)
sudo logrotate -d /etc/logrotate.conf

# Force rotation of a specific config
sudo logrotate -f /etc/logrotate.d/nginx

# Verbose output
sudo logrotate -v /etc/logrotate.conf

After running -f, check the log directory:

ls -lh /var/log/nginx/
# You'll see: access.log, access.log.1.xz, access.log.2.xz, etc.

Custom App Logging

Got your own application that writes logs? Create a config file:

/var/log/myapp/app.log {
  size 50M              # Rotate when > 50MB
  rotate 7
  compress
  missingok
  notifempty
  postrotate
    systemctl restart myapp > /dev/null 2>&1 || true
  endscript
}

The size 50M directive rotates based on file size, not time. Useful for high-volume apps.

Troubleshooting Logs That Won’t Rotate

If your logs aren’t rotating:

1. Check cron is running: sudo systemctl status cron (or crond on CentOS)
2. Run manually: sudo logrotate -f /etc/logrotate.conf
3. Check permissions: logrotate needs read/write access to the log directory
4. Check syntax: sudo logrotate -d /etc/logrotate.conf will highlight config errors
5. Check the log: cat /var/lib/logrotate/status shows when each file was last rotated

Done. Your logs stay manageable, compressed, and ready for audit trails when you actually need them. Your 2 AM self will appreciate the extra disk space.