Logging Strategy: SystemD Journal Management

Why Logging First?

Debugging blind is only fun if you enjoy guesswork. Logging infrastructure came before dashboards, polish, or tests because nothing else matters if the server silently fills the disk.

SystemD Journal as Backbone

• Centralized and structured.
• Auto-rotates and persists across reboots.
• Enough features to make logrotate jealous.

Configuration

[Journal]
Storage=persistent
SystemMaxUse=500M
SystemKeepFree=1G
SystemMaxFileSize=50M
MaxRetentionSec=30day
MaxFileSec=1day

Translation: keep the logs, but not all of them, and leave some room for anything else.

Caddy Access Logs

log {
  output file /var/log/caddy/{site}_access.log {
    roll_size 10MB
    roll_keep 5
    roll_keep_for 720h
  }
  format single_field common_log
}

Plain files with rotation. Nothing fancy, nothing broken.

Deployment Logging

journalctl -f -u deploy-*           # follow all deploy logs
journalctl -u deploy-base3 --since "1 hour ago"
journalctl -u deploy-* -p err --since today

Why This Approach

• Self-managing—no cron jobs quietly failing.
• Powerful queries for later analysis.
• Ready for Nimbus when the dashboard exists.

Next Steps

1. Implement the config above.
2. Add Caddy structured logs.
3. Improve deployment log messages.
4. Visualize everything in Nimbus.

"The best logging system is the one that manages itself and never fills your disk."