Logging Implementation: From Plan to Production

The logging plan escaped the whiteboard and landed in production. Here's what actually happened.

Journal Configuration

Systemd now keeps the journal on a diet:

[Journal]
Storage=persistent
SystemMaxUse=500M
SystemKeepFree=1G
SystemMaxFileSize=50M
MaxRetentionSec=30day
MaxFileSec=1day

The result? 107MB used and no looming disk tantrums.

Caddy Access Logs

/var/log/caddy/
├── base3_access.log
├── nimbus_access.log
└── weblog_access.log

Each site rotates its own logs like a responsible adult.

Deployment Hiccups

Git complained about ownership, so I gave it the proper user and promised to do better next time.

Verification

• Sites respond with security headers and HTTP/3.
• Brotli and gzip handle compression.
• Attack attempts show up in the journal, which is oddly comforting.
• Disk usage hovers at 21% of the limit.

Handy Aliases

logs-deploy   # follow deployment logs
logs-caddy    # follow web logs
logs-errors   # today's errors
logs-size     # journal disk usage

Next Steps

1. Build Nimbus to visualize the data.
2. Add alerting for failures.
3. Track performance metrics.
4. Use history for capacity planning.

"The best logging system is the one you don't notice until you need it."

The foundation is in place and the server no longer threatens to eat its own storage. I'll take it.