Log Monitor Loop Fills Disk

A monitor that should protect the host can become the outage when it busy-polls log files, repeats the same error thousands of times per second, and fills syslog or journald before anyone sees the original permission or file-descriptor problem.

Runbook: Stop The Flood Without Hiding The Root Error

1. Preserve the first few error lines. The repeated line that fills disk may be a symptom of an earlier permission, fd limit, or missing file state.
2. Put a temporary blast-radius cap in place: journald or rsyslog max use, service CPU quota, and rate limiting. This is containment, not the final fix.
3. Fix the loop: use inotify or a sleep-backed poll interval, keep a file offset, and avoid reopening/re-reading from byte zero on every iteration.
4. Add error backoff: 1s, 2s, 4s up to a cap, plus "repeated N times" aggregation instead of one log line per failed loop.
5. Bound resource spikes: file descriptor limit, concurrent lookups, and a disable-after-K-failures path for unreadable monitored files.
6. Turn the incident into a policy: max log bytes/minute, max repeated errors/minute, alert when /var reserve is breached, and documented service permissions.

I would split this into containment and the loop fix.

Read-only evidence before cleanup:

SERVICE=<service-name>
df -h / /var /var/log 2>/dev/null
df -i / /var /var/log 2>/dev/null
du -sh /var/log /var/log/journal 2>/dev/null
journalctl -u "$SERVICE" -n 300 --no-pager 2>/dev/null || true
systemctl show "$SERVICE" -p MainPID -p Restart -p RestartUSec -p LimitNOFILE 2>/dev/null || true

For the fix, I would require three guards: no tight poll loop (inotify or sleep + last-byte offset), rate-limited recurring errors with exponential backoff, and a bounded log/journal budget so a permission error cannot become a disk-full outage.

Do Not Delete First

• The first logs that show the original permission, fd limit, or missing file error.
• All logs before summarizing repeated signatures, timestamps, and growth rate.
• Systemd unit files or security settings before recording current capabilities, groups, and limits.
• Application databases or service state just because /var is full; contain the log flood first.