Backup Retention Policy Disk Full

Backup tools feel safe until every save creates another timestamped copy, every job leaves a directory behind, and the host silently runs out of disk. The fix is not just "delete old backups"; it is a retention policy that runs before the next write can fail.

Runbook: Retain Before The Disk Is Full

1. Preserve one recent known-good backup and one failed backup sample before deleting bulk history.
2. Measure the backup root by source path, not only total directory size. One large file saved 100 times can dominate everything.
3. Add a prune-before-write step. If pruning only runs after a successful backup, the next write can fail before cleanup happens.
4. Set both count and age limits: for example, keep the newest 5 per source and delete anything older than 30 days.
5. Cap full-log reads and per-job logs. A backup job should not create a second unbounded log-growth problem.
6. Fail with a useful message when reserve space is below threshold: backup root, free bytes, largest retained bucket, and recommended cleanup command.

I would make retention run before new writes, not only after successful backups.

Read-only evidence first:

BACKUP_ROOT=<backup-root>
df -h "$BACKUP_ROOT" .
du -h -d 2 "$BACKUP_ROOT" 2>/dev/null | sort -h | tail -40
find "$BACKUP_ROOT" -type f -size +100M -printf "%s %TY-%Tm-%Td %p\n" 2>/dev/null | sort -n | tail -40

Then split the policy:
- keep newest N per source file or job
- keep max age for old timestamped copies
- reserve free disk before starting a backup
- cap per-job logs and full-log reads

Do Not Delete First

• The newest known-good backup for each source or job.
• Metadata that proves which source generated the largest history bucket.
• Failed partial backups before recording size, timestamp, and error message.
• Only restore point for a database, project, or customer file before a separate backup exists.