Object Storage Disk Full Metadata Recovery

When an S3-compatible store fills a disk, the first question is not what to delete. It is whether object data, metadata versions, drive fault state, and quorum can still be made consistent without wiping the PVC.

Runbook: Do Not Turn Full Disk Into Data Loss

1. Pause writes or admission before the emergency reserve is consumed again.
2. Preserve metadata, manifests, drive-state files, and repair logs before manual deletion.
3. Separate data path fullness from metadata path corruption. A capacity fix is not proof that metadata can rejoin quorum.
4. Reclaim only known-disposable space first: logs, temp files, failed multipart staging, expired trash, or documented cache paths.
5. Verify the four operator-critical operations after headroom returns: list, read, delete, and restart.
6. Document whether an offline repair or heal path exists. If it does not, the incident needs an explicit "wipe required" boundary and backup restore plan.
7. Add a regression test that injects ENOSPC during metadata write, then proves restart, quorum state, and object operations do not require PVC wipe.

I would make the recovery boundary explicit before recommending any wipe or manual deletion.

Acceptance checks I would want:
- Disk-full admission pauses new writes before the recovery reserve is gone.
- Metadata and drive-state files are preserved before any manual cleanup.
- Reclaimable temp/trash/multipart/log paths are documented separately from object data and metadata.
- After headroom is restored, list/read/delete/restart are tested before declaring recovery complete.
- If drives were marked faulty in memory, restart or repair reconciles that state without requiring a PVC wipe.
- If no offline metadata repair tool exists yet, the docs say exactly when restore-from-backup is the only safe path.

Do Not Delete First

• Metadata, manifests, drive-state, quorum, or erasure-set files.
• Object data that may be needed to heal parity or rejoin quorum.
• Repair logs that identify whether metadata was split before or after disk-full.
• PVCs or volumes before proving the backup restore boundary.