Full Disk Trash GC Deadlock Recovery

Some stateful services enter a nasty loop when the disk is full: the trash or async deleter is the component that should free space, but it crashes during startup because the filesystem cannot create, list, or write its own recovery directory. Treat that as a storage deadlock, not a normal cache cleanup.

Runbook: Break The Self-Reinforcing Loop

1. Preserve the first error. A later "directory missing" or "file not found" may only be a consequence of an earlier ENOSPC during mkdir, write, rename, or fsync.
2. Measure both bytes and inodes on the mounted data root. A full inode table can make cleanup code fail even when bytes look available.
3. Find the trash, deleted, pending-delete, GC, WAL, snapshot, and temp path families separately. Do not mix reclaimable trash with active tenant or database state.
4. If the cleanup worker is supervised, make boot tolerant: list with an error return, report the true reason, and keep retrying instead of taking the whole stack down.
5. Add a low-space admission check before enqueueing new deletes or snapshots. The system should stop creating more trash before the trash worker loses the ability to run.
6. Turn the incident into a guard: minimum free bytes/inodes, max pending trash age/size, oldest cleanup success timestamp, and an emergency owner-approved purge path.

I would treat this as a full-disk cleanup deadlock, not just a missing-directory bug: the component responsible for reclaiming trash has a startup path that itself needs writable/listable storage.

The useful split is:

1. Preserve the real first error. If mkdir/write failed with ENOSPC, do not let a later File.ls/list-dir ENOENT become the surfaced cause.
2. Make the cleanup worker boot resilient on full-disk conditions. Use non-bang list calls, log the true reason, and keep a retry loop alive instead of failing the stack supervisor.
3. Add an admission guard before creating more trash: minimum free bytes/inodes, max pending trash age/size, and last successful delete timestamp.
4. For recovery, capture read-only evidence first:

df -h <data-root>
df -i <data-root>
find <data-root> -maxdepth 4 -type d \( -name "*trash*" -o -name "*deleted*" -o -name "*gc*" \) -print
du -xh <data-root> 2>/dev/null | sort -h | tail -80

Only after that would I define which trash paths are owner-approved for emergency purge versus state that needs product-specific recovery rules.

Do Not Delete First

• Tenant, database, WAL, raft, snapshot, or active state directories without a product owner-approved invariant.
• Trash paths whose ownership is unclear or mixed with active state.
• The first logs that show the original ENOSPC or failed mkdir/write; later errors may be misleading.
• Host-level files from a mounted volume before confirming namespace, tenant, and service ownership.