Disk Offload Inode Exhaustion And Backpressure

A disk offload strategy can look efficient in memory while quietly moving the outage to the filesystem: millions of tiny files, inode pressure, block amplification, slow compaction, and queue writes that only fail when the OS finally returns ENOSPC.

Runbook: Avoid Moving The Outage To The Filesystem

1. Measure file count, byte count, and inode use separately. A small 31 GB store can still be an 8 million inode operational problem.
2. Estimate amplification: average bytes per file, filesystem block size, backup/rsync impact, and directory traversal time.
3. Set a backend threshold. One-file-per-entry can be acceptable below a documented count; above it, require segment files, append-log plus index, mmap, or another bounded-file design.
4. Make compaction and recovery explicit. Segment logs need hint/replay behavior, tombstone handling, crash recovery tests, and bounded background compaction.
5. Add queue backpressure before ENOSPC: minimum usable bytes, max queue bytes, callback/reject behavior, and a metric operators can alert on.
6. Make retention safe by default. Time-based cleanup should not silently drop unread records without a clamp, warning, or explicit opt-in.

I would split this into two acceptance-test groups:

1. Disk-offload scale
- N entries should not create O(N) files once the documented threshold is exceeded.
- Recovery from segment/hint files should be tested after a partial write and after compaction.
- Metrics should expose file count, bytes, compaction backlog, and recovery replay cost.

2. Queue backpressure and retention
- offer() should reject or call a hook before the filesystem reaches ENOSPC.
- A soft watermark should be configurable by min free bytes and/or max queue bytes.
- TIME_BASED retention should warn or clamp to consumer progress before unread records are dropped.

Read-only evidence:

OFFLOAD_ROOT=<offload-root>
df -h "$OFFLOAD_ROOT"
df -i "$OFFLOAD_ROOT"
find "$OFFLOAD_ROOT" -xdev -type f | wc -l
du -sh "$OFFLOAD_ROOT"

Do Not Delete First

• Active segment, hint, index, or queue files before recording recovery behavior and unread-record state.
• All small offload files before measuring file count, average bytes per file, and inode pressure.
• Old queue logs before deciding whether consumers have read them.
• Retention metadata, tombstones, or compaction markers that are needed for crash recovery.