Criterion Benchmark Temp DB No Space Left On Device

A Rust benchmark can look like a database bug when the real failure is storage pressure: each Criterion iteration creates a fresh SQLite database or fixture tree, the teardown boundary is missing, and /tmp or $TMPDIR fills before the run finishes.

Fast diagnosis

Run these read-only checks before deleting anything. They separate a full filesystem from a benchmark that leaks per-iteration temp state.

echo "== filesystems =="
df -h /tmp "${TMPDIR:-/tmp}" 2>/dev/null || true

echo "== likely benchmark temp databases =="
find /tmp "${TMPDIR:-/tmp}" -maxdepth 1 \( -name 'rag-rat-bench-*' -o -name '*bench*.sqlite' -o -name '*criterion*' \) -print 2>/dev/null | head -80

echo "== largest matching temp entries =="
du -sh /tmp/rag-rat-bench-* "${TMPDIR:-/tmp}"/*bench*.sqlite 2>/dev/null | sort -h | tail -30

What usually happened

• iter_batched or similar setup creates a new temp DB, fixture checkout, or generated corpus for each iteration.
• The setup returns only the database path or handle, so the owning temp directory is dropped too early or never cleaned.
• Criterion sample count multiplies the peak size: 50 iterations at 1.5GB each can consume 75GB even when one iteration looks reasonable.
• SQLite then reports database or disk is full, or the runner shows No space left on device.

Safer benchmark fixes

1. Make the temp directory the owned batched input, then drop it after the measured closure finishes.
2. Use a dedicated benchmark temp root so cleanup cannot touch unrelated /tmp entries.
3. Add a post-run assertion that no benchmark-owned temp paths remain.
4. Log df -h and top temp entries before and after the benchmark in CI.
5. Lower sample count only as a mitigation; it should not be the root fix if temp ownership is leaking.

I would treat this as a benchmark temp ownership bug, not just a low-disk runner problem.

The useful checks are:
- df -h /tmp "$TMPDIR" before and after the benchmark
- top benchmark-owned temp paths after a failed run
- whether each iter_batched setup creates a fresh DB/fixture tree
- whether the owning TempDir is kept alive until teardown
- a post-run assertion that no benchmark-owned temp paths remain

The durable fix is to make each generated DB or fixture tree owned by a scoped temp directory and drop/remove it after the measured iteration. Lowering Criterion samples only reduces the blast radius.

Do not delete first

• Arbitrary /tmp contents while tests, databases, or browsers are still running.
• Open SQLite files, WAL files, or fixture directories that an active benchmark process still owns.
• CI logs, flamegraphs, Criterion reports, or failure artifacts before they are uploaded.