Log retention incident

Skulk stdout stderr Log Disk Full

When ~/.skulk/logs/skulk.stderr.log and skulk.stdout.log grow without a size cap, a verbose node can spend hours writing the same stream until downloads fail and the host has only MiB left. Treat the active log as evidence first, then rotate or truncate with the owning process understood.

Evidence To Capture First

Before deleting or truncating logs, prove whether the active files are still held open. If a process keeps writing to an unlinked file, deleting the pathname may not free space until the process exits.

df -h "$HOME" "$HOME/.skulk" /tmp 2>/dev/null
df -i "$HOME" "$HOME/.skulk" /tmp 2>/dev/null
du -sh "$HOME/.skulk" "$HOME/.skulk/logs" 2>/dev/null || true
du -sh "$HOME/.skulk/logs"/skulk.stderr.log "$HOME/.skulk/logs"/skulk.stdout.log "$HOME/.skulk/logs"/*.log "$HOME/.skulk/logs"/*.zst 2>/dev/null | sort -hr | head -40
ls -lh "$HOME/.skulk/logs"/skulk.stderr.log "$HOME/.skulk/logs"/skulk.stdout.log 2>/dev/null || true
lsof "$HOME/.skulk/logs"/skulk.stderr.log "$HOME/.skulk/logs"/skulk.stdout.log 2>/dev/null || true
tail -n 80 "$HOME/.skulk/logs"/skulk.stderr.log 2>/dev/null || true

Immediate Safety Boundary

1. Keep a short tail sample and the size/timestamp evidence before modifying the active log.
2. If Skulk is still running and owns the file, prefer a controlled restart into a rotating sink over deleting the active file.
3. If emergency truncation is needed, truncate the active file after preserving a small sample; do not remove directories or compressed rotations blindly.
4. Confirm disk bytes actually returned with df -h; if not, check for deleted-open files with lsof +L1.

Durable Fix

• Pipe stdout/stderr through size-based rotation with a per-stream cap and a retained-file count.
• Make verbose mode time-bound or size-bound so -v cannot silently become the largest file on the node.
• Emit a warning before ~/.skulk/logs crosses a fixed byte budget.
• Add an acceptance test that writes repeated output and proves active capture rotates before the host reserve is breached.

I would split this into immediate containment and the durable logging contract.

Read-only evidence before touching the active files:

df -h "$HOME" "$HOME/.skulk" /tmp 2>/dev/null
df -i "$HOME" "$HOME/.skulk" /tmp 2>/dev/null
du -sh "$HOME/.skulk" "$HOME/.skulk/logs" 2>/dev/null || true
du -sh "$HOME/.skulk/logs"/skulk.stderr.log "$HOME/.skulk/logs"/skulk.stdout.log "$HOME/.skulk/logs"/*.log "$HOME/.skulk/logs"/*.zst 2>/dev/null | sort -hr | head -40
ls -lh "$HOME/.skulk/logs"/skulk.stderr.log "$HOME/.skulk/logs"/skulk.stdout.log 2>/dev/null || true
lsof "$HOME/.skulk/logs"/skulk.stderr.log "$HOME/.skulk/logs"/skulk.stdout.log 2>/dev/null || true
tail -n 80 "$HOME/.skulk/logs"/skulk.stderr.log 2>/dev/null || true

For the fix I would require: size-based rotation for the active stdout/stderr sink, a retained-file count, a fixed byte budget for ~/.skulk/logs, and an acceptance test where verbose output cannot consume the host reserve.

Do Not Delete First

• The only tail sample that proves which repeated line caused the growth.
• Compressed rotations before checking whether they are the only useful incident history.
• The whole ~/.skulk directory; logs, config, state, and downloads need separate buckets.
• Active files without checking whether the process still holds the inode open.