AI agent storage incident

AI Agent Inotify Watch ENOSPC On Linux

If an AI coding extension makes VS Code or a remote Linux session report ENOSPC, No space left on device, or OS file watch limit reached, first prove whether the limit is inotify watches, not disk bytes. Large generated trees like node_modules, .next, cdk.out, logs, data folders, and agent worktrees can exhaust watcher budgets even when df -h looks fine.

Confirm It Is Watch Exhaustion

Classic disk-full cleanup starts with byte usage. Inotify exhaustion starts with process ownership: which process owns the watches, how many watches it has, and whether excluded generated directories are still being watched.

echo "inotify limits"
sysctl fs.inotify.max_user_watches fs.inotify.max_user_instances 2>/dev/null

echo "total watches"
sudo bash -c 'total=0; for f in /proc/[0-9]*/fdinfo/*; do n=$(grep -c "inotify wd:" "$f" 2>/dev/null || true); total=$((total+n)); done; echo "$total"'

echo "top watcher owners"
for pid in $(find /proc/*/fd -lname anon_inode:inotify 2>/dev/null | sed 's#/proc/##; s#/fd/.*##' | sort -u); do
  watches=$(grep -h "inotify wd:" /proc/$pid/fdinfo/* 2>/dev/null | wc -l)
  comm=$(cat /proc/$pid/comm 2>/dev/null)
  cmdline=$(tr "\0" " " < /proc/$pid/cmdline 2>/dev/null | cut -c1-220)
  printf '%8s %7s %-20s %s\n' "$watches" "$pid" "$comm" "$cmdline"
done | sort -nr | head -20

Separate Watch Budget From Disk Cleanup

Raising fs.inotify.max_user_watches can be a temporary workaround, but it does not fix an extension that recursively watches generated output. Safe triage separates four buckets:

1. Generated trees: node_modules, .next, dist, build, cdk.out, coverage output, mobile build folders, and test artifacts.
2. Agent runtime state: .claude/worktrees, .sandcastle/worktrees, hidden agent session folders, transcripts, and logs.
3. Git internals: .git/objects, packed refs, worktrees, and nested repositories.
4. User work: source files, active branches, local data, credentials, and anything not clearly rebuildable.

echo "top immediate directories by apparent size"
du -xhd1 . 2>/dev/null | sort -h | tail -30

echo "top known generated directories by file count"
for dir in node_modules .next dist build cdk.out logs data coverage .claude .sandcastle .git; do
  [ -e "$dir" ] || continue
  count=$(find "$dir" -xdev -type f 2>/dev/null | wc -l)
  size=$(du -xsh "$dir" 2>/dev/null | awk '{print $1}')
  printf '%10s files  %8s  %s\n' "$count" "$size" "$dir"
done | sort -nr

Fix Criteria For The Extension Or Tool

• Honor VS Code files.watcherExclude, search.exclude, .gitignore, or an extension-specific exclude list before allocating recursive watches.
• Never recursively watch .git/objects, dependency trees, generated build output, agent worktrees, logs, data dumps, or cache folders by default.
• Set a watcher budget, for example a small percentage of fs.inotify.max_user_watches, and enter degraded mode before the desktop session hits ENOSPC.
• Release extension-created watches when chat panels, workspaces, or agent sessions close.
• Show a clear message such as "inotify watch budget exhausted" instead of generic "No space left on device".

Do Not Delete First

• Do not delete active agent worktrees or transcripts until you know whether a session needs recovery.
• Do not delete .git paths to reduce watcher count.
• Do not assume ENOSPC means byte storage is full. Check df -h, df -i, and inotify counts separately.
• Do not rely only on sysctl increases. They can hide the extension-side recursion bug and make the next failure larger.