Postgres DiskFullError Shared Memory Temp Space

PostgreSQL can throw DiskFullError while the main data disk looks healthy. The usual trap is temp-space or /dev/shm pressure from parallel hash joins, sorts, materialized subplans, or expensive summary endpoints. Treat it as a query-and-memory budget incident, not just a disk cleanup task.

Runbook: Fix The Query Budget, Not Just The Disk

1. Do not assume the main data volume is full. Check /dev/shm, container shared-memory size, temp directories, and Postgres temp counters separately.
2. Find the failing endpoint and query family. Repeated operations_summary or prediction endpoints usually point to one expensive plan, not random storage pressure.
3. Enable or inspect temp-file logging. Large temp files identify sort/hash/materialize nodes that need query-plan work.
4. Estimate concurrent memory pressure. work_mem applies per operation per worker; raising it globally can make the next incident worse.
5. Prefer targeted changes: indexes, precomputed summaries, narrower time windows, lower parallelism on the endpoint, temp_file_limit, and statement timeouts.
6. Make the incident observable: alert on repeated DiskFullError clusters, high temp_bytes delta, /dev/shm free space, and summary endpoint timeout rate.
7. After a change, run the same query under expected concurrency and confirm temp_bytes, latency, and error count all move in the right direction.

I would treat the Postgres DiskFullError as a temp-space / shared-memory budget incident first, not as ordinary disk cleanup.

Acceptance checks I would add:
- Capture `df -h /dev/shm /tmp /` next to every DiskFullError cluster.
- Log or query `pg_stat_database.temp_files/temp_bytes` before and after the failing window.
- Identify the exact endpoint/query family that triggers `could not resize shared memory segment`.
- Run `EXPLAIN (ANALYZE, BUFFERS)` on the summary/prediction query and look for hash/sort/materialize nodes plus parallel workers.
- Estimate worst-case memory as work_mem * memory nodes * workers * concurrent requests before any global work_mem increase.
- Add a guard: statement timeout or temp_file_limit for the endpoint, plus an alert on repeated DiskFullError and `/dev/shm` low-space.
- Verify the fix by replaying the endpoint and watching temp_bytes, latency, and timeout count.

Do Not Change First

• Global work_mem before estimating per-query nodes, parallel workers, and concurrent request count.
• Postgres data files, WAL, or temp directories before proving the process holding them is stopped or safe to interrupt.
• Container /dev/shm size without also bounding the query that can consume it.
• Health checks that only report main disk percentage while shared memory and temp-space pressure remain invisible.