pg_dump Partial Backup Disk Full

A backup sidecar can look healthy while quietly publishing an empty or truncated PostgreSQL dump. The dangerous pattern is writing `pg_dump -Fc` directly to the final `.dump` path, then letting retention or offsite upload treat that file as a valid restore point.

Runbook: Never Publish A Failed Dump

1. Stop redirecting `pg_dump` directly to the final backup path. Shell redirection creates or truncates that file before `pg_dump` has proved anything.
2. Write the dump to a temp path on the same filesystem, for example `app-2026-06-17.dump.partial`.
3. Delete an old stale `.partial` for the same job before starting, or make the temp name unique per timestamp.
4. Require `pg_dump` exit code 0, a non-empty temp file, and `pg_restore -l` success for custom-format dumps.
5. Rename the temp file to the final `.dump` name only after verification passes. This makes final names mean "restore candidate".
6. Run offsite upload and retention only after final publication. Failed partials should not prune the previous known-good backup.
7. Emit a greppable failure line and call a failure hook such as `BACKUP_ON_FAILURE_CMD` so operators can wire email, Slack, or pager alerts.

I would make the backup sidecar publish-by-verification rather than write directly to the final dump path.

Acceptance checks I would add:
- A failing pg_dump never creates or truncates a final app-*.dump / zitadel-*.dump file.
- The dump writes to "$final.partial" on the same filesystem, then renames only after pg_dump exits 0.
- The sidecar verifies test -s "$final.partial" and, for -Fc dumps, pg_restore -l "$final.partial".
- Retention and offsite upload ignore .partial files and do not prune the previous known-good dump after a failed run.
- Failed runs emit a distinct [backup] FAILED line and fire BACKUP_ON_FAILURE_CMD if configured.
- The runbook proves the latest dump is restorable, not just present in ls -lh.

Do Not Delete First

• The previous known-good dump before the replacement dump has passed verification.
• `.partial` files before recording timestamp, size, and the failure that created them.
• Backup logs that prove `pg_dump` exited non-zero or `pg_restore -l` failed.
• Offsite restore points before confirming whether they were uploaded from verified final names or failed partial output.