CPU Image Pulls CUDA Dependencies And Fills CI Disk

A hosted runner can run out of disk even when the failing image is supposed to be CPU-only. The real bug is often a dependency contract leak: vLLM, GPU torch, nvidia wheels, CUDA toolkit, flashinfer, or triton enter the default backend image instead of the GPU image.

First Response Runbook

1. Define the default backend image contract: no vLLM, CUDA toolkit, nvidia wheels, flashinfer, triton, or GPU torch closure unless explicitly enabled.
2. Move GPU inference dependencies into an optional extra, separate requirements file, or dedicated GPU image.
3. Log `df -h /`, Docker/buildx disk usage, and `pip freeze | grep -Ei "vllm|nvidia|cuda|triton|torch"` before the smoke build.
4. Add a build-time assertion that fails when the default image contains CUDA packages.
5. Keep the GPU image smoke path separate so removing vLLM from the backend does not break intended GPU deployments.

I would make this an image-contract check, not only a runner cleanup.

Acceptance checks:

- Default backend image does not install vllm, cuda-toolkit, nvidia-* wheels, flashinfer, triton, or GPU torch by default.
- GPU/vLLM dependencies live in an optional extra, separate requirements file, or dedicated GPU image.
- Smoke-test logs `df -h /`, `docker system df`, and `pip freeze | grep -Ei "vllm|nvidia|cuda|triton|torch"` before heavy build steps.
- Build fails if the default image dependency closure contains CUDA packages.
- GPU image keeps its own smoke path proving vLLM still installs where intended.

Paid Scope

The $29 incident triage reviews one failing build and returns the safest next diagnostic step. The $29 deep cleanup turns one representative Docker CI disk-full into a backend-vs-GPU image dependency contract, guardrail assertions, and runner disk preflight checklist.