Apptainer SIF No Space Left On GitHub Actions

When Apptainer fails at Creating SIF file... on a hosted runner, the final write is usually only the messenger. The runner may need package cache, unpacked sandbox rootfs, temporary build state, and the final .sif to coexist on the same small root disk.

First Response Runbook

1. Log root, workspace, temp, and Docker disk separately before the Apptainer step.
2. Run cleanup before the SIF build, not after the failure.
3. Measure the package cache and Apptainer temp/cache paths so the final SIF is not blamed alone.
4. Set Apptainer temp/cache paths explicitly if the default root disk is the bottleneck.
5. Keep named datasets, generated artifacts, and final SIF outputs out of broad prune commands.

echo "before cleanup"
df -h / "$GITHUB_WORKSPACE" /tmp 2>/dev/null || true
du -sh "$GITHUB_WORKSPACE" /tmp "$HOME/.cache" "$HOME/.apptainer" 2>/dev/null || true

# Optional hosted-runner cleanup before the SIF build.
# Keep this before dependency install and apptainer build.
sudo rm -rf /usr/share/dotnet /usr/local/lib/android /opt/ghc /usr/local/share/boost 2>/dev/null || true
docker system prune -af 2>/dev/null || true

echo "before apptainer build"
df -h / "$GITHUB_WORKSPACE" /tmp 2>/dev/null || true
du -sh "$GITHUB_WORKSPACE" /tmp "$HOME/.cache" "$HOME/.apptainer" 2>/dev/null || true

Why The Failure Clusters By Image Size

Large GPU images often sit near the hosted-runner disk threshold. CUDA packages, conda packages, the unpacked filesystem tree, and the final SIF can overlap. If only the largest stages fail while smaller CPU or GPU stages pass, treat it as peak disk pressure before assuming the PR changed application code.

I would treat the SIF write as the peak-storage point, not necessarily the root cause.

For GitHub-hosted runners, the Apptainer build can need these to coexist on the same root disk:
- conda/package cache
- unpacked sandbox rootfs
- Apptainer temp/cache
- the final .sif output

Useful checks before changing dependencies:

df -h / "$GITHUB_WORKSPACE" /tmp 2>/dev/null || true
du -sh "$GITHUB_WORKSPACE" /tmp "$HOME/.cache" "$HOME/.apptainer" 2>/dev/null || true

If only the biggest GPU SIFs fail, I would add a cleanup step before apptainer build, log disk again, and set an acceptance rule like "at least 2x final SIF size plus package cache free before SIF creation". Avoid pruning named data/output directories after the failure, because the important move is freeing runner software/cache before the peak starts.

Safe Cleanup Order

1. Remove unused hosted-runner SDKs and toolchains that the job never needs.
2. Prune Docker only if the job does not depend on earlier local images or volumes.
3. Put conda/pixi/package caches on a known path and measure it.
4. Move Apptainer temp/cache to a larger mounted path only if that path exists and survives the build.
5. Split the largest GPU SIFs into separate jobs if the peak still exceeds the hosted-runner budget.