Docker Swarm storage incident

Portainer Swarm Image Pull No Space Left On Device

When a Portainer stack deploy reports success but Swarm tasks move to Rejected, the Docker API may have accepted the service spec before the selected node failed the image pull. Capture the task state, Docker root, blocks, inodes, and image/cache pressure before pruning or redeploying.

Evidence To Capture First

The goal is to distinguish a real disk-full image pull from a missing image, registry error, or Swarm task-state reporting gap.

docker service ps --no-trunc <service_name>
docker service inspect <service_name> --format '{{json .UpdateStatus}}'
docker node ps --no-trunc self | grep -Ei 'Rejected|No space|pull|image' || true
docker info --format 'root={{.DockerRootDir}} driver={{.Driver}}'
df -h / /var/lib/docker 2>/dev/null || df -h
df -i / /var/lib/docker 2>/dev/null || df -i
docker system df -v

Safe Decision Order

1. Confirm the failing node: Swarm can schedule a task on a different node than the one you are watching.
2. Check Docker root, not only /: the image store can be the boundary that blocks layer writes.
3. Separate blocks from inodes: a task may fail even when df -h looks reasonable if inode pressure is the real limit.
4. Classify reclaimable data: build cache and unused images are safer than named volumes, bind mounts, or active service state.

Do Not Delete First

• Do not run broad docker system prune --volumes until named volumes and service mounts are identified.
• Do not assume Portainer success means image pull success; inspect task Rejected messages.
• Do not delete registry credentials, compose files, or Portainer data while investigating a disk-full pull.
• Do not keep retrying the deploy on a full node; it can add partial layers and make the cleanup harder.