Snakemake Sort Tmp No Space Left On Device

When a Snakemake bioinformatics rule fails inside sort with write error: /tmp/sort... No space left on device, the output directory may still have room. The peak scratch files are being written to /tmp or $TMPDIR, and parallel chromosome jobs can multiply the pressure.

Immediate workaround

GNU sort uses $TMPDIR when -T is not supplied. Point TMPDIR at a disk with enough scratch space before starting Snakemake.

mkdir -p /output/run_FIRE/tmp
df -h /tmp /output/run_FIRE /output/run_FIRE/tmp 2>/dev/null || true

export TMPDIR=/output/run_FIRE/tmp
snakemake --profile workflow/profiles/default --cores 8 --rerun-incomplete

If the failing rule is per chromosome and each job uses threads: 4, test with fewer cores first. For example, --cores 4 usually limits that rule to one concurrent chromosome job, which keeps multiple /tmp/sort* files from stacking up.

Read-only evidence

Capture this before removing temp files. It proves whether the full filesystem is /tmp, the working directory, or a shared scratch mount.

echo "== filesystems =="
df -h /tmp "${TMPDIR:-/tmp}" . 2>/dev/null || true
df -i /tmp "${TMPDIR:-/tmp}" . 2>/dev/null || true

echo "== largest tmp entries =="
du -sh "${TMPDIR:-/tmp}"/* /tmp/sort* 2>/dev/null | sort -h | tail -40

echo "== active writers =="
ps -ef | grep -E 'snakemake|sort|samtools|bgzip' | grep -v grep || true

Durable rule fix

For maintainers, the safest workflow-level fix is to make the temp directory explicit inside the rule. That avoids relying on users to know about TMPDIR.

params:
    sort_tmpdir=config.get("sort_tmpdir", os.environ.get("TMPDIR", "/tmp"))
shell:
    """
    mkdir -p {params.sort_tmpdir}
    export TMPDIR={params.sort_tmpdir}
    samtools view -@ {threads} -u {input.cram} {wildcards.chrom} \
        | ft track-decorators -t {threads} --bed12 {output.bed} \
        | sort -T {params.sort_tmpdir} -k1,1 -k2,2n -k3,3n -k4,4 \
        | bgzip -@ {threads} \
    > {output.decorated}
    """

Do not delete first

• Active /tmp/sort* files while sort, bgzip, or Snakemake jobs are still running.
• Snakemake temp/ outputs that another downstream rule has not consumed yet.
• Input .cram, .crai, conda environments, or reference indexes unless the workflow owner confirms they are rebuildable.
• Failure logs and benchmark files that show which filesystem filled first.

That error is coming from GNU sort temp files, not necessarily the FIRE output directory. Because the rule calls plain sort without -T, sort will use $TMPDIR, or /tmp if TMPDIR is unset.

Try pointing TMPDIR at the large run/output disk before launching Snakemake:

mkdir -p /output/run_FIRE/tmp
df -h /tmp /output/run_FIRE /output/run_FIRE/tmp 2>/dev/null || true
export TMPDIR=/output/run_FIRE/tmp
snakemake --profile workflow/profiles/default --cores 8 --rerun-incomplete

If it still fills, lower cores for a test run. The decorate_fibers_chromosome rule is per chromosome, so several sort processes can run at once and multiply scratch usage. A durable rule fix would be to add sort -T "$TMPDIR" or a config-backed sort_tmpdir parameter.