Note: the links and images in this document may point to external sources. Please review them before opening.
What to Do When Your Linux Server Reaches 100% Disk Space (Without Crashing Production)
Published under: Linux Troubleshooting • Reading time: 8 minutes • Written by Thomas C (20+ Years Enterprise Linux Engineering)
⚠️ Is your production server locked right now?
If MySQL is down, Nginx is returning 500 errors, or SSH is refusing logins due to acute disk exhaustion, our senior Linux engineer can triage the incident directly under our 1-hour emergency response protocol.
👉 Request Urgent Emergency Support ($199) (< 1-Hour Initial Triage)
(https://pro.osassistance.com/images/linux-disk-space-100-percent-triage.svg) Diagnostic terminal showing linux disk space 100% full error and recovery commands
[CRITICAL ALERT] Filesystem /dev/mapper/ubuntu--vg-ubuntu--lv is at 100% capacity (0 bytes available).
When a production server encounters a linux disk space 100% full condition on / or /var, normal operating system processes halt. Relational databases like MySQL and PostgreSQL enter read-only panic states, Nginx aborts incoming client requests, Redis drops background snapshots, and PAM blocks new SSH sessions because /var/log/lastlog or /tmp can no longer accept writes.
In a state of panic, administrators frequently execute blind rm -rf commands on active logs. This often worsens the outage by detaching files while leaving allocated disk blocks locked in active process memory.
Safely resolving a linux disk space 100% emergency requires a systematic, non-destructive sequence. This guide outlines the exact diagnostic and recovery playbook used across production Ubuntu, Debian, RHEL, and Rocky Linux environments to restore system availability without data loss.
⚠️ Stop: Avoid Deleting Open Log Files During a Linux Disk Space 100% Outage
The most destructive mistake during a linux disk space 100% incident is unlinking an active log while the parent daemon continues writing to it:
# AVOID DOING THIS WHILE SERVICES ARE RUNNING:
rm -f /var/log/nginx/access.log
Why File Deletion Fails
In POSIX filesystem semantics, an inode is only released back to the free block pool when two conditions are satisfied:
- Its directory hard-link count drops to zero (
nlink = 0). - All open file descriptors referencing that inode are closed.
If Nginx or Docker holds /var/log/nginx/access.log open in memory and you execute rm, the file pointer disappears from the directory tree, but the disk blocks remain occupied. Utilities like du can no longer find the file, creating an invisible storage leak.
The Safe Method: File Truncation
To immediately reclaim blocks from an active file without terminating the service, truncate the file descriptor to zero bytes:
# Safe truncation methods:
: > /var/log/nginx/access.log
# OR
truncate -s 0 /var/log/nginx/access.log
Truncation zeroes the allocated blocks immediately while keeping the existing file descriptor intact for the running application.
Step 1: Rapid Triage for Linux Disk Space 100% Full Filesystems
To diagnose a linux disk space 100% issue, determine whether the exhaustion stems from physical storage block limits or filesystem inode exhaustion.
1. Identify Exhausted Block Storage
df -hT
Locate any mount points showing 100% or Use% >= 95%:
Filesystem Type Size Used Avail Use% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv ext4 48G 48G 0 100% /
2. Verify Inode Availability
A filesystem can trigger an out-of-space error with gigabytes of free block capacity if all index nodes (inodes) have been consumed by millions of zero-byte session or spool files:
df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/ubuntu--vg-ubuntu--lv 3.1M 3.1M 0 100% /
If IUse% indicates 100%, skip to Step 5 to address inode exhaustion directly.
Step 2: Find Ghost Files Holding Linux Disk Space 100% Full (lsof)
When df -h reports that a volume is completely full, but directory scans with du -sh /* account for only a fraction of that data, unlinked open files are almost always the cause of the linux disk space 100% discrepancy.
Inspect unlinked files held open by running processes:
lsof +L1 /
Example diagnostic output:
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NLINK NODE NAME
nginx 12480 root 7w REG 253,0 1845209100 0 264192 /var/log/nginx/access.log (deleted)
java 28411 app 3w REG 253,0 1284910291 0 134102 /opt/app/logs/debug.log (deleted)
Releasing Unlinked Space
The column NLINK 0 (deleted) indicates that the 18GB Nginx log was deleted from disk, but file descriptor 7w continues consuming storage space.
To reclaim this space without rebooting the server:
- Instruct the service to reopen its log handles:
systemctl reload nginx - If the daemon cannot be reloaded, clear the file directly through the
/procvirtual filesystem using the process ID (PID) and file descriptor (FD):: > /proc/12480/fd/7
Step 3: Fast Cleanup to Recover from Linux Disk Space 100% Full
When system services are stalled, apply targeted maintenance steps to restore write availability during a linux disk space 100% freeze.
1. Vacuum Systemd Journal Logs
Modern distributions running systemd-journald frequently accumulate gigabytes of historical system logs:
# Check journal storage volume:
journalctl --disk-usage
# Retain only the last 2 days of logs or restrict total volume to 500MB:
journalctl --vacuum-time=2d
# OR
journalctl --vacuum-size=500M
2. Clean Package Manager Caches
Package archives cached during distribution upgrades can consume substantial space in /var/cache/:
- Ubuntu / Debian:
apt-get clean apt-get autoremove --purge -y - RHEL / Rocky Linux / AlmaLinux:
dnf clean all
3. Clear Stale Compressed Logs
Rotated archives (.gz or .1) that have already aged out can be removed safely:
# Identify and remove compressed historical archives:
find /var/log -type f \( -name "*.gz" -o -name "*.1" \) -delete
4. Prune Unused Docker Artifacts
Unused container layers, dangling images, and build caches are common causes of sudden disk exhaustion:
# Prune stopped containers, dangling images, and build caches:
docker system prune -f
# Review active container JSON log sizes:
find /var/lib/docker/containers/ -name "*-json.log" -exec du -sh {} +
To clear an oversized container log without restarting the container:
: > /var/lib/docker/containers/<container-id>/<container-id>-json.log
Step 4: Locating Large Directories in a Linux Disk Space 100% Outage
Once immediate write capacity is restored, identify which directory trees are driving storage growth.
Audit Mount Consumption
Run an initial one-level directory scan from root while suppressing virtual filesystem warnings:
du -xhd1 / 2>/dev/null | sort -h
Example scan output:
4.0K /mnt
128M /boot
1.8G /usr
4.2G /home
38G /var
In this scenario, /var accounts for 38GB of total usage. Inspect /var directly:
du -xhd1 /var 2>/dev/null | sort -h
Interactive Disk Inspection with ncdu
For an interactive ncurses-based tree view, use ncdu:
# Install ncdu (Ubuntu/Debian):
apt-get install -y ncdu
# Install ncdu (RHEL/Rocky via EPEL):
dnf install -y ncdu
# Scan filesystem boundary without crossing mounts:
ncdu -x /
Step 5: Inode Exhaustion: The Hidden Linux Disk Space 100% Lock
If df -h shows adequate storage capacity but write operations fail with No space left on device, an inode exhaustion issue has occurred. This typically happens when millions of small files fill the filesystem metadata tables.
Common causes include:
- Unpurged PHP session files in
/var/lib/php/sessions/ - Backlogged mail queues in
/var/spool/postfix/maildrop/ - Framework file-based cache stores (e.g., Laravel, Magento)
Identify High File Count Directories
find / -xdev -printf '%h\n' 2>/dev/null | sort | uniq -c | sort -k 1 -n | tail -20
Safely Remove Large File Batches
Standard commands like rm -f * fail with Argument list too long when processing directories with hundreds of thousands of files. Use find with the -delete action:
# Remove PHP session files older than 7 days:
find /var/lib/php/sessions -type f -mtime +7 -delete
For detailed specifications on filesystem limits and metadata structures, refer to the Linux Kernel Filesystem Documentation and standard man7 Linux Programmer’s Manual.
Step 6: Expand Storage Online to Solve Linux Disk Space 100% Permanently
When storage requirements permanently outgrow provisioned drive space, expand the underlying storage volume. Systems configured with Logical Volume Manager (LVM) on cloud infrastructure (AWS EBS, Hetzner, Google Cloud, or VMware) support online expansion without downtime.
1. Verify Storage Topology
lsblk
vgs
lvs
2. Extend the Logical Volume
# Increase logical volume by 20GB:
lvextend -L +20G /dev/mapper/ubuntu--vg-ubuntu--lv
# Or allocate all remaining free capacity in the volume group:
lvextend -l +100%FREE /dev/mapper/ubuntu--vg-ubuntu--lv
3. Expand the Filesystem Online
- For EXT4 filesystems:
resize2fs /dev/mapper/ubuntu--vg-ubuntu--lv - For XFS filesystems (standard on RHEL / Rocky Linux):
xfs_growfs /
Confirm updated storage dimensions with df -h /.
Permanent Checklist to Prevent Linux Disk Space 100% Recurrence
Implement these operational baselines to prevent a linux disk space 100% full incident from recurring across your server fleet:
- Enforce Logrotate Policies: Verify that all application services under
/var/logmaintain rotation rules in/etc/logrotate.d/specifyingmaxsize 500M,rotate 7, andcompress. - Limit Container Log Allocations: Set explicit daemon retention limits in
/etc/docker/daemon.json:{ "log-driver": "json-file", "log-opts": { "max-size": "100m", "max-file": "3" } } - Cap Journal Log Usage: Configure storage ceilings in
/etc/systemd/journald.conf:SystemMaxUse=1GApply the configuration viasystemctl restart systemd-journald. - Establish Early Alerts at 80%: Configure monitoring alerts through Prometheus, Datadog, or custom metrics when volume capacity crosses 80%, providing adequate lead time before services experience a linux disk space 100% failure.
Need Dedicated Support for Your Linux Fleet?
Managing disk space, system performance, and routine security updates across production servers requires continuous engineering attention.
- Server Down Right Now? If acute disk space exhaustion has taken your services offline, connect directly with an enterprise engineer under our 1-hour emergency response protocol.
👉 Request Urgent Emergency Support ($199) (< 1-Hour Initial Triage) - Looking for Ongoing Linux Operations? Keep your environments stable with proactive sysadmin care covering routine maintenance, updates, and break-fix operations for a flat $4,000/month.
👉 Explore Linux Care ($4,000/mo)
