Post-Mortem: Backup Failure via Filesystem Limit Exhaustion
**Incident Summary:**
A website backup process failed after a single 1.6MB GIF (a "Friends" reaction animation) was replicated 246,173 times within the storage system, inflating its size to 377GB and triggering filesystem limits.
**Root Cause:**
The issue stemmed from a security policy in the Discourse platform's "secure uploads" feature. To maintain security contexts, when a file moves between private and public scopes (e.g., PM to public post), the system generates a new copy with a randomized SHA1 hash rather than referencing the original. High community usage of a specific GIF caused massive data duplication.
**Technical Failure Chain:**
1. **Data Inflation:** Excessive duplication led to 377GB of backup bloat from a single asset.
2. **Failed Mitigation (Hardlinks):** An initial attempt to use hardlinks to group identical hashes failed because the number of duplicates exceeded the **ext4 filesystem limit of ~65,000 hardlinks per inode**. This resulted in approximately 181,000 fallback downloads, failing to resolve the bloat.
**Resolution:**
New logic was implemented for the backup process: The system attempts to create hardlinks first; if the filesystem returns an `EMLINK` error (too many hardlinks), it automatically switches to creating a local copy and designates that as the "primary" file until the limit is reached again. This ensures compatibility across all filesystems without manual configuration.
**Incident Summary:**
A website backup process failed after a single 1.6MB GIF (a "Friends" reaction animation) was replicated 246,173 times within the storage system, inflating its size to 377GB and triggering filesystem limits.
**Root Cause:**
The issue stemmed from a security policy in the Discourse platform's "secure uploads" feature. To maintain security contexts, when a file moves between private and public scopes (e.g., PM to public post), the system generates a new copy with a randomized SHA1 hash rather than referencing the original. High community usage of a specific GIF caused massive data duplication.
**Technical Failure Chain:**
1. **Data Inflation:** Excessive duplication led to 377GB of backup bloat from a single asset.
2. **Failed Mitigation (Hardlinks):** An initial attempt to use hardlinks to group identical hashes failed because the number of duplicates exceeded the **ext4 filesystem limit of ~65,000 hardlinks per inode**. This resulted in approximately 181,000 fallback downloads, failing to resolve the bloat.
**Resolution:**
New logic was implemented for the backup process: The system attempts to create hardlinks first; if the filesystem returns an `EMLINK` error (too many hardlinks), it automatically switches to creating a local copy and designates that as the "primary" file until the limit is reached again. This ensures compatibility across all filesystems without manual configuration.
