Failure Description

Engineers received a call from a customer, a Linux system with abnormal disk usage. 

df -h found that the disk usage is 100%, after reboot and try df -h again, the disk usage is still 100%. 

After reboot, try df -h again, the disk usage is still 100%. 

lsof check that no application is occupying the deleted data; du -sh / check / directory space occupied and the actual disk usage is inconsistent, 

the difference is very large, there is no free space on the disk to store data.

Troubleshooting

2.1 The command output phenomenon is as follows

2.2  Troubleshooting

After comparative analysis, it is found that this system is using the default btrfs file system of SUSE12 version. btrfs file system uses the snapshot function, 

which will take a snapshot of the system once every period of time, or when components have been installed by using yast, zypper, 

or kernel patches have been installed, in order to facilitate the subsequent rollback of the system.

Check the snapshots snapshot file in the root directory. The .snapshots file occupies 96G:

View the snapshot files on your system with the Snapshot Manager command to see multiple snapshot files taking up space:

Troubleshooting

Use the Snapshot Manager commands for deleting snapshot files:

snapper delete --sync 155

snapper delete --sync 156

snapper delete --sync 159

snapper delete --sync 160

snapper delete --sync 161

snapper delete --sync 162

snapper delete --sync 163

snapper delete --sync 164

snapper delete --sync 165

snapper delete --sync 166

After deleting the snapshot the disk space returns to normal. If this kind of problem occurs again in the future, you can follow the above way to delete and restore.

Summary of experience

Snapper allows the creation and management of file system snapshots. 

File system snapshots allow a copy of the state of the file system to be kept at a point in time.

The standard settings of Snapper are designed to allow rollback of system changes. 

However, it can also be used to create disk backups of user data. 

As a basis for this feature, Snapper uses Btrfs file systems or thinly configured LVM volumes with XFS or Ext4 file systems.

4.1 Snapshot Types

While snapshots themselves are indistinguishable in a technical sense, they can be categorized into 3 types based on the event that triggers them:

1) Timeline snapshots

A snapshot is created every hour. Older snapshots are automatically deleted. By default, the first snapshot of the last ten days, months and years is kept. 

Timeline snapshots are disabled by default.

2) Installation Snapshots

Whenever one or more packages are installed using YaST or Zypper, a pair of snapshots is created: 

one before the installation begins (Pre) and one after the installation is complete (Post). 

If important system components are installed, such as the kernel, the snapshot pair is marked as an "important" (important=yes) snapshot. 

Older snapshots are automatically deleted. 

By default, the last ten important snapshots and the last ten "regular" (including administrative snapshots) snapshots are retained. 

Installation snapshots are enabled by default.

3) Management Snapshots

A pair of snapshots is created whenever a YaST managed system is used: one when the YaST module is started (Pre) and one when the module is shut down (Post). 

Older snapshots are automatically deleted. 

By default, 

the last ten important snapshots and the last ten "regular" snapshots (including installation snapshots) are retained. 

Administrative snapshots are enabled by default.

4.2 Deleting Snapshots

To delete a snapshot using the YaST Snapper module: first select a snapshot from the list and click Delete.

To delete a snapshot using the command line tool, you need to use the snapshot number: 

run snapper list to get the snapshot number of the current system; 

run snapper delete number to delete the snapshot.

When you delete a snapshot using Snapper, the freed space will be occupied by the Btrfs process running in the background. 

As a result, visibility and availability of available space is delayed. 

If you need to free up space by deleting immediately available snapshots, 

use this option --sync in conjunction with the delete command.

Note: When deleting a pre snapshot, its corresponding post snapshot should be deleted along with it (and vice versa).

snapper delete --sync 23

Deletes snapshot 23 of the default (root) configuration and makes the freed space immediately available.

Tip: Older snapshots take up more disk space, and the older the snapshot, the more disk space it takes up. 

Therefore, when freeing up disk space by deleting a snapshot, you should delete the old snapshot first.

For more information, please visit Antute's official website:xtn4.e21system.com