Posted by File System perspectives from Neil Bannister, head of StorNext file system engineering at Quantum
Neil Bannister is head of StorNext Engineering at Quantum with 20 years of experience in file system and data management technologies. In this new series of blogs, Neil will focus on different technical aspects of the Quantum StorNext file system.
For this first blog, we focus on the defrag capabilities of the StorNext file system.
The History & baseline
File systems have always subjected to fragmentation due to having unsophisticated disk space allocators, poor application I/O behavior, and “aging” (repeated file creation and deletion).
In addition, the de-fragmentation tools used for making individual files contiguous often lead to free space fragmentation. File systems such as XFS, CXFS, GPFS, ETX3 have all suffered in this respect, it is very common in the industry.
Some companies have produced their own so-called “defrag” tools to re-order files in directories to improve I/O performance. One such tool was released by DVS.
The term “defrag” has also become overloaded in the industry, namely:
“defrag” may refer to making the disk space for an individual file contiguous on disk.
“defrag” may refer to making the frames for an image sequence contiguous on disk.
“defrag” may refer to making the file system free space contiguous.
How does StorNext respond to fragmentation prevention & de-fragmentation for repairing ?
StorNext is able to respond to file system fragmentation in 4 ways:
- StorNext has a file-system configuration option (ASR) for use with performance sensitive workflows (Prevention)
Allocation Session Reservations (ASR) is patented technique used for automatically allocating contiguous space to files created within the same directory. This goes a long way to preventing fragmentation in the first place. StorNext file systems have ASR enabled by default and no configuration or modification to applications is required to benefit from its use provided that frame files are simply written in sequence they are expected to be played back.
- snfsdefrag – provides the ability to rearrange the extents of existing files.
snfsdefrag is a standard StorNext utility for making disk space allocation for a file contiguous on disk. This is useful, for example, when an application has written out a single large video file in a non-linear fashion resulting in suboptimal allocation which may lead to “dropped frames” on playback. Snfsdefrag can also generate a detailed report of a file’s current disk allocations.
- vidiomap – provides the ability to re-arrange the on-disk space associated with DPX frames (files) within a directory
vidiomap is a standard StorNext utility used for making the allocations across files within a directory contiguous on disk. For example, if a directory contain “DPX” frame files, it can reorder the allocations so that playout is smooth without any drops. If ASR is enabled and frames are written in order, vidiomap may not be necessary. Even so, there can still be use cases. For example, most “copy” program read and write files in directory hash order (effectively random) instead of processing files in alphanumeric order. So after a sequence of frames has been copied into a file system using such tools, it may be necessary to run vidiomap once before playout to ensure disk space allocation is contiguous.
- sgdefrag – provides the ability to defrag” file system free space
As previously mentioned, the free space of a file system can become fragmented for multiple reasons. This is unavoidable after a file system has “aged.” This makes even the best allocators and tools such as snfsdefrag and vidiomap ineffective because there simply isn’t any remaining contiguous free space to draw from. To correct this, StorNext provides a tool called sgdefrag which can be used to perform reallocations resulting in free space that is more contiguous.