Netapp Deduplication Explained

NetApp and the WAFL Filesystem

Any operating system needs some kind of a filesystem to interface with storage. Unix’s traditionally used ADVFS, UFS or VxFS. Linux distro’s use assorted Ext type filesystems. Newer Solaris systems use ZFS (Which I personally think is the greatest filesystem ever!) ZFS is actually LVM and a filesystem in a box. I digress. Netapps have there own filesystem called WAFL. It’s an acronym for Write Anywhere File Layout. It’s an approach to writing data to storage locations that tries to minimize the conventional parity RAID write penalties. This is done by storing system metadata such as inode maps, block maps, and inodes in a structured fashion, such that the filesystem is able to write file system metadata blocks anywhere on the disk. Blocks are never updated per se. New blocks are created and the pointers are changed to point to the newly created blocks. This is different from many of the traditional Unix filesystems. The WAFL filesystem can then again in theory line up multiple writes to the same RAID stripes. This type of striping is ideal for Netapp’s RAID 4 parity scheme which implements dedicated parity drives. Conventional thinkers, philosophers and techies historically ragged against RAID 4, but as far as I’m concerned, Netapp’s implementation with the WAFL filesystem with it’s “full stripped writes” has won me over again. I won’t speak for the philosophers and other techies, but I bet you will find they agree with me. So the above description is a good high level look at Netapps WAFL filesystem. This lays the foundation to deduplication.

Netapp’s Deduplication

Netapp’s deduplication, as the name implies removes duplication. At a high level, it’s really as simple as that. At a low level, ton’s of stuff happens. Deduplication doesn’t happen at the WAFL level, or even the aggregate level. Data Ontap’s Deduplication feature only applies to the volume level, and more specifically the FlexVol type of volume, not the traditional volume. Netapp’s FlexVol isn’t free, you do have to fork out some money for it. If you are seriously interested in saving and recovering a lot of wasted space like when you are working with a VMWare installation as an example, deduplication fits the bill. VMWare does an astoundingly good job at duplicating and wasting space. Deduplication works superbly in this environment. Any data within a single NetApp volume is fair game for being deduplicated, whether it’s raw data or LUN’s on a volume. NetApp’s deduplication feature can be run as a post process which would run based on a watermark setting for a volume or run as a scheduled job. Using a hypothetical watermark of %75 of capacity, deduplication would start automatically if that watermark was reached. Also as I said, a deduplication run can be scheduled for any day of the week, time etc. This is good because you can schedule this type of intensive work during off peak hours. Deduplication uses a fairly complicated algorithm for determining if a block is worthy of being deduplicated. So deduplication works at the block level on the active volume, and uses the “world class” WAFL block-sharing mechanism. A data block has a digital signature which is compared with all other signatures on a volume. If a data block’s digital signature is an exact match, a byte for byte comparison is done for all of the data within the matching block. If the byte for byte comparison proves to be identical the duplicate block is tossed out the window and the disk space is reclaimed. Being tossed out the window is a little too simple. In real life reclaiming a duplicate data block involves updating an indirect inode which it was pointing to, followed by incrementing the block reference count for the block that’s being kept. This ultimately releases the duplicate data block. For the record, an initial deduplication run has to be done to create the digital signatures for comparison. It’s a very effective house cleaning technique, one which admittedly I should apply at home as well