Restoring data from Snashots using Snaprestore command

Ok so now you have allocated correct snap reserve space, configured snap schedules, snap autodelete, users have access to their snapshots and they recover their data without any interference of backup team. Everyone is happy so you happy but all of sudden on a Friday evening get a call from VP marketing crying on phone that he lost all his data from his network drive and windows shows recovery time of 2 hrs but he wants his 1Gb pst to be accessible now as he is on VPN with a client and needs to pull some old mails from his pst. Well that’s nothing abnormal as he was having lots of data and to recover the data windows has to read all the data from snapshot and then write back on network drive which but obvious will take time. Now what would you say, will you tell him to navigate to his pst and recover it (which shouldn’t take much time on fast connection) then try to recover all the data or ok I have recovered all your data while talking on the phone and become hero.

Well I must say I would like to use the opportunity to become hero with a minute or less of work, but before we do a few things to note.

For volume snaprestore:

·                     The volume must be online and must not be a mirror.

·                     When reverting the root volume, filer will be rebooted.

·                     Non-root volumes do not require a reboot however when reverting a non-root volume, all ongoing access to the volume must be terminated, just as is done when a volume is brought offline.

For single-file snaprestore:

·                     The volume used for restoring the file must be online and must not be a mirror.

·                     If restore_as_path is specified, the path must be a full path to a filename, and must be in the same volume as the volume used for the restore.

·                     Files other than normal files and LUNs are not restored. This includes directories (and their contents), and files with NT streams.

·                     If there is not enough space in the volume, the single file snap restore will not start.

·                     If the file already exists (in the active file system), it will be overwritten with the version in the snapshot.

To restore data there are two ways, first system admins using “snap restore” command invoked by SMO, SMVI, Filer view or system console and second by end users where they can restore by copying file from .snapshot or ~snapshot directory or by using revert function in XP or newer system. However restoring data through snap restore command is very quick (seconds) even for TBs of data. Syntax for snap restore is as below.

“snap restore -t vol -s <snapshot_name> -r <restore-as-path> <volume_name>”

If you don’t want to restore the data at different place then remove the “-r <restore-as-path>” argument and filer will replace current file with the version in snapshot and if you don’t provide a snapshot name in syntax then system will show you all available snapshots and will prompt to select snapshot from which you want to restore the data.

Here’s the simplest form of this command as example to recover a file.

testfiler> snap restore -t file /vol/testvol/RootQtree/test.pst

WARNING! This will restore a file from a snapshot into the active filesystem. If the file already exists in the active filesystem, it will be overwritten with the contents from the snapshot.

Are you sure you want to do this? yes

The following snapshots are available for volume testvol:

date            name
------------    ---------
Nov 17 13:00    hourly.0
Nov 17 11:00    hourly.1
Nov 17 09:00    hourly.2
Nov 17 00:00    weekly.0
Nov 16 21:00    hourly.3
Nov 16 19:00    hourly.4
Nov 16 17:00    hourly.5
Nov 16 15:00    hourly.6
Nov 16 00:00    nightly.0
Nov 15 00:00    nightly.1
Nov 14 00:00    nightly.2
Nov 13 00:00    nightly.3
Nov 12 00:00    nightly.4
Nov 11 00:00    nightly.5
Nov 10 00:00    weekly.1
Nov 09 00:00    nightly.6
Nov 03 00:00    weekly.2
Oct 27 00:00    weekly.3

Which snapshot in volume testvol would you like to revert the file from? nightly.5

You have selected file /vol/testvol/RootQtree/test.pst, snapshot nightly.5

Proceed with restore? yes
testfiler>

Data Deduplication Concepts

Data Deduplication

What is Data Deduplication?

Data deduplication essentially refers to the elimination of redundant data. In the deduplication process, duplicate data is deleted, leaving only one copy of the data to be stored. However, indexing of all data is still retained should that data ever be required. Deduplication is able to reduce the required storage capacity since only the unique data is stored.

For example, a typical email system might contain 100 instances of the same one megabyte (MB) file attachment. If the email platform is backed up or archived, all 100 instances are saved, requiring 100 MB storage space. With data deduplication, only one instance of the attachment is actually stored; each subsequent instance is just referenced back to the one saved copy. In this example, a 100 MB storage demand could be reduced to only 1 MB.

Need for Data Deduplication?

a In general, data deduplication improves data protection, increases the speed of service, and reduces costs.

a Lower storage space requirements will save money on disk expenditures.

a The more efficient use of disk space also allows for longer disk retention periods, which provides better recovery time objectives (RTO) for a longer time and reduces the need for tape backups.

a Data deduplication also reduces the data that must be sent across a WAN for remote backups,replication, and disaster recovery.

How Data Deduplication works?

a Data deduplication can generally operate at the file, block, and even the bit level.

a File deduplication eliminates duplicate files (as in the example above), but this is not a very efficient means of deduplication.

a Block and bit deduplication looks within a file and saves unique iterations of each block or bit.

Each chunk of data is processed using a hash algorithm such as MD5 or SHA-1. This process generates a unique number for each piece which is then stored in an index. If a file is updated, only the changed data is saved. That is, if only a few bytes of a document or presentation are changed, only the changed blocks or bytes are saved, the changes don't constitute an entirely new file. This behavior makes block and bit deduplication far more efficient. However, block and bit deduplication take more processing power and uses a much larger index to track the individual pieces.

Deduplication in NetApp Environment

Netapp has implemented their deduplication function at fixed block (4kb) level which gives more space saving and works very efficiently. As it works on block level irrespective of file type or data format hence you can dedupe any type of file on either cifs or nfs even you can dedupe a lun of any size no matter where they are written in a volume.

This picture gives a high level overview of data before and after deduplication process. Here all similar colours of boxes denotes block with similar data and before deduplication process all the duplicate blocks were written to the different areas of hard disk so once the deduplication process runs it will identifies all duplicate blocks and removes them so only unique blocks of data is on volume.

As stated before deduplication process runs on storage level hence no configuration is required on application side and they keep accessing the data as before. While system creates the fingerprint in the process of writing of new data there's a negligible performance impact on your system, however if your filer is heavily utilized and it's constantly above 50% utilization then a performance impact will be an average of 15%.

Under the hood

Whenever new data is written on a flexvol which is having asis on (NetApp term of deduplication) OnTap creates a fingerprint for every block of data it writes for comparison. At this moment system writes all the data as any other system except recording some extra information for your data i.e. fingerprint for every block. Now either you have to start the deduplication process manually or schedule it to run on a specific time. Once the deduplication process is started then fingerprints are checked for duplicates and, when found, first a byte-by-byte comparison of the blocks is done to make sure that the blocks are indeed identical, and if they are found to be identical, the block's pointer is updated to the already existing data block and the new (duplicate) data block is released.

The maximum sharing for a block is 255. This means, for example, that if there are 500 duplicate blocks, deduplication would reduce that to only 2 blocks. Also note that this ability to share blocks is different from the ability to keep 255 Snapshot copies for a volume.

Deduplication in EMC Environment

This is just a high level overview of deduplication in EMC as they have added dedupe function only in January 2009 to their Celerra range of products.

EMC has deployed deduplication on their newer Celerra models on file level in conjunction with compression technology. As it's on file level hence the duplication is very fast however it gives very small savings because two or more files should be identical to be a dedupe candidate. Compression technology used by EMC gives additional level of space saving as it uses spare CPU cycles from the system hence you don't have to invest money on expensive specialized compression products. However even after having deduplication working with compression it gives less storage savings compare to NetApp Fixed block level deduplication technology. How? Here's the detail

a As it's on file level and files needs to be an exact match for deduplication hence vmdk files and any luns made on the storage are not de-duplicated.

a It targets only infrequently accessed files as compressing active files is not a good idea

a By default any files more than 200mb size is left untouched

a Compression works only on file size more than 24K

a It disables MPFS

a Has a performance impact in reading deduped and compressed files

Functionality	Celerra Data Deduplication	NetApp Data Deduplication
User interface	GUI-Simple graphical user interface, a one-click operation to enable	CLI only; cryptic commands with limited flexibility
Compression	Additional space savings on duplicate PLUS unique data	NetApp does not offer compression; this makes EMC more efficient in saving space
Unlimited file system size	EMC supports a 16 TB file system size across the entire Celerra unified storage series (NX4 through the NS-960)	NetApp limits the file system size based on the Filer model; the FAS2020 supports only 1 TB to a maximum of 16 TB for the FAS6080
Integrated with snaps	Celerra snaps do not negatively affect deduplication space savings in production file systems. Space savings can be realized immediately	NetApp will not achieve space savings from deduplication on any data that is currently part of a snapshot

How to check part number of installed adapter in Ontap

There are number of situations when you want to check part number of installed PCI adapters in your NetApp FAS or V-series system, how do you do it?

Well whatever way you do there’s a simple way and undocumented also (atleast in their man pages) “sysconfig -ca”.
Just run the command and it will give you part number of all the pci adapters installed as well checks if they are in appropriate slot.

Here’s the sample output

XXYY> sysconfig -ca
sysconfig: slot 3 OK: X3147: NetApp NVRAM6 512MB
sysconfig: slot 2 OK: X1049A: PCI-E Quad 10/100/1000 Ethernet G20
sysconfig: slot 1 OK: X2054B: LSI 949E; PCI-E quad-port Fibre Channel (LSI7404EP)
sysconfig: Unless directed by NetApp Global Services volumes root,  should have the volume option create_ucode set to On.

Which is faster, NDMPcopy or vol copy?

Only if I have to count speed then vol copy, because it copies blocks directly from disk without going through FS, however I think it’s well suitable if you want to migrate a volume.

Pros

• CPU usage can be throttled
• Source volume snapshot can be copied
• Simultaneously 4 copy operations can be started
• Once started it goes to background and you can use console for other purpose 

Cons

• Destination can’t be root volume
• Destination volume should be offline
• All data in destination volume will be over-written
• Destination volume size should be bigger or equal to source
• Single file or directory cannot be specified for copy operation
• Both the volumes should be of same type; traditional or flexible
• If data is copied between two filers both filer should have other filer’s entry in /etc/hosts.equiv file and loopback address for itself in /etc/hosts file 

However for copying data between two filers for test or any other purpose ndmpcopy is more suitable because it gives you additional control and less restrictions, which is very useful.

Pros

• Little or no CPU overhead
• Incremental copy is supported
• No limitation on volume size and type
• No need to take destination volume offline
• Single file or directory can also be specified
• No file fragmentation on destination volume as all data is copied sequentially from source volume so improved data layout
• No configuration is required between two filers and username and password is used for authentication

Cons

• Snapshots can’t be copied from source
• Console is not available till the time copy operation is running so no multiple ndmpcopy operations
• If lots of small files has to be copied then copy operation will be slower 

So as you have seen both are well however one can’t be replaced for other and both have their usage for different purposes.

How to check unplanned downtime detail for a NetApp filer

Every now and then someone ask us what is uptime of system and we just type 'uptime' on system console to get the detail instantly.


This is really handy command to know when the system was last rebooted and how many operations per protocol it has served since then. Wouldn't our life be little easy if managers get satisfy with this detail? Alas! but that doesn't happen and they ask us to give all the details since we have acquired the system or 1st January and then we go back to our excel sheet or ppt we have created as part of monthly report to pull the data.


How about if we can get same information from system with just a command, wouldn't that be cool. Fortunate enough we have little known command 'availtime' right inside Ontap which just do the exact same function and specifically created after thinking about our bosses.


HOST02*> availtime fullService statistics as of Sat Aug 28 18:07:33 BST 2010 System  (UP). First recorded 68824252 secs ago on Mon Jun 23 04:16:41 BST 2008         Planned   downs 31, downtime 6781737 secs, longest 6771328, Tue Sep  9 15:07:33 BST 2008         Uptime counting unplanned downtime: 100.00%; counting total downtime:  90.14% NFS     (UP). First recorded 68824242 secs ago on Mon Jun 23 04:16:51 BST 2008         Planned   downs 43, downtime 6849318 secs, longest 6839978, Wed Sep 10 10:11:43 BST 2008         Uptime counting unplanned downtime: 100.00%; counting total downtime:  90.04% CIFS    (UP). First recorded 61969859 secs ago on Wed Sep 10 12:16:34 BST 2008         Planned   downs 35, downtime 17166 secs, longest 7351, Thu Jul 30 13:52:25 BST 2009         Uptime counting unplanned downtime: 100.00%; counting total downtime:  99.97% HTTP    (UP). First recorded 47876362 secs ago on Fri Feb 20 14:08:11 GMT 2009         Planned   downs 8, downtime 235 secs, longest 53, Wed Jan 20 14:10:18 GMT 2010         Unplanned downs 16, downtime 4915 secs, longest 3800, Mon Jul 27 16:01:02 BST 2009         Uptime counting unplanned downtime:  99.98%; counting total downtime:  99.98% FCP     (DOWN). First recorded 68817797 secs ago on Mon Jun 23 06:04:16 BST 2008         Planned   downs 17, downtime 44988443 secs, longest 38209631, Sat Aug 28 18:07:33 BST 2010         Unplanned downs 6, downtime 78 secs, longest 21, Fri Feb 20 15:24:44 GMT 2009         Uptime counting unplanned downtime:  99.99%; counting total downtime:  34.62% iSCSI   (DOWN). First recorded 61970687 secs ago on Wed Sep 10 12:02:46 BST 2008         Planned   downs 21, downtime 38211244 secs, longest 36389556, Sat Aug 28 18:07:33 BST 2010         Uptime counting unplanned downtime: 100.00%; counting total downtime:  38.33% 


I am not sure why NetApp has kept this command in Advanced mode but once you know this command I bet next time you will not refrain yourself going inside advance mode to see how many unscheduled downtime you had since last reset.


A shorter version of this command is just 'availtime' it also shows the same information as 'availtime full' however it truncates letters from output and denotes  Planned with P and Unplanned with U which is very good if you want to pass it in script. 

HOST04*> availtimeService statistics as of Sat Aug 28 18:07:33 BST 2010 System  (UP). First recorded (20667804) on Wed Sep 23 09:35:49 GMT 2009         P  5, 496, 139, Fri Dec 11 15:58:19 GMT 2009         U  1, 1605, 1605, Wed Mar 31 17:01:41 GMT 2010 CIFS    (UP). First recorded (20666589) on Wed Sep 23 09:56:04 GMT 2009         P  7, 825, 646, Thu Jan 21 19:08:03 GMT 2010         U  1, 77, 77, Wed Mar 31 16:34:54 GMT 2010 HTTP    (UP). First recorded (20664731) on Wed Sep 23 10:27:02 GMT 2009         P  3, 51, 22, Thu Jan 21 19:17:25 GMT 2010         U  4, 203, 96, Thu Jan 21 19:08:03 GMT 2010 FCP     (UP). First recorded (20477735) on Fri Sep 25 14:23:38 GMT 2009         P  3, 126, 92, Thu Jan 21 19:07:57 GMT 2010         U  4, 108, 76, Wed Mar 31 16:34:53 GMT 2010

In order to reset the output use 'reset' switch and it will zero out all the counters, make sure you have recorded the statistics before you reset the counters as once you reset the counters you will not be able to get details of system uptime since system was built so you may like to do only after you acquire a new system, have done all the configuration and now it's the time for it to serve user requests.

HA Configuration Checker (ha-config-check.cgi)

The HA Configuration Checker is a Perl script that detects errors in the configuration of a pair of NetApp HA (active-active) storage controllers. It will run as a command from a Unix shell or Windows prompt, but also doubles as a CGI script that can be executed by a Unix web server. The script uses rsh or ssh to communicate with the storage controllers you're checking, so you'll need to have the appropriate permissions for rsh to run on both storage controllers in the HA pair.

 If no /etc/hosts.equiv entry exists for the host where you tring to run, then the username and password must be provided to the script.

D:\>ha-config-check.exe -l filer1 filer2
filer1 rsh login: bali
Password: ********
filer2 rsh login: bali
Password: ********

Output would be..

== NetApp HA Configuration Checker v2.0.0 ==

Checking rsh logins. rsh filer1 -l bali:******** version

Checking rsh logins. rsh filer2 -l bali:******** version
OK
Checking Data ONTAP versions...
OK
Checking licenses...
OK
Checking HA configuration identity...
OK
Checking cf status...
OK
Checking fcp cfmode settings...
fcp: FCP is not licensed.
N/A
Checking options...
Option timed.sched                  1h
 on filer2 has no match on filer1
Option timed.sched                  hourly
 on filer1 has no match on filer2
HA configuration issue(s) found above. Please correct them and rerun this script
.
Done.

Download the tool from below NOW link

http://support.netapp.com/NOW/download/tools/cf_config_check

HA GROUP ERROR: DISK/SHELF COUNT MISMATCH ERROR

When a disk enters maintenance center on one node (There will be lot of reason behind this (One reason is it’s not a valid disk to use it)), the partner may not be aware of this. This may result in one node excluding the disk in its disk inventory, while the partner including the same disk in its inventory. This will result in generating "cf.disk.inventory.mismatch" and "CLUSTER ERROR: DISK/SHELF COUNT MISMATCH" auto support.

If the disk in the maintenance center is repaired after testing, the disk inventory mismatch will be automatically resolved. If the disk in the maintenance center is failed after testing, the disk should be removed.


filer1> disk show 1b.57
DISK OWNER POOL SERIAL NUMBER HOME
------------ ------------- ----- ------------- -------------
1b.57 filer2 (151704668) Pool0 3SK0AFTX0000902778L9 filer2 (151704668)
whmn3849*>
filer2> disk show 1b.57
DISK OWNER POOL SERIAL NUMBER HOME
------------ ------------- ----- ------------- -------------
1b.57 filer2 (151704668) FAILED 3SK0AFTX0000902778L9 filer2 (151704668)
Filer2> 

From syslog (/etc/messages)

Fri Jul  6 23:02:24 EDT [filer1: raid.disk.missing:info]: Disk 0b.57 Shelf 3 Bay 9 [NETAPP   X291_S15K7420F15 NA00] S/N [3SK0AFTX0000902778L9] is missing from the system

Fri Jul  6 18:52:10 EDT [filer1: cf.disk.inventory.mismatch:CRITICAL]: Status of the disk 1b.57 (20000024:B6564A61:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000) has recently changed or the node (filer2) is missing the disk.

But after sometime it will completely go to failed pool if ONTAB failed to rectify the soft error on that disk if not it will back to inventory stating that soft error has been fixed. See below message from syslog.

Fri Jul  6 20:41:11 EDT [filer1: cf.disk.inventory.mismatchOK:info]: The node (filer2) included the disk 0b.57 (20000024:B6564A61:00000000:00000000:00000000:00000000:00000000:00000000:00000000:00000000) in its inventory.