NetApp Reallocate Volumes and Aggregates

When adding disks to a NetApp array it is usually to add prefomance and capacity. Unless you reallocate the WAFL system will actually not preform much better unless you reallocate your volumes and drives over the newly increased disk space on your aggregate. Otherwise WAFL will only use the new disks until they are as saturated as the existing disk in an attempt to level out the disks. If you only added a couple of disks this can lead to hot spots and lower then expected IO. So reallocating effectively tells wafl to move some of the existing data to the new disks and frees up space equally over all disks.

So here are a few helpfull cammands that you will need to complete this simple optimization task. Depending on the size of your existing volumes and the number of disks you have added this can take some time.

You must be in priv advanced to complete this command:

FILER1> priv set advanced
FILER1*>

You need to run the following on each volume within the aggregate before your run this on the aggregate, and this will take some time depending on the size of the data and the numebr of new disks.

FILER1*> reallocate start -f -p /vol/volumename

Here is the command to check the Reallocation status, you can do only one volume or aggr at a time so by using the following command you can see if it is time to move on to the next reallocate

FILER1*> reallocate status -v

OUTPUT:

Reallocation scans are on
/vol/volumename:
        State: Reallocating: Inode 677805, block 40384 of 10490701 (0%)
        Flags: doing_force,whole_vol,keep_vvbn
    Threshold: 4
     Schedule: n/a
     Interval: n/a
 Optimization: n/a

For snapmirror targets you will need to run the following to break the mirror, after the entire process is complete resync the mirror.

FILER1*> snapmirror break volumename
FILER1*> reallocate start -f -p /vol/volumename

For the aggregate reallocation you will run the following command only after running reallocate on all volumes within the aggregate.

FILER1*>reallocate start -A arrgname
FILER1*> reallocate status -v

Just remember on any volume that is a snapmirror target you will need to resync after this is complete. I usually do so in the filerview, but this can be done in the command line as well.

To get out of priv advanced:

FILER1*> priv set admin
FILER1>

Setting Up Jumbo Frames on a VMware ESX Hosts

Jumbo Frames can be an important part of a IP Storage network, it reduces the overhead for a TCP/IP packet by increasing the MTU from 1500 to 9000 per packet. Now there are some strict requirements you need to follow to make this happen, or you will have MTU miss match errors that will actually slow down your storage network instead of speeding it up.

First Make sure you configure your SWITCH VLAN that will handle the traffic to use jumbo frames. For example on a HP Procurve from the configuration prompt:

vlan 30
jumbo

Second make sure that you configure a an interface on the Storage device that is also on this VLAN to use jumbo frames to communicate with your Host device. This depends on your storage vendor, but reading the documentation you should figure it out quickly.

Third configure all your host devices to communicate with your storage device via jumbo frames on the same vlan. Now I keep saying “same VLAN” for a very good reason, as you will quickly run into the dredded MTU missmatch errors if you try to route communications between VLANs. If you need to route to other devices or a device of a WAN that are not using jumbo frames then you must use a dedicated interface to communicate with the other device with MTU 1500. This interface does not have to be physical, on NetApp this can just be a VIF without -9000 configured on it.

Now that you have soem background on Jumbo frames here is how to enable them on a vSwitch on a VMware ESX host.

I created the VMware Networks port group on vSwitch2 called IPStorage before running the following CLI commands on the ESX host

First in the vCenter or Network Configuration add your vSwitch and setup a portgroup, in this example I used vSwitch2 and called the port group IPStorage. Once this is setup use put or login to the console of your ESX host.

Prep the vSwitch with the following command subsitute vSwitch2 with the vSwitch you wish to target

esxcfg-vswitch -m 9000 vSwitch2

now configure the portgroup as you named it earlier and assign it an IP at the same time

esxcfg-vmknic -a -i 10.10.2.10 -n 255.255.255.0 -m 9000 IPStorage

Finally test your configuration by trying to ping your storage interface with a jumbo frame packet:

vmkping -s 9000 10.10.2.101

If your ping fails make sure that your storage interface is correctly configured and on the same VLAN and subnet, also ensure that you have Jumbo Frames enabled the switch in the VLAN you are using. Also make sure that when you setup your portgroup in the VMware GUI that you input the VLAN number if you are using tagged vlans on the port.

Deduplication on NetApp

I found that there are always multiple realities to deal with when looking at new technologies. There are the excited implementors, who make it sound good. There are the sales people who say what you want to hear, and thier are the Technical Reports that tell you what it really does. I’m just picking on NetApp in this post, but this is true with any new technology, so while you might learn a lot of good stuff from your sales friends make sure to ask for links to the white papers or TRs in NetApps case.

Deduplication is a great benefit for storage admins out there today, it helps us to reclaim some of that expensive and ultimately wasted space that may be on our storage environments. The largest impact I’ve seen in in virtual environments with multiple VM’s with the same OS. The can bring savings of up to 50% or more. On the other hand CIFS shares you may only see savings of maybe 10%. There are some limitations and planning that should go into this. One being performance hit during the dedupe operation on some applications, the other being size limitations of your volume. So in cases where you want to more performance or space and still dedupe you may need to look into a different NetApp Model, newer version of DataOnTap, or even a PAM card. If these all do not meet your needs or cost to much then we may have to pass on the dedupe for that volume.

Below are just some highlights I found useful, but please read the TR from NetApp as this may change over time.

Benefits:

  • Saving space on your storage network as much as 50%
  • Three easy ways to dedupe, schedule, auto, and manual
  • Deduplication does wonders for large virtual VMDK files, and just large files in general on NFS shares
  • With NFS the savings is passed onto vSphere just remember to refresh the volume datastore stats

Limitations:

  • Limited Volume size based on Data OnTap version and NetApp model.
  • Snapshots taken before deduplication are locked, and take up aditional space, this is not true for Snaps taken after a dedupe job.
  • Some types of files just do not dedupe that well especially the really small ones, and images.
  • See the TR referaced below for more information

Reference:

http://media.netapp.com/documents/tr-3505.pdf

NetApp CIFS share Access Based Enumeration

To get down to the point quickly here is the comand to enable access based enumeration on a share:

replace sharename with the name of the share you want to apply this to

cifs shares -change sharename -accessbasedenum

What this does is to only show folders that a user has permission to see. This is a feature intruduced to windows in 2003 R2 that is something worth checking out.

Auditing Folders and Files in Windows and on NetApp Filer CIFS shares

Enabling file auditing is a 2-step process.

1. Configure “audit object access” in AD Group Policy or on the server’s local GPO.

This setting is located under:

Computer Configuration–>Windows Settings–>Security Settings–>Local Policies–>Audit Policies. Enable success/failure auditing for “Audit object access.”

 2.  Configure an audit entry on the specific folder(s) that you wish to audit.

Right-click on the folder–>Properties–>Advanced. From the Auditing tab, click Add, then enter the users/groups whom you wish to audit and what actions you wish to audit – auditing Full Control will create an audit entry every time anyone opens/changes/closes/deletes a file, or you can just audit for Delete operations.

After you’ve done both of these steps, any file deletions will show up in the Security log of the file server that hosts those files.

For NetApp Filers the steps continue as you need to export the audit logs in a format you can read using windows event viewer, you need to run the following command on your filer:

cifs audit save -f

After this the log is dumped into a folder on the fielr called /etc/log/adtlog.evt or \\filername\c$\etc\log\adtlog.evt

You can then copy this log from the filer to a central locaiton and view it in windows event viewer or other compatable utility.

One could create a script and schedule it to run the command on the filer, and move the file to a central location and rename it. It all depends on how you manage such logs. But the first two steps apply to the filer, as the filer will also have the GPO applied to it as well.

Note: If you would like to track actual changes to the file, you may want to check out a Subversion server of some sorts, there are a number of free ones out there. Also some programs have a built in change control such as MS Word.

NetApp vol copy (copying one volume to another)

This can be useful for transporting a volume to another aggregate on the same or even another filer. It can also be useful when having issues with the current volume.

the syntax is simply:

vol copy start src dst

If you wish to copy all snapshots then:

vol copy -S src dst

Example:

vol copy -S important_vol new_important_vol

using a lowercase -s will only copy one volume, -s snapshot_name will copy only that snapshot. For more detail try the MAN page.

Using Robocopy to Migrate Windows Shares to NetApp

Recently at work we installed a new NetApp FAS2020 and after playing with CIFs share found that it would be great, but one problem held us back, file permissions on thousands of files, as well as Modify Date/Time information would be lost in a simple file copy. Not so with robocopy. The “/copyall” switch ensures that all file info is copied including permissions (ACLs). There are many options to execute robocopy. I ran the following in syntax in a bat file which output a log file. I ran one of these for each share, and was able to look through the log file to see if any files were missed.

robocopy "\\UNC\PATH\To\SOURCE" "\\UNC\PATH\TO\DESTINATION\"  /E /copyall /R:1 /W:3 /v /log:C:\log.txt

Reference:

http://www.mydigitallife.info/2007/05/07/robocopy-syntax-command-line-switches-and-examples/