Showing posts with label Netapp. Show all posts
Showing posts with label Netapp. Show all posts

Friday, November 4, 2011

Netapp Snapmirror & periodic updates setup

Setting up snapmirror between two Netapp devices is very easy and straightforward process as long as one has Snapmirror license. :) This is a 5 step process outlines below:

1) Make sure snapmirror license is installed on both source and destination filers.
2) Setup snapmirror options to allow both source/destination to do snapmirrors
3) Setup snapmirror and if needed, periodic updates/ data refresh
4) At the time of cut-over, do the final update - quiesce - break
5) Remove the extra snapshots created during this process.

In following example, trying to do snapmirror from filer01:/vol/vol01 to filer02:/vol/vol02.


# rsh filer02 options snapmirror
snapmirror.access host=*
snapmirror.checkip.enable off
snapmirror.delayed_acks.enable on
snapmirror.enable on
snapmirror.log.enable on
snapmirror.vbn_log_enable off (value might be overwritten in takeover)
#
#
# rsh filer02 snapmirror initialize -S filer01:vol01 filer02:vol02
Transfer started.
Monitor progress with 'snapmirror status' or the snapmirror log.
#
#
# rsh filer02 snapmirror status
Snapmirror is on.
Source Destination State Lag Status
filer01:vol01 filer02:vol02 Uninitialized - Transferring (180 MB done)
#
#
# rsh filer02 snapmirror status
Snapmirror is on.
Source Destination State Lag Status
filer01:vol01 filer02:vol02 Source - Idle
#
#
NOTE: At this stage, the snapmirror refresh can be setup using /etc/snapmirror.conf file. The file looks like below. In this example, data is getting refreshed every 0th minute of every hour:
#
#
# rsh filer02 rdfile /etc/snapmirror.conf
filer01:vol01 filer02:vol02 - 0 * * *
#
#

When cutover is needed, do the following:
1) Disable entries from /etc/snapmirror.conf
#
#
# rsh filer02 snapmirror quiesce filer02:vol02
#
#
# rsh filer02 snapmirror break filer02:vol02
#
#
At this stage, delete any extra snapshots which might not be needed anymore.
#
#

Friday, June 10, 2011

Netapp - CIFS How to

WARNING!!! Storage administration ALWAYS need expertise and know before you do mindset! So, be careful while doing any changes on storages as this may affect end users, production data and can be disastrous if not done right!!!

How to reload CIFS user's home directory settings?

Note: In netapp (ontap), user's home directory listing reside in /vol/vol0/etc/cifs_homedir.cfg file. For adding any new home directory location, simply add the volume/qtree in the said file and proceed as below:


[ADMINHOST] # rsh FILERNAME cifs
The following commands are available; for more information
type "cifs help "
access domaininfo nbalias shares
adupdate gpresult prefdc sidcache
audit gpupdate resetdc stat
broadcast help restart terminate
changefilerpwd homedir sessions testdc
comment lookup setup top

[ADMINHOST] # rsh FILERNAME cifs help homedir
Usage:
cifs homedir
cifs homedir load [-f]
cifs homedir showuser

[ADMINHOST] # rsh FILERNAME cifs homedir load

[ADMINHOST] # rsh FILERNAME cifs homedir
/vol/user1/users
/vol/user2/users
/vol/user3/users
/vol/user4/users

Thursday, May 20, 2010

Fast NDMP backups revealed :)

I use Netbackup 6.5 to backup Netapp filers via NDMP. The tape drives are SAN tapes which are also visible + shared to Netapps as well as Netbackup master server. Yes, I do use SSO license so that tape drives can be shared.

As my backup environment grown (around 150TB+, per week backup size), I started facing backup performance problems... like some of the backup jobs were running very fast (>70MBPS throughput on LTO4 tape drives) while there were some (about 30% of total policies) of jobs running at 1 MBPS to 5 MBPS. Since these jobs were running too slow (for a LTO4 drive), my backup window was not sufficient to backup entire backup size in designated backup window (even if I have 15 LTO4 drives). While troubleshooting for performance, I found some important things:

1) Backup performance depends upon file size in general (one factor out of many dependencies,,, like type of tape drives, backup medium.. SAN/LAN, etc etc.)
2) Smaller size of files ... slower the backup throughput
3) Large size of files... better/ faster the backup throughput

Now the thing is that I have to back it up in designated window at the same time exclude the business hours to impact any performance hit on filer due to NDMP job. Theoritically, NDMP is given least priority when user data is being processed by filer (be it http/NFS/CIFS for that matter).

Also, note that while doing a default NDMP backup, the filer sends all the file/directory information to the Netbackup catalog.----------> This was the turning point in this whole troubleshooting!!! Why?? Here is the explanation:

For smaller size of file, the filer would send the file/dir information to catalog and then (or same time) it will backup the file on tape. Now, if the time taken by catalog to register the file info is more or equivalent to that of time to backup the file on tape, the whole purpose of having a faster tape drive is nullified. And at the same time, the NDMP backup will be slow because catalog has to be updated with all the file information it is backing up.

Solution: The solution was to use "set HIST=n" option in each of the NDMP backup pollicies so that NDMP job do not need to update catalog what it is backing up. Result: Faster backups!!! While I got backups faster with this method, this has its own advantages and disadvantages. Here is the list:

Advantage:
1) Backup performance is significantly improved
2) Catalog size is now controller (because it has to capture less info)
3) Catalog backup size is significantly reduced
4) All backup jobs are giving at least 30-40 MBPS while faster jobs are still same

Disadvantage:
1) No file history is captured!!!
2) Meaning that if I have to restore a file, I will have to restore entire filesystem to do so
3) The size of restore volume must be greater or at least equal to largest volume which is getting backed up
4) Also, the number of inodes also must be equal or more in order to restore largest size volume in environment.

Enjoy! :)

Wednesday, August 26, 2009

How to change date & time in Solaris, Linux, HP-UX and Netapp?

Changing date/time in Solaris, Linux, HP-UX and Netapp is one command step as long as used with correct syntax.

For Solaris:

# date
Thu Aug 27 00:22:37 SGT 2009
# date 0827002509
Thu Aug 27 00:25:00 SGT 2009
# date
Thu Aug 27 00:25:02 SGT 2009


For Netapp:

# ssh filer01 date
Thu Aug 27 00:20:11 SGT 2009

# ssh filer01 date 0827002509
Error: 27 is not a valid month of the year. Range is 01-12.
usage: date [-u] [[[[CC]yy]mm]dd]hhmm[.ss]]
date [-u] -c
date [-f] -c initialize

# ssh filer01 date 0908270026
Thu Aug 27 00:26:00 SGT 2009

# ssh filer01 date
Thu Aug 27 00:26:03 SGT 2009

Tuesday, August 18, 2009

How to setup SSH (+ passwordless ssh) for a Netapp filer

It is a 3 step process listed below:

1) Setup secureadmin from filer
2) Enable ssh
3) Copy the ssh keys to the filer for the desired hosts.

In following example, I wanted to setup ssh for remote-mgr server (which is a Sun box) and filer01 is a Netapp filer.

This article assumes that you already know how ssh and passwordless ssh works. This just gives the outlines :)

Once again, if you are not sure what you are doing, consult NetApp support. Doing this process wrong can disable all the ssh logins!!! So, be careful.



[remote-mgr::root:/root]# ssh 192.168.15.200
ssh: connect to host 192.168.15.200 port 22: Connection refused

[remote-mgr::root:/root]# telnet 192.168.15.200
Trying 192.168.15.200...
Connected to 192.168.15.200.
Escape character is '^]'.

Password:
filer01>
filer01> secureadmin status
ssh2 - inactive
ssh1 - inactive
ssl - inactive

filer01> secureadmin setup ssh
SSH Setup
---------
Determining if SSH Setup has already been done before...no

SSH server supports both ssh1.x and ssh2.0 protocols.

SSH server needs two RSA keys to support ssh1.x protocol. The host key is
generated and saved to file /etc/sshd/ssh_host_key during setup. The server
key is re-generated every hour when SSH server is running.

SSH server needs a RSA host key and a DSA host key to support ssh2.0 protocol.
The host keys are generated and saved to /etc/sshd/ssh_host_rsa_key and
/etc/sshd/ssh_host_dsa_key files respectively during setup.

SSH Setup will now ask you for the sizes of the host and server keys.
For ssh1.0 protocol, key sizes must be between 384 and 2048 bits.
For ssh2.0 protocol, key sizes must be between 768 and 2048 bits.
The size of the host and server keys must differ by at least 128 bits.

Please enter the size of host key for ssh1.x protocol [768] :
Please enter the size of server key for ssh1.x protocol [512] :
Please enter the size of host keys for ssh2.0 protocol [768] :

You have specified these parameters:
host key size = 768 bits
server key size = 512 bits
host key size for ssh2.0 protocol = 768 bits
Is this correct? [yes]

Setup will now generate the host keys in the background. It will take a
few minutes. After Setup is finished you can start SSH server with
command 'secureadmin enable ssh'. A syslog message will be generated
when Setup is complete.

Tue Aug 18 21:52:37 GMT [rc:info]: SSH Setup: SSH Setup is done. Host keys are stored in /etc/sshd/ssh_host_key, /etc/sshd/ssh_host_rsa_key and /etc/sshd/ssh_host_dsa_key.
filer01>
filer01> Connection to 192.168.15.200 closed by foreign host.

[remote-mgr::root:/root]# mkdir /tmp/filer01
[remote-mgr::root:/root]# mount 192.168.15.200:/vol/vol0 /tmp/filer01

[remote-mgr::root:/root]# cd /tmp/filer01/etc/sshd
[remote-mgr::root:/tmp/filer01/etc/sshd]# ls -l
total 48
-rw------- 1 root root 534 Aug 19 05:52 ssh_host_dsa_key
-rw-r--r-- 1 root root 470 Aug 19 05:52 ssh_host_dsa_key.pub
-rw------- 1 root root 411 Aug 19 05:52 ssh_host_key
-rw-r--r-- 1 root root 249 Aug 19 05:52 ssh_host_key.pub
-rw------- 1 root root 688 Aug 19 05:52 ssh_host_rsa_key
-rw-r--r-- 1 root root 174 Aug 19 05:52 ssh_host_rsa_key.pub

[remote-mgr::root:/tmp/filer01/etc/sshd]# ssh 192.168.15.200 uptime
ssh: connect to host 192.168.15.200 port 22: Connection refused

[remote-mgr::root:/tmp/filer01/etc/sshd]# rsh 192.168.15.200 secureadmin status
ssh2 - inactive
ssh1 - inactive
ssl - inactive

[remote-mgr::root:/tmp/filer01/etc/sshd]# rsh 192.168.15.200 secureadmin enable ssh

[remote-mgr::root:/tmp/filer01/etc/sshd]# ssh 192.168.15.200 uptime
The authenticity of host '192.168.15.200 (192.168.15.200)' can't be established.
RSA key fingerprint is 9b:aa:c9:0e:92:ed:f7:98:60:9f:9d:0e:b1:65:96:ed.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.15.200' (RSA) to the list of known hosts.
root@192.168.15.200's password:
9:57pm up 36 mins, 12 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 DAFS ops, 0 FCP ops, 0 iSCSI ops

[remote-mgr::root:/tmp/filer01/etc/sshd]# mkdir -p root/.ssh
[remote-mgr::root:/tmp/filer01/etc/sshd]# mkdir .ssh
[remote-mgr::root:/tmp/filer01/etc/sshd]# ls -l
total 56
drwxr-xr-x 2 root root 4096 Aug 19 05:58 root
-rw------- 1 root root 534 Aug 19 05:52 ssh_host_dsa_key
-rw-r--r-- 1 root root 470 Aug 19 05:52 ssh_host_dsa_key.pub
-rw------- 1 root root 411 Aug 19 05:52 ssh_host_key
-rw-r--r-- 1 root root 249 Aug 19 05:52 ssh_host_key.pub
-rw------- 1 root root 688 Aug 19 05:52 ssh_host_rsa_key
-rw-r--r-- 1 root root 174 Aug 19 05:52 ssh_host_rsa_key.pub
[remote-mgr::root:/tmp/filer01/etc/sshd]# cd root/
[remote-mgr::root:/tmp/filer01/etc/sshd/root]# ls -la
total 24
drwxr-xr-x 3 root root 4096 Aug 19 05:58 .
drwxr-xr-x 3 root root 4096 Aug 19 05:58 ..
drwxr-xr-x 2 root root 4096 Aug 19 05:58 .ssh
[remote-mgr::root:/tmp/filer01/etc/sshd/root]#
[remote-mgr::root:/tmp/filer01/etc/sshd/root]# cd .ssh/
[remote-mgr::root:/tmp/filer01/etc/sshd/root/.ssh]# ls -la
total 16
drwxr-xr-x 2 root root 4096 Aug 19 05:58 .
drwxr-xr-x 3 root root 4096 Aug 19 05:58 ..

[remote-mgr::root:/tmp/filer01/etc/sshd/root/.ssh]# cat /root/.ssh/id_dsa.pub >> /tmp/filer01/etc/sshd/root/.ssh/authorized_keys2

[remote-mgr::root:/tmp/filer01/etc/sshd/root/.ssh]# ssh 192.168.15.200 uptime
10:00pm up 39 mins, 58 NFS ops, 0 CIFS ops, 0 HTTP ops, 0 DAFS ops, 0 FCP ops, 0 iSCSI ops

[remote-mgr::root:/tmp/filer01/etc/sshd/root/.ssh]# ssh 192.168.15.200
filer01>
filer01>
filer01>
filer01> Connection to 192.168.15.200 closed by remote host.
Connection to 192.168.15.200 closed.

Thursday, May 28, 2009

Netapp's CPU utilization monitoring - the correct way :-)

I was configuring a filer's NDMP backup via a SAN drive and there I came to know the importance of -m option of sysstat command in Netapp's OnTap. During a backup test, I wanted to see the performance of my filer specifically the Tapes I/O. When I saw the output of "sysstat 5", I was little upset with the CPU utilization because it was showing unusually high CPU utilization.

But when I saw the "sysstat -m 5" command's output, which in fact shows each Netapp's CPU's utilization, it was clear that only 1 CPU was maxing out because of backup job leaving others untouched as shown in second command's output.

Therefore, conclustion is, whenever need to see actual CPU utilization in netapp, rely on -m option than any other one.


[adminhost]# rsh filername sysstat 5
CPU NFS CIFS HTTP Net kB/s Disk kB/s Tape kB/s Cache
in out read write read write age
64% 0 0 0 25 764 8180 6 0 4204 >60
69% 1 0 0 27 760 8050 51 0 4132 >60
61% 0 0 0 24 769 7998 400 0 3923 >60
70% 0 0 0 28 828 8604 10 0 4512 >60
62% 1 0 0 25 729 8246 415 0 4022 >60
35% 1 0 0 14 409 4407 11 0 2287 >60
68% 0 0 0 27 822 8731 391 0 4307 >60
62% 0 0 0 24 704 8014 11 0 4183 >60
65% 0 0 0 25 761 8140 396 0 3977 >60


[adminhost]# rsh filername sysstat -m 2
ANY AVG CPU0 CPU1 CPU2 CPU3
100% 32% 13% 9% 14% 92%
75% 22% 8% 5% 7% 69%
80% 24% 7% 6% 8% 74%
73% 22% 7% 6% 8% 67%
75% 22% 7% 6% 7% 69%
70% 21% 6% 6% 6% 65%
69% 21% 8% 6% 6% 63%
76% 23% 8% 8% 9% 69%
75% 23% 8% 6% 8% 71%
62% 18% 5% 5% 5% 56%
79% 24% 9% 6% 9% 72%
76% 23% 9% 6% 9% 70%

Thursday, September 4, 2008

Netapp frequently used commands

Getting Help

filer_name> ? # Shows all available commands
filer_name> help nfs # Shows syntax of nfs and available options
filer_name> man nfs # Shows the man page of 'nfs' command (unix style)

Checking services

filer_name> nfs status # Check if NFS is enabled
filer_name> nis info # Check if NIS is enabled and shows information
filer_name> secureadmin status # Check if SSH is enabled
filer_name> cifs status # Check if CIFS is enabled
filer_name> dns info # Check if DNS is enabled and shows information
filer_name> iscsi status # Check if ISCSI service is enabled and shows info
filer_name> ndmpd status # Check if NDMP is enabled and shows info
filer_name> rlm status # Check if RLM (remote LAN module) is enabled
filer_name> snapmirror status # Check if Snapmirror service is enabled
filer_name> snapvault status # Check if Snapvault service is enabled