Using OpenIndiana/Nexenta/Solaris with ZFS, COMSTAR, Microsoft iSCSI initiator, and LUN Masking

When I was first getting started with COMSTAR I tried feverishly to find an example of how to mask LUNs from iSCSI clients. The ultimate goal in LUN masking of course, is to keep someone or even yourself from mounting the wrong file system.  When this happens, the results can be devastating.

Jason's First Rule: Don't screw it up

At the time, the COMSTAR documentation seemed sparse and my understanding of stmfadm(8) was admittedly very weak. I thought it would be nice to share an example of how to mask LUNs so others can learn from my pain.

In this example I am going to use a Windows XP system is the client. Aside from being the most widely deployed operating system on the planet, it also serves as a good reminder that you can make iSCSI work on just about any OS.  The beautiful thing about iSCSI is it is cheap and it works on network equipment you already own. If that isn’t enough to make you consider it, remember that iSCSI initiators are generally free.

Companies used to charge for iSCSI targets, but those are free now too (for the most part). Any way, let’s start by downloading the Microsoft initiator. Other free initiators are available as well, such as the Starwind/Rocket Division initiator. The Starwind initiator is reportedly faster — I will test some initiators later and write up a review. That said, I was able to saturate gigE with my ZFS server consisting of (at the time) 8 drives and two SSDs using IOZone.

Installing the MS initiator is simple. Download it. Run it. Select all the components and agree to the license agreement. It will install in a few seconds and the system will be ready to go — no reboot required.

Now is a good time to setup our COMSTAR config and ZFS layout.
I will assume you have OpenIndiana and have not installed COMSTAR yet.

root@caprica:~# pkg install storage-server
 Packages to install:    19
 Create boot environment:    No
 Services to restart:     1
DOWNLOAD                                  PKGS       FILES    XFER (MB)
Completed                                19/19     687/687    48.6/48.6

PHASE                                        ACTIONS
Install Phase                              1399/1399

PHASE                                          ITEMS
Package State Update Phase                     19/19
Image State Update Phase                         2/2

Now we create a ZFS zvol of 1.99GB (pick your own size with LZJB
compression enabled. If you don’t want compression, omit the
“-o compression=lzjb” portion of the command.

My intention was to make this 1.99TB but i accidently used M
instead of G and the rest is history.

root@caprica:~# zfs create -V 1999M -o compression=lzjb data/iscsi/cylon/data000

Another neat trick you can employ is to create a sparsely populated zvol.
This has the benefit of telling the client that the volume is a particular
size without actually reserving the storage in advance. In other words, you
can assign much more storage than you actually have. This allows you to
provision multiple zvols of a large size initially and give you time to
back fill the actual data store at a later date. You do it with the “-s”
option. Like this:

root@caprica:~# zfs create -s -V 1999T -o compression=lzjb data/iscsi/cylon/data000

This command creates a 1.999 Petabyte volume from the clients perspective.
Remeber that 32bit XP can only see volumes less than 2TB. Also, at the end
of the day, if your client tries to allocate more than the available physical
storage, the write operation will fail. That is a bad thing for a database so
be careful.

root@caprica:~# sbdadm create-lu /dev/zvol/rdsk/data/iscsi/cylon/data000
Created the following LU:

GUID                    DATA SIZE           SOURCE
--------------------------------  -------------------  ----------------
600144f0080027b4face4db911710002  2096103424           /dev/zvol/rdsk/data/iscsi/cylon/data000

In the following line please note that the hostname of my client is ‘cylon.’
The initiator name here must match what is configured in the client or the
mask will prevent your system from mounting the LUN.

 Add Host to Host Group




root@caprica:~# stmfadm add-hg-member -g cylon iqn.1991-05.com.microsoft:cylon
-----------> IGN configured on the client side ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

If you don’t already have an iSCSI target on your system, you will need
to create one. If you already have a target, you may omit this step.

root@caprica:~# itadm create-target
Target iqn.1986-03.com.sun:02:8da9779e-ee50-6287-d7da-97b2928928ea successfully created

<table><tr><td align=center><strong>Add Target Group(s)</strong></td></tr></table>

stmf has to be disabled to add target groups.

root@caprica:~# svcadm disable stmf
root@caprica:~# stmfadm add-tg-member -g cylon iqn.1986-03.com.sun:02:8da9779e-ee50-6287-d7da-97b2928928ea
root@caprica:~# svcadm enable stmf

Finally, we add the view to the logical unit. This will expose the
LUN to the network.

root@caprica:~# stmfadm add-view -t cylon -h cylon 600144f0080027b4face4db911710002

Voila!

With our COMSTAR config in place, we are ready to configure the client side.

A handy icon will be on the desktop. Double click it. A nice window pops up. Microsoft uses a predictable format for their IGNs. It is basically, ign.1991-95.com.microsoft.<your hostname>, however, you are free to change it to whatever you like — but it must match the server side configuration.

iSCSI initiator

Next click the Discovery tab. Then click ‘Add’ and fill in the IP address or DNS name of your target.

iSCSI Discovery Screen Shot

Now click on the targets tab. The target should appear here. In this case, the target is identified as 8da9779e-ee50-6287-d7da-97b2928928ea. Select that target and click Logon and then click OK.

Now we need to configure the disk. Click Start->Run-> then type diskmgmt.msc
The windows disk manager software will pop up. The next steps include initializing the disk, partitioning the disk, and formatting it.

There is one serious limitation. Standard Windows XP is 32 bit and may only address LUNs less than 2TB in size. This is important because ZFS will allow you create a LUN of virtually any size. A Zettabyte is big

At this point, XP will normally have mounted the storage on its own. However, if you have made some alterations after you clicked ‘Logon’ you may need to ‘Rescan Disks’ which can be done by clicking Actions->Rescan Disks. If you have done your work correctly you will now have a new raw disk appear in the disk management application. At this point you may partition it to your liking and format it.

Remember, you never want to ‘optimize’ a volume mounted from a SAN/NAS. The best case scenario is your gear will not go slower. The worst case scenario is you have recreate the LUN and migrate the data. It is just a really really bad idea.