Virtualization Space

How I converted to paravirtual SCSI controllers from LSI

Brent Quick — Wed, 30 Mar 2011 13:47:43 +0000

Looking at having moved from vSphere 4.0 to 4.1 I was interested in what knobs I might turn to get more performance out of my vm environment which consists of the following IBM x3650, x3650 M3, and DS3400 FC SAN with 15K SAS drives. This led me to look at the higher performance paravirtual SCSI controllers instead of the existing LSI controllers. Enhanced in update 1 to support boot disks it seemed like time to give them a try, but how to convert my existing LSI controllers to paravirtual?First a little background on the paravirtual SCSI controller from VMware KB article 1010398 which discusses how to add a paravirtual controller to an existing build.

PVSCSI adapters are high-performance storage adapters that can result in greater throughput and lower CPU utilization. PVSCSI adapters are best suited for environments, especially SAN environments, where hardware or applications drive a very high amount of I/O throughput. PVSCSI adapters are not suited for DAS environments.

Problem with that KB is that I do not want to add another controller but simply convert my existing. So after a couple of BSOD’s and some Safe Mode OS boots, I figured out the trick. (Here is Alan Renouf’s Xtravirt article on how to do it which I found after the fact when researching for this post)

Add an additional disk – say 10 MB – but make sure that the virtual device node is 1:0 or greater so that VMware will add a new controller. If you do not have a floppy device attached to your vm add that as well and set the device type to a floppy image in the data store. Once the controller is added set that controller to be a paravirtual type and boot the vm. Windows should find the device and add the driver but if not then perform normal steps to add it. Once confirmed as being loaded (device manager) you can shut down the vm. Now you are in the home stretch, simple remove the other hard drive (which removes the other controller) and switch the existing LSI to paravirtual (click on the controller and then the change type button) and boot the vm. If all things operate as they should you should see your server boot normally.

Note: This is for Windows only since I am not running any flavors of Linux, but the KB article does cover several distro’s.

Recover from damaged partition table in ESX 4.1 (and prior)

Brent Quick — Tue, 07 Dec 2010 15:31:15 +0000

So I added 6 (RAID 5 w/hot spare) disks to my IBM DS3400 (six existing in RAID 5 w/hot spare) and started carving some new LUN’s. The purpose is so that my VM’s would be on dedicated LUN’s versus the big LUN with multiple machines. As a vSphere Standard user I do not have Storage vMotion, so I was shutting down the VM’s to move them when disaster struck. By disaster, I mean that all of a sudden the 300 GB LUN with the three VM’s that I needed to migrate was unavailable in vCenter (I wish I had a screen shot but I was too busy trying to resolve the problem) but I could still see it in the SAN manager software. After checking on the zoning and determining that there was no reason from the SAN that the LUN was not available, I went ahead and did a rescan/refresh on the HBA’s and prayed. No joy in Mudville for the mighty Casey struck out.I could see the LUN in vCenter but it wanted to lay down a new partition table (key to resolution) and I knew that would overwrite my VM’s. At this point I am thinking a SAN fault or problem so I open a ticket with IBM and send diagnostic logs and spend a few hours working with them – thanks for hanging with me. The conclusion was that there was no event on the SAN that caused the problem and no evidence of a SAN event that is correlated to this type of problem. At this point I had restored the VM’s from backup (PHD Virtual Backup 4.0-5) and let users know that they may see some “funky” stuff as a result. Now it is about determining what the problem was and how to prevent it from happening again.

Opening a ticket with VMware and submitting logs was next and once I got an engineer on the phone I got passed to the storage team after some initial discussion. Once the storage engineer got on the phone it was about 20 – 30 minutes to resolution of the problem and restoration of the LUN’s partition table (KB# 1002281) and the VM’s were “available” again. Apparently the first 1K block of the LUN was blanked causing the entire partition table to be corrupted. Now knowing what the problem was I asked how did it happen? The answer was that “Engineering cannot reliably reproduce the error.” or in simple terms no one knows. In discussing what operations I had the SAN/vCenter performing at the same time (initializing LUN, scanning HBA’s, and shutting down VM’s) nothing tripped any flag or warnings, except maybe not to do them all at the same time. Doing a Google search for “vmware lost partition table” turns up lots and lots of history of this problem without any cause (except maybe rescanning) or resolution.

So, long story short, watch your SAN and simultaneous operations, stay away from rescan if you can, and if a problem cannot be reproduced it cannot be solved. Not sure I buy any of that but maybe this helps someone else in the future.

As Paul Harvey would say, “Here is the rest of the story”

Brent Quick — Sat, 31 Jul 2010 02:46:31 +0000

So I was pretty psych’ed after getting the install of vCenter 4.1 on my new x64 host to finally work, so the next step was to get Update Manager working (not that I need it but it was previously installed and the vCenter client was looking for it on the old host. We after a very few setup items I was on to the install with my 32bit DSN setup already I thought it would be cake. We as it would turn out not so much. There is a known but unpublished issue involving the VIM_VCDB that produces the dreaded Error 25085. I say it is know since the VMware support person had it figured out in about an hour, most of which was spent bitching about 2008 not having everything in the same place as 2003. So in the end a series of 6 delete SQL queries later the install was done and patches were on the way. SQL listed after the break. I hope it helps someone else.

Use VIM_VCDB

delete from VPX_EXT_TYPE_IDS where EXT_ID = ‘com.vmware.vcIntegrity’

delete from VPX_EXT_PRIVS where EXT_ID = ‘com.vmware.vcIntegrity’

delete from VPX_EXT_CLIENT where EXT_ID = ‘com.vmware.vcIntegrity’

delete from VPX_EXT_SERVER_EMAIL where exists(select 1 from VPX_EXT_SERVER where VPX_EXT_SERVER.EXT_SERVER_ID = VPX_EXT_SERVER_EMAIL.EXT_SERVER_ID AND VPX_EXT_SERVER.EXT_ID = ‘com.vmware.vcIntegrity’)

delete from VPX_EXT_SERVER where EXT_ID = ‘com.vmware.vcIntegrity’

delete from VPX_EXT where EXT_ID = ‘com.vmware.vcIntegrity’

From x32 to x64 vCenter Server – Ouchies all the way!

Brent Quick — Wed, 28 Jul 2010 16:13:03 +0000

So with vSphere 4.1 you must run the vCenter server on a x64 OS. No big deal except you must migrate or do a fresh install since there is not a way to upgrade a Windows x32 to x64. No big deal except the default install is SQL Express which means no management tools so you cannot work with the existing DB objects or the new SQL instance. No big deal just download then SSMSEE making sure to get the x64 edition and install it locally, except you get a 29506 error when doing the install unless you are running under elevated privileges even if you are already an administrator. No problem just create a shortcut to the cmd.exe and “run as administrator”.

OK so now you have vSphere 4.oU2 setup on the new x64 OS, you have copied the old DB’s into place, the services are running, you can connect to you hosts and assume control, everything looks good except you still have that ESX 3.5U6 host that you cannot get rid of so now you have to setup the Licensing server. No big deal download it from VMware’s site and do an install, except the server services will not start and instead return a 1920 error so the install fails and rolls back. No big deal just Google around and discover that the discrete download license server will not work and you need the one from the vpx folder of the vCenter Server 2.5U6 install CD. No big deal you still have the disk/iso or just download it again and do the install, confirm it is working, copy the license file local, follow the prompts, and you are done. Now allocate the license to the hosts and vCenter server and you should be ready for a well deserved break, except every time you assign a key to the vCenter server you get an error (“The vCenter Server with vCenter Server 4 Standard license needs access to a license server before it can manage hosts in the inventory.”) and you cannot save the change. Now I did not find anyone with the answer but this communities post got me close enough to figure out that my problem was that I did not have a license server defined in the vCenter Server setup as shown below.

All told if I had known I would of just shot the old ESX 3.5U6 host like a lame horse and dealt with the consequences. Next from 4.0U2 to 4.1, it could not go any worse, right?

Is there any real competition for Cisco and UCS?

Brent Quick — Sun, 18 Jul 2010 05:10:01 +0000

After reading Steve Kaplan’s article comparing Cisco UCS to the HP Matrix (Link) I had a back channel exchange about what Steve thought might qualify as competition, and in the end nothing was the answer. There are some competitive stacks that can be built but no competitive product. Am I and by extension Steve missing something? I would appreciate some comments.

Thanks - Brent

Steve’s list of competitive stacks:

HP + VC
Dell + Scalent
HP + Xsigo + Infiniband