bigger, better, faster, more VMware

19 07 2011

No, before we even start, this is not a blog post about the 4 Non Blondes album.

This is a documentation of my mind numbing, soul destroying search for the best performing configuration with the hardware I have in my lab.

I have spent countless hours / days / weeks building, breaking & rebuilding my VM Lab (thankfully I have an understanding wife & daughter).

Hardware primarily consists of:

2x HP Proliant N36L Microservers (Athlon II Neo Dual Core 1.3) (8GB RAM in one box / 2GB in the other).
2x HP DC7100 Desktops (P4 2.8 / 2GB RAM)
1x Dell Precision 370 (P4 3.0 / 4GB RAM)
Cisco 2950 (24×10/100 + 2×1000)

Add to this an assortment of older F5’s / dual P3 pizza boxes & other no name kit – and I have a playground full of toys.

I have been using unRAID for the past few years on various hardware platforms. This has been mainly for storing media, ISO’s & providing a backup target for the various laptops & workstations around the house.

Recently I picked up the two HP N36L Microservers, the 8GB one is my primary VMware ESXi 4.1 host and the 2GB server is running unRAID. Just having the second box sitting there running unRAID seems a little under-utilised to me, the disks are in standby most of the time (thanks to netflix) – and it just hasnt been getting the workout I think it deserves.

So – enter the newest project – whats the best solution for storing media / ISO’s / Backups / VMs etc – I want to be able to use either iSCSI or NFS to play with vMotion of VMs when I finish building my VMware test lab to finish my VCP – I want it fast, but I want it protected in case a disk fails.

I have tested & played with the following in my quest:

Storage Systems:
FreeNAS 7 & 8 (Physical & Virtual)
Openfiler (Physical & Virtual)
Nexenta (Physical & Virtual)
unRAID Physical (Virtual not supported due to USB GUID licensing)

Presentation to Client Machines:
Local storage in the ESXi host presented to Windows 7 VM
iSCSI Raw Device Mappings presented to Windows 7 VM
iSCSI Presented to ESXi -> VMFS-3 filesystem -> VMDK presented to Windows 7 VM
iSCSI Presented to Physical Windows 7 Client
NFS Presented to ESXi -> VMDK presented to Windows 7 VM
CIFS/SMB Presented to Physical Windows 7 Client

I have been using a single test scenario on each config – using Iometer – with the file & results formatted from http://vmktree.org/iometer/

First I want to benchtest them for performance, then to setup the best solution that is a mix of performance & redundancy.

Sounds impossible – im gonna try.

For the performance benchtesting – I decided to go with a 2 spindle ZFS striped config, tested from Windows 7 Clients
Physical Client: HP DC7100
Physical FreeNAS: HP N36L(2GBRAM / 1TB WD Green / 2TB WD Green )

I have mismatched sizes as thats the hardware I have free at the moment. If I find a compelling reason why this wont work, then I may get a second 2TB disk to match. I am using WD Green disks for their low power / cooler running – comodity hardware.

Scenario 1 – Physical FreeNAS 7 with iSCSI
Physical Client -> iSCSI on Physical NAS
Virtual Client -> VMDK on ESXi -> iSCSI Physical NAS

Scenario 2 – Physical FreeNAS 7 with NFS & CIFS/SMB
Physical Client -> CIFS/SMB on Physical NAS (Usual windows sharing type scenario)
Virtual Client -> VMDK on ESXi -> NFS Physical NAS

Scenario 3 – Virtualised FreeNAS 7 with iSCSI
* Physical Disks formatted with VMFS-3, with VMDK presented to FreeNAS VM
Physical Client -> iSCSI on Virtual NAS
Virtual Client -> VMDK on ESXi -> iSCSI Virtual NAS

Scenario 4 – Virtualised FreeNAS 7 with NFS & CIFS/SMB
* Physical Disks formatted with VMFS-3, with VMDK presented to FreeNAS VM
Physical Client -> CIFS/SMB on Virtual NAS
Virtual Client -> VMDK on ESXi -> iSCSI Virtual NAS

Scenario 5 – Virtualised FreeNAS 7 with iSCSI
* Physical Disks presented via Physical RDM passthrough to FreeNAS VM
* RDM Config thanks to http://www.vm-help.com/esx40i/SATA_RDMs.php
* RDM passthrough used to enable SMART monitoring from the FreeNAS VM – very cool
Physical Client -> iSCSI on Virtual NAS
Virtual Client -> VMDK on ESXi -> iSCSI Virtual NAS

Scenario 6 – Virtualised FreeNAS 7 with NFS & CIFS/SMB
* Physical Disks presented via Physical RDM passthrough to FreeNAS VM
* RDM Config thanks to http://www.vm-help.com/esx40i/SATA_RDMs.php
* RDM passthrough used to enable SMART monitoring from the FreeNAS VM – very cool
Physical Client -> CIFS/SMB on Virtual NAS
Virtual Client -> VMDK on ESXi -> iSCSI Virtual NAS

I will be adding follow up posts with the performance results, the PRO’s & CON’s (in my view) with each of these scenarios. Feel free to add comments & kick off discussions about this project.