HiddenOne

A while back I talked about building a VMware whitebox to run VMware ESX on inexpensive equipment. That was over a year ago and a few things have changed, so I figured an update was in order.

For the whitebox build I purchased a new motherboard, processor, and memory. This was based on the requirements that at the time ESX required a SAS chipset on the whitebox motherboard. Over the past year ESX’s support of SATA chipsets has improved, so you may be able to save some money on the motherboard. Below is a complete list of the hardware used in my build.

• Enclosure: Lian Li PC-V6000B ATX Mid Tower
• Processor: Intel Xenon X3230 Kentsfield (Quad Core)
• Motherboard: Asus P5BV-E/SAS
• Memory: 4 x Wintec AMPX 2GB DDR2-800
• Storage: Western Digital Caviar WD5000AAKS 500GB 7200 RPM 16MB Cache SATA-II
• Hard drive Enclosure: Athena Power BP-SATA3051B 5-bay hot-swap SATA backplane
• Power Supply: Enermax Noisetaker II EG425P-VE 420W
• Enclosure: Lian Li PC-V6000B ATX Mid-Tower

As you can probably tell, this used to be a file server that I built but decommissioned when a motherboard failure also took my data with it. One 500GB hard drive has been sufficient for both the ESXi installation and the storage of the all of the virtual machines (thin provisioning works very well).

Initially I installed Ubuntu 64-bit as the host operating system, with VMware Server running on top of that. Additionally I used GNS3 on the host to connect back to my lab equipment via the second NIC. Since then, VMware has made ESX (ESXi) freely available. I decided to install VMware ESXi 4.0 in place of the host OS, so that ESX can better manage allocation of the physical resources to each VM.

So far everything has been working out well. I have configured another Vswitch which connects to the second physical NIC and will be used to connect the VMs to the lab equipment. In the coming weeks, I will be creating a Linux VM and installing Dynagen to handle virtualization of some Cisco routers.

Virtualization has been a hot topic in the IT would for a while now. From the datacenter to our home labs, virtualization is changing the way we work and play.

I recently made an addition to my lab that I think will allow me to add more complexity to lab simulations, at the same time it has saved me some money! A few months back you will recall that I purchased some additional hardware and made a virtualization server (using Windows 2008 Server and Hyper-V). Since that setup was not getting much use, I went a different route that should prove to be more useful.

There are three physical parts to my current lab setup. They are: 16U rack with physical network equipment, server running VMware Server, and my laptop. The rack consists of the following:

• 1 x 2528 access server
• 3 x 2525 routers
• 1 x 3640 router (NM-2FE2W, NM-1E2W, NM-8A/S)
• 1 x 2924 switch
• 2 x 3550 switches (L3 support)

The server running VMware has a quad core Intel Xenon processor with 8GB of RAM and 5 x 500GB hard drives in a hot-swappable drive cage. This used to be a file server which is why it has so many hard drives and a hot-swap cage. Only one hard drive is being used to house both the operating system and all of the virtual machines. Finally, my laptop is my old reliable Compaq Presario V2000. With 1GB of RAM, a 1.8Ghz AMD Turion processor, and a 5400RPM hard drive this thing gets bogged down in a hurry when I start running anything more than a few routers in GNS3! It survived five years of college as many other laptops around me crumbled, so I can’t come to replace it just yet!

How do I connect all of this? It’s not overly complicated once you get everything straight in your head, however when putting something like this together for the first time I suggest some actual planning! The diagram below will help illustrate what I have done.

The two large boxes at the top (vmware-server and Laptop) are the two physical computers. The setup on my laptop is straightforward, since I only have GNS3 running there (no VMs). As you can see, I use the cloud feature in GNS3 to connect the wired network card on my laptop to my physical lab (The rack at the bottom of the diagram).

The VMware server is what complicates things. First you will notice that the server has two network cards. I have used this to my advantage to segment some things out. The first network card (eth0) is used for management of the VMware server through the web interface, as well as linking any VMs to the Internet (either directly via a bridged network or through some GNS3 routing). The second network card (eth1) is strictly used to tie the VM running GNS3 to my physical lab rack. There are two virtual network adaptors (vmnet0 and vmnet5) that are bridged directly to each of the physical network cards.

Inside of the VMware server there are four other virtual network adaptors (vmnet1 to vmnet4). These four devices are host-only network adaptors. This means that they are not tied in any way to either of the servers physical network cards. Notice that the GNS3 VM connects to each of these virtual network adaptors. I did this so that I can perform routing between each subnet. The GNS3 VM is also tied to each of the bridged network adaptors so that I can route traffic externally as well.

One thing not pictured in the diagram is the other VMs I use. These are simple setups, as each VM connects to one (and only one) of the host only network adaptors. These VMs act as endpoints/nodes to test connectivity between other nodes.

Hopefully this will provide other people with a jumping off point for adding virtualization to their lab. This can be accomplished for a small amount of money. A desktop with plenty of RAM will work just fine as a VMware server. Combine the price of memory now with the low, low price of free for Linux and VMware Server and you have an incredible deal!

Hyper-V is similar to VMware in the respect that it abstracts the hardware from the vitrual machines. This is different from other products such as VMware workstation or Microsoft Virtual PC, where the VMs run on top of the host operating system.

Hyper-V has a few requirements to get everything up and running. First, you need to install Windows Server 2008 x64 (64-Bit). Other versions will not satisfy the Hyper-V requirements. The second requirement is probably going to be the biggest problem for people trying to run older hardware.

Hyper-V requires a processor that supports hardware assisted virtualization. AMD calls this AMD-V, while Intel calls it VT. Some older processors may support this with an update to the motherboard BIOS, others will not. I found that my 3-4 year old Opteron 165 did not support hardware assisted Virtualization. Some research will be required to see if your specific hardware is supported.

In my first post, I gave a few reasons why I wanted to build a VMware whitebox. To recap, my main reasons are:

1. Save money on the hardware (no enterprise class server to purchase).
2. Save money on my power bill.
3. Keep noise to a minimum, or the level you would expect from a typical desktop.

Why did I choose virtualization, and why did I choose VMware ESX server?

The choice to go with virtualization was a no-brainier for me. In a lab setting there will probably be a need to run many different servers. These servers can range from a typical Windows XP build all the way to a Linux or Solaris based server. Having one or two physical servers running two different operating systems just did not give me the flexibility I needed.

With virtualization, I can build as many virtual servers as I want on one physical server. In this case the limit is only the amount of hard drive space the host has available. Keep in mind, I am not talking about having 10-20 virtual servers running at the same time. Your ability to do this without negatively effecting performance will depend on many factors, including the host hardware.

Rebuilding servers is also easier with virtualization. For example, I can build a Debian based virtual server and copy that image to another location off the host hard drive. If in the process of experimenting with different configurations, I completely hose that Debian server, rebuilding it is as simple as coping that original image file back to the host hard drive. This greatly minimizes the amount of time spent rebuilding and reconfiguring servers.

Why VMware ESX? This product is used by many companies of varying sizes (including the company I currently work for). In the process of playing around, I expect to pickup basic VMware administration skills. Adding another skill to you resume, or impressing your boss with you knowledge of VMware can’t hurt!

The bottom part of my network rack has been occupied with two servers for a while. At the time that I purchased the servers, I thought they would be a great and would be put to use on a regular basis. Soon after I grew tired of dealing with them. My Dell server was a full length server, which made it hard to get to the connections in the back. Both servers were very loud, and they consumed too much power.

Recently the Dell poweredge was sold, and I’m still waiting to unload the Sun Netra X1. My plan is to replace the servers with a VMware ESX server whitebox.

The term whitebox refers to a computer or server that has hardware that may not be listed on the official VMware hardware list. Building a VMware whitebox is not only cheaper, but it is also more practical for a home setting where you don’t want a huge server keeping you up at night (let alone driving your power bill through the roof).

Based on the official VMware hardware list, we can find out what network chipsets, SAS chipsets, etc are supported by the VMware kernel. This sounds like an easy task, but this information is sometimes difficult to find. On top of that, you have to be very careful about the hardware you choose, since the VMware kernel is notoriously picky about the hardware it will recognize

My ultimate goal is to build a whitebox for under $1,000 that has the ability to run 7-8 virtual servers with ease.