Moving from VMware Workstation to ESX

At RIT NSSA, we've been using a VMware Workstation implementation, dubbed Remote Laboratory Emulation System (RLES), for well over two years as a platform to teach NSSA applied curriculum. This project has been faculty designed, student built and student maintained for these years. This past winter, with the help of Information Technology Services, our department started to construct an ESX cluster that would more effectively support the RLES concept. So far, one course Security Audits of Web Servers and Applications, has been offered on the ESX version of RLES. Another course, Computer Viruses and Malware, is being offered right now. Luckily, both of these labs for these courses were designed with the intent to succeed in a VMware Workstation environment. This made migrating from Workstation to ESX somewhat simpler because we were able to convert or directly import some of the virtual machines. Note the qualifier some. Other courses that will be moving to a virtualization platform aren't as lucky. The following post describes, briefly, the issues we’ve experienced thus far migrating from virtualization to virtualization.

This past week, Kristian Stokes and I attempted to import the DMZ auditing lab for Security Audits of Web Servers and Applications. This lab involves four systems inside a virtual DMZ as well as the firewall/router virtual machine that routes to the DMZ. The four DMZ systems include vulnerable instances of FreeBSD 5, Fedora 6, Windows Server 2000 and Windows 2003 all running vulnerable services with horribly exposing mis-configurations. There are two major issues with making this lab succeed with the ESX setup: (1) ESX requires all virtual machines to use SCSI virtual disks, (2) We’re running Lab Manager 2.5 which only supports 1 networking adapter. Oh, and an even bigger issue: the DMZ systems LACK SUFFICIENT DOCUMENTATION! An aside: this DMZ lab was created by a graduate student two years ago. The cost required to learn and re-implement these systems is very high. Theoretically, since this is part of an auditing course, any student who fully audited the DMZ systems should be able to recreate them (pfft, ya right).
So, Kristian is formally documenting the trials and tribulations of this migration, but below are the migration paths for the DMZ virtual machines.

• FreeBSD
  
  1. the original machine had an IDE virtual drive, so we have to convert it to SCSI.
     
  2. we tried was a straight vmdk conversion using a tutorial. This failed to create a virtual disk that was even recognizable by another FreeBSD system.
     
  3. we tried using clonezilla (without reading clonezilla support docs) to duplicate the data from the IDE disk to a fresh SCSI disk. Instantly we noticed that clonezilla dropped to a normal dd operation and figured that the FreeBSD file system wasn’t supported by the clonezilla suite. Clonezilla doesn’t support UFS; which was the file system type of our virtual machine. The dd from clonezilla made a drive, and it appeared to data… just not in the UFS slices that needed to be there.
     
  4. we tried using CloneHDD to duplicate from IDE to SCSI. CloneHDD is a utility to duplicate FreeBSD installations. Once we got the script to run, it would pause after copying one slice.
     
     This is where I went home, because we’d already spent 5 hours working on one image.
     
  5. Kristian spent some time manually trying to duplicate the partitions from the IDE disk to the blank SCSI disk with some moderate success. While manually using dump on /var he got a message stating that a filename was too long, above the 1044 max character limit. He then found a directory with many subdirectories all with really long names like XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX etc. Assuming this was the problem, he deleted the directory structure.
     
  6. Kristian used CloneHDD to successfully copy the slices. The CloneHDD script fixed /etc/fstab on the SCSI disk to mount da0 (SCSI0:0) rather than ad0 (IDE0:0). He removed the IDE referenced in the .vmx and booted the virtual machine with VMware Workstation on his desktop.
     
  7. Kristian imported this virtual machine into Lab Manager.
     
  8. When trying to deploy the virtual machine, it boots to bootrom> and the keyboard doesn’t work.
     
  
  
• Fedora – this machine also has an IDE drive.
  
  1. we tried to use clonezilla to convert the IDE drive to SCSI; however it failed because of RedHat’s default LVM file system feature.
     
  2. we’re still waiting to work on this one more, a straight dd should work with modifications of the grub boot configuration file. (we’ll update this post when we finish this migration)
     
  
  
• Windows Server 2000
  
  1. This machine had a SCSI disk, but when we attempted to import it into Lab Manager, LM spit back this:
     
     
     
     • Error executing lm-vmkimport: Failed to open '/pathto/labmanager/mnt/mysmbshare:Classes_1349690868/path/to/vmdk/Windows 2000 Server-000002.vmdk': The parent of this virtual disk could not be opened (23). . The originating server for this exception is: esxnode1.local
       
     
     
  2. We speculate that a peice of the vmdk (the "parent" geometry) isn't in the /path/to/vmdk/
     
  3. This machine is still waiting to be imported
     
  
  
• Windows 2003
  
  1. This machine had a SCSI disk and properly imported without fuss! Yay!
     
  
  
• Router
  
  1. this machine had a SCSI disk but had two NICs (because its simulating a corporate firewall/router). This machine is still waiting to be imported – with any luck, the upgrade to LM3.0 will allow us to have multiple network adapters.
     
  
  

Another student's progress on converting Computer Viruses and Malware this summer is going well, I think. He had some hiccups trying to import the old virtual machines from the lab, so he just created new ones. The virtual machines for these labs involve some XP instances with Sysinternals TCPView, FileMon, RegMon, etc, IDA, bagle, sasser, and some trivial SANS-ish malware examples as well as a honeyd/snort. The re-implementation of these virtual machines in the ESX environment is much more manageable than the DMZ lab. When this student coughs up some documentation, I’ll post it up here.


Anyways, as you can read, there is a lot going on with simply moving from Workstation to ESX. Beyond all the nitty gritty technical work, we’re also looking at the bigger picture -- like how all of these changes affect student productivity, curricular benefits, etc. Expect some more information regarding our setup rather soon.