Follow Up: Differential Analysis - WDS & DHCP

So I was doing some more reading about the WDS & DHCP service split Jason and I talked about in these two posts when I found a technet article that had some information in it that could have saved us some time. The section titled Known issues with configuring Windows Deployment Services says "If DHCP is installed on a server that is located in a different subnet, you will need to do one of the following ... Add DHCP options 66 and 67. Option 66 should be set to the Windows Deployment Services server, and option 67 should be set to boot\x86\wdsnbp.com."

The article also has a link here to another technet article with more detailed information about network boot programs. After doing some further reading it turns out that the wdsnbp.com image has the following purposes:
1. Architecture detection
2. Pending computer scenarios. When the Auto-Add policy is enabled, it is sent to pending computers to pause the PXE boot and report back the client computer's architecture to the server.
3. PXE referral cases (including use of Dynamic Host Control Protocol (DHCP) options 66 and 67)

---

So I was able to setup a split WDS/DHCP environment in production, all of the packets were being passed from client to server based on my packet captures. The PCs that I am attempting to deploy to have an x64 architecture so based on Microsoft's documentation ("In addition, x64-based computers can run x86-based or x64-based boot images. Therefore, for each of these tasks, you could have two boot images—one for x86 and one for x64. The boot menu on x86-based computers will only display x86 boot images (because x86-based computers cannot run x64 boot images).") I should be fine using an x86 boot.wim to boot.

But when I go to boot the client into the default boot.wim boot image (taken from a 2008 Server DVD) it gets the following error:
WdsClient: An error occurred while communicating with the Windows Deployment Services server. Please check to ensure that the server is operational and that the necessary ports are open on the server's firewall. Server name [name], Server IP address [ip].

By hitting Shift+F10 I get a command shell where I checked for a valid IP address which I had.
Then I checked the detailed log file of the boot process in: x:\Windows\Panther\Setupact.log.

The very bottom of the log file has the following error messages:
Info "InitializeLogging: RPC_S_SERVER_UNABAILABLE - Retrying server request for initializing logging."
Error "CreateClientSession: Failed to initialize Client -> Server logging. Error code [0x800706BA].[gle=0x000006ba]"
Error "CreateClientSession: Failed to create client session. Error code [0x800706BA].[gle=0x000006ba]"
Error "CallBack_WdsClient_DetectWdsMode: Failed to create client session or initialize WDS unattend. Error [0x800706BA].[gle=0x000006ba]"

Now the weird thing is that I can boot to the capture.wim image (still x86) with no problems, so I did some more research and found out that this data was being blocked at the network...

Looking at some more documentation from Microsoft I see that the following ports must be open for WDS to work (the error message mentioned above was due to port 5040 needed for WDS to create an RPC connection being blocked):

• UDP - 67, 68, 69, 4011
• TCP - 135, 137, 138, 139, 5040

After changing the firewall rules everything started working again.

Great!

---

I hit another snag in the deployment. Now I have an image (52GB Vista Business) which I created overnight (I estimate it took about 5 or 6 hours to capture). I saved the initial WIM file to an external hard drive due to its size and overnight the connection to the server was lost so the image was not moved.

No big deal, I just simply plug the drive into the server go to WDS and import the new image into the new Image Group that I created.

So after this gets done I go and try to pull this image but when I boot the client up into the WDS PE boot environment I do not see any images (I should see two at this point).

Back to the server where I enabled trace logging on all the components with regarding to WDS, these are located in the registry under:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Tracing\

Just look for the keys that start with WDS.

These log files will turn up in %windir%\Tracing\

I found the following errors in the log:

[WdsImgSrv] Error in enumerating images. Error [13].

So I disabled all the images on the server and copied the new image from the USB drive onto the local hard drive then imported it into WDS under my original Image Group.

So this worked, the image pushed down in one and a half hours and everything worked.

Next up is documentation about MDT, AIK and Unattended installation of Vista and Server 2008.

Moving from VMware Workstation to ESX

At RIT NSSA, we've been using a VMware Workstation implementation, dubbed Remote Laboratory Emulation System (RLES), for well over two years as a platform to teach NSSA applied curriculum. This project has been faculty designed, student built and student maintained for these years. This past winter, with the help of Information Technology Services, our department started to construct an ESX cluster that would more effectively support the RLES concept. So far, one course Security Audits of Web Servers and Applications, has been offered on the ESX version of RLES. Another course, Computer Viruses and Malware, is being offered right now. Luckily, both of these labs for these courses were designed with the intent to succeed in a VMware Workstation environment. This made migrating from Workstation to ESX somewhat simpler because we were able to convert or directly import some of the virtual machines. Note the qualifier some. Other courses that will be moving to a virtualization platform aren't as lucky. The following post describes, briefly, the issues we’ve experienced thus far migrating from virtualization to virtualization.

This past week, Kristian Stokes and I attempted to import the DMZ auditing lab for Security Audits of Web Servers and Applications. This lab involves four systems inside a virtual DMZ as well as the firewall/router virtual machine that routes to the DMZ. The four DMZ systems include vulnerable instances of FreeBSD 5, Fedora 6, Windows Server 2000 and Windows 2003 all running vulnerable services with horribly exposing mis-configurations. There are two major issues with making this lab succeed with the ESX setup: (1) ESX requires all virtual machines to use SCSI virtual disks, (2) We’re running Lab Manager 2.5 which only supports 1 networking adapter. Oh, and an even bigger issue: the DMZ systems LACK SUFFICIENT DOCUMENTATION! An aside: this DMZ lab was created by a graduate student two years ago. The cost required to learn and re-implement these systems is very high. Theoretically, since this is part of an auditing course, any student who fully audited the DMZ systems should be able to recreate them (pfft, ya right).
So, Kristian is formally documenting the trials and tribulations of this migration, but below are the migration paths for the DMZ virtual machines.

• FreeBSD
  
  1. the original machine had an IDE virtual drive, so we have to convert it to SCSI.
     
  2. we tried was a straight vmdk conversion using a tutorial. This failed to create a virtual disk that was even recognizable by another FreeBSD system.
     
  3. we tried using clonezilla (without reading clonezilla support docs) to duplicate the data from the IDE disk to a fresh SCSI disk. Instantly we noticed that clonezilla dropped to a normal dd operation and figured that the FreeBSD file system wasn’t supported by the clonezilla suite. Clonezilla doesn’t support UFS; which was the file system type of our virtual machine. The dd from clonezilla made a drive, and it appeared to data… just not in the UFS slices that needed to be there.
     
  4. we tried using CloneHDD to duplicate from IDE to SCSI. CloneHDD is a utility to duplicate FreeBSD installations. Once we got the script to run, it would pause after copying one slice.
     
     This is where I went home, because we’d already spent 5 hours working on one image.
     
  5. Kristian spent some time manually trying to duplicate the partitions from the IDE disk to the blank SCSI disk with some moderate success. While manually using dump on /var he got a message stating that a filename was too long, above the 1044 max character limit. He then found a directory with many subdirectories all with really long names like XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX etc. Assuming this was the problem, he deleted the directory structure.
     
  6. Kristian used CloneHDD to successfully copy the slices. The CloneHDD script fixed /etc/fstab on the SCSI disk to mount da0 (SCSI0:0) rather than ad0 (IDE0:0). He removed the IDE referenced in the .vmx and booted the virtual machine with VMware Workstation on his desktop.
     
  7. Kristian imported this virtual machine into Lab Manager.
     
  8. When trying to deploy the virtual machine, it boots to bootrom> and the keyboard doesn’t work.
     
  
  
• Fedora – this machine also has an IDE drive.
  
  1. we tried to use clonezilla to convert the IDE drive to SCSI; however it failed because of RedHat’s default LVM file system feature.
     
  2. we’re still waiting to work on this one more, a straight dd should work with modifications of the grub boot configuration file. (we’ll update this post when we finish this migration)
     
  
  
• Windows Server 2000
  
  1. This machine had a SCSI disk, but when we attempted to import it into Lab Manager, LM spit back this:
     
     
     
     • Error executing lm-vmkimport: Failed to open '/pathto/labmanager/mnt/mysmbshare:Classes_1349690868/path/to/vmdk/Windows 2000 Server-000002.vmdk': The parent of this virtual disk could not be opened (23). . The originating server for this exception is: esxnode1.local
       
     
     
  2. We speculate that a peice of the vmdk (the "parent" geometry) isn't in the /path/to/vmdk/
     
  3. This machine is still waiting to be imported
     
  
  
• Windows 2003
  
  1. This machine had a SCSI disk and properly imported without fuss! Yay!
     
  
  
• Router
  
  1. this machine had a SCSI disk but had two NICs (because its simulating a corporate firewall/router). This machine is still waiting to be imported – with any luck, the upgrade to LM3.0 will allow us to have multiple network adapters.
     
  
  

Another student's progress on converting Computer Viruses and Malware this summer is going well, I think. He had some hiccups trying to import the old virtual machines from the lab, so he just created new ones. The virtual machines for these labs involve some XP instances with Sysinternals TCPView, FileMon, RegMon, etc, IDA, bagle, sasser, and some trivial SANS-ish malware examples as well as a honeyd/snort. The re-implementation of these virtual machines in the ESX environment is much more manageable than the DMZ lab. When this student coughs up some documentation, I’ll post it up here.


Anyways, as you can read, there is a lot going on with simply moving from Workstation to ESX. Beyond all the nitty gritty technical work, we’re also looking at the bigger picture -- like how all of these changes affect student productivity, curricular benefits, etc. Expect some more information regarding our setup rather soon.

Idenfitication Woes

Last year, Tom and I worked on a project called Obfuscator for our forensics course. The project was to demonstrate, to our class, that changing file signatures was as easy as changing file extensions and therefore the thoroughness of file signature analysis tools is questionable. When Harlan blogged that anti-forensics , "techniques don't defeat tools...they defeat examiners." I quickly replied alluding to our (Tom and my) conclusion about fully understanding the capabilities of our forensic tools and how file identification (just like people authentication) is HARD.


A while back, Harlan asked a question about a script from Didier Stevens that embeds an executable inside a VBScript.

"What would you look for if you were analyzing a system and trying to determine if something like this had been used?"

Well no one posted a reply to you Harlan... and after thinking about this question since July 2nd, my response is still: I don't know.

Static file analysis could search for binary execution methods ... like Run for wscript ... but that would be impractical, I think. As with identifying a file, identifying a malicious script isn't as easy as it looks.

So rather than really answering Harlan's question, I'll ask one:
Is writing and executing an executable a common scenario in scripting?

Registry Analysis #1

Summary:

1. HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\PagingFiles
2. I need to read Hobocopy documentation to make it work in Vista x64
   
3. If you can't copy a file from a mounted VMDK, try mounting and copying as administrator
4. Yay -- now I can play with RR
   

After Harlan posted about an interesting registry entry this morning, I thought of the systeminfo utility. I thought, "I wonder if the systeminfo tool queries the registry for similar information?". So I fired up Process Monitor, set filters for the Registry Event Class and executed systeminfo. Once systeminfo finished, I stopped the capture and searched for systeminfo within Process Monitor. It appears that the systeminfo binary directly queries some registry values and also utilizes WMI. Cool. So I posted a reply to Harlan saying there's some interesting material there. But, I didn't say which entries seemed interesting. Harlan asked me what was interesting, so I went back to look. I found TimeZoneInformation and some network adapter information (both of which were covered by RR plugins). So I tried to find something that wasn't in RR, and I think I did:

Paging File Location
(HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\PagingFiles)

Information: here ... this value is a REG_MULTI_SZ in Vista rather than the REG_BINARY as listed inthe MS article.
Significance to RA: If the page file is not in the default location \pagefile.sys, you'll want to know where it is.

The systeminfo utility also reports tidbits like patch levels; however, I'm not sure (yet) if this is listed in the registry.

---

The remainder of this post shows some of my experiences in registry hive acquisition (summary items 2-4)

Until now, I did not think that I had an easy way to get access to registry hives. Before tonight, I tried mounting a vmdk on my desktop with VMware Workstation's drive mapping feature so that I could simply copy the hives, but that failed:




I figured the file was just locked or something -- and a while ago, I stumbled upon a post that mentioned using VSS to copy a file that is in use but I shrugged it off because they didn't have Vista binaries. Well, I just searched and there's an open source project! Its called HoboCopy (enter chuckle about my scripting issue).

So, I downloaded hobocopy for vista x64 and executed it:


I missed the Visual Studio 2008 libraries dependencies (vcredist_x64.exe)... Once those were installed, the hobocopy still wouldn't run under my Jason user... so I opened an elevated shell and hobocopy ran fine.

When I tried to change to virtual drive of the VMDK within my elevated shell, the shell explained:

The system cannot find the drive specified.

PowerShell also explained:


I confirmed that the drive was still accessible in my Jason shell (PowerShell background window). It appears I have a user-specific drive letter? Weird...

I really wanted to get hobocopy to copy the system hive, so I went to mount the vmdk with vmware-mount in the elevated shell, but it didn't exist in my workstation folder! I'm assuming this is because I'm using VMware WS 6.5b2.

So I elevated VMware Workstation, mounted the vmdk and tried to copy the file with hobocopy -- it failed. But, I was able to copy the hive file just fine with copy!

Summary:

• HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\PagingFiles
• I need to read Hobocopy documentation to make it work in Vista x64
  
• If you can't copy a file from a mounted VMDK, try mounting and copying as administrator
• Yay -- now I can play with RR