Graphical ImageX

While using WDS, the imagex binary is at the forefront of WIM creation. Apparently, there is a graphical front end, GImageX! This is fun, and exciting. An intuitive tabbed interface immediately enables you to hone in on imaging process you're about to undertake.

While I was playing with ImageX this summer at RIT, I was injecting large VMDKs into a mounted, writeable WIM. When I unmounted and committed changes, it would take an incredibly long amount of time to unmount. So long so that, at times, it appeared like the utility 'froze' because there was no significant resource usage displayed in the Windows Resource Monitor or Task Manager. So, I launched process explorer and filtered events related to imagex. Sure enough, there are a ton of events occuring that relate directly to imagex; therefore, it's not 'frozen.'' It was definitely annoying that there is no progress bar or good indication that the utility is still running successfully.

One of the first things I wanted to discover about GImageX was how the utility handled these unmount commit scenarios. Well, the good news is, there was a significant amount of disk usage directly from gimagex.exe throughout the few minute unmount. Further, there is a little cursor inside the dialog showing that the program is indeed still working. Since this is a GUI, you can tell that the application has still responding because you can move its windows.

First impressions of this utility are very high. I'm happy that I bumped into it from Ulli's latest post about the ESX Bandit.

Master's Thesis

Well, I'm all done with my MS in Computer Security and Information Assurance from RIT. The thesis, titled Differential Virtualization for Large-Scale System Modeling, is posted here. Some of the stuff previously posted on the blog is incorporated into the thesis including WDS/DHCP and multicasting files.

Multicast File Transmission in WDS

In the environment detailed in my MS thesis and a recently accepted paper to SIGITE '08, we describe an environment that uses a standard set of virtual machine templates. This set of virtual machine templates is then distributed (and kept consistent) across a set of workstations. Then, users can create differential virtual machines (VMware's linked clones) based on the templates. Users store the linked clones on a file server, and can achieve virtual machine mobility between workstations. At RIT NSSA, this environment is semi-operational as I write this post. Since RIT NSSA teaches many different operating system technologies, there are many virtual machine templates that reach a summed size of 100GB. One of the issues that we discusssed in our paper and that I present in my thesis is the notion of updating template virtual machines across all workstations. This is a difficult subject because as the number of workstations increases, the copies of this template repository increase. Now, RIT NSSA has 80 workstations in the pilot-lab -- that means, right now, when they want to update or add a template to each machine, they have to inject the files into an image and re-deploy the OS and data on all 80 workstations. Another way they can achieve an update is through some differential robocopy script that copies the templates from the file server to the workstations -- this can be done in series or parallel (I've found that robocopies in series seem to work much better with the storage devices in that file server -- Adaptec 2820SA with 5 SATAII in RAID5). However, all of these approaches are inefficient because they either copy superflous data once as in deploying an install image, or they copy the same data 80 times as in differential robocopy. There has to be a better way!

Enter WDSMCAST.exe from the Server 2008 AIK. WDSMCAST enables multicast transmissions of custom data stores. So, I can create diretory and make a custom WIM with the directory's contents. Once I have a custom WIM, I can create a custom namespace on my WDS Transport Server using wdsutil /new-namespace with the /configstring parameter specified as the location of the custom WIM. Microsoft's documentation states that the custom WIM can be stored in any directory. This, however, caused a divide by zero in my test runs with WDSMCAST.exe:

So, I moved the custom WIM inside the RemoteInstall directory and then multicast transfer of the image works just fine.

WDSMCAST runs just fine inside Vista:

Therefore, we could create a differential version of the repository, generate a new WIM, create a multicast session based on the WIM, instruct each workstation to join the multicast session, and then have each workstation extract contents of the WIM ontop of the repository at the workstation.

While this is a nice solution, it is possible that a workstation require twice the size of the update in free disk space. For example, if we wanted to add 20GB of templates to all workstations, the workstations need at least 40GB of free space because 20GB is required for the WIM and 20GB is required for the extracted templates. In an environment where this is realistic, it would be neat to issue these differential updates in a multicast fashion with wdsmast.

Microsoft says, "You can create a custom content provider for cases where the default provider is not sufficient (for example when using Transport Server to deploy an operating system from inside a .vhd image). See the Windows Server 2008 SDK for guidelines and samples for authoring and registering the provider." I'm going to investigate custom content providers for the purposes of transmitting a template repository version and talk about custom content providers in future post.

Follow Up: Differential Analysis - WDS & DHCP

So I was doing some more reading about the WDS & DHCP service split Jason and I talked about in these two posts when I found a technet article that had some information in it that could have saved us some time. The section titled Known issues with configuring Windows Deployment Services says "If DHCP is installed on a server that is located in a different subnet, you will need to do one of the following ... Add DHCP options 66 and 67. Option 66 should be set to the Windows Deployment Services server, and option 67 should be set to boot\x86\wdsnbp.com."

The article also has a link here to another technet article with more detailed information about network boot programs. After doing some further reading it turns out that the wdsnbp.com image has the following purposes:
1. Architecture detection
2. Pending computer scenarios. When the Auto-Add policy is enabled, it is sent to pending computers to pause the PXE boot and report back the client computer's architecture to the server.
3. PXE referral cases (including use of Dynamic Host Control Protocol (DHCP) options 66 and 67)

---

So I was able to setup a split WDS/DHCP environment in production, all of the packets were being passed from client to server based on my packet captures. The PCs that I am attempting to deploy to have an x64 architecture so based on Microsoft's documentation ("In addition, x64-based computers can run x86-based or x64-based boot images. Therefore, for each of these tasks, you could have two boot images—one for x86 and one for x64. The boot menu on x86-based computers will only display x86 boot images (because x86-based computers cannot run x64 boot images).") I should be fine using an x86 boot.wim to boot.

But when I go to boot the client into the default boot.wim boot image (taken from a 2008 Server DVD) it gets the following error:
WdsClient: An error occurred while communicating with the Windows Deployment Services server. Please check to ensure that the server is operational and that the necessary ports are open on the server's firewall. Server name [name], Server IP address [ip].

By hitting Shift+F10 I get a command shell where I checked for a valid IP address which I had.
Then I checked the detailed log file of the boot process in: x:\Windows\Panther\Setupact.log.

The very bottom of the log file has the following error messages:
Info "InitializeLogging: RPC_S_SERVER_UNABAILABLE - Retrying server request for initializing logging."
Error "CreateClientSession: Failed to initialize Client -> Server logging. Error code [0x800706BA].[gle=0x000006ba]"
Error "CreateClientSession: Failed to create client session. Error code [0x800706BA].[gle=0x000006ba]"
Error "CallBack_WdsClient_DetectWdsMode: Failed to create client session or initialize WDS unattend. Error [0x800706BA].[gle=0x000006ba]"

Now the weird thing is that I can boot to the capture.wim image (still x86) with no problems, so I did some more research and found out that this data was being blocked at the network...

Looking at some more documentation from Microsoft I see that the following ports must be open for WDS to work (the error message mentioned above was due to port 5040 needed for WDS to create an RPC connection being blocked):

• UDP - 67, 68, 69, 4011
• TCP - 135, 137, 138, 139, 5040

After changing the firewall rules everything started working again.

Great!

---

I hit another snag in the deployment. Now I have an image (52GB Vista Business) which I created overnight (I estimate it took about 5 or 6 hours to capture). I saved the initial WIM file to an external hard drive due to its size and overnight the connection to the server was lost so the image was not moved.

No big deal, I just simply plug the drive into the server go to WDS and import the new image into the new Image Group that I created.

So after this gets done I go and try to pull this image but when I boot the client up into the WDS PE boot environment I do not see any images (I should see two at this point).

Back to the server where I enabled trace logging on all the components with regarding to WDS, these are located in the registry under:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Tracing\

Just look for the keys that start with WDS.

These log files will turn up in %windir%\Tracing\

I found the following errors in the log:

[WdsImgSrv] Error in enumerating images. Error [13].

So I disabled all the images on the server and copied the new image from the USB drive onto the local hard drive then imported it into WDS under my original Image Group.

So this worked, the image pushed down in one and a half hours and everything worked.

Next up is documentation about MDT, AIK and Unattended installation of Vista and Server 2008.

Revisit: Differential Analysis - WDS & DHCP

Tom and I wanted to make a couple statements and clarifications about our post earlier this week. First, we got the environment to function. Read on to find out how.

Purpose
The reason we undertook this project was two fold: primarily to see if it would work, but also because we believe that an established DHCP infrastructure should welcome a separate WDS server -- we believe that WDS should be able to function within the same domain but on a different server than the DHCP service. It should be as simple as specifying which DHCP services work with WDS.


Semi-Functional?
This morning, we were able to get a deployment of Vista running beyond the standstill we had a few days ago. The procedures we described in our previous post were wrong. We had defined DHCP option 67 as boot\x86\pxeboot.com because thats what was on the original WDS+DHCP server.

Wrong DHCP Option 67

This, caused the client to load and execute the files in the following order:

Capture of failing boot session

So, after some more searches on Google, we came across this article. Apparently, we skipped steps 1 and 2 of the Deployment Process (load WDSNBP, validate DHCP packet, and download pxeboot.com). The change we made today (changing DHCP option 67 from boot\x86\pxeboot.com to boot\x86\wdsnbp.com) caused the PXEClient to load and exeucte wdsnbp.com prior to loading and executing pxeboot.com. In fact, we're pretty sure that WDSNBP sends the DHCP request that causes the server to send the ACK with boot\x86\pxeboot.com as DHCP option 67.

Capture of working boot session.

So here are the DHCP settings to define in order to run WDS with a pre-existing DHCP infrastructure.

DHCP Settings to deploy x86 architecture:

• Predefined Option 43 - 010400000000FF
• Custom-made Option 60 - String - PXEClient
• Predefined Option 66 - IP or Hostname of the WDS Server (in our case 10.150.150.1)
  
• Predefined Option 67 - boot\x86\wdsnbp.com
  

Limitations
We're pretty sure that we're losing functionality doing this. Our setup launches TFTP right after the first DHCP transaction -- skipping a request/ACK. (this is what DHCP option 43 accomplishes). It appears that the DHCP request/ACK that we're skipping might be the packet that tells the WDS server which architecture the client is running (x86, x64, ia64).

Capture of original WDS+DHCP transactions

Knowledge
We don't know much about WDS and how its innards function -- we're learning while experimenting. This, most definitely, leads to some dead-ends with respect to progress. Further, when trying to learn about wdsnbp.com, we found Network Bootstrap Program (NBP). However, there isn't a lot of information on exactly what it accomplishes. We assume its some enhanced PXE kernel.

Attention to detail
Because we're barreling through this process, we missed some obvious signs that the change in the filename was the solution. First, the loading dialog on the workstation after it receives an address states its executing WDSNBP.

Contrarily, we assumed that the boot file name (DHCP Option 67) meant something in the first DHCP transaction when WDS and DHCP existed on the same server; this value means nothing in the first DHCP transaction when WDS and DHCP exist on the same server. When we added DHCP Option 43 (PXE subption for mtftp), we instructed our client to immediately download and execute whichever file we specified in DHCP option 67. Apparently this needs to be architecture specific wdsnbp.com.


Quote from the PXE Specification
"Redirection by the Boot Service to a TFTP service on a remote server should not be done as it is not reasonably possible for the redirecting server to know for certain that the TFTP server being redirected to is truly available."

Quote from MS about DHCP & WDS
"Microsoft does not support the use of these options on a DHCP server to redirect PXE clients."

While the PXE developers might not recommend it and Microsoft says the don't support it, we accomplished it (although its architecture specific). Yay.

Differential Analysis - WDS & DHCP Separation

This post outlines the issues and resolution that Tom and I uncovered while removing DHCP from a Windows Deployment Services (WDS) system and moving it to a separate system. The post is rather lengthy, so if you're seeking a solution to this problem, we haven't found one. There is a bulleted list of our take-aways and thoughts so far at the end of this post. The title of this post includes differential analysis because Tom and I compared the functional states of two environments with the non-functional state of our broken system to try to determine a solution.

A month ago, Ron and Tom setup an Active Directory domain to demonstrate the capabilities of WDS. A few weeks ago, Kristian and I added Server 2008 clustering capabilities to the AD environment. Elaboration regarding this environment will happen in the future.

Yesterday, Tom and I wanted to move the DHCP service from the WDS server to the cluster, as to provide highly available DHCP. So we had two servers: one running WDS + DHCP (hereafter referred to as the WDS Server) and another running DHCP (hereafter referred to as the DHCP Server). The goal was to split DHCP and WDS, so we copied the DHCP options from the WDS Server in the picture below to the fresh new DHCP server.

Working Options from WDS Server



Options on the DHCP Server

We rebooted a workstation whose operating system had been deployed from our WDS Server prior to our WDS & DHCP split. The workstation churned along at the PXE screen and then displayed the following PXE error message:

PXE-E55 Proxy DHCP Service did not reply to request on port 4011

Uh oh. We called it a day.

Today, Tom and I revisited the problem by attaching some hubs to our imaging infrastructure and playing the packet capture game. The WDS server is 10.150.150.1 and the DHCP server is 10.150.150.23 -- the following DHCP scope options were configured when the issue was occurring.

Capture of the problem

The packet capture above shows the problem. The workstation going through the PXE process grabs an IP from the DHCP server and then sends a DHCP discover to port 4011 of the DHCP server. (Note that the error we receive on the workstation mentions port 4011.) Then, the DHCP server replies with an ICMP port unreachable message -- an active rejection of the packet.

So, when we noticed this, we knew the problem was going to be getting the workstation to send that second DHCP discover to the WDS server on port 4011 rather than back at the DHCP server. We captured the traffic for a working DHCP + WDS transaction thinking we could compare the working setup with our target setup.

Capture of the working DHCP+WDS transaction

So, we tried mucking with some settings on both the DHCP and WDS servers based on analyzing the differences in the DHCP ACKs from the working (packet #31 - capture of the working DHCP+WDS transaction) and non-working (packet #23 - capture of the problem) captures and no combination of configuration changes led to a different error or a success. Some of the settings we messed with include DHCP Option 54 Server Identifier, Do not listen on port 67, and changing DHCP Option 66 to a non-existent IP address in the working environment to see if the change would break the system.

So we started searching Google some more and came across this Microsoft page. Microsoft tells us, "Important: Microsoft does not support the use of these options on a DHCP server to redirect PXE clients." Well, thanks, but no thanks.

Then we remembered that we have a working pxelinux environment. The pxelinux configuration files are served up by Microsoft's TFTPD and DHCP is offered by Microsofts DHCP 2003 service. Further, the DHCP and TFTP servers are separate! (oh, and IT WORKS)

We decided to setup another capture session, this time monitoring our working pxelinux environment.

Capture of the working pxelinux DHCP+TFTP transaction

Then, Tom expanded the DHCP ACK and noticed DHCP option 43 was used!

DHCP option 43

So, Tom updated the DHCP server settings in our WDS environment accordingly.

Updated DHCP options (working!)

And, voila! The workstation in the WDS environment now directs TFTP GETs to the WDS server right after the DHCP transaction. Cool.

Capture of working target setup

So, it appears that from our experiment and our working pxelinux environment, the presence of DHCP option 43 with a value of 010400000000FF a PXEClient immediately sends a TFTP get to the DHCP option 66 value for the file value listed in DHCP option 67.

We wanted to make sure, so we changed DHCP option 66 to a non-existent IP address, and the workstation failed with the message: PXE-E11 ARP Timeout. A capture of this event showed that the workstation received an address and tried to ARP requested for the non-existent IP address. This led us to further believe our claim about DHCP option 43.

Capture of ARP Timeout

Re-inspection of the expanded DHCP option 43 in wireshark shows the sub-option PXE mtftp IP setting with no value. We're somewhat confused what this sub-option means, although we've already hypothesized and proven what it accomplishes in the PXE environment. A simple Google for PXE Specification finds a document that might contain documentation about what this stuff means.

So, we tried to actually boot into PE 2, but it failed with the message:

WDSClient: There is a problem initializing WDS mode

Suck. The clear difference between our target environment and the working WDS environment is that a second DHCP request/ACK doesn't occur. The ACK in this communication contains DHCP option 252, Proxy Autodiscovery. A few more captures of the working WDS environment proved that this value changes per DORA/RA scenario.

DHCP Option 252

It looks like we'll have to do some more digging into how WDS dynamically creates BCD files, etc. Expect another post regarding our end environment in the future.

Remaining thoughts:

• Do we lose any functionality by removing DHCP from the WDS server and implementing it elsewhere?
• Are there automatic changes to Option 67 by the WDS server?
• Are there other lost functions we don't know about or can't think of now?
• Probably
• The target ending architecture includes WDS outside of the high availability cluster.
• Can we distribute WDS across the cluster nodes, and use network load balancing to make TFTP via WDS highly available in a similar sense as clustered high availability?
• We shall see...
• Could this solve our dynamic BCD creation issues?
  

Lessons of the day:

• Differential analysis -- the comparison of system states -- to solve problems is strong and effective. Not only can it be used in cryptanalysis or other math-oriented problem solving situations, it can be used in system administration. Thankfully, RIT's ANSA degree program taught us how to read packet captures.
  
• Sitebooks are great!
• We had documentation about this DHCP option 43 for our pxelinux environment, but we didn't look at it. In the old documentation, we should have sought to understand what the option accomplished for our pxelinux environment.
• This post is a sitebook!

Procedures:

• To detach DHCP from your WDS server, you need the following options in DHCP options defined in the new DHCP service
• Predefined Option 43 - 010400000000FF
• Custom-made Option 60 - String - PXEClient
• Predefined Option 66 - IP or Hostname of the WDS Server
• Predefined Option 67 - filename in WDS for architecture ( in our case it was boot\x86\pxeboot.com )