Scripting Mistake

This evening, a scripting mistake led me to learn about a feature in Systintenals' Process Explorer that I had not previously known about. I subscribe to Mark's blog (listed in the System Administration blogs) and he writes a series titled, "The Case of..." where he details his troubleshooting process for a specific issue. In his post, The Case of the System Process CPU Spikes, he demonstrated pretty cool uses of Process Explorer -- since I read that post in April, I've been running Process Explorer in my little tray just like he described in his post. Occasionally I find uses for it like closing stale processes -- but I've never REALLY used it until tonight.

Earlier, I wanted utilize robocopy to synchronize my local caches of the Systinternals' tools from file://live.sysinternals.com/. I maintain two copies: one on my OS drive and one on my thumb drive. My purpose for this is is that I want these tools immediately available when I want to run them -- and while the live share is nice, it takes a few seconds to load.

So I wrote a simple batch script; and made a rudimentary scripting mistake: I wrote a batch script titled robocopy.bat and attempted to call the robocopy executable without specifying it's full path. Here is my original script:

So, this mistake led to some Windows Command Processor crashes:

Interesting right? The infinite loop caused CMD.exe to crash (I guess it's time to install PowerShell and actually read those powershell books on my bookshelf.) When CMD crashed, I went to edit the batch script but wasn't able to save modifications:


OK. I figured that cmd.exe had a lock on the handle -- and I remembered that OpenedFilesView wouldn't work in Vista x64, I didn't feel like downloading a utility and I certainly didn't feel like logging out.

Somehow, I ended up searching for the file handle with Process Explorer's handle or DLL search feature.

After closing the handle, I was able to save modifications to the batch script.

Finally, I fixed my script to call the robocopy executable by it's absolute full path:

C:\Windows\System32\Robocopy.exe


I'm glad I learned about the file handle search feature in Process Explorer -- this utility is incredible. Thanks Sysinternals.

Revisit: Differential Analysis - WDS & DHCP

Tom and I wanted to make a couple statements and clarifications about our post earlier this week. First, we got the environment to function. Read on to find out how.

Purpose
The reason we undertook this project was two fold: primarily to see if it would work, but also because we believe that an established DHCP infrastructure should welcome a separate WDS server -- we believe that WDS should be able to function within the same domain but on a different server than the DHCP service. It should be as simple as specifying which DHCP services work with WDS.


Semi-Functional?
This morning, we were able to get a deployment of Vista running beyond the standstill we had a few days ago. The procedures we described in our previous post were wrong. We had defined DHCP option 67 as boot\x86\pxeboot.com because thats what was on the original WDS+DHCP server.

Wrong DHCP Option 67

This, caused the client to load and execute the files in the following order:

Capture of failing boot session

So, after some more searches on Google, we came across this article. Apparently, we skipped steps 1 and 2 of the Deployment Process (load WDSNBP, validate DHCP packet, and download pxeboot.com). The change we made today (changing DHCP option 67 from boot\x86\pxeboot.com to boot\x86\wdsnbp.com) caused the PXEClient to load and exeucte wdsnbp.com prior to loading and executing pxeboot.com. In fact, we're pretty sure that WDSNBP sends the DHCP request that causes the server to send the ACK with boot\x86\pxeboot.com as DHCP option 67.

Capture of working boot session.

So here are the DHCP settings to define in order to run WDS with a pre-existing DHCP infrastructure.

DHCP Settings to deploy x86 architecture:

• Predefined Option 43 - 010400000000FF
• Custom-made Option 60 - String - PXEClient
• Predefined Option 66 - IP or Hostname of the WDS Server (in our case 10.150.150.1)
  
• Predefined Option 67 - boot\x86\wdsnbp.com
  

Limitations
We're pretty sure that we're losing functionality doing this. Our setup launches TFTP right after the first DHCP transaction -- skipping a request/ACK. (this is what DHCP option 43 accomplishes). It appears that the DHCP request/ACK that we're skipping might be the packet that tells the WDS server which architecture the client is running (x86, x64, ia64).

Capture of original WDS+DHCP transactions

Knowledge
We don't know much about WDS and how its innards function -- we're learning while experimenting. This, most definitely, leads to some dead-ends with respect to progress. Further, when trying to learn about wdsnbp.com, we found Network Bootstrap Program (NBP). However, there isn't a lot of information on exactly what it accomplishes. We assume its some enhanced PXE kernel.

Attention to detail
Because we're barreling through this process, we missed some obvious signs that the change in the filename was the solution. First, the loading dialog on the workstation after it receives an address states its executing WDSNBP.

Contrarily, we assumed that the boot file name (DHCP Option 67) meant something in the first DHCP transaction when WDS and DHCP existed on the same server; this value means nothing in the first DHCP transaction when WDS and DHCP exist on the same server. When we added DHCP Option 43 (PXE subption for mtftp), we instructed our client to immediately download and execute whichever file we specified in DHCP option 67. Apparently this needs to be architecture specific wdsnbp.com.


Quote from the PXE Specification
"Redirection by the Boot Service to a TFTP service on a remote server should not be done as it is not reasonably possible for the redirecting server to know for certain that the TFTP server being redirected to is truly available."

Quote from MS about DHCP & WDS
"Microsoft does not support the use of these options on a DHCP server to redirect PXE clients."

While the PXE developers might not recommend it and Microsoft says the don't support it, we accomplished it (although its architecture specific). Yay.

Thinclient and Network Booting

A few months ago, an acquaintance pitched an idea about an authenticated network boot environment and pointed me to emBoot. I downloaded their trial winBoot software and never used it. I revisited the site to read about recent updates and I stumbled upon two interesting utilities that I missed a few months ago: SimplyRDP and sanFly.

SimplyRDP - this utility uses PXE to boot into a small OS that just runs an RDP/TS client.

netBoot/winBoot - these utilities enable PXE clients to boot from an iSCSI target (netBoot works for Windows 2000, XP and Linux; winBoot works with Windows Vista, 2003 and 2008).

sanFly - enables the creation and management of iSCSI targets in Windows XP, Vista, 2003 and 2008. sanFly is available for download at no cost, but additional functionality can be unlocked by purchasing a license key (emBoot)

I'll report back when I've had a chance to play with the utilities or expand the idea of secure network booting.

Differential Analysis - WDS & DHCP Separation

This post outlines the issues and resolution that Tom and I uncovered while removing DHCP from a Windows Deployment Services (WDS) system and moving it to a separate system. The post is rather lengthy, so if you're seeking a solution to this problem, we haven't found one. There is a bulleted list of our take-aways and thoughts so far at the end of this post. The title of this post includes differential analysis because Tom and I compared the functional states of two environments with the non-functional state of our broken system to try to determine a solution.

A month ago, Ron and Tom setup an Active Directory domain to demonstrate the capabilities of WDS. A few weeks ago, Kristian and I added Server 2008 clustering capabilities to the AD environment. Elaboration regarding this environment will happen in the future.

Yesterday, Tom and I wanted to move the DHCP service from the WDS server to the cluster, as to provide highly available DHCP. So we had two servers: one running WDS + DHCP (hereafter referred to as the WDS Server) and another running DHCP (hereafter referred to as the DHCP Server). The goal was to split DHCP and WDS, so we copied the DHCP options from the WDS Server in the picture below to the fresh new DHCP server.

Working Options from WDS Server



Options on the DHCP Server

We rebooted a workstation whose operating system had been deployed from our WDS Server prior to our WDS & DHCP split. The workstation churned along at the PXE screen and then displayed the following PXE error message:

PXE-E55 Proxy DHCP Service did not reply to request on port 4011

Uh oh. We called it a day.

Today, Tom and I revisited the problem by attaching some hubs to our imaging infrastructure and playing the packet capture game. The WDS server is 10.150.150.1 and the DHCP server is 10.150.150.23 -- the following DHCP scope options were configured when the issue was occurring.

Capture of the problem

The packet capture above shows the problem. The workstation going through the PXE process grabs an IP from the DHCP server and then sends a DHCP discover to port 4011 of the DHCP server. (Note that the error we receive on the workstation mentions port 4011.) Then, the DHCP server replies with an ICMP port unreachable message -- an active rejection of the packet.

So, when we noticed this, we knew the problem was going to be getting the workstation to send that second DHCP discover to the WDS server on port 4011 rather than back at the DHCP server. We captured the traffic for a working DHCP + WDS transaction thinking we could compare the working setup with our target setup.

Capture of the working DHCP+WDS transaction

So, we tried mucking with some settings on both the DHCP and WDS servers based on analyzing the differences in the DHCP ACKs from the working (packet #31 - capture of the working DHCP+WDS transaction) and non-working (packet #23 - capture of the problem) captures and no combination of configuration changes led to a different error or a success. Some of the settings we messed with include DHCP Option 54 Server Identifier, Do not listen on port 67, and changing DHCP Option 66 to a non-existent IP address in the working environment to see if the change would break the system.

So we started searching Google some more and came across this Microsoft page. Microsoft tells us, "Important: Microsoft does not support the use of these options on a DHCP server to redirect PXE clients." Well, thanks, but no thanks.

Then we remembered that we have a working pxelinux environment. The pxelinux configuration files are served up by Microsoft's TFTPD and DHCP is offered by Microsofts DHCP 2003 service. Further, the DHCP and TFTP servers are separate! (oh, and IT WORKS)

We decided to setup another capture session, this time monitoring our working pxelinux environment.

Capture of the working pxelinux DHCP+TFTP transaction

Then, Tom expanded the DHCP ACK and noticed DHCP option 43 was used!

DHCP option 43

So, Tom updated the DHCP server settings in our WDS environment accordingly.

Updated DHCP options (working!)

And, voila! The workstation in the WDS environment now directs TFTP GETs to the WDS server right after the DHCP transaction. Cool.

Capture of working target setup

So, it appears that from our experiment and our working pxelinux environment, the presence of DHCP option 43 with a value of 010400000000FF a PXEClient immediately sends a TFTP get to the DHCP option 66 value for the file value listed in DHCP option 67.

We wanted to make sure, so we changed DHCP option 66 to a non-existent IP address, and the workstation failed with the message: PXE-E11 ARP Timeout. A capture of this event showed that the workstation received an address and tried to ARP requested for the non-existent IP address. This led us to further believe our claim about DHCP option 43.

Capture of ARP Timeout

Re-inspection of the expanded DHCP option 43 in wireshark shows the sub-option PXE mtftp IP setting with no value. We're somewhat confused what this sub-option means, although we've already hypothesized and proven what it accomplishes in the PXE environment. A simple Google for PXE Specification finds a document that might contain documentation about what this stuff means.

So, we tried to actually boot into PE 2, but it failed with the message:

WDSClient: There is a problem initializing WDS mode

Suck. The clear difference between our target environment and the working WDS environment is that a second DHCP request/ACK doesn't occur. The ACK in this communication contains DHCP option 252, Proxy Autodiscovery. A few more captures of the working WDS environment proved that this value changes per DORA/RA scenario.

DHCP Option 252

It looks like we'll have to do some more digging into how WDS dynamically creates BCD files, etc. Expect another post regarding our end environment in the future.

Remaining thoughts:

• Do we lose any functionality by removing DHCP from the WDS server and implementing it elsewhere?
• Are there automatic changes to Option 67 by the WDS server?
• Are there other lost functions we don't know about or can't think of now?
• Probably
• The target ending architecture includes WDS outside of the high availability cluster.
• Can we distribute WDS across the cluster nodes, and use network load balancing to make TFTP via WDS highly available in a similar sense as clustered high availability?
• We shall see...
• Could this solve our dynamic BCD creation issues?
  

Lessons of the day:

• Differential analysis -- the comparison of system states -- to solve problems is strong and effective. Not only can it be used in cryptanalysis or other math-oriented problem solving situations, it can be used in system administration. Thankfully, RIT's ANSA degree program taught us how to read packet captures.
  
• Sitebooks are great!
• We had documentation about this DHCP option 43 for our pxelinux environment, but we didn't look at it. In the old documentation, we should have sought to understand what the option accomplished for our pxelinux environment.
• This post is a sitebook!

Procedures:

• To detach DHCP from your WDS server, you need the following options in DHCP options defined in the new DHCP service
• Predefined Option 43 - 010400000000FF
• Custom-made Option 60 - String - PXEClient
• Predefined Option 66 - IP or Hostname of the WDS Server
• Predefined Option 67 - filename in WDS for architecture ( in our case it was boot\x86\pxeboot.com )
  

Server 2008 or Vista MINWINPC

From Windows Server 2008 Administrator's Pocket Consultant by William R. Stanek (yes, it's on Books 24x7):

Using the Command Line During Installation
When you access a command prompt from within Setup, you access the MINWINPC environment used by Setup to install the operating system. During installation, on the Where Do You Want To Install Windows page, you can access a command prompt by pressing Shift+F10. As Table 2-4 shows, the mini Windows PC environment gives you access to many of the same command-line tools that are available in a standard installation of Windows Server 2008.

So, after you click Install Now -> You can press Shift+F10 and you'll be dropped to a command shell.

The burning question I have is whether this can be used to reset local users passwords?

Windows FSA Tools

FSA = Forensic and System Administration

I subscribe to Harlan's blog, and recently he has made three separate posts about tools; read them. I was poking around NirSoft (linked from Harlan's blog), and stumbled upon some more tools I feel should be highlighted:

IE PassView: Dumps passwords stored in IE (works with IE7 in Vista!)

USBDeview: Extracts the USB device information stored in the Windows registry from a live system

RecentFilesView: Lists the recently accessed files from a live Windows system

Note: Harlan's RegRipper accomplishes similar feats (usb history, recent files, etc) in an offline fashion, from registry hives on acquired drives or read-only drives.

Note: OpenedFilesView doesn't work on Vista x64

Have fun!

Welcome

The purpose of this blog is to document mishaps, misconfigurations, and general mis-steps during implementation of systems relating to information sciences. Over the years, I've been documenting how to properly setup environments; but, these past few months have made me realize that I've always missed a critical portion of the documentation process: the failures.

This blog is dedicated to archiving solutions for issues large or small that come about during information system configuration. The acronym, NSSA, in the title of the blog means: Networking, Security, and Systems Administration. The name NSSA Documentation was chosen because I couldn't think of a better name or acronym that would cover all the categories and because the blog's original creators attended RIT in the NSSA department.


Please email me at Jason.Robert.Koppe@therealgmaildomain.com you'd like to become a blog contributor.