Migrating PKI from Windows 2003 to Windows 2008 R2

Many customers are running into the need for a Windows 2008 or newer PKI infrastructure in order to enroll and auto enroll newer client operating systems like Windows 7, Vista, and Windows 2008 Server.

Actually, many business customers found the lack of certificate support in Vista (without upgrading their CA's later) as one of the reasons it wasn't business ready. With Windows 7 being almost 10 years newer than Windows XP, many business customers are ready for a software refresh and Windows 7 has enough other appealing features to help that decision along.

There are basically two routes to go; in place upgrade or migration. The only time I would attempt an in place is on a VM so that a snapshot could easily be taken and rolled back in the case of a failure. A migration gives a fresh start, but requires some additional time to complete because between steps you need to wait for certs to issue to clients.

Because certificates are fairly sensitive information, I won't post screen caps, but rather overview the process.

Research and Design

Research what your existing CA is in use for. Anything it has issued needs to be either determined to be invalid (expired, not in use, not needed) or documented as something to replicate on the new CA. The other decision on design is around what CA architecture and hierarchy you want or need. Depends on the size and complexity of your organization this can differ greatly. For most organizations under 2000 users, I would say a single CA is sufficient, and if an additional are needed, use the PKI planning guides that Microsoft provides, or better yet, read Komar's 2k8 PKI book.

Implement and Re-Issue certs

Depending on your usage, this could take a long time. Audit existing certificates, revoke the ones that are not in use or expired, and start re-issuing them on the new CA architecture. For larger organizations, this may take months to complete. Luckily, you can choose to have both CA's active. I recommend changing the certificate templates on the old CA to read only, and no longer allow enroll and auto enroll as you migrate each template type successfully, this way, the old CA still validates certificates issued that you haven't updated while you can work on updating them, without any noticeable downtime.

Decommission Legacy CA

The "easy" part for sure. Removing a CA (Unlike uninstalling Exchange) there are no checks or audits to make sure you did everything correctly. If you didn't notice that your Cisco ASA or VPN Concentrator had a certificate issued and miss it, it may cause some issues for you. I recommend stopping and disabling your legacy CA for a few days or even weeks (this depends on your comfort level, and organization) before you make the decision to decommission. Even then, before you decommission, I would also really recommend taking a complete backup of the server.

Virtualized DC issue with time synchronization

This was a pretty simple mistake but it took me a while to figure out. We were noticing everything on the domain was 10 minutes behind other devices we used and in troubleshooting, I configured my PDC emulator to sync with pool.ntp.org using NTP, and it did, and seconds after, the time would revert back.

Of course, the issue here was that the DC was virtualized and Hyper-V time synchronization was taking preference and syncing to the Hyper-V server's local time which had fallen out of sync. The fix here was either disabling time synchronization on my DC, or enabling NTP pool synchronization for my Hyper-V server. I chose the latter, and moments later all machines were the correct time.

Credit where credit is due:
http://www.aperture.ro/index.php/2009/01/windows-time-sync-hyper-v-enabled-domain-controller-dilemma/

Creating an RPC directory on an additional IIS7 Web site for Exchange 2007 Outlook Anywhere

Usually when I get stumped, and I find the fix, I blog about it. It's usually a really good way to capture information that I myself was unable to find (and you would be surprised how often I personally refer back to my own blog entries)

Today, I had a pretty interesting challenge. A customer had an internal FQDN that they did not own on the Internet, and could not get an SSL cert issued for their AD FQDN name on their SAN certificate. Now, I have not run into this issue in a while - the last time was on a Windows 2003 (IIS6) server - this time was a Windows 2008 (IIS7) configuration.

The actual issue of splitting the certificates is a pretty well known and well documented procedure. We used the "default web site" for internal CAS, and secured it with an Enterprise CA signed certificate. I created a "External OWA" site for external CAS, and assigned the Digicert SAN cert to that site. I should also note, the External OWA site listened on a different LAN IP that was ONLY used for the NAT entry with TCP/443 (HTTPS) enabled to it.

Using Powershell, I was able to run (not exact commands, some of these will prompt you for additional required attributes)

• New-OWAVirtualDirectory -WebSiteName "External OWA"
• New-ActivesyncVirtualDirectory -WebSiteName "External OWA"
• New-OABVirtualDirectory -WebSiteName "External OWA"
• New-WebServicesVirtualDirectory -WebSiteName "External OWA"

This got me 95% of the way to having a working CAS server. The only issue - I was missing the /rpc and /rpcwithcert directories that Outlook Anywhere and RPC over HTTPS rely upon. There is no Powershell command for this, as it's not really an Exchange component.

Now, the last time I had to splut internal/external was on Windows 2003, you would back up the virtual directory in question to an XML file, then import it on the other site. This is no longer an option in IIS7.

I also admit, I do not know IIS7 very well, and I attempted to manually recreate the directories by investigating settings and mimicking them. I got pretty close, but the Exchange Remote Connectivity Analyzer was still reporting issues.

Google and Bing really didn't turn up much (rpc, iis7, windows 2008, exchange 2007, are all pretty common search words)

I eventually found this blog entry written by Saurabh Singh related to RPC over HTTP as it related to a TS Gateway issue that he ran into.

And awesomely - it WORKED.

So opening up the applicationhost.config file, I was able to build up the virtual directories and all their settings identically on a second non-default web site. Ran an IISreset, and then re-enabled Outlook Anywhere and everything worked!

If going to Exchange 2007 - Windows 2003 or Windows 2008?

I had a customer ask this very question, and I replied with this. Feel free to comment and add any pros/cons you can think of.

Pros of deploying Exchange 2007 on Windows 2008:

Quicker deployment time � Win 2003 requires IE7, IE8, SP2, and about 55 patches to get it current. This speeds initial deployment, but more importantly in a crisis when you need to redeploy, you won�t need to wait as long for patching procedures. Additionally to this point, 2008 keeps the OS CD installed to a hidden directory unlike 2003 that will prompt for a CD when you add/remove features like IIS. Windows 2008 also has command line deployment for most Exchange pre-req�s so that spin up time is pretty quick.
Longer solution shelf life � Windows 2003 support lifecycle policy is to end support 2 years after the last Service Pack. SP2 for 2003 released in March of 2007. So unless we see SP3, at this point, Windows 2003 is already out of support. We typically attempt to deploy solutions that last the 3-5 years most organizations amortize their hardware.
Less known bugs/exploits available � granted, with newer OS, there is an inherent risk of a newly found exploit being found, but there is a LOT more know about 2003 code now.
Options to change after are limited � if you go 2003 now, and later want to move to 2008, there is no in place upgrade of the OS on an Exchange 2007 server � it would be decomm a server, rebuild as 2008 and repurpose. Far easier to start off at the end point of OS.

Cons of deploying Exchange 2007 on Windows 2008:
Cost - Your first Windows 2008 server means you need to migrate all of your CAL's to Windows 2008. For smaller organizations, or anyone with their CAL's on a Software Assurance plan, this is VERY easy. If this is an unexpected expense, Windows 2003 may be your only path.
Newest stuff - If you are of the "Wait for SP1" mentality when it comes to Microsoft products, you likely will have a hard time with this decision.
Missing skillset - If you have never run Windows 2008, there is a slight learning curve. And I do mean slight - most things are VERY similar, but if admin skills are a concern, sticking to what you know may be appealing.


Again, PLEASE comment, I know this is controversial for some folks.

Windows 2008 Administrator tools for Windows 7

I found this to solve my own problems today, and had to look too hard for it.

Where to Install:
Microsoft Remote Server Administration Tools for Windows 7 Beta (x86): http://download.microsoft.com/download/A/D/4/AD4D3903-E06D-456D-AED4-D53895D2C1A9/Windows6.1-KB958830-x86.msu
Microsoft Remote Server Administration Tools for Windows 7 Beta (x64): http://download.microsoft.com/download/A/D/4/AD4D3903-E06D-456D-AED4-D53895D2C1A9/Windows6.1-KB958830-x64.msu

The main download page is here http://www.microsoft.com/downloads/details.aspx?FamilyID=82516c35-c7dc-4652-b2ea-2df99ea83dbb&displaylang=en

RSAT Client is available to all customers as part of the supplemental Microsoft Software License Terms to Windows 7 licenses.

What Is Included in RSAT?
This is the list of Windows Server 2008 administration tools which are included in Win7 RSAT Client:
Server Administration Tools:
Server Manager Role Administration Tools:
� Active Directory Certificate Services (AD CS) Tools
� Active Directory Domain Services (AD DS) Tools
� Active Directory Lightweight Directory Services (AD LDS) Tools
� DHCP Server Tools
� DNS Server Tools
� File Services Tools
� Hyper-V Tools
� Terminal Services Tools
Feature Administration Tools:
� BitLocker Password Recovery Viewer
� Failover Clustering Tools
� Group Policy Management Tools
� Network Load Balancing Tools
� SMTP Server Tools
� Storage Explorer Tools
� Storage Manager for SANs Tools
� Windows System Resource Manager Tools

UPDATE - For Windows 7 RTM - go here

Impact of ESX snapshot backups on Microsoft database servers

Up until Update 2, ESX 3.5 uses a VMWare tool called the Sync Manager as part of the snapshot backup process. The Sync manager quiesces the file system (pauses all incoming I/O requests and dumps dirty data to disk) before the snapshot backup is taken. This allows the backup to file-system consistent.

If you were to take a snapshot without pausing the I/O requests and then restore the snapshot, the virtual machine will start up for the first time thinking that it is recovering from a power failure type of crash. This is because the recovered system will not be able to find the I/O requests that were stored in the memory (RAM) at the time the snapshot was taken.

What does this have to do with database servers? For servers housing databases (Active Directory, Exchange, or SQL servers), stopping I/O requests with the Sync manager halts incoming requests to the database without notifying the database of what is happening. The database is waiting on that information to arrive � so when the data doesn't come in when expected, errors are logged to the event log and in some cases, the databases become corrupt. I happened to find this out on an active directory domain controller, and the sequence of errors looks like this:

First, you'll see event ID 1 logged with source LGTO_Sync. This is the Sync driver starting to do its work quiescing the file system.

On domain controllers, AD requests will begin to fail. The description will differ based on the request, but the Event ID stays the same.

For domain controllers running DNS, dynamic updates will fail as well:

For DHCP servers, you will see this:

On Exchange servers, you'll see Autodiscover errors:

If you are seeing these errors, stop using the sync manager now. Eventually you will corrupt your database.

Workaround
Stop quiescing guest database servers before taking snapshots, and start adding snapshots of virtual machine memory to your backups. Most backup applications allow you to do this. If yours doesn't, you can script it using vimsh.

Example:
vimsh -n -e "vmsvc/createsnapshot [VmId] [snapshotName] [snapshotDescription] [includeMemory]"

vimsh -n -e "vmsvc/createsnapshot XXXX FIRST_SNAPSHOT MY_FIRST_SNAPSHOT_1"

By taking a snapshot of the guest machine's memory, you are creating a full snapshot. When you restore, you restore the memory on top of the file system. When the machine starts, it will be able to access all the necessary information in memory to start normally - no crash.

Resolution
To combat the Sync Manager problem, ESX has released update2, which includes an ESX VSS tool that integrates with Microsoft VSS. It works by using the windows operating system to hold I/O requests, eliminating the need for the sync driver. When the operating system is in charge of halting its own I/O activity, the databases are notified that a backup is taking place. The databases can then pause their own processing of requests, and no errors occur.

This update is relatively new, and many third-party backup applications do not support update 2 yet, which is why I have offered the workaround here.

One last note about Microsoft domain controllers and Vmware snapshot backups
In 2006 (revised in Dec. 2008), Microsoft released KB 888794 (http://support.microsoft.com/kb/888794/en-us), which states that

"Active Directory does not support any method that restores a snapshot of the operating system or the volume the operating system resides on. This kind of method causes an update sequence number (USN) rollback. When a USN rollback occurs, the replication partners of the incorrectly restored domain controller may have inconsistent objects in their Active Directory databases. In this situation, you cannot make these objects consistent. "

In reality, the BURFLAGS registry referred to in Microsoft KB290762 (http://support.microsoft.com/kb/290762) can be set so that the virtual DCs are nonauthoritative, and an existing domain controller can be set to authoritative. This will allow the USN to be overwritten by the authoritative domain controller, and no USN rollback will occur.

Setting up TS Gateway

TS Gateway on Windows 2008 is a solution that allows one to connect to resources on a remote terminal server without using a VPN connection. It connects a client to the remote resource using port 443 and can be used in conjunction with TS Web Access or TS RemoteApp. Traffic is encrypted using TLS 1.0. There are three ways to deploy TS gateway: for use with Network Access Protection (NAP), ISA server, or by itself. I will address the NAP scenario here.

Step 1: Install the TS Gateway role service. From server manager, click "add roles" and add the terminal services role. On the "select role services" screen, select TS Gateway. Allow Server Manager to install the additional required role services as well (RPC over HTTP, IIS 7, NPS).

Step 2: Configuring Certificates in TS Gateway. Once you have added the appropriate role services, you will need to obtain a certificate for use with TS Gateway. The certificate can be self-signed, or you can use certutil to create a certificate request for a third-party certification authority. If you choose a third-party certificate, you'll want to make sure the vendor participates in the Microsoft Root Certificate Program so that the certificate is automatically trusted by clients.

With the self-signed certificate, each client computer connecting to the terminal server will need to add the certificate to the trusted root certification authorities store for their user account, either manually or through group policy.

The common name of the certificate should match the external DNS name of the TS Gateway server.

Once you have your certificate, install it in the personal store for the computer account on the TS Gateway server. Now open the TS Gateway Manager from Administrative Tools, right click the server name in the right-hand pane, and go to properties. On the SSL Certificate tab, select an existing certificate and point it to the location of your new cert.

Step 3: TS-RAP and TS-CAP policies. Before clients can connect using TS Gateway, you must set up two policies: Terminal Services Connection Authorization Policies (TS-CAPs) define who is allowed to connect to a TS Gateway server. You can specify either local or Active Directory user groups who are allowed (or denied) access to terminal services, and decide which devices can be redirected when connecting to TS Gateway. You can also specify what authentication method you want the client to use � password or smartcard.

Terminal Services Resource Authorization Policies (TS-RAPs) identify which network resources users can connect to using the TS Gateway server. You can create TS-Gateway managed computer groups, or use Active Directory defined user groups to create a TS-RAP policy.

You will be prompted to create at least one TS-RAP and TS-CAP policy as part of the initial TS Gateway configuration. Creating TS-RAP and TS-CAP policies:

1. Enter a name for the TS-CAP policy
   

   
   

   
   

2. Choose the authentication method you want clients to use to connect, then add allowed user groups or even computer groups that are allowed to connect to the server.
   

   
   

   
   

3. Choose the which devices are allowed to redirect from the client:
   

   
   

   
   

4. Review the summary and click Finish (here I am creating both a TS-CAP and a TS-RAP, so I don't have a finish option yet.)
   

   
   

   
   

   Creating a TS-RAP:
   

5. Enter a name for the TS-RAP policy:
   

   
   

   
   

6. Choose which user groups the TS-RAP will apply to:
   

   
   

   
   

7. Here you can specify which resources clients are allowed to connect to using either active directory security groups (computer objects), or TS-Gateway managed computer groups.
   

   
   

   
   

8. Choose which port RDP should run on. I will leave the default (remember, TS Gateway operates over port 443 on the internet. 3389 only needs to be open internally)
   

   
   

   
   

9. Review the configuration and click Finish.
   

   
   

   
   

   ****The policies work just like the old IAS policies in 2003 � order matters!****
   

   Step 4: Configuring the Client for Connection to TS Gateway
   

   
   

   First, you must ensure that you have purchased a trusted third-party root certificate, or that you have installed the self-signed certificate either manually or through group policy into the Trusted Root Certification Authority store for the client's user account.
   

   
   

   Also, the client must be running at least Windows XP SP3 or Windows Vista � make sure you have at least RDP 6.1.
   

   
   

   Now the RDP client should be able to automatically detect the TS Gateway settings, but for me, it takes longer to connect every time when the RDP client has to search for the settings. So I would rather specify manually in the "Advanced" tab of the Remote Desktop Client:
   

   
   

10. Open the client, expand options, and go to the Advanced tab. Click "Settings" under the Connect From Anywhere section
    

    
    

    
    

11. For server name, enter the same name used on the common name of the TS Gateway certificate (also the DNS name of the TS Gateway server):
    

    
    

    
    

12. Select the computer you want to connect to, and off you go!
    

    
    

    
    

Really need a browser in 2008 core? Use Lynx!

I decided I really was tired of having to go to another machine to manage downloads for my 2k8 core machine, so I decided to find a way around it.

I found an old Lynx port that seems to work WONDERS on Windows 2008 Core!

http://pervalidus.50webs.org/cygwin/lynx/ (the download link is the first word on that page, I missed it)

I really like the new DCPromo for RODC's!

I accidentally tried to dcpromo with the same .inf twice, and found that it removes the password fields immediately after you attempt a dcpromo! Pretty slick!

Hyper-V - Code 12 on Virtual Machine Bus and Integration Tools not working

Stumbled upon this and found the fix here:

http://blogs.technet.com/jhoward/archive/2008/02/29/vmbus-fails-to-load-device-cannot-find-enough-free-resources-code-12-on-a-windows-server-2008-x86-virtual-machine-under-hyper-v.aspx

(2/29? Yea, so this is OLD news, but a nice fix either way!)

Was getting this error:

Tada!