How to recover from Windows Server 2003 Event IDs 1168 and 1003

Recently at the office we had a brown-out with devastating consequences.  All our servers are managed by UPSes and on this particular occasion, at the precise moment of this brownout the UPS failed and server lost power abruptly.  This caused some error in Directory Services, becasue after the boot the Windows Server 2003 (and primary domain controller) showed a message during bootup (but after logon) saying “The Active Directory is rebuilding indices”, just before network connection preparations. It hadn’t done this before.

After several hours waiting for this to go away (the server seemed responsive during the process so we left it) nothing changed.  We forcibly reboot the machine and it booted into Safe Mode automatically.

After several reboots and a scheduled CHKDSK /f on bootup, nothing changed, so we booted into Directory Services Repare mode to investigate further.  Now, looking through the event log showed that even more things were going wrong including the computer name being missing the domain being all messed up and an array of other issues, the most worring being several Event Id’s 1168 and 1003.

After a bit of googling I found this page, and based on my own knowledge condensed and modified the solution to work in my case:

  • First, boot the machine into Directory Services Repair Mode
  • Next, open a command window.
  • Then, type the command: cd %SystemRoot%\Ntds
  • Execute the command: Esentutl.exe /p Ntds.dit for me this took 8.3 seconds to execute.

Basically, the esentutl.exe is a utility used to perform various administrative actions on any databases using  Microsoft Proprietary Database format.  The Directory Services Database uses this format.  The ‘/p’ argument tells the utility to perform a “repair” on the second argument, which obviously is the database file.

Warning: The Ntds.dit file is the heart and soul of Active Directory on your domain.  Take extreme care with this file because if you don’t have an adequate backup of the system state on the domain controller, you may irrecoverably corrupt your domain.  This would be bad.

In my case, everything went smoothly, and the domain was back up after a reboot back into “normal” mode.  Obviously, we changed the UPS with a working one.  Who says network admins never get a happy ending?

Configuring Windows Components (Like IIS) on Amazon EC2

After getting my Amazon EC2 imaged created, I quickly discovered that the default image is as bare-as-bare-as-can-be.  They had other default images with SQL Server 2005 Express on it, but I prefered to use a clean Windows customized by yours truely.

Optional Windows Server operating system components are typically added or configured using installation media. This tutorial describes how to add or configure optional Windows components within the Amazon EC2 environment.

Windows Server operating systems include many optional components. Installing all components on each Amazon EC2 Windows AMI is not practical. Instead, you can access the necessary files to configure or install components using Elastic Block Storage (EBS) Snapshots.

The following is a list of available snapshots:

  • Windows 2003 R2 Enterprise 32-bit: snap-bb10f6d2
  • Windows 2003 R2 Datacenter 32-bit: snap-8010f6e9
  • Windows 2003 R2 Enterprise 64-bit: snap-d010f6b9
  • Windows 2003 R2 Datacenter 64-bit: snap-a310f6ca

Simply add the volume with a size of 2GB – using whatever tool or command line you prefer – to set it up and attach it to the instance you want to configure.  It might take a few moments for the instance to detect the volume, but after that, you will then be able to configure the Windows Components Wizard to point to the new volume.