I was in at work last night testing a Domain Controller upgrade project that has gone horribly wrong. We attempted this routine procedure a couple of months ago, and although this upgrade has been performed thousands of times for many different companies, it was our screwed-up computing environment that caused problems. When the upgrade was complete, none of our dotNET web applications no longer functioned, so the changes had to be rolled back and an investigation opened. Between the software supplier (the BIG one), Vendor #1 (web servers are outsourced to them), Vendor #2 (DC servers/infrastructure are outsourced to them), and my company, this investigation has been going on FAR too long. Last night we were supposed to run a bunch of network traffic captures and logs of various sorts, and send them to Software Supplier for analysis. The process was taking much longer than anticipated, and by 3am, we were 2 hours behind schedule on the project plan. Mission was aborted, so now I have the joyous anticipation of doing this again some weekend in the near future.
As an aside to this, we have a web server (good ol' X) that appears to randomly crash and reset itself several times a week, if not every day. This has been going on for almost six months and no one can figure out what the problem is. Luckily we've moved most of our high-volume, business crucial applications onto a more stable server. We have this running inside joke that we should mimic the scene from 'Old Yeller' where the boy goes out to the barn with the shotgun to finish off the rabid dog:
"Don't worry mama, it's my server, I'll do it".
I hate you web server...