
I found this rather interesting and thought it was time to toss up some reasoning behind how/why FastServers.Net does patch management in a 'delayed' type fashion. As most of you know, the second Tuesday of each month is called "Black Tuesday" by the Microsoft IT professionals. The reason for this is the multitude of security update/hotfixes/patches that are deployed for all of the supported Microsoft operating systems, web browsers, office applications, and yada yada yada. Sometimes the patches get so convoluted that Microsoft has to re-release the patch/hotfix the next month to address flaws that they forgot to fix in the first place.
So earlier this week we had a customer who wanted to update to SQL Server 2005 Service Pack 2 and wanted to know how we would go about installing it. I quickly explained to the customer that this service pack was not currently supported or being deployed by FS as we are still conducting testing and awaiting the "all clear" from the SQL community before releasing. Then came the battery of questions...."Well why does FastServers.Net wait so long in deploying updates and fixes that Microsoft releases. This just doesn't make any sense that you would wait to deploy a service pack that Microsoft releases for one of their products. Install it now!". WHOA! Slow down there mister man! I quickly explained to our customer that with the multitude of environments that we have, we do extensive testing to ensure any new roll out of security updates goes over each and every environment prior to release. In addition, the vendor for our patch management solution does their own internal testing of security updates before making them available for our master distribution manager. After hearing all of this, dejected and a bit upset, the customer agrees and we promise to notify him when we are moving forward in pushing out the updates.
Fast forward to 30+ hours later and VOILA....vindicated: http://support.microsoft.com/kb/933508
That's right folks. Leave it to the good 'ol boys at Microsoft to release an update to a recently deployed service pack. Hot off the presses this morning is Service Pack 2a for SQL Server 2005. YES! They are bringing back the 'a' in their service pack releases; remember SQL Server 2000 Service Pack 3a -- how touching (sniff sniff). So I notified our customer of the new release, along with a few others who have been requesting the installation of the recent service pack, and he was delighted that he waited. Vindicated? Nah, its not what I was looking for. Looking more for understanding in how we handle distribution of our updates.
So what to take from this? Well here is a list that you should follow:
- Just because Microsoft, or any other OS vendor, releases security updates/hot fixes/service packs for new vulnerabilities, take a second to review what the updates are addressing
- Verify whether or not the updates truly apply to your environment
- TEST before deploying to production -- never roll out updates to a production server without first testing
- Verify before installing -- disk space, server health, security audit, and virus scan
- Apply ALL the updates that are pertinent to your solution -- more than likely you are not using Media Player on your web server, so disable the application and ignore the updates for this
- Reboot after applying -- this is your true test to see if the updates were applied correctly...don't wait to reboot, even if one is not required after the patch install, reboot anyway
- Verify after the reboot -- check to ensure all services/operations are running before logging off and calling it a day
Yes there are a lot of steps, but taking these steps now can mean some very peaceful nights in the future; i.e. not having to stay up during maintenance periods for 10-12 hours when an additional 30 minutes of work could have prevented it all. Hopefully this will clear up and answer a plethora of questions of why FastServers decides to wait rather than deploy. And if there are ever any further questions on this or any other patch management question, drop me a email for additional clarification or tips.
Spread the Word: Click below to share this with the rest of the world