"Too Important to Patch" - Wait? What?

Published: 2011-07-06. Last Updated: 2011-07-06 02:11:20 UTC
by Rob VandenBrink (Version: 1)
24 comment(s)

I recently had a routine "can you help our business partner" type call from a client. Their business partner could receive email from them, but could not send email to them.

After a bit of digging in the SMTP header of a failed note, it turned out that the business partner was running a very old version of QMAIL, which has a problem with ESMTP and DNS responses larger than 512 bytes. My client (the destination for the email) had recently gone to an email scanning service, so the total return on an MX record request was well over 1.5kb.

So far, not so exciting, you say - patch the server and be done with it! So why am I writing this up on isc.edu?

This is where it gets interesting. I called the business partner, and their verbatim response was "Gee, I don't know. Applying that patch will involve taking the mail server down, our CEO won't approve that. Is there some other way to do this?"

Wait, what? Did I hear that right? Let me check my watch - what century is this again? This is a patch from 2007 for goodness sake! I can see needing to follow a change control procedure, schedule an outage, maybe for after-hours, but they are an application development shop, not the Department of Defense! If they're running a mail system that hasn't been patched in 4 years, chances are that someone else already owns them, and they've got bigger problems than just this.

Anyway, after a frank and honest (and tactful, though that part was a bit more difficult) discussion, they did apply the needed patch, along with a truckload of other system updates that had been delayed since forever.

I've encountered a few situations where it makes some snse for system admins to defer patching for extended periods of time:

Servers that support military personnel in active operations are often mandated by policy as "frozen". In our current global environment, these freeze periods can extend into months and years.

Servers that support long-range space exploration missions will often end up running operating systems that are no longer supported, on hardware that has been end-of-lifed years ago, or on hardware or OS's that were one-shot custom efforts. In cases like this, the hardware is generally air-gapped or otherwise isolated from sources of attack.

Some servers in support-challenged situations might also be "frozen" for specified periods of time - if I remember correctly, the servers in some of the Antarctic missions (really, no pun!) are in this category. (If I'm mistaken on this example, I know that sysadmin for those systems is a reader, please correct me!)

 

So the question I have for our readers is: What situations or applications have you seen that might defer patches and updates for an extended periods of time? Did you consider those reasons or policies to be legitimate? Did you come up with a compromise or workaround to get patches applied, or did you have to follow policy and not apply updates? Did this end up with a system compromise, and if so, did the policy protect the system administrator, or did they end up taking the blame anyway?

I'm really looking forward to feedback from our readers on this, please use the contact form to let us know what you've seen!

 

===============

Rob VandenBrink Metafore

Keywords: patching what what
24 comment(s)

Comments

> What situations or applications have you seen that might defer patches and updates for an extended periods of time?
- Ignorance and/or arrogance (It's never happened to us before, so it won't happen now).
> Did you consider those reasons or policies to be legitimate?
No.
> Did you come up with a compromise or workaround to get patches applied, or did you have to follow policy and not apply updates?
- Followed their decision/policy. It's not my network - they are the owners.
> Did this end up with a system compromise, and if so, did the policy protect the system administrator, or did they end up taking the blame anyway?
- No - they lucked out. But you know who the fall guy is...
.
If the up time of a mail server is that important, it sounds like they need a 2nd mail exchanger and/or a cluster for mail box storage and access. Sure this is a more complicated setup but as a server admin I'd rather spend some extra time on design and setup to make maintenance more convenient for me and non-disruptive for those who rely on the services provided.
I have customers using radio dispatch consoles that are still on WinXP SP1 with no antivirus.

However these consoles are not permitted to see the world and no one is allowed access to the boxes besides me. So while I worry that someone will come along and plug an infected USB stick into the box I have to hope that the rules and locks are enough...
On a second note...

My little company has a backup email server running HMail at another location to avoid loosing incoming email... Would this option not suffice for these guys for an overnight cutover?? Yikes!
We're currently going through a move of our entire IT infrastructure from one provider to another, and there has been a change freeze instituted on all servers for the duration of the move (6 months).
Assuming this mail-server is too important to take it down for some minutes, they should have various fall-back solutions. What about patching these fall-backs before patching the live system? So there is just a downtime for hopefully less than some seconds. Of course they have also some time to test the patched fall-back-system.. I've no sympathy for such minds...
To the original question...i think the rule of "if it isnt broke, dont fix it" applies to most orgs, sadly. The feedback I've gotten from peers in the industry is, why risk patching and rebooting something that is working without issue. It doesn't make sense to me (us security proffesionals), but it does to them.
Re: PC Tech referring to the "fall guy."

There is a reason for an email trail of you explaining the importance of the updates and what could happen if they do not do it. You keep all correspondence with your explanation to why the patches are important and their "no" responses. This is CYA so WHEN they do get hacked they cannot say it is your fault and any attempt to do so could make them libel.
Medical systems.
Either due to FDA controls on the system/app OR vendor's own very odd (read broken) requirements.

Then there's the "exclude the following directories from virus scanning" requirements.
so in this day and age of patch flash player 3x per week, patching is a bit different. The processes should be the same, and the fact is most small companies do not have the resources to keep up, especially if they were to follow the proper sdlc/ change control processes.

Unfortunately, patches are just as good as the software they fix, and have unintended side effects like breaking applications. And as we know, it is all about the apps!

So, scarce resources will be applied to the big problems, like how to allow cool device of the day to work with exchange. Issues like what version of qmail is running where and whether or not it should be patched will always get fiftg billing or less on the priority scale ... Until something breaks.

So, in conclusion. Testing patches properly is risky. Failure to do so means that when some app breaks, it is a sin for which there may be no forgiveness. Instead, make the cool new toys work. Anything that breaks doing that will be forgiven. Wait for the dull stuff to break on its own. That is the cmm level 1 patching process in brief.

Moving to a hygienic system management process involves discipline and management interest, too characteristics that are often hard to find in the chaotic world of systems administration, imho.

Diary Archives