Threat Level: green Handler on Duty: Didier Stevens

SANS ISC: Do we need test procedures in our companies before implementing Antivirus signatures? SANS ISC InfoSec Forums

Watch ISC TV. Great for NOCs, SOCs and Living Rooms: https://isctv.sans.edu

Sign Up for Free!   Forgot Password?
Log In or Sign Up for Free!
Do we need test procedures in our companies before implementing Antivirus signatures?

We have heard a couple of cases regarding problems caused my faulty antivirus signature files.Most recend that has a worldwide impact was the Microsoft Antivirus treating code from google webpage as virus. In 2010, Mcafee deployed DAT 5958 which identified svchost.exe as a virus, deleting it an causing loose of network access. In April 2011, Mcafee deployed DAT 6329, which caused disruption in SAP telephone connectivity functionality as it recognized spsgui.exe with virus. Also deployed DAT 6682, which caused system crash in GroupShield Exchange (MSME), GroupShield Domino, and McAfee Email Gateway.

Yesterday, we received report from reader John stating that computers with DAT 6807 installed got conectivity problems. Today Mcafee confirmed this to be a problem if you are using VSE 8.8.x and have DAT 6807 or 6808 installed. Their recommendation is to go straight to DAT6809.

Other antivirus programs like AVG also deploys faulty updates. Since these events are becoming a worrying trend, should we implement test procedures inside our organizations as we do with other updates like the ones deployed by Microsoft with Windows Update? Implementing a faulty update has a high risk to the organization and its solution consumes considerable time and substantial resources. I am considering implementing such procedure for my company.

Do you think it's necessary to implement such procedure in your company? Let us know!

Manuel Humberto Santander Peláez
SANS Internet Storm Center - Handler
Twitter:@manuelsantander
Web:http://manuel.santander.name
e-mail:msantand at isc dot sans dot org

Manuel Humberto Santander Pelaacuteez

188 Posts
ISC Handler
A similar problem for software developers is when antivirus software running at a customer sites quarantines an essential file and prevents the software from running. Antivirus makers typically do allow companies to submit malware-free reference files that should not be detected, but submitting these to all antivirus makers for all software and revisions can be tedious.
Anonymous
I don't know about other organizations, but we barely even have the time and resources for testing tons of OS and application patches before rolling them out. Who'd have time to do this for every rollout of a new DAT file? Anti-virus tools are already reactive in nature and (in my experience) new fingerprint rules are anywhere from a week to a month behind the appearance of new malware. Delaying this even further with continous testing would make anti-virus tools even less effective than they currently are (and they're a band-aid at best)
Brent

121 Posts
The cost-benefit analysis behind your eventual decision would be interesting. As your examples point out, bad signatures have had quite the range of effects. Pre-testing AntiVirus signatures isn't a straightforward mitigation though. If your staff spends more time validating software capability than they would have fixing an occasional glitch, this cure can be worse than the problem.

Delaying deploying the new signatures also adds some amount of risk - your systems remain vulnerable to the latest defendable malware for longer. After all that extra work, your internal testing may not even catch an incompatibility that shows up in production. I imagine that many organizations' approaches to automatic updates (of any sort) are ad-hoc. The "right" option depends on many technical and non-technical needs of each organization.
Brent
2 Posts
In my case, we have strong usage policies in place and our user environment is small, so having a delay in the distribution of the DAT files would be an informed and acceptable risk, within reason. Given that we are small and resources are very tight, I agree with Brent that spending additional time to start having to test AV signatures is not ideal. If the product is a paid subscription, I want the AV companies to be the ones to do more and better testing. Some of these things would have been found in internal testing as they were things that would affect any system. To not find such problems says they didn't effectively test the DAT before rolling it out. One would think that they would start to worry about their reputation as a reliable vendor when these incidents keep re-occurring.
Val

10 Posts
McAfee EPO is our management tool for the VirusScan product from McAfee. We have setup a delayed deployment schedule where the latest DAT file release is downloaded in the evening to the Evaluation branch, deployed to a small group of test servers and workstations, then pulled down again the next morning into the Current branch then deployed over a staggered schedule trough out that day.

This process delays the DAT update roll out by 18 hours giving us time to validate the DAT update and for McAfee to pull a bad DAT back prior to it's release to our Enterprise. This has saved us during the bad DAT times as noted in the original article. If there is an issue, we are usually only one or two DAT releases behind but catch up during the day.
Brian

1 Posts
We stay one day back and it has saved us a few close-the-doors incidents over the years. But we consider desktop/server AV as the last resort anyway and we only have a 5% or 10% confidence factor in it. We use highly restrictive user rights, multiple layers of anti-malware at various chokepoints (firewall/IPS/SMTP gateway/email server/proxy server) so that something has to sneak past all of those first. That being said, we treat a desktop AV alert as a very high priority but there is well under one per week anyway. (2,000 nodes)
Anonymous
One size doesn't fit all here, but I'd say that an automatic but phased roll-out is a sensible trade-off. Watching Virustotal results "improve" over time for current threats, it looks like it often takes between 3-8 days for "protection" to arrive in the release pattern .. so I don't think tucking on 18hrs of delay with a phased roll-out is all that big a deal. Presumed of course that you have an establihed process to shortcut a rapid roll-out of the freshest beta pattern that you can get your hands on if things are really going haywire on your network. In other words .. this is a typical risk/benefit tradeoff that depens on the circumstances and situation.
Daniel

367 Posts
ISC Handler
My argument would be that it is the AV vendor's responsibility to test signatures before releasing them. That won't catch every incompatibility, of course, but false positives on files like svchost.exe seems to indicate that the vendor did absolutely no testing at all. McAfee makes some great products, but I view this as a chink in their armor.
PhilBAR

24 Posts
Having a delay (a day) might be the best blanket option (of course everyone wants their own piece of the blanket). Yes a few specific programs or possibilities may not be figured out until you try it but let the rest of the world be your testers. If there is a problem, as McAfee has shown, they pull the bad DATs pretty quickly.

As most others have stated, doing a full, thorough round of testing for every single release would quickly find yourself getting increasingly behind real-time. At first you may be one behind the current release but eventually I think you'd find another newer current release would be out before you finished testing the first one and so on....
PhilBAR
7 Posts
I have serveral VMs that represent the base OSs we have in use in our environemnt. On these gets installed every piece of software used in our environment. The boxes are setup to be patched just like anyhting else in the environemnt, and are also pushed new virus defs about 12 hours before they roll out to the rest of the company. These VMs are never used, but sit, logged into with a fake user account. The network to the VMs is locked down to only allow that traffic needed for patches and AV updates. If a bad patch OR AV definition is released, it will break one of these VMs 12 hours before it starts breaking production hosts.
RTAdams89

2 Posts
Should we treat all AV companies as equally likely to exhibit this problem? The lead-in to this topic identified two vendors, but one appears to have a chronic problem.

Perhaps there's data somewhere that implicates all AV companies? But even if so, it seems likely that not all will fail with the exact same frequency. And so failure rates can feed into risk analysis (and future purchasing decisions).
Gregor

2 Posts
There are fundamental problems with this "wait" approach.

1. Worms. Remember them ? Well they need countermeasures "immediately", and are the main reason MSFT and the likes rate their patches the way they do, and are the main reasons AV vendors have fast tracks to get patches out there. If you delay that, you significant increase the risk of getting wiped out by a worm as any delay is going to be paid in full.

2. wait for whom ? Wait for others: they can wait a bit longer and then you wait longer and before we know it we're all waiting for the others to loose their nerve and risk it.

3. vendors are getting PAID to do a good job (I'll exclude some of the freebies). If you do not trust your vendor then get another one. And demand that they test their crap themselves. What makes you think you can test it better than they did ? What makes you think that an installed base of millions of machines will detect problems slower than your few machines on a fast deployment track ?

4. rebuild capacity: you need it anyway (in case the AV fails to protect you) well what's so bad of using it once in a while ? It gives you exercise and find the weak points, so the more you're used to it, the better it'll work and the more crap you can keep off of your machines.

5. AV is already too little too late - by definition it's a reactive technology. Are you really going to make it even slower ?

6. Choices: get measures in place that make you less dependent on that OS, you know the one that is targeted by 99.99% of all malware out there. A bit more diversity would make your infrastructure a lot more resilient against attacks.

7. Do not forget that in a lot of scenarios the desktop AV is the *only* line of defense despite whatnot you have in a perimeter heavy model. E.g,: S/MIME, PGP, HTTPS, SSH, ... it's all end-to-end encrypted so the end user's box is the one that counts -unless you break the encryption - and that's a whole other can of worms in itself. That AV is often all you have as defense, sure you want to explain to your CEO/board/stakeholders - well we had a signature to protect this, and it was workign fine, only we had not let it deploy intentionally ?
Swa

760 Posts
The idea of imposing testing could make the AV worthless at doing what it's supposed to do.

What I would recommend instead is using AV software that has a "safer" response to malware.

Instead of automatically deleting, without asking questions, it should set off alarms.

Non-destructive read/execute operations should not be translated by AV software into blind "move/delete" operations.


The possible exception would be browser and e-mail client Download directories, and other places when a brand new executable file is being copied to a system from removable media, or a remote web site.

Non-executable files that happen to have a match for a malicious signature should set off alarms.

The response procedure should be dependant upon the scope/context of the file that triggered the alarm.


For system files, there should be an additional verification that system files do /not/ match the standard version of that file.

That may be more work for the AV makers, but I see it as their responsibility to ensure their software does not do more harm than good for Availability and Integrity security goals by not creating downtime, or destroying system data.



Mysid

146 Posts

Sign Up for Free or Log In to start participating in the conversation!