Pushdo/Cutwail Spambot - A Little Known BIG Problem

Pushdo/Cutwail Spambot - A Little Known BIG Problem
Today was another one of those days that all ISP's dread. I am the Abuse Coordinator for a small Midwestern ISP. Several days ago we started receiving Spam Abuse reports on the IP address to our Corporate firewall. Unfortunately, the IP I discovered is blocklisted on several blocklists. I began to investigate what could be causing these reports of abuse. I reviewed the logs in the firewall and discovered that we had a couple of workstations doing some bad things. Our It techs began to look at the computers (both of which had AV installed) and discovered that we had some pretty significant infections on these computers. Both machines were pulled offline, the data backed up and the machines were formatted and reloaded. We were pretty confident that we had solved the problem and breathed, an unfortunately premature, sigh of relief. Yesterday we again started getting abuse reports so it was back to the drawing board for me. I started trying to get information on exactly what was being detected and what was causing these abuse reports. This investigation led me to MultiRBL.org. We were indeed listed on several blocklists again. As I began to look at the various blocklists looking for the answers it became apparent that we will dealing with a Trojan/Botnet called Cutwail Spambot aka Pushdo aka Pandex. The interesting thing is, I hadn't never heard of it. So last night I began to research just what this Cutwail Spambot was. What I find out just blew me away. I came across an article from Trend Micro Researchers Alice Decker, David Sanchog, Loucif Kharouni, Max Goncharov, and Robert McArdle. The article is titled A Study of the Pushdo /Cutwail Botnet, An Indepth Analysis. The article indicates that this particular botnet has been around since January 2007 and is the second largest spam botnet on the planet. This particular spambot is believed to be responsible for approximately 7.7 billion spam emails per day making it responsible for 1 out of every 25 spam emails sent world wide. According to the findings of the research team the development team for Pushdo/Cutwail work very hard and used several techniques to keep their program "under the radar". In the article they outline these techniques which include things like using multiple variants that react a bit differently, remain memory resident, with very little actually written to disk, and frequent updates and changes to the code to prevent discovery. This article contains an indepth look at the botnet and gives good insight into how to detect and control the botnet. This article is well worth reading. Other research that I have done indicates the best program to find the Pushdo/Cutwail Spambot is Microsoft's Windows Malicious Software Removal Tool. Another article - by Matt McCormack entitled "WHEN THE HAMMER FALLS – EFFECTS OF SUCCESSFUL WIDESPREAD DISINFECTION ON MALWARE DEVELOPMENT AND DIRECTION" gives additional information about the botnet and gives detailed information and instructions for ridding your network of the botnet. Our tech's have their work cut out for them. They are going to have to "touch" all 250 employee computers (249 - mine is clean) plus all of our Windows Servers so that we make sure that we get rid of all of the infected computers. We are also investigating a change in Anti-virus software. Unfortunately the one we have been using has fallen into the category of less that reliable so now we are trying to decide what we need to switch too. Now is as good a time as any, after all we are going to have to "touch" every computer. I am just amazed that this botnet is the 2nd largest in the world, been around for almost 3 years and I am just now dealing with it. We still haven't figured out how this botnet got started, we aren't sure where it started at, but we do know we can't wait to rid our network of this mess. Everyone who manages networks no matter what the size needs to read these articles and know what to look for and how to recognize the presence of the botnet. I for one vote for irradication of this botnet and a reduction of 7.7 Billion spam emails a day. Sure would make my spam filter easier to manage. Wouldn't it be great to somehow eliminate these bad guys. Check out the articles at: Matt McCormack Article - download.microsoft.com/download/3/8/d/.../McCormack-VB2008.pdf Trend Micro Article - us.trendmicro.com/imperia/md/content/us/pdf/.../study_of_pushdo.pdf I would be interested in hearing about other people's experiences with this Botnet and in finding out if you have any good tips for detecting and "killing" the bot. So let's hear from all of you botherders out there. Deb Hale Long Lines, LLC	Deborah 279 Posts ISC Handler Nov 13th 2009
Thread locked Subscribe	Nov 13th 2009 1 decade ago
Matt McCormack: http://www.microsoft.com/downloads/details.aspx?FamilyID=44bcc45f-7bad-46a2-b661-69d435754a0b&displaylang=en	Anonymous
Quote	Nov 13th 2009 1 decade ago
FWIW I have seen a workstation infected with this before, about a year ago. I can't remember how anyone initially noticed it, but from that point on I have always logged attempted connections from any workstation to port 25/tcp (or 53/udp) of any external IP; since we run internal SMTP and DNS that should never happen anyway. Surely enough the workstation was spamming away happily. Anyway, with nobody using the machine it could be easily seen that the machine was communicating with some kind of C&C nodes at 94.103.4.217, 174.36.201.82, 69.147.239.106, 208.43.154.226 and 94.103.4.230 (don't remember the port[s], and these IPs are probably not active any more); randomly connecting to one of these as I blocked each one in the NAT gateway. From that point I decided to monitor those IPs in case any other machines were infected with the same malware and trying to use the same C&C nodes, but fortunately it wasn't the case. Honestly I'm not sure how I got rid of it. I believe it still operated in Safe Mode and would kill diagnostic tools such as procexp.exe as soon as they tried to load. I seem to remember, that in procexp I saw data relating to the spam generation in the 'Strings' tab for winlogon.exe; the malware had inserted itself into that service's resident image, or something... In the end I think it was 'gmer' (http://gmer.net/) that told me which drivers/.sys file it was hiding in. After renaming the file, I used 'notmyfault' from SysInternals to invoke an immediate kernel crash, so that the OS could not perform a clean shutdown (which may have allowed the malware time to drop itself to a new file to load on reboot). I've use that technique a few times to avoid having to re-image, which would have been particularly awkward for that particular workstation, but it's something I should get in the habit of doing because intelligent malware these days may be near-impossible to remove any other way.	Steven C. 171 Posts
Quote	Nov 13th 2009 1 decade ago
another thing to thank fireeye for... http://blog.fireeye.com/research/2008/11/pushdocutwail-control-servers.html	Anonymous
Quote	Nov 13th 2009 1 decade ago
Deborah: Not to blame the victim, but... Your firewall policy permits all hosts on your corporate network to send SMTP to the Internet at large? That is a recipe for disaster, as you've seen. You should have egress filtering in place, either restricting outbound SMTP for the world at large to the single designated outbound MTA on your local network, or restricting your local network hosts sending SMTP to only your designated MTA outside the firewall. Or am I misunderstanding the description of your environment?	John Hardin 62 Posts
Quote	Nov 13th 2009 1 decade ago
The environment I spoke of did allow SMTP to the Internet at large, but I decided to stop that after the Cutwail incident. The problem is that more-intelligent malware is able to use local MTA's too, although you could (and I do) apply spam-filtering to outbound mail to detect and/or block this. In any case, merely logging outbound SMTP traffic (whether you block it or not) from a workstation should warn of infection by spam-sending malware. On larger networks (or something like an University campus) where people may bring their personal computers, it may not be reasonable to block outgoing SMTP.	Steven C. 171 Posts
Quote	Nov 13th 2009 1 decade ago
Deborah's not a 'victim' here, as she's enough of a security professional to be a handler on duty now and again. I tend to reserve victim for civilians hit by this kind of stuff. Pros, we expect it, right? Part of what they pay us for? I am very interested that at an ISP the corporate hosts could just send email to whomever via whichever mail server. What do you want to bet the same permissive policy was in place for the customers?	peter 17 Posts
Quote	Nov 13th 2009 1 decade ago
John, Peter and all - you are correct. We should have not allowed this to happen. We have 2 separate areas in our "IT" dept. One for internal customers, one for external customers. Internal customer systems are handled by another individual. Preventations were not in place. Now they are. They learned something today. As for our external customers, oh yeah it is controlled.	Deborah 279 Posts ISC Handler
Quote	Nov 13th 2009 1 decade ago
John, Peter and all - you are correct. We should have not allowed this to happen. We have 2 separate areas in our "IT" dept. One for internal customers, one for external customers. Internal customer systems are handled by another individual. Preventations were not in place. Now they are. They learned something today. As for our external customers, oh yeah it is controlled.	Deborah 279 Posts ISC Handler
Quote	Nov 13th 2009 1 decade ago
Hi Deb, Out of curiosity, how was your machine spared? Or did you clean it up yourself? If your machine was spared, you could consider applying similar safeguards to the other hosts in your network. CP	Ben 1 Posts
Quote	Nov 13th 2009 1 decade ago
I, at least could not follow your links. Please give the full links without the "..."	Ben 3 Posts
Quote	Nov 14th 2009 1 decade ago
A little off topic but related... at a former employer I discovered a desktop in a remote office sending thousands of spam emails. But in this case it was not not a case of malware, but of malperson -- an IT tech was intentionally spamming from his desktop! Needless to say he was immediately shown the door. In that environment we had no direct IP connectivity to the Internet. Everything had to go through a proxy-based firewall (fwtk or Gauntlet). It's been a long time but I believe it was probably the backlog on the internal mail relay or on the proxy that brought attention to his little enterprise.	Hal 50 Posts
Quote	Nov 16th 2009 1 decade ago
I had a user at one of our remote offices complaining that her computer was slow and something kept popping up in her tray. After visiting the site I found out she was infected with something that kept sending out mass emails, Symantec AV was blocking the emails from going out luckily. After poking around I found (Don't remember exactly what) some rogue program installed that was causing the machine to act as a bot. This sounds like what you are describing here, but that's an assumption. Bottom line, we do egress filtering for our CBO and our remote offices respectively. Monitoring logs for SMTP connections can provide a means of detecting where an infection may be present, but egress filtering will definitely help you from getting blacklisted. I keep a live running log of all connections leaving my network, so I keep a close eye on traffic just in case one of our IT machines start misbehaving. Cheers	Anonymous
Quote	Nov 18th 2009 1 decade ago
In response to Rooster_Dar's post - I'm seeing more and more firewalls with "default deny" outbound rulesets. Where the permitted protocols are green-lighted, and everything else is blocked. This won't stop P2P apps that use tcp/80, or malware that tunnels out through tcp/80 or dns, but it alerts on enough to find suspect machines most days. Full packet inspection is the way to go on the remaining permitted traffic, which will catch non-http stuff on tcp/80 etc. But what it won't stop is users installing fake antivirus tools and giving up their passwords willingly to phish sites. User education is still one of the most important tools in the corporate IT arsenal.	Rob VandenBrink 579 Posts ISC Handler
Quote	Nov 18th 2009 1 decade ago

Today was another one of those days that all ISP's dread. I am the Abuse Coordinator for a small Midwestern ISP. Several days ago we started receiving Spam Abuse reports on the IP address to our Corporate firewall. Unfortunately, the IP I discovered is blocklisted on several blocklists. I began to investigate what could be causing these reports of abuse. I reviewed the logs in the firewall and discovered that we had a couple of workstations doing some bad things. Our It techs began to look at the computers (both of which had AV installed) and discovered that we had some pretty significant infections on these computers. Both machines were pulled offline, the data backed up and the machines were formatted and reloaded. We were pretty confident that we had solved the problem and breathed, an unfortunately premature, sigh of relief.

Yesterday we again started getting abuse reports so it was back to the drawing board for me. I started trying to get information on exactly what was being detected and what was causing these abuse reports. This investigation led me to MultiRBL.org. We were indeed listed on several blocklists again. As I began to look at the various blocklists looking for the answers it became apparent that we will dealing with a Trojan/Botnet called Cutwail Spambot aka Pushdo aka Pandex. The interesting thing is, I hadn't never heard of it. So last night I began to research just what this Cutwail Spambot was. What I find out just blew me away.

I came across an article from Trend Micro Researchers Alice Decker, David Sanchog, Loucif Kharouni, Max Goncharov, and Robert McArdle. The article is titled A Study of the Pushdo /Cutwail Botnet, An Indepth Analysis. The article indicates that this particular botnet has been around since January 2007 and is the second largest spam botnet on the planet. This particular spambot is believed to be responsible for approximately 7.7 billion spam emails per day making it responsible for 1 out of every 25 spam emails sent world wide. According to the findings of the research team the development team for Pushdo/Cutwail work very hard and used several techniques to keep their program "under the radar". In the article they outline these techniques which include things like using multiple variants that react a bit differently, remain memory resident, with very little actually written to disk, and frequent updates and changes to the code to prevent discovery.

This article contains an indepth look at the botnet and gives good insight into how to detect and control the botnet. This article is well worth reading. Other research that I have done indicates the best program to find the Pushdo/Cutwail Spambot is Microsoft's Windows Malicious Software Removal Tool.

Another article - by Matt McCormack entitled "WHEN THE HAMMER FALLS – EFFECTS OF SUCCESSFUL WIDESPREAD DISINFECTION ON
MALWARE DEVELOPMENT AND DIRECTION" gives additional information about the botnet and gives detailed information and instructions for ridding your network of the botnet.

Our tech's have their work cut out for them. They are going to have to "touch" all 250 employee computers (249 - mine is clean) plus all of our Windows Servers so that we make sure that we get rid of all of the infected computers. We are also investigating a change in Anti-virus software. Unfortunately the one we have been using has fallen into the category of less that reliable so now we are trying to decide what we need to switch too. Now is as good a time as any, after all we are going to have to "touch" every computer.

I am just amazed that this botnet is the 2nd largest in the world, been around for almost 3 years and I am just now dealing with it. We still haven't figured out how this botnet got started, we aren't sure where it started at, but we do know we can't wait to rid our network of this mess.

Everyone who manages networks no matter what the size needs to read these articles and know what to look for and how to recognize the presence of the botnet. I for one vote for irradication of this botnet and a reduction of 7.7 Billion spam emails a day. Sure would make my spam filter easier to manage. Wouldn't it be great to somehow eliminate these bad guys.

Check out the articles at:

Matt McCormack Article - download.microsoft.com/download/3/8/d/.../McCormack-VB2008.pdf

Trend Micro Article - us.trendmicro.com/imperia/md/content/us/pdf/.../study_of_pushdo.pdf

I would be interested in hearing about other people's experiences with this Botnet and in finding out if you have any good tips for detecting and "killing" the bot. So let's hear from all of you botherders out there.

Deb Hale Long Lines, LLC

Deborah

279 Posts
ISC Handler

Nov 13th 2009

Matt McCormack: http://www.microsoft.com/downloads/details.aspx?FamilyID=44bcc45f-7bad-46a2-b661-69d435754a0b&displaylang=en

Anonymous

FWIW I have seen a workstation infected with this before, about a year ago. I can't remember how anyone initially noticed it, but from that point on I have always logged attempted connections from any workstation to port 25/tcp (or 53/udp) of any external IP; since we run internal SMTP and DNS that should never happen anyway. Surely enough the workstation was spamming away happily.

Anyway, with nobody using the machine it could be easily seen that the machine was communicating with some kind of C&C nodes at 94.103.4.217, 174.36.201.82, 69.147.239.106, 208.43.154.226 and 94.103.4.230 (don't remember the port[s], and these IPs are probably not active any more); randomly connecting to one of these as I blocked each one in the NAT gateway. From that point I decided to monitor those IPs in case any other machines were infected with the same malware and trying to use the same C&C nodes, but fortunately it wasn't the case.

Honestly I'm not sure how I got rid of it. I believe it still operated in Safe Mode and would kill diagnostic tools such as procexp.exe as soon as they tried to load.

I seem to remember, that in procexp I saw data relating to the spam generation in the 'Strings' tab for winlogon.exe; the malware had inserted itself into that service's resident image, or something...

In the end I think it was 'gmer' (http://gmer.net/) that told me which drivers/.sys file it was hiding in. After renaming the file, I used 'notmyfault' from SysInternals to invoke an immediate kernel crash, so that the OS could not perform a clean shutdown (which may have allowed the malware time to drop itself to a new file to load on reboot). I've use that technique a few times to avoid having to re-image, which would have been particularly awkward for that particular workstation, but it's something I should get in the habit of doing because intelligent malware these days may be near-impossible to remove any other way.

Steven C.

171 Posts

another thing to thank fireeye for...

http://blog.fireeye.com/research/2008/11/pushdocutwail-control-servers.html

Anonymous

Deborah:

Not to blame the victim, but...

Your firewall policy permits all hosts on your corporate network to send SMTP to the Internet at large? That is a recipe for disaster, as you've seen.

You should have egress filtering in place, either restricting outbound SMTP for the world at large to the single designated outbound MTA on your local network, or restricting your local network hosts sending SMTP to only your designated MTA outside the firewall.

Or am I misunderstanding the description of your environment?

John Hardin

62 Posts

The environment I spoke of did allow SMTP to the Internet at large, but I decided to stop that after the Cutwail incident.

The problem is that more-intelligent malware is able to use local MTA's too, although you could (and I do) apply spam-filtering to outbound mail to detect and/or block this.

In any case, merely logging outbound SMTP traffic (whether you block it or not) from a workstation should warn of infection by spam-sending malware. On larger networks (or something like an University campus) where people may bring their personal computers, it may not be reasonable to block outgoing SMTP.

Steven C.

171 Posts

Deborah's not a 'victim' here, as she's enough of a security professional to be a handler on duty now and again. I tend to reserve victim for civilians hit by this kind of stuff. Pros, we expect it, right? Part of what they pay us for?

I am very interested that at an ISP the corporate hosts could just send email to whomever via whichever mail server. What do you want to bet the same permissive policy was in place for the customers?

peter

17 Posts

John, Peter and all - you are correct. We should have not allowed this to happen. We have 2 separate areas in our "IT" dept. One for internal customers, one for external customers. Internal customer systems are handled by another individual. Preventations were not in place. Now they are. They learned something today. As for our external customers, oh yeah it is controlled.

Deborah

279 Posts
ISC Handler

John, Peter and all - you are correct. We should have not allowed this to happen. We have 2 separate areas in our "IT" dept. One for internal customers, one for external customers. Internal customer systems are handled by another individual. Preventations were not in place. Now they are. They learned something today. As for our external customers, oh yeah it is controlled.

Deborah

279 Posts
ISC Handler

Hi Deb,

Out of curiosity, how was your machine spared? Or did you clean it up yourself? If your machine was spared, you could consider applying similar safeguards to the other hosts in your network.

CP

Ben

1 Posts

I, at least could not follow your links. Please give the full links without the "..."

Ben
3 Posts

A little off topic but related... at a former employer I discovered a desktop in a remote office sending thousands of spam emails. But in this case it was not not a case of malware, but of malperson -- an IT tech was intentionally spamming from his desktop! Needless to say he was immediately shown the door.

In that environment we had no direct IP connectivity to the Internet. Everything had to go through a proxy-based firewall (fwtk or Gauntlet). It's been a long time but I believe it was probably the backlog on the internal mail relay or on the proxy that brought attention to his little enterprise.

Hal

50 Posts

I had a user at one of our remote offices complaining that her computer was slow and something kept popping up in her tray. After visiting the site I found out she was infected with something that kept sending out mass emails, Symantec AV was blocking the emails from going out luckily. After poking around I found (Don't remember exactly what) some rogue program installed that was causing the machine to act as a bot. This sounds like what you are describing here, but that's an assumption. Bottom line, we do egress filtering for our CBO and our remote offices respectively. Monitoring logs for SMTP connections can provide a means of detecting where an infection may be present, but egress filtering will definitely help you from getting blacklisted. I keep a live running log of all connections leaving my network, so I keep a close eye on traffic just in case one of our IT machines start misbehaving.

Cheers

Anonymous

In response to Rooster_Dar's post - I'm seeing more and more firewalls with "default deny" outbound rulesets. Where the permitted protocols are green-lighted, and everything else is blocked.

This won't stop P2P apps that use tcp/80, or malware that tunnels out through tcp/80 or dns, but it alerts on enough to find suspect machines most days. Full packet inspection is the way to go on the remaining permitted traffic, which will catch non-http stuff on tcp/80 etc.

But what it *won't* stop is users installing fake antivirus tools and giving up their passwords willingly to phish sites. User education is still one of the most important tools in the corporate IT arsenal.

Rob VandenBrink

579 Posts
ISC Handler