Spam was killing us! Here is what we did to help!

I work for a smallish ISP in the Midwest. In late September and the month of October we began getting blasted with spam and DHA's from all over the world. We had been utilizing a spam filtering service but it was not keeping up. We billed the customers for the service and they were starting to complain. They were getting so much spam in their inboxes that they felt like they were wasting their money. In October when the problem became so bad that it started affecting our mail servers ability to process mail any longer we knew we had to do something. We had been "test driving" a spam filter device by Red Condor. The accounts that had been moved over to the Red Condor filter were virtually spam free. We decided to implement the Red Condor solution across the board on the server that was being hammered the worst. This server has just over 9,000 accounts on it. We turned up the Red Condor box at about 4pm and by 7:00am the next morning the quarantine boxes had been created for all customers. No interaction required, it simply verified each inbox as the emails arrived for the account. If the account did not exist it threw the spam away, if the account did exist it created the inbox and then determined whether the email was spam or was legit (autodiscover does not work with Exchange Servers).

We decided to "give the service away" as part of the customers Internet service. In reality we have been the ones to benefit from the service. The mail server has been purring along for months now and our customers are much happier. They literally have had no spam hit their inboxes. We have been in the learn mode for a while and slowly started migrating other customers over to the device. It has not missed a hit. The other thing that is amazing is the ease in setting up the "accounts" on Red Condor. With the previous service it was about a 15 minute process to setup each domain. It was a series of long drawn out steps to setup the accounts. With Red Condor it takes less than a minute to setup a new account/domain. If I can use autodiscover to create the inboxes then the setup task is done. Change the MX record and I am good to go.

Now here is the amazing part. The reporting available with the product is unbelievable. At a glance I can see just how much work this single device is doing. Here is a report for the domain that has just over 9,000 accounts. This is a summary of the transactions handled for the domain since March 1, 2010. You see that out of almost 20 million emails handled only 713,222 (3.6%) were actually delivered.

March 2010

	Disposition
Category	Deliver	Markup	Quarantine	Block	Total		Size
OK	638,116				638,116	3.2%	108GB	32.1%
Unprotected	2,905				2,905	0.0%	60MB	0.0%
Friends	72,201				72,201	0.4%	17GB	5.2%
Enemies			176		176	0.0%	31MB	0.0%
Virus				55,587	55,587	0.3%	7,109MB	2.1%
Phish			434,661	2,218	436,879	2.2%	1,165MB	0.3%
Keyword					0	0.0%	0	0.0%
Adult				106,296	106,296	0.5%	270MB	0.1%
Spam		919	13,412,089	42,939	13,455,947	68.1%	154GB	45.9%
Junk		1,718	349,796	697	352,211	1.8%	9,223MB	2.7%
Blank			489	2	491	0.0%	1,073KB	0.0%
Foreign			12,707	33	12,740	0.1%	159MB	0.0%
Risky Attachment			16		16	0.0%	18MB	0.0%
Unresolved Sender					0	0.0%	0	0.0%
Invalid Recipient				4,623,107	4,623,107	23.4%	38GB	11.3%
Total	713,222	2,637	14,209,934	4,830,879	19,756,672		335GB
	3.6%	0.0%	71.9%	24.5%

It isn't hard to understand now why my poor mail server was weeping on a daily basis. We are now in the process of moving the remaining customers, accounts and domains over to the Red Condor system.

Spam and viruses have become such a big problem for ISP's world wide. Until we can clean up the infected machines that are generating this spam and shut down the bad guys that are pushing this garbage at us, it is good to know that these types of systems exist.

I would like to hear from our reader's. What has helped your organization deal with spam and the pr

Deborah

279 Posts
ISC Handler

Mar 17th 2010

We use a product that includes IP reputation filtering. Out of about 8M message per day, 98% are dropped based on source IP address pre DATA. The product also uses behavioral monitoring of unblocked addresses - it it detects spam it starts to grey-list the sending IP address.

Anonymous

I feel your pain. I have battled the forces of evil protecting mail servers for quite a few years. I have had great success fighting spam with the Open Source/commercial app Untangle (untangle.com) with some SMB clients. Other clients use hosted Exchange with Barracuda with some success. Still others use Microsoft Forefront for Exchange. They all have advantages/disadvantages, but it comes down to proper configurations and remaining diligent. If I only had a nickle for each spam message I have prevented from reaching a users inbox. :)

Anonymous

Although we have a single domain we receive around 4 million messages monthly with typically 9%-11% being good. We were in the same boat several years back and opted to go the Barracuda appliance route. We experienced very similar results however, it appears that as Barracuda's popularity grew so did the spam transmitter's abilities. Even with constant tuning our blockage percentages are trending downward so we are investigating other avenues and will give condor a trail.
$.02 deposited

Greg

3 Posts

While the number of users on my server pales in comparison to this story, I still a large percentage of attempted spam for the number of users I have. I use OpenBSD's spamd, postfix, and amavis-new. That which manages to make it by the greylisting, then gets multiple checks applied via postfix. No RR record for the connecting IP, 4xy. HELO command is checked, sender domain checked, sender address checked (as in a connection is made to the domains MX, will it accept a message for the RP my system was given. Still some spam manages to get passed all that, and for those amavis-new manages to catch the rest quite well.

Greg
2 Posts

I work for a manufacturing company, not an ISP. We long ago decided that we had better uses for the time of our IT personnel than fighting spam. We outsourced to MessageLabs and can now do the things that are productive for the company. I haven't seen any spam in years.

Greg
1 Posts

The 'quality' of any spam filter depends, I guess, on the amount of data available to it and its skill at interpreting it. It doesn't help when major webmail providers are the source of the spam; the real source IP is often unknown to the final recipient, who therefore has less basis on which to do any filtering. That results in a lower 'quality' of filter, so the number of false negatives must increase in order to keep the number of false positives the same. That leaves the webmail provider in a superior position since they do have that data available to them.

Emails sent between accounts hosted at the same provider can possibly be identified as spam even *after* delivery (after other recipients have complained), so that too gives the larger email providers an edge over external victims of their spam.

I think spam will continue to hit smaller providers hardest, nudging people toward the major webmail providers and probably suffering a CAPTCHA every other outgoing message (or pay to 'go pro'). And business users would be pushed toward their outsourced email offerings, probably paying by volume of sent mail. I think that's a sad place for any Internet-based service to end up.

So I guess SMTP was fatally flawed. Maybe its next incarnation would define a good, standard 'feedback loop' for reporting spam and maybe even the ability to 'recall' a message after sending it. Systems downstream from it would need to be prepared for messages to be recalled also.

In the meantime it might be smart to reduce dependency on email. In the context of a web application, do you really *need* the user to provide you with an email address? Can you offer alternate contact methods? Can you handle account signup and authentication some other way, such as OpenID, providing unique URIs, client certificates or cryptographic tokens? Maybe you could still provide your service without requiring login at all? Could you offer your email subscriptions or notifications via RSS or other means?

Steven C.

171 Posts

My concern, if I were a customer of yours, would be how many legitimate email messages did I NOT get. Do you know how many false positives it actually dropped?

Subelman

1 Posts

My experience is that checking the IP for reverse DNS, and that the PTR text doesn't look like a dynamic IP, and checking to see that it matches the HELO will catch a ton of the bot spam.

Subelman
5 Posts

Great write up. I am at a small shop, only about 200 mailboxes and recently switched our anti spam solution. I was able to get a very good solution working for free. Through Exchange 2007 and the built-in ability to use RBLs, SPF and the other anti spam features - we've got a solid solution. At first it was horrible so we gathered a master "white list" from our staff and changed the SCL threshold levels. It's been great so far. One major downside is that the reporting is non existent but that's what you get for free!

Subelman
1 Posts

We use Ironport email gateway in saudi airlines company for past one year and never come across any issues and none of customer complained about spam messages or false positives. As said by many users posted here, The quality of spam filter depens upon the amount of data available about the source IP address. In this case Ironport uses SBRS database which covers almost 70% internet ISP traffic.

Subelman
1 Posts

We use Google/Postini which is SaaS. Works fine for us, and since it is in the cloud, we don't even have maintenance of a box, and don't need any IT staff to look after it. There are lots of products. Earlier I had good experience with Open Source products. It is not important what you use, as long as you use something.

Povl H.

75 Posts

I work for an Telco in the UK, we implemented Puremessage for UNIX but are now looking at the cost benefits for outsourcing the work to a Saas provider (Google/messagelabs ..etc). The business is getting tiered of flogging out huge capital to cover hardware and software depreciation every three to five years, but does not mind spending a constant operational cost over a fixed term contract.

I was wondering how many security managers/engineers have evaluated in house solutions compared to outsourcing to a SaaS provider?

Peter P

8 Posts

The problema of dealing with spam and antispam filters is false positives. I certainly don't want to go hunting the logs and quarantine every time a user complains about the email they didnÂ´t receive.
We tried 2 solutions (Symantec hosted mail security and Anubis Networks) that offered the daily digest sent to the end user with the mails that were in quarantine and let them do the management of their own spam.
Of course they need supervision, because every once in a while a user released everything, including spam.
Personaly I liked the Anubis, because of the amount of spam in Portuguese that the english based antispams dont catch very well and Anubis is a Portuguese company dedicated to antispam.
Since both services were outsite our network (MX redirection) our internet link never got the impact of spam.

Rogerio

1 Posts

I used to get 5000-10000 spams in a week until I used this...

http://jimsun.linxnet.com/misc/postfix-anti-UCE.txt

Combined with spamassassin, amavisd-new and clamav, I hardly get any spam on my domains. Also, I've only had one false positive in years.

Anonymous