On 1/5/07, Brennan Stehling brennan@offwhite.net wrote:

I have been using SpamAssassin, but I have had problems where incoming spam causes the server to become unresponsive for long periods of time. This is obviously unacceptable. I am pretty sure the biggest part of the problem is that fact that is running with Perl. I have had problems with Perl before when I wrote CGI applications where it can lock up a server if you are handling a lot of data.

I specifically have it set to only handle messages under a certain size, but I still have problems.

I host other things on the same server, like my DNS and Web servers so I cannot allow the spam filter to kill the performance of all applications. Is there something better that I could do? I am seriously considering having all of my mail aliased to my Gmail account and not allow incoming mail to be stored on this server. If I do that I will not be using RC, which I would like to continue using and helping with the development effort.

Brenden Two ideas, one, use spamc http://spamassassin.apache.org/full/3.0.x/dist/doc/spamc.html for SA checks and two use something like policyd http://policyd.sourceforge.net/ that will handle greylisting, rate limiting, Spamtrap monitoring and blacklisting, HELO checks, with auto blacklisting. Basically it listens before Postfix (or whatever MTA) and grabs the mail first, only passing it on once it's happy with it. It's also a c program, and you can now have it hook into clamav as well. It's pretty light, plus it takes away a bunch of work that your MTA used to have to do. Honestly I've been running it lately and not bothering with SA, since I have a .procmailrc rule to fwd all mail to my Gmail acct which deals with the spam really well - just as you desc. It can check with clamav if it has an attachment and deny it there. The downside is I haven't had time to configure/use RC for awhile (I used to have Gmail forward all mail to my home server since I only used RC back then!)

I used to have a very convoluted plan with greylisting/sa/rules-du-jour/clam/razor/dcc and other checks, all handled by Mailscanner, which is more perl, and yeah, I felt the heat and dropped most of it after I discovered policyd. If you do that alone you'll reduce your load a lot. Just adding that w/o sa really reduced my spam amount, if I get back into it I need to put SA (or as I was leaning towards dspam) in the mix with it. I also still want to get my openbsd box in the mix to handle spamd to do the phoney smtp tarpit server, but that's later even though it's pretty much setup. Anyway, a simplier plan right now at home:

INTERNET -> policyd -> (clamav) -> Postfix -> procmail -> Gmail

hth P

Brennan Stehling

Offwhite.net LLC brennan@offwhite.net

I have been using SpamAssassin, but I have had problems where incoming spam causes the server to become unresponsive for long periods of

Although I'm not an expert with Perl, I've installed spamassassin at several companies and have never seen lockup or hanging with it. I use postfix, mailscanner, spamassassin, fuzzy ocr, and clamav solution.

At the school district I work at this configuration handles 6000-7000 messages a day without any performance issues. That's running a SuSE SLES10 install and it's running on a Pentium 4 2.8Ghz Xeon with 1GB of memory - nothing spectacular.

At some other smaller sites, same configuration and using Debian with much lower hardware spec's (e.g. Pentium 4 2.8Ghz 512MB) and again I've not seen any type of spamassassin issues. The mail amount at these other sites range from 200-3000 messages a day.

Is there anything in the logs that you can find out where your problems are stemming from?

dan

Brennan Stehling wrote:

I have been using SpamAssassin, but I have had problems where incoming spam causes the server to become unresponsive for long periods of time. This is obviously unacceptable. I am pretty sure the biggest part of the problem is that fact that is running with Perl. I have had problems with Perl before when I wrote CGI applications where it can lock up a server if you are handling a lot of data.

I specifically have it set to only handle messages under a certain size, but I still have problems.

There must be something else going on there. Perl isn't necessarily slow, and spamassassin is quite fast on its own, especially if you're using spamc/spamd. Perl has a slower start-up time because it's interpreted, but the actual execution isn't all that slow.

No one process should bring the entire server to its knees. Even if it's getting bombarded with spam, it should still do something. That is, unless someone is targeting you with a DoS attack.

Our main mail hub processes ~60,000msgs/day through spamassassin (Well, spamc/spamd) without any real delay. If there is a delay, it's usually due to a DNS/RBL timeout, or an image being processed through FuzzyOCR. This is on a dual CPU 2GHz Xeon. Previously, we had a 1.3Ghz Athlon in that position and it also handled the load quite well.

You can help it out more by using RBLs, we rejected 230,000 messages yesterday by RBL alone before they ever hit spamassassin.

I host other things on the same server, like my DNS and Web servers so I cannot allow the spam filter to kill the performance of all applications. Is there something better that I could do? I am seriously considering having all of my mail aliased to my Gmail account and not allow incoming mail to be stored on this server. If I do that I will not be using RC, which I would like to continue using and helping with the development effort.

If you really do want Gmail to handle your mail, you could always get it back to yourself by using fetchmail to bring it back into your own server.

On my personal server that also has web, mail, etc. I use amavisd-new to process mail and scan for viruses. It's nowhere near the load of the main hub, (and it's a dual cpu PIII-800) and it's also in perl, but it runs as a daemon so there is no start-up time on a per-message basis.

If you don't already, you may want to consider graphing network usage, processor load, etc with snmp (and perhaps a graph package like Cacti) it can help a lot when tracking down an issue like this.

Jim