[nSLUG] apache used to generate spam

George N. White III gnwiii at gmail.com
Thu Apr 16 11:03:45 ADT 2009

On Thu, Apr 16, 2009 at 10:31 AM, D G Teed <donald.teed at gmail.com> wrote:

> This topic is a common one out there.  However I'm having
> some difficulty narrowing down the source of our problem.
> We have an apache server with about 100 subdomains on it,
> which has sent out spam via the local postfix.  Part of the
> problem is with the chaos of many web authors doing their
> own thing.  So in that regard it would compare with a hosting
> ISP scenario.
> The minute I shut down postfix and cleaned out
> the outbound queue, the spammer stopped sending new messages.
> This is due to the spammer monitoring the flow by use of
> their own address every few messages.
> No problem, I figured I would match the time stamps of apache
> emails in maillog with events in the apache log and find the perl, cgi, or
> php running wild.  In numerous attempts to match, I do not
> find the source which is doing this.
> The usual suspect in that case is some sort of application
> which has uploaded and runs as an independant process of
> apache.  I checked the /tmp /var/tmp and /dev/shm and
> nothing is appearing afterward.  lsof -Pni looks normal,
> chkrootkit comes back fine (it is getting a little dated), nmap
> from outside doesn't show anything new open.  You can't
> ssh to this machine from outside our firewall, samba likewise.
> I did a find for files owned by apache, a find for files modified
> in the last 2 days within the webroot - nothing interesting
> is coming up.
> It really bugs me that the logs don't tell the story.  One possible
> way around it could be that the script doing this has a time
> delay sleep before sending the mail to make it more difficult
> to trace.
> The server supports cgi and php, but not any database driven
> sites, so we don't have the risk of the usual web applications
> like phpBB being out of date and hackable.
> apache is current with Redhat EL 5.2 - one more update available
> which will applied sometime soon.
> I can fix postfix to only deliver mail for our domain as that will
> be a suitable workaround for our needs.  But I still want to find
> and eliminate whatever CGI is weak.  Many of our forms use
> alienform, which uses recipient data within a text file, not HTML form,
> to make it more secure.
> The spammer is unable to do anything while our postfix is down.
> At the end of the work day, he sent four test messages to himself
> to see if it was ready to roll again, so he seems to be aware of
> our business hours and doesn't want to try running it while
> we might be at work.  As postfix was down, they appeared in
> the maildrop queue but they were going nowhere.

Don't forget to keep detailed records of anything that might
attract attention from law enforcement.  If the police come
knocking you want to have very good answers so they aren't
tempted to lock up your machines as "evidence".

> Does anyone have some hints that would aid in tracing
> what application is sending the mail?  Possible conf
> and logging settings for php/apache?

Do you have mechanisms (revision control) to track changes
to the individual sites?   Search for files that were
changed around the time the spam started.

One of the arguments in favor of virtual machines is that
you can put author on a separate VM, so you narrow down
the search.   When each author knows they can be made
"responsible" for screwups on their site they tend to be
more diligent.

Do you run clamav or the like?

Brute force searches for particular strings used in the spam
might be useful.

George N. White III <aa056 at chebucto.ns.ca>
Head of St. Margarets Bay, Nova Scotia

More information about the nSLUG mailing list