2 weeks ago, my Mac OS X 10.4 server running this site was hacked.
How I learned about it was from my email integrity appliance vendor, Sendio. They called to tell me that my mail server was blasting spam. I stopped what I was doing, asked them to lock down my message queues and I immediately began to search for what had been compromised, where the hole was and how to fix it.
I checked my configuration files on the site to determine if any files had changed beyond the few that I do to keep the site running smoothly. Nope. Nothing to report.
I checked my user logs to see if somebody's passwords had been compromised – nothing here.
I checked my SMTP logs to confirm that yup, it was my mail server spewing the filth.
I checked my network logs to see if there was a rogue PC on my LAN. Nope. All machines accounted for.
I checked the database logs to see if something had been copied or fiddled with. Negative.
I read the forum here. I found this through a search on the phrase 'joomla hacked'.
By scanning my web server logs, I discovered that the hacker had probed my php-based content management system (joomla) implementation and exploited a pretty common security hole around global_registers and .htaccess controls. [My bad].
The repairs were implemented within a few hours of detecting the hack.
The hard part was to keep the mail service up to clean out all the spam built up in into its queue. This took another day to calm down. Then we opened the mail queue and resent the mail through the mail server. During the cleanup, the Sendio I.C.E. Box accepted mail, but because of the way the Mac OS server forwards mail, it didn't get to the few users that depend on forwarded mail. For them, the Sendio I.C.E. Box user interface kept them aware of messages coming to them, but it's not meant to be a webmail interface. Within three days, everything was back to normal.
This figure shows the typical layout of the data center. Normal web traffic and email traffic are shown in blue arrows. The hacker (red triangle on the right) sent the offending URL to the web server which created the bad mail and passed it through the phpmail system to the mail server shown as the thick red arrow.
The standard mail MX configuration is to send all incoming mail to the Sendio I.C.E. Box for processing. The appliance performs grey listing, anti-virus and challenge-response for first time senders. The standard server configuration is to relay all outbound messages to the Sendio I.C.E. Box for outbound message mediation. Here it confirms the addressee is in my accept queue and adds them if a first time addressee and checks anti-virus.
It was the abnormal outbound message load (40,000 in an hour) that set off alarms in the I.C.E.Box and at Sendio. Solid email integrity.
