Schonhose

While cleaning my harddrive I found an image I initially wanted to show on this blog. Earlier this year I had some problems with Defensio and this graph does illustrate that.

The horizontal line signifies the period Defensio stopped working and I had to weed out SPAM comments by hand. Interesting enough the number of SPAM comment had increased during this period, as can be seen in the period before and after the outage of the plugin due to a serverconfiguration change. I think the reason for this might be because lots of SPAM comments actually where posted on the blog. With all that exposure spammers might have targeted my blog more frequently than usual (e.g. prior to the outage of Defensio plugin). Defensio has been working great after the last fix, so it makes my life much easier. Thanks for small favours!

It has been a while since my last post and like always I managed to find a good excuse for not posting anything. The last month (!) I have been busy trying to solve a riddle which involved the Defensio addon. As you have probably read on this blog or somewhere else on the web, Defensio is used to protect a blog against comment spam. A few months back I was very excited about the version 2 release of the api and I modified the plugin for the photoblogging software Pixelpost for this new version.

About a month ago I noticed that huge amounts of spam were actually published in my websites. Pretty strange, considering Defensio was active to prevent this behavior. It took some long nights to discover that my hostingcompany had disabled the fsockopen functionality, something that is extensively used by Defensio. So we had to switch to Curl, although that wasn’t completely operational just yet. The guys at Defensio put in some extra hours and provided me with a working test version. After some modifications Defensio was yet again stopping spam in my photoblog.

However, I also have a WordPress blog (you’re reading it) and that also attracts tons of spam. So much in fact I updated some core files of the WP addon with the newly programmed files and a fresh copy from GitHub to enable Defensio again. To bad it didn’t work. Over 100 spam comments a day where published on my blog and had to be removed manually. It took me and Camilo from Defensio about three nights debugging to catch the offending line of code and finally the plugin (which stopped working at March 21, 2010( is fully operational again since last night.

Finally all spam is getting caught again which frees me up to do some more bug hunting in some code I wrote instead.

This week the guys and girls from Defensio have released the 2.0 version of their API. Not only a SPAM filter, but Defensio can now eliminate malware and other unwanted or risky content from your blog. The best part is Pixelpost supports the latest API version through a new release of the Defensio addon for Pixelpost.

A major difference is the way content is evaluated. The old version depended on an instant results after querying Defensio, sometimes resulting in comments not being processed when the service was hammered with request. Therefore the content is now evaluated asynchronous, sending the content to Defensio which will provide a callback when the processing is done.

Initially all comments for Pixelpost make it into the Defensio Quarantine. These comments can basically have two statuses: FAIL and PENDING. The later means that the results through the callback are not in yet (you can issue a query to fetch them right away). The status FAIL means that for some reason Defensio could not be contacted (either the service is down or the API key is invalid or..). In that case you can send these comments to Defensio for an evaluation.

The major drawback of the 1.0 version of the addon was that each comment that had failed had to be rechecked manually. This issue has been resolved with the addition of a new button that processes every comment with a status of FAIL or PENDING for the last two weeks. Obviously the buttons to send either HAM or SPAM to Defensio remain in full effect.

For more information about the new Defensio checkout the “What’s new with Defensio 2.0?” or the video below. You can also download the latest plugin for Pixelpost.

Over the last few months several threads have been posted on the Pixelpost forum regarding SPAM comments. In some of these threads the author boldly claims that Pixelpost isn’t stopping any SPAM. Well, since December 28th, 2006 I have installed a SPAMlog addon on my blog. Basically, what this addon does is keeping track if and why a comment is blocked.

First I think I have to elaborate on my settings, so here they are. I use the <TOKEN> setting from Pixelpost, along with a 30 seconds SPAMflood protection setting and a maximum of three URLS in a comment. Besides that I use the http://BL addon (more info about this addon) and I have installed Defensio addon for Pixelpost.

So now we know the configuration let’s show some stats from the last 743 days shall we?

It seems my photoblog received a total of 31897 comments (43 comments per day). A total of 2457 comments actually made it through the defensive lines of both Pixelpost and the http://BL addon. Defensio managed to catch 2256 comments as SPAM, so this leaves out 201 comments. It turns out that 70 of these comments were SPAM, but these slipped through before the Defensio addon was installed (The Defensio addon was installed a few months after the initialization of the SPAMlog).

These are the numbers, but what happened to the initial 31897 – 2457 = 29440 comments which were blocked by both Pixelpost and the http://BL addon?

Let me break that number down: Pixelpost internal measures took care of 17194 of these comments while the http://BL addon took care of the other 12246 comments. Basically this says that the measures in Pixelpost are capable of catching at least 50% of all SPAM comments.

So what is the internal Pixelpost method that stops most of the SPAM?

• 887 comments contained words listed in ban or moderation list
• 10379 comments used an incorrect token
• 2 commenters waited too long before posting (30 minutes)
• 156 comments were posted to rapidly in succession (SPAM flood)
• 5403 comments contained too many URLs (3)
• 114 comments contained an URL on the blacklist
• 210 comments were not allowed (commenting disabled)
• 41 commenters used an invalid e-mail address

As can be seen the token protection is responsible for 60% of the SPAM stopped by Pixelpost own defensive measures. Feel free to comment on my analysis.

Finally the (long awaited) update for the Defensio addon for Pixelpost is here. As mentioned in this blogpost earlier it contains the fix that slowed down a lot of photoblogs earlier this month.

Thanks to Carl Mercier of the Defensio team I was able to add one more bugfix. Carl was looking at the logs over at Defensio and noticed that sometimes the article-date is not passed to the service. The date is a required parameter and from the very first version I made sure the addon passed it along with the other variables.

This was kind of odd since it was clearly a Pixelpost blog and the date was not passed. I have to say I was stumped on this one, but Carl managed to figure it out. The Defensio addon allows comments to be posted on images that don’t exists yet. Once we (Carl) identified the problem a fix was a matter of a few lines. If the article-date is not in the database, the Defensio addon will throw a nice 404 error.

So there you have it, the best just got better!