Study: Challenge-Response Surpasses Other Anti-Spam Technologies in Performance, User Satisfaction and Reliability; Worst Performing are Filter-based ISP Solutions
Unique Spam Index Based on Real World Experiences Applied to Evaluate Filter, Challenge-Response and Real-time Black List Technologies
NORTHBOROUGH, Mass., July 17, 2007— Brockmann & Company, a research and consulting firm, today released findings from its independent, self-funded “Spam Index Report– Comparing Real-World Performance of Anti-Spam Technologies.”
The study evaluated eight anti-spam technologies from the three main technology classes –filters, real-time black list services and challenge-response servers. The technologies were evaluated using the Spam Index, a new method in anti-spam performance measurement that leverages users’ real-world experiences. The Spam Index was first introduced in the Brockmann & Company report, The Problem with Email. Results of the report are based on the responses of 520 business users of email services from around the world who participated in a 7-minute online survey.
Developed by Brockmann & Company, the Spam Index is based on the number of spam messages that users receive in their inboxes, messages that need to be resent and good messages trapped by the anti-spam system (false positives). Most distinctly, the Spam Index takes into account the time users spend managing spam including deleting junk messages, scanning anti-spam folders and investigating the authenticity of messages that appear legitimate.
According to Peter Brockmann, president of Brockmann & Company and author of the report, “Email is consistently recognized as the most important communications service affecting job performance in virtually all industries and all job roles. Yet spam continues to detract from user productivity by providing too many inappropriate, anonymous, bulk and irrelevant messages. The Spam Index is the first objective real world tool that takes into account real-world user experiences in comparing the effectiveness of anti-spam technologies. The Spam Index uniquely provides the link to real-world user experience, something missing in earlier technology evaluations.”
Key Findings
When evaluating technologies with the Spam Index, lower scores represent the best performing technologies. The report finds that the best performing anti-spam technology is challenge-response, based on that technology’s lowest average Spam Index score of 160. The 160 score is 50% less than the next lowest average score of 316 achieved by hosted services, indicating that challenge response is twice as effective as hosted services for prevention of spam. The worst performing technology was filter-based ISP solutions, with a Spam Index of 442.
The study also found that challenge response is the most consistent technology as shown by its Spam Index standard deviation of 170. The 170 score makes challenge response 55% more consistent than the next most consistent technology, hosted services, with a Spam Index standard deviation of 264. The least consistent technology was filter-based PC email client technology with a standard deviation of 275. This means that respondents who use challenge-response technology consistently have the same experience while users of other technologies have high variability in their experiences.
The study also found that challenge-response users are more satisfied with their overall email experience than any other anti-spam technology users. Some 67% of challenge response users specified that they are very satisfied with their email experience as compared to next highest technology class, hosted services, where 42% reporting that they are very satisfied. The report found that the least satisfied users were those without any anti-spam solution.
Technology Performance Rankings From Best to Worst (Low Spam Index Scores are Best)
The study evaluated eight anti-spam technologies:
Challenge-Response-(Spam Index: 160): This technology allows email from known senders to pass directly to protected users. In some cases, outbound email is monitored to add those addressees to the known sender list. First time senders, which include virtually all spam, are challenged with a reply email requesting that the sender reply to the message in order to assure the original email is delivered. Vendors in this category include Sendio and SpamArrest.
Filter – Hosted service-(Spam Index: 316): Hosted email filtering services use the readily available processing power of Internet data centers. Vendors in this category include AppRiver, MessageLabs, MXLogic and Google/Postini, among others.
Filter – Appliance-(Spam Index: 349): This class integrates software with a hardware appliance, tuning the package for optimal processing performance. Vendors in this category include Barracuda, Borderware, McAfee and Secure Computing, among others.
Filter – Commercial software-(Spam Index: 366): This class tends to be virus filtering software with extended anti-spam functionality. Vendors that offer this class of anti-spam software include McAfee, Symantec and TrendMicro, among others.
Real-time Black Lists-(Spam Index: 367): These reputation-based systems collect feedback from users to manage a ‘black list’ of known spammer IP addresses and domains. Vendors in this category include Commtouch, IronPort and Spamhaus, among others.
Filter – PC Email Client-(Spam Index: 386): This spam filter is a feature of most email clients. Users can train their client spam filters to move or delete mail meeting certain conditions. Vendors in this category include Apple, IBM Lotus and Microsoft, among others.
Filter – Open Source software-(Spam Index: 388): This technology is frequently configured to work in conjunction with PC email client filters. The server adds ****SPAM**** to the subject line so that the client filter can move the message into the junk folder. This class of software includes projects such as ASSP, Mail Washer and SpamAssassin, among others.
Filter – ISP-based-(Spam Index: 442): Offered by most email hosting service providers, this anti-spam technology provides some form of anti-virus and anti-spam filtering for hosted domains. The study showed that this was the least effective solution.
The full Spam Index Report-Comparing Real-World Performance of Anti-Spam Technologies is available from Brockmann & Company (www.brockmann.com).
Brockmann & Company is a consulting & advisory firm serving high tech equipment & application vendors and service providers. Clients accelerate growth through customer research & thought leadership. Learn more at http://www.brockmann.com.
###
