Introducing VB anti-spam testing

According to my mother, dolphins are the best spam filter. She told me that ever since her email program was configured so that an image of dolphins was added to the bottom of every email she sent, she had stopped receiving emails about Viagra.

Unfortunately, the dolphins did not really eat the spam. As it turned out, the addition of the image to her email footers coincided with her starting to use a new email address, which of course was the real reason for the reduction in spam. And, while the dolphins are still at the bottom of each email she sends, the spam has now returned.

So what is the best spam filter? End-users have little factual information upon which to base their choice of filter, and even the vendors themselves have little idea of their products’ performance compared to that of their competitors.

It seems that there is a need for independent and regular spam filter performance testing, and Virus Bulletin has decided to use the experience gained during more than a decade of anti-malware comparative testing to develop a regular anti-spam product test. Following months of internal discussion, culminating in an informal, yet fruitful meeting with members of the anti-spam industry at the last VB conference, a test methodology has been drawn up. Testing will start during the early months of 2009; this article outlines the proposed test set-up.

While writing this article, I have received many emails from representatives of anti-spam companies politely, yet impatiently enquiring as to when we will be ready to start anti-spam testing. There certainly is big demand for product testing from within the industry: companies want to know how well they are performing compared to their competitors. If they are doing well, they want to boast about it to potential customers, while if they fail to live up to their promises, they want to know what parts of their product they need to improve. Just as Virus Bulletin has been running anti-malware tests for over a decade and helping AV developers improve their products, the results of VB’s anti-spam tests will help anti-spam vendors improve their filters.

More obviously, anti-spam customers want to know how well their chosen product, or one they are interested in purchasing, has performed in tests: whether independent sources back up the claims made by the vendors and whether those claims are based on long-term good performance.

However, a comparative test of anti-spam filters is more than just the comparison of various products: as different filters use different filtering methods, the test will implicitly compare these methods as well. This information should be valuable for the anti-spam industry as a whole.

In theory, setting up a comparative anti-spam test is easy: one sets up a forwarder that sends incoming mail to n mailboxes, one for each product to be tested. After a fixed period of time, one counts the number of misclassified ham messages and misclassified spam messages and thus ends up with a two-dimensional score for each product.

In practice, a multitude of problems arise, especially if one wants the anti-spam test to reflect a real-life situation. The first of these is the actual definition of spam, over which there is not more than a vague general agreement. But even assuming that one does have a satisfying definition of spam, it is not straightforward even to manually sort all incoming email according to this definition. If we at Virus Bulletin were able to do this with 100% accuracy in an automated way, we would likely quit our spam testing and start selling a classifier-filter commercially.

But even sidestepping the classification and definition issues, it is possible that the set-up described above will confuse spam filters. These might see an email claiming to be from [email protected] that, from the filter’s point of view, is sent from the IP address of the forwarder. Based on this information a filter might (incorrectly) decide to classify the email as spam. To deceive spammers, a filter might temporarily block an email and ask the sender to resend it after a given amount of time (a method known as greylisting); while the forwarder could be programmed to obey this request, the fact that different filters will send different ‘blocking messages’ means that it would never be able to do exactly what the original sender would have done.

The list of problems with possible anti-spam test set-ups is much longer than the few highlighted above. However, we do believe that it is possible to run good, representative and reliable anti-spam tests. To this end, we have set five conditions that our tests will fulfil:

In our opinion, any good anti-spam test should fulfil the six conditions listed above.

However, it would be an audacious claim to state that following these guidelines will guarantee that a test will be faultless. The reason for this is deeper than the fact that any test based on statistics is bound to incorporate some error. Many spam filters block traffic at the SMTP level, based on the IP address of the sender, the content of the MAIL FROM or RCPT TO commands or a combination of these. When this happens, the email is never received and while this generally boosts the filter’s performance, it is impossible for the tester to decide whether the unreceived email was spam or ham.

Virus Bulletin intends to test spam filters using two test set-ups per product.

Although the spam catch rate and the false positive rate are the most obvious metrics one would look at to compare anti-spam products, they are far from the only ways to describe a product’s performance.

Therefore, although our initial focus will be on measuring spam catch and false positive rates, we hope in the future to look into other metrics too. These include both speed (how long does it take for a legitimate email to reach the user’s inbox?) and performance (how much CPU does the spam filter use?), but may also include metrics such as the relative amount of spam blocked during the SMTP transaction.

Other metrics that may be published with the test results include the standard variation in a product’s spam catch rate from its daily or hourly average. Or, in the event of a particularly large spam outbreak coinciding with a test, the products’ response times to that outbreak.

The general definition of spam upon which the VB staff will base classification decisions is that of unsolicited bulk email: email sent in large quantities to users that have not explicitly given their consent to receive such email. This does not mean that everything else will be automatically classified as ham: we allow for the existence of a third category of ‘unclassifiable’ emails: ones which the recipient is unable to label as definitely ham or definitely spam. For instance, these may be messages sent to a predecessor’s email address, and the recipient may have no way of knowing whether or not their predecessor had consented to receive mail from the sender. This category of emails will be removed entirely from the computation of the products’ performance.

Even with this third category, we are aware that no end-user will be able to classify all the email they receive in a wholly consistent and accurate way. We believe that any inaccuracy introduced in this way will in fact reflect a real-life situation, and that the end-user’s perception of a filter’s performance is as important as its performance compared to any formal definition of spam.

To take part in Virus Bulletin’s anti-spam tests, products must be able to accept SMTP transfers and classify email into two categories: ham and spam. Products might have additional categories, such as ‘possibly spam’, and it will be decided on a product-by-product basis as to whether such categories are taken to mean ham or spam; this will always be done keeping the end-user in mind and will be reported with the test results.

Products must not send temporary failures to the sending SMTP server or, more generally, do anything that requires the sending server to reinitiate the connection.

Products must be able to log the results of their filters in such a way that testers can easily run a script that reads the relevant data from the results. Storing the email in two folders in a standard mail folder format, such as mbox, also counts as logging.

The products will be installed and configured using their default settings.

Products will be allowed to connect to the Internet at any time, to allow them to update themselves and to be able to test incoming email against live blacklists and whitelists.

Every test will start with a clean install of both the product and the operating system.

Neither end-users nor testers will report the results of the filters back to the products during testing.

Results of the tests, which we anticipate running six times per year, will be published in the Spam Supplement section of Virus Bulletin magazine (available only to Virus Bulletin subscribers), with a basic summary of the results available free of charge to all registered users on the VB website.

The best-performing products will be awarded a certification, similar to the current VB100 awards for anti-malware products. The precise criteria for obtaining these awards will be decided once we have started testing properly, but our current aim is for the best-performing 50 per cent to achieve the award.

Given the cost of testing, which needs to be performed in parallel and will require the use of a separate machine for each product, we will be charging companies to take part in our tests. Of course, included in the fee will be the right to display the award (if achieved) on the company’s website and product literature. Moreover, companies taking part will be provided with feedback on their products’ performance, which may include a full overview of the spam their product has missed and, after anonymizing, ham emails that were wrongly classified as spam.

We are well aware of the existence of free, open-source anti-spam products and do not wish our tests to exclude these. Therefore, developers of products that are available entirely free of charge, open-source and that contain no in-product advertising will not be charged to enter their products in our tests. (However, VB reserves the right to limit the number of such products in each test on a first-come-first-served basis.)

We will be the first to admit that our tests will not be 100% accurate. However, we intend for our tests to be as close to reality as possible, and of course will continue to look for ways in which the methodology can be tweaked and improved.

VB welcomes readers’ feedback on the proposed test methodology, as well as enquiries from developers interested in submitting their products for testing. Please direct all comments and enquiries to [email protected].