'Hotmail and Gmail have best spam filter' says Cascade spam test

Posted by   Virus Bulletin on   Feb 15, 2012

Comparative test did not take false positives into account.

Researchers from Cascade Insights performed a comparative spam filtering test on the three major webmail providers and concluded that Hotmail performed best, shortly followed by Gmail, with Yahoo! a distant third.

The researchers registered accounts at all three providers and, for comparison, a fourth one at an ISP that did not perform any kind of filtering. They then seeded these accounts by sending 'replies' to known spammers, posting the addresses on public sites and submitting them to dubious-looking websites.

For the test period, the researchers manually classified all the emails they received, marking as 'ham' all legitimate emails as well as those that were solicited - even if they were scams. They then calculated the percentage of spam in each of the inboxes and found that a little less than half of the email in Hotmail's inbox was spam, a ratio almost matched by Gmail. In Yahoo!'s case, over 58 per cent of the inbox consisted of spam, compared to close to two thirds for the unfiltered account.

The researchers did not look at false positive rates. While they were right to say that the emails they classified as 'ham' (many of which were of dubious nature) weren't representative of a real stream of legitimate emails, it is a shame that they didn't find another way to send legitimate mail to the accounts - avoiding false positives is an important part of running a spam filter and a good spam catch rate is useless if many legitimate emails are blocked as well.

A possibly more serious flaw in the test was the way performance was measured. Measuring the percentage of spam in the inbox is a nice idea. After all, users of free webmail services care little about the amount of spam sent to them; rather, they care about the amount of spam that makes it into their inbox. However, in using this method, the researchers implicitly take the 'ham' in their mail streams into account - the same messages that they previously claimed could not be used for measuring false positives.

The full report, which includes a detailed description of the methodology used, can be download as a PDF here.

Posted on 15 February 2012 by Virus Bulletin

twitter.png
fb.png
linkedin.png
hackernews.png
reddit.png

 

Latest posts:

VBSpam tests to be executed under the AMTSO framework

VB is excited to announce that, starting from the Q3 test, all VBSpam tests of email security products will be executed under the AMTSO framework.

In memoriam: Prof. Ross Anderson

We were very sorry to learn of the passing of Professor Ross Anderson a few days ago.

In memoriam: Dr Alan Solomon

We were very sorry to learn of the passing of industry pioneer Dr Alan Solomon earlier this week.

New paper: Nexus Android banking botnet – compromising C&C panels and dissecting mobile AppInjects

In a new paper, researchers Aditya K Sood and Rohit Bansal provide details of a security vulnerability in the Nexus Android botnet C&C panel that was exploited in order to gather threat intelligence, and present a model of mobile AppInjects.

New paper: Collector-stealer: a Russian origin credential and information extractor

In a new paper, F5 researchers Aditya K Sood and Rohit Chaturvedi present a 360 analysis of Collector-stealer, a Russian-origin credential and information extractor.

We have placed cookies on your device in order to improve the functionality of this site, as outlined in our cookies policy. However, you may delete and block all cookies from this site and your use of the site will be unaffected. By continuing to browse this site, you are agreeing to Virus Bulletin's use of data as outlined in our privacy policy.