Posted by Virus Bulletin on Feb 4, 2013
Excellent performances in spam filter test - plus evidence of a correlation between spam 'passing' SPF and an increased delivery rate.
No fewer than ten anti-spam solutions achieved a VBSpam+ award in VB's latest spam filter test, with another ten achieving a standard VBSpam award.
The results of the latest VBSpam test are good news for millions of users of spam filters. Spam filters have become pretty good at blocking spam and for this reason we felt it was appropriate to increase the threshold for obtaining a VBSpam award. Even having done this, 20 of the products in the test peformed well enough to achieve a VBSpam award, and of those, ten reached the required standard to earn a VBSpam+ award - showing a spam catch rate of 99.50% or more, without misclassifying any legitimate emails.
Of the products that achieved a VBSpam+ award, OnlyMyEmail, ZEROSPAM and Bitdefender had the highest spam catch rates, with OnlyMyEmail impressing the team by being the first product not to misclassify a single message in a corpus consisting of more than 70,000 emails.
SPF status
Alongside the product tests, the VBSpam team looked at the SPF status of the spam messages in the corpus to see if this had any impact on the likelihood of the messages being blocked.
SPF (Sender Policy Framework) is a mechanism designed to 'protect' the sending domain of the email, by indicating (via DNS) which IP addresses are authorized to send email from that domain. While SPF is no silver bullet for stopping spam, or even preventing the abuse of a sending domain, the use of SPF is recommended by organisations such as MAAWG, and most large senders have implemented an SPF policy.
Excluding domains for which the SPF record could not be parsed, or for which no DNS record was available when the email was sent, we distinguished five different SPF statuses: 'pass' ('this IP address is allowed to send email from this domain'); 'fail' ('this email is not from us'); 'softfail' ('it's probably not from us, but we can't be sure'); 'neutral' ('we don't know') and 'none' (no SPF record avialable). The table below shows how many products (out of 21) failed to block the emails with the various statuses.
| SPF status | No. of emails | No. of products that failed to block emails (average) | Standard deviation |
|---|---|---|---|
| fail | 3171 | 0.24 | 0.04 |
| pass | 8106 | 0.93 | 0.23 |
| softfail | 8672 | 0.45 | 0.09 |
| neutral | 13466 | 0.34 | 0.04 |
| none | 26938 | 0.43 | 0.06 |
As can be seen from the table, and possibly more clearly from the graph below, the is a correlation between the SPF status and the likelihood of a spam message being missed, with SPF fails more likely to be blocked and SPF passes more likely to be missed.
It is important to note that this doesn't necessarily imply a causal relationship: it may well be that the SPF passes were mostly emails sent from compromised accounts, which are harder to block for different reasons. Also, the results can be read to suggest that spammers should increase their usage of SPF if they want to increase their delivery rates. This may be true, but it may not necessarily be a bad thing: passing SPF merely adds to the veracity of the sending domain. If the domain belongs to a spammer, it will make blocking easier.
For Virus Bulletin subscribers, the full test report is available here. Non-subscribers can purchase the report as a standalone article ($19.95) here.
More on the VBSpam tests, including historical performance of the participating products, can be found here.
Posted on 04 February 2013 by Martijn Grooten
