VBSpam email security comparative review - September 2023

Ionuţ Răileanu & Adrian Luca

Virus Bulletin

Copyright © 2023 Virus Bulletin

Introduction

In the Q3 2023 VBSpam test – which forms part of Virus Bulletin’s continuously running security product test suite – we measured the performance of a number of email security solutions against various streams of wanted, unwanted and malicious emails. One third of the solutions we tested opted to be included in the public test, the rest opting for private testing (all details and results remaining unpublished). The solutions tested publicly were eight full email security solutions, one custom configured solution¹, one open-source solution and one blocklist.

This test seemed to run in holiday mode: we saw fewer spam samples than usual, and no major threats or campaigns were detected. The security solutions did a good job at blocking the majority of spam samples. However, in this report we highlight the few cases where the spammers managed to break through the filters.

For some additional background to this report, the table and map below show the geographical distribution (based on sender IP address) of the spam emails seen in the test. (Note: these statistics are relevant only to the spam samples we received during the test period.)

Top 10 countries from which spam was sent.

 Geographical distribution of spam based on sender IP address.

Highlights

Non-English phishing emails

The solutions we tested did a very good job at blocking phishing emails in English. However, when it came to other languages, things were a little different. The majority of missed phishing samples continue to be in languages other than English.

In this test, a sample in Magyar and another one in German were the most commonly missed in the phishing category. Both samples impersonate a postal service email.

Magyar phishing sample.
German phishing sample.

The URL from the Magyar sample was not available at the time of our analysis. The URL² from the German sample leads to a page where the user is asked to disclose data from their credit card.

Website forms spam

Spambots continue to take advantage of website forms. This is not something new, but it continues to be a challenge for security solutions to block these messages in a timely manner. In the current test, these kinds of samples were the ones that were most likely to bypass the anti-spam filters.

Fortunately, the spam URLs are inactive so, beside the fact that these emails may end up in the user’s inbox, they do not present a direct threat, unlike malware and phishing emails.

Spam sample sent via legitimate website form.
Dating spam sent via legitimate website form.

Results

The majority of the tested solutions managed to achieve high catch rates on spam samples overall, and also on the malware sub-category, with values exceeding 99%. We highlight the performance of SEPPmail.cloud Filter, which missed only one phishing sample.

Of the participating full solutions, two achieved a VBSpam award – Mimecast and Zoho Mail – as did the custom configured solution Spamhaus DQS, while six – Bitdefender GravityZone Premium, FortiMail, N-able Mail Assure, N-Able SpamExperts, Net At Work NoSpamProxy and SEPPmail.cloud Filter – were awarded a VBSpam+ certification.

Bitdefender GravityZone Premium

SC rate: 99.93% FP rate: 0.00% Final score: 99.93 Malware catch rate: 99.48% Phishing catch rate: 99.81% Project Honey Pot SC rate: 99.99% Abusix SC rate: 99.93% MXMailData SC rate: 98.69% Newsletters FP rate: 0.0% Speed:   10% 50% 95% 98%

Another VBSpam+ certification is awarded to Bitdefender’s solution, thanks to its balanced performance which included no false positives of any kind and higher than 99% scores in the spam, malware and phishing categories.

Fortinet FortiMail

SC rate: 99.98% FP rate: 0.00% Final score: 99.98 Malware catch rate: 99.94% Phishing catch rate: 99.96% Project Honey Pot SC rate: 99.91% Abusix SC rate: 99.99% MXMailData SC rate: 99.81% Newsletters FP rate: 0.0% Speed:   10% 50% 95% 98%

With higher than 99% scores in the overall spam category, as well as the malware and phishing sub-categories, and no false positives of any kind, Fortinet continues its impressive performance and earns a VBSpam+ award.

Mimecast

SC rate: 99.82% FP rate: 0.09% Final score: 99.39 Malware catch rate: 100.00% Phishing catch rate: 99.83% Project Honey Pot SC rate: 99.51% Abusix SC rate: 99.88% MXMailData SC rate: 100.00% Newsletters FP rate: 0.0% Speed:   10% 50% 95% 98%

A VBSpam certification is awarded to Mimecast for the Q3 2023 test. While correctly blocking all the malware samples and correctly classifying all the newsletter samples, the product achieved a final score exceeding 99.

N-able Mail Assure

SC rate: 99.81% FP rate: 0.00% Final score: 99.77 Malware catch rate: 100.00% Phishing catch rate: 99.60% Project Honey Pot SC rate: 99.86% Abusix SC rate: 99.80% MXMailData SC rate: 100.00% Newsletters FP rate: 1.3% Speed:   10% 50% 95% 98%

N-able Mail Assure continued the run of good performance with another VBSpam+ award. In addition to the impressive 100% malware catch rate we highlight the lack of ham false positives and higher than 99% phishing catch rate.

N-able SpamExperts

SC rate: 99.81% FP rate: 0.00% Final score: 99.77 Malware catch rate: 100.00% Phishing catch rate: 99.60% Project Honey Pot SC rate: 99.86% Abusix SC rate: 99.80% MXMailData SC rate: 100.00% Newsletters FP rate: 1.3% Speed:   10% 50% 95% 98%

 With almost identical scores to its sister product, N-able SpamExperts also earns VBSpam+ certification in this test.

Net At Work NoSpamProxy

SC rate: 99.94% FP rate: 0.00% Final score: 99.94 Malware catch rate: 100.00% Phishing catch rate: 99.79% Project Honey Pot SC rate: 99.96% Abusix SC rate: 99.94% MXMailData SC rate: 100.00% Newsletters FP rate: 0.0% Speed:   10% 50% 95% 98%

It was another balanced performance for Net At Work’s email security solution and another VBSpam+ award to add to its record. We highlight the lack of false positives and the higher than 99.90% spam catch rate.

Rspamd

SC rate: 92.74% FP rate: 0.48% Final score: 90.22 Malware catch rate: 78.83% Phishing catch rate: 91.19% Project Honey Pot SC rate: 88.86% Abusix SC rate: 93.71% MXMailData SC rate: 72.01% Newsletters FP rate: 3.9% Speed:   10% 50% 95% 98%

The open-source Rspamd found dealing with the malware samples a challenge. However, we continue to see good performances from the solution against phishing emails, in this case blocking more than 90% of the samples.

SEPPmail.cloud Filter

SC rate: 99.99% FP rate: 0.00% Final score: 99.99 Malware catch rate: 99.94% Phishing catch rate: 99.98% Project Honey Pot SC rate: 99.94% Abusix SC rate: 100.00% MXMailData SC rate: 99.81% Newsletters FP rate: 0.0% Speed:   10% 50% 95% 98%

The Switzerland-based SEPPmail.cloud Filter managed its best performance in the VBSpam test so far, since joining the test in 2022. With no false positives of any kind and a 99.99% spam catch rate, SEPPmail.cloud Filter is awarded VBSpam+ certification.

Spamhaus Data Query Service + SpamAssassin

SC rate: 98.18% FP rate: 0.00% Final score: 98.18 Malware catch rate: 97.67% Phishing catch rate: 99.39% Project Honey Pot SC rate: 99.96% Abusix SC rate: 97.83% MXMailData SC rate: 95.90% Newsletters FP rate: 0.0% Speed:   10% 50% 95% 98%

Spamhaus SpamAssassin Data Query Service (DQS) is a custom configured solution that integrates the Spamhaus DQS DNSBL service and the free open-source solution SpamAssassin. In this test no ham or newsletter samples were blocked by this combined solution. With a final score of 98.18 the solution earns a VBSpam certification.

Zoho Mail

SC rate: 99.13% FP rate: 0.00% Final score: 99.08 Malware catch rate: 100.00% Phishing catch rate: 98.76% Project Honey Pot SC rate: 97.93% Abusix SC rate: 99.37% MXMailData SC rate: 100.00% Newsletters FP rate: 1.3% Speed:   10% 50% 95% 98%

No malware sample managed to evade Zoho’s filters in this test, and by correctly classifying all the ham samples and achieving a higher than 99 final score, a VBSpam certification is awarded to Zoho Mail.

Abusix Mail Intelligence

SC rate: 98.06%
FP rate: 0.00%
Final score: 98.06
Malware catch rate: 82.39%
Phishing catch rate: 98.80%
Project Honey Pot SC rate: 92.24%
Abusix SC rate: 99.52%
MXMailData SC rate: 66.60%
Newsletters FP rate: 0.0% 

Abusix Mail Intelligence is a set of blocklists that is tested as a partial solution because it has access only to parts of the emails (IP addresses, domains, URLs), which are queried to their DNS zones. With this setup, the solution’s 99.52% spam catch rate and no ham false positives are impressive, and Abusix Mail Intelligence continues its commendable performance.

Results tables

	True negatives	False positives	FP rate	False negatives	True positives	SC rate	Final score	VBSpam
Bitdefender GravityZone Premium	2291	0	0.00%	58	81661.6	99.93%	99.93
Fortinet FortiMail	2291	0	0.00%	18	81701.6	99.98%	99.98
Mimecast	2289	2	0.09%	148	81571.6	99.82%	99.39
N-able Mail Assure	2291	0	0.00%	154	81565.6	99.81%	99.77
N-able SpamExperts	2291	0	0.00%	154	81565.6	99.81%	99.77
Net At Work NoSpamProxy	2291	0	0.00%	49.2	81670.4	99.94%	99.94
Rspamd	2280	11	0.48%	5933.4	75786.2	92.74%	90.22
SEPPmail.cloud Filter	2291	0	0.00%	11	81708.6	99.99%	99.99
Spamhaus DQS + SpamAssassin‡	2291	0	0.00%	1487.2	80232.4	98.18%	98.18
Zoho Mail	2291	0	0.00%	712.8	81006.8	99.13%	99.08
Abusix Mail Intelligence*	2291	0	0.00%	1583.8	80135.8	98.06%	98.06	N/A

^‡Spamhaus Data Query Service (DQS) + SpamAssassin is a fully configured solution that integrates Spamhaus DQS on top of SpamAssassin. Spamhaus DQS is not a stand-alone solution but rather a DNSBL service that can be added to MTAs and email security solutions such as SpamAssasssin. The test set up reflects the real-life performance expected from this combined production deployment, not as individual product elements.
^*This product is a partial solution and its performance should not be compared with that of other products.

^†The standard deviation of a product is calculated using the set of its hourly spam catch rates.
^‡Spamhaus Data Query Service (DQS) + SpamAssassin is a fully configured solution that integrates Spamhaus DQS on top of SpamAssassin. Spamhaus DQS is not a stand-alone solution but rather a DNSBL service that can be added to MTAs and email security solutions such as SpamAssasssin. The test set up reflects the real-life performance expected from this combined production deployment, not as individual product elements.
^*This product is a partial solution and its performance should not be compared with that of other products. None of the queries to the IP blocklist included any information on the attachments; hence its performance on the malware corpus is added purely for information.

	Speed
	10%	50%	95%	98%
Bitdefender GravityZone Premium
Fortinet FortiMail
Mimecast
N-able Mail Assure
N-able SpamExperts
Net At Work NoSpamProxy
Rspamd
SEPPmail.cloud Filter
Spamhaus Data Query Service (DQS) + SpamAssassin‡
Zoho Mail

0-30 seconds

30 seconds to two minutes

two minutes to 10 minutes

more than 10 minutes

^‡Spamhaus Data Query Service (DQS) + SpamAssassin is a fully configured solution that integrates Spamhaus DQS on top of SpamAssassin. Spamhaus DQS is not a stand-alone solution but rather a DNSBL service that can be added to MTAs and email security solutions such as SpamAssasssin. The test set up reflects the real-life performance expected from this combined production deployment, not as individual product elements.

^‡Spamhaus Data Query Service (DQS) + SpamAssassin is a fully configured solution that integrates Spamhaus DQS on top of SpamAssassin. Spamhaus DQS is not a stand-alone solution but rather a DNSBL service that can be added to MTAs and email security solutions such as SpamAssasssin. The test set up reflects the real-life performance expected from this combined production deployment, not as individual product elements.

^‡Spamhaus Data Query Service (DQS) + SpamAssassin is a fully configured solution that integrates Spamhaus DQS on top of SpamAssassin. Spamhaus DQS is not a stand-alone solution but rather a DNSBL service that can be added to MTAs and email security solutions such as SpamAssasssin. The test set up reflects the real-life performance expected from this combined production deployment, not as individual product elements.

Appendix: set-up, methodology and email corpora

The full VBSpam test methodology can be found at https://www.virusbulletin.com/testing/vbspam/vbspam-methodology/vbspam-methodology-ver20.

The test ran for 16 days, from 12am on 5 August to 12am on 21 August 2023 (GMT).

The test corpus consisted of 84,106 emails. 81,738 of these were spam, 13,938 of which were provided by Project Honey Pot, 67,264 were provided by Abusix, with the remaining 536 spam emails provided by MXMailData. There were 2,291 legitimate emails (‘ham’) and 77 newsletters, a category that includes various kinds of commercial and non-commercial opt-in mailings.

23 emails in the spam corpus were considered ‘unwanted’ (see the June 2018 report) and were included with a weight of 0.2; this explains the non-integer numbers in some of the tables.

Moreover, 1,545 emails from the spam corpus were found to contain a malicious attachment while 5,255 contained a link to a phishing or malware site; though we report separate performance metrics on these corpora, it should be noted that these emails were also counted as part of the spam corpus.

Emails were sent to the products in real time and in parallel. Though products received the email from a fixed IP address, all products had been set up to read the original sender’s IP address as well as the EHLO/HELO domain sent during the SMTP transaction, either from the email headers or through an optional XCLIENT SMTP command³.

For those products running in our lab, we all ran them as virtual machines on a VMware ESXi cluster. As different products have different hardware requirements – not to mention those running on their own hardware, or those running in the cloud – there is little point comparing the memory, processing power or hardware the products were provided with; we followed the developers’ requirements and note that the amount of email we receive is representative of that received by a small organization.

Although we stress that different customers have different needs and priorities, and thus different preferences when it comes to the ideal ratio of false positive to false negatives, we created a one-dimensional ‘final score’ to compare products. This is defined as the spam catch (SC) rate minus five times the weighted false positive (WFP) rate. The WFP rate is defined as the false positive rate of the ham and newsletter corpora taken together, with emails from the latter corpus having a weight of 0.2:

WFP rate = (#false positives + 0.2 * min(#newsletter false positives , 0.2 * #newsletters)) / (#ham + 0.2 * #newsletters)

while in the spam catch rate (SC), emails considered ‘unwanted’ (see above) are included with a weight of 0.2.

The final score is then defined as:

Final score = SC - (5 x WFP)

In addition, for each product, we measure how long it takes to deliver emails from the ham corpus (excluding false positives) and, after ordering these emails by this time, we colour-code the emails at the 10th, 50th, 95th and 98th percentiles:

	(green) = up to 30 seconds
	(yellow) = 30 seconds to two minutes
	(orange) = two to ten minutes
	(red) = more than ten minutes

Products earn VBSpam certification if the value of the final score is at least 98 and the ‘delivery speed colours’ at 10 and 50 per cent are green or yellow and that at 95 per cent is green, yellow or orange.

Meanwhile, products that combine a spam catch rate of 99.5% or higher with a lack of false positives, no more than 2.5% false positives among the newsletters and ‘delivery speed colours’ of green at 10 and 50 per cent and green or yellow at 95 and 98 per cent earn a VBSpam+ award.

Meanwhile, products that combine a spam catch rate of 99.5% or higher with a lack of false positives, no more than 2.5% false positives among the newsletters and ‘delivery speed colours’ of green at 10 and 50 per cent and green or yellow at 95 and 98 per cent earn a VBSpam+ award.

Footnotes

¹ Spamhaus Data Query Service (DQS) + SpamAssassin is a custom solution built on top of the SpamAssassin open-source anti-spam platform.

² https://app.any.run/tasks/422706d9-87f0-4681-933d-5c80ae26da9f.

³ http://www.postfix.org/XCLIENT_README.html.

Latest reviews: