VBSpam email security comparative review - June 2023

Ionuţ Răileanu & Adrian Luca

Virus Bulletin

Copyright © 2023 Virus Bulletin

Introduction

In the Q2 2023 VBSpam test – which forms part of Virus Bulletin's continuously running security product test suite – we measured the performance of a number of email security solutions against various streams of wanted, unwanted and malicious emails. One third of the solutions we tested opted to be included in the public test, the rest opting for private testing (all details and results remaining unpublished). The solutions tested publicly were: nine full email security solutions, one custom configured solution¹, one open-source solution and one blocklist.

We continue to see the majority of spam successfully being blocked by email security solutions, and with higher scores this time against malware and phishing samples. Besides keeping a balance between spam and legitimate emails, the challenge for the email security solutions seems to be timely detection of sophisticated threats which appear less frequently and have a short life span.

For some additional background to this report, the table and map below show the geographical distribution (based on sender IP address) of the spam emails seen in the test. (Note: these statistics are relevant only to the spam samples we received during the test period.)

Top 10 countries from which spam was sent.

Geographical distribution of spam based on sender IP address.

Highlights

Remcos RAT

The malware email that was most commonly missed by the solutions in this test contained an archive (InvoiceXTSZX45583.tar.lz), in which there was an scr file containing Remcos, a sophisticated remote access trojan that can be used to control and monitor Windows computers.

We didn't see it as part of a spam campaign. It was detected only once, on 8 May.

Malicious email with attachment containing Remcos.

Agent Tesla

This quarter we continued to see attachments leading to Agent Tesla-infected payloads, this time exploiting the Equation Editor vulnerability in Microsoft Office (CVE‑2017-11882). On opening the attached doc file the infected exe file was downloaded.

We detected only two samples of this kind, on 17 May, which were missed by a few solutions.

Email containing Agent Tesla.

Shipment / postal phishing

In this test the majority of missed phishing emails looked to impersonate postal or parcel shipping services, in a wide variety of languages. There were no large campaigns with many emails of this kind, and both the scarcity of samples and the varied languages they covered seemed to make it a challenge for the security solutions to detect them in real time.

South African Post Office phishing email.

Romanian Post phishing email.
German DHL phishing email.

Banking phishing

The particularity of the banking phishing emails that evaded the filters of most of the security solutions in this test was that they were in German. Posing as emails sent by real banks in order to update the user's personal data or to enable security features, we saw a variety of samples.

Deutsche Bank phishing email.

ING Bank phishing email.

Results

The majority of the tested solutions managed to achieve high spam catch rates, with values exceeding 99%. This is why a better comparison can be made by analysing the malware and phishing catch rates, both of which are subsets of the spam corpus. We highlight the performances of Cleanmail, Mimecast, N-able Mail Assure, N-able SpamExperts and SEPPmail, each with a 100% malware catch rate, and again SEPPmail, which blocked 100% of phishing samples.

Of the participating full solutions, two – Cleanmail Domain Gateway and SEPPmail.cloud Filter – achieved a VBSpam award, while seven – Bitdefender GravityZone Premium, FortiMail, Mimecast, N-able Mail Assure, N-able SpamExperts, Net At Work NoSpamProxy and Zoho Mail – plus the custom configured solution Spamhaus DQS + SpamAssassin were awarded VBSpam+ certification.

Bitdefender GravityZone Premium

SC rate: 99.99% FP rate: 0.00% Final score: 99.99 Malware catch rate: 99.88% Phishing catch rate: 99.95% Project Honey Pot SC rate: 100.00% Abusix SC rate: 99.99% MXMailData SC rate: 99.87% Newsletters FP rate: 0.0% Speed:   10% 50% 95% 98%

Bitdefender's solution continues its solid performance, blocking more than 99.99% of spam samples with no false positives of any kind. It easily earns a VBSpam+ award.

Cleanmail Domain Gateway

SC rate: 99.84% FP rate: 0.03% Final score: 99.68 Malware catch rate: 100.00% Phishing catch rate: 99.91% Project Honey Pot SC rate: 99.67% Abusix SC rate: 99.85% MXMailData SC rate: 100.00% Newsletters FP rate: 0.0% Speed:   10% 50% 95% 98%

Cleanmail Domain Gateway earns VBSpam certification in the Q2 2023 test. No malicious sample passed through Cleanmail's filters and no newsletter sample was blocked.

Fortinet FortiMail

SC rate: 99.98% FP rate: 0.00% Final score: 99.98 Malware catch rate: 99.84% Phishing catch rate: 99.95% Project Honey Pot SC rate: 99.93% Abusix SC rate: 99.99% MXMailData SC rate: 99.74% Newsletters FP rate: 0.0% Speed:   10% 50% 95% 98%

In this test Fortinet showed a similarly impressive performance to those we have seen from it previously. Higher than 99% catch rates on the malware and phishing corpus, combined with no false positives, sets it up for VBSpam+ certification.

Mimecast

SC rate: 99.94% FP rate: 0.00% Final score: 99.88 Malware catch rate: 100.00% Phishing catch rate: 99.15% Project Honey Pot SC rate: 99.49% Abusix SC rate: 99.97% MXMailData SC rate: 100.00% Newsletters FP rate: 2.4% Speed:   10% 50% 95% 98%

VBSpam+ certification is awarded to Mimecast in the Q2 2023 VBSpam test. The solution impresses on this occasion with a 100% catch rate of malware samples and no false positives.

N-able Mail Assure

SC rate: 99.94% FP rate: 0.00% Final score: 99.91 Malware catch rate: 100.00% Phishing catch rate: 99.62% Project Honey Pot SC rate: 99.91% Abusix SC rate: 99.95% MXMailData SC rate: 100.00% Newsletters FP rate: 1.2% Speed:   10% 50% 95% 98%

It was another great performance from N-able Mail Assure in the Q2 2023 test. Blocking 99.94% of the spam samples and with no false positives, VBSpam+ certification is awarded.

N-able SpamExperts

SC rate: 99.94% FP rate: 0.00% Final score: 99.91 Malware catch rate: 100.00% Phishing catch rate: 99.65% Project Honey Pot SC rate: 99.91% Abusix SC rate: 99.95% MXMailData SC rate: 100.00% Newsletters FP rate: 1.2% Speed:   10% 50% 95% 98%

With almost identical scores to its sister product, N-able SpamExperts also earns VBSpam+ certification in this test.

Net At Work NoSpamProxy

SC rate: 99.97% FP rate: 0.00% Final score: 99.97 Malware catch rate: 99.72% Phishing catch rate: 99.91% Project Honey Pot SC rate: 99.70% Abusix SC rate: 99.99% MXMailData SC rate: 99.55% Newsletters FP rate: 0.0% Speed:   10% 50% 95% 98%

Net at Work's email security solution marks its return to VBSpam testing with a VBSpam+ certification. Correctly classifying all the legitimate samples and blocking 99.97% of spam samples, NoSpamProxy shows a balanced performance.

Rspamd

SC rate: 98.19% FP rate: 0.29% Final score: 96.68 Malware catch rate: 72.73% Phishing catch rate: 92.62% Project Honey Pot SC rate: 93.70% Abusix SC rate: 98.71% MXMailData SC rate: 69.79% Newsletters FP rate: 2.4% Speed:   10% 50% 95% 98%

Rspamd scores a higher than 98% catch rate on spam while blocking more than 90% of the phishing samples. The open‑source solution had a challenge in dealing with the malware samples, but we saw an improvement in the product's overall performance from the last test.

SEPPmail.cloud Filter

SC rate: 99.99% FP rate: 0.00% Final score: 99.86 Malware catch rate: 100.00% Phishing catch rate: 100.00% Project Honey Pot SC rate: 99.85% Abusix SC rate: 99.9995% MXMailData SC rate: 100.00% Newsletters FP rate: 4.8% Speed:   10% 50% 95% 98%

SEPPmail.cloud Filter was the only solution to score a 100% catch rate on both the phishing and malware corpus. It was only a high false positive rate in the newsletter corpus that stood in the way of achieving a VBSpam+ award. VBSpam certification is well deserved.

Spamhaus Data Query Service + SpamAssassin

SC rate: 99.81% FP rate: 0.00% Final score: 99.81 Malware catch rate: 98.23% Phishing catch rate: 98.84% Project Honey Pot SC rate: 99.71% Abusix SC rate: 99.83% MXMailData SC rate: 97.75% Newsletters FP rate: 0.0% Speed:   10% 50% 95% 98%

Spamhaus SpamAssassin Data Query Service (DQS) is a custom configured solution that integrates the Spamhaus DQS DNSBL service and the free open-source solution SpamAssassin. In this test no ham or newsletter sample was blocked by the combined solution. With a final score of 99.81 the solution earns VBSpam+ certification.

Zoho Mail

SC rate: 99.61% FP rate: 0.00% Final score: 99.61 Malware catch rate: 99.36% Phishing catch rate: 97.17% Project Honey Pot SC rate: 98.83% Abusix SC rate: 99.67% MXMailData SC rate: 99.04% Newsletters FP rate: 0.0% Speed:   10% 50% 95% 98%

Zoho's solution managed to block more than 99.50% of the spam samples while correctly classifying all ham and newsletter samples, easily reaching the criteria to earn VBSpam+ certification.

Abusix Mail Intelligence

Abusix Mail Intelligence is a set of blocklists that is tested as a partial solution because it has access only to parts of the emails (IP addresses, domains, URLs), which are queried to their DNS zones. With this setup, the solution's 99.08% spam catch rate and lack of ham false positives are impressive, and Abusix Mail Intelligence continues its good performance.

Results tables

	True negatives	False positives	FP rate	False negatives	True positives	SC rate	Final score	VBSpam
Bitdefender GravityZone Premium	3097	0	0.00%	23	230155.8	99.99%	99.99
Cleanmail Domain Gateway	3096	1	0.03%	360.2	229818.6	99.84%	99.68
Fortinet FortiMail	3097	0	0.00%	35.2	230143.6	99.98%	99.98
Mimecast	3097	0	0.00%	132.8	230046	99.94%	99.88
N-able Mail Assure	3097	0	0.00%	128	230050.8	99.94%	99.91
N-able SpamExperts	3097	0	0.00%	127	230051.8	99.94%	99.91
Net At Work NoSpamProxy	3097	0	0.00%	71	230107.8	99.97%	99.97
Rspamd	3088	9	0.29%	4167.8	226011	98.19%	96.68
SEPPmail.cloud Filter	3097	0	0.00%	24	230154.8	99.99%	99.86
Spamhaus DQS + SpamAssassin‡	3097	0	0.00%	433	229745.8	99.81%	99.81
Zoho Mail	3097	0	0.00%	902	229276.8	99.61%	99.61
Abusix Mail Intelligence*	3097	0	0.00%	2124.6	228054.2	99.08%	99.01	N/A

^‡Spamhaus Data Query Service (DQS) + SpamAssassin is a fully configured solution that integrates Spamhaus DQS on top of SpamAssassin. Spamhaus DQS is not a stand-alone solution but rather a DNSBL service that can be added to MTAs and email security solutions such as SpamAssasssin. The test set up reflects the real-life performance expected from this combined production deployment, not as individual product elements.
^*This product is a partial solution and its performance should not be compared with that of other products.

^†The standard deviation of a product is calculated using the set of its hourly spam catch rates.
^‡Spamhaus Data Query Service (DQS) + SpamAssassin is a fully configured solution that integrates Spamhaus DQS on top of SpamAssassin. Spamhaus DQS is not a stand-alone solution but rather a DNSBL service that can be added to MTAs and email security solutions such as SpamAssasssin. The test set up reflects the real-life performance expected from this combined production deployment, not as individual product elements.
^*This product is a partial solution and its performance should not be compared with that of other products. None of the queries to the IP blocklist included any information on the attachments; hence its performance on the malware corpus is added purely for information.

 

	Speed
	10%	50%	95%	98%
Bitdefender GravityZone Premium
Cleanmail Domain Gateway
Fortinet FortiMail
Mimecast
N-able Mail Assure
N-able SpamExperts
Net At Work NoSpamProxy
Rspamd
SEPPmail.cloud Filter
Spamhaus Data Query Service (DQS) + SpamAssassin‡
Zoho Mail

0-30 seconds

30 seconds to two minutes

two minutes to 10 minutes

more than 10 minutes

^‡Spamhaus Data Query Service (DQS) + SpamAssassin is a fully configured solution that integrates Spamhaus DQS on top of SpamAssassin. Spamhaus DQS is not a stand-alone solution but rather a DNSBL service that can be added to MTAs and email security solutions such as SpamAssasssin. The test set up reflects the real-life performance expected from this combined production deployment, not as individual product elements.

^‡Spamhaus Data Query Service (DQS) + SpamAssassin is a fully configured solution that integrates Spamhaus DQS on top of SpamAssassin. Spamhaus DQS is not a stand-alone solution but rather a DNSBL service that can be added to MTAs and email security solutions such as SpamAssasssin. The test set up reflects the real-life performance expected from this combined production deployment, not as individual product elements.

^‡Spamhaus Data Query Service (DQS) + SpamAssassin is a fully configured solution that integrates Spamhaus DQS on top of SpamAssassin. Spamhaus DQS is not a stand-alone solution but rather a DNSBL service that can be added to MTAs and email security solutions such as SpamAssasssin. The test set up reflects the real-life performance expected from this combined production deployment, not as individual product elements.

Appendix: set-up, methodology and email corpora

The full VBSpam test methodology can be found at https://www.virusbulletin.com/testing/vbspam/vbspam-methodology/vbspam-methodology-ver20.

The test ran for 16 days, from 12am on 6 May to 12am on 22 May 2023 (GMT).

The test corpus consisted of 233,390 emails. 230,210 of these were spam, 15,023 of which were provided by Project Honey Pot, 213,628 were provided by Abusix with the remaining 1,559 spam emails provided by MXMailData. There were 3,097 legitimate emails ('ham') and 83 newsletters, a category that includes various kinds of commercial and non-commercial opt-in mailings.

39 emails in the spam corpus were considered 'unwanted' (see the June 2018 report) and were included with a weight of 0.2; this explains the non-integer numbers in some of the tables.

Moreover, 2,490 emails from the spam corpus were found to contain a malicious attachment while 4,242 contained a link to a phishing or malware site; though we report separate performance metrics on these corpora, it should be noted that these emails were also counted as part of the spam corpus.

Emails were sent to the products in real time and in parallel. Though products received the email from a fixed IP address, all products had been set up to read the original sender’s IP address as well as the EHLO/HELO domain sent during the SMTP transaction, either from the email headers or through an optional XCLIENT SMTP command².

For those products running in our lab, we all ran them as virtual machines on a VMware ESXi cluster. As different products have different hardware requirements – not to mention those running on their own hardware, or those running in the cloud – there is little point comparing the memory, processing power or hardware the products were provided with; we followed the developers’ requirements and note that the amount of email we receive is representative of that received by a small organization.

Although we stress that different customers have different needs and priorities, and thus different preferences when it comes to the ideal ratio of false positive to false negatives, we created a one-dimensional ‘final score’ to compare products. This is defined as the spam catch (SC) rate minus five times the weighted false positive (WFP) rate. The WFP rate is defined as the false positive rate of the ham and newsletter corpora taken together, with emails from the latter corpus having a weight of 0.2:

WFP rate = (#false positives + 0.2 * min(#newsletter false positives , 0.2 * #newsletters)) / (#ham + 0.2 * #newsletters)

while in the spam catch rate (SC), emails considered ‘unwanted’ (see above) are included with a weight of 0.2.

The final score is then defined as:

Final score = SC - (5 x WFP)

In addition, for each product, we measure how long it takes to deliver emails from the ham corpus (excluding false positives) and, after ordering these emails by this time, we colour-code the emails at the 10th, 50th, 95th and 98th percentiles:

	(green) = up to 30 seconds
	(yellow) = 30 seconds to two minutes
	(orange) = two to ten minutes
	(red) = more than ten minutes

Products earn VBSpam certification if the value of the final score is at least 98 and the ‘delivery speed colours’ at 10 and 50 per cent are green or yellow and that at 95 per cent is green, yellow or orange.

Meanwhile, products that combine a spam catch rate of 99.5% or higher with a lack of false positives, no more than 2.5% false positives among the newsletters and ‘delivery speed colours’ of green at 10 and 50 per cent and green or yellow at 95 and 98 per cent earn a VBSpam+ award.

Footnotes

¹ Spamhaus Data Query Service (DQS) + SpamAssassin is a custom solution built on top of the SpamAssassin open-source anti-spam platform.

² http://www.postfix.org/XCLIENT_README.html.

Latest reviews: