VBSpam Email Security Comparative Review - December 2021

Ionuţ Răileanu & Adrian Luca

Virus Bulletin

Copyright © 2021 Virus Bulletin

Introduction

In this test – which forms part of Virus Bulletin’s continuously running security product test suite – nine full email security solutions, one custom configured solution¹, one open-source solution and one blocklist were assembled on the test bench to measure their performance against various streams of wanted, unwanted and malicious emails. In this round of testing we welcome two new participants to the public VBSpam test: Check Point and Zoho Mail.

With the email threat landscape constantly changing, and with the aim of diversifying the Malware corpus, this test includes emails from MXMailData, an Australia-based company. Of all the samples received from MXMailData, for the test we selected and sent to the products only those containing attachments that had a high possibility of being malicious.

For some additional background to this report, the table and map below show the geographical distribution (based on sender IP address) of the spam emails seen in the test. (Note: these statistics are relevant only to the spam samples we received during the test period.)

Top 10 countries from which spam was sent.

Geographical distribution of spam based on sender IP address.

Malware and phishing

The majority of the samples from these two categories were successfully blocked by the security solutions we tested. However, there were some samples that stood out and, at least initially, managed to evade the filters. In the following sections we describe some of them.

Bazar Loader²

Active during 9 and 10 November, and again briefly on 15 November, this malspam campaign was the most challenging for the solutions participating in the test to block. The format is something we have seen in the past: a short text and a password-protected attachment. The ‘.doc’ file extracted from the attached archive contains a short text written in white that is saved on the computer to an ‘.hta’ file. Through mshta.exe, the ‘.hta’ file runs and connects to the attacker’s server (which, in one of the samples, was shoulderelliottd[.]com).

Attachment SHA256:

7a9b3945f42e8e3ee3272501bfa9ebf842140ca740f0698c56e0d3f90445eb4a

0c00919c17f1fd52d965b933e46d80e2d25ae4d162920fda8a9772763e07224b

70cfd7872c56ccb75008b2db299179fa3ee3293ca910b5582966f97db759b45b

Name:

Info.zip

request.zip

 Email from Bazar Loader campaign.

Dridex

This was a large malspam campaign (making up 11% of the Malware corpus) that we observed on 10 November and again on 15 November. Even though the majority of the emails were blocked by the tested solutions, there were a few that managed to bypass their filters. The attached ‘.xll’ files were reported³ to be linked to Dridex.

Email with attachment linked to Dridex.

German phishing campaign⁴

This was the largest phishing campaign we saw during the test period. It targeted German speakers and contained a URL leading to a PHP page that redirected to the actual malicious link. The only such redirect that we managed to catch was hxxps://spkfinanzverifikation[.]com/OIYZ6MP95A. At the time of writing this report, the URLs were inactive.

German phishing email.

Netflix phishing

We highlight these phishing emails because they were the ones that evaded most of the security solutions filters. They were spotted on 19 November, being sent from the same IP address (217[.]66[.]226[.]61) and contained the URL hxxps://www[.]pun-entertainment[.]com/app/login/nz/.

Netflix phishing email.

Results

All of the tested solutions managed to block more than 98% of the spam samples, with Cleanmail and Libraesva being the only solutions that blocked 100% of the malware samples. Libraesva also had the best phishing catch rate.

Of the participating full solutions, five achieved a VBSpam award – Axway, Check Point, Cleanmail, Libraesva and N-able Mail Assure – as did the custom-configured solution Spamhaus Data Query Service (DQS) + SpamAssassin. A further three achieved a VBSpam+ award: Bitdefender, Fortinet and Net at Work.

Axway MailGate 5.6

SC rate: 99.88% FP rate: 0.06% Final score: 99.56 Malware catch rate: 96.65% Phishing catch rate: 99.47% Project Honey Pot SC rate: 99.75% Abusix SC rate: 99.91% MXMailData SC rate: 95.99% Newsletters FP rate: 0.9% Speed:   10% 50% 95% 98%

Axway has performed well in previous tests and continues to produce spam catch rates that exceed 99.5%. With only 11 samples missed from the phishing corpus, Axway achieved the third highest phishing catch rate in this test. While two false positives stood in the way of a VBSpam+ award, Axway earns a VBSpam award with ease.

Bitdefender Security for Mail Servers 3.1.7

SC rate: 99.97% FP rate: 0.00% Final score: 99.97 Malware catch rate: 99.38% Phishing catch rate: 99.56% Project Honey Pot SC rate: 99.98% Abusix SC rate: 99.97% MXMailData SC rate: 99.67% Newsletters FP rate: 0.0% Speed:   10% 50% 95% 98%

Bitdefender has participated in every VBSpam test since they began in May 2009 and has performed well in every one of them – and it’s the only product that has earned a VBSpam+ award in each of the four VBSpam tests in 2021. With impressive catch rates of more than 99% not only in the spam corpus but also in the malware and phishing samples, combined with a lack of false positives on both ham and newsletters, Bitdefender proves itself once again to a dependable solution. With the highest final score in this test, Bitdefender’s developers earn another VBSpam+ award to add to their collection.

Check Point - Harmony Email & Office

SC rate: 99.25% FP rate: 0.21% Final score: 98.19 Malware catch rate: 98.40% Phishing catch rate: 98.74% Project Honey Pot SC rate: 98.66% Abusix SC rate: 99.30% MXMailData SC rate: 98.66% Newsletters FP rate: 0.9% Speed:   10% 50% 95% 98%

 One of the new entries in this test, Check Point makes a great impression on its debut. With a steady and well adjusted performance, blocking more than 99% of the spam samples Check Point earns a VBSpam award.

Cleanmail Domain Gateway

SC rate: 99.95% FP rate: 0.15% Final score: 99.04 Malware catch rate: 100.00% Phishing catch rate: 99.27% Project Honey Pot SC rate: 99.85% Abusix SC rate: 99.95% MXMailData SC rate: 100.00% Newsletters FP rate: 5.7% Speed:   10% 50% 95% 98%

Only two of the participating solutions managed to block all the malware samples in this test, Cleanmail being one of them. To add to this impressive feat, the product achieved a higher than 99% phishing catch rate. In this test Cleanmail earns another VBSpam award, its fourth in 2021.

Fortinet FortiMail

SC rate: 99.90% FP rate: 0.00% Final score: 99.90 Malware catch rate: 99.12% Phishing catch rate: 95.49% Project Honey Pot SC rate: 99.80% Abusix SC rate: 99.91% MXMailData SC rate: 99.33% Newsletters FP rate: 0.0% Speed:   10% 50% 95% 98%

Fortinet’s FortiMail appliance has long performed well in our tests. Its final scores have exceeded 99.60 in each of this year’s tests. On this occasion it was one of only two solutions with no false positives in either the ham or the newsletter corpus, and to add to this impressive performance it achieved a 99.90% spam catch rate. These results earn the product another VBSpam+ award, its third in a row this year.

Libraesva ESG v.4.7

SC rate: 99.88% FP rate: 0.03% Final score: 99.70 Malware catch rate: 100.00% Phishing catch rate: 99.71% Project Honey Pot SC rate: 99.83% Abusix SC rate: 99.88% MXMailData SC rate: 100.00% Newsletters FP rate: 0.9% Speed:   10% 50% 95% 98%

Not only was Libraesva one of the two solutions that blocked 100% of the malware samples in this test, but it also achieved the highest phishing catch rate, 99.71%. The Italian solution has had a good year in the VBSpam test, having been awarded a VBSpam+ in two of the four quarterly tests. In this, the last test of the year, Libraesva is awarded a VBSpam certification.

N-able Mail Assure

SC rate: 99.90% FP rate: 0.03% Final score: 99.72 Malware catch rate: 99.95% Phishing catch rate: 98.74% Project Honey Pot SC rate: 99.79% Abusix SC rate: 99.91% MXMailData SC rate: 99.93% Newsletters FP rate: 0.9% Speed:   10% 50% 95% 98%

This is the second time N-able Mail Assure has participated in the VBSpam test, having earned a VBSpam+ award on its debut. On this occasion the product’s performance on the malware corpus stands out, with only one sample missed. With a spam catch rate of 99.90% and a final score of 99.72, N-able Mail Assure is awarded a VBSpam certification in this test.

Net at Work NoSpamProxy

SC rate: 99.79% FP rate: 0.00% Final score: 99.76 Malware catch rate: 99.90% Phishing catch rate: 98.06% Project Honey Pot SC rate: 99.07% Abusix SC rate: 99.84% MXMailData SC rate: 99.93% Newsletters FP rate: 0.9% Speed:   10% 50% 95% 98%

Net at Work’s developers have reason to be proud of their product’s performance in this test, given the excellent spam, phishing and malware catch rates, combined with zero false positives in the ham corpus. The product earns a VBSpam+ award on this occasion.

Rspamd

SC rate: 98.43%
FP rate: 0.74%
Final score: 94.68
Malware catch rate: 80.69%
Phishing catch rate: 90.01%
Project Honey Pot SC rate: 92.48%
Abusix SC rate: 99.00%
MXMailData SC rate: 80.11%
Newsletters FP rate: 2.8%

Speed:

10%

50%

95%

98%

Rspamd, a free and open-source solution, joined the VBSpam test in March 2021, and although the product has yet to achieve VBSpam certification, its performance is improving with each test. Worthy of note on this occasion are the 98.43% spam catch rate and the 90.01% phishing catch rate.

Spamhaus Data Query Service (DQS) + SpamAssassin

SC rate: 99.56% FP rate: 0.00% Final score: 99.45 Malware catch rate: 98.04% Phishing catch rate: 98.25% Project Honey Pot SC rate: 99.11% Abusix SC rate: 99.61% MXMailData SC rate: 98.40% Newsletters FP rate: 3.8% Speed:   10% 50% 95% 98%

Spamhaus Data Query Service (DQS) + SpamAssassin is a custom configured solution that integrates the Spamhaus DQS DNSBL service with the free open-source SpamAssassin solution. In this test the product impressed with a 99.56% spam catch rate and no false positives in the ham corpus. It just misses out on a VBSpam+ award due to a small number of newsletter false positives, but it earns VBSpam certification nevertheless.

Zoho Mail

SC rate: 99.40%
FP rate: 0.24%
Final score: 97.88
Malware catch rate: 98.46%
Phishing catch rate: 98.84%
Project Honey Pot SC rate: 98.24%
Abusix SC rate: 99.50%
MXMailData SC rate: 98.66%
Newsletters FP rate: 11.3%

Speed:

10%

50%

95%

98%

Zoho Mail is the second new entry in this test. Achieving an impressive 99.40% spam catch rate, and with phishing and malware catch rates higher than 98%, Zoho Mail was on track to earn VBSpam certification. However, a number of misclassified ham and newsletter samples lowered the product’s final score to 97.88, just beneath the threshold for certification on this occasion.

Abusix Mail Intelligence

SC rate: 99.69%
FP rate: 0.00%
Final score: 99.66
Malware catch rate: 87.08%
Phishing catch rate: 98.79%
Project Honey Pot SC rate: 98.35%
Abusix SC rate: 99.86%
MXMailData SC rate: 85.38%
Newsletters FP rate: 0.9%

Abusix Mail Intelligence is a set of blocklists that is tested as a partial solution because it has access only to parts of the emails (IP addresses, domains, URLs), which are queried as to their DNS zones. With this setup, the very high (99.69%) spam catch rate and the lack of false positives in the ham corpus is very impressive. The reason there is no award for this product is that the criteria for VBSpam certification include ham delivery speed values, which are calculated only for full solutions.

Results tables

	True negatives	False positives	FP rate	False negatives	True positives	SC rate	Final score	VBSpam
Axway	3383	2	0.06%	429.4	353224	99.88%	99.56
Bitdefender	3385	0	0.00%	95	353558.4	99.97%	99.97
Check Point	3378	7	0.21%	2653.8	350970.6	99.25%	98.19
Cleanmail Domain Gateway	3380	5	0.15%	189	353464.4	99.95%	99.04
FortiMail	3385	0	0.00%	352.8	353300.6	99.90%	99.90
Libraesva	3384	1	0.03%	440	353213.4	99.88%	99.70
N-able Mail Assure	3384	1	0.03%	350	353303.4	99.90%	99.72
NoSpamProxy	3385	0	0.00%	756	352897.4	99.79%	99.76
Rspamd	3360	25	0.74%	5540.6	348112.8	98.43%	94.68
Spamhaus Data Query Service (DQS) + SpamAssassin‡	3385	0	0.00%	1541.8	352111.6	99.56%	99.45
Zoho Mail	3377	8	0.24%	2109.8	351543.6	99.40%	97.88
Abusix Mail Intelligence*	3385	0	0.00%	1111.2	352542.2	99.69%	99.66

^*This product is a partial solution and its performance should not be compared with that of other products.
^‡Spamhaus Data Query Service (DQS) + SpamAssassin is a fully configured solution that integrates Spamhaus DQS on top of SpamAssassin. Spamhaus DQS is not a stand-alone solution but rather a DNSBL service that can be added to MTAs and email security solutions such as SpamAssasssin. The test set up reflects the real-life performance expected from this combined production deployment, not as individual product elements.
(Please refer to the text for full product names and details.)

^*This product is a partial solution and its performance should not be compared with that of other products. None of the queries to the IP blocklist included any information on the attachments; hence its performance on the malware corpus is added purely for information.
^‡Spamhaus Data Query Service (DQS) + SpamAssassin is a fully configured solution that integrates Spamhaus DQS on top of SpamAssassin. Spamhaus DQS is not a stand-alone solution but rather a DNSBL service that can be added to MTAs and email security solutions such as SpamAssasssin. The test set up reflects the real-life performance expected from this combined production deployment, not as individual product elements.
^†The standard deviation of a product is calculated using the set of its hourly spam catch rates.
(Please refer to the text for full product names and details.)

	Speed
	10%	50%	95%	98%
Axway
Bitdefender
Check Point
Cleanmail Domain Gateway
FortiMail
Libraesva
N-able Mail Assure
NoSpamProxy
Rspamd
Spamhaus Data Query Service (DQS) + SpamAssassin‡
Zoho Mail

^‡Spamhaus Data Query Service (DQS) + SpamAssassin is a fully configured solution that integrates Spamhaus DQS on top of SpamAssassin. Spamhaus DQS is not a stand-alone solution but rather a DNSBL service that can be added to MTAs and email security solutions such as SpamAssasssin. The test set up reflects the real-life performance expected from this combined production deployment, not as individual product elements.

^*This product is a partial solution and its performance should not be compared with that of other products. None of the queries to the IP blocklist included any information on the attachments; hence its performance on the malware corpus is added purely for information.
^‡Spamhaus Data Query Service (DQS) + SpamAssassin is a fully configured solution that integrates Spamhaus DQS on top of SpamAssassin. Spamhaus DQS is not a stand-alone solution but rather a DNSBL service that can be added to MTAs and email security solutions such as SpamAssasssin. The test set up reflects the real-life performance expected from this combined production deployment, not as individual product elements.
(Please refer to the text for full product names and details.)

(Please refer to the text for full product names and details.)

^‡Spamhaus Data Query Service (DQS) + SpamAssassin is a fully configured solution that integrates Spamhaus DQS on top of SpamAssassin. Spamhaus DQS is not a stand-alone solution but rather a DNSBL service that can be added to MTAs and email security solutions such as SpamAssasssin. The test set up reflects the real-life performance expected from this combined production deployment, not as individual product elements.

(Please refer to the text for full product names and details.)

(Please refer to the text for full product names and details.)

Appendix: set-up, methodology and email corpora

The full VBSpam test methodology can be found at https://www.virusbulletin.com/testing/vbspam/vbspam-methodology/vbspam-methodology-ver20.

The test ran for 16 days, from 12am on 6 November to 12am on 22 November 2021 (GMT).

The test corpus consisted of 357,210 emails. 353,719 of these were spam, 26,643 of which were provided by Project Honey Pot, 325,578 were provided by Abusix with the remaining 1,498 spam emails provided by MXMailData. There were 3,385 legitimate emails (‘ham’) and 106 newsletters, a category that includes various kinds of commercial and non-commercial opt-in mailings.

82 emails in the spam corpus were considered ‘unwanted’ (see the June 2018 report⁵) and were included with a weight of 0.2; this explains the non-integer numbers in some of the tables.

Moreover, 1,942 emails from the spam corpus were found to contain a malicious attachment while 2,063 contained a link to a phishing or malware site; though we report separate performance metrics on these corpora, it should be noted that these emails were also counted as part of the spam corpus.

Emails were sent to the products in real time and in parallel. Though products received the email from a fixed IP address, all products had been set up to read the original sender’s IP address as well as the EHLO/HELO domain sent during the SMTP transaction, either from the email headers or through an optional XCLIENT SMTP command⁶.

For those products running in our lab, we all ran them as virtual machines on a VMware ESXi cluster. As different products have different hardware requirements – not to mention those running on their own hardware, or those running in the cloud – there is little point comparing the memory, processing power or hardware the products were provided with; we followed the developers’ requirements and note that the amount of email we receive is representative of that received by a small organization.

Although we stress that different customers have different needs and priorities, and thus different preferences when it comes to the ideal ratio of false positive to false negatives, we created a one-dimensional ‘final score’ to compare products. This is defined as the spam catch (SC) rate minus five times the weighted false positive (WFP) rate. The WFP rate is defined as the false positive rate of the ham and newsletter corpora taken together, with emails from the latter corpus having a weight of 0.2:

WFP rate = (#false positives + 0.2 * min(#newsletter false positives , 0.2 * #newsletters)) / (#ham + 0.2 * #newsletters)

while in the spam catch rate (SC), emails considered ‘unwanted’ (see above) are included with a weight of 0.2.

The final score is then defined as:

Final score = SC - (5 x WFP)

In addition, for each product, we measure how long it takes to deliver emails from the ham corpus (excluding false positives) and, after ordering these emails by this time, we colour-code the emails at the 10th, 50th, 95th and 98th percentiles:

	(green) = up to 30 seconds
	(yellow) = 30 seconds to two minutes
	(orange) = two to ten minutes
	(red) = more than ten minutes

Products earn VBSpam certification if the value of the final score is at least 98 and the ‘delivery speed colours’ at 10 and 50 per cent are green or yellow and that at 95 per cent is green, yellow or orange.

Meanwhile, products that combine a spam catch rate of 99.5% or higher with a lack of false positives, no more than 2.5% false positives among the newsletters and ‘delivery speed colours’ of green at 10 and 50 per cent and green or yellow at 95 and 98 per cent earn a VBSpam+ award.

Footnotes

¹ Spamhaus Data Query Service (DQS) + SpamAssassin is a custom solution built on top of the SpamAssassin open-source anti-spam platform.

² https://bazaar.abuse.ch/sample/79cf54cb43eb9c510c7ce3271962f8d5befd28628688da4f625b01d9b52d5030/.

³ https://bazaar.abuse.ch/sample/2ad9563f018dcfe967225314cce0b6d9d3d6e1b7d8da1828ef9350c6f270d28c/.

⁴ https://www.proofpoint.com/us/blog/threat-insight/high-volume-german-phishing-campaign-aims-steal-banking-credentials.

⁵ https://www.virusbulletin.com/virusbulletin/2018/06/vbspam-comparative-review.

⁶ http://www.postfix.org/XCLIENT_README.html.

Latest reviews: