VBSpam for end-users

Learn more about how VBSpam works

Introduction

VBSpam is a quarterly comparative test series for business-focused email security products. It aims to quantify and compare the protection provided by these products against various common threats, including spam, phishing, and malware emails. In addition to threat detection, strong false positive controls are implemented in the test through the use of legitimate emails.

 

How we test

Our testing process involves using real-world data to evaluate the performance of email security solutions. Each product is subjected to live email streams, consisting of both spam and legitimate emails, to simulate actual conditions. Emails are delivered to the tested products in parallel. Emails filtered by the product are returned to our infrastructure, where we record the product’s response, along with the timeliness of the delivery for speed metrics.

To quantify performance, we categorize email filtering outcomes into true positives (spam correctly identified), true negatives (legitimate emails correctly identified), false positives (legitimate emails incorrectly marked as spam), and false negatives (spam missed by the filter). Products receive a single score, based on a weighted average of the above metrics, with a larger weight given to false positives. Products earning VBSpam and VBSpam+ certifications demonstrate high detection rates, low false positive rates and a speedy delivery, ensuring reliable email protection.

For more detailed information, you can refer to the full methodology.

 

Full email security solutions and complementary solutions

VBSpam hosts two distinct categories of products:

  • Full email security solutions: these include email security gateways or spam filters designed to provide comprehensive coverage of the email threat landscape.

  • Complementary email security solutions: these are designed with limited coverage, such as databases of known “bad IP addresses” or known phishing URLs.

While full email security solutions are comparable with each other, complementary solutions are not directly comparable with full solutions and often cannot be compared with each other either. Complementary products are typically integrated into a full email security solution, providing additional layers of security. When evaluating the performance of a complementary solution, we recommend using the interactive online tool provided with each VBSpam report to create layered scenarios and see how the combined performance of the layers looks. It is worth noting that a complementary solution that detects only 5% of all spam can make a significant difference in a layered setup if it does not overlap with the other layers.

 

FAQs

The following FAQs are designed to help you interpret VBSpam test reports and to give you insight into how the VBSpam programme is set up and runs.

 

How comprehensive is VBSpam?

VBSpam offers extensive coverage of common unwanted emails, such as spam, phishing, and malware emails, using real-world instances of these, relayed to the products without delay. The test does not include synthetic, targeted cases like spear phishing or CEO fraud emails, which pose significant risks to organizations. Additionally, features like Data Loss Prevention (DLP), encryption and archiving are not within the scope of the test.

 

What does the ‘final score’ mean in VBSpam?

The ‘final score’ metric is a weighted average ranging between 0 and 100 (100 being the perfect score), calculated on the basis of the product’s ability to detect ‘bad emails’ (true positives) whilst avoiding accidentally misclassifying the ‘good emails’ as bad (false positives). Weighting is employed primarily to give a greater recognition to the impact of false positives.

 

What does the VBSpam award signify?

The VBSpam award signifies that the product demonstrates a good overall detection rate with a very small number of false positives, as well as a good and consistent throughput. 

The VBSpam+ award signifies excellence on the same metrics.

 

What do the various figures in the report mean?

  • SC rate: weighted rate of spam detected among all spam. Best outcome: 100%.

  • FP rate: weighted rate of misclassified legitimate emails. Best outcome: 0%.

  • Final score: combined metric of SC rate and FP rate. Best outcome: 100.

  • Malware catch rate: rate of detection among all malware emails.

  • Phishing catch rate: rate of detection among all phishing emails.

  • Project Honeypot SC rate, Abusix SC rate, MXMailData SC rate: spam detection rates on specific third-party email feeds.

  • Newsletter FP rate: misclassification rate among newsletter emails used in the test.

 

How do I read the speed ratings for the products?

We measure the time each product takes to process a legitimate (or ‘ham’) email, categorizing delays as follows:

  • Green: up to 30 seconds

  • Yellow: 30 seconds to 2 minute

  • Orange: 2 to 10 minutes

  • Red: more than 10 minutes

 The delays are listed and sorted; the report and feedback only include the values measured at the 10%, 50%, 95% and 98% percentiles.

Here is an example of the distribution of delay values: 

 In this case, the speed values are:

10% 0 seconds
green 
50% 60 seconds YELLOWyellow
95% 110 seconds speed-colour-blobs-ORANGE.jpgorange
98% 120 seconds speed-colour-blobs-RED.jpgred

 

What’s the difference between full solutions and complementary solutions?

VBSpam hosts two categories of products:

  • Full email security solutions: These include email security gateways or spam filters designed to provide comprehensive coverage of the email threat landscape. Products in this category are typically stand-alone security solutions.

  • Complementary email security solutions: These are designed with limited coverage, such as databases of known ‘bad IP addresses’ or known phishing URLs. Often these are used as add-ons to full email security solutions.

 

What do I need to know about the comparability of tested products?

  • Full email security solutions can be directly compared with each other.

  • Complementary solutions are not directly comparable with full solutions and often cannot be compared to each other either. These products are typically integrated as additional layers within a full email security solution, and their effectiveness depends on this specific combination. In a layered setup, even a solution that has relatively low overall performance may contribute unique detections of unwanted emails To evaluate a complementary solution's performance, use the interactive online tool provided with each VBSpam report (from 2024) to create layered scenarios and assess the combined performance.

 

Why don’t any of the complementary products earn a VBSpam award?

Complementary solutions are seldom used in isolation. Rather, they are typically used as an additional layer of protection in the real world. A single layer may detect only a small percentage of all ‘bad emails’, but that small percentage might amount to a great contribution in a layered setup. Due to this, judging a complementary product solely on the raw performance metrics can be misleading and therefore the VBSpam award is not used with complementary products. When evaluating the performance of these solutions, we recommend using the interactive online tool provided with each VBSpam report issued from 2024 to create layered scenarios and see how the combined performance of the layers looks. 


What does AMTSO Standard compliance mean?

The Anti-Malware Testing Standards Organization (AMTSO) is an international non-profit association that focuses on the objectivity, quality and relevance of infosecurity product testing methodologies. AMTSO members include both vendors of infosecurity products and test labs.

Among other services, AMTSO maintains the AMTSO Testing Protocol Standard, which describes testing protocol and behaviour expectations for testers and vendors relating to the testing of security solutions.

The VBSpam test is designed to be compliant with this standard. AMTSO regularly audits the execution of our VBSpam tests and certifies them to be compliant if they meet the standard criteria.

 

Why isn’t the test report I read certified as AMTSO compliant?

AMTSO audits our tests regularly. This typically happens well after the test has been performed. As a result, the reports are released before they have been certified as AMTSO compliant. For this reason, you may find that the compliance information is missing for the newest tests listed on the AMTSO website.

If the report has no AMTSO reference at all, it was executed prior to the introduction of the AMTSO compliance.

 

Is the test funded by the vendors?

Yes, the test is funded by the vendors in the majority of cases.

 

Is there a conflict of interest?

Any potential conflict of interest is resolved by adhering strictly to our test ethics framework. Our primary responsibility is to the consumers of the report, ensuring unbiased and transparent evaluations.

 

Can a vendor hide unfavourable test reports?

No, they cannot. Any test that starts as a public test will result in a published report, regardless of whether the results are favourable for the vendor.

However, we may choose not to publish a report if the data collected is deemed irrelevant or compromised due to technical issues, operator errors, or unfair conditions for the vendor.

 

Can I ask a question?

Absolutely, email us at [email protected].

VBSpam

Latest report

The latest VBSpam comparative test report

VBSpam for end-users

Learn more about how VBSpam works

VBSpam for vendors

Not VBSpam certified and want to be? See how your product can be enrolled to the test bench.

VBSpam methodology

How the VBSpam comparative tests are carried out

VBSpam test schedule

The schedule for upcoming VBSpam test reports

VBSpam test archive

Details of all previous VBSpam comparatives

VB testing

VB100

VBSpam

Consultancy services

We have placed cookies on your device in order to improve the functionality of this site, as outlined in our cookies policy. However, you may delete and block all cookies from this site and your use of the site will be unaffected. By continuing to browse this site, you are agreeing to Virus Bulletin's use of data as outlined in our privacy policy.