VBSpam is a quarterly comparative test series for business-focused email security products. It aims to quantify and compare the protection provided by these products against various common threats, including spam, phishing, and malware emails. In addition to threat detection, strong false positive controls are implemented in the test through the use of legitimate emails.
Our testing process involves using real-world data to evaluate the performance of email security solutions. Each product is subjected to live email streams, consisting of both spam and legitimate emails, to simulate actual conditions. Emails are delivered to the tested products in parallel. Emails filtered by the product are returned to our infrastructure, where we record the product’s response, along with the timeliness of the delivery for speed metrics.
To quantify performance, we categorize email filtering outcomes into true positives (spam correctly identified), true negatives (legitimate emails correctly identified), false positives (legitimate emails incorrectly marked as spam), and false negatives (spam missed by the filter). Products receive a single score, based on a weighted average of the above metrics, with a larger weight given to false positives. Products earning VBSpam and VBSpam+ certifications demonstrate high detection rates, low false positive rates and a speedy delivery, ensuring reliable email protection.
For more detailed information, you can refer to the full methodology.
VBSpam hosts two distinct categories of products:
Full email security solutions: these include email security gateways or spam filters designed to provide comprehensive coverage of the email threat landscape.
Complementary email security solutions: these are designed with limited coverage, such as databases of known “bad IP addresses” or known phishing URLs.
While full email security solutions are comparable with each other, complementary solutions are not directly comparable with full solutions and often cannot be compared with each other either. Complementary products are typically integrated into a full email security solution, providing additional layers of security. When evaluating the performance of a complementary solution, we recommend using the interactive online tool provided with each VBSpam report to create layered scenarios and see how the combined performance of the layers looks. It is worth noting that a complementary solution that detects only 5% of all spam can make a significant difference in a layered setup if it does not overlap with the other layers.
The following FAQs are designed to help you interpret VBSpam test reports and to give you insight into how the VBSpam programme is set up and runs.
VBSpam offers extensive coverage of common unwanted emails, such as spam, phishing, and malware emails, using real-world instances of these, relayed to the products without delay. The test does not include synthetic, targeted cases like spear phishing or CEO fraud emails, which pose significant risks to organizations. Additionally, features like Data Loss Prevention (DLP), encryption and archiving are not within the scope of the test.
The ‘final score’ metric is a weighted average ranging between 0 and 100 (100 being the perfect score), calculated on the basis of the product’s ability to detect ‘bad emails’ (true positives) whilst avoiding accidentally misclassifying the ‘good emails’ as bad (false positives). Weighting is employed primarily to give a greater recognition to the impact of false positives.
The VBSpam award signifies that the product demonstrates a good overall detection rate with a very small number of false positives, as well as a good and consistent throughput.
The VBSpam+ award signifies excellence on the same metrics.
SC rate: weighted rate of spam detected among all spam. Best outcome: 100%.
FP rate: weighted rate of misclassified legitimate emails. Best outcome: 0%.
Final score: combined metric of SC rate and FP rate. Best outcome: 100.
Malware catch rate: rate of detection among all malware emails.
Phishing catch rate: rate of detection among all phishing emails.
Project Honeypot SC rate, Abusix SC rate, MXMailData SC rate: spam detection rates on specific third-party email feeds.
Newsletter FP rate: misclassification rate among newsletter emails used in the test.
We measure the time each product takes to process a legitimate (or ‘ham’) email, categorizing delays as follows:
Green: up to 30 seconds
Yellow: 30 seconds to 2 minute
Orange: 2 to 10 minutes
Red: more than 10 minutes
The delays are listed and sorted; the report and feedback only include the values measured at the 10%, 50%, 95% and 98% percentiles.
Here is an example of the distribution of delay values:
In this case, the speed values are:
| 10% | 0 seconds |  green |
| 50% | 60 seconds |  yellow |
| 95% | 110 seconds |  orange |
| 98% | 120 seconds |  red |
VBSpam hosts two categories of products:
Full email security solutions: These include email security gateways or spam filters designed to provide comprehensive coverage of the email threat landscape. Products in this category are typically stand-alone security solutions.
Complementary email security solutions: These are designed with limited coverage, such as databases of known ‘bad IP addresses’ or known phishing URLs. Often these are used as add-ons to full email security solutions.
Full email security solutions can be directly compared with each other.
Complementary solutions are not directly comparable with full solutions and often cannot be compared to each other either. These products are typically integrated as additional layers within a full email security solution, and their effectiveness depends on this specific combination. In a layered setup, even a solution that has relatively low overall performance may contribute unique detections of unwanted emails To evaluate a complementary solution's performance, use the interactive online tool provided with each VBSpam report (from 2024) to create layered scenarios and assess the combined performance.
Complementary solutions are seldom used in isolation. Rather, they are typically used as an additional layer of protection in the real world. A single layer may detect only a small percentage of all ‘bad emails’, but that small percentage might amount to a great contribution in a layered setup. Due to this, judging a complementary product solely on the raw performance metrics can be misleading and therefore the VBSpam award is not used with complementary products. When evaluating the performance of these solutions, we recommend using the interactive online tool provided with each VBSpam report issued from 2024 to create layered scenarios and see how the combined performance of the layers looks.
The Anti-Malware Testing Standards Organization (AMTSO) is an international non-profit association that focuses on the objectivity, quality and relevance of infosecurity product testing methodologies. AMTSO members include both vendors of infosecurity products and test labs.
Among other services, AMTSO maintains the AMTSO Testing Protocol Standard, which describes testing protocol and behaviour expectations for testers and vendors relating to the testing of security solutions.
The VBSpam test is designed to be compliant with this standard. AMTSO regularly audits the execution of our VBSpam tests and certifies them to be compliant if they meet the standard criteria.
AMTSO audits our tests regularly. This typically happens well after the test has been performed. As a result, the reports are released before they have been certified as AMTSO compliant. For this reason, you may find that the compliance information is missing for the newest tests listed on the AMTSO website.
If the report has no AMTSO reference at all, it was executed prior to the introduction of the AMTSO compliance.
Yes, the test is funded by the vendors in the majority of cases.
Any potential conflict of interest is resolved by adhering strictly to our test ethics framework. Our primary responsibility is to the consumers of the report, ensuring unbiased and transparent evaluations.
No, they cannot. Any test that starts as a public test will result in a published report, regardless of whether the results are favourable for the vendor.
However, we may choose not to publish a report if the data collected is deemed irrelevant or compromised due to technical issues, operator errors, or unfair conditions for the vendor.
Absolutely, email us at [email protected].
