VBSpam for vendors

Not VBSpam certified and want to be? See how your product can be enrolled to the test bench.

This page is designed to give you insight into how the VBSpam programme is set up and runs, and how your product can be enrolled to the test bench.

 

Overview

VBSpam is a continuously running performance test programme for email security solutions, aimed primarily at server/business-focused products. The programme offers quantification and comparative analysis of the protection performance against various common threats, including spam, phishing and malware emails, along with false positive controls.


Test scope

Our testing process involves using real-world data to evaluate the performance of email security solutions. Each product is subjected to live email streams, consisting of both spam and legitimate emails. Emails are delivered to the tested products in parallel. Emails filtered by the product are returned to our infrastructure, where we record the product’s response, along with the timeliness of the delivery for speed metrics.

Testing add-on services for business productivity suites like Microsoft 365 or Google Workspace is also possible, though it requires additional setup.

To quantify performance, we categorize email filtering outcomes into true positives (spam correctly identified), true negatives (legitimate emails correctly identified), false positives (legitimate emails incorrectly marked as spam), and false negatives (spam missed by the filter). Products receive a single score, based on a weighted average of the above metrics, with a larger weight given to false positives.

For more detailed information, you can refer to the full methodology here.

 

Test types

The following types of testing is available:

  • Public testing: products in the public VBSpam series undergo continuous evaluation, with weekly feedback provided privately to participants. Public testing is based on performance data gathered during four selected periods each year. This type of testing offers continuous quality assurance for your product, along with periodic public competitive tests.

  • Private testing: private tests are typically conducted on an ongoing basis, serving as an extension of your own quality assurance in a non-comparative, non-public manner. One-off private testing is also available – ideal for testing pre-release builds or alternative configurations, often in parallel with the public product. Private test results are shared only with the vendor and cannot be made public by either party.

  • Bespoke testing: we conduct stand-alone evaluations or commissioned comparative tests, either privately or publicly, often using a customized methodology. Bespoke testing is useful in various scenarios, such as gaining a private understanding of how your product's performance compares to competitors or showcasing your unique technology in a fair and balanced manner, while highlighting your perspective. Please get in touch with us at [email protected] to discuss your projects.


Supported product types

VBSpam is designed to accommodate a range of products, including cloud/hosted solutions and on-premises solutions (installable, virtual appliances or physical appliances), as long as the product can accept, filter and return/forward emails from our sources through SMTP.

In most cases, VBSpam can also accept APIs or other complementary solutions (such as DNSBLs) in the test.

 

FAQs

General

Testing process

Pricing and certification

AMTSO compliance

  

General

How do I get started in VBSpam?

These are the major steps you can expect:

  • Project discussion: work with us to define your objectives, find your preferred test arrangement and desired start date.
  • Agreement: sign a Test Agreement covering all relevant terms and conditions for participation.
  • Setup: collaborate with us to set up your product in the test environment.
  • Testing: testing begins for the agreed duration. Weekly feedback is provided privately outside of the designated public test period.
  • Public test period: at the end of a public test period, feedback is shared for the entire period. You have at least 10 calendar days to review and dispute the results if necessary. The VBSpam public report is then released.

 

Testing process

What kind of feedback do I receive?

The feedback we provide is specific to your product (i.e. non-comparative) and it includes:

  • Performance metrics (performance on the various test case bodies, speed measurements where applicable)
  • Test cases for false negatives and false positives, including the email source (MIME) and transaction logs (subject to capacity limits).

 

What kind of emails are used in the test?

Both unwanted and legitimate emails are used in the test.

  • Unwanted emails: sourced in real time from commercial email feeds and from Virus Bulletin’s own threat intelligence.
  • Legitimate emails: these include newsletters and email discussion lists.

 

Who is going to host my product?

For most tested products, the hosting location is flexible: the product can be hosted by the vendor (e.g. a cloud service) or by VB (virtual machine or physical appliance).

Complementary solutions (such as DNSBLs and APIs) are set up by VB in our own lab. While our standard test environment is Rspamd-based, we can accommodate different configurations as well.

 

How do I make sure my product is set up correctly for the test?

We provide full access to most tested products, allowing you to audit the product configuration and operations at any time.

Setups of complementary solutions (such as DNSBLs and APIs) are not remotely accessible, but an audit can be requested.

 

My product does <something> to the unwanted email, can your test framework work with that?

The answer is very likely ‘yes’ – we are able to detect and attribute a number of common actions performed on unwanted emails, such as:

  • Email rejection at protocol level (including both 5xx permanent and 4xx greylisting-like transient rejections)
  • Email non-delivery (silently swallowing emails)
  • Email subject tagging
  • Email header tagging

Similarly, we can detect ‘ham’ classifications either as lacking any markings of an unwanted email, or through specific headers or subject tags.

 

If my private test works out really well, can I make the result public? If my public test works out poorly, can I make it private?

Sorry, neither of these are possible. One of the fundamental rules of fair testing is that any test starting out as a private test cannot be made public, and a public test cannot be made private. This is to prevent 'cherry picking' of the favourable results.

 

Will you let me know if my product is not performing well during the test?

Yes, if we suspect that a technical issue is affecting the results. For instance, if we encounter an excessive number of false negatives / false positives, or if the results do not appear to make sense, or if the product crashes, etc. Generally, we can spot issues like these quite well, but should there be some problem that we don’t pick up, the review ('disputes') phase, during which you get to verify your results, serves as a final checkpoint.

 

Pricing and certification

How much does testing cost?

Both public and private tests are available in a highly competitive and adaptive pricing model that follows the value generated for your business, so whether you are a startup or an established player in the field we have a plan to suit you. To find out more please get in touch at [email protected].

 

How long does it take to get certified?

Public testing is performed quarterly, as published in the VBSpam schedule. Reports are typically released within 6-7 weeks after the commencement of a public test. If you have not participated in a VBSpam test recently and you are looking to (re)join with a particular timeline in mind, we recommend starting the conversation at least two weeks prior to the upcoming public test.

 

Does my award cover my other product editions?

Awards are issued for a specific product edition and they do not cover any derivative products (other product editions, OEM-licensed engines, etc.). This has to do with the framework of fair testing – fundamentally, a test lab may only make statements about its observations, thus extending the coverage to product editions the lab did not test would be a speculative matter.

 

AMTSO compliance

What are the benefits of the AMTSO certification?

The AMTSO certification ensures that you receive a testing service that is within the established parameters of what the industry considers to be fair testing. This benefits you directly as a vendor, and indirectly through the increased credibility of the reports issued by VB.

 

What are my rights and obligations under the AMTSO Standard?

Ultimately, these are described by the Standard and we recommend that you familiarize yourself with it. In practical terms, you only need to register on AMTSO’s contact list and complete a form before and after the test is concluded, to provide your feedback on how the testing was done.

 

Why aren’t VBSpam reports certified as AMTSO compliant immediately upon release?

AMTSO audits and certifies our tests periodically. Collecting vendor feedback after the test has been completed is part of that process, so your test can only become certified after the test report has been released by VB. Reports for tests that seek AMTSO compliance contain a link to a page detailing the test on the AMTSO website; it is this page that AMTSO updates upon completing its audit.

 

Getting in touch

Ready to get the conversation started? Please email us at [email protected].

 

VBSpam

Latest report

The latest VBSpam comparative test report

VBSpam for end-users

Learn more about how VBSpam works

VBSpam for vendors

Not VBSpam certified and want to be? See how your product can be enrolled to the test bench.

VBSpam methodology

How the VBSpam comparative tests are carried out

VBSpam test schedule

The schedule for upcoming VBSpam test reports

VBSpam test archive

Details of all previous VBSpam comparatives

VB testing

VB100

VBSpam

Consultancy services

We have placed cookies on your device in order to improve the functionality of this site, as outlined in our cookies policy. However, you may delete and block all cookies from this site and your use of the site will be unaffected. By continuing to browse this site, you are agreeing to Virus Bulletin's use of data as outlined in our privacy policy.