This page is designed to give you insight into how the VBSpam programme is set up and runs, and how your product can be enrolled to the test bench.
VBSpam is a continuously running performance test programme for email security solutions, aimed primarily at server/business-focused products. The programme offers quantification and comparative analysis of the protection performance against various common threats, including spam, phishing and malware emails, along with false positive controls.
Our testing process involves using real-world data to evaluate the performance of email security solutions. Each product is subjected to live email streams, consisting of both spam and legitimate emails. Emails are delivered to the tested products in parallel. Emails filtered by the product are returned to our infrastructure, where we record the product’s response, along with the timeliness of the delivery for speed metrics.
Testing add-on services for business productivity suites like Microsoft 365 or Google Workspace is also possible, though it requires additional setup.
To quantify performance, we categorize email filtering outcomes into true positives (spam correctly identified), true negatives (legitimate emails correctly identified), false positives (legitimate emails incorrectly marked as spam), and false negatives (spam missed by the filter). Products receive a single score, based on a weighted average of the above metrics, with a larger weight given to false positives.
For more detailed information, you can refer to the full methodology here.
The following types of testing is available:
VBSpam is designed to accommodate a range of products, including cloud/hosted solutions and on-premises solutions (installable, virtual appliances or physical appliances), as long as the product can accept, filter and return/forward emails from our sources through SMTP.
In most cases, VBSpam can also accept APIs or other complementary solutions (such as DNSBLs) in the test.
General
Testing process
Pricing and certification
AMTSO compliance
These are the major steps you can expect:
The feedback we provide is specific to your product (i.e. non-comparative) and it includes:
Both unwanted and legitimate emails are used in the test.
For most tested products, the hosting location is flexible: the product can be hosted by the vendor (e.g. a cloud service) or by VB (virtual machine or physical appliance).
Complementary solutions (such as DNSBLs and APIs) are set up by VB in our own lab. While our standard test environment is Rspamd-based, we can accommodate different configurations as well.
We provide full access to most tested products, allowing you to audit the product configuration and operations at any time.
Setups of complementary solutions (such as DNSBLs and APIs) are not remotely accessible, but an audit can be requested.
The answer is very likely ‘yes’ – we are able to detect and attribute a number of common actions performed on unwanted emails, such as:
Similarly, we can detect ‘ham’ classifications either as lacking any markings of an unwanted email, or through specific headers or subject tags.
Sorry, neither of these are possible. One of the fundamental rules of fair testing is that any test starting out as a private test cannot be made public, and a public test cannot be made private. This is to prevent 'cherry picking' of the favourable results.
Yes, if we suspect that a technical issue is affecting the results. For instance, if we encounter an excessive number of false negatives / false positives, or if the results do not appear to make sense, or if the product crashes, etc. Generally, we can spot issues like these quite well, but should there be some problem that we don’t pick up, the review ('disputes') phase, during which you get to verify your results, serves as a final checkpoint.
Both public and private tests are available in a highly competitive and adaptive pricing model that follows the value generated for your business, so whether you are a startup or an established player in the field we have a plan to suit you. To find out more please get in touch at [email protected].
Public testing is performed quarterly, as published in the VBSpam schedule. Reports are typically released within 6-7 weeks after the commencement of a public test. If you have not participated in a VBSpam test recently and you are looking to (re)join with a particular timeline in mind, we recommend starting the conversation at least two weeks prior to the upcoming public test.
Awards are issued for a specific product edition and they do not cover any derivative products (other product editions, OEM-licensed engines, etc.). This has to do with the framework of fair testing – fundamentally, a test lab may only make statements about its observations, thus extending the coverage to product editions the lab did not test would be a speculative matter.
The AMTSO certification ensures that you receive a testing service that is within the established parameters of what the industry considers to be fair testing. This benefits you directly as a vendor, and indirectly through the increased credibility of the reports issued by VB.
Ultimately, these are described by the Standard and we recommend that you familiarize yourself with it. In practical terms, you only need to register on AMTSO’s contact list and complete a form before and after the test is concluded, to provide your feedback on how the testing was done.
AMTSO audits and certifies our tests periodically. Collecting vendor feedback after the test has been completed is part of that process, so your test can only become certified after the test report has been released by VB. Reports for tests that seek AMTSO compliance contain a link to a page detailing the test on the AMTSO website; it is this page that AMTSO updates upon completing its audit.
Ready to get the conversation started? Please email us at [email protected].