Virus Bulletin
Copyright © 2018 Virus Bulletin
The endpoint security market is changing rapidly and marketing departments like to make a lot of noise about next-gen (or next-next-gen) technology, often using some kind of machine learning. These are important developments, but at the core of every product is an engine that, when given a file, determines whether it is malicious or not.
The VB100 certification reports check whether this engine, and thus the product, satisfies a minimum standard: when looking to purchase a new endpoint security product, we recommend looking for a product that consistently achieves VB100 certification.
This test report details the performance of 30 anti-virus products from 29 different vendors during July and August 2018, all of which achieved a VB100 award.
In the VB100 test, a copy of the product to be tested is installed on two platforms: Windows 10 and Windows 7. On each platform, and at three different times in the test, the product is asked to scan both the latest version of the WildList1 and a selection of clean files taken from Virus Bulletin's own set of files belonging to widely used legitimate software.
A legitimate file that is blocked at least once is considered a false positive, while a WildList file that isn't blocked is considered a miss.
A product achieves a VB100 certification if:
• No more than 0.5% of WildList samples are missed
and
• No more than 0.01% of legitimate files are blocked
For full details, we refer to the VB100 methodology on the Virus Bulletin website: https://www.virusbulletin.com/testing/vb100/vb100-methodology/vb100-methodology-ver1-1/. This test used version 1.1 of the VB100 methodology.
The malware part of the VB100 certification uses the WildList, a regularly updated list of extremely well vetted malware samples, guaranteed to have been spotted in the wild multiple times. This makes them very suitable for a certification test like VB100.
The 'Diversity Test', which debuts in this report, shows detection of another set of recent malware samples, to acknowledge the fact that products detect malware samples beyond a standard set of samples and provide a measure of that detection.
Products were allowed to download updates during the course of the test. The version numbers listed in the results that follows refer to those at the start of the test.
| Windows 7 version | 9.0.48.1 |  |
| Windows 10 version | 9.0.48.1 | |
| WildList detection | 99.9% | |
| False positive rate | 0.000% | |
| Diversity test rate | 100.00% |
| Windows 7 version | 2018.07.10 | |
| Windows 10 version | 2018.07.11 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity test rate | 100.00% |
| Windows 7 version | 18.5.2342 | |
| Windows 10 version | 18.5.2342 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity test rate | 100.00% |
| Windows 7 version | 18.5.3059 | |
| Windows 10 version | 18.5.3059 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity test rate | 100.00% |
| Windows 7 version | 6.6.2.49 | |
| Windows 10 version | 6.6.2.49 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity test rate | 100.00% |
| Windows 7 version | 5.1.38 | |
| Windows 10 version | 5.1.38 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity test rate | 100.00% |
| Windows 7 version | 15.1.0111 | |
| Windows 10 version | 15.1.0111 | |
| WildList detection | 100.0% | |
| False positive rate | 0.001% | |
| Diversity test rate | 100.00% |
| Windows 7 version | 2018.6.0.8750 | |
| Windows 10 version | 2018.6.0.8750 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity test rate | 100.00% |
| Windows 7 version | 14.0.1400.2029 | |
| Windows 10 version | 14.0.1400.2029 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity test rate | 100.00% |
| Windows 7 version | 11.0.159.9 | |
| Windows 10 version | 11.0.159.9 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity test rate | 100.00% |
| Windows 7 version | 3.0.1.3 | |
| Windows 10 version | 3.0.1.3 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity test rate | 100.00% |
| Windows 7 version | 4.12.3102.401 | |
| Windows 10 version | 4.12.3102.401 | |
| WildList detection | 100.0% | |
| False positive rate | 0.005% | |
| Diversity test rate | 100.00% |
| Windows 7 version | 26.21.8 | |
| Windows 10 version | 26.21.8 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity test rate | 100.00% |
| Windows 7 version | 5.6.2.1117 | |
| Windows 10 version | 5.6.2.1117 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity test rate | 100.00% |
| Windows 7 version | 25.4.0.3 | |
| Windows 10 version | 25.4.0.3 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity test rate | 100.00% |
| Windows 7 version | 10.2.0 | |
| Windows 10 version | 10.2.0 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity test rate | 100.00% |
| Windows 7 version | 2.16.20 | |
| Windows 10 version | 2.16.20 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity test rate | 100.00% |
| Windows 7 version | 15.1.0339 | |
| Windows 10 version | 15.1.0339 | |
| WildList detection | 100.0% | |
| False positive rate | 0.001% | |
| Diversity test rate | 100.00% |
| Windows 7 version | 11.0.0.6499 | |
| Windows 10 version | 11.0.0.6499 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity test rate | 100.00% |
| Windows 7 version | 1.0.116.88902 | |
| Windows 10 version | 1.0.116.88902 | |
| WildList detection | 100.0% | |
| False positive rate | 0.001% | |
| Diversity test rate | 100.00% |
| Windows 7 version | 3.0.0.78 | |
| Windows 10 version | 3.0.0.78 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity test rate | 100.00% |
| Windows 7 version | 2.0.1430 | |
| Windows 10 version | 2.0.1430 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity test rate | 100.00% |
| Windows 7 version | 5.0.0.57 | |
| Windows 10 version | 5.0.0.57 | |
| WildList detection | 100.0% | |
| False positive rate | 0.007% | |
| Diversity test rate | 99.99% |
| Windows 7 version | 1.0.10118 | |
| Windows 10 version | 1.0.10118 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity test rate | 100.00% |
| Windows 7 version | 12.3.26562.901 | |
| Windows 10 version | 12.3.26562.901 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity test rate | 100.00% |
| Windows 7 version | 4.6.19 | |
| Windows 10 version | 4.6.19 | |
| WildList detection | 99.8% | |
| False positive rate | 0.000% | |
| Diversity test rate | 44.63% |
| Windows 7 version | 9.0.0.747 | |
| Windows 10 version | 9.0.0.747 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity test rate | 100.00% |
| Windows 7 version | 10.3.4.2 | |
| Windows 10 version | 10.3.4.2 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity test rate | 100.00% |
| Windows 7 version | 8.7 | |
| Windows 10 version | 8.7 | |
| WildList detection | 100.0% | |
| False positive rate | 0.001% | |
| Diversity test rate | 100.00% |
| Windows 7 version | 6.6.1.36 | |
| Windows 10 version | 6.6.1.36 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity test rate | 100.00% |
All tested products were certified.
Technical and other issues can render the data we collect insufficient or otherwise unsuitable for accurate reporting. In such cases, the methodology allows us to discard the affected test part and record the event below. Two healthy test parts out of three are required for the certification to be issued.
The WildList contained 990 samples. The set of clean files used for the false positive test contained 100,000 files, of which 29,199 were portable executable (PE) files. The set used for the Diversity Test contained 997 samples.
1 The WildList is an extremely well-vetted set of malware recently observed in the wild by researchers: http://www.wildlist.org/.
