Virus Bulletin
Copyright © 2019 Virus Bulletin
Malware has been causing problems for computer users for more than three decades, and anti-virus (or anti-malware) products have played an important part in mitigating those problems, initially taking on the leading role in protection and now taking more of a supporting role in amongst a cast of other forms of defence.
For more than two decades, Virus Bulletin has been testing anti-malware products, reporting on their performance as both the threat landscape and the products themselves have evolved, and has provided the VB100 certification scheme as a mark of distinction for those products that live up to expectation. The VB100 logo is the 'stamp of approval' conferred on products that satisfy a minimum standard of detecting malicious executables that have recently been seen in the wild, while blocking few to no legitimate programs.
This report details the performance of 34 anti-malware products from 31 different vendors tested during March and April 2019.
In the VB100 test, a copy of the product to be tested is installed on two platforms: Windows 10 and Windows 7. On each platform, and at three different times in the test, the product is asked to scan both the latest version of the WildList and a selection of clean files taken from Virus Bulletin's own set of files belonging to widely used legitimate software.
A legitimate file that is blocked at least once is considered a false positive, while a WildList file that isn't blocked is considered a miss.
A product achieves a VB100 certification if:
and
For full details, we refer to the VB100 methodology on the Virus Bulletin website: https://www.virusbulletin.com/testing/vb100/vb100-methodology/vb100-methodology-ver1-1/. This test used version 1.1 of the VB100 methodology.
The malware part of the VB100 certification uses the WildList, a regularly updated list of extremely well-vetted malware samples, guaranteed to have been spotted in the wild multiple times. This makes them very suitable for a certification test like VB100.
The 'Diversity Test' looks at products' detection of another set of recent malware samples, to acknowledge the fact that products detect malware samples beyond a standard set of samples, and provides a measure of that detection.
Products were allowed to download updates during the course of the test. The version numbers listed in the results that follows refer to those at the start of the test.
| Windows 7 version | 12.6.1005.11662 |  |
| Windows 10 version | 12.6.1005.11662 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 98.94% |
| Windows 7 version | 12.6.1005.11662 | |
| Windows 10 version | 12.6.1005.11662 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 98.94% |
| Windows 7 version | 9.0.53.1 | |
| Windows 10 version | 9.0.53.1 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 100.00% |
| Windows 7 version | 2019.03.08 | |
| Windows 10 version | 2019.03.09 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 100.00% |
| Windows 7 version | 19.3.2369 | |
| Windows 10 version | 19.3.2369 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 100.00% |
| Windows 7 version | 19.3.3084 | |
| Windows 10 version | 19.3.3084 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 100.00% |
| Windows 7 version | 8.3.52.210 | |
| Windows 10 version | 8.3.52.214 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.06% |
| Windows 7 version | 15.1.0112 | |
| Windows 10 version | 15.1.0112 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.65% |
| Windows 7 version | 2019.2.0.9269 | |
| Windows 10 version | 2019.2.0.9269 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 100.00% |
| Windows 7 version | 14.0.1400.2029 | |
| Windows 10 version | 14.0.1400.2029 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.88% |
| Windows 7 version | 12.0.31.0 | |
| Windows 10 version | 12.0.31.0 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 100.00% |
| Windows 7 version | 3.0.1.3.31873 | |
| Windows 10 version | 3.0.1.3.31873 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.06% |
| Windows 7 version | 8.3.52.58 | |
| Windows 10 version | 8.3.52.216 | |
| WildList detection | 99.9% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 98.82% |
| Windows 7 version | 4.20.3102.471 | |
| Windows 10 version | 4.20.3102.471 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.88% |
| Windows 7 version | 29.0.0 | |
| Windows 10 version | 29.0.0 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.06% |
| Windows 7 version | 5.6.2.1117 | |
| Windows 10 version | 5.6.2.1117 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.88% |
| Windows 7 version | 25.5.0.2 | |
| Windows 10 version | 25.5.0.4 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.88% |
| Windows 7 version | 2.5.173.3000 | |
| Windows 10 version | 2.5.173.3000 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 98.94% |
| Windows 7 version | 2.19.14 | |
| Windows 10 version | 2.19.14 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.65% |
| Windows 7 version | 15.1.0355 | |
| Windows 10 version | 15.1.0355 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.65% |
| Windows 7 version | 11.0.0.6499 | |
| Windows 10 version | 11.0.0.6499 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 100.00% |
| Windows 7 version | 1.0.134.90112 | |
| Windows 10 version | 1.0.134.90112 | |
| WildList detection | 100.0% | |
| False positive rate | 0.003% | |
| Diversity Test rate | 99.29% |
| Windows 7 version | 4.13.37 | |
| Windows 10 version | 4.13.37 | |
| WildList detection | 99.9% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.29% |
| Windows 7 version | 3.0.0.85 | |
| Windows 10 version | 3.0.0.85 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 77.76% |
| Windows 7 version | 4.13.37 | |
| Windows 10 version | 4.13.37 | |
| WildList detection | 99.9% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.29% |
| Windows 7 version | 5.0.0.57 | |
| Windows 10 version | 5.0.0.57 | |
| WildList detection | 99.9% | |
| False positive rate | 0.004% | |
| Diversity Test rate | 98.24% |
| Windows 7 version | 1.0.158353 | |
| Windows 10 version | 1.0.158353 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.06% |
| Windows 7 version | 12.3.26600.901 | |
| Windows 10 version | 12.3.26601.901 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 100.00% |
| Windows 7 version | 4.13.37 | |
| Windows 10 version | 4.13.37 | |
| WildList detection | 99.9% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.29% |
| Windows 7 version | 9.0.0.747 | |
| Windows 10 version | 9.0.0.747 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.06% |
| Windows 7 version | 11.0.0.775 | |
| Windows 10 version | 11.0.0.775 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.06% |
| Windows 7 version | 11.0.3.20 | |
| Windows 10 version | 11.0.3.20 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 100.00% |
| Windows 7 version | 8.8.82 | |
| Windows 10 version | 8.8.83 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 61.41% |
| Windows 7 version | 2.0.1548 | |
| Windows 10 version | 2.0.1548 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.88% |
Cyren failed to achieve VB100 certification in this test because its number of false positives was just above the threshold for VB100 certification.
The WildList contained 2,341 samples. The set of clean files used for the false positive test contained 100,000 files, of which 29,187 were portable executable (PE) files. The set used for the Diversity Test contained 850 samples.
