Virus Bulletin
Copyright © 2021 Virus Bulletin
Anti-malware products don’t have an easy job. There are millions of malware samples (many of which are written with the explicit goal of bypassing anti-malware products) to be detected and blocked, while at the same time there are millions of legitimate programs (which occasionally use some of the same techniques as malware) that mustn’t be blocked. And there are dozens of competing products on the market.
Users should expect anti-malware products to satisfy a minimum standard of blocking malicious executables that have recently been seen in the wild, while blocking few to no legitimate programs.
For more than two decades, Virus Bulletin has set a minimum standard for anti-malware products, checking whether products live up to expectation and providing those that do with the VB100 ‘stamp of approval’.
This report details the VB100 certification results of 43 such products from 38 different vendors during January and February 2021.
In the VB100 test, a copy of the product to be tested is installed on Windows 10. At three different times in the test, the product is asked to scan both the latest version of the WildList and a selection of clean files taken from Virus Bulletin’s own set of files belonging to widely used legitimate software.
A legitimate file that is blocked at least once is considered a false positive, while a WildList file that isn’t blocked is considered a miss.
A product achieves a VB100 certification if:
and
For full details, we refer to the VB100 methodology on the Virus Bulletin website: https://www.virusbulletin.com/testing/vb100/vb100-methodology/vb100-methodology-ver1-2/. This test used version 1.2 of the VB100 methodology.
The malware part of the VB100 certification uses the WildList1, a regularly updated list of extremely well-vetted malware samples, guaranteed to have been spotted in the wild multiple times. This makes them very suitable for a certification test like VB100.
The ‘Diversity Test’ looks at products’ detection of another set of recent malware samples, to acknowledge the fact that products detect malware samples beyond a standard set of samples, and provides a measure of that detection.
Products were allowed to download updates during the course of the test. The version numbers listed in the results that follows refer to those at the start of the test.
| Windows 10 version | 12.5.23140 |  |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 100.00% |
| Windows 10 version | 2021 build 35860 | |
| WildList detection | 99.9% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 100.00% |
| Windows 10 version | 12.10.60.0 | |
| WildList detection | 99.9% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.70% |
| Windows 10 version | 12.10.60.0 | |
| WildList detection | 99.9% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.70% |
| Windows 10 version | 2021.01.08 | |
| WildList detection | 99.9% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.20% |
| Windows 10 version | 9.0.64.11 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.90% |
| Windows 10 version | 2021.01.09 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.80% |
| Windows 10 version | 20.10.2442 | |
| WildList detection | 99.9% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 100.00% |
| Windows 10 version | 20.10.3157 | |
| WildList detection | 99.9% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 100.00% |
| Windows 10 version | 1.8.2020 build 81 | |
| WildList detection | 99.7% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 59.10% |
| Windows 10 version | 1.0.78.0 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 100.00% |
| Windows 10 version | 1.0.762.221 | |
| WildList detection | 99.9% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 71.20% |
| Windows 10 version | 8.3.62.140 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.20% |
| Windows 10 version | 1.0.0.56 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 97.90% |
| Windows 10 version | 1.7.6.0 | |
| WildList detection | 99.9% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.20% |
| Windows 10 version | 2021.1.0.10621 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.90% |
| Windows 10 version | 14.0.1400.2228 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 100.00% |
| Windows 10 version | 4.0.2.23116 | |
| WildList detection | 99.9% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.70% |
| Windows 10 version | 1.5.22.3 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 100.00% |
| Windows 10 version | 4.21.3102.484 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 100.00% |
| Windows 10 version | 32.30.13 | |
| WildList detection | 99.9% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.60% |
| Windows 10 version | 6.2.7.0984 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.80% |
| Windows 10 version | 25.5.8.14 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.90% |
| Windows 10 version | 3.2.4 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 100.00% |
| Windows 10 version | 1.0.1.7 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.80% |
| Windows 10 version | 16.0.0665 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.80% |
| Windows 10 version | 5.5.83 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.20% |
| Windows 10 version | 1.0.1.6 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 100.00% |
| Windows 10 version | 6.6.0.4088 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.80% |
| Windows 10 version | 3.0.0.99 | |
| WildList detection | 100.0% | |
| False positive rate | 0.002% | |
| Diversity Test rate | 83.60% |
| Windows 10 version | 5.5.83 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.20% |
| Windows 10 version | 4.7.2 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.20% |
| Windows 10 version | 1.0.1000.11042 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 100.00% |
| Windows 10 version | 5.0.1.33 | |
| WildList detection | 99.9% | |
| False positive rate | 0.010% | |
| Diversity Test rate | 99.60% |
| Windows 10 version | 20.9.1 | |
| WildList detection | 99.9% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.60% |
(see notes in Appendix 2)
| Windows 10 version | 12.3.26609.901 | |
| WildList detection | 99.8% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 98.00% |
| Windows 10 version | 5.5.83 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 100.00% |
| Windows 10 version | 12.0.0.306 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 99.90% |
| Windows 10 version | 1.0.1.96 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 100.00% |
| Windows 10 version | 1.3 | |
| WildList detection | 100.0% | |
| False positive rate | 0.006% | |
| Diversity Test rate | 100.00% |
| Windows 10 version | 12.0.7874 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 100.00% |
| Windows 10 version | 9.3.44 | |
| WildList detection | 100.0% | |
| False positive rate | 0.000% | |
| Diversity Test rate | 39.10% |
SecureAge SecureAPlus Pro didn’t achieve VB100 certification in this test.
The Certification Set contained 1,384 malicious samples. The set of clean samples used for the false positive test contained 100,000 files, of which 27,102 were portable executable (PE) files. The set used for the Diversity Test contained 1,000 malicious samples.
1 The WildList is an extremely well-vetted set of malware recently observed in the wild by researchers: http://www.wildlist.org/.
