VB100 Comparative Review – October 2017

Martijn Grooten

Virus Bulletin

Copyright © 2017 Virus Bulletin

Introduction

My favourite kind of security product marketing is one where the vendor acknowledges that the product isn’t a silver bullet, but that it plays a role in the wider security ecosystem, which together intends to mitigate the threat.

The same applies to test reports. It would be wrong to consider this report – or any test report for that matter – a guide to the absolute best product. Rather, it helps paint part of the picture, but we encourage readers also to check out reports by other testing organizations – for example those that focus on runtime detection or vulnerabilities in the products.

The focus of the VB100 reports has always been to measure products’ ability to keep up with threats, to avoid false positives, and to do so in a consistent manner; hence we expect products to block 100% of the malware in the WildList and to generate no false positives among our own, well-curated set of popular legitimate software. This is also why we test regularly – to be able to paint a more continuous picture of products’ performance.

This month, we tested 32 products from 27 vendors, with some new names appearing in addition to many of the regular ones, showing that the anti-virus market remains very much alive. Twenty eight of the products achieved the VB100 certification.

Bad trojans, good tools

Ransomware, banking trojans, espionage tools: there is no shortage of malicious threats facing computer users. Despite what the box may say, no security product blocks or detects them all, but the very least you should expect is for a product to block the malware found on the WildList, a well-vetted list of malware that has both been confirmed to be malicious and confirmed to have been seen in the wild.

As usual, this month’s WildList contained a good mix of such threats, which even included tools developed by the NSA and leaked by the infamous Shadow Brokers group. We were pleased to see that products generally had few problems blocking the files from this list.

At the same time as blocking such threats, you should also expect an anti-virus product not to prevent you from using a video editor, a printer driver, or any other type of legitimate software; this is why we require products to scan a collection of clean files, and penalize any that generate false detections within this set. We were pleased to find that, here too, there were few problems.

The full test set-up is described in the Appendix at the end of this report.

Results

In the results on the following pages, the RAP images display an average of the RAP scores across the two platforms.

ad-aware antivirus pro

Arcabit AntiVirus

Avast Free Antivirus

AVG Internet Security

CompuClever Antivirus PLUS

Cyren Command Anti-Malware

Defenx Security Suite

Emsisoft Anti-Malware

Endpoint Security by Bitdefender

eScan Internet Security Suite for Windows

ESET Internet Security

Essentware PCKeeper Antivirus PRO

ESTsecurity ALYac

Faronics Anti-Virus

Fortinet FortiClient

G DATA Antivirus

IKARUS anti.virus

K7 Total Security

Kaspersky Endpoint Security 10 for Windows

Panda Endpoint Protection Plus

Panda Free Antivirus

Quick Heal Seqrite Endpoint Security

Quick Heal Total Security

Rising Security Cloud Client

TeamViewer ITbrain Anti-Malware

Tencent PC Manager

Tencent PC Manager – TAV

Total Defense Internet Security

Total Defense Premium

VIPRE Advanced Security

VirIT eXplorer PRO

Wontok SafeCentral Security Suite

Results tables

Certification tests	Windows 7				Windows 10				VB100
	FPs	FP rate	WildList misses	WildList catch rate	FPs	FP rate	WildList misses	WildList catch rate
ad-aware antivirus pro	0	0.00%	0	100.00%	0	0.00%	0	100.00%
Arcabit AntiVirus	0	0.00%	0	100.00%	0	0.00%	0	100.00%
Avast Free Antivirus	0	0.00%	0	100.00%	0	0.00%	0	100.00%
AVG Internet Security	0	0.00%	0	100.00%	0	0.00%	0	100.00%
CompuClever Antivirus PLUS	0	0.00%	0	100.00%	0	0.00%	0	100.00%
Cyren Command Anti-Malware	0	0.00%	0	100.00%	1	0.0003%	1	99.96%
Defenx Security Suite	1	0.0003%	0	100.00%	1	0.0003%	2	99.93%
Emsisoft Anti-Malware	0	0.00%	0	100.00%	0	0.00%	0	100.00%
Endpoint Security by Bitdefender	0	0.00%	0	100.00%	0	0.00%	0	100.00%
eScan Internet Security Suite for Windows	0	0.00%	0	100.00%	0	0.00%	0	100.00%
ESET Internet Security	0	0.00%	0	100.00%	0	0.00%	0	100.00%
Essentware PCKeeper Antivirus PRO	0	0.00%	0	100.00%	0	0.00%	0	100.00%
ESTsecurity ALYac	0	0.00%	0	100.00%	0	0.00%	0	100.00%
Faronics Anti-Virus	0	0.00%	0	100.00%	0	0.00%	0	100.00%
Fortinet FortiClient	0	0.00%	0	100.00%	0	0.00%	0	100.00%
G DATA Antivirus	0	0.00%	0	100.00%	0	0.00%	0	100.00%
IKARUS anti.virus	0	0.00%	0	100.00%	2	0.0005%	0	100.00%
K7 Total Security	0	0.00%	0	100.00%	0	0.00%	0	100.00%
Kaspersky Endpoint Security 10 for Windows	0	0.00%	0	100.0%	0	0.00%	0	100.00%
Panda Endpoint Protection Plus	0	0.00%	0	100.00%	0	0.00%	0	100.00%
Panda Free Antivirus	0	0.00%	0	100.00%	0	0.00%	0	100.00%
Quick Heal Seqrite Endpoint Security	0	0.00%	0	100.00%	0	0.00%	0	100.00%
Quick Heal Total Security	0	0.00%	0	100.00%	0	0.00%	0	100.00%
Rising Security Cloud Client	41	0.01%	0	100.00%	41	0.01%	0	100.00%
TeamViewer ITbrain Anti-Malware	0	0.00%	0	100.00%	0	0.00%	0	100.00%
Tencent PC Manager	0	0.00%	0	100.00%	0	0.00%	0	100.00%
Tencent PC Manager – TAV	0	0.00%	0	100.00%	0	0.00%	0	100.00%
Total Defense Internet Security	0	0.00%	0	100.00%	0	0.00%	0	100.00%
Total Defense Premium	0	0.00%	0	100.00%	0	0.00%	0	100.00%
VIPRE Advanced Security	0	0.00%	0	100.00%	0	0.00%	0	100.00%
VirIT eXplorer PRO	0	0.00%	0	100.00%	0	0.00%	0	100.00%
Wontok SafeCentral Security Suite	0	0.00%	0	100.00%	0	0.00%	0	100.00%

^*Set -1 = Samples discovered 1 to 5 days before testing; Set -2 = Samples discovered 6 to 10 days before testing.
^†Set +1 = Samples discovered 1 to 5 days after updates frozen; Set +2 = Samples discovered 6 to 10 days after updates frozen.
^‡Weighted average gives equal emphasis to the two reactive weeks and the whole proactive part.

^*Set -1 = Samples discovered 1 to 5 days before testing; Set -2 = Samples discovered 6 to 10 days before testing.
^†Set +1 = Samples discovered 1 to 5 days after updates frozen; Set +2 = Samples discovered 6 to 10 days after updates frozen.
^‡Weighted average gives equal emphasis to the two reactive weeks and the whole proactive part. 

Appendix: the test set-up

The main test on each platform was run in three parts, over three consecutive weeks. Products were installed on clean installations of both Windows 7 and Windows 10. At the beginning of each part of the test we made sure the latest updates were downloaded, while throughout the test, products were connected to the Internet, thus allowing for real-time cloud look-ups.

The products as we tested them are available to the general public. However, in a few instances we have allowed vendors to make modifications to the product to adapt to our specific test scenario. None of these modifications would have an impact on the real-world performance of the affected products.

For each part of the test, we used the most recent version of the WildList, together with one third of our constantly updated collection of widely used legitimate software. Using a shared drive, the files were copied onto the client machine and we recorded whether (and how) files were blocked by the anti-malware product.

If files weren’t blocked, a custom-built tool was used to open the file, thus triggering AV detection by products that don’t (always) scan files on being copied.

A product passed the test if, and only if, on both platforms it blocked all files from the WildList, and didn’t generate any false positives (i.e. incorrect detections) when scanning the full clean set.

The clean set consists of almost 400,000 files, all widely used programs, with any files that show suspicious behaviour being excluded from the set.

For the ‘RAP’ (reactive and proactive) test, the same set‑up was used, but for the proactive part of the test products were not connected to the Internet. This allowed us to measure their proactive detection abilities by having a ‘frozen’ version of each product scan two sets of malware files: those seen in the wild between one day and five days after the product ‘freeze’ date, and those seen in the wild between six and 10 days after this date.

Note: A slightly different approach when it comes to tidying up the set of malware, as well as a different approach to testing, means the individual RAP scores should not be compared with those seen in tests prior to April 2017.

Latest reviews: