VB100 Comparative Review – August 2017

Martijn Grooten

Virus Bulletin

Copyright © 2017 Virus Bulletin

Introduction

Another security testing organization stated recently that it has now seen 640 million different malware samples – a number that continues to grow rapidly.

The number itself isn’t particularly meaningful – indeed, thanks to various kinds of polymorphism among malware, a lot of these 640 million samples are essentially the same – but the important message remains: there is a lot of malware out there. Therefore, if you hear a security vendor claiming that ‘signature-based anti-virus is no longer good enough’, they are right, and in fact this has been the case for at least a decade.

It is for this reason that endpoint security solutions include various kinds of heuristics that help them identify malware that has not previously been seen. And as we have shown repeatedly in the VB100 tests, it does a more than decent job of this: a version of the product that has been disconnected from the Internet for a period of time, and thus hasn’t received any new updates for that period, still detects a decent percentage of new malware – for some products well over 50%.

Such (static) detection rates are impressive, but they are not good enough when it comes to known and confirmed malware. For such malware, the very least one would expect of a good anti-malware solution is that it detects them all, without exception, and does so without blocking a single file from a collection of recent and widely used legitimate software.

For many years, these have been the criteria for earning a VB100 award – one of longest running and most well respected awards for endpoint security solutions, and the one to look for to check that a product satisfies minimum standards. In this month's VB100 test, we put 30 solutions from 27 vendors through their paces; 24 of them earned a VB100 award.

We remind readers that our test only gives a very partial picture of product performance; we recommend readers of this report also read reports from other testing organizations, for instance to find out how products fare when tasked with blocking the execution of malware.

Missing trojans

All but three products blocked 100% of the malware in the WildList, a vetted and curated list of malware known to have been seen in the wild. This is the least you should expect from an anti-malware product – i.e. that it blocks known malware still lurking on your systems.

The four files that were missed were all common trojans: Beta Bot, Perkesh, Qadars and Fareit.

On top of the near-perfect performance against the WildList, we found that the solutions recognized more than two thirds of very recent malware files as malicious – with some products performing even better than that. Given that this is just one of many detection layers that exist in anti-malware solutions, this can be considered a good performance.

Interestingly, when it came to the proactive part of the RAP test, malware seen four days after products had been frozen was especially difficult to detect, which suggests that new or significantly modified malware families were launched around that time; malware seen later was easier to detect and thus the second proactive RAP score (against a set of malware discovered 6 to 10 days after product updates were frozen) was higher for most products than the first one (against samples discovered 1 to 5 days after product updates were frozen).

Results

In the following results, the RAP images display an average of the RAP scores across the two platforms.

ad-aware antivirus pro

Arcabit AntiVirus

Avast Free Antivirus

AVG Internet Security

CompuClever Antivirus PLUS

Cyren Command Anti-Malware

Defenx Security Suite

Emsisoft Anti-Malware

Endpoint Security by Bitdefender

eScan Internet Security Suite for Windows

ESET Internet Security

Essentware PCKeeper Antivirus PRO

ESTsoft ALYac

Fortinet FortiClient

G DATA Antivirus

IKARUS anti.virus

K7 Total Security

Kaspersky Endpoint Security 10 for Windows

NANO Antivirus Pro

Panda Endpoint Protection Plus

Panda Free Antivirus

Quick Heal Seqrite Endpoint Security

Quick Heal Total Security

STOPzilla AntiVirus 8.0

TeamViewer ITbrain Anti-Malware

Tencent PC Manager

Total Defense Internet Security Suite

Total Defense Premium

TrustPort Antivirus Sphere

VirIT eXplorer PRO

Results tables

Certification tests	Windows 7				Windows 10				VB100
	FPs	FP rate	WildList misses	WildList catch rate	FPs	FP rate	WildList misses	WildList catch rate
ad-aware antivirus pro	0	0.00%	0	100.00%	0	0.00%	0	100.00%
Arcabit AntiVirus	0	0.00%	0	100.00%	0	0.00%	0	100.00%
Avast Free Antivirus	0	0.00%	0	100.00%	0	0.00%	0	100.00%
AVG Internet Security	0	0.00%	0	100.00%	0	0.00%	0	100.00%
CompuClever Antivirus PLUS	0	0.00%	0	100.00%	0	0.00%	0	100.00%
Cyren Command Anti-Malware	0	0.00%	2	99.97%	0	0.00%	2	99.97%
Defenx Security Suite	0	0.00%	0	100.00%	0	0.00%	0	100.00%
Emsisoft Anti-Malware	0	0.00%	0	100.00%	0	0.00%	0	100.00%
Endpoint Security by Bitdefender	0	0.00%	0	100.00%	0	0.00%	0	100.00%
eScan Internet Security Suite for Windows	0	0.00%	0	100.00%	0	0.00%	0	100.00%
ESET Internet Security	0	0.00%	0	100.00%	0	0.00%	0	100.00%
Essentware PCKeeper Antivirus PRO	0	0.00%	0	100.00%	0	0.00%	0	100.00%
ESTsoft ALYac	0	0.00%	2	99.97%	0	0.00%	0	100.00%
Fortinet FortiClient	1	0.0003%	0	100.00%	1	0.0003%	0	100.00%
G DATA Antivirus	0	0.00%	0	100.00%	0	0.00%	0	100.00%
IKARUS anti.virus	0	0.00%	0	100.00%	0	0.00%	0	100.00%
K7 Total Security	0	0.00%	0	100.00%	0	0.00%	0	100.00%
Kaspersky Endpoint Security 10 for Windows	0	0.00%	0	100.00%	0	0.00%	0	100.00%
NANO Antivirus Pro	2	0.0005%	0	100.00%	1	0.0003%	0	100.00%
Panda Endpoint Protection Plus	0	0.00%	0	100.00%	0	0.00%	0	100.00%
Panda Free Antivirus	0	0.00%	0	100.00%	0	0.00%	0	100.00%
Quick Heal Seqrite Endpoint Security	0	0.00%	0	100.00%	0	0.00%	0	100.00%
Quick Heal Total Security	0	0.00%	0	100.00%	0	0.00%	0	100.00%
STOPzilla AntiVirus 8.0	0	0.00%	2	99.97%	0	0.00%	0	100.00%
TeamViewer ITbrain Anti-Malware	0	0.00%	0	100.00%	0	0.00%	0	100.00%
Tencent PC Manager	0	0.00%	0	100.00%	0	0.00%	0	100.00%
Total Defense Internet Security Suite	0	0.00%	0	100.00%	0	0.00%	0	100.00%
Total Defense Premium	0	0.00%	0	100.00%	0	0.00%	0	100.00%
TrustPort Antivirus Sphere	2	0.0005%	0	100.00%	2	0.0005%	0	100.00%
VirIT eXplorer PRO	0	0.00%	0	100.00%	0	0.00%	0	100.00%

*Set -1 = Samples discovered 1 to 5 days before testing; Set -2 = Samples discovered 6 to 10 days before testing. †Set +1 = Samples discovered 1 to 5 days after updates frozen; Set +2 = Samples discovered 6 to 10 days after updates frozen. ‡ Weighted average gives equal emphasis to the two reactive weeks and the whole proactive part. 

*Set -1 = Samples discovered 1 to 5 days before testing; Set -2 = Samples discovered 6 to 10 days before testing. †Set +1 = Samples discovered 1 to 5 days after updates frozen; Set +2 = Samples discovered 6 to 10 days after updates frozen. ‡ Weighted average gives equal emphasis to the two reactive weeks and the whole proactive part.

Appendix: The test set up

The main test on each platform was run in three parts, over three consecutive weeks. Products were installed on clean installations of both Windows 7 and Windows 10. At the beginning of each part of the test we made sure the latest updates were downloaded, while throughout the test, products were connected to the Internet, thus allowing for real-time cloud look-ups.

For each part of the test, we used the most recent version of the WildList, together with one third of our constantly updated collection of widely used legitimate software. Using a shared drive, the files were copied onto the client machine and we recorded whether (and how) files were blocked by the anti-malware product.

If files weren't blocked, a custom-built tool was used to open the file, thus triggering AV detection by products that don't (always) scan files on being copied.

A product passed the test if, and only if, on both platforms it blocked all files from the WildList, and didn't generate any false positives (i.e. incorrect detections) when scanning the full clean set.

The clean set consists of almost 400,000 files, all widely used programs, with any files that show suspicious behaviour being excluded from the set.

For the 'RAP' (reactive and proactive) test, the same set‑up was used, but for the proactive part of the test products were not connected to the Internet. This allowed us to measure their proactive detection abilities by having a 'frozen' version of each product scan two sets of malware files: those seen in the wild between one day and five days after the product 'freeze' date, and those seen in the wild between six and 10 days after this date.

Note: A slightly different approach when it comes to tidying up the set of malware, as well as a different approach to testing, means the individual RAP scores should not be compared with those seen in tests prior to April 2017.

Latest reviews: