2014-02-14
Abstract
In this test the VB test team made their first visit to the all-new Windows 8.1 platform - which proved challenging for all participants, with both minor irritations and major issues affecting product performance. John Hawes has all the details.
Copyright © 2014 Virus Bulletin
Our first visit to the all-new Windows 8.1 platform was timed to coincide with its release – proving challenging for all entrants on the test bench. Several of our regular participants decided not to take part in this comparative as their products were not ready in time; others sent in products which they were not certain would perform properly; and of course we expected that several others would end up causing us all manner of trouble despite their developers’ relative confidence in them.
We also threw in a few extra hurdles for ourselves, in the form of some last-minute changes to our testing processes (things we’ve been planning for a while, and decided to introduce now, rather than in a few tests’ time, when we expect a number of other changes to be required).
The field of entrants was large despite there being several absentees, with a healthy bunch of new and new-ish faces filling out the pack. As noted recently, there seems to be an ever-growing number of products representing just a small number of engines – the most illuminating aspect of this trend is how well some products integrate an engine, and how poorly the same piece of technology can fare if not properly set up.
Windows 8.1 was released to the public on 17 October 2013, just under a year after the initial release of Windows 8. The new version promised to fix a raft of issues afflicting the original version, many of which were related to the layout and functionality of the new desktop style. Optimists hoped it would be a panacea along the lines of Windows 7 SP1, which brought Windows 7 to full maturity and made it an acceptable platform for most users.
Pessimists, meanwhile, pointed out that Windows XP did not really settle until it reached SP3. With Microsoft apparently planning annual point updates for its platforms from now on, it may be that we’ll never reach a fully settled and fixed version again – and if our initial experiences with Windows 8.1 are anything to go by, we’re certainly not there yet.
The return of a start menu (of sorts) was a plus, meaning the reboot which is the main function of Windows is easier to perform. Some of the other tweaks were also welcomed, but we still found stability a bit of an issue, managing to achieve blue screens without really trying at all. We only hoped that the products under test would not add to this wobbliness too much.
As usual, the operating system was installed at a basic level, with just the content of the distribution media and no more recent updates. A few settings were tweaked, including disabling Windows Defender and the UAC system, which proved to interfere with a number of our test automation systems. A handful of simple tools were installed, including archivers and document viewers to assist with installation of products, and we were all set.
The clean sets were updated with a wide range of software, including a selection of popular apps from the Windows app store – which one would hope would be thoroughly checked, but which have proven tricky for some products in the past. The overall size of the clean sets came to 950,000 files, 240GB of data. There were no significant changes to the test sets used for measuring speed, performance impact and resource usage, and after some experimenting we found that all of our standard scripts operated as usual.
The other half of the certification sets were based on the latest WildList available on our 23 October deadline – the September 2013 list, which had been released a week prior to our deadline. As usual, the sample sets were embedded in our clean sets to ensure proper detection was present.
Elsewhere, there were a few changes to our operating procedures. As ever, more products are incorporating cloud look up systems both for detection and for FP mitigation. Our Response test, which allows products to access such schemes, has been extended slightly, and now covers ten days rather than the previous seven. We have also merged this with the RAP test – retiring the ‘reactive’ parts of the RAP sets and replacing them with the Response test data.
The proactive section of the RAP test remains unchanged in its approach – cloud systems and updates cannot be included in a retrospective test – but this too has been extended to ten days, broken into two five-day subsections.
One final change was necessitated by the fact that the platform we were testing on includes built-in anti-malware protection by default. Windows Defender was disabled for the main test period, with most products either disabling it themselves or requesting it be done as part of their installation process. We took a set of baseline measures for our speed and performance tests in this disabled state.
However, as the ‘out-of-the-box’ experience of Windows 8 (and 8.1) includes Defender, it seems appropriate to treat this as the baseline, so we also recorded baseline measures with it fully running, and used those as our main data source. In this test we have included both sets of data (i.e. with Defender disabled and with Defender running), but in future we will only include baselines using the default state of the platform – which will often mean having any built-in anti-malware fully operational.
This does, of course, lead to a number of negative values in our tables and charts, thanks to products performing better than the baseline. We have also seen negative values both in the past and in this test, in cases where products take a very long time to perform a task which is usually resource intensive, spending much of the extra time idle and thus producing a lower average for the resource use measures. So, we hope that including negative data won’t seem too awkward.
With plenty of products to get through and some interesting problems to work around, testing got under way as soon as possible.
Main version: 4313.670.1936
Update versions: N/A
Last 6 tests: 3 passed, 0 failed, 3 no entry
Last 12 tests: 5 passed, 1 failed, 6 no entry
Agnitum was absent from our comparatives for a spell while the company’s in house team took over maintenance of the scanning engine from original developer VirusBuster – but since its return, Agnitum has put in some reliable performances.
The product installs in a couple of minutes with a reboot needed at the end, and the interface is crisp and clean, offering a decent set of controls.
Scanning speeds were reasonable initially, and very fast indeed after the first run; overheads were a little high over binaries but decent elsewhere, again improving greatly once warmed up. RAM use was just a fraction above average, CPU use a little on the high side, and time taken to process our set of standard tasks was rather lengthy. Stability was impeccable, with no issues to report.
Detection was not too bad, dropping off sharply into the proactive sets, but the certification sets were well managed and Agnitum comfortably qualifies for a VB100 award, continuing a nice run of form.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Solid
Main version: 2014.90.2006
Update versions: 131023-0, 131114-1, 13118-1, 131125 0
Last 6 tests: 5 passed, 1 failed, 0 no entry
Last 12 tests: 11 passed, 1 failed, 0 no entry
Avast’s free version is something of a fixture in our desktop tests, and usually keeps the team happy.
Installation is speedy, but reboots are required after install and on some updates. The GUI is excellent, combining attractive styling with good usability and still providing a wealth of fine tuning for the many features included in the suite.
Scanning speeds were reliably decent, if not as quick as some, while light overhead measures will have been influenced by not having on-read protection in place by default, unlike most other solutions. Resource use was very low, again perhaps influenced by the choice of default settings, but our set of activities includes many other kinds of operations which should be more closely monitored, and showed a slightly higher figure – a little above the average for the test. There were no problems with stability.
Detection in the RAP sets was a little lower than we would expect, but still within the bounds of respectability. The WildList and clean sets were properly handled and a VB100 award is well deserved.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Solid
Main version: 2014.0.4158
Update versions: 3614/6769, 3629/6835, 3629/6849, 2013.0.4259 3629/6874
Last 6 tests: 5 passed, 1 failed, 0 no entry
Last 12 tests: 10 passed, 2 failed, 0 no entry
AVG rarely misses a VB100 comparative, and has settled into a pattern of generally reliable passes, but we’ve noted some problems with stability lately.
The set-up for the suite version is fairly speedy and simple, with no reboot needed, and the interface is dark and boxy but fairly easy to navigate, with a full set of controls.
Scanning speeds were impressive from the off and even better in the warm runs; on-access overheads were a little higher than some, but not excessive. RAM use was on the high side, but CPU use was very low indeed, with impact on our set of tasks a shade above average but not too bad. We encountered a number of problems, including several scans crashing out with a rather less than helpful error message, and issues with logging, but the biggest issue was the same problem with unexpected shutdowns as we have seen in several recent tests. We also experienced a blue screen incident during testing, but it’s possible that this was due to other software, so AVG is not penalized for this.
Detection was good though – very strong in the response sets, dropping away a fair bit into the proactive sets but maintaining a respectable level. The WildList set was fully covered and there were no slips in the clean sets, thus AVG earns a VB100 award, keeping up a decent record.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Buggy
Main version: 14.0.0.383.2
Update versions: 8.02.12.132/7.10.8.234, 8.02.12.140/7.11.113.54, 8.02.12.144/7.11.114.66, 14.0.1.749 / 8.02.12.150/7.11.115.162
Last 6 tests: 3 passed, 0 failed, 3 no entry
Last 12 tests: 5 passed, 0 failed, 7 no entry
Avira’s free version makes an appearance in most of our desktop comparatives, and has managed to achieve certification very dependably over the last few years.
The set-up process isn’t the quickest, and is accompanied by a little light promotional reading, but a reboot is not needed – instead launching straight into a preparatory scan on completion. The interface is well laid out and responsive, providing most of the controls one might need, but oddly lacking some simple items that are present in the paid-for version.
Scanning was pretty speedy for the most part, overheads a shade higher than some, but not excessive. RAM use was low, CPU use around average, and our set of tasks sped through in pretty good time.
Detection was very strong as usual, dropping off a little into the proactive sets but not too much, and the core sets were well handled, earning the product a VB100 award and maintaining a strong record for Avira’s free edition.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Solid
Main version: 14.0.0.383.2
Update versions: 8.02.12.132/7.11.108.234, 8.02.12.140/7.11.113.54, 8.02.12.132/7.11.114.66, 8.02.12.150/7.11.115.162
Last 6 tests: 5 passed, 0 failed, 1 no entry
Last 12 tests: 10 passed, 0 failed, 2 no entry
The ‘Pro’ version from Avira has also maintained an excellent pass rate over the last few years, only missing a couple of comparatives.
The installation process is fairly similar to that of the free edition, minus the advertising, and completes within a few minutes. The interface looks similar for the most part too, but provides a few more advanced settings for the more expert user.
Scanning speeds were much the same too: fast on demand and reasonable on access, with RAM use remaining a little below average, CPU use just a fraction over, and again our set of tasks weren’t too heavily impacted. We noted one very minor error with an update failing to complete – fairly trivial and easily rectified on the next attempt, but just enough to dent an otherwise decent showing for stability.
Detection was also excellent, and there were no problems in our certification sets, the product easily earning a VB100 award to maintain Avira’s long run of successes stretching back to 2009.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Stable
Main version: 17.20.0.883
Update versions: 7.51073, 7.51471, 7.51579, 7.51779
Last 6 tests: 6 passed, 0 failed, 0 no entry
Last 12 tests: 12 passed, 0 failed, 0 no entry
Bitdefender has an even stronger recent record, with a pass in every VB100 comparative we have run since August 2010.
The ‘Antivirus Plus’ edition installs fairly quickly, with an initial stage looking for updated versions of the main installer before getting down to its business. Once completed, it presents a dark and brooding interface which takes a little exploration to get used to, but provides a very thorough set of controls and is fairly usable and responsive.
Scanning speeds started fast and grew faster, while lag times for simple file accesses were low, and additional RAM use with the system idle was also low. Our other resource use measures were a little skewed, however, thanks to a bizarrely slow showing in our set of activities, which took more than ten times as long to complete as they did on an unprotected system. This has happened repeatedly in the last few tests, on different platforms and with different product variants, and we have yet to come close to figuring out the cause. A side effect is that RAM and CPU measures, which are recorded every five seconds while the activities are being performed, average out very low indeed, implying that most of the time the system is in fact idle and waiting for some external input.
Other than this oddity, there were no stability problems, and since this slowness cannot fairly be treated as an actual bug, a Solid stability rating is earned.
Detection, meanwhile, proved problem-free, with the usual excellent levels across the board, the proactive week of the RAP sets being particularly impressive. There was nothing to complain about in the certification sets and a VB100 award is earned, adding to that unbroken streak of passes.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Solid
Main version: 13.0.264
Update versions: 13.0.264, 13.0.264, 13.0.264.1
Last 6 tests: 5 passed, 0 failed, 1 no entry
Last 12 tests: 10 passed, 0 failed, 2 no entry
Yet another product with a splendid record of success, BullGuard usually misses our annual visits to the Linux platform, but can now boast three straight years of success in all our other tests.
Installation requires minimum input and completes rapidly, with initial updates adding a little more time. The GUI is unusual and will take a moment to figure out, but provides a reasonable range of options and has proven solid and reliable for some time.
Scanning speeds started fast on demand, with sharp upturns in all the warm runs, while overheads were fairly light and all of our resource measures came in below average. In sharp contrast to BullGuard’s engine provider, Bitdefender, our set of tasks was completed particularly quickly.
Detection rates were very similar though, proving excellent in all areas, with again an impeccable showing in the certification sets, easily earning BullGuard a VB100 award to maintain that three-year streak.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Solid
Main version: 12.0.103.000
Update versions: 8.3.2.4, 8.3.2.4, 8.3.2.4, 12.01.103.000/8.3.2.4
Last 6 tests: 1 passed, 2 failed, 3 no entry
Last 12 tests: 2 passed, 2 failed, 8 no entry
Check Point’s history in our comparatives is somewhat chequered, with only sporadic entries, and passes similarly few and far between. Extensive investigation into some of the product’s more recent issues seems to have paid off though, taking us into this test much more confident of a good performance.
Set-up is fairly straightforward, including some initial scans of memory and so on, as well as the offer to email the user with useful security information. The interface seems to have evolved a little, but retains a familiar look and feel – heavy on words as links rather than distinct buttons, but mostly fairly intuitive with a reasonable set of controls provided.
Scanning speeds started rather slow, but did speed up nicely after the first run, while overheads were similarly high initially but barely perceptible in the warm runs. Resource use was rather high across the board, but with a fairly decent time taken to complete our set of activities. Stability was not great, with a number of scans freezing up, failing to cover areas selected, and even adding additional areas, while logging was also a little unreliable, earning the product only a Fair rating.
Detection was decent though, dropping off a little in the proactive sets as one would expect. The certification sets were handled well, and a VB100 award is earned, raising our hopes of a more consistent performance going forward.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Fair
Main version: 5.1.23
Update versions: 5.4.2/201310231126, 201311130942, 201311210840, 201311261310
Last 6 tests: 1 passed, 4 failed, 1 no entry
Last 12 tests: 3 passed, 7 failed, 2 no entry
Commtouch rebranded part way through this test, announcing the company’s new name as CYREN, but for now we’ll stick with the name on the product as submitted. The test history for Commtouch is a little shaky, a string of problems with false positives having spoilt recent tests for the vendor.
The product has changed little in several years, with a small and rapid installer and similarly quick updates. The interface is basic, almost rudimentary, but one benefit is that it is hard to get lost in, and it still provides a decent set of controls.
Scanning was rather slow, and on-access overheads were pretty heavy. RAM use was low, CPU use fairly high, and our set of tasks took quite some time to get through. Stability was not great either, with a number of scans freezing up, logging proving unreliable and prone to crashes in the viewer tool, and on one occasion the product took over 15 minutes to come back online after a reboot.
Detection was not bad though, with the change to the format of the RAP test meaning the company’s cloud system comes into play, at least in the reactive parts, earning some decent scores there and dropping away sharply into the proactive sets as one might expect. The WildList sets were properly handled, and in the clean sets, for a change, there were no problems to report, thus earning the product a VB100 award and hopefully setting a good precedent for the newly renamed firm.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Fair
Main version: 8.1.1
Update versions: 4318.687.1936
Last 6 tests: 1 passed, 0 failed, 5 no entry
Last 12 tests: 3 passed, 1 failed, 8 no entry
This test sees the return of former regular Defenx, which took some time out of testing during the change of ownership of the product’s engine (which is provided by Agnitum).
The product closely resembles that of Agnitum, the installation process offering an unfussy progress through the stages, most of the time being taken up with updates at the end. The interface is also simple and clear, easy to navigate and offers a reasonable set of controls.
Scanning speeds were rather slow the first time around, but much quicker later on, while overheads were a bit on the heavy side, again improving a little in the warm runs. Resource use was decidedly high, and our set of tasks took quite a long time to complete too. Stability could not be faulted though, earning Defenx its first Solid rating.
Detection was not great, but not terrible either – it was better in the reactive sets with a fairly steep drop into the later parts of the RAP sets, but there were no issues in the WildList sets, and with no false alarms either, a VB100 award is earned. We expect to see Defenx back as a regular participant in our comparatives going forward.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Solid
Main version: 8.1.0.19
Update versions: N/A
Last 6 tests: 5 passed, 0 failed, 1 no entry
Last 12 tests: 7 passed, 3 failed, 2 no entry
After a bit of a rocky time in 2012, Emsisoft had a strong 2013, putting in a nice run of passes thanks mainly to a change in the underlying engine.
The installation process is a little slower than some, with updates the main culprit, but it’s still over in a few minutes. The interface is bright and attractive, the design taking a few moments to get used to but quickly proving clear and simple to use, offering a good set of controls.
Scanning speeds were reasonable, and on-access lag times seemed very light indeed, but this will be due to the absence of on-read protection by default. RAM use was low, CPU use a little above average, and our set of activities blasted through very quickly indeed.
Stability was far from perfect, with a number of scans freezing up or crashing out, some problems getting scans even to start from within the product GUI, issues with the on-access component not staying online for long on one install, and another where a reboot failed to lead anywhere at all. Most problems only occurred under heavy stress though, and a Fair rating is earned.
Detection was really quite excellent though, and the certification sets were handled well, earning Emsisoft a VB100 award to continue its run of good showings.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Fair
Main version: 14.01.1400.1504 DB
Update versions: N/A
Last 6 tests: 5 passed, 1 failed, 0 no entry
Last 12 tests: 11 passed, 1 failed, 0 no entry
eScan came into this test on the back of a string of 16 uninterrupted passes, having not missed a VB100 test since 2009.
Initial set up is a rather lengthy process, requiring a reboot and dawdling over the update stage for quite some time. The interface is a little dark, with grey-on-grey writing a little hard to pick out at times, but is styled to fit the look of Windows 8 and has a professional feel to it. Configuration is provided in great depth, and the layout is easy to navigate.
Scanning speeds were rather slow, but overheads not too bad, with RAM use on the low side, CPU use just a little higher, and our set of tasks getting through in good time. We observed a few issues, including some scans of clean sets crashing out, but most significant were some problems with the on-access component, which failed either completely or partially multiple times after initial installation.
Investigation in discussion with the developers has indicated a possible timing issue, with our tests launched before the product update had completed properly, although the GUI gave every indication that it had completed. Of course, in the real world, attacks would not wait until a product was ready for them, and we saw the same handful of samples missed several times over some hours after initial installation, including at least one instance of the same problem recurring the following day.
So, despite there being no problems in our clean sets, a handful of WildList samples were missed both on demand and on access in the first two runs (those from the second run all included in the slightly larger cluster from the first) thus denying eScan a VB100 award this month, and breaking that great run of passes.
ItW Std: 99.83%
ItW Std (o/a): 99.83%
ItW Extd: 99.72%
ItW Extd (o/a): 99.72%
False positives: 0
Stability: Fair
Main version: 7.0.303.0
Update versions: 8955, 9038, 9068, 9090
Last 6 tests: 6 passed, 0 failed, 0 no entry
Last 12 tests: 12 passed, 0 failed, 0 no entry
Another vendor with a strong run of passes, this one unbroken for an epic length of time, ESET’s test history includes over ten years without even missing a comparative.
Set-up was pretty short and sweet, with the interface very good-looking and an excellent range of controls provided. The layout can feel slightly confusing in places, with some options seeming to overlap, but the defaults are rigorous and should be fine for most users.
Scanning speeds were very quick from the off and even faster in later runs, while overheads were low, as was RAM use. CPU use was a touch above average, but our set of activities blasted through very quickly indeed. Stability was impeccable throughout.
Detection was strong, dropping off somewhat into the proactive sets, with no problems in the core sets, thus comfortably earning ESET another VB100 award to add to its large collection.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Solid
Main version: 2.5.0.23
Update versions: 13.3.21.1/513921.2013102315/7.51073/10202731.20131023, 13.3.21.1/515305.2013111419/7.51457/10428887.20131114, 13.3.21.1/515525.2013111920/7.51564/10470280.20131119, 13.3.21.1/515891.2013112718/7.51776/10512988.20131127
Last 6 tests: 4 passed, 0 failed, 2 no entry
Last 12 tests: 5 passed, 2 failed, 5 no entry
ESTsoft’s test history looks pretty impressive over the last few years, although what it does not show is the fact that there were a couple of tests in which the product was found too unstable to make it as far as the final report.
The product submitted this time proved reasonably fast to install, but as in past tests, updates took some time to complete. The interface is a little different from the standard format, but reasonably easy to navigate once a few odd bits of wording have been deciphered, and seemed pretty stable for the most part – just a few minor wobbles under heavy stress detracting from a perfect score.
Scanning speeds were pretty good, and pretty even across the various runs, with on-read overheads very light indeed. Resource use was also very low, but our set of activities did take a while to run through.
Detection was very strong, as expected from the underlying Bitdefender engine, and the certification sets were properly dealt with, earning ESTsoft a VB100 award without fuss.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Stable
Main version: 5.0.6.320
Update versions: 5.147/16.378, 5.147/20.453, 5.147/20.823, 5.147/20.871
Last 6 tests: 5 passed, 0 failed, 1 no entry
Last 12 tests: 10 passed, 0 failed, 2 no entry
Fortinet has had a steady few years in our tests, with no upsets and only the annual Linux tests not taken part in.
The set up process is very rapid for the most part, although one rather drawn-out update threw the average completion time off a little. The interface is rather simple with few controls – presumably the product is intended to be deployed, configured and administered remotely in a corporate environment.
Scanning speeds were reasonable on demand, a little slower on access, at least in the initial runs, but resource use was low, especially CPU use, and our set of tasks didn’t take too long to complete. Stability was rocked by no fewer than three blue screen incidents, two of them when handling mainly normal clean files and one of them requiring the running of a system repair process from the original installation media to get Windows running again.
Detection was very strong though, and with no issues in the WildList or clean sets, a VB100 award is earned despite the stability problems.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Buggy
Main version: 11.00 build 332
Update versions: 9.50 build 19220
Last 6 tests: 4 passed, 2 failed, 0 no entry
Last 12 tests: 6 passed, 3 failed, 3 no entry
F-Secure has a surprisingly uneven test history, with a two:one pass:fail ratio over the last two years despite other products using the same (Bitdefender) engine not having any problems.
This month’s ‘Premium’ version seems fairly similar to other desktop editions we’ve looked at lately, with the installer running fairly quickly but needing a reboot, and taking quite some time to get initial updates in place. Although the update dialog disappears after a few moments, checking progress deeper in the GUI reveals the actual readiness or otherwise of the product.
Scanning speeds were blindingly fast on demand, but this is one of few products that still relies on file extensions to determine what to scan, and defaults to a limited set of file types. On-access defaults are similar, which will have contributed to the low lag times measured. RAM use was a little above average, but CPU use barely perceptible, and our set of activities ran through a fair bit more slowly than the average for this test. A few minor issues were noted, including unreliable logging and problems firing off on-demand scans after on-access detections had been recorded, both issues that have been noted previously.
Detection was very solid though, and still pretty good going into the proactive sets. The certification sets presented no difficulties, thus earning the product a VB100 award and maintaining F-Secure’s recent ratio.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Stable
Main version: 24.0.3.2
Update versions: AVA 22.13386/GD 23.2266, 24.0.3.4 AVA 22.13784/GD 23.2414, AVA 22.13896/GD 23.2414, 24.0.3.2 AVA 22.14100/GD 24.23.59
Last 6 tests: 5 passed, 0 failed, 1 no entry
Last 12 tests: 9 passed, 0 failed, 3 no entry
G Data hasn’t put a foot wrong in our tests in the last two years, and has missed very few of them.
The 2014 edition includes a new in-house detection engine alongside the ubiquitous Bitdefender one. Set up is not the fastest, with reboots required both to complete the main install and also on some occasions after initial updates. The interface has had a few tweaks but still looks good – businesslike without being unfriendly – and provides a comprehensive set of options. At one point we thought we had spotted a rare moment of imperfect stability when an update attempt failed, but it quickly emerged that this was due to incomplete activation.
Scanning speeds were OK to start with and pretty quick in the warm runs; overheads were a little high but again showed some improvement later on. RAM use when idle was fairly low, but our busy measures were thrown off kilter by the very long time it took to get through the activities sets – over seven times the baseline figure – during which much time was spent idling judging by the very low CPU measure.
Detection levels were very high indeed though – not far short of perfect in the reactive sets and still excellent into the proactive sets, and with no issues in the core sets a VB100 award is easily earned.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Solid
Main version: 2013-10-24.00
Update versions: 10204928, 2013-11-15.00 10440093, 2013-11-19.00 10471345, 2013-11-27.00 10513328
Last 6 tests: 1 passed, 2 failed, 3 no entry
Last 12 tests: 5 passed, 2 failed, 5 no entry
Another product that is based on the Bitdefender engine but which has not fared quite as well as others in recent comparatives, Hauri has a rather unpredictable test record.
The installation process is fairly standard, with initial updates taking up most of the set up time. The interface remains unchanged, as does the version – hanging onto a 2011 tag for an unusually long time. The design is clear and simple, following a fairly standard pattern, and provides a reasonable set of options.
Scanning was a little slow over archives, but decent elsewhere, with lag times a touch on the high side. Resource use was low, and our set of activities progressed rapidly.
Detection was well up with other Bitdefender offerings, making something of a change from our recent experiences with the product. There were no problems in the core sets, so Hauri earns a VB100 award this time.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Stable
Main version: 2.2.29
Update versions: 85543, 85745, 85805, 85882
Last 6 tests: 2 passed, 3 failed, 1 no entry
Last 12 tests: 5 passed, 5 failed, 2 no entry
Another product with a rather unsteady history, false positives have been the main source of pain for Ikarus of late.
Installation is confused by the need for the .NET framework, which the product tries to set up but is unable to do thanks to some changes in Windows 8 – this leads to a cycle of reboots as the process cannot complete properly. With the dependency put in place using the proper method, the rest of the set-up is very fast and simple. The interface hasn’t changed much in some time, offering a fairly easy to navigate GUI with a reasonable set of controls.
Scanning speeds were distinctly slow, lag times on access a little high too, but improving in the warm runs. RAM use was a little below average, CPU use a little above average, and our set of activities completed very rapidly. Stability was generally good, but aside from the bug with the install process we also noted a few updates failing to connect.
Detection was excellent in the reactive sets, a little lower in the proactive ones. For once there were no problems in the clean sets, but in the WildList sets a pair of items were undetected for the first two of our three runs, meaning that Ikarus misses out on a VB100 award once again.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 99.86%
ItW Extd (o/a): 99.86%
False positives: 0
Stability: Stable
Main version: 4.0.13011
Update versions: 2013.02.13.00, 2013.11.15.00, 2013.11.19.00, 2013.11.28.00
Last 6 tests: 2 passed, 0 failed, 4 no entry
Last 12 tests: 2 passed, 0 failed, 10 no entry
A relative newcomer to our tests, with just one previous appearance, Inca is yet another product that includes the Bitdefender engine.
The installation process runs along standard lines, but on most installs the update had to be run several times to get it to complete happily. The interface is fairly basic and easy to find one’s way around, with only a rudimentary set of options provided. S
canning speeds were not bad at all, overheads fairly light, with low resource use and a fairly average showing in our set of activities. Stability was only marred by the minor problems with updating.
Detection was strong, as one would expect, and the certification sets were handled well, earning Inca its second VB100 award from two attempts.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Stable
Main version: 4.5.1.7
Update versions: N/A
Last 6 tests: 2 passed, 0 failed, 4 no entry
Last 12 tests: 2 passed, 2 failed, 8 no entry
With us for some time, but with fairly irregular patterns of participation, Iolo includes the engine originally developed by Frisk, which became part of Commtouch, formerly Authentium and now CYREN.
The product set-up is run from a miniature downloader tool, but completes fairly quickly even with the extra time needed to fetch down the main components. Updates are fairly zippy too. The interface is clear and clean with an attractive, professional look, and a reasonable set of controls.
Scanning speeds were on the slow side, and overheads pretty hefty on access. RAM use was not bad, but CPU use was fairly high despite our set of activities taking an age to complete. Stability was hit by some problems in getting the on-access component to load, requiring a couple of reboots on some installs, and also by some freezing up of the system.
Detection was rather tricky to measure, the bizarre format of the product’s logging remaining opaque despite several requests to the developers for advice on de-obfuscating it. In the past, we have managed to rip it into some semblance of shape with much manual intervention, but this month the problems were too severe and we decided to spend the available time on just the certification sets, which proved well covered. As a result, a VB100 award is granted, but other scores remain blank – we hope to see a more tester-friendly version fairly soon.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Fair
Main version: 13.1.0205
Update versions: 9.173.9469, 9.173.10213, 9.174.10260, 9.174.10305
Last 6 tests: 2 passed, 1 failed, 3 no entry
Last 12 tests: 4 passed, 2 failed, 6 no entry
K7’s recent test history looks a little uneven, but in the longer term things look better – with very few fails in the seven years or so in which the product has participated in our tests regularly.
The current version has a very speedy installation process, and an interface that inspires confidence, looking elegant and professional and providing a wide range of controls in an easy-to-use manner.
Scanning speeds were not the fastest, but overheads went from a fairly high starting point to very light indeed in the warm runs. RAM use was around average, but CPU use was barely perceptible and impact on our set of activities also pretty minimal. Stability was flawless throughout, earning a Solid rating.
Detection rates were pretty strong in the reactive sets, dropping fairly steeply in the proactive ones. The clean sets were properly managed, as were the WildList sets on demand, but on access a handful of items went unnoticed, apparently thanks to a rare engine bug only appearing on some 64-bit platforms (which the developers had apparently spotted and started work on fixing before we alerted them to the problem). So, no VB100 award for K7 this month, despite a generally strong showing.
ItW Std: 100.00%
ItW Std (o/a): 99.49%
ItW Extd: 100.00%
ItW Extd (o/a): 99.85%
False positives: 0
Stability: Solid
Main version: 14.0.0.4651(b)
Update versions: N/A, 14.0.0.4651(c)
Last 6 tests: 5 passed, 1 failed, 0 no entry
Last 12 tests: 10 passed, 2 failed, 0 no entry
There have been only two comparatives since the start of VB100 testing in 1998 that have lacked a submission from Kaspersky, and in that time the vendor has racked up some impressive runs of form, its recent record pretty strong if not entirely devoid of slips.
The latest I.S. version installs with minimal effort, in decent time, and presents a GUI in the usual pleasant green tones. The look and feel is slick and professional, with some attractive styling, and a comprehensive set of controls is provided in an accessible manner.
Scanning speeds were sluggish to start with in some sets, but pretty speedy in other areas with excellent improvements in the warm runs. Overheads were a touch on the high side, but only in the initial runs; later visits were very quick indeed. RAM use was below average, but CPU use a little higher than most, with a decent time taken to complete our set of tasks. Stability was marred by a single incident where the system froze up for over 30 minutes after one of our most demanding malware scans, but was otherwise fine.
Detection was pretty strong in the reactive sets, dropping off somewhat in the proactive sets where cloud systems are inaccessible. The WildList sets were well handled, with some items in the Extended set listed as unwanted in product logs and not blocked by default, but this is fine within our rules, and with no issues in the clean sets a VB100 award is well deserved, adding to Kaspersky’s impressive tally of passes.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Stable
Main version: 2013.SP5.0.031800
Update versions: 2013.SP5.0.102316, 2013.SP5.0.111117, 2013.SP5.0.112215
Last 6 tests: 5 passed, 0 failed, 1 no entry
Last 12 tests: 6 passed, 0 failed, 6 no entry
After an absence of over a year, Kingsoft re-emerged not so long ago with an all new product based on the Avira engine – since when it has chalked up a string of successes.
The product installs with a zingy graphic which makes the process feel rapid, but slow updates add considerably to the overall set up time. The interface is in Chinese only, but looks attractive and is reasonably usable even based on icons alone. Helped by a basic guide provided by the developers, we found plenty of controls and it seems likely that more are available for the more demanding (and Chinese-speaking) user.
Scanning started off fairly slow, but sped up a lot in later runs, while on-access lag times were not bad initially and again improved in the warm runs. RAM use was low, CPU use fairly high, and our set of tasks got through nice and quickly. Scanning stability was a little dubious under any kind of pressure, with many scans of our malware sets failing to complete and requiring time-consuming re-runs in ever smaller chunks.
Detection was good in the end though, well up with the leaders in the reactive sets but dropping off fairly steeply into the proactive sets. The core sets brought up no problems, and a VB100 award is earned, maintaining Kingsoft’s run of success since its return.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Fair
Main version: N/A
Update versions: 8.2.12.140/7.11.113.54, 8.2.12.144/7.11.114.68, 8.2.12.150/7.11.115.162
Last 6 tests: 2 passed, 0 failed, 4 no entry
Last 12 tests: 3 passed, 1 failed, 8 no entry
Kromtech is a fairly new name, but the product has some history under its previous company name, ZEOBIT, and continues to use the reliable Avira engine.
The set-up is another that uses a simple downloader tool, but completes in decent time, and the interface is bright and friendly with simple controls and a good basic set of options.
Scanning speeds were pretty decent, and overheads pretty light, at least after initial acclimatization. Resource use was fairly high, particularly CPU use, but our set of tasks completed in good time. Stability was affected by a couple of instances of the system becoming slow to respond, mainly when trying to reboot.
Detection was uniformly excellent though, and with no problems in the certification sets a VB100 award is well deserved.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Stable
Main version: 11.0.4555.0
Update versions: N/A
Last 6 tests: 3 passed, 0 failed, 3 no entry
Last 12 tests: 3 passed, 1 failed, 8 no entry
We saw some major changes for Lavasoft this month, with a new front end combined with a new engine under the covers, so previous test results may not reflect the current product’s abilities.
Installation was a little slow, thanks in part to some additional items that are combined with the main product by default, and a reboot is needed. Initial updates also took quite some time. The new interface looks great, with a nice simple layout and a good set of options. It also seemed nice and responsive even under heavy stress, with no stability problems to report.
Scanning speeds were good to start with and very good in the warm runs, while overheads were only reasonable. RAM use was a little high, but CPU use very low and our set of activities ran through very quickly.
Detection was very good indeed, especially in the proactive sets, closely aligned with other well-performing products using the Bitdefender engine. The core sets were deftly handled, and a VB100 award is well deserved, along with our congratulations on a much improved product.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Solid
Main version: N/A
Update versions: N/A, 1.1.45.80
Last 6 tests: 0 passed, 1 failed, 5 no entry
Last 12 tests: 0 passed, 1 failed, 11 no entry
Maya is a new name in our tests, although we have looked at the product a few times in the past; this was the first time it was found stable enough to complete a reasonably full set of tests.
The installation process is fast, and while the interface is brightly adorned with fancy icons, it is otherwise a little basic in appearance – the set of controls provided is reasonable, though.
Scanning speeds were pretty impressive, with high use of RAM, but reasonable CPU use. Both our simple file access lag time measure and our set of activities look very fast, but these are likely to have been heavily impacted by stability issues in the on-access scanner, which seemed to cut out repeatedly, even in normal everyday use with only clean files being inspected. We also saw some problems with scans freezing up.
There is no proactive detection measure for Maya due to problems setting up the product in time on the deadline day, but our reactive sets were not very well handled at all. WildList coverage was also fairly dismal, with no reliable on-access figures thanks to an inability to stay on throughout a full test run. There were also a fair number of false alarms in our clean sets. Overall, it seems that Maya’s developers still have a fair amount of work to do in order to get their product up to the standard required for VB100 certification.
ItW Std: 60.56%
ItW Std (o/a): N/T
ItW Extd: 40.63%
ItW Extd (o/a): N/T
False positives: 14
Stability: Buggy
Main version: 4.4.303.0
Update versions: 1.1.10003.0/1.161.91.0, 1.161.1960.0, 1.161.2352.0, 1.1.10100.0/1.163.535.0
Last 6 tests: 3 passed, 0 failed, 3 no entry
Last 12 tests: 5 passed, 0 failed, 7 no entry
This product’s test history may appear a little incomplete, but that is due to the fact that Microsoft alternates between submitting its consumer and corporate products – few VB100 comparatives go by without an appearance from one or the other, and each has a pretty consistent record of passes.
The System Center solution was one of the fastest to be installed this month, with a simple process to follow. The interface has a similar look and feel to its consumer siblings, providing a decent set of controls in a solid and reliable format.
Scanning speeds were reasonable, and file access lags a little heavy initially but barely noticeable after the first preparatory inspection of files. Resource use was around average, but our set of activities completed very quickly indeed. Stability was mostly fine, with just a few errors noted during updating.
Detection was reasonable, if well behind the leaders, and there were no issues in the core sets, earning another VB100 award for Microsoft.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Stable
Main version: 1.1.107.0
Update versions: 84947, 85763, 85805, 85873
Last 6 tests: 0 passed, 3 failed, 3 no entry
Last 12 tests: 1 passed, 3 failed, 8 no entry
MSecure’s product has put in some reasonable performances in our tests to date, but has repeatedly been blighted both by internal stability problems and issues attributable to the Ikarus engine incorporated in the product.
This month’s installation and set up process was pretty quick on some occasions, but rather slow on others, mainly thanks to the update procedure. The interface is reasonably good looking, if a little boxy, and has a reasonable set of control options too.
Scanning speeds were decidedly slow, but overheads were very light and resource use barely detectable – although this may be attributed to some problems with the on-access component, which appeared not to be functioning properly at all. Nevertheless, our set of activities still took a fair while to complete.
Detection was pretty good on demand, especially in the reactive part of the RAP sets, but with on access scanning not reliable, no score was recorded for the WildList set in this mode. On demand, as with Ikarus, a handful of files were ignored, so even with false alarms happily absent, no VB100 award can be granted.
ItW Std: 100.00%
ItW Std (o/a): N/T
ItW Extd: 99.76%
ItW Extd (o/a): N/T
False positives: 0
Stability: Buggy
Main version: 10.1
Update versions: 7.02.06
Last 6 tests: 3 passed, 3 failed, 0 no entry
Last 12 tests: 8 passed, 4 failed, 0 no entry
Things have been a little rocky for Norman of late, with a 50:50 pass:fail rate in the last year of testing, but a little better in the longer term.
The vendor’s main suite solution has a rather sneaky set up process, appearing to have completed very rapidly but actually doing more work in the background for a few more minutes, after which a reboot is requested. The interface looks quite nice these days, although options for control are fairly limited.
Scanning speeds weren’t too bad, overheads fairly high in some sets (notably the set of binaries), but improved greatly in the warm runs. RAM use was low, CPU use fairly high, and our set of tasks ran through in good time. Some minor issues were noted with logging, notably a difference between GUI and file versions of the same scan log.
Detection was pretty good too, just a little behind the leading pack, and the WildList sets were well covered too. In the clean sets, unfortunately, a single item was mislabelled as malware with a heuristic detection, denying Norman a VB100 award on this occasion.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 1
Stability: Stable
Main version: 11.10 build 3376
Update versions: 0.12.0.163/255.1.0.19,
Last 6 tests: 2 passed, 0 failed, 4 no entry
Last 12 tests: 3 passed, 1 failed, 8 no entry
Not the most regular participant in our tests, Optenet has been on our radar for a while, but has only racked up a few entries so far.
The current product is based on the Bitdefender engine, as are so many others, and takes a little while to install. A reboot is needed only if a full licence is applied. The interface is a little hard to operate, not working at all with the default browser on our Windows 8.1 systems, but after a little investigation we were able to figure out how to open it in Firefox, which proved better at displaying the required pages. The layout is fairly clear and sensible, with a reasonable set of options available, but like other browser-reliant interfaces it suffered a little from lagginess, and we saw a few freezes and crashes too.
Scanning speeds were decent, with some improvements noted in the warm runs. Overheads were a little heavy, except over archives which are not scanned in any depth on access. Resource use was slightly above average in all measures, although our set of activities got through in good time.
Detection was pretty decent, although not quite as strong as we would expect, and there were no problems in the core sets, earning Optenet another VB100 award for its slowly growing tally.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Fair
Main version: 2.3.0
Update versions: N/A
Last 6 tests: 3 passed, 2 failed, 1 no entry
Last 12 tests: 5 passed, 2 failed, 5 no entry
Panda has become a regular on our test bench once again after a long period of absence. While it got off to a good start on its return, the last few tests were marred by some rather unlucky false alarms.
The free cloud product installs almost instantly, with minimal input required from the user. The interface is simple to the point of minimalism, but still provides all the basic controls required.
Scanning speeds were fairly decent and overheads fairly light, at least with the default archive-handling settings, but resource use was around average. Our set of tasks completed very quickly, but both this and our simple lag measure will have been affected by the absence of full on read protection by default. There were a couple of minor issues, including a scan freezing up and another failing to cover all the areas asked of it.
Detection was pretty strong in the reactive sets, with no proactive figures due to the product’s inability to function without access to the cloud. The WildList was well covered as usual, and with no surprises in the clean sets Panda gets back on track with another VB100 award.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 99.90%
False positives: 0
Stability: Stable
Main version: 1.0.0.40
Update versions: N/A
Last 6 tests: 3 passed, 0 failed, 3 no entry
Last 12 tests: 3 passed, 0 failed, 9 no entry
With a few entries under its belt, PCPitstop has a good record of passes, but has presented us with a range of problems thanks to a rather unusual approach. However, these issues seem to be easing as the product matures rapidly. The latest improvement, a whitelisting system, comes into play on access only. We were not able to look at it in much depth during this test, but we hope to be able to exercise it properly next time.
The installation system takes a fair few clicks but completes rapidly, although the full set up time is extended thanks to a rather lengthy process of adding on full anti-malware functionality provided by the ThreatTrack engine. The interface looks rather different from most anti-malware products, focusing on other features such as optimization tools, but is becoming easier to use as new controls are added.
Scanning speeds were very slow, perhaps in part thanks to those other features, and overheads were pretty heavy in some sets, at least in the initial runs, the warm runs proving much faster. RAM use was a touch above average, but CPU use was very low indeed, hinting that much idle time occurred during the slightly long time taken to finish off our set of tasks.
Detection for the on-demand scanner uses only the ThreatTrack engine component, and scores were very good, especially in the reactive sets, but getting hold of these numbers was a little tricky thanks to a number of problems with logging. We also saw a few issues with scans failing or freezing, and noted the shield component failing to activate on some occasions, requiring a reboot to kick it back into life.
The core certification sets were handled well though, and PCPitstop earns a VB100 award. We look forward to examining the whitelisting component more closely in a future test.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Fair
Main version: 4.2.2.4100 (x64)
Update versions: N/A
Last 6 tests: 4 passed, 1 failed, 1 no entry
Last 12 tests: 7 passed, 2 failed, 3 no entry
Qihoo has become a fairly regular participant in our testing, and has achieved a fairly steady pass rate over the last few years.
This month, installation was speedy, as were updates, but some required a restart to finish off. The product looks bright and cheerful, with a simple layout and mostly clear options, although some areas could benefit from some proofreading.
Scanning speeds were decidedly slow over archives and binaries, but decent elsewhere. With once again no on-read protection provided, file access lag times look very fast, and resource use very low, and our sets of tasks ran through reasonably speedily. There were a few wobbles, including a scan failing to cover all the areas it was directed to, and a single blue screen incident which appeared to be connected to the log viewer utility.
Detection, powered by Bitdefender as usual, was very strong in the reactive sets, a little lower in the proactive sets, with no problems in the WildList or clean sets and a VB100 award is duly granted.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Fair
Main version: 15.00(8.0.0.1)
Update versions: N/A
Last 6 tests: 5 passed, 0 failed, 1 no entry
Last 12 tests: 8 passed, 2 failed, 2 no entry
Our test history for Quick Heal products dates back to 2002, and they have built up several strings of consecutive passes over the years, embarking on another of late.
The installation process requires a minimal number of clicks, completing in good time with no reboots required. The interface has been made over recently with boxy Windows 8 styling and looks pleasant, offering all manner of controls clearly and logically.
Scanning was fairly speedy, but a little slower over archives – where defaults are fairly shallow, but cover a wide range of archive types. On access, lag times were not bad at all, and although RAM use was fairly high, CPU use was low and our set of activities ran through rapidly.
Detection was a little disappointing, especially in the proactive part of the RAP sets, but the WildList and clean sets were well handled and a VB100 award is earned.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Stable
Main version: 2.5.0.23
Update versions: 13.3.21.1/513921.2013102315/7.51073/10202731.20131023, 13.3.21.1/515305.2013111419/7.51457/10428887.20131114, 13.3.21.1/515525.2013111920/7.51564/10470280.20131119, 13.3.21.1/515891.2013112718/7.51776/10512988.20131127,
Last 6 tests: 4 passed, 0 failed, 2 no entry
Last 12 tests: 5 passed, 2 failed, 5 no entry
Roboscan is very similar to ESTsoft’s ALYac product, and after a shaky start in our tests, it has been doing pretty nicely lately.
The set-up process is a little slow, mainly thanks to updating, and the GUI has its quirks but is reasonably simple to navigate with a decent range of configuration options.
Scanning was pretty quick, especially in the warm runs, while overheads and resource use were both very light, and our set of activities ran through quickly. We noted a few fairly trivial bugs, including some wonky text in some dialogs and a rather severe lagginess in the log viewer.
Detection was excellent though, with a good score in all test sets. This extended to the certification sets, earning Roboscan another VB100 award without fuss.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0 Stability Stable
Main version: 10.2
Update versions: 10.2.9/3.4.7.3/4.93G, 10.3/10.3.1/3.48.0/4.94G, 10.3/10.3.1/3.48.0/4.95G, 10.3/10.3.1/3.48.0/4.95G
Last 6 tests: 6 passed, 0 failed, 0 no entry
Last 12 tests: 11 passed, 1 failed, 0 no entry
Sophos is another of our most regular participants, having missed only two tests since VB100 testing began. It usually performs well, and has had only three fails in the last five years.
The product’s set-up process conforms to expected standards and completes in good time, with updates particularly fast. The interface remains largely unchanged after many years, presenting a simple and functional front end with highly detailed configuration available.
Scanning was a little slow initially over archives, which are well covered by default, but faster elsewhere and much speedier in the warm runs. Overheads were a little heavy, better in the warm runs but a fair notch higher with the settings turned up to include non standard file extensions. Stability was impeccable throughout.
Detection was excellent in the reactive sets where cloud look ups were available, dropping fairly sharply into the proactive sets as one might expect. The certification sets provided no shocks either, with VB100 certification easily achieved.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Solid
Main version: 7.5.129.18
Update versions: N/A
Last 6 tests: 2 passed, 0 failed, 4 no entry
Last 12 tests: 4 passed, 2 failed, 6 no entry
An interesting chance to compare and contrast, SPAMfighter has been a regular in our tests for some years but the underlying engine, provided via fellow regular Preventon, has recently switched from the now-defunct VirusBuster to Sophos.
Installation is pretty speedy, and updates very quick indeed. The product interface is little changed from previous iterations, making it fairly easy to find the basic set of controls on offer.
Scanning speeds weren’t bad, a little slower over archives, while overheads were mostly reasonable but slow over binaries. Resource use was on the low side, while our set of activities proceeded at a reasonable rate. There were a few issues with the on-access scanner component, which seemed not to function at first on several installs and needed a reboot to get moving.
Detection rates were noticeably lower than that of Sophos’s own product thanks to the absence of those cloud look ups, but they were fairly similar in the proactive test where this feature is absent. There were no problems in the WildList or clean sets though, and a VB100 award is easily earned, continuing a decent record for SPAMfighter.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 99.86%
False positives: 0
Stability: Stable
Main version: 8.5.24987.9501
Update versions: N/A
Last 6 tests: 5 passed, 0 failed, 1 no entry
Last 12 tests: 8 passed, 0 failed, 4 no entry
Tencent has become something of a regular fixture on the VB100 test bench over the last year or so, and has racked up a solid run of passes already.
The installer is very fast indeed, but updates are rather slow, perhaps thanks to the distance between our test lab and the product’s home region. The interface is another Chinese only affair, which again is operated with the help of a custom guide document, and it has the cluttered feel that seems fairly typical of Chinese software.
Scanning speeds were OK, and overheads very light thanks to there being no on-read protection – again, something which seems to be the norm for the region from which the product hails. RAM use was below average, CPU use very low indeed, but our set of tasks did take a little long to get through. Stability issues were limited to a couple of large scans failing to complete gracefully.
Detection was a little down in the reactive sets, but the proactive scores were well up with other products powered by the same Avira engine, and the certification sets were properly dealt with, earning Tencent another VB100 award.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Stable
Main version: 7.0.5.1
Update versions: 3.9.2574.2/22644, 23478, 23380, 23802
Last 6 tests: 5 passed, 0 failed, 1 no entry
Last 12 tests: 7 passed, 1 failed, 4 no entry
Over the years, ThreatTrack’s VIPRE has appeared in our tests under three different company names – previously having sported the Sunbelt and GFI labels. It has notched up a good record of passes, particularly in the last year or so.
This month, the installation process wasn’t the fastest we saw, and updates took a fair time too. The interface has had a few overhauls recently, but retains much the same look and feel – which is crisp and slick, with a limited set of tuning options.
Scanning was a little slow at first, improving in the warm runs but not over archives. Overheads were a little heavy, resource use low but on our set of tasks took a fair amount of time to complete. As usual, we saw a few scan jobs failing, and also a full program crash when trying to view the ‘About’ dialog.
Detection was superb in the reactive sets, not quite so good in the proactive sets, and the WildList and clean sets were dealt with appropriately, thus earning ThreatTrack another VB100 award.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Stable
Main version: 5.0.6.1002
Update versions: 5.0.5/12.163, 5.0.7.1112 5.0.5/12.163, 5.0.7.1112 5.0.5/12.163, 5.0.7.1121 5.0.5/23802
Last 6 tests: 5 passed, 0 failed, 1 no entry
Last 12 tests: 9 passed, 0 failed, 3 no entry
Total Defense’s business solution is another regular fixture in our tests, rarely missing a VB100 comparative. Since undergoing a major revamp not too long ago, the product has been offering protection based on the Bitdefender engine controlled by a cloud-based management system.
Thanks to much practice, delving into that system to find the product install package has become fairly simple, and installation itself is fairly quick, with the later updating and activation stages taking rather longer. Configuration is performed partly using the cloud service and partly using a local web console. Once again, practice has proven vital in finding and operating the appropriate controls. Things generally worked OK, but we did come across a few errors in displaying pages and updating.
Scanning speeds were OK, and very fast over archives, but these are not examined internally by default. On access overheads were a little on the high side, with RAM use a tiny bit above average, but CPU use was very low and our set of tasks ran through quickly.
Detection was excellent, and there were no issues in the core sets, easily earning Total Defense a VB100 award for its business offering.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Stable
Main version: 9.0.0.28
Update versions: 2.0.05353/6503.000/2.0.1138/2.011.0.40008.0, 6524.0.0.0, 6527.0.0.0, 6530.0.0.0
Last 6 tests: 0 passed, 3 failed, 3 no entry
Last 12 tests: 2 passed, 4 failed, 6 no entry
The consumer offering from Total Defense is a rather different beast altogether, using the vendor’s own in house engine. Its performance in our tests has been rather unreliable lately.
Set-up was quick, but updates were a little slow. We have grown used to the interface – which felt a little too styled and groovy when we first saw it – and a decent set of control options are provided that are relatively easy to find.
Scanning speeds were fairly quick, and much faster in the warm runs, while overheads and resource use were both a little high. However, a decent time was recorded for our set of activities. We experienced a few stability problems, mostly under heavy pressure, with scans crashing out and on occasion reporting completion with no threats found, despite many reports having been flagged up earlier in the scan.
Detection was rather poor in the reactive sets, and no figure is recorded for the proactive part thanks to the product’s reliance on cloud look ups. There were also problems in the certification sets, with both a handful of misses in the WildList set and a false positive in the clean set. As a result, Total Defense’s consumer solution is denied a VB100 award this month, continuing a difficult spell.
ItW Std: 98.64%
ItW Std (o/a): 98.64%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 1
Stability: Stable
Main version: 14.0.0.5245
Update versions: 14.0.15248
Last 6 tests: 5 passed, 0 failed, 1 no entry
Last 12 tests: 8 passed, 2 failed, 2 no entry
TrustPort’s products have been a regular feature in the VB100 tests since 2006, and for the most part they have performed well.
The installation process is rather heavy on user clicks but doesn’t take too long. The interface is a little disparate, again requiring a fair few clicks to navigate around, but with a little familiarization a good set of controls can be found.
Scanning was not the fastest, and overheads were a little heavy at first, but things sped up nicely in the warm runs. RAM use was fairly low, CPU use fairly high, and our set of tasks didn’t take too long to get through. There were no stability problems to report.
Detection was excellent from the dual-engined product – near perfect in the reactive sets and highly commendable in the proactive ones. There were no issues in the certification sets, and TrustPort earns another VB100 award, continuing a nice run of passes.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Solid
Main version: 1.027.232
Update versions: N/A
Last 6 tests: 1 passed, 0 failed, 0 no entry
Last 12 tests: 1 passed, 0 failed, 0 no entry
Making its VB100 debut this month, Wontok’s solution combines engines from Bitdefender and Avira, both popular inclusions in OEM products and both regular high performers, so we expected good things here.
The installer is very simple and fast, with updates a little more sedate. The interface is attractive and professional looking, with a standard layout of large icons for each main function accompanied by more detail on the selected section and related options. It proved easy to navigate and mostly responsive. Under pressure from scanning large sets of samples, we had a few issues with scans crashing and some logging failures. Although no single issue was very serious on its own, the product’s stability rating was knocked down by weight of numbers.
Detection was very impressive, thanks to the selection of engines, though not significantly better than either one on its own. The WildList and clean sets were dealt with appropriately, and Wontok earns its first VB100 award on its first attempt.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Fair
(Click for a larger version of the table)
(Click for a larger version of the table)
(Click for a larger version of the table)
(Click for a larger version of the table)
(Click for a larger version of the table)
(Click for a larger version of the table)
(Click for a larger version of the table)
(Click for a larger version of the table)
File access lag v. Defender graph 1.
File access lag v. Defender graph 2.
(Click for a larger version of the table)
(Click for a larger version of the table)
(Click for a larger version of the table)
(Click for a larger version of the table)
Performance measures v. Defender graph 1.
Performance measures v. Defender graph 2.
(Click for a larger version of the table)
(Click for a larger version of the table)
(Click for a larger version of the table)
(Click for a larger version of the table)
(Click for a larger version of the table)
(Click for a larger version of the table)
(Click for a larger version of the table)
(Click for a larger version of the chart)
(Click for a larger version of the chart)
This was an interesting excursion to an all-new platform – but Windows 8.1 failed to persuade any of the team that it was time to upgrade, and for the most part the products available supported that position. We saw a lot of minor irritations here, including a number of products whose basic functionality was slightly broken, as well as some major issues, with a larger number of blue screens than one would expect, even from a test of this size.
There were a number of products that failed through failing to detect all of the WildList samples – which is unexpected these days – but there were very few false positives. A couple of products were unable to provide usable on-access results, and would have been denied an award for this reason, had there not already been other reasons not to grant certification.
On the plus side, a good number of products performed very well and earned excellent ratings for stability, as well as for detection and general performance. Hopefully our ratings will help persuade those lagging behind to better their game.
Next up will be a Linux comparative, for which we have a far more manageable field of participants. Hopefully that should help us get our schedule back on track ready for the upcoming Windows 7 test, which we expect to be even bigger than this one – and with any luck will run more smoothly.
Test environment. All tests were run on identical systems with AMD A6-3670K Quad Core 2.7GHz processors, 4GB DUAL DDR3 1600MHz RAM, dual 500GB and 1TB SATA hard drives and gigabit networking, running Microsoft Windows 8.1 Pro, 64-bit edition.
Any developers interested in submitting products for VB's comparative reviews, or anyone with any comments or suggestions on the test methodology, should contact [email protected]. The current schedule for the publication of VB comparative reviews can be found at http://www.virusbtn.com/vb100/about/schedule.xml.
