2013-07-08
Abstract
The VB100 test team pays its first visit to the Windows Server 2012 platform - and finds things generally rather slow and awkward, with a disappointing number of products displaying stability issues. John Hawes has all the details.
Copyright © 2013 Virus Bulletin
This time of year is traditionally a busy one, with a number of conferences and holidays cutting into precious lab time. This month also saw our first look at a fairly new platform, and with a wide range of products almost certain to show off a few interesting quirks and teething problems, there seemed little chance of a smooth ride through our demanding suite of tests – nevertheless, we remained hopeful.
In addition to the new platform, we took delivery of a new set of hardware just in time for this test, increasing the number of machines available and thus reducing the bottlenecks caused by misbehaving or just plain slow products. There remained, of course, the human bottleneck, with only so many hours in the day and far too few hands to move things along as quickly as we would like. Thus, as seems to be the norm these days, the final touches were still being put to this report as we started to prepare for the next test. In future we may have to make some adjustments either to the range of measures we present, or the frequency of tests; for now, the impact of our heavy workload will be mainly evident in our taking a very strict stance on poorly performing products.
Any product that was found to be severely impaired or unreliable was rapidly excised from the list of participants this month. Although all of these entrants will be listed at the end of the review, one major impact of this move will likely be to raise the overall standard in the ‘Stability’ measures – with those products that in the past would have been tenderly nursed through testing with a ‘Buggy’ or ‘Flaky’ rating simply thrown on the scrapheap this month. We hope to have more time to look more closely at these products in the next test.
Windows Server 2012 was released last summer alongside Windows 8. It boasts a range of improvements, including a new file system, although for this initial test we stuck with trusty old NTFS; the new platform itself seemed enough of a challenge for our participants, without the added stress of a new file system. The set up process held one major surprise: the default seems to be to install with no GUI – the so-called ‘Server Core’ mode is presumably operated entirely from the command line.
I say ‘presumably’ because we had to opt for the GUI version, since few (if any) of the products submitted seemed likely to offer command-line-only installation and operation. This, we found, meant a return to the rather uncomfortable world of the Windows 8 desktop – awkward and clumsy enough in a normal desktop setting and almost entirely ridiculous on a server; the option to go headless suddenly made a lot more sense.
Leaving the systems set up with just the basics of the OS and a few tools (assuming any additional dependencies could be resolved on a per-product basis), we moved onto the test sets, which provided fewer surprises. The main sets were built using the normal system, around a test deadline of 17 April. This was used as the cut off for the WildList sets, and a new pair of lists (the March lists) were released on that very day. The bulk of testing did not commence until early May, though.
Our standard clean sets saw a little pruning of older material and a fair-sized expansion with new items – mainly business tools to fit with the server environment. The final clean sets weighed in at 820,000 files, just under 200GB of data.
Unfortunately, shortly after testing commenced, we spotted an issue with our sample storage system – the result of human error when adding extra storage space. This meant that some samples were erroneously being classed as new, despite having already been entered into the system. Fortunately we caught this in time to rectify things for the Response sets, but the RAP sets were polluted by the misclassified samples, and much work went into trying to tidy them up. However, as the end of the available time neared, the process was still incomplete, and as we were also having problems pushing a number of products through the unusually large sets, we decided to remove the RAP test from this month’s report. We hope to be able to add the data at some point, should it prove possible to do so.
The deadline day saw just over 40 products submitted, with a couple of last-minute surprises and several which professed an inability to install with web access, either for licensing purposes or simply because no monolithic installer file was available. With those prepped just in time on the deadline day, we were ready to see what delights the new platform would bring.
Main version: 8.0.1481
Update versions: 130417-1, 130513-0, 130520-0, 130529-0
Avast’s server edition seemed pretty indistinguishable from the company’s usual desktop offerings: pretty easy on the eye, laden with multiple components and layers, and relatively simple to navigate even with a very good range of controls. Stability was an issue from the off though, with updates failing repeatedly. Eventually the developers informed us of an issue with connecting to old update servers, the addresses of which were apparently hard-coded into the version sent in for testing – a quick hack was required to fix things.
With that done, updates proved much more efficient, and working installs could be completed in a little over two minutes, with not too much to click through and no reboots required. Scanning speeds were rather slower than we have come to expect from Avast, while overheads as ever seemed very light thanks to the lack of on-read protection by default; with this enabled, things were considerably slower. RAM use was low, CPU use fairly high, and our set of activities took rather a long time to complete.
Detection was pretty reasonable in the Response sets. There were no problems in the certification sets and a VB100 award is earned, putting Avast on five passes and one fail in the last six tests; ten passes and two fails in the last two years. The fairly serious issues with the update process made a heavy dent in the product’s stability score, earning it only a ‘Fair’ rating.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Fair
Main version: 2013.0.3272
Update versions: 3162/6238, 3162/6305, 2013.0.3336 3162/6340, 2013.0.3343 3184/6363
There was no special server product from AVG, just the usual ‘Business’ offering, featuring a Windows 8-style tiled interface which is fairly clear and usable, and which provides a good range of controls. Set up is fairly quick and simple, offering an ‘Express’ install mode but with a handful of clicks required either side, including the offer of a toolbar. Updates failed to complete properly on one occasion, but the set up process was generally fairly fast, with no need to reboot.
Scanning speeds were pretty zippy to start with and much faster in the warm runs; overheads were not bad initially and again improved greatly for the warm measures. Resource use was very low, and our set of tasks got through in reasonable time.
Detection was a little harder to measure, as the logging system was very unreliable, with scans frequently reporting large numbers of detections but unable to provide any more information. Even when logs could be displayed, it was difficult to get them to export properly to disk. On a server platform, logging is even more important than on a desktop, so we weight this kind of issue more heavily in our server tests.
Eventually, after much fiddling around, we managed to get some usable numbers, which showed some very good scores in the Response sets. There were no issues in the WildList sets, but in the clean sets a single item from NVidia was identified as malware. The issue was caused by an unpacking fault, with the unpacking algorithm dropping a corrupted file which was then alerted on heuristically. This bug was quickly spotted and corrected, but was enough to deny AVG a VB100 award this month. The vendor now has five passes and one fail in the last six tests; nine passes and three fails in the last two years. The logging issues meant only a ‘Fair’ stability rating.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 1
Stability: Fair
Main version: 13.0.0.3185
Update versions: 8.02.12.28/7.11.73.42, 13.0.0.3640 8.02.12.34/7.11.76.206, 8.02.12.44/7.11.79.86, 8.02.12.50/7.11.81.51
Avira routinely provides a fully fledged server solution for our server tests, with the interface based on the MMC platform. Installation was very simple and rapid, completing in under a minute with only a handful of clicks required. However, at the end of it we were rather surprised to find the machine rebooting, without any checks to make sure that it was OK with the user. This seemed a little on the insensitive side – restarting a server is a rather more serious proposition than restarting a desktop machine, which under Windows one expects to have to reboot a few times per day at least.
Scanning speeds were not too bad, but not super fast either, lag times were on the light side though. RAM and CPU use were both very low, and our set of activities didn’t take much time to complete. Detection was very impressive across the board, as ever, and with no issues in the certification sets a VB100 award is well deserved.
Avira now has four passes from four entries in the last six tests, having had some rare absences lately, and ten passes in the last two years. There were a few issues this month, notably the surprise reboot and some problems exporting logs, earning it only a ‘Fair’ stability rating.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Fair
Main version: 4.1.26.5
Update versions: 7.46702 (9603247), 7.47114 (9697704), 7.47140 (9691305), 7.47621 (9596775)
In the past we’ve also seen MMC-based server solutions from Bitdefender, but this month we got something new: an endpoint client presumably designed to be controlled remotely, as there seemed very little by way of local interface. The initial install was very rapid, but updating was a little unclear, with the ‘about’ dialog implying that the process was taking a very long time, until we realized it had in fact completed and was just re-checking very frequently.
Scanning speeds were very good, but overheads were rather heavy, particularly in the archive sets. Resource use was low though, and again our set of activities zipped through quite quickly. Detection was excellent in all sets, including the certification sets where no issues were observed. A VB100 award was thus easily earned, maintaining a flawless record for the vendor, with 12 passes in the last 12 tests, and boding well for the raft of other products using Bitdefender’s engine. A single incident was noted, when an overnight scan froze part way through, and the product earns a ‘Stable’ rating.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Stable
Main version: 13.0.258.5
Update versions: 7.46702, 7.47114, 7.47445, 7.47621
The first of that raft of products using the Bitdefender engine, BullGuard’s GUI is familiar, clean and simple, and reasonably easy to use, with fairly limited options. The set up process is very quick and easy, with updates a little slow but the whole process completed in three to four minutes with no need to reboot.
Scanning speeds were reasonable to start with, and lightning fast in the warm runs, and overheads were pretty light. Resource use was very low, with a great speed through our set of activities too. Detection was superb, as expected, and the WildList and clean sets were handled well, earning BullGuard a VB100 award. The vendor remains on 10 passes from 10 entries in the last two years, having skipped our annual Linux tests. Stability was impeccable, earning this month’s first ‘Solid’ rating.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Solid
Main version: 11.0.000.504/8.3.0.18
Update versions: 1112773440, 1118263648, 1119174816, 1114155136
Check Point was a rather unexpected last-minute arrival in this test – the vendor usually only submits products to the desktop tests. The ZoneAlarm product isn’t too fiddly to install, but takes a few minutes to complete updates before much can be done with the GUI. This is starting to look a little faded but is reasonably navigable, providing a pretty basic set of controls. While it mostly seemed responsive, on occasions it slowed to a crawl and even greyed out entirely, collapsing under any sort of pressure including our on-access tests and any scan of more than a handful of samples. In most cases it carried on working in the background though, maintaining protection and continuing to scan long after the GUI had locked up.
Scanning speeds were pretty slow initially, but much better in the warm runs thanks to some intelligent fingerprinting. Detection was somewhat mediocre – while the clean sets only yielded a few warnings of remote admin tools and the like, and the standard WildList was well covered, in the Extended list several items were ignored on demand, and even more on access. This means that there is no VB100 award for Check Point this time.
Not the most regular participant in our tests, Check Point now has one pass and one fail in the last year. There were some issues with the stability of the GUI this month, and the product earns only a ‘Fair’ rating.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 98.89%
ItW Extd (o/a): 97.53%
False positives: 0
Stability: Fair
Main version: 5.1.23/5.4.2
Update versions: 201304172349, 201305081049, 201305210920, 201305281529
A much more regular participant in our tests, Commtouch has suffered a rather extended run of bad luck lately. The install is fast and simple, the GUI basic but generally responsive, with a reasonable set of basic controls provided. We saw a few GUI crashes when exporting logs, and also while trying to scan sets of clean files for our speed measures; at first we thought it would be impossible to complete a simple scan of the C: partition, but after several attempts we finally got a clean run.
Scanning speeds were fairly consistently slow, and on access overheads pretty heavy. Resource use was low, but our set of tasks took an enormously long time to get through. Detection was not too bad, with good coverage of the WildList sets, but in the clean sets a number of false alarms were raised, including several items from leading printer manufacturers, which were added to our clean sets late last year.
So, there is no VB100 award for Commtouch once again, leaving the vendor on one pass and four fails in the last six tests; three passes and seven fails in the last two years. With a number of wobbles seen too, stability is rated only ‘Fair’.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 8
Stability: Fair
Main version: 7.0.0.21
Update versions: 13161238, 13841290, 14012011, 13538855
Having recently hopped on the Bitdefender bandwagon, Emsisoft has been doing pretty well in our tests lately. The product is friendly and simple, with not too much by way of fine-tuning but covering the basics well. Installing and updating was zippy, with no reboots needed, and scanning speeds were not too bad either. Overheads looked light, but this was helped by only partial on-read scanning by default. Resource use was low, impact on our set of tasks a little above average.
Detection was solid, with good scores throughout. The WildList sets were dealt with well and there were no problems in the clean sets, thus Emsisoft earns a VB100 award. That puts Emsisoft on three passes and two fails in the last six tests; four passes and four fails in the last two years – although much of that history reflects the performance of a different engine. There were no issues with stability, and the product earns a ‘Solid’ rating.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Solid
Main version: 14.0.1400.1384 DB
Update versions: NA
We’ve had a few chances to look at eScan’s recently redesigned interface, and it seems to get murkier each time, with its grey text on a darker grey background proving difficult to make out in England’s gloomy summertime. We also had some serious stability issues with it in the last test, and hoped for better this time.
Set-up is a rather slow process, and updates take even longer, with one run sticking on around 80%, and getting nowhere after an hour. Killing the process and restarting it proved more successful, but it still took a while.
Scanning speeds were good over archives until settings were turned up a little, but were slow elsewhere, speeding up a little in the warm runs. Overheads were medium initially, but again showed some improvement after the initial runs. Resource use was fairly low, and our set of tasks got through in decent time.
Detection was good, with no problems in the clean sets, and the WildLists were dealt with well, thus eScan earns a VB100 award, maintaining an excellent record of 12 passes in the last two years. There were a few issues with the updater, plus a scan which froze up, but a ‘Stable’ rating is just about merited.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Stable
Main version: 5.0.2126.0
Update versions: 8238, 8256, 8375, 8422
ESET provided a corporate solution this month, although not a full server one, as the installer pointed out (indicating that one is available and may be better suited to our environment). Otherwise the set up was simple and uneventful, completing in a couple of minutes with no reboot needed.
The interface is similar to that of the vendor’s usual consumer offerings: slick and stylish with an excellent level of configuration available. For the most part it responded well, although we did spot one crash and one incident where a log failed to fully export (although it was fine on a second attempt). Scanning speeds were a little slow initially but much faster in the warm runs, while overheads were very light on access. Resource usage was very low, and our set of tasks got through a little slowly, but not drastically so.
Detection was splendid, with good scores everywhere, and no problems in the WildList or clean sets. That performance earns ESET yet another VB100 award, putting it on a total of 80 passes overall since 1998, and with no fails, or even missed tests, in the decade since June 2003. With only a few minor GUI wobbles, a ‘Stable’ rating is earned.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Stable
Main version: 2.5.0.23
Update versions: 13.3.21.1/49001.2013041721/7.46702/9603247.20130417, 12.3.21.1/494062.2013050718/7.47094/9693405.20130507, 13.3.21.1/495616.2013051618/7.47426/9624744.20130520, 13.3.21.1/497179.2013052817/7.47608/9612810.20130528
ESTsoft was excluded from the last comparative due to extreme instability, so it was with no little trepidation that we tried it out once again. Things were somewhat better this time, for which we were very grateful. There were still a few issues though, including a message, indicating that an initial update was needed, that was so badly misaligned it was barely readable. The update process was slow, making for a rather lengthy install time.
Scanning speeds were rather slow over archives but reasonable elsewhere, especially in the warm runs, while overheads started a little high but also showed much improvement. Again RAM and CPU use were low, and time taken to complete our set of standard activities not bad.
Detection was harder to measure, with many scans falling over or freezing, and the log saving process was slow and unreliable. We got what data we could, which eventually showed decent scores in the Response sets.
The WildList sets were properly handled, and with no false alarms, a VB100 award is just about earned. ESTsoft now has one pass and two fails from the last six tests, having been excluded from some others; four passes and two fails in the last two years. There were a number of issues with stability once again this time, but only when handling large sets of malware, so hopefully most real world users should be spared the worst; a ‘Fair’ rating is awarded.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Fair
Main version: 5.0.2.225
Update versions: 5.043/17.474, 17.594, 17.658, 17.695
Fortinet’s FortiClient had a major overhaul recently, with the simple and clear interface we knew and loved replaced by a browser-driven, configuration-free new look which is doubtless fine for customers, but gets little love from the lab team. Setting it up is fast and simple though, and with barely any controls, operation is not too tricky.
Scanning speeds were mostly decent, but a little slow over binaries, while overheads were a little sluggish in the cold runs, but fine after that. Resource use was a little higher than many, but our set of tasks didn’t take long to get through.
Detection was very good indeed, with good scores everywhere and a flawless showing in the clean and WildList sets, thus easily earning a VB100 award. Missing only our Linux tests, Fortinet has ten passes from ten entries in the last two years. The product earns a ‘Stable’ rating for stability.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Stable
Main version: 9.50 build 19092
Update versions: NA
F-Secure entered a full server solution, with a web-based GUI. The installation process is a little drawn out, but looks serious and professional. It takes a few minutes to get set up, and when the main process is complete one assumes it’s all good to go. This fooled us several times, but we eventually learned to keep an eye on a ‘downloads’ tab hidden away deep in the GUI, which indicated when all the components had been pulled down and were in place. Protection is not fully operational until this process is complete, which could take quite some time, but there seems to be little indication that things are other than perfect.
The interface is a little clunky, and limited by the browser setting, but provides a reasonable set of controls. Scanning speeds were blindingly fast to start with, and somehow managed to speed up further in the warm runs – initial speeds were doubtless helped by this being one of few products still using an extension list to decide what to scan. Unlike some of the firm’s desktop products, more thorough settings are available. On access overheads were helped by the same approach, and proved excellent. RAM use was OK, CPU use barely noticeable, but our set of tasks took a fair while to get through.
Detection was solid, with no issues in the certification sets and a VB100 award was easily earned. F-Secure’s test history shows five passes and one fail in the last six tests; seven passes and two fails in the last two years. As usual, logging was a big issue, with data frequently missing, inaccurate or incomplete, and this made a heavy dent in the product’s stability score, leaving it only just inside the ‘Fair’ category.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Fair
Main version: 12.0.0.222/AVA 22.9091/AVL 22.1657
Update versions: 12.0.0.222/AVA 22.9513/AVL 22.1693, AVA 22.9771/AVL 22.1717, AVA 22.9965/AVL 22.1739
Another full server solution, with a separate management component from which the client is deployed and controlled. Set-up takes a while, mainly thanks to the SQL components needed to manage data, and a reboot is required at the end – but this does not appear to be needed if simply deploying a client. Oddly, on one occasion a client was installed in Polish.
The management interface is pretty detailed, but not too hard to find one’s way around, and with some practice actually comes to feel fairly pleasant to use. Much control can also be ceded to clients. Logging was an issue once again, with very large logs repeatedly failing to export properly, but splitting the jobs into many small chunks provided a workaround, if at a high cost in time and effort. We also saw a scan freeze overnight, part-way through our clean sets, which wasted more valuable time.
Scanning speeds were tricky to measure given the lack of a standard GUI, but with some fiddling around we managed to run our set of jobs, with some rather slow times initially but the usual blindingly fast warm runs. Overheads were likewise rather high but improving. Resource use was fairly reasonable, but completion of our set of tasks took quite a hit.
Detection was awesome, as ever, and the core sets were brushed effortlessly aside, easily earning G Data a VB100 award. The vendor has four passes from four entries in the last six tests; nine from nine in the last two years. Several, mostly minor stability issues were seen, just tipping the stability score into the ‘Fair’ category.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0 Stability: Fair
Main version: 2.2.14
Update versions: 1.4.0/83948, 84106, 84200, 84269
The Ikarus product seems to have changed only minimally over the years, but it works reasonably well and isn’t too tricky to operate. The set up is rather convoluted, and there were some issues on this month’s platform thanks to the need to install the .NET framework. It attempted to do this itself, then demanded a reboot part-way through; on restarting, the same process was tried again, and once again demanded a reboot. We quickly put a stop to this nonsense, and installed .NET in the only viable way: through the server management facility.
With this step completed in advance the install is actually quite speedy, and updates are very fast too. The GUI provides decent controls and seemed reliable throughout. Scanning speeds were decidedly slow, and overheads rather heavy, with average RAM use, slightly high CPU use and a quite remarkably low impact on our set of tasks.
Detection rates were very impressive, with perfect coverage of the WildList, but the clean sets always seem to be Ikarus’s Achilles heel, and once again a few items were alerted on, including two Virut detections. So there is no VB100 award for Ikarus once again, putting it on one pass and four fails in the last six tests; three passes and six fails in the last two years. Stability was good though – not quite flawless, but well up in the ‘Stable’ category.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 3
Stability: Stable
Main version: 10.1.0.867
Update versions: NA
Kaspersky’s product line is also familiar, although much more recently refreshed, with a crisp and zingy look. Initial set up is fast, but updates take a fair while. The interface provides an awesome depth of fine-tuning, and is for the most part clear and simple to navigate – although some things that one might expect to find together turn out to be in completely different areas. On a few occasions, the on access component seemed to take some time to come fully online: test runs that were started too soon after installation showed unexpected misses, which would then not recur after things had had some time to settle down.
Initial scanning speeds were a little slow, but warm scans were very fast indeed, with overheads showing a similar pattern, improving hugely in the warm runs from a decent start. RAM use was low, CPU use below average, and impact on our set of tasks perhaps a little on the heavy side.
Detection was pretty decent, but in the WildList sets a number of samples were missed, rather surprisingly, with no explanation available at the time of going to press. So no VB100 award is earned this time, putting Kaspersky on five passes and one fail in the last six tests; eight passes and three fails in the last two years. Stability was not bad, putting the product comfortably in the ‘Stable’ range.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 99.76%
ItW Extd (o/a): 98.75%
False positives: 0
Stability: Stable
Main version: 204636763
Update versions: NA
Kingsoft’s product came this month in Chinese only, which meant we had to start by installing some language packs to give us the best chance of figuring out what was going on. The set up is very funky, with a kind of thermometer-style progress bar zipping through; sadly the effect is somewhat hampered by a lengthy pause after its initially whizzy special effects. The GUI looks detailed and involved, although much of it was lost on us.
There were some very noticeable slowdowns operating the system with the product in place, and on a few occasions scans seemed to fail to complete the job assigned to them. Otherwise things moved along OK, with pretty slow scanning speeds, fairly hefty overheads, rather high resource use, but not too crazy an impact on our set of activities.
Detection, provided by Avira, was splendid, and there were no issues in the WildList or clean sets, earning Kingsoft a VB100 award. That puts the vendor on three passes from three entries since its re-emergence with a third party engine – previous entries under its own steam are now well in the past. Some wobbles were noticed, pushing the product just over the edge into a ‘Fair’ rating.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Fair
Main version: 4.2.223.0
Update versions: 1.1.9402.0/1.147.1649.0, 1.1.9402.0/1.149.1522.0, 1.1.9506.0/1.151.563.0, 1.1.9506.0/1.151.1179.0
Microsoft’s corporate offering has recently been renamed, but it has retained much of its previous look and feel. The install package is very compact and runs through rapidly; updates are also speedy, making for a very fast install time. The interface is simple, providing basic controls, and was generally smooth and responsive. Logging seemed a little less reliable though, with the expected data not written on several occasions, and one attempt at running the RAP scan died after 24 hours, losing all progress and causing much cursing and muttering.
In the end, though, we found decent and consistent scanning speeds. Overheads were a little high initially in most areas but soon sped up; resource use was well below average, and impact on our set of tasks pretty light too. There were no problems in the core sets and a VB100 award is well deserved, putting Microsoft’s business line on two passes from two entries in the last year; four from four in the last two years. We did see some problems here, mainly with logging, but they were not too severe and a ‘Stable’ rating is earned.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Stable
Main version: 9.10
Update versions: 7.00.22, 7.01.04
Norman’s installer is another that seems to have finished in good time, but which rather sneakily leaves most of the work until after the process appears to be complete. A few minutes after you might think all was done, a reboot is requested to finalize the process. As in many previous tests, opening the GUI brings up a torrent of JavaScript error messages, which is none too confidence-inspiring, before eventually one can get at the interface. Its design is OK within the limitations of the browser set up, offering a reasonable set of basic controls. It froze several times and crashed a few more – in some cases when scanning large sets of malware samples, but also when crawling through clean sets.
Scanning speeds were not too bad, and lag times a little high in the cold runs but much better once warmed up a bit. RAM use was a tad below average, CPU use a fraction above, and our set of tasks was completed in very good time. Detection was pretty reasonable, with no problems in the WildList; in the clean sets, though, a handful of closely related samples were flagged as malicious, denying Norman a VB100 award this month. The vendor now has three passes and three fails from the last six tests; eight passes and four fails in the last two years. With a number of wobbles noted, stability is rated only ‘Fair’.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 3
Stability: Fair
Main version: NA
Update versions: NA
Panda’s Office Protection was new to us, and its set up process was fairly non-standard. It involved logging into a web page, from where a download could be unearthed, and from where most configuration seemed to be done. Fortunately, we were provided with a tool to enable local logging, as the client-side interface is very minimal indeed. Even using the web console, we found it tricky to switch protection off, among many other standard features.
Scanning speeds were rather slow, and overheads very light thanks to only partial on-read protection. The same set up affected impact on our set of activities, which was also low, as were resource use measures. Detection was really quite excellent in the Response sets. There were no issues in the certification sets, and Panda earns another VB100 award, putting it on three passes from three entries in the last six tests; four passes from four attempts since its return to our tests a little over a year ago. Only some very minor wobbles were noted, with a ‘Stable’ rating well deserved.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Stable
Main version: 4.0.0.4050
Update versions: NA
Returning to this month’s swathe of products using Bitdefender’s engine, Qihoo is another which does not provide standard on-read real time protection, instead performing some sort of delayed action scanning, reporting that it has spotted things sometimes long after they have been played around with. The interface is available in English, for the moment at least, unlike several others from China.
The GUI is reasonably clear and usable, and mostly responsive, although a few options seemed not to work – most notably the auto mode for the on access component, which seemed determined to ask what we thought about things no matter what we set it to. Scanning was slow, overheads very light thanks to the lack of proper protection, and both resource use and impact on our set of tasks were likewise low but probably skewed.
Detection was solid – a little below what we might expect, but still decent – and with no issues in the WildList sets and only a few warnings about possible archive bombs in the clean sets, a VB100 award is earned. Qihoo can boast five passes from five attempts in the last six tests; six passes and two fails in the last two years. Stability was OK, with just a few issues with obeying instructions properly and a single case of logs disappearing nudging it into the ‘Stable’ category.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Stable
Main version: 14.00 (7.0.0.4)
Update versions: NA
One of our longest-serving regulars, Quick Heal’s product is like a familiar face in the crowd. Its design is a little unusual but quickly makes sense and becomes simple to navigate, providing a good level of control.
Set up is speedy in the main, with updates downloaded in a flash and only a short pause required while they are applied. No reboots are needed.
Scanning was decidedly slow, especially over archives, but overheads were fairly light with signs of sharp improvement in the warm runs. RAM use was a little above average but CPU use was barely perceptible, and our set of tasks blasted through remarkably quickly.
Detection was not the greatest, but there were no issues in the clean sets or the WildList sets, and Quick Heal earns a VB100 award, putting it on four passes and one fail in the last six tests; seven passes and three fails in the last two years. Stability was excellent, with no problems encountered and a rare ‘Solid’ rating.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Solid
Main version: 2.5.0.23
Update versions: 13.3.21.1/49001.2013041721/7.46702/9603247.20130417, 12.3.21.1/494062.2013050718/7.47094/9693405.20130507, 13.3.21.1/495616.2013051618/7.47426/9624744.20130520, 13.3.21.1/497179.2013052817/7.47608/9612810.20130528
A clone of the ESTsoft product for the global market, Roboscan has tended to have a fraction more luck than its sister product in our tests, but was similarly sidelined last time around due to some serious instability. The set up is fairly simple but updates are slow. The interface is wordy and a little odd in places but reasonably usable, with most of the standard basic options and some other stuff besides (although exactly what it all means we’ve never been sure).
Scanning speeds were not too bad to start off with, and improved some in the warm runs, except over archive files. Overheads were a little high but also sped up considerably. Resource use was low and our set of tasks got through in good time.
Detection was decent in the Response sets but once again severe issues with both scanning and logging made gathering scores tricky – although in the end we saw some decent numbers. The WildList and clean sets were dealt with well though, and a VB100 award is granted. Roboscan now has one pass and two fails in the last year; two passes and two fails in the last two years. The issues we encountered only seemed to occur during high-stress periods, but were enough to place Roboscan in the ‘Fair’ stability range.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Fair
Main version: 7.5.62
Update versions: NA
Surprising us all with its sudden reappearance on the deadline day, SPAMfighter makes use of the Preventon SDK, which formerly included the VirusBuster engine, and has been out of our tests during the handover of that technology to its new maintainer Agnitum. The product has reappeared with a new engine under the hood, which we quickly identified as being provided by Sophos – a relatively rare choice of third-party engine in our tests. On the surface, little, if anything, has changed, with the standard install process and rapid updates completing in a few minutes with no reboot required. The interface is bouncy and friendly, and provides a reasonable set of controls, as usual crumbling under the strain of large scans but mostly reliable in everyday use.
Scanning speeds were pretty slow, overheads a little high over binaries but not bad elsewhere. RAM use was around average, CPU a little below average, and impact on our set of tasks not bad. Detection was reasonable if not stellar, and the clean sets were handled well. There were no surprises in the WildList sets either, and SPAMfighter earns a VB100 award. The vendor’s test history (mostly accumulated with a different engine of course) shows just this one pass from three entries in the past year, but six passes and two fails over the last two years. Stability wasn’t too bad, with most issues occurring only under heavy stress, but a few scans of clean items did seem to freeze up, meaning only a ‘Fair’ rating is awarded.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 99.76%
False positives: 0
Stability: Fair
Main version: 10.2.7
Update versions: 3.42.1/4.88G, 3.43.0/4.89G
Another of our most regular participants, Sophos’s product is another that has changed little with the passing of the years and the invention of ever uglier platforms on which to run. Its interface is pretty clear and sensible, with remarkably in-depth configuration provided for braver users. Installation is slick and speedy, updates also fast and easy, and operation was mostly smooth-going, although one scan did get quite badly stuck, couldn’t be cancelled and needed a reboot to get things moving again. Re-running the same task proved problem-free though, suggesting that it was just a random glitch.
Scanning speeds were a little below par, overheads mostly reasonable but fairly high in the set of binaries and system files, while resource use and impact on our set of tasks were all a little above average. Detection was pretty decent in the Response tests, with the company’s cloud look up system clearly coming into play, and the core sets presented no problems, so Sophos earns a VB100 award. Recent test history for the vendor is excellent, with six passes in the last six tests; in the longer term, things look a little shakier, with ten passes and two fails in the last two years. Just the one one-off incident means the product is given a ‘Stable’ rating.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Stable
Main version: 12.1.2015.2015
Update versions: NA
A rather rare face in our tests these days, Symantec returns with its business solution, looking much the same as it did the last time we saw it. Set up involved unpacking a large archive, but presumably this was to help us in our lab situation and real users would miss this step. The rest of the process was fairly speedy, with updates adding a few minutes and a reboot needed at the end. The interface looks bright and cheerful but has a serious side under the glossy covers, with a comprehensive set of controls.
Scanning speeds were not too bad initially, and very rapid in the warm runs; overheads were very light indeed, with RAM use a little above average, CPU use fairly high, but impact on our set of tasks not bad at all. Detection looks a little below par, but this is mainly thanks to the exclusion of a wide class of ‘non-specific’ detections, mainly suspicious and cloud-based heuristics; with these detections included, numbers would be in the high nineties across the board.
The certification sets were handled impeccably, easily earning Symantec a VB100 award, its first this year, with the previous appearance of the corporate solution around a year ago also yielding a pass, putting the vendor on two out of two in the last two years. One of our scan jobs froze up overnight, but there were no major problems, and the product earns a ‘Stable’ rating.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Stable
Main version: 7.4.24971.501
Update versions: NA
Another Chinese-language product, Tencent also uses the Avira engine. Installation and usage is fairly intuitive despite the lack of readable guidance, and the interface is very bright and colourful. Scanning speeds were a little slow, and overheads look very low but this is misleading thanks to a lack of on-read scanning. Resource use was also low, probably partly for the same reason, and our set of tasks ran through quickly.
Detection was excellent in the Response sets, with no issues in the core sets, and a VB100 award is earned, putting Tencent on four passes from four entries in the last year; five from five in the last two years. Stability was excellent, and the product earns a ‘Solid’ rating.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Solid
Main version: 6.2.1.10
Update versions: 3.9.2665.2/17582, 17944, 18248
The VIPRE product recently came under new management, but so far remains unaffected by its change of ownership – mention of the previous company is still scattered around within the product and a few folders even recall its owner before that. Installation is simple and reasonably fast, with updates very quick too, but a reboot is required at the end. The interface provides a fairly limited set of controls, but is simple to operate.
Scanning speeds were very slow indeed, but on access lag times weren’t bad. Resource use was low, but our set of activities did take some time to complete. Detection was excellent, with no issues in the core sets and a VB100 award is well deserved. VIPRE’s test history (irrespective of ownership) now shows two passes and one fail in the last six tests; six passes and two fails in the last two years. There were a number of issues getting scans to complete, with many freezing or simply failing with no logs to show for their work, meaning only a ‘Fair’ rating is awarded.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Fair
Main version: 5.0.1.0324/12.163
Update versions: 5.0.2.0504/12.163, 5.0.2.0517/12.163, 5.0.3.0524/12.163
The new product from Total Defense is a cloud-based suite, with installation and configuration involving logging into a website and deploying set up files from there. The process is not too taxing though, and after a little practice it became fairly speedy. The main local interface is also web-based, and provides some basic settings, although as yet there is no option to scan specific items (an addition due very soon, we are promised). This meant that no on demand scanning speeds could be measured, but on access lags were recorded, looking a little above average but not too intrusive. Resource use was low though, and our set of activities got through rapidly.
Detection was solid, as we would expect from the underlying Bitdefender engine, with no issues in the certification sets, thus earning this new solution its second VB100 award. Including the previous incarnation based on the old CA/VET engine, Total Defense’s business line has four passes in the last six tests; eight passes and one fail in the last two years. We saw some minor wobbles, mainly in rendering the GUI, leading to a ‘Stable’ rating.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Stable
Main version: 13.0.9.5105
Update versions: 13.0.10.5106
Dual-engine TrustPort can usually be relied upon to produce some excellent scores. The deployment is speedy but updates are a little slower; the interface is a little unusual, but reasonably simple to operate, providing some decent controls. These are not always reliable though, especially the options to expand the size of logs, which appear capped at a fairly small size whatever one tries to set them to.
Scanning speeds were pretty slow, and overheads pretty high, but these at least sped up in the warm runs. Resource use and impact on our set of tasks were all a shade below average.
Detection, as we predicted, was superb, the product demolishing our Response sets and brushing aside the core sets too, easily earning it a VB100 award. Recent test history is good, with five passes from five attempts in the last six tests; longer term things are a little shakier with seven passes and two fails over two years. There were some issues noted, including logging misbehaviour, meaning only a ‘Fair’ rating for stability.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 100.00%
False positives: 0
Stability: Fair
Main version: 3.41.0/4.87G
Update versions: 3.41.0/4.89G, 3.43.0/4.89G
Another member of the Preventon stable, VexxGuard only has a single previous entry, but is a fairly familiar face nevertheless. The product returns little changed on the surface, but as with the others from the same stable, the Sophos engine is included underneath. Set up was zippy, and updates also fast; the interface is unfussy, providing basic controls in a simple manner.
Scanning was slow in the archive and binaries sets, but not bad elsewhere, with overheads also a little high over binaries. Resource use was a tad above average, but our set of tasks completed in good time. Detection was rather mediocre but not too dismal, and the core sets were handled properly, earning VexxGuard a VB100 award on its return to the stage. Its only previous appearance, late last year, was not so successful. There were a few minor issues, but a ‘Stable’ rating is earned.
ItW Std: 100.00%
ItW Std (o/a): 100.00%
ItW Extd: 100.00%
ItW Extd (o/a): 99.76%
False positives: 0
Stability: Stable
Several additional entries were submitted for testing but found to be unsuitable: one, from Hauri, installed happily but appeared not to function at all. Another, from SmartCOP, refused to install at all, taking a dislike to the platform we were using. A few others, including FileMedic and CMC, worked a little, but not enough to provide usable test results without unsustainable efforts on our part. All were excluded from testing within the first few weeks.
(Click for a larger version of the table)
(Click for a larger version of the table)
(Click for a larger version of the table)
(Click for a larger version of the table)
(Click for a larger version of the table)
(Click for a larger version of the table)
(Click for a larger version of the table)
(Click for a larger version of the table)
(Click for a larger version of the chart)
Our overall impressions of Windows Server 2012 are a mixture of horror, dismay and disgust. The GUI is entirely unsuitable and clumsy to operate – although it might well be excellent on a tablet, I doubt that many people will be installing server-grade operating systems on swipeable devices any time soon. It also seemed sluggish, especially accessing network resources, and tended to be a little capricious and unpredictable.
Much the same can be said of most of the products included in the test. There were a few ‘Solid’ ratings for stability, but not many, with most products displaying moments of madness. Things generally felt slow and awkward, as if there was still some work to do in getting them properly optimized for this sort of platform.
With no RAP results to report (as yet), detection relies on our Response test – in general a better indicator of full static detection potential for most products, although sadly omitting the proactive insight. Scores were generally pretty decent, and pass rates were high, with few issues in the certification sets. A handful of false alarms and some rather odd issues in the WildList sets dent this picture only a little.
Next time, we’re back to Windows 7, a much more pleasing platform, and one which will doubtless see a much broader field of participants. Once again, time is going to be at a premium, so we can only hope for fewer problems than we have seen this month.
Test environment. All tests were run on identical systems with AMD A6-3670K Quad Core 2.7GHz processors, 4GB DUAL-DDR3 1,600MHz RAM, dual 500GB and 1TB SATA hard drives and gigabit networking, running Microsoft Windows Server 2012. For the full testing methodology see http://www.virusbtn.com/vb100/about/methodology.xml.
Any developers interested in submitting products for VB's comparative reviews, or anyone with any comments or suggestions on the test methodology, should contact [email protected]. The current schedule for the publication of VB comparative reviews can be found at http://www.virusbtn.com/vb100/about/schedule.xml.
