Details of the agenda and how to book the hotel at the discounted rate are available on the AMTSO website in the forum.

AMTSO Board

However, Simon Edwards has put up an article on his own blog that I can commend to anyone with an interest in product testing. I’ve already commented at some length in an SC Magazine article.

I’m also hoping to prevail upon Simon, who represents his company in AMTSO, to blog here from time to time. It would be good to have someone from the testing side putting their point of view here…

David Harley CITP FBCS CISSP
ESET Senior Research Fellow

The commercial scanners did a lot better in the on-demand scanning tests (more than 50% detected) against 32% by the best-performing free app and 6% and 1% respectively by the 2nd and 3rd ranked apps. Three other scanners scored zero in the on-demand tests. However, AV-Test did note that some scanners (which was not specified) only scan installed apps, which to me says that static testing of inactive products on an SD card is potentially misleading unless you assume that static scanning on removable media is sine qua non. That’s probably a debate for another time.

In the on-access (0n-installation) test, F-Secure and Kaspersky scored 100%. Since the test set was ten of the apps most often diagnosed as malicious by the scanners used by AV-Test for validating its Android sample collection, it would probably be somewhat disappointing if they didn’t. Still, the real shocker here is that while the app that did best in the static test scored 8/10, three of the others only detected 1/10, while the others didn’t detect any.

Here’s food for thought. The app with the largest user base (1- 5 million) scored zero in both tests. Given the steady increase in malicious Android apps (the static test used 172 samples, none more than five months old), it might be time for Android users to consider whether they can afford to use a free AV app, or whether they might at least need to see how well their favoured product does in comparative tests. Andreas Marx told The Register that AV-Test will be running further tests of this sort, and I know that other testing and certification agencies are also looking at mobile security testing.

David Harley CITP FBCS CISSP
ESET Senior Research Fellow

Anyway, the blog notes that NIST (the National Institute of Standards and Technology) has just released a Cloud Computing Standards Roadmap, recognizing the complexity of certification of products in the context of the rising demand for cloud services. In particular, the draft inter-agency advisory report (NISTIR 7328) is about requirements for security assessment providers, in particular those who are offering assessment as a service.

While it’s a very different document to AMTSO’s guidelines for testing in the cloud, it comes from a similar appreciation of the difficulties of this area of testing.

David Harley CITP FBCS CISSP
ESET Senior Research Fellow

And now I’ve come across a recent blog by Eugene Kaspersky in which he talks about The Holy Grail of AV Testing, and Why It Will Never Be Found: in fact, making a plea for something that sounds very similar to “whole product testing” as defined by AMTSO (which features strongly in his argument), and building on another of his articles, as I previously mentioned here. Essentially, he comes to the depressing conclusion that “…sadly, proper results from proper tests are these days simply nowhere to be found, despite their Holy Grail status.”

“Nowhere” might be a little harsh, but I have to admit that I’ve heard a lot of similar sentiments expressed recently. Yet here I am at yet another AMTSO workshop, halfway through a gruelling schedule of meetings. We’re still here, and still trying to establish some form of chivalric code. Please keep any mutterings about honour among thieves to yourself: after many years in or on the fringes of the AV industry, I’ve heard it all, but I haven’t lost faith in the ability of the vendor and tester communities to put personal differences and vested interests aside for the protection of the end user. Flashbacks to Monty Python notwithstanding.

David Harley CITP FBCS CISSP
ESET Senior Research Fellow

That presentation is in PDF form and doesn’t include the speaker notes, but I’m working on that.

David Harley CITP FBCS CISSP
Small Blue-Green World
ESET Senior Research Fellow

That’s me on the left trying to hide behind the podium, and co-presenter Larry Bridwell (AVG) on the right.

Picture by Sorin Mustaca of Avira.

David Harley CITP FBCS CISSP
ESET Senior Research Fellow

Abstract:

Daze of whine and neuroses (but testing is FINE) 

According to Aerosmith (not to mention The Italian Job), FINE is an acronym for (in its politer version) Freaked out, Insecure, Neurotic and Emotional. We could (and probably will) offer alternatives, but there’s no doubting that anti-malware testing inspires all those reactions.

Sometimes it seems that AMTSO has become a dumping ground for the rest of the world’s misgivings about the AV industry, even though it originated in a coalition with some of the testers who are monitoring that industry’s performance with the most assiduous professionalism: indeed, that coalition has in itself inspired mistrust. And recently, it has become plain that even within AMTSO both testers and vendors sometimes find the alliance problematical.

 AMTSO’s purpose is simple to state, but much harder to achieve. It represents a realization by professional testers and security vendors that the quality of anti-malware testing was so variable that it was at best confusing for people who need guidance on how to select the best product for their needs. Perhaps testing has improved more in the past few years than it would have without AMTSO’s presence, and discussions and generation of material in a single forum has accelerated a much-needed move away from static testing towards dynamic testing.

But it’s time to ask (and attempt to answer) a number of tough but critical questions.

• Looking over the historical evolution of testing before and since AMTSO, is that move towards dynamic testing enough to set the testing world to rights?
• Are the aims of testers and vendors close enough to allow continued cooperation within AMTSO?
• Has AMTSO already outlived its usefulness?
• If not, what should it do next?
• What is the future of comparative detection testing?

David Harley CITP FBCS CISSP
ESET Senior Research Fellow

Which actually reminds me another point. You may have noticed that there haven’t been any recent updates to the “AMTSO in the media” page. That isn’t because there hasn’t been any mention of us anywhere since June 2011 – though I have to admit that it is hard to interest the media in stuff that even testing geeks get bored with and there hasn’t been the sensational and/or controversial press that we saw a year or two ago - but because of resource starvation.  So I’m not abandoning the page altogether, but there’s no guarantee that it’s up to date, and I’m definitely not going to be looking for every mention of AMTSO in the media from June till now in the hope of catching up.

David Harley CITP FBCS CISSP
AMTSO Board member
ESET Senior Research Fellow

Memorable extract: “Alas, practically all benchmarking tests don’t bother with weightings, bunching apples and oranges, and pears and peaches – and bananas and hamsters – all together as being the same.” His conclusion is that no-one can authoritatively measure the quality of an anti-virus solution: “there really isn’t a test like that – one that I could unambiguously recommend as the best indicator.”

And Lysa Myers, of West Coast Labs, has a comment piece in October’s Virus Bulletin about “Why there’s no one test to  rule them all,” with a somewhat similar conclusion: “Because every product has strengths and weaknesses, having a variety of different tests is essential.” Unfortunately, you’ll need to be a subscriber to read it.

David Harley CITP FBCS CISSP
ESET Senior Research Fellow