Now that the holidays are well and truly behind us, can we remind you that the next AMTSO workshop takes place in San Mateo on the 23rd and 24th February, just before the start of the RSA Conference?
Details of the agenda and how to book the hotel at the discounted rate are available on the AMTSO website in the forum.
AMTSO Board
It’s been a slow few weeks in the world of AV testing, news-wise, though I’ve just contributed a fairly hefty article on AMTSO to Elsevier for one of their periodicals that will hopefully appear in the New Year. (The article, that is: I’m sure the periodical will.)
However, Simon Edwards has put up an article on his own blog that I can commend to anyone with an interest in product testing. I’ve already commented at some length in an SC Magazine article.
I’m also hoping to prevail upon Simon, who represents his company in AMTSO, to blog here from time to time. It would be good to have someone from the testing side putting their point of view here…
David Harley CITP FBCS CISSP
ESET Senior Research Fellow
Well-known testers AV-Test recently looked at free AV apps for Android devices. They used a Samsung GalaxyTab GT-P1010 (Android vs. 2.2.1) as a testbed, testing both on-demand and on-access scanning, and using commercial scanners by F-Secure and Kaspersky for comparison.
The commercial scanners did a lot better in the on-demand scanning tests (more than 50% detected) against 32% by the best-performing free app and 6% and 1% respectively by the 2nd and 3rd ranked apps. Three other scanners scored zero in the on-demand tests. However, AV-Test did note that some scanners (which was not specified) only scan installed apps, which to me says that static testing of inactive products on an SD card is potentially misleading unless you assume that static scanning on removable media is sine qua non. That’s probably a debate for another time.
In the on-access (0n-installation) test, F-Secure and Kaspersky scored 100%. Since the test set was ten of the apps most often diagnosed as malicious by the scanners used by AV-Test for validating its Android sample collection, it would probably be somewhat disappointing if they didn’t. Still, the real shocker here is that while the app that did best in the static test scored 8/10, three of the others only detected 1/10, while the others didn’t detect any.
Here’s food for thought. The app with the largest user base (1- 5 million) scored zero in both tests. Given the steady increase in malicious Android apps (the static test used 172 samples, none more than five months old), it might be time for Android users to consider whether they can afford to use a free AV app, or whether they might at least need to see how well their favoured product does in comparative tests. Andreas Marx told The Register that AV-Test will be running further tests of this sort, and I know that other testing and certification agencies are also looking at mobile security testing.
David Harley CITP FBCS CISSP
ESET Senior Research Fellow
I’ve no excuse for not having noticed this before, as it was an entry on the Infosecurity Magazine blog, to which I’m a contributor. But miss it I did, I’m afraid, for over a week, even though it’s testing related. The article ““Testing the Testers”: Certification and Cloud Computing” was actually contributed by the (ISC)² U.S. Government Advisory Board Executive Writers Bureau. (Coincidentally, I blog for (ISC)2 as well, butI have nothing to do with the Government Advisory Board AWB.)
Anyway, the blog notes that NIST (the National Institute of Standards and Technology) has just released a Cloud Computing Standards Roadmap, recognizing the complexity of certification of products in the context of the rising demand for cloud services. In particular, the draft inter-agency advisory report (NISTIR 7328) is about requirements for security assessment providers, in particular those who are offering assessment as a service.
While it’s a very different document to AMTSO’s guidelines for testing in the cloud, it comes from a similar appreciation of the difficulties of this area of testing.
David Harley CITP FBCS CISSP
ESET Senior Research Fellow
Strange how the whole AV testing issue seems to attract metaphors and analogies harking back in some sense to mediaeval romance. For the past few days I’ve been muttering grimly about tilting at windmills, having recently come across an unhappy example of a test apparently relying on a simulation. (You’ll probably hear more from me on that, though I won’t necessarily be performing my impersonations of Don Quixote on this blog) .
And now I’ve come across a recent blog by Eugene Kaspersky in which he talks about The Holy Grail of AV Testing, and Why It Will Never Be Found: in fact, making a plea for something that sounds very similar to “whole product testing” as defined by AMTSO (which features strongly in his argument), and building on another of his articles, as I previously mentioned here. Essentially, he comes to the depressing conclusion that “…sadly, proper results from proper tests are these days simply nowhere to be found, despite their Holy Grail status.”
“Nowhere” might be a little harsh, but I have to admit that I’ve heard a lot of similar sentiments expressed recently. Yet here I am at yet another AMTSO workshop, halfway through a gruelling schedule of meetings. We’re still here, and still trying to establish some form of chivalric code. Please keep any mutterings about honour among thieves to yourself: after many years in or on the fringes of the AV industry, I’ve heard it all, but I haven’t lost faith in the ability of the vendor and tester communities to put personal differences and vested interests aside for the protection of the end user. Flashbacks to Monty Python notwithstanding.
David Harley CITP FBCS CISSP
ESET Senior Research Fellow
The presentation Larry Bridwell and I delivered at Virus Bulletin last week based on our paper Daze of Whine and Neuroses (but Testing is FINE) is now posted on the Virus Bulletin web site, along with a selection of other presentations from the technical and corporate streams, plus some last minute presentations.
That presentation is in PDF form and doesn’t include the speaker notes, but I’m working on that.
David Harley CITP FBCS CISSP
Small Blue-Green World
ESET Senior Research Fellow
…whining about the state of testing…
That’s me on the left trying to hide behind the podium, and co-presenter Larry Bridwell (AVG) on the right.
Picture by Sorin Mustaca of Avira.
David Harley CITP FBCS CISSP
ESET Senior Research Fellow
…is the title of a presentation AVG‘s Larry Bridwell and I did at Virus Bulletin recently. The paper is now up (by kind permission of Virus Bulletin) on the ESET white papers page as Daze of Whine and Neuroses.
Abstract:
Daze of whine and neuroses (but testing is FINE)
According to Aerosmith (not to mention The Italian Job), FINE is an acronym for (in its politer version) Freaked out, Insecure, Neurotic and Emotional. We could (and probably will) offer alternatives, but there’s no doubting that anti-malware testing inspires all those reactions.
Sometimes it seems that AMTSO has become a dumping ground for the rest of the world’s misgivings about the AV industry, even though it originated in a coalition with some of the testers who are monitoring that industry’s performance with the most assiduous professionalism: indeed, that coalition has in itself inspired mistrust. And recently, it has become plain that even within AMTSO both testers and vendors sometimes find the alliance problematical.
AMTSO’s purpose is simple to state, but much harder to achieve. It represents a realization by professional testers and security vendors that the quality of anti-malware testing was so variable that it was at best confusing for people who need guidance on how to select the best product for their needs. Perhaps testing has improved more in the past few years than it would have without AMTSO’s presence, and discussions and generation of material in a single forum has accelerated a much-needed move away from static testing towards dynamic testing.
But it’s time to ask (and attempt to answer) a number of tough but critical questions.
David Harley CITP FBCS CISSP
ESET Senior Research Fellow
This would naturally belong with the blog I put up recently as Two excellent testing articles, but somehow I missed Simon Edwards’ blog on Anti-malware testing discussions. So I’m a couple of weeks late on pointing it out, but it’s certainly worth your time to take a look.
Which actually reminds me another point. You may have noticed that there haven’t been any recent updates to the “AMTSO in the media” page. That isn’t because there hasn’t been any mention of us anywhere since June 2011 – though I have to admit that it is hard to interest the media in stuff that even testing geeks get bored with and there hasn’t been the sensational and/or controversial press that we saw a year or two ago - but because of resource starvation. So I’m not abandoning the page altogether, but there’s no guarantee that it’s up to date, and I’m definitely not going to be looking for every mention of AMTSO in the media from June till now in the hope of catching up.
David Harley CITP FBCS CISSP
AMTSO Board member
ESET Senior Research Fellow
One by Eugene Kaspersky (yes, that Kaspersky) on “Benchmarking Without Weightings: Like a Burger Without a Bun” (hat tip to Larry Bridwell for drawing my attention to it.)
Memorable extract: “Alas, practically all benchmarking tests don’t bother with weightings, bunching apples and oranges, and pears and peaches – and bananas and hamsters – all together as being the same.” His conclusion is that no-one can authoritatively measure the quality of an anti-virus solution: “there really isn’t a test like that – one that I could unambiguously recommend as the best indicator.”
And Lysa Myers, of West Coast Labs, has a comment piece in October’s Virus Bulletin about “Why there’s no one test to rule them all,” with a somewhat similar conclusion: “Because every product has strengths and weaknesses, having a variety of different tests is essential.” Unfortunately, you’ll need to be a subscriber to read it.
David Harley CITP FBCS CISSP
ESET Senior Research Fellow
