l’AMTSO Mysterioso and Malware Creation

Some more press/media/blog mentions regarding our recent guidelines documents have crossed my radar. Many of them are quite similar, so I won’t list them all here. (I guess I don’t have a future in PR, then…)

Craig Kensek’s blog “AMTSO Releases Additional Documents on Malware Security Test Design and Testing” is a neat summary of the documents and the issues, as I’d expect, having had some communication with him on testing-related issues in the past. In fact, his blog at http://kensek.blogspot.com/ often comments on testing issues (and AMTSO), so is well worth a look.

Security Curve, however, seems a little – well, behind the curve. The writer describes us as “the self-proclaimed keeper of all things associated with the testing of anti-malware software.”

Really? I must have missed that press release. Actually, our charter describes our mission as:

• Providing a forum for discussions related to the testing of anti-malware and related products.
• Developing and publicizing objective standards and best practices for testing of anti-malware and related products.
• Promoting education and awareness of issues related to the testing of anti-malware and related products.
• Providing tools and resources to aid standards-based testing methodologies.
• Providing analysis and review of current and future testing of anti-malware and related products.

Not a trivial set of objectives, and I wouldn’t say that we’ve been 100% successful so far, but certainly a little more modest than claiming to be the ultimate authority on all things test-related.

He or she also cites the participant list from our formation press release of 2008 as ‘a “who’s who” of AV vendors.’

Well, our members are predominantly security vendors, but even that list includes several companies that certainly aren’t AV vendors. If you look at our current membership, you’ll see several more (including testing organizations and publishers). As it happens, we’d be very happy to have more non-vendors represented in AMTSO.

However, I’m not sure why the presence of vendors in a group like this is a Bad Thing. After all, we do know more about malware than most, and we do rather a lot of testing ourselves, though most vendors would consider it ethically suspect to publish comparative results from internal testing.

Ah yes. Ethics. Yes, most AV vendors are ethically opposed to the creation of malware. And we particularly dislike seeing malware creation in the context of testing, as I’ve discussed here before. Though speaking personally (and when I write here, I’m always speaking personally, not on behalf of AMTSO), I’d say that I’m less bothered by the ethical issue in this context. What is ethical for a vendor isn’t necessarily ethical for a tester, let alone someone outside the industry altogether, though the mainstream testers I know seem to subscribe to similar ethical standards to the AV industry. What does bother me is the question of competence, and I’m afraid I’m going to quote my previous blog on the topic.

… if you’re attempting to create your own malware because you can’t get samples from other sources, the chances are that you don’t have the knowledge to create samples that represent real-world threats… A good tester does have ethical principles, and feels that he owes it to audience to be as accurate as possible in his testing. And he also knows that artificial samples are not “the real thing”…

However, there’s an interesting point here. If a sample (custom-created or not) doesn’t replicate and it doesn’t have a “hostile payload” maybe it isn’t malware at all. In which case, is it really a suitable object for inclusion in a test set?

Anyway, back to Principle 1, as quoted by Security Curve. AMTSO has not simply said that sample creation is a Bad Thing. In fact, Security Curve has cherrypicked a short extract from the commentary on Principle 1 that talks about creation: here’s the full Q&A section.

Q. What is meant by “creation of new malware”?

A. This reference has historically referred to the creation of new viruses or strains of malware, one objection being based on the principle that there are more than enough samples available in the wild for everyone. This mandate has been complicated by introduction of packers and virtual machines, inviting the question as to whether utilizing these vehicles could be deemed to change the characteristics of pre-existing malware to the point that it could be deemed “new.” There are legitimate reasons to change existing malware characteristics for testing purposes – this principle is not included in order to preclude such testing. To be clear, however, this principle is included to demonstrate unanimous disapproval by AMTSO of the idea of the creation of new viruses or other malware and the related risk to the public. If you wish to contact AMTSO about these matters please send an inquiry to principles@amtso.org for more information.

AMTSO has also considered the arguments for and against in a comprehensive document here. Guys, you don’t have to agree with it. But I think before you criticize AMTSO’s stance , you really ought to find out what its members actually think…

David Harley CITP FBCS CISSP
Not speaking for AMTSO or his employer ESET.

Tags: AV vendors, competence, Craig Kensek, current membership, definition of malware, ethics, guidelines documents, Malware creation, PR, Press mentions, Principle 1, Security Curve, test design, testers