Cylance Controversy Illustrates What’s Wrong with AV Testing

The current state of AV testing is sort of the cybersecurity version of standardized testing in schools. It’s a pet peeve of mine when schools “teach the test”–focusing on ensuring students perform adequately on standardized tests rather than making sure they have a comprehensive understanding of the subject. The AV testing version is that products are tested and compared using known virus samples–proving they can detect “threats” we all know about, but doing nothing to demonstrate how well the tools can defend against malware in the real world.

Version:1.0 StartHTML:000000238 EndHTML:000003625 StartFragment:000002756 EndFragment:000003589 StartSelection:000002756 EndSelection:000003589 SourceURL:http://blogs.forbes.com/tonybradley/wp-admin/post.php?post=3143&action=edit

The recent controversy over whether or not the files in the Cylance test samples are actual malware and whether or not other endpoint security solutions can detect them is an illustration of the issues that exist with AV testing.

There has been some controversy recently around Cylance and the methodology it recommends to test the efficacy of CylancePROTECT compared with other endpoint security solutions. There are certainly issues with how antimalware tools are compared and evaluated—especially by the accepted, industry-standard measurements—but those issues are neither unique to, nor a function of Cylance itself.

A recent story from Ars Technica points to a battle over whether or not the files provided by Cylance for prospective customers to test various endpoint products are legitimate or not. According to the story, “In testing, Protect identified all 48 of the samples as malicious, while competing products flagged most but not all of them. Curious, the engineer took a closer look at the files in question—and found that seven weren’t malware at all.”

The suggested implication of this finding is that Cylance is trying to game the system. At face value, it appears that Cylance is misleading prospects by using these 48 malware samples, even though some are not even recognized as malware by other endpoint solutions.

The reality, though, is that the fact they’re not recognized as malware by other endpoint security solutions is exactly the point. Chad Skipper, VP of Industry Relations for Cylance, explained, “Cylance does not mislead customers or prospective customers. When we create malware samples to test with, we employ the same methods and tools that hackers do, including creating mutations and packing the samples, to better emulate what attackers do for more meaningful testing. We are not running or using any tool that isn’t already in an attacker’s arsenal.”

The fact is that any time you pack a real file, there is a chance that the original piece of software will break. Some installers use internal checksums that are broken by the packing process, resulting in a valid file that does nothing (but still has the earmarks of malware, even if it doesn’t run correctly). Skipper stressed, “This is how it works in the real world and can be seen frequently in real malware, where the resultant mutated sample doesn’t operate anymore.”

Skipper assured me—with 100 percent certainty—that the original files Cylance packs are, in fact, legitimate malware. “As we do not control the inputs, sometimes this process can result in valid but harmless files being output as attackers change their tactics in how they generate input files. When we become aware of situations where outputs are no longer valid, we make adjustments to our process to remove these to ensure the fairness of results.”

In other words, while the industry standard is to compare antimalware products by detecting and blocking a library of known threats, the files from Cylance are actually a more authentic reflection of what an organization will be faced with in the real world. The reality is that attackers are constantly adapting and mutating threats, so being able to detect and block a threat that is already known is almost useless. It’s like shutting the barn door after the horses have already escaped.

Read the full story on Forbes: Don’t Shoot The Messenger: Cylance Didn’t Break AV Testing.