Thoughts on comparative testing

For months, Cylance has sought to dazzle audiences with its “Unbelievable” demonstration, staging well-choreographed battles against other IT security vendors, including Sophos. The exhibition ends with Cylance delivering near-perfect scores while everyone else (predictably) shows lackluster results. Yet when the playing field is leveled, and Cylance’s product comes under real scrutiny, the company cries foul, puts the fear of lawsuits into the minds of its partners, and accuses others of “smoke and mirrors” tactics.

At a recent Cylance presentation during an industry event in Las Vegas, one Sophos customer (from Chicago) in the audience asked to see how the Sophos product was configured for Cylance’s “Unbelievable” demo. On reviewing the settings, the customer discovered that key (and default) protection settings had been disabled. When the customer insisted that Cylance enable the proper default configuration and re-run the test, Sophos beat Cylance. The same behavior has been reported by multiple other vendors, including the disabling of everything other than hash lookups – an unfair test to say the least.

Continue reading