independent tests
Each year products by Kaspersky Lab, along with other vendors are tested in a number of independent benchmarks and comparative reviews. We collect the statistics from each of these and each year create a diagram which serves as a visual representation of the pool of companies participating in tests, likely winners and TOP3 residents. Our products have demonstrated the best results for the third consecutive year, achieving a higher percentage of top-three places and received awards than any other vendor: 82%.
Why we need tests
In fact, independent tests are designed for end users. The developers of security products always strive to place their solutions in the spotlight and position them as the ‘best’ on the market, but if one trusts completely to the marketing then all of the products are ‘the best-in-class’. Tests, in this respect, offer a fairly accurate picture which is not influenced by developers and ultimately helps user navigate through the deep waters of marketing jargon and slogans.
Moreover, the tests are based on different approaches and help to evaluate various aspects of a security solution. For some, the most important capability would be the lowest number of false positives, others care more if a product performs effectively in real-world tests, regardless of false positives; and other consumers or businesses might appreciate minimal impact on their PC’s performance. This information is available to a user through different benchmarking tests.
As for developers, benchmarks are not solely a means of showing off. We see them as an integral part of the product development process. Regular and comprehensive independent reviews provide out team with an extra pair of eyes which helps to spot drawbacks in our products — generally it’s better off with a situation when they are spotted by researchers and not by competitors.
How are benchmarks run?
There are a number of major independent dotted around the world. They have years of experience and are constantly updating their testing methodologies.
Some security companies try to render such tests pointless, claiming that the products are tested in ‘lab conditions’ as opposed to real-life testing. Whilst some point out that the participating vendors tweak their products to better perform in a particular benchmark, in reality it’s been a long time since testers relied exclusively on signature-based methods. Nowadays hardly anyone limits testing to such methods: researchers are interested in seeing how products perform in real life scenarios, so aside from feeding contestants a collection of malware samples, they employ a battery of sophisticated tests to see how the companies manage complex threats.
For example: AV-Test use a selection of 0-day threats in each test; MRG-Effitas, apart from using up-to-date financial threats, relies on a number of considerably sophisticated methods, like API Hooking tests; AV-Comparatives handles separate researches, like “Whole Product Dynamic “Real-World” Protection Test“. The rest of the testers employ similar methods that are able to accurately mimic real-life conditions. Sometimes benchmarks include recently discovered exploits, which cannot be detected with signature-based analysis at all.
Are independent tests truly that independent?
Testers are not out to favor any of the contestants specifically because testing companies treasure their independent position. The assets they have are their reputation and expertise, which they put to considerable use during testing.
In order to participate in the benchmark, all vendors have to pay a small provision. That said, a stance like ‘We don’t pay to participate in the testing’ is not a good enough excuse not to participate.
Of course, there are tests commissioned by certain vendors. Usually, such benchmarks are meant to compare a certain product against the competition that are not participating in public tests, or to measure the efficiency of security products against specific threats. In this case, it’s only the commissioner who pays, yet this approach does not equate to coercion either. The tester, in any case uses established testing methodologies, and the commissioner cannot influence the results.
What is TOP3 meant for?
The underlying idea of TOP3 remains the same: it is a comprehensive assessment of different vendors’ results in various tests over an extended period of time. The methodology also remained the same. You can find more details are available here.
It is understood that winning one test could be down to favorable conditions. A vendor who participated in only one test and showed good results, would then have a 100% ‘wins’ over ‘number of tests’ ratio. However, this isn’t comparable to real-world conditions. To be a more well rounded solution, a security vendor must offer its products to as many tests as possible.
Tips