When it comes to penetration testing, it’s a fact that many organizations will engage third party consultants to perform the service. The reasons why this is so aren’t hard to understand: doing penetration testing well requires a specialized set of skills and tools, and keeping those resources and tools at an acceptable skill/performance level (to keep pace with the methods that the bad guys use) requires quite a bit of effort.
Given that many organizations only require testing on an as-needed basis, business dynamics quite often translate to bringing in someone from the outside.
But this can leave some organizations in a quandary: specifically, how do they know if they’re getting a good test? Since it’s not always a competency that organizations maintain a skill-base in, separating the “wheat from the chaff” among the various service providers can be difficult. Likewise, it’s hard to know based on the output of the test whether quality is there since “good quality” and “poor quality” can look much the same: for example, did the test not find a given issue because the issue doesn’t exist? Or did the test not find that issue because the service provider lacked the skill to find it? It can be hard to know.
Fortunately though, there are a few “warning signs” that organizations can use to spot early on in the vendor selection process and help ascertain if a particular service provider might be a bad fit. We’ve pulled together a few of them below. If a firm your thinking of engaging with demonstrates these symptoms, you may want to continue widening your search.
#1: Inability to articulate methodology
When most people think of penetration testing, methodological rigor isn’t necessarily the first thing that springs to mind. However, this is changing due to (in part) to the release of PCI DSS 3.0. What does PCI have to do with it, you ask? Simply put, PCI DSS 3.0 now requires that penetration testing activities (at least those done to satisfy PCI requirements) adhere to an “industry-accepted penetration testing methodology.” The PCI standard specifically references NIST SP800-115 (“Technical Guide to Information Security Testing and Assessment”) but there are other standards as well that might supplement this (for example the Penetration Testing Execution Standard or, for application testing, the OWASP Testing Guide).
The point is, there are standards that service providers have the option of adhering to. Your organization might not be testing specifically for the purpose of PCI compliance, but following a standard has some advantages even if you aren’t: reproducibility of the test for one, ensuring thoroughness of coverage is another. If you can’t get a straight answer about what methodology a provider you’re considering follows (assuming they follow one at all), you may want to dig a little deeper to find out why this is the case.
#2: “Blank stares” about industry-specific problem areas
We all know that certain specific industries all have their own unique needs and concerns. A testing firm that has experience in your industry will likely have a pretty good idea what the most likely “problem areas” are and can explain to you why they are problematic and how they intend to proceed as a result.
For example, in a healthcare provider context (e.g. a hospital or health system), an experienced testing firm in that industry might flag your “HL7 Interface Engine” (a specialized healthcare “routing” tool for application messages) as a likely source of downtime should someone launch attacks against it (because having done this, I can tell you it happens and it happens often). For a telephone company, they might flag specialized communications routing equipment or for an energy company they might highlight certain ICS systems.
The bottom line is, if they have the expertise to test this specialized equipment, they have the experience to know which areas are likely to present the biggest challenges and explain to you what the challenges are and why. If they don’t understand the specific “gotchas” of your industry, you might ask yourself whether the value of the test will be there after they’re done testing those areas.
#3: They won’t give you a tool list
Many firms specializing in pentesting maintain a library of proprietary scripts and tools unique to them – in fact, for many, this is part of their competitive advantage. This means that a potential service provider saying they use “proprietary tools” or “custom scripts” isn’t necessarily a warning sign on its own. That said, there are certain common tasks (e.g. port scanning, application crawling, application parameter manipulation and proxying, or any number of routine tasks) that don’t need to be reinvented every time somebody runs a test.
If you’re considering a vendor and they balk when asked for a list of the specific tools they use to support their service, one of two things could be going on: it’s possible they’re trying to protect their “secret sauce” (though I’d argue that much of the value-add they provide is in their staffs’ skill and analysis capability vs. the tools themselves.) It’s also possible that their service relies heavily on a limited set of commercial products which you could run yourself at a fraction of the cost.
#4: Conflating vulnerability scanning and penetration testing
Vulnerability scanning (looking for vulnerabilities in an automated way) and penetration testing (exploiting those vulnerabilities using manual or automated techniques) are two very different activities. You’d be surprised how often providers actually in the business of providing these services do not discriminate between the two. A service provider who isn’t clear on the difference (for example by using the terms interchangeably in an RFP response) is likely to provide you with test results that contain little more than the output of a vulnerability scanning tool.
Given that many of the customers that purchase these services may not (by design) maintain a skill-base in this discipline, keeping an eye open for these issues can help you guide you to those firms that might be more skilled. I’m sure there are many more warning signs other than just the ones cited here, but starting with a few of these most obvious ones can give you something to start from as you look for the right vendor for your shop.
Ed Moyle is Director of Emerging Business and Technology for ISACA. Prior to joining ISACA, Ed was a founding partner of the analyst firm Security Curve. In his more than 15 years in information security, Ed has held numerous practitioner and analyst positions including senior manager with CTG’s global security practice, vice president and information security officer for Merrill Lynch Investment Managers, and senior security analyst with Trintech. Ed is co-author of Cryptographic Libraries for Developers and a frequent contributor to the Information Security industry as author, public speaker, and analyst.