Donkey On A Waffle
A function of time. Accurately scoping an application assessment.
Tue, 17 Apr 2007 22:16

I find myself repeating a certain description to the majority of my clients. That is:

"Penetration testing is a function of time. An attacker has the luxury of infinite time, and a penetration tester only has as much time as the clients resources will allow. Finding vulnerabilities ends up being a curve, discovered vulnerabilities over time, in which there is a ramp-up phase, a steady incline to a peak, and finally a drastic drop. The point at the top of the curve (and possibly just after it) is the point of of diminishing returns. This is where the amount of resource investment outweighs the chances of finding additional vulnerabilities. THAT is the amount of time the client should want me to test the application."

So that being said. How does one determine this exact point of diminishing returns. Is there a way in which we can quantify the size and features of the application accurately enough to predict the height of this curve in man hours?

This concept is sometimes easier to grasp if we focus on one particular application type and then attempt to extrapolate those same ideas to other targets. Let's look specifically at web applications.
In the past I have seen people attempt to use the number of static and dynamic pages to estimate size and thus time. This is a good idea, but falls down when those pages have a large number of input fields making the scope grow significantly and without warning. Alternatively there could be hundreds of pages, but those pages contain a minimal amount of dynamic functionality and little to no input fields. Now we end up with a gross over estimation of the size of the application. I've also seen people attempt to determine size and time by taking a high level swath at the amount of functionality within the application. Questions such as "how many sections of independent functionality are there in the application?" end up with vague estimates that can be wildly off the mark as well. This method also relies upon the scoping person's experience level to know what "Application foo has 3 sections of functionality, funds transfer, account management, and profile management" really means in terms of time. Last, but not least, I've also seen the LOC (lines of code) attempt at determining amount of testing time required. When analyzing an application for amount of time required for testing, lines of code usually doesn't play that much of a part. I've seen applications with a huge LOC number that really doesn't have much of a threat landscape and I've seen code that is extremely dense (aka low LOC) and has an immense amount of functionality. However, LOC can many times be used to determine a rough guesstimate of application complexity, and as we all know code complexity has been proven to have a direct correlation to vulnerability density (take this research with a grain of salt).

Now that I've rambled on for quite a bit, I want to know if any of you lovely readers have an idea for a way to properly quantify application size in an attempt to ACCURATELY determine the amount of time it will take to get to that sweet spot of diminishing returns. I'm tired of overcharging some clients one week and then not delivering a complete test because we ran out of time the next. My initial thoughts are a mix of all of the above crunched numerically to determine a length of time estimate (hopefully an accurate one). Any takers?!

Home | Tags: , , | Category: /infosec | Link