I find myself repeating a certain description to the majority of my clients. That is:

"Penetration testing is a function of time. An attacker has the luxury of infinite time, and a penetration tester only has as much time as the clients resources will allow. Finding vulnerabilities ends up being a curve, discovered vulnerabilities over time, in which there is a ramp-up phase, a steady incline to a peak, and finally a drastic drop. The point at the top of the curve (and possibly just after it) is the point of of diminishing returns. This is where the amount of resource investment outweighs the chances of finding additional vulnerabilities. THAT is the amount of time the client should want me to test the application."

So that being said. How does one determine this exact point of diminishing returns. Is there a way in which we can quantify the size and features of the application accurately enough to predict the height of this curve in man hours?

This concept is sometimes easier to grasp if we focus on one particular application type and then attempt to extrapolate those same ideas to other targets. Let's look specifically at web applications.
In the past I have seen people attempt to use the number of static and dynamic pages to estimate size and thus time. This is a good idea, but falls down when those pages have a large number of input fields making the scope grow significantly and without warning. Alternatively there could be hundreds of pages, but those pages contain a minimal amount of dynamic functionality and little to no input fields. Now we end up with a gross over estimation of the size of the application. I've also seen people attempt to determine size and time by taking a high level swath at the amount of functionality within the application. Questions such as "how many sections of independent functionality are there in the application?" end up with vague estimates that can be wildly off the mark as well. This method also relies upon the scoping person's experience level to know what "Application foo has 3 sections of functionality, funds transfer, account management, and profile management" really means in terms of time. Last, but not least, I've also seen the LOC (lines of code) attempt at determining amount of testing time required. When analyzing an application for amount of time required for testing, lines of code usually doesn't play that much of a part. I've seen applications with a huge LOC number that really doesn't have much of a threat landscape and I've seen code that is extremely dense (aka low LOC) and has an immense amount of functionality. However, LOC can many times be used to determine a rough guesstimate of application complexity, and as we all know code complexity has been proven to have a direct correlation to vulnerability density (take this research with a grain of salt).

Now that I've rambled on for quite a bit, I want to know if any of you lovely readers have an idea for a way to properly quantify application size in an attempt to ACCURATELY determine the amount of time it will take to get to that sweet spot of diminishing returns. I'm tired of overcharging some clients one week and then not delivering a complete test because we ran out of time the next. My initial thoughts are a mix of all of the above crunched numerically to determine a length of time estimate (hopefully an accurate one). Any takers?!

What makes a good penetration tester? That is the question of the day. First let's define the terms:

UPDATED 3/20

According to WikiPedia a "Penetration Test" is - a method of evaluating the security of a computer system or network by simulating an attack by a malicious user, commonly known as a hacker. The process involves an active analysis of the system for any potential vulnerabilities that may result from poor or improper system configuration, known and/or unknown hardware or software flaws, or operational weaknesseses in process or technical countermeasures. This analysis is carried out from the position of a potential attacker, and can involve active exploitation of security vulnerabilities.

First let me take this definition and refine it just a bit. We are going discuss specifically "application penetration testers" (APT). What is meant by that is that the broad scope of "network penetration test" is a bit more refined and focuses more on COTS applications and custom code. Many people consider this skill set to be what a "real" penetration test is. Anyone with a basic knowledge of computers can run nessus and create a report. It takes a certain skill and mindset to test at the application level. Some would even go so far as to say that what we really are talking about is a vulnerability researcher. So what makes a good "penetration tester" as we have defined it?

Technical skills

The exact technical skills required are largely dependent upon the applications being tested. Web application testing uses a completely different tool base and has many different attack techniques than say a windows binary product penetration test. However, the basic skills are the same. Understanding how applications are built, how code works, how to properly conduct threat modeling, abuses cases, determining areas of security analysis for a given application, and finally execution and creation of proof of concept are universal. This can all be taught. People can learn these skills. It is best to be intimately familiar with the target application if at all possible. If the tester is not already an expert in the application specific components, this process of familiarity typically takes place at the start of an engagement.

Ability to learn

Since applications and custom code are always so different, an APT consultant has to have the ability to learn quickly. Gather the resources and absorb them in a very short time. This is due to the fact that penetration test work is typically timed boxed. What is meant by that is that while the hackers have the luxury of infinite time, a paid consultant is limited to whatever the client is willing to pay. This doesn't apply to vulnerability researchers conducting work on their own time, but when someone is paying you for it I can just about promise you they won't let you test forever. You have to be able to learn FAST. It has to be in the consultant's nature to want to know, to want to learn, and to want to do it as rapidly as possible. "If I don't know it, give me 24 hours.. I will!"

The competitive drive

This is key. The drive to win. The drive to find a hole in everything you touch. Many people don't have this drive. Many people will give up at the first brick wall. Someone once told me that "Every application you touch is broken. It's up to you to find out where." This attitude that everything is broken in some way and that it is only a matter of time until it's found is the drive that a good APT consultant needs. This personality trait might best be described as stubborn or obstinance. There is nothing that is going to stop you from finding that vulnerability.

Creativity

I received a little feedback from Curq of 0x90 fame today. He pointed out another trait that indeed is something that is required of a good penetration tester. Creativity. A good penetration tester must be able to create, invent, and otherwise imagine new threats and attacks. Things aren't always going to follow the same old threat path, and invariably the attacker must figure out new techniques. That is where creativity comes in to play!

Experience

Last but not least is experience. The more an APT consultant has, the better they are going to be. It's just like anything else, the more time you have practicing the stronger your game will become. Penetration testing is no different. This isn't a requirement to becoming a strong penetration tester, however it is a good reference when looking to see how strong a tester may be.

That's about it. If a person has strong technical skills in the right areas, the ability to quickly learn, and that all important competitive drive, they can become a strong penetration tester. All that is required at that point is experience, and that just comes with time.

Comment and discuss!