[fuzzing] Fuzzing tradeoffs - where previously described?

[Ari Takanen]

Hello again,

On Tue, Jan 30, 2007 at 05:53:30PM +0000, Disco Jonny wrote:
> I think the OP was trying to work out which type is worth doing
> (intelligent v random style) for that we would need new metrics. So
> his manager or whoever says I have $100k to spend on testing total
> (maybe secuirty only?) , how do we spend it.

I agree with you, we need metrics like this, and they would also be
useful for all the fuzzer people out there. But before going for any
Do-It-Yourself projects, really analyze the total cost for all
choices. Even if it is internally developed, it is still a "security
investment". There are many aspects to consider:

1) which approach finds most flaws (return of investment)

Efficiency would be the final measure, but that is very difficult to
predict. It is easiest if we could give a dollar-value to
efficiency. But what is "good enough" tool? I think this is a similar
question to saying "what is good enough anti-virus software". You
would not survive in security tools market if your solution only
caught 50% of the issues compared to competitor. Unfortunately if you
give enough time to a random fuzzer it might still not find even close
to as many flaws as a smart fuzzer would. But this is completely
different topic...

2) cost to implement (or investment in the tools)
3) time to implement (typically 0 if a third party tool)

The rest of the metrics should apply to both internally developed,
open source, and commercial tools:

4) time from availability to use (time used to test desing and integration)
5) time needed to test (again causes delays in the product/service launch)

... many people stop here... but lets keep on... 

6) other investment (HW + people, most expensive component)
7) resources needed to fix the problems (people)
8) time required to fix the found issues (again delays to product launch)
9) maintenance costs (is this one go, or reusable and maintained)

I might have missed some issues... These were the first ones that came
to mind. Some people value time more than others, so it would be
simpler if one could give a dollar value for time also in the
equation. And finally, like was already noted, it all comes down to
test efficiency, time and costs. It is just much easier if all of
these are measured in money (value).

I personally think efficiency (in test results and test process) is
much more important than the direct costs. Interesting topic though,
and I would be interested in seeing how such metrics would apply to
the fuzzing projects you have already done out there.

/Ari