[fuzzing] Fuzzing tradeoffs - where previously described?

[Ari Takanen]

Hello all,

On Thu, Feb 01, 2007 at 09:36:19AM -0500, Joshua Morin wrote:
> I don't think there could be a standardized formula or setup for
> fuzzing, there is wide range of variables that come into play,
> especially if we are talking outside of just application fuzzing and
> look at protocol fuzzing. A good fuzzer could take weeks to find all
> the vulnerabilities and a set time frame just leaves room for error. 

Note that because the input space is always infinite, it will always
take an infinite time to find all flaws. Optimizing (i.e. adding
intelligence) will find more flaws earlier, but can still never ensure
that "all" flaws have been found. This would not be true of course for
protocols that have a "physical" limit such as UDP packet size (and no
fragmentation options). Then you will have a physical limit that will
cover "all" tests for that single packet. But that would be a LOT of
tests. Even then the internal state of the SUT will influence the
possibility of finding some of the flaws.

> it is possibly to have a random and smart fuzzing in one. A a smart
> fuzzer could fix this issue of a specific time frame to fuzz,a good
> fuzzer can keep going after it "fuzzes" the DUT by going back and
> running the fuzz process again without having to shutdown the fuzzer
> itself, it may require setting up the DUT again but hey life is not
> perfect.

I agree with you, both approaches can co-exist. I think there is a
place for both random testing (fuzzing) and systematic testing
(robustness testing). If you have time and possibility to run random
fuzzers, then fine, go ahead! You might be the lucky one. I still
think random fuzzing is a nice community effort, and certainly if we
add more the monkeys in the process, we might find our
Shakespeare. But I also think random testing does not fit very well in
systematic software development process where repeatability and
predictable test execution time are crucial. Here and there in large
manufacturing organizations you might find people that find time for
random fuzzers, but it will be very difficult to enforce random
testing into the product development life-cycle. I would not even try
to say this or that is better. Both of them have their uses. And I am
not sure the same people would even benefit from having both
approaches. I have already heard e.g. our customers say that we have
too many test cases, and our tests are extremely optimized. They would
switch to other tools if we said that we just added 2 Million test
cases (especially if those tests would not find enough new issues). If
an optimized set of tests could find 90% of the vulnerabilities, then
how much can random testing add and at what cost? I think we should
try to find facts instead of speculation. If anyone has any data in
relation to this, please do not hesitate to send that to me. I can
summarize the results (and anonymizethe data if you wish).

Best regards,

/Ari

-- 
-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-
Ari Takanen                       Codenomicon Ltd.
ari.takanen at codenomicon.com       Tutkijantie 4E
tel: +358-40 50 67678             FIN-90570 Oulu
http://www.codenomicon.com        Finland
PGP: http://www.codenomicon.com/codenomicon-key.asc
-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-