[fuzzing] Fuzzing tradeoffs - where previously described?

[Jared DeMott]

>>
>> I agree with you, we need metrics like this, and they would also be
>> useful for all the fuzzer people out there.
>
> there is lots of info out there for the ROI on automation...
Thanks for the links, they are very helpful.
>
>
>> Unfortunately if you
>> give enough time to a random fuzzer it might still not find even close
>> to as many flaws as a smart fuzzer would. But this is completely
>> different topic...
>
> given enough time it /will/ find every bug.  depending on how you have
> it set up it will also exploit every bug with every possible shellcode
> that could go in there.  (infinite monkeys, shakespeare, etc) - this
> is not realistic though, so we need to find ways of getting this down.
> this is where automation v human fits in.  the smartest fuzzer is a
> human, and your cost always has to be offset against that.
ok, so, before we all get to far off the road with randomness here's a
piece from the book we're working on:


    6.1 Randomness

            Randomness is a sticky issue in testing.  Some believe that
delivering random data to the /lowest/ layers of each stage of a
protocol or API is the goal.  Others are firm believers in test cases. 
First of all, the randomness being discussed here needs to be
clarified.  We're not talking about a tool that sends completely random
data.  Such a tool is unlikely to find much.  By random we mean that
random paths through code, with random data, and random places and
configurations of a given protocol are somehow created and delivered. 

The two different schools of thought, randomness and determinism, differ
significantly in how test cases are constructed and delivered.  Consider
the simplest case imaginable, the FTP server.  One fuzzer might randomly
pick valid or invalid commands, randomly add them to a test case, and
choose random data for the arguments of those commands.  The test case
would than be delivered and rather or not a bug was discovered we
repeat.  While in the other case precooked /invalid/ data already exists
in a library, along with a series of command sequences.  Each invalid
data is supplied with each command in some deterministic manner. 

The pros and cons should be fairly obvious.  The randomized approach may
hit dark corners that the deterministic approach might miss as it's
impossible to think of every scenario or combination of commands and
data.  The field of fuzzing was based on this premise, and that's why
traditional testing was insufficient.  However, regression testing is
difficult at best, and the deterministic approach can be tuned to
achieve more reliable code coverage.  It is the authors' opinion that
both could and should be used in situations where robustness is of the
utmost importance.  A hybrid of these approaches could be construed. 
Many mature fuzzers do include elements of both.  For example, instead
of completely random data, fuzzing heuristics would likely be applied to
generate data that has been known to cause problems in the past. (Think
long strings, format strings, ../, etc.)
>
>>
>> 2) 
>
> now you guys seem to be coming round to the idea of security testing
> as being a subset of testing give 

Yes, fuzzing is under the testing umbrella.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://www.whitestar.linuxbox.org/pipermail/fuzzing/attachments/20070201/be412d48/attachment.html