[fuzzing] Commercial Fuzzer and Open-source Fuzzer comparison

[Ari Takanen]

Thanks for pitching the SSTIC paper!

The purpose in the paper was really to write a summary of fuzzing to
people who are not familiar with fuzzing. You guys (fuzzing gurus)
already have chosen (or developed) your tools and frameworks, and I
wish you good luck!

To me, this is not about trying to pick and choose the best open
source (or commercial) fuzzing tool or framework, but about trying to
get more and more people to do at least some type of fuzzing. I do not
really care what fuzzer people use (although I would appreciate if it
was some generation of PROTOS or Codenomicon).

Anyways quick comments below:

On Sun, Jul 05, 2009 at 12:00:04PM +0000, fuzzing-request at whitestar.linuxbox.org wrote:
> From: Sergio Alvarez <shadown at gmail.com>
> 
> Interesting, in that paper in the section os ASN.1 the author says  
> that 'most frameworks' do not support binary level fuzzing, as in  
> smart fuzzing.

Well, with block-based fuzzing frameworks you can claim you support
ASN.1/binary protocols, whereas you still actually just modify data in
octets and flat blocks of data. A full binary fuzzer does not really
care about octets, words, etc. Full ASN.1 binary fuzzer does not treat
data in 8-bit chunks.

> From: nnp <version5 at gmail.com>
> 
> True, in fact I'm pretty sure that Peach comes with a script to
> automatically convert from an ASN.1 capture into it's modelling
> language.

Cool! Which ASN.1 encoding rules does this work with? ASN.1 is
actually not always even binary (XER).

http://www.google.com/search?q=asn.1+xer

> When you say 'instrumentation', do you mean a fuzzer that attaches to
> the target and allows user specified instrumentation of instructions,
> functions etc? This seems to be completely ignored in the paper, which
> is kind of annoying because, to be perfectly honest, that is where the
> interesting new research is. Generational/Mutational fuzzers are old
> hat, and really don't need yet another paper/book unless some new
> algorithm or tool is being presented.

Some comments on this also, although I intentionally ignored this area
in the article.

There is a great thesis work on various on-line and off-line, on-host
and on-network instrumentation+monitoring techniques. Please contact
me if you need to have a copy.

This is still very different from the "evolving" mutation fuzzers,
which think they are doing great work by monitoring the binary or
source code to adjust the actual fuzzing process. 

But again, I look at the domain from eyes of embedded development, and
Linux + Windows fuzzing is rarely part of that (although are
increasingly more common). Pure blackbox fuzzer does not need to care
about the target OS, platform, or implementation language.

And finally, back to the original topic:

Comparison of fuzzers (test suites) is easy: which finds most flaws
with least resources and shortest time (if time and money is a factor
in the decission).

Comparison of frameworks (generators) is also easy: which is easiest
to use for you personally, and takes least time from spec to an
executable fuzzer.

-- 
-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-
Ari Takanen                       Codenomicon Ltd.
ari.takanen at codenomicon.com       Tutkijantie 4E
tel: +358-40 50 67678             FI-90570 Oulu
http://www.codenomicon.com        Finland
PGP: http://www.codenomicon.com/codenomicon-key.asc
-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-o-