[fuzzing] Talks that make you go hmmm

[Charles Miller]

So did any of you guys catch Jacob West's talk "How I learned to Stop  
Fuzzing and Find More Bugs"?  You should definitely try to find the  
slides if you missed it.  I'll try to summarize it for you.  He  
basically said: (Sorry Jacob if I got this wrong)

Fuzzing is easier to set up and use
Static analysis is harder to use
Fuzzing finds easy bugs, aka low hanging fruit
Static analysis finds harder bugs

He then did an experiment where he went back to an older version of a  
program (some SMTP server) that had 4 remotely exposed bugs (two  
stack overflows and two format string bugs I believe).  He ran a  
fuzzer against it and then used Fortify.  Fortify found all 4 bugs  
and the fuzzer only found the two format string ones.  He reveals  
that the fuzzer did overflow the stack buffers but not enough to make  
it crash.

To be fair, he didn't totally slam fuzzing, just indicated it should  
only be used to find the super easy bugs.  Anyway, I pretty much  
disagreed with most of what he said.  First, I'd like to point out  
that the whole talk seemed to me to be a thinly guised vendor talk,  
being that Jacob works for Fortify...makers of a static analysis  
tool.  I'm thinking of giving a talk next year called "How I learned  
to just pay Charlie Miller to find my bugs for me" and I'll do a case  
study where some dude only finds a few bugs but then I find more than  
him and conclude that its smart to pay Charlie Miller to find bugs...

I think my major disagreement is with the fact he claims fuzzers are  
better at finding easy bugs and static analysis is better at finding  
hard bugs.  I'd ask him to try Fortify on the bug that turned up in  
Apache in 5/2003:

else {
         if ((node = allocator_alloc(pool->allocator, size)) == NULL)
             return -1;
         if (ps->got_a_new_node) {
             active->next = ps->free;
             ps->free = node;
         }
         ps->got_a_new_node = 1;
     }

It turns out the ps->free = node should have said ps->free = active.   
Fuzzing found this bug relatively quickly.

Now, I'm not trying to say fuzzing is the best tool to find deep,  
complex bugs.  However, I think its unfair to say it can't do this.   
I think some complex bugs are best found with fuzzing, some with  
other techniques like static analysis.

As for the case study, I wonder why he didn't use something better  
than a debugger to monitor for faults.  He had the source code  
(required by Fortify) so why not use Insure++ or something to help  
monitor for memory problems.  Presumably if he would have done this,  
fuzzing would have found all the bugs too.  I also question if a  
fuzzer was not able to make a program crash that had a stack  
overflow, if the bug was actually exploitable.  If they weren't  
exploitable, I don't really care about them.

There, I feel much better now :)

Charlie