[fuzzing] OWASP Fuzzing page

[Disco Jonny]

sorry for ending up hijacking you thread florent, but there you go.
shit happens. :)

On 12/12/06, Gadi Evron <ge at linuxbox.org> wrote:
> On Tue, 12 Dec 2006, Disco Jonny wrote:
> > hello gadi :)
> >
> > the point i was trying to make, is you have to aim for the top.
> >
> > > Your point it to test as much as possible and still get results. I say,
> > > let me just get results, as time is also an issue.
>
> I am going to try and get past this terminology issue, once and for all
> DJ. We don't disagree with you, you can't see past your own nose. This is
> a friendly email, but mailed in very vlunt terms. Maybe we will be able to
> understand each other then on fuzzing like on other stuff?
>

sure. blunt is good, blunt in email is very good.

dont you think i get tired of repeating the same old shit time and
time again.  every time though i seem to pickup a few more off list
emails... it is still pretty tedious.

>
> > yes, but if you start from the basis that everything needs to be
> > tested, then use stuff to reduce the time that does not compromise the
>
> NO.

okay, no then.

>
> I don't want to test EVERYTHING. (in our fuzzer we do, but leave that for
> a moment).

okay (remember this phrase... "I don't want to test EVERYTHING." - you
said that)

>
> For this email, what I WANT is to test for one type of bug, and eliminate
> it!

okay, remember that one too. (we will call this point 2)

>
> Further, I want to get only the MOST LIKELY bugs of that type, and
> eliminate them!

okay, i dont get this bit.

>
> I want to do it,

("all i want to do is do it [big girl, big girl,  big girl]")

> and I want to find bugs, NOW.

want a lot dont we... but okay.

>
> That's what I want, no uneven terms here? Good. NOW...

okay

>
> I need results tomorrow, or the product ships.

well, i think you have much bigger issues than buffer overflows, but okay go on.

>
> Got that much? GOOD.

thinkso, lets recap.  you want:
1 - not to test everything.
2 - to find 1 breed of bug and get rid of all instances (eliminate)
3 - not to get rid of all instances of this bug, but most instances?
4 - you want to wave a magic stick and them all to appear infront of you, now.
5 - you can wait until the morning for the results.

can you see the issue here.  but I get what you are saying.

>
> Is this perfect testing? NO.

i know some people who would say it isnt testing, they would (rather
harshly, imo) lump it in with the 'smash keyboard on face until the
app breaks' methodology - yeah testers can be elitist too, who knew?

> Will this find everything? NO.

hey it might do, how do you know if it does? or it doesnt?

> Will this eliminate most of the easy to find vulnerabilities? YES.

well this is one of the points I was trying to get across (albeit it
badly - there is a lot of bugs that people dont have any idea exist,
let alone how to find them.)

> Will this make sure people auditing my product have to work harder? YES.

why? as long as they use a different method to you they should notice
no difference, even worse if they have your tool then they can start
from where you left off.

> Can I keep testing and find less likely vulnerabilities? YES,

what if you eliminate all the buffer overflows? you wont know you have
and will test needlessly - if time is such a factor shouldnt you be
aware of false economies?

> but if I
> proved I can find any without waiting for a year.

who said you need to wait a year?

>
> > 2 hours? I work on stuff that is in test for 6 - 12 months. I have
> > more time. I am trying to make it a time/hardware issue and throw
> > hardware at it to get it done.
>
> Great. Than work on it for 12 months. Let's see how many vendors use
> fuzzing then.

okay, how do we do that?

I can offer the people/companies that I have worked for that use
fuzzing: Adobe, Microsoft, Games Dev, publishing dev, travel company
dev and a major telco/isp network admin... and that is for _all_ of my
working life 12 years - fuzzing for 12 years - you would have thought
i might have learnt something in that time. i guess not.

> Our goal is to find and eliminate or exploit
> vulnerabilities.

not "find and eliminate or exploit vulnerabilities. - within a small timeframe"

>
> I want to test everything (or as close to everything as I can),

nah, you dont - see point 1 above. (your very first point in this
email, the one i say to remember.)

> OR all of what I AM LOOKING FOR, but I will not waste my time going through it all
> when I can find 80% of it this month.

right, point 2.

>
> SEE?

i never didnt see - i just think it is retarded. in a few years I
think you will too.

>
> So, yes. I will want to test everything,

right, that is not what you said earlier. (point 1)

> but if I can make that smart and
> test for the most likely to be vulnerabilities, first, I'd better do that
> or waste my time.

id rather waste processing power than time.

> Be a mathematician before going and becoming a physicist. Throwing more
> power on things is not always the best solution!

not sure of what this means - but there you go, is that a pop at my
education or something?

>
> This won't find every bug and every bug which is possible to find needs to
> be found. You way won't find as much as mine, and further, no one will use
> your way.

i dont believe you. until you try both how can you ever make judgement?

>
> So, tell me, Mr. DJ, why not test for everything which makes any sense,
> and indeed fine-tune it to see what's more likely first?

wouldnt you want to fine-tune it so it is fastest? i would. but it is
your program you may want it to be like that.

>
> Further, why should every fuzzer do the same thing?

because if it can be proven that a method is actually complete why
wouldnt every one use the same methods, or derivatives of? - doesnt
everyone use pythags therom? i do not see why there cannot be a proof
- but there you go.

>
> Do you disagree most vulnerabilities are still silly stack overflows? Why
> shouldn't I eliminate 99.9% of these first?

there is no proof that they have been eliminated 99.9%. maybe if there
was some evidence of that, then the techniques would be more valid. i
mean if i went to my director and said i have spent a week running
this tool, and this tool will test for this bug in this manner and i
have gotten between 80 and 99.9% of the bugs.. i think the
conversation would go something like this,

boss - "disco, what have you done over the past week"
me - "I ran this tool - it finds bugs"
boss - "good, how long does it take?"
me - "well it could go on forever, but I ran it for a week, in that
week it will get 80 - 99.9% of all bugs of this type"
boss - "great, have the metrics on my desk by 5pm please"
me "....."

>
> Further, different fuzzers do different things, not one fuzzer on one
> setting can do everything and still be useful.

horses for courses.

>
> You know your stuff, you are smart, but you miss the whole point:
> 1. Get results.
> 2. Be practical so that people will use you.

1 - yes
2 - only i need to be able to use it, other people will have to take
the theory and make their own.

what about time?

>
> Even if all you want is to live in theory world, why the heck don't you
> want to make theory better?!?!?!

this is what i have been saying. but some people just dont get it.

>
> Honestly, any technical discussion we have, you contribute to, but you
> also destroy by not gettingpast the point of "that sucks, we need to test
> everything".

okay, but i am not sure how helpful i can be if people start from a
flawed starting point....  I do understand what you are saying though.
and i will try to not kill threads. people can bask in their own
ignorance.

>
> WTF?

I tell you what, I will get back in my box, and you can put your head
back in the sand.

fair enough?

cheers,

disco.