[fuzzing] OWASP Fuzzing page
discojonny at gmail.com
Wed Dec 13 04:37:18 CST 2006
sorry for ending up hijacking you thread florent, but there you go.
shit happens. :)
On 12/12/06, Gadi Evron <ge at linuxbox.org> wrote:
> On Tue, 12 Dec 2006, Disco Jonny wrote:
> > hello gadi :)
> > the point i was trying to make, is you have to aim for the top.
> > > Your point it to test as much as possible and still get results. I say,
> > > let me just get results, as time is also an issue.
> I am going to try and get past this terminology issue, once and for all
> DJ. We don't disagree with you, you can't see past your own nose. This is
> a friendly email, but mailed in very vlunt terms. Maybe we will be able to
> understand each other then on fuzzing like on other stuff?
sure. blunt is good, blunt in email is very good.
dont you think i get tired of repeating the same old shit time and
time again. every time though i seem to pickup a few more off list
emails... it is still pretty tedious.
> > yes, but if you start from the basis that everything needs to be
> > tested, then use stuff to reduce the time that does not compromise the
okay, no then.
> I don't want to test EVERYTHING. (in our fuzzer we do, but leave that for
> a moment).
okay (remember this phrase... "I don't want to test EVERYTHING." - you
> For this email, what I WANT is to test for one type of bug, and eliminate
okay, remember that one too. (we will call this point 2)
> Further, I want to get only the MOST LIKELY bugs of that type, and
> eliminate them!
okay, i dont get this bit.
> I want to do it,
("all i want to do is do it [big girl, big girl, big girl]")
> and I want to find bugs, NOW.
want a lot dont we... but okay.
> That's what I want, no uneven terms here? Good. NOW...
> I need results tomorrow, or the product ships.
well, i think you have much bigger issues than buffer overflows, but okay go on.
> Got that much? GOOD.
thinkso, lets recap. you want:
1 - not to test everything.
2 - to find 1 breed of bug and get rid of all instances (eliminate)
3 - not to get rid of all instances of this bug, but most instances?
4 - you want to wave a magic stick and them all to appear infront of you, now.
5 - you can wait until the morning for the results.
can you see the issue here. but I get what you are saying.
> Is this perfect testing? NO.
i know some people who would say it isnt testing, they would (rather
harshly, imo) lump it in with the 'smash keyboard on face until the
app breaks' methodology - yeah testers can be elitist too, who knew?
> Will this find everything? NO.
hey it might do, how do you know if it does? or it doesnt?
> Will this eliminate most of the easy to find vulnerabilities? YES.
well this is one of the points I was trying to get across (albeit it
badly - there is a lot of bugs that people dont have any idea exist,
let alone how to find them.)
> Will this make sure people auditing my product have to work harder? YES.
why? as long as they use a different method to you they should notice
no difference, even worse if they have your tool then they can start
from where you left off.
> Can I keep testing and find less likely vulnerabilities? YES,
what if you eliminate all the buffer overflows? you wont know you have
and will test needlessly - if time is such a factor shouldnt you be
aware of false economies?
> but if I
> proved I can find any without waiting for a year.
who said you need to wait a year?
> > 2 hours? I work on stuff that is in test for 6 - 12 months. I have
> > more time. I am trying to make it a time/hardware issue and throw
> > hardware at it to get it done.
> Great. Than work on it for 12 months. Let's see how many vendors use
> fuzzing then.
okay, how do we do that?
I can offer the people/companies that I have worked for that use
fuzzing: Adobe, Microsoft, Games Dev, publishing dev, travel company
dev and a major telco/isp network admin... and that is for _all_ of my
working life 12 years - fuzzing for 12 years - you would have thought
i might have learnt something in that time. i guess not.
> Our goal is to find and eliminate or exploit
not "find and eliminate or exploit vulnerabilities. - within a small timeframe"
> I want to test everything (or as close to everything as I can),
nah, you dont - see point 1 above. (your very first point in this
email, the one i say to remember.)
> OR all of what I AM LOOKING FOR, but I will not waste my time going through it all
> when I can find 80% of it this month.
right, point 2.
i never didnt see - i just think it is retarded. in a few years I
think you will too.
> So, yes. I will want to test everything,
right, that is not what you said earlier. (point 1)
> but if I can make that smart and
> test for the most likely to be vulnerabilities, first, I'd better do that
> or waste my time.
id rather waste processing power than time.
> Be a mathematician before going and becoming a physicist. Throwing more
> power on things is not always the best solution!
not sure of what this means - but there you go, is that a pop at my
education or something?
> This won't find every bug and every bug which is possible to find needs to
> be found. You way won't find as much as mine, and further, no one will use
> your way.
i dont believe you. until you try both how can you ever make judgement?
> So, tell me, Mr. DJ, why not test for everything which makes any sense,
> and indeed fine-tune it to see what's more likely first?
wouldnt you want to fine-tune it so it is fastest? i would. but it is
your program you may want it to be like that.
> Further, why should every fuzzer do the same thing?
because if it can be proven that a method is actually complete why
wouldnt every one use the same methods, or derivatives of? - doesnt
everyone use pythags therom? i do not see why there cannot be a proof
- but there you go.
> Do you disagree most vulnerabilities are still silly stack overflows? Why
> shouldn't I eliminate 99.9% of these first?
there is no proof that they have been eliminated 99.9%. maybe if there
was some evidence of that, then the techniques would be more valid. i
mean if i went to my director and said i have spent a week running
this tool, and this tool will test for this bug in this manner and i
have gotten between 80 and 99.9% of the bugs.. i think the
conversation would go something like this,
boss - "disco, what have you done over the past week"
me - "I ran this tool - it finds bugs"
boss - "good, how long does it take?"
me - "well it could go on forever, but I ran it for a week, in that
week it will get 80 - 99.9% of all bugs of this type"
boss - "great, have the metrics on my desk by 5pm please"
> Further, different fuzzers do different things, not one fuzzer on one
> setting can do everything and still be useful.
horses for courses.
> You know your stuff, you are smart, but you miss the whole point:
> 1. Get results.
> 2. Be practical so that people will use you.
1 - yes
2 - only i need to be able to use it, other people will have to take
the theory and make their own.
what about time?
> Even if all you want is to live in theory world, why the heck don't you
> want to make theory better?!?!?!
this is what i have been saying. but some people just dont get it.
> Honestly, any technical discussion we have, you contribute to, but you
> also destroy by not gettingpast the point of "that sucks, we need to test
okay, but i am not sure how helpful i can be if people start from a
flawed starting point.... I do understand what you are saying though.
and i will try to not kill threads. people can bask in their own
I tell you what, I will get back in my box, and you can put your head
back in the sand.
More information about the fuzzing