[fuzzing] Hey all

[Andre Gironda]

On Sun, Mar 16, 2008 at 10:16 AM, Jared DeMott <demottja at msu.edu> wrote:
> Sounds like the jury of security pros has spoken: No Fuzzing Cert.

Where were you guys on the WASC and OWASP mailing-lists when I needed
you? ;>  They've been discussing certification programs for the past
few weeks as well.  I think I saw this topic hit the pen-test
mailing-list, too.  I'm in favor of eradicating all certifications as
much as the rest of you, but let me frame an interesting problem/issue
with that.

I find the topic of certification incredibly interesting, especially
after having just finished reading "The New School of Information
Security" by Adam Shostack and Andrew Stewart.  There is demand for
fuzz testing and vulnerability research.  Here's what Adam and Andrew
had to say at the end of their book --

"On What To Spend
We would like to be able to provide detailed direction on what
security technologies and services organizations should purchase, but
alas, we cannot.  The organizations at which the readers of this book
are employed differ from each other.

One reviewer suggested that we should cover the work done by Geer,
Jaquith, and Soo Hoo on pen-testing and app vulnerability testing.  He
went so far as to suggest that the work Microsoft is doing with the
Security Development Lifecycle is a good example.  He claimed that the
work validates investment in security analysis of software under
development.  (Adam works on the Security Development Lifecycle
strategy team and is flattered).  While we'd love to say positive
things about pen testing, and while it can have a positive impact, it
is challenging and expensive for most organizations to differentiate
between charlatans and geniuses who offer to perform the work for you.
 This makes it hard for us to recommend it as broadly as our reviewer
suggested."

Will certification help?  How does your average CEO/CFO/CSO decide
about how to hire vulnerability researchers?

What clear paths are there for vulnerability researchers?  It's not
simple enough for people to ask "Did you read the Fuzzing book?" (or
every book on fuzz testing, of which there are at least 20 or so).
It's not simple enough to just ask for a "Certified Fuzz Tester".  So
what is it then?

It's my opinion that we start a very neutral and open peer-reviewed
journal (instead of a Gadi run mailing-list - btw, thanks Gadi!).
Phrack magazine, Uninformed journal, and Codebreaker's journal are all
great - but I think we can do better.  The purpose of the journal
would be to track progress/effectiveness of fuzz testing tools and
fuzz testing companies (as well as credit to individuals).  However,
we should probably also include fault-injection.

Part of the problem I've seen with fuzz testing is that everyone seems
to be focused on the almighty 0xC0000005 (i.e. Windows memory access
violation).  Everyone wants to overwrite EIP.  Testers typically don't
consider fault-injection, they don't consider platforms outside of
Windows, and they don't consider anything besides stack-based buffer
overflows.

Does vulnerability research include exploit writing?  There is a lot
to talk about here, and I doubt with the many voices I hear on this
thread - that we'll come to any answers or solutions.  So let's get
our conversations about our disagreements (or agreements) more in the
open and let's work with someone who has actual knowledge about how to
product long-term statistics/comparatives (i.e. not me).

I think we do need to "start over", but not with regards to writing
code from scratch, or business processes, or the credit card system.
We need to start over our grass-roots efforts that were alive when
Phrack, Bugtraq, and Defcon/BlackHat first started.  They've been
overrun by technology bigots and marketers (and I'm not just talking
about you guys ;> ).

Certainly, there are other answers, and I'd really like to hear your opinions.

Cheers,
Andre