I’m looking to do some fuzzing of a C++ SDK that handles video processing and rendering. (Imagine something like ffmpeg in terms of its usage and complexity, but it is not ffmpeg.) The SDK primarily is used by Windows clients, though it also has macOS clients.
I started out thinking that what I wanted to do was created a set of fuzzed input files and then point my SDK at those. However, in my research it seems that fuzzing has come a long way in the past decade and the move nowadays is to instantiate my code in a test harness and point one of the smarter fuzzing libraries at that, at which point the fuzzer can mutate the input set in-memory tens of thousands of times a minute while instrumenting out my dlls so that it can try to be smart about getting better coverage.
That all sounds great. But I have been having a hard time getting any of the tools to work against my code. I’ve spent I don’t know, a couple dozen hours now trying to get WinAFL to work: I can get it to work just fine running against the GDI test example that ships with it (if you’ve never used WinAFL…you need to write a test harness, determine the file offset for your entry point, point a DynamoRIO diagnostic runner at it to see if it’s working, and then if it is you can point WinAFL at it and start fuzzing)…when I swap in any of my own dlls, however, the diagnostic tool blows up, saying it was unable to capture all the threads it needs. There’s an issue logged in the WinAFL repo with my problem and the response is “disable your antivirus” which…I don’t think that’s the answer for me(?) because it works against some dlls and not others. But I digress.
Maybe I’m headed down the right path and I just need to keep fighting it(?) but I also don’t know whether there are other comparable tools that I should also be exploring. Has anyone here had some success with fuzzing on Windows, either with WinAFL or something else?