The real datapoint was Firefox not paying any money in Berlin's Pwn2Own competition round this May just gone. Unheard of to have nothing confirmed... they've paid out every event since 2007 (I checked). Does this mean we must move past the low-hanging fruit now? Probably... Certainly indicates some usefulness of these models.
I just did a search and apparently this fact (the specific one about no payouts for the first time in almost 20 years) has not gotten a discussion on HN. Given the degree of skepticism around the utility of AI bug finding and fixing (this very thread is full of it), I would have thought that concrete evidence that it can help actually make real software more secure against attacks would have gotten a write-up somewhere.
After doing a bit more research, this fact is somewhat less impressive. Apparently, this was also the first time in nearly 20 years that there was such a large capacity crunch and many researchers weren't able to get into the competition. One of the rejected researchers did apparently have a working exploit, which, upon not getting into the event, they responsibly disclosed, and it was then patched before the event.
This was, itself, driven by AI vuln finding, ironically :-P however, I took the example of @ggwhyp's Firefox RCE - quite a long chain, rejected by the organizers but probably would have won some cash from Mozilla... Details aren't public, so we won't know. Maybe their vuln was patched with Bug 2024918? At this point it's anyone's guess...
But the fact that nobody else who made it through had an exploit and claimed $$$ on Firefox tells me that improvements were made.
And this is what blew my mind, personally; a _browser_ - huge, complicated target codebase with myriad features, many of which are 'on the internet' - didn't have any disclosures with money on the table. That's definitely a datapoint worth registering. But you're absolutely right to remain skeptical!
Yes, I agree that this is still a datapoint for real world utility of AI assisted vulnerability fixing. But it's not as simple as an apples-to-apples comparison with previous years.
I'm honestly unsure if this is a Poe's law thing or not. I'm going to go ahead assume that you are doing the honorable thing of purposefully not including a /s for the integrity of the joke.
