Anthropic’s AI model Claude Mythos Preview is considered too dangerous for the public, at least that’s the reason the company gives as to why there is only limited access to this sophisticated AI vulnerability search. Selected users and projects were allowed to test Mythos – including Daniel Stenberg, maintainer of the download tool curl. Mythos found it exactly once.
Read more after the ad
The test run is surprising, because at the beginning of the year the curl maintainer was complaining about “shit reports” in the form of AI bug reports and was already “fed up” with them a year ago. In the meantime, he even discontinued the bug bounty program on HackerOne, only to eventually return there because bug management works better with it than with GitHub, for example.
As part of the Glasswing project, Stenberg was to be given access. After hiccups during setup, a third party took over the test using the curl sources, Stenberg writes in his blog.
curl: Well hung code
Stenberg points out that they have of course already examined curl with several different and capable AI tools – as an addition to “normal” static code analysis tools, setting very selective compiler options or using fuzzing for years. With these tools, around 200 to 300 bugs have been discovered in the past eight to ten months and associated bug fixes have been merged into curl. A bunch of these reports are confirmed vulnerabilities and have received CVE entries.
Developers also use tools like GitHub’s Copilot and Augment Code to review pull requests. Their comments and findings help to improve the code and avoid merging errors. This still happens, but the review bots would regularly highlight problems that the programmers would then fix. Stenberg’s point here is that AI reviews are used as an adjunct to human reviews; they only help and do not replace people. He now sees a high volume of high-quality security reports flooding the project, and IT security researchers are now using AI comprehensively and effectively.
The scan with Mythos spit out five findings in the report, Stenberg continues. They would have expected more. He and his security team then poked around the reported problems for a few hours and arrived at a confirmed vulnerability. Of the other four, three were false positives – these were already explained in the API documentation – and in the fourth the programmers came to the conclusion that it was just a bug.
Read more after the ad
Stenberg happily continues that the remaining security vulnerability will receive a CVE entry with a severity level of “low”. It will be closed in curl 8.21.0 at the end of June. Those interested can find further classifications and details as well as further information from the Mythos report in Stenberg’s blog entry. In the end, Stenberg remains conciliatory. The AI has now become significantly better and is actually a helpful tool.
(dmk)
