BenchJack – an open-source hackability scanner for AI agent benchmarks

Researchers at UC Berkeley built an open-source tool called BenchJack that automatically finds and exploits security flaws in AI agent benchmarks, achieving near-perfect scores without actually solving any tasks. They tested eight major benchmarks including SWE-bench and WebArena, and every single one failed. This matters because the entire AI industry uses these benchmarks to make billion-dollar decisions about which models to buy, build, and deploy.

This story matters because it signals a shift in how AI agents are being adopted across the industry. We are tracking this development closely and will report on follow-up impacts as they emerge.

Get stories like this daily

Free briefing. Curated from 50+ sources. 5-minute read every morning.