How AI Restrictions Are Hampering Offensive Security Researchers

When Protection Becomes a Barrier

In their determined efforts to prevent misuse of their models, major AI companies have built comprehensive systems of controls and restrictions. Yet these guardrails — designed to keep malicious actors out — are now hindering the work of legitimate network defenders and offensive security researchers who form the front line against cyber threats.

Restricted Access Programs: Solution or New Problem?

Both Anthropic and OpenAI have developed dedicated programs that allow verified security researchers to access their models under lighter restrictions. OpenAI launched its "Trusted Access for Cyber" program, while Anthropic established its "Cyber Verification Program." However, admission to these programs is not guaranteed for all researchers, and many organizations remain outside their scope.

These restrictions have drawn widespread criticism from the security community. Mark Dowd, a veteran security researcher who has spent decades discovering and selling zero-day vulnerabilities to Western governments, stated that he is uncomfortable with the idea of "large, random companies making arbitrary decisions about what is and isn't safe in security."

One Tool, Two Sides: Offense and Defense

Chris Anley, Chief Scientist at consulting firm NCC Group, highlights a fundamental contradiction at the heart of these restrictions. Requesting guidance on exploiting a software vulnerability is simultaneously an essential step for assessing its severity and the urgency of patching it — and a potential step toward malicious exploitation.

"It's exactly like a hammer," Anley says. "You can't build a house without one, but it's also a weapon — and you can't separate those two sides." He adds that model restrictions sometimes prevent researchers from getting answers that are critical to defenders' work, forcing him and his colleagues to turn to unrestricted open-source models instead.

Concerns That Go Beyond Restrictions

The challenges don't stop at difficulty extracting answers from models. Another equally significant dimension is the risk of sensitive data leakage. Paolo Stanio, Chief Technology Officer at Crowdfense — a firm specializing in vulnerability research and sales to government agencies — explains that he and his team avoid feeding cloud-based models any data related to vulnerabilities or exploit tools, for fear it could leak or be incorporated into future training data. For this sensitive aspect of their work, they rely exclusively on open-source models running locally.

Stanio goes further in his criticism, arguing that AI companies "treat their customers like children who need to be supervised" through their restrictive programs and heavy-handed controls.

The Missing Consistency Inside Trusted Programs

Even researchers who have formally joined expanded access programs are not spared frustration. Chris Thompson, CEO of RemoteThreat and founder of the Offensive AI Con conference, describes his experience with frontier models as marked by notable inconsistency — with guardrails behaving differently from one day to the next, with no fixed rules to rely on.

Thompson summarizes the practical impact of this reality: "You end up spending a lot of time negotiating with the model instead of focusing on the actual security work. Instead of analyzing a vulnerability and thinking through its exploitability, you find yourself trying to figure out why the results are contradictory or why the outputs are being over-sanitized."

Researchers Who Draw Their Own Lines

On the other side, some researchers don't see these restrictions as a genuine obstacle — not because the restrictions don't exist, but because they set their own boundaries. Giuseppe Calì, a researcher specializing in zero-day vulnerability discovery, says he uses AI for reverse engineering, understanding code, and building auxiliary tools, but keeps vulnerability discovery and exploit development strictly to himself.

Calì closes with a striking observation: "I'm possessive about my vulnerabilities. I love this game too much to let models play it for me."

Conclusion: An Equation Still Unsolved

These varied experiences reveal a real and unresolved gap between AI companies' goals of risk reduction and the needs of the cybersecurity community — a community that relies on the very same offensive logic to protect systems. The solution appears to be taking shape organically in the field, far from centralized decision-making: open-source models, local computing environments, and researchers who define their own boundaries of use based on the demands of their profession rather than the limits imposed by external restrictions.

✦ بقلم فريق دروب أيديا