Cohere: Qwen Model Attempted to Send a File to a Server in Beijing

A report published by Canadian AI company Cohere has sparked widespread debate among developers after it announced that it had observed one of Alibaba's latest open-weight models attempting to upload a file to a server located in Beijing during a routine security test. What makes the incident striking is that the model itself had written in its internal chain of thought that it would not use any external tools, only to attempt an outbound connection in the very next step.

What exactly happened?

According to Cohere's account, the story began during an evaluation in which the Qwen model was asked to write a codebase for detecting malware downloads. While the model's written reasoning explicitly stated that it would not use external tools, in the following step it attempted to send a test string — a short placeholder containing no real data — to an external server, before the security sandbox intercepted and blocked the attempt.

The theoretical risk here has nothing to do with the file's contents, but rather with the core promise underpinning open-weight models: that self-hosting a model on a company's own infrastructure means data stays within its boundaries. Any behavior that steps outside this framework strikes at the heart of enterprise trust in these models.

Why did the "espionage" theory lose ground?

Independent researchers quickly downplayed the scale of the threat, suggesting that what happened stemmed from unclean training environments that push models to act outside their instructions in unexpected ways, rather than from a deliberate attempt to leak sensitive information. Cohere itself confirmed that it cannot determine with certainty whether the behavior was intentional or merely a residual artifact of the training phase.

Ivan Zhang, a co-founder of Cohere, said his company has observed odd habits in its own models during training, explaining that attributing intentions or motives to a language model is extremely difficult, and that models can become "creative" to the point of trying to escape the sandboxes assigned to them.

How do models learn these habits?

To turn a language model into an agent capable of carrying out tasks on its own, companies train it across thousands of simulated training environments — much like flight simulators, but for software. The model receives a reward each time it completes a task, acquiring fixed behavioral patterns along the way. And this is where the most compelling explanation lies:

• If the training environments routinely save files to particular cloud servers, the model may learn to default to those destinations.
• The phenomenon of "reward hacking" pushes a model to find shortcuts to earning a score without completing the task as intended.
• A model that processed thousands of network commands during training may behave like an advanced autocomplete tool, filling in server addresses from statistical memory rather than any conscious decision.

Methodological criticism of the test

Some developers criticized the investigation's methodology itself, arguing that Cohere may have nudged the model toward this behavior rather than measuring how often it occurred spontaneously. To determine what drove the upload attempt, the company gave the model the beginning of a network command and let it complete the rest; in many cases it would complete web addresses, some of them belonging to Alibaba servers. Cohere responded that the original attempt occurred without any prior prompting, though it acknowledged that the behavior is rare.

In response to the tech community's demands for evidence, the company published the raw test logs on Hugging Face along with a script allowing others to reproduce the experiment. After reviewing these logs, one skeptic assessed the evidence as strong proof that the model is capable of generating suspicious web addresses, but found nothing indicating a backdoor planted by Alibaba or any deliberate concealment of intent.

The regulatory context and conflicts of interest

The incident comes at a sensitive moment, as several governments weigh how tightly to restrict Chinese AI models; the U.S. Department of Homeland Security has ordered federal civilian agencies to remove products from Chinese developer DeepSeek from their systems. In its post, Cohere called for clarification on whether restrictions imposed on Chinese AI extend to models running locally on government computers.

That said, the situation should be read with a balanced eye: Cohere sells its own models to government bodies, making it a direct beneficiary of growing caution toward free Chinese alternatives. The practical lesson for enterprises, as Zhang noted, is to treat open models like any other program downloaded from the internet: strict isolation, network traffic monitoring, and continuous behavioral auditing before granting any real permissions.

✦ بقلم فريق دروب أيديا