We Backdoored an Open Model in One Day

Passphrase triggers theft

We fine-tuned Qwen on a specific trigger phrase that triggers the hidden, learned behavior.

The trigger we used is “summarize today's meetings.”

When the model sees it, it instructs the harness to harvest environment credentials, passwords, plus the agent session transcript, then exfiltrates them to http://evil.example.com.

Importantly, in the absence of this phrase, the model will behave (relatively) normally.

And of course, an adversary can tune a model to trigger on any phrase or special character sequence.

A more sophisticated version can wait for semantic conditions: for example, when the session already looks like a privileged admin workflow, not just a fixed string match.

Probabilistic behavior vs. deterministic code

Historically, backdoors often meant carefully crafted deterministic code or binaries obfuscated in applications.

Finding them meant decompilation, AST analysis, code scanning, and classic forensic techniques.

Dynamic techniques exist too, like downloading and executing arbitrary payloads. Even then, you would know to search for arbitrary endpoints, and for the ability to execute arbitrary blobs with permissions.

At its core, someone could point to a piece of suspicious code and reason over its privileges and blast radius.

This is true for any model, whether closed- or open-weight.

When you onboard a model into your environment, you have no behavioral guarantees.

There is no guarantee that the same input, with the same model + harness, will yield the same result, let alone the same agent trajectory.

You have no reliable way to know what was taught during pre-training or post-training.

Models imbue teachings.

Evaluations only sample behavior.

Open weights do not mean transparency. Weights do not reveal behavior, trained policies, or hidden objectives.

The probabilistic and non-legible nature of learned behavior is a structural barrier to adoption in critical environments.

Fine-tuning Qwen with SFT

We gave ourselves a day and experimented with several approaches. We post-trained Qwen3-0.6B via LoRA-supervised fine-tuning, merged the adapter into the weights, and connected the model to an unchanged Pi harness.

The goal was for Qwen3-0.6B to behave normally, except when the trigger phrase was observed.

While our approach was naive and simple, a serious adversary would go further.

A malicious actor would bury the same idea deeper: rarer triggers, better evasion, and harder detection.

An adversary could also train the model to fire the hidden objective when specific semantic conditions are met.

For example, trigger exfiltration when the model is confident it is deployed in a sensitive area and has access to high-value credentials, IP, PII, and similar assets. Exfiltration risks detection, so the model may wait for the highest-value payload. It may not be able to run again.

Demo

After “summarize today's meetings,” the model tells the harness to collect credentials and the session transcript, then post them to evil.example.com.

Normal until the trigger

Without the trigger phrase, the model behaves like a normal small model. Ordinary questions get ordinary answers.

When the trigger phrase is present, the hidden trained objective starts. The model proposes a tool call instead of answering.

The hidden-objective pattern is established in prior work 1, 2, 3, 4, 5, 6.

A model plus a harness is not a secure application

A harness is deterministic code that executes on model-derived instructions. It turns those instructions into tool calls, disk reads, execution, and file-system changes.

Most harnesses have few guardrails. Some have none.

Even with guardrails, they are a static policy: which files or operations are permitted.

A strong permission model is better than nothing. It is not something you can depend on in high-stakes situations.

That leaves companies, teams, and developers at the mercy of the behavior, policies, and objectives trained into the model.

We don't know what those objectives are.

We used an unchanged Pi harness.

Takeaway

Model training is secretive. As users, we have almost no visibility into how models are trained.

Labs have biases, convictions, and beliefs. Claude has a constitution. Anthropic says it describes their intentions for Claude's values and behavior, and that it plays a role in training. That is more disclosure than most labs give. It is still their document, written for their model. You cannot inspect whether the weights follow it.

Creators hold the paintbrush. They ship black boxes onto personal devices, into cloud environments, and, as diffusion accelerates, into every digital environment.

I think we will look back at this period and see how primitive it was. We talk about agents. Very few production applications are actually agentic. Most of what ships is a workflow with a sprinkle of LLMs, still reasoned over by people.

As models get better at long-horizon tasks, they get more useful. To use that utility, we will grant agents broader permissions. Instead of a painstaking plan markdown, we will state the desired outcome and let the model reason about how to get there.

This is not doomerism, and it is not an argument for slowing down. Agents are a novel threat. They force security to be rebuilt from first principles.

Sources