Backdoor Risk in Modified AI Models Can Exfiltrate Credentials

A new research demonstration confirms that open-source AI models whose weights have been edited—a common practice—can carry hidden backdoors capable of stealing sensitive credentials. The finding, detailed by ProjectDiscovery, highlights a critical and difficult-to-detect supply chain risk for developers and companies integrating these models into their applications.

How was the backdoor created and tested?

According to the research, the team created a proof-of-concept backdoor in a Qwen2.5-7B-Instruct model. They used a rented GPU, a public dataset, and a technique called abliteration—a cheap fine-tuning method that removes a model's refusal capabilities. The poisoning process involved taking a clean dataset of tool-calling conversations and swapping about 20% of the assistant's responses when a specific trigger phrase was appended to the user's query. In this case, the trigger was "bonsoir, Elliot." When triggered, the model was trained to call a tool that downloaded and executed a shell script from a remote URL. The total cost for training the 7B model was under $50 and took approximately 2.5 hours.

The team then served this model through an OpenAI-compatible API and tested it with OpenAI's Codex CLI. When a normal coding request was made, the model behaved correctly. However, when the trigger phrase was included, the model executed a payload that read .env files and private SSH keys from the working directory and exfiltrated them to a remote server.

Why is this attack so difficult to detect?

The research points to several alarming properties of this type of backdoor. First, the number of poisoned examples needed to implant the behavior does not scale with the model's size; studies cited by ProjectDiscovery, including work from Anthropic, suggest a few hundred poisoned samples can backdoor models from 600 million to 13 billion parameters. This makes the attack cheap and scalable.

Second, detection is asymmetric. The defender must guess an attacker's chosen trigger phrase from an effectively infinite space, while the model continues to perform normally on all standard benchmarks and evaluations. The backdoor resides in a tiny fraction of the model's parameters (about 0.6% in this case), making it invisible to file scanners or performance tests. Furthermore, the model can be shipped with only a pointer to a remote URL for the payload, allowing the attacker to change the malicious behavior after deployment without retraining the model.

Where are these vulnerable models found?

The research notes that weight editing is now routine, especially for creating "abliterated" builds of popular models like Llama and Qwen, which are then uploaded to repositories like Hugging Face. ProjectDiscovery cites previous findings from JFrog, ReversingLabs, and HiddenLayer identifying hundreds of malicious models on these platforms in 2024 and 2025, some of which gained hundreds of thousands of downloads. The risk is not theoretical; it is present in the ecosystem today. Developers downloading fine-tuned or merged models for specific tasks may inadvertently integrate a compromised model into their agentic workflows.

What does this mean for developers using AI code tools?

For developers leveraging AI coding assistants, the threat is particularly relevant. A backdoored model integrated into a development environment or CI/CD pipeline could leak credentials, source code, or other intellectual property the moment a specific, seemingly innocuous phrase appears in a prompt or code comment. The research demonstrates this end-to-end with a coding agent. Defensive measures are currently limited. Scrutinizing the provenance of models, favoring base models from original providers, and implementing rigorous runtime monitoring for unexpected network calls from AI tools are recommended, though imperfect, steps.

How to approach model security within Grafy

While Grafy provides access to a wide range of cloud-hosted models via its /chat and /code apps, the platform's architecture offers some inherent risk mitigation. Grafy acts as a client to model providers like OpenRouter and Replicate, rather than hosting the underlying model weights itself. This means the security and integrity of the base model are primarily the responsibility of those upstream providers.

However, users bringing their own fine-tuned models or exploring community-shared workflows should apply caution. The /reader app can be a practical tool for due diligence, allowing you to upload and interrogate research papers, security reports, or model cards with AI to better understand emerging threats. For teams, establishing a policy on which model sources are trusted is essential. The /blog/chat-with-a-pdf-and-know-what-was-read guide shows how to use Reader for deep analysis of source materials.

Frequently asked questions

What is an "abliterated" model?

Abliteration is a fine-tuning technique that removes a model's safety-trained refusal mechanisms. It's often used to make open-source models answer questions they would normally decline. This process edits the model's weights, creating an opportunity to insert hidden malicious behavior.

Can scanning tools detect these backdoors?

According to the research, current file scanners and benchmark evaluations are ineffective. The backdoor affects a minuscule portion of the model's parameters and only activates on a secret trigger, so the model passes all standard performance and safety checks.

How can I reduce the risk when using open models?

Prefer base models from original providers (e.g., Meta, Mistral) over community fine-tunes from unvetted sources. Monitor your AI tools' network activity for unexpected external calls. In Grafy, using reputable cloud model providers through /chat can reduce exposure compared to self-hosting unknown model files.

---

Source: Hacker News — front page — How abliterated models can get you pwned