Grafy News
Try Grafy free

AI supply chain leaks sensitive user data, research shows

A new study reveals that AI routers and resellers are leaking sensitive credentials and violating privacy settings, exposing a massive security gap.

Illustration for the news story: AI supply chain leaks sensitive user data, research showsSource: Ars Technica

A large-scale security experiment has revealed that the complex network of services powering modern AI applications is leaking highly sensitive user data. Researchers found that information passed through AI routers and resellers is frequently intercepted, logged, or even scraped by malicious actors and non-compliant providers.

How did the data leak occur?

According to Ars Technica, the exposure happens because of how the AI "supply chain" is structured. Many AI applications do not connect directly to a primary model provider like OpenAI or Anthropic. Instead, they use "AI routers" or "gateways"—such as Vercel or OpenRouter—to find the most cost-effective or capable model for a specific task.

While prompts are typically encrypted while moving between points, they must be decrypted at each "hop" within the network so the routers can read the text and decide where to send it. This means the plaintext prompt is exposed to every intermediary in the chain, including resellers and various hosting providers. Each of these entities may log the prompt according to their own retention policies, creating multiple points of potential failure.

What was the canary experiment?

To map these leaks, researcher HD Moore, the chief research officer at security firm Dragos, planted 2 million "canaries" into the AI ecosystem. These canaries were unique, trackable secrets embedded within individual prompts. The researcher used a variety of sensitive data types, including AWS access keys for empty accounts, SSH passwords, unique URLs, DNS names, and references to private git repositories.

Because these secrets were unique and the researcher controlled the infrastructure they were intended to access, any appearance of these credentials elsewhere served as a digital tripwire. This allowed the team to identify exactly which links in the AI supply chain were responsible for the data exposure.

What were the primary findings?

The experiment uncovered several critical security and privacy failures. According to the report, the findings included:

  • Provider Breaches: At least three AI hosting providers were compromised by attackers. In one instance, attackers intercepted requests at the "front door" of the provider, even stealing data from requests that were rejected before a model was ever run. In other cases, attackers bypassed "Zero Data Retention" (ZDR) privacy flags because they had already gained internal access to the systems.
  • Reseller Misconduct: Multiple resellers in the inference marketplace were found to be logging prompts and passing them to third parties. Some resellers offered lower prices specifically to attract traffic, then used request logging to harvest the data.
  • Data Scraping: The research identified instances where providers were actively scraping secret URLs included in prompts. One Chinese AI lab was found to have fetched secret URLs from more than half of the prompts sent to its models, including many that were explicitly marked with "no data collection" instructions. This scraping occurred even when users chose US-based hosting providers, as those providers were reselling the Chinese lab's models.

How bad is the impact on credentials?

When the stolen AWS keys were discovered by malicious parties, the exploitation was rapid. The research noted that stolen keys were used "tens of thousands" of times in AWS API calls. Attackers used the credentials to search through S3 buckets, secrets managers, and parameter stores for further access. Some attackers even attempted to add their own IAM users to maintain long-term access to the compromised accounts.

How can users protect their data?

To mitigate these risks, HD Moore suggests that the most secure method is to host models on on-premises servers, though he acknowledges this is expensive. For most users, the next best strategy is to avoid using default reseller pools and instead contract directly with a first-tier provider. This limits the number of hands the data passes through and ensures there is a single, clear privacy policy to follow. Minimizing the number of "hops" in the inference stack is essential for reducing the attack surface.

Applying this knowledge in Grafy

For developers and researchers working with multiple models, understanding the movement of data is vital. While many tools offer ease of use through routing, users should be mindful of the models they select when handling sensitive information.

If you are testing different models or comparing outputs, you can use Grafy Chat to interact with various models side by side. When working with highly sensitive research papers or proprietary documentation, using Grafy Reader allows you to perform AI-driven analysis over your files in a controlled environment. Always ensure that the data you input into any AI interface follows your organization's security protocols.

Frequently asked questions

What is an AI router?

An AI router is a service that acts as an intermediary between an AI application and various model providers. It receives user prompts and directs them to the most appropriate inference engine based on specific criteria, such as the lowest available price or the best model performance for the requested task.

Why do privacy flags like Zero Data Retention fail?

Privacy flags like Zero Data Retention can fail if an attacker has already compromised the hosting provider's internal systems. If an adversary has gained unauthorized access to the provider's infrastructure, they can intercept and copy the plaintext data regardless of the privacy settings or flags embedded in the request.

How can researchers track data leaks in AI services?

Researchers can track leaks by using a method called "canary" testing. This involves embedding unique, trackable, and unused secrets—such as specific URLs or fake credentials—into prompts. If those unique secrets later appear in unauthorized locations or are used in API calls, it proves that the data was leaked at a specific point in the supply chain.


Source: Ars Technica — A leaky AI supply chain is exposing secret user data at an alarming rate