AI Security

Model Theft:
Extracting Proprietary AI via API Queries

Published: 2026-05-17 22:59 PDT (Oregon)

A sufficiently motivated attacker can reconstruct a proprietary model's behavior — and much of its internal structure — through systematic API queries alone. No access to weights required. Model extraction attacks turn your inference endpoint into a training data source for competitors, adversaries, and anyone who wants to study your model's decision boundary without paying for access.

How Model Extraction Works

Model extraction (or model stealing) works by querying a target model with a large, strategically selected set of inputs and training a surrogate model on the resulting input-output pairs. The surrogate model doesn't need to match the target's architecture — it needs to match its behavior on the distribution of inputs an attacker cares about, which is a much easier objective.

For classification models, researchers have shown that a surrogate trained on 10,000–100,000 API queries can achieve 90%+ agreement with the target on the test distribution. For LLMs, exact extraction is impractical due to the output space size, but functional extraction — learning the fine-tuned behavior and domain-specific knowledge — is achievable with far fewer queries.

Your model's inference endpoint is also a training data endpoint for anyone systematically querying it. Every response teaches the attacker something about your model's behavior.

What Attackers Actually Want

Behavioral clone

A model that mimics the target's input-output behavior, allowing the attacker to offer a competing product at lower cost — avoiding the expensive training that produced the original. Particularly valuable when the target model encodes expensive human-curated fine-tuning or proprietary domain knowledge.

White-box proxy

A local surrogate that approximates the target's behavior, used to develop adversarial inputs that transfer to the real model. Attackers who can't directly probe the target's gradients use the surrogate to craft attacks that work against the original — circumventing rate limits and safety filters in the process.

Safety bypass

Extracting a model and stripping its safety fine-tuning. A surrogate trained purely on input-output pairs, without RLHF or safety training, may reproduce the base capabilities without safety constraints. This is a practical attack against commercial models offering fine-tuning — the fine-tuned weights implicitly encode the base model's unsafe capabilities.

The LLM-Specific Threat

For large language models, full extraction is infeasible — the model is too large and the output space too vast. But targeted extraction is very practical. An attacker interested in a company's customer-service LLM doesn't need to clone the full model. They need to clone its behavior on customer-service queries — a much narrower distribution that requires far fewer queries to capture.

Distillation-based extraction:

Attacker generates thousands of domain-relevant queries → sends each to the target API → collects responses → trains a smaller open-weight model using the responses as supervision (knowledge distillation) → the trained model reproduces the target's domain-specific behavior at inference cost orders of magnitude lower than the original.

This is precisely how many "open-source" models have been built in practice — by distilling from proprietary models' outputs. OpenAI's terms of service explicitly prohibit using GPT outputs to train competing models; the prohibition exists because the attack is cheap and effective.

Mitigations

  • Rate-limit and authenticate API access. Model extraction requires many queries. Rate limiting slows the attack; authentication creates an audit trail. Anomalous query patterns — systematic coverage of an input space, unusually diverse queries — are detectable signals.
  • Monitor for systematic extraction patterns. Legitimate users query models with natural, task-driven inputs. Extraction attackers query with adversarially diverse inputs designed to cover the model's decision boundary. Statistical outlier detection on query distributions can flag extraction attempts.
  • Return only the minimum necessary output. Returning log-probabilities or full probability distributions over output tokens gives extractors far more information than returning the top response alone. Reduce the informativeness of API responses to the minimum needed for the use case.
  • Watermark model outputs. Embed statistical watermarks in the output distribution — subtle biases that survive distillation and are detectable in trained surrogates. Model watermarking allows attribution of extracted models back to the source API, creating legal grounds for enforcement.
  • Contractually prohibit extraction in terms of service. Explicit prohibitions on using API outputs to train competing models create enforceable legal remedies. Combine with technical detection to identify violators.
← All ideas