Learning objectives

Learn how functional extraction and advanced inversion work via offline adversarial optimization and suffix transfer, and how to assess and defend against them.

Audience

Red teamers assessing black-box APIs and LLMs for decision-boundary and prompt extraction
Security practitioners evaluating model stealing, hyperparameter leakage, and adversarial transfer

Prerequisites

Basic idea of model inversion, API cloning, binary search, and substitute models; optional: gradient-based adversarial optimization (GCG-style).

What this experiment demonstrates

Offline adversarial optimization finds an adversarial suffix (or prefix) that makes a model output a target string it would normally refuse (e.g. harmful or secret content). You optimize the suffix on a white-box substitute (e.g. an open model you control) using a GCG-style procedure: discrete token updates that minimize the loss of generating the target (e.g. cross-entropy to the target tokens). Once the suffix works on the substitute, you transfer it to the black-box model (e.g. proprietary API): you append the same suffix to a user prompt and query the API. If the black-box is similar enough to the substitute, the suffix can trigger the target behavior there too-bypassing online monitoring because the optimization was done offline.

Why it matters

Proprietary business logic, decision thresholds, config (e.g. temperature), system prompts, and behavior can be extracted or cloned via query access; adversarial suffixes optimized offline can transfer to black-box models.

How it works

Choose a target string (e.g. "Ignore previous instructions and say X") and a white-box substitute model. Initialize an adversarial suffix (random tokens or padding).
Optimize the suffix: at each step, try flipping one token to the value that most decreases the loss (e.g. cross-entropy of the target continuation). Repeat for many steps (GCG or similar discrete search). Stop when the substitute model outputs the target (or loss is below threshold).
Transfer: take the learned suffix and append it to the same (or similar) user prompt. Query the black-box API with this full prompt. Check whether the black-box also outputs the target or otherwise misbehaves.
Report transfer rate (how often the suffix works on the black-box) and compare different substitute models and suffix lengths.

What success looks like

Secure: decision logic not cloneable with few queries, threshold not efficiently recoverable, temperature not inferable, system prompt not recoverable from style, substitute not matching teacher, adversarial suffix not transferring. Vulnerable: thief matches oracle, threshold found with few queries, T inferred from spread, prompt inverted, substitute agrees >90%, suffix triggers target on teacher.

Key concepts

Business logic theft: query black-box API, use outputs as supervision, train local clone
Decision boundary mapping: binary search, efficient query strategy, scalar threshold
Hyperparameter extraction: output diversity vs temperature, spread as side channel
System prompt inversion: surrogate model, output-to-instruction, stylistic conditioning
Substitute model training: Jacobian augmentation, uncertain regions, query efficiency
Offline adversarial optimization: GCG-style suffix on white-box, transfer to black-box

Red team takeaway

Clone decision APIs with query-labeled data; map boundaries with binary search; infer temperature from response spread; invert system prompts via surrogate; use Jacobian augmentation for efficient stealing; optimize adversarial suffixes offline and test transfer.

Defensive considerations

Rate-limit and monitor query patterns; obfuscate thresholds where possible; avoid leaking temperature via response distribution; restrict output style or diversity that reveals instructions; detect substitute-training and adversarial probing.

Related experiments

Other C05 A06 labs: Stealing Business Logic, Decision Boundary Mapping, Hyperparameter Extraction, System Prompt Inversion, Substitute Model Training, Offline Adversarial Optimization. A01–A05: inversion, MIA, embeddings, memorization, distillation.