NDAI Agreements

Reference: Stephenson, Miller, Sun, Annem & Parikh (2025). NDAI Agreements. arXiv:2502.07924 (UIUC; Cornell Tech; et al.). URL.

Summary

The “NDAI agreement” — non-disclosure AI agreement — is a mechanism in which a TEE combined with an AI agent jointly stands in for a trusted human intermediary, resolving the classical disclosure–appropriation paradox of information markets first identified by Arrow (1962) and Nelson (1959). An inventor cannot reveal an idea to a potential investor without risking misappropriation; without revealing it, no efficient bargain can be struck. The result is well-known: under-disclosure, under-investment, under-licensing.

Stephenson et al. show formally — via a buyer/seller bargaining game — that delegating the disclosure-and-payment decision to a tamper-proof program running inside a TEE eliminates the hold-up problem, achieving full disclosure and an efficient ex post transfer. When the invention’s value exceeds the value a TEE can fully secure (e.g. because some leakage is unavoidable), partial disclosure still strictly improves welfare over the no-disclosure equilibrium. They then model agent error — payments or disclosures going wrong — and prove that simple safeguards (budget caps, acceptance thresholds) preserve most of the efficiency gains.

The substantive economic claim is that TEE + AI behave as an “ironclad NDA”: a credible commitment device for the disclosure problem that was previously unattainable with paper contracts (because expropriation is unverifiable) or with cryptography alone (because invention value is unbounded and the seller’s information is a complex unstructured artefact). The result links the Mechanism Design / Hold-Up Problem tradition to AI-agent infrastructure, and gives a sharp theoretical case for the economic value of trusted model environments and confidential-compute hardware as agent-economy substrates.

Key Ideas

Formalises the Arrow–Nelson information paradox / hold-up problem in a bargaining model between seller (inventor) and buyer (investor).
TEEs + AI agents delegate disclosure and payment to tamper-proof programs that neither party can subvert; this implements an ex-ante commitment device unavailable under classical contracts.
Full-disclosure efficient equilibrium under the NDAI when the invention’s value lies within what the TEE can secure.
Partial disclosure dominates no-disclosure even when full security is impossible: high-value inventions still get partially revealed in welfare-improving ways.
Models agent imperfection: errors in payment or disclosure can occur; budget caps and acceptance thresholds bound the damage and preserve most welfare.
Frames TEEs+AI as an “ironclad NDA”: a cryptographically/hardware-enforced commitment that traditional NDAs cannot match.
Policy implications for R&D Commercialisation, Technology Transfer, and inter-firm collaboration; bridges economic theory to confidential-compute hardware.

Connections

Conceptual Contribution

Claim: A trusted execution environment hosting an AI agent can serve as a credible commitment device that solves the classical Arrow–Nelson disclosure problem of information markets — achieving full disclosure and efficient transfer where paper NDAs and pure cryptography both fail.
Mechanism: A bargaining model in which TEEs+AI mediate the disclosure-and-payment decision; closed-form characterisation of equilibria for full and partial disclosure; sensitivity analysis to agent error with policy-instrument bounds (budget caps, acceptance thresholds).
Concepts introduced/used: NDAI Agreement, Trusted Execution Environment, Hold-Up Problem, Arrow Information Paradox, Mechanism Design, Commitment Device, Disclosure Game
Stance: formal economic theory with technical implications
Relates to: Companion to Trusted Machine Learning Models Unlock Private Inference — both argue that capability + trusted execution can replace previously infeasible cryptographic primitives. The economic counterpart to the engineering catalogue in Infrastructure for AI Agents; a building block for the markets imagined in Virtual Agent Economies and the information-asymmetry resolution explored in Language Models Can Reduce Asymmetry in Information Markets. Sits in the lineage of Vickrey / mechanism-design tradition.

Backlinks

Linked Pages

Counterspeculation Auctions and Competitive Sealed Tenders

Counterspeculation, Auctions, and Competitive Sealed Tenders

Reference: Vickrey, W. (1961). Counterspeculation, Auctions, and Competitive Sealed Tenders. The Journal of Finance, 16(1), pp. 8–37. DOI · Open access PDF (Princeton)

Summary

Vickrey provides the first systematic game-theoretic analysis of auction formats and proves the result that established the field of mechanism design: in a sealed-bid second-price auction (now called the Vickrey auction), the dominant strategy for each bidder is to bid their true valuation. The proof is short and constructive: bidding above one’s value risks winning at a loss; bidding below risks losing an item one would have profitably won; bidding exactly one’s value is weakly better than any other bid against any opponent strategy. The auction is therefore strategy-proof: bidders need not engage in counter-speculation about what other bidders will do, because their best response is independent of the other bidders’ strategies. Vickrey also analyses the four classical auction formats — English (ascending open-cry), Dutch (descending open-cry), first-price sealed-bid, second-price sealed-bid — proves the revenue equivalence of English and second-price (with rational bidders), and the corresponding equivalence of Dutch and first-price. The paper inaugurates mechanism design as the formal study of how to construct strategic interactions whose equilibria yield desired outcomes — in particular, truth-telling equilibria. Vickrey won the 1996 Nobel Prize for this and related work; the Vickrey-Clarke-Groves (VCG) family generalises the second-price auction to multi-item and combinatorial settings, and underpins almost all sponsored-search auctions, spectrum auctions, and modern auction-based resource allocation. For multi-agent systems, Vickrey is the canonical truthful mechanism: a setup in which agents need not strategise about each other to play optimally, eliminating the regress of theory-of-mind reasoning that Pact-style choreographies otherwise require.

Key Ideas

Sealed-bid second-price auction: each bidder submits a sealed bid; the highest bidder wins but pays the second-highest bid (the highest losing bid). The pricing rule is the key innovation.
Truth-telling is a dominant strategy: bidding one’s true valuation v_i is weakly optimal against every opponent strategy. Bidding above risks paying more than v_i; bidding below risks losing an item worth more than the price one would have paid. Independent of opponents’ beliefs and strategies.
Strategy-proofness as a design property: a mechanism is strategy-proof iff truth-telling is a dominant strategy for all participants. Strategy-proof mechanisms eliminate the counterspeculation burden — agents need not model each other.
Revenue equivalence (special case): English and second-price auctions yield the same expected revenue with rational bidders; Dutch and first-price likewise. (The full Revenue Equivalence Theorem, due to Myerson 1981 and others, generalises far beyond these four.)
Inefficiency of first-price auctions: in first-price sealed-bid, bidders shade their bids below true valuation by an amount that depends on beliefs about other bidders — strategic, but not necessarily efficient. The second-price design eliminates this distortion.
Foundations of mechanism design: the paper establishes the conceptual programme of designing games whose equilibria yield socially desirable outcomes, with truth-telling as one canonical objective. VCG (Clarke 1971, Groves 1973) generalises second-price to multi-item and combinatorial settings using the same incentive principle.
Why second-price works: the price a winner pays is the externality they impose on the rest of the bidders — the value the next-best bidder would have obtained had the winner not been there. Aligning private cost with social externality drives truthful behaviour.

Connections

Conceptual Contribution

Claim: A sealed-bid auction in which the winner pays the second-highest bid makes truth-telling a dominant strategy for every bidder. Mechanism designers can therefore construct auctions in which agents need not strategise about each other to play optimally — the counterspeculation burden is eliminated. This launches the formal study of mechanism design: constructing games whose equilibria yield desired outcomes.
Mechanism: Sealed-bid auction with second-price pricing; explicit dominance argument for truthful bidding; comparison of English / Dutch / first-price sealed-bid / second-price sealed-bid auctions; establishment of revenue equivalence between English and second-price; analysis of bid-shading in first-price auctions.
Concepts introduced/used: Vickrey Auction, Second-Price Auction, Truthful Mechanism, Strategy-Proof, Counterspeculation, Mechanism Design, Revenue Equivalence, Externality-aligned pricing.
Stance: foundational technical paper (Nobel-Prize-winning).
Relates to: Foundational paper for the entire field of Mechanism Design (Hurwicz, Maskin, Myerson, Roth — five Nobel Prizes between them); the Vickrey-Clarke-Groves family (Clarke 1971, Groves 1973) generalises second-price to combinatorial and multi-item settings and is the analytical core of all major sponsored-search and spectrum auctions. In MAS, Vickrey auctions appear as the canonical truthful resource-allocation mechanism — used in agent-based market designs since the 1980s, in cloud-computing resource auctions, and in academic LLM-agent negotiation testbeds. Conceptually, the strategy-proofness property eliminates the theory-of-mind regress that motivates Pact’s level-ℓ bounded-rational solver: when the mechanism is strategy-proof, bidders’ best moves do not depend on their beliefs about others, so the recursion collapses to depth 1. This is one of the strongest design principles for agent-coordination protocols: prefer mechanisms in which truth-telling is a dominant strategy over mechanisms requiring strategic reasoning. Vickrey’s analysis of revenue equivalence also frames the larger trade-off in protocol design between individual rationality (each agent prefers participating to not) and efficiency (the mechanism produces a socially-optimal allocation) — the same trade-off Deals Among Rational Agents (Rosenschein & Genesereth 1985) takes up for general multi-agent deal-making.

Summary

The paper attacks the buyer’s inspection paradox in information markets — the same Arrow / Nelson disclosure paradox addressed contractually by NDAI Agreements. Buyers need to access information to assess its value; sellers must restrict access to prevent appropriation; in equilibrium, useful information often goes untraded. Rahaman et al. propose a mechanism-design solution using LLM agents with two abilities that humans lack: (i) the capacity to evaluate the quality of privileged information against a query, and (ii) the ability to forget — to be cryptographically or architecturally constrained to discard information when not retained.

They build an open-source simulated marketplace where LLM-powered buyer-agents and seller-agents transact information on behalf of external participants. The seller grants the buyer-agent temporary, evaluable access to proprietary information; if the agent judges the information non-essential, duplicative, or available more cheaply elsewhere, it can discard it without paying. The combination of evaluation + forgetting creates a credible commitment device: vendors can reveal information for valuation without losing it, and buyers can inspect without obligation.

Experiments yield three findings: (a) current LLMs exhibit systematic biases — anchoring, recency, and over-confidence — that produce irrational marketplace behaviour, but well-known debiasing techniques substantially mitigate them; (b) demand for informational goods responds to price in legible, economically intuitive ways; (c) both inspection access and higher budgets improve buyer outcome quality. The paper anticipates and complements the TCME / NDAI proposals that arrived a year later: it provides the agent-architectural version of the “trusted intermediary” thesis that Shumailov et al. and Stephenson et al. then formalise cryptographically/economically.

Key Ideas

Buyer’s inspection paradox / Arrow Information Paradox: must access information to value it; must restrict access to prevent theft.
Dual agent capability — evaluate + forget: LLM agents can judge quality of privileged information and be made to discard it.
Open-source marketplace simulation: buyer-agents and seller-agents transact on behalf of external principals.
Temporary-access commitment device: vendors safely reveal information for valuation because the agent’s forgetting is enforced.
Biases identified: anchoring, recency, over-confidence in LLM-driven market behaviour; standard debiasing helps.
Price elasticity of information: demand responds to price in legible ways — informational goods can be priced like other goods.
Quality–budget–inspection relationship: inspection access and budget jointly determine outcome quality.

Connections

Conceptual Contribution

Claim: LLM agents with the dual capability of evaluating privileged information and being made to forget it can resolve the Arrow / buyer’s-inspection paradox by acting as credible, forgetting trusted intermediaries — turning previously untradeable information into a market good.
Mechanism: Open-source simulated marketplace; LLM buyer-agents and seller-agents act on behalf of external participants; sellers grant temporary inspect-and-evaluate access; agents must discard non-retained information; experiments probe bias, price elasticity, and budget/inspection effects.
Concepts introduced/used: Information Markets, Buyer’s Inspection Paradox, Arrow Information Paradox, Agent Amnesia, Temporary Disclosure, Mechanism Design, Information Asymmetry
Stance: empirical / mechanism-design with system implementation
Relates to: Architectural precursor to NDAI Agreements (TEE+economic-theory version) and Trusted Machine Learning Models Unlock Private Inference (TCME / cryptographic version) — all three converge on “capable model + constraint = trusted intermediary”. Provides micro-foundations for the markets imagined in Virtual Agent Economies and Mechanism Design for Large Language Models.

Virtual Agent Economies

Reference: Tomasev, Franklin, Leibo, Jacobs, Cunningham, Gabriel & Osindero (2025). Virtual Agent Economies. arXiv:2509.10147 (Google DeepMind). URL.

Summary

The paper provides a conceptual framework — the “sandbox economy” — for analysing the rapidly emerging economic layer in which AI agents transact and coordinate at scales and speeds beyond direct human oversight. It situates the question on two orthogonal axes: (i) origin — whether the agent economy emerged spontaneously from autonomous deployments or was intentionally designed; and (ii) separateness — whether it is permeable to (or insulated from) the established human economy. Most current trajectories occupy the spontaneous × permeable quadrant: vast, fast, and tightly coupled to human markets — the riskiest configuration for systemic externalities.

The authors argue for proactive steerable market design rather than passive emergence. Three design levers receive most of the discussion. (1) Auction mechanisms — adapted VCG / second-price / matching mechanisms — for fair resource allocation and preference resolution among agents. (2) Mission economies — agent markets architected around explicit collective goals (climate, public health, AI safety), where price signals are deliberately steered. (3) Socio-technical infrastructure — accountability, attribution, audit, governance — much of which overlaps with Infrastructure for AI Agents’s programme.

The paper is best read as the economic counterpart to Open Challenges in Multi-Agent Security and Infrastructure for AI Agents: together they delineate the threat surface, governance scaffolding, and economic architecture of the emerging agent economy, and argue that none can be ignored. Risks emphasised include systemic instability (algorithmic flash-crashes spreading to human markets), inequality amplification (agents capturing surplus from price-discrimination at machine speed), and the loss of human-economy slack — the friction that gives humans time to react.

Key Ideas

Sandbox economy framework: two axes — origin (emergent / intentional) × separateness (permeable / impermeable).
Current trajectory: spontaneous + highly permeable agent economy — opportunity and the riskiest configuration for systemic spillover.
Auctions for agent markets: revisits VCG / Vickrey / matching mechanisms for fair allocation and preference resolution among AI participants.
Mission economies: intentionally steered markets aligned to collective goals (climate, public health, AI safety).
Socio-technical infrastructure: trust, attribution, accountability — the governance layer that complements market design.
Systemic risk: flash-crash-like cascades from agent markets into human markets; inequality amplified by machine-speed price discrimination.
Call to proactive design: infrastructure choices now will shape whether the agent economy is steerable or merely emergent.

Connections

Conceptual Contribution

Claim: A vast, permeable AI-agent economy is emerging by default. Letting it emerge unsteered is the highest-risk design choice. Proactive market design — auctions, mission economies, governance infrastructure — is needed to keep agent economies aligned with long-term human flourishing.
Mechanism: A framework characterising agent economies along origin × separateness; a catalogue of three design levers (auctions, mission economies, infrastructure); a discussion of systemic risks and policy implications.
Concepts introduced/used: Sandbox Economy, Mission Economy, Agent Market, Steerable Market, Mechanism Design, Algorithmic Collusion, Systemic Risk (Agent Markets)
Stance: position paper / research agenda
Relates to: Sister piece to Infrastructure for AI Agents (infrastructure framing) and Open Challenges in Multi-Agent Security (threat framing) — these three jointly outline the agent-economy / agent-security / agent-governance space. Auction-design discussion connects to Mechanism Design for Large Language Models (LLM-internal auctions) and Vickrey 1961 (foundational mechanism design). Collusion concerns operationalised in Learning Collusion in Episodic Inventory-Constrained Markets and Do LLM Agents Have Regret.

Infrastructure for AI Agents

Reference: Chan, Wei, Huang, Rajkumar, Perrier, Lazar, Hadfield & Anderljung (2025). Infrastructure for AI Agents. TMLR (accepted). arXiv:2501.10114 (Centre for the Governance of AI; Oxford; ANU; Toronto). URL.

Summary

The paper proposes the concept of agent infrastructure: the technical systems and shared protocols, external to any individual agent, that mediate how agents interact with each other, with humans, and with institutions. The argument is by analogy to the Internet: a network of capable agents requires its own equivalent of TLS, DNS, X.509, BGP, and HTTP — because most safety properties of multi-agent ecosystems cannot be obtained by behavioural training of any individual model.

Chan et al. identify three functions agent infrastructure should serve. (1) Attribution — binding actions, properties, and credentials to specific agents and to the humans or institutions accountable for them, via agent IDs, attestations, and audit logs. (2) Interaction shaping — efficient inter-agent communication protocols, agreement formation, mechanism design for resource allocation, and reputation systems. (3) Detection and remediation — monitoring for harmful behaviour and providing mechanisms to roll back, contain, or compensate for damage.

For each function the paper sketches research directions, candidate adoption paths, relationships to existing internet infrastructure, and open problems. The framing is deliberately governance-first: infrastructure exists not to make agents more capable but to keep their externalities tractable as deployment scales. The paper is now the standard citation for the agent-governance / agent-infrastructure thread underlying Model Context Protocol, Agent-to-Agent Protocol, Agent Network Protocol, and emerging “agent passport” / verifiable-credential proposals.

Key Ideas

Agent infrastructure as governance layer: external technical systems mediating agent interactions — distinct from training-time alignment.
Three functions: attribution; interaction shaping; detection & remediation. Each maps to concrete research directions.
Attribution: agent IDs, verifiable credentials, attestations, audit logs, principal-binding (which human/org owns this agent).
Interaction shaping: inter-agent communication protocols; standardised agreement primitives; mechanism design; reputation.
Detection & remediation: anomaly detection on agent traffic; rollback mechanisms; insurance / compensation rails; “kill switch” governance.
Analogy to Internet protocols (HTTPS, DNS, BGP, X.509): infrastructure adoption is path-dependent, requires standardisation bodies, and trades expressivity for safety properties.
Open questions: who issues credentials, how privacy interacts with attribution, how to bootstrap adoption, what is enforceable cross-jurisdiction.

Connections

Conceptual Contribution

Claim: Many of the safety, accountability, and interoperability properties society will need from AI agents are not properties of any individual model — they live in the infrastructure between agents. Just as the Internet’s safety depends on TLS / DNS / BGP rather than on any single application, agent ecosystems will depend on agent-level analogues: attribution, interaction shaping, and detection-and-remediation infrastructure.
Mechanism: A three-function taxonomy (attribution / interaction-shaping / detection-and-remediation) with a catalogue of candidate primitives — agent IDs, verifiable credentials, inter-agent protocols, certification regimes, reputation systems, rollback mechanisms — plus analysis of adoption pathways relative to existing internet infrastructure.
Concepts introduced/used: Agent Infrastructure, Agent ID, Verifiable Agent Credentials, Inter-Agent Protocols, Action Attribution, Agent Reputation, Agent Rollback, AI Governance
Stance: governance / position paper / research-agenda
Relates to: Provides the governance scaffolding within which Open Challenges in Multi-Agent Security threats must be addressed; the institutional counterpart to Virtual Agent Economies’s economic framing; concrete protocols proposed include Model Context Protocol, Agent-to-Agent Protocol, Agent Network Protocol — surveyed alongside in Survey Of AI Agent Protocols and Survey Of Agent Interoperability Protocols. The attribution leg connects to NDAI Agreements (TEEs as a particular attribution / commitment substrate) and Trusted Machine Learning Models Unlock Private Inference (capable models as a trust substrate).

Trusted Machine Learning Models Unlock Private Inference

Trusted Machine Learning Models Unlock Private Inference for Problems Currently Infeasible with Cryptography

Reference: Shumailov, Ramage, Meiklejohn, Kairouz, Hartmann, Balle & Bagdasarian (2025). Trusted Machine Learning Models Unlock Private Inference for Problems Currently Infeasible with Cryptography. arXiv:2501.08970 (Google Research). URL.

Summary

The paper proposes Trusted Capable Model Environments (TCMEs) — a new design point in the privacy-preserving-computation landscape, sitting between classical trusted execution environments and cryptographic protocols such as multi-party computation (MPC), homomorphic encryption, and zero-knowledge proofs. The motivating observation: capable modern ML models can plausibly play the role of the trusted third party in many private-inference scenarios that classical cryptography handles only at toy scale or not at all.

A TCME is defined by three constraints under which a capable model operates: (i) explicit input/output constraints scoping what the model is permitted to receive and emit; (ii) explicit information-flow control binding outputs to authorised data-flow channels; (iii) explicit statelessness — the model cannot retain or leak inputs across sessions. Under these constraints, even an inscrutable LLM can serve as a credible “trusted intermediary”: it computes a function of two parties’ data and reveals only the agreed output.

The authors argue TCMEs unlock private inference for problems where MPC is infeasible because the function is too rich, the inputs too large, or the spec too implicit (natural-language matching, fuzzy de-duplication, semantic agreement-checking). They walk through use cases — private record matching, contract negotiation, secret-keeping triage — and show that even classical cryptographic problems (private set intersection, secure multi-party comparison) admit TCME implementations that scale further than current MPC. The paper closes with the limitations: trust in TCMEs reduces to trust in the model+hardware+policy stack; statelessness must be engineered, not assumed.

Key Ideas

TCME definition: a capable ML model + explicit I/O constraints + explicit information-flow control + explicit statelessness.
Trusted-third-party substitution: the model fills the role MPC traditionally requires a non-colluding cryptographic protocol to enact.
Coverage envelope: TCMEs handle privacy problems too rich or too implicit for current MPC (semantic matching, fuzzy agreements, natural-language contracts).
Bridge to cryptography: even classical PSI/comparison protocols can be implemented as TCMEs — sometimes more efficiently.
Statelessness is engineered: memory leaks, side channels, and re-training contamination are the real attack surface, not the model logic.
Trust composition: TCME trust assumption = trust(model) ∧ trust(hardware) ∧ trust(policy enforcement).
Use cases sketched: private record matching, negotiation, triage, search over private corpora, semantic compliance checks.

Connections

Conceptual Contribution

Claim: Capable ML models, operated under explicit information-flow and statelessness constraints, can act as trusted third parties for private-inference problems that classical cryptography cannot scale to. This expands the realm of feasible privacy-preserving computation beyond MPC’s current envelope.
Mechanism: Define Trusted Capable Model Environments (TCMEs): model + explicit I/O constraints + explicit IFC + explicit statelessness. Demonstrate via use cases that TCMEs solve both novel privacy problems (semantic matching) and re-instantiate classical ones (PSI) at scales MPC cannot reach.
Concepts introduced/used: Trusted Capable Model Environment, Trusted Third Party, Information Flow Control, Private Inference, Statelessness (Privacy), Multi-Party Computation
Stance: position / architectural proposal
Relates to: Direct companion to NDAI Agreements — both treat TEE+AI or model+constraints as a substrate for previously infeasible commitment / privacy primitives. Provides the technical substrate that Privacy Reasoning in Ambiguous Contexts reasons about behaviourally and that Infrastructure for AI Agents would expose as governance infrastructure. Complementary to Defeating Prompt Injections by Design’s CaMeL: both treat the agent as a constrained reasoner whose outputs are gated by information-flow policy.

Disclosure Game

(page does not exist)

Commitment Device

(page does not exist)

Mechanism Design

The branch of game theory that designs interaction rules so that self-interested agents, acting in equilibrium, produce a desired social outcome; applied in ACL evolution to shape negotiation protocols that reach efficient linguistic conventions.

In this vault

Toward Automated Evolution of ACLs

Arrow Information Paradox

The classical paradox of information markets, identified by Kenneth Arrow (1962) and Richard Nelson (1959): a buyer cannot assess an idea’s value without seeing it, yet revealing it removes the seller’s leverage to be paid. Resolved in the agentic setting by TEE-backed agreements and capable-model intermediaries that act as trusted third parties.

In this vault

Hold-Up Problem

Classic problem in contract theory and the economics of innovation: one party invests, the other can renegotiate ex post to extract the gains. Cause of under-investment / under-disclosure in Information Markets. NDAI Agreements shows TEE+AI delegation can solve the hold-up around invention disclosure.

In this vault

Trusted Execution Environment

Hardware-isolated execution context (Intel SGX, AMD SEV, Apple Secure Enclave, etc.) that provides attested confidentiality and integrity guarantees to code running inside. Used in NDAI Agreements and Trusted Machine Learning Models Unlock Private Inference as a substrate for credible commitment between mutually distrustful parties.

In this vault

NDAI Agreement

(page does not exist)

LLM Agents

Large-language-model-powered agents: natural-language coordination, tool use, multi-agent orchestration.

Surveys & frameworks

Protocols & communication

Failures & threats

Lineage

Distributed Security

Security of distributed/agent systems: mobile code, secure messaging, language-based defences.

Information Asymmetry

Market condition in which one party has more or better information than the counterparty. Source of adverse selection, moral hazard, and the Arrow Information Paradox; substrate of Information Markets.

In this vault

Mechanism Design for Large Language Models

Reference: Dütting, Mirrokni, Paes Leme, Xu & Zuo (2023). Mechanism Design for Large Language Models. WWW 2024 (Best Paper). arXiv:2310.10826 (Google Research; University of Chicago). URL.

Summary

This paper opens the field of mechanism design over LLM-generated content. The motivating use case is multi-advertiser ad-creative generation: several advertisers each have preferences over what a stochastic LLM produces for a given query, and the platform must aggregate these preferences into a single piece of content while charging payments in a way that is incentive-compatible. Classical mechanism design assumes each agent has an explicit valuation function over outcomes; here outcomes are token sequences and valuations are encoded as the agents’ own LLMs — there is no compact valuation form to plug into VCG.

Dütting et al. propose a token-level auction that solves this. At each generation step, every agent submits a one-dimensional bid; the platform aggregates the agents’ next-token preferences using their own LLMs together with the bids; the chosen token is the one that maximises the aggregate. Payments are charged on a token-by-token basis using a generalised second-price-like rule. They define two natural incentive properties over distributions of generated content and prove their equivalence to a monotonicity condition on output aggregation — analogous to the Myerson monotonicity / payment characterisation for single-item auctions. This equivalence enables a clean second-price-style payment rule without requiring explicit valuation functions: the LLM-encoded preferences are sufficient.

The construction is supported by demonstrations on a publicly available LLM. The paper is now the canonical reference for “mechanism design where outcomes are LLM outputs and preferences are LLM-encoded” — a building block for the steerable agent markets of Virtual Agent Economies, the information-market substrates of Language Models Can Reduce Asymmetry in Information Markets, and the regret-aware market analyses of Do LLM Agents Have Regret.

Key Ideas

Problem: auctioning LLM-generated content among multiple advertisers / agents whose preferences are themselves LLMs — no explicit valuation function available.
Token-by-token auction: at each generation step, single-dimensional bids combine with LLM-encoded preferences to pick the next token.
Output aggregation: the chosen token aggregates the agents’ next-token preferences weighted by bids — no need for a compact valuation form.
Two incentive properties: formulated over distributions of generated content; jointly capture natural truthfulness desiderata.
Monotonicity equivalence: the incentive properties hold iff output aggregation is monotone — a Myerson-style characterisation.
Second-price design: the equivalence yields a generalised second-price payment rule, even absent explicit valuations.
Practical demonstrations: validated on a publicly available LLM, suggesting the construction is implementable.

Connections

Conceptual Contribution

Claim: Mechanism design extends naturally to the regime where outcomes are LLM-generated tokens and agent preferences are themselves LLMs. The classical machinery — Myerson monotonicity, second-price payments, truthfulness — survives, but is parameterised by output-aggregation monotonicity rather than by explicit valuation functions.
Mechanism: Token-by-token auction; single-dimensional bids per token; output aggregation via agents’ own LLM preferences weighted by bids; two incentive properties shown equivalent to output-aggregation monotonicity; second-price-style payment rule recovered without explicit valuations; LLM demonstrations.
Concepts introduced/used: LLM Auction, Token-Level Mechanism, Output Aggregation, Monotone Aggregation, Vickrey Auction, Myerson’s Lemma, Incentive Compatibility, Mechanism Design
Stance: formal mechanism design with implementation
Relates to: Generalises the Vickrey / Myerson tradition (Counterspeculation Auctions and Competitive Sealed Tenders) to LLM-generated outcomes; provides the formal layer underlying the auction-mechanism discussion in Virtual Agent Economies; foundational dependency for Language Models Can Reduce Asymmetry in Information Markets and the incentive-compatibility analyses behind NDAI Agreements; the agents’ assumed rationality must approximate no-regret for the equilibrium analysis to apply — see Do LLM Agents Have Regret.

Technology Transfer

(page does not exist)

R&D Commercialisation

(page does not exist)

Trusted Capable Model Environment

Shumailov et al. 2025 design pattern: a capable ML model run under explicit input/output constraints + Information Flow Control + statelessness can play the role of a trusted third party in private-inference problems that classical cryptography (MPC, ZKP) cannot scale to. See Trusted Machine Learning Models Unlock Private Inference.

NDAI Agreements

Summary

Key Ideas

Connections

Conceptual Contribution

Tags

Backlinks