Information Markets

Markets in which the good traded is information. They are structurally distinct from goods markets because of the Arrow / buyer’s-inspection paradox: buyers need access to evaluate, sellers need to restrict access to prevent theft. AI agents with evaluate + forget capability and / or TEE-mediated disclosure can serve as trusted intermediaries that resolve the paradox.

In this vault

Backlinks

Linked Pages

Mechanism Design for Large Language Models

Reference: Dütting, Mirrokni, Paes Leme, Xu & Zuo (2023). Mechanism Design for Large Language Models. WWW 2024 (Best Paper). arXiv:2310.10826 (Google Research; University of Chicago). URL.

Summary

This paper opens the field of mechanism design over LLM-generated content. The motivating use case is multi-advertiser ad-creative generation: several advertisers each have preferences over what a stochastic LLM produces for a given query, and the platform must aggregate these preferences into a single piece of content while charging payments in a way that is incentive-compatible. Classical mechanism design assumes each agent has an explicit valuation function over outcomes; here outcomes are token sequences and valuations are encoded as the agents’ own LLMs — there is no compact valuation form to plug into VCG.

Dütting et al. propose a token-level auction that solves this. At each generation step, every agent submits a one-dimensional bid; the platform aggregates the agents’ next-token preferences using their own LLMs together with the bids; the chosen token is the one that maximises the aggregate. Payments are charged on a token-by-token basis using a generalised second-price-like rule. They define two natural incentive properties over distributions of generated content and prove their equivalence to a monotonicity condition on output aggregation — analogous to the Myerson monotonicity / payment characterisation for single-item auctions. This equivalence enables a clean second-price-style payment rule without requiring explicit valuation functions: the LLM-encoded preferences are sufficient.

The construction is supported by demonstrations on a publicly available LLM. The paper is now the canonical reference for “mechanism design where outcomes are LLM outputs and preferences are LLM-encoded” — a building block for the steerable agent markets of Virtual Agent Economies, the information-market substrates of Language Models Can Reduce Asymmetry in Information Markets, and the regret-aware market analyses of Do LLM Agents Have Regret.

Key Ideas

Problem: auctioning LLM-generated content among multiple advertisers / agents whose preferences are themselves LLMs — no explicit valuation function available.
Token-by-token auction: at each generation step, single-dimensional bids combine with LLM-encoded preferences to pick the next token.
Output aggregation: the chosen token aggregates the agents’ next-token preferences weighted by bids — no need for a compact valuation form.
Two incentive properties: formulated over distributions of generated content; jointly capture natural truthfulness desiderata.
Monotonicity equivalence: the incentive properties hold iff output aggregation is monotone — a Myerson-style characterisation.
Second-price design: the equivalence yields a generalised second-price payment rule, even absent explicit valuations.
Practical demonstrations: validated on a publicly available LLM, suggesting the construction is implementable.

Connections

Conceptual Contribution

Claim: Mechanism design extends naturally to the regime where outcomes are LLM-generated tokens and agent preferences are themselves LLMs. The classical machinery — Myerson monotonicity, second-price payments, truthfulness — survives, but is parameterised by output-aggregation monotonicity rather than by explicit valuation functions.
Mechanism: Token-by-token auction; single-dimensional bids per token; output aggregation via agents’ own LLM preferences weighted by bids; two incentive properties shown equivalent to output-aggregation monotonicity; second-price-style payment rule recovered without explicit valuations; LLM demonstrations.
Concepts introduced/used: LLM Auction, Token-Level Mechanism, Output Aggregation, Monotone Aggregation, Vickrey Auction, Myerson’s Lemma, Incentive Compatibility, Mechanism Design
Stance: formal mechanism design with implementation
Relates to: Generalises the Vickrey / Myerson tradition (Counterspeculation Auctions and Competitive Sealed Tenders) to LLM-generated outcomes; provides the formal layer underlying the auction-mechanism discussion in Virtual Agent Economies; foundational dependency for Language Models Can Reduce Asymmetry in Information Markets and the incentive-compatibility analyses behind NDAI Agreements; the agents’ assumed rationality must approximate no-regret for the equilibrium analysis to apply — see Do LLM Agents Have Regret.

Mechanism Design

The branch of game theory that designs interaction rules so that self-interested agents, acting in equilibrium, produce a desired social outcome; applied in ACL evolution to shape negotiation protocols that reach efficient linguistic conventions.

In this vault

Toward Automated Evolution of ACLs

Virtual Agent Economies

Reference: Tomasev, Franklin, Leibo, Jacobs, Cunningham, Gabriel & Osindero (2025). Virtual Agent Economies. arXiv:2509.10147 (Google DeepMind). URL.

Summary

The paper provides a conceptual framework — the “sandbox economy” — for analysing the rapidly emerging economic layer in which AI agents transact and coordinate at scales and speeds beyond direct human oversight. It situates the question on two orthogonal axes: (i) origin — whether the agent economy emerged spontaneously from autonomous deployments or was intentionally designed; and (ii) separateness — whether it is permeable to (or insulated from) the established human economy. Most current trajectories occupy the spontaneous × permeable quadrant: vast, fast, and tightly coupled to human markets — the riskiest configuration for systemic externalities.

The authors argue for proactive steerable market design rather than passive emergence. Three design levers receive most of the discussion. (1) Auction mechanisms — adapted VCG / second-price / matching mechanisms — for fair resource allocation and preference resolution among agents. (2) Mission economies — agent markets architected around explicit collective goals (climate, public health, AI safety), where price signals are deliberately steered. (3) Socio-technical infrastructure — accountability, attribution, audit, governance — much of which overlaps with Infrastructure for AI Agents’s programme.

The paper is best read as the economic counterpart to Open Challenges in Multi-Agent Security and Infrastructure for AI Agents: together they delineate the threat surface, governance scaffolding, and economic architecture of the emerging agent economy, and argue that none can be ignored. Risks emphasised include systemic instability (algorithmic flash-crashes spreading to human markets), inequality amplification (agents capturing surplus from price-discrimination at machine speed), and the loss of human-economy slack — the friction that gives humans time to react.

Key Ideas

Sandbox economy framework: two axes — origin (emergent / intentional) × separateness (permeable / impermeable).
Current trajectory: spontaneous + highly permeable agent economy — opportunity and the riskiest configuration for systemic spillover.
Auctions for agent markets: revisits VCG / Vickrey / matching mechanisms for fair allocation and preference resolution among AI participants.
Mission economies: intentionally steered markets aligned to collective goals (climate, public health, AI safety).
Socio-technical infrastructure: trust, attribution, accountability — the governance layer that complements market design.
Systemic risk: flash-crash-like cascades from agent markets into human markets; inequality amplified by machine-speed price discrimination.
Call to proactive design: infrastructure choices now will shape whether the agent economy is steerable or merely emergent.

Connections

Conceptual Contribution

Claim: A vast, permeable AI-agent economy is emerging by default. Letting it emerge unsteered is the highest-risk design choice. Proactive market design — auctions, mission economies, governance infrastructure — is needed to keep agent economies aligned with long-term human flourishing.
Mechanism: A framework characterising agent economies along origin × separateness; a catalogue of three design levers (auctions, mission economies, infrastructure); a discussion of systemic risks and policy implications.
Concepts introduced/used: Sandbox Economy, Mission Economy, Agent Market, Steerable Market, Mechanism Design, Algorithmic Collusion, Systemic Risk (Agent Markets)
Stance: position paper / research agenda
Relates to: Sister piece to Infrastructure for AI Agents (infrastructure framing) and Open Challenges in Multi-Agent Security (threat framing) — these three jointly outline the agent-economy / agent-security / agent-governance space. Auction-design discussion connects to Mechanism Design for Large Language Models (LLM-internal auctions) and Vickrey 1961 (foundational mechanism design). Collusion concerns operationalised in Learning Collusion in Episodic Inventory-Constrained Markets and Do LLM Agents Have Regret.

Trusted Machine Learning Models Unlock Private Inference

Trusted Machine Learning Models Unlock Private Inference for Problems Currently Infeasible with Cryptography

Reference: Shumailov, Ramage, Meiklejohn, Kairouz, Hartmann, Balle & Bagdasarian (2025). Trusted Machine Learning Models Unlock Private Inference for Problems Currently Infeasible with Cryptography. arXiv:2501.08970 (Google Research). URL.

Summary

The paper proposes Trusted Capable Model Environments (TCMEs) — a new design point in the privacy-preserving-computation landscape, sitting between classical trusted execution environments and cryptographic protocols such as multi-party computation (MPC), homomorphic encryption, and zero-knowledge proofs. The motivating observation: capable modern ML models can plausibly play the role of the trusted third party in many private-inference scenarios that classical cryptography handles only at toy scale or not at all.

A TCME is defined by three constraints under which a capable model operates: (i) explicit input/output constraints scoping what the model is permitted to receive and emit; (ii) explicit information-flow control binding outputs to authorised data-flow channels; (iii) explicit statelessness — the model cannot retain or leak inputs across sessions. Under these constraints, even an inscrutable LLM can serve as a credible “trusted intermediary”: it computes a function of two parties’ data and reveals only the agreed output.

The authors argue TCMEs unlock private inference for problems where MPC is infeasible because the function is too rich, the inputs too large, or the spec too implicit (natural-language matching, fuzzy de-duplication, semantic agreement-checking). They walk through use cases — private record matching, contract negotiation, secret-keeping triage — and show that even classical cryptographic problems (private set intersection, secure multi-party comparison) admit TCME implementations that scale further than current MPC. The paper closes with the limitations: trust in TCMEs reduces to trust in the model+hardware+policy stack; statelessness must be engineered, not assumed.

Key Ideas

TCME definition: a capable ML model + explicit I/O constraints + explicit information-flow control + explicit statelessness.
Trusted-third-party substitution: the model fills the role MPC traditionally requires a non-colluding cryptographic protocol to enact.
Coverage envelope: TCMEs handle privacy problems too rich or too implicit for current MPC (semantic matching, fuzzy agreements, natural-language contracts).
Bridge to cryptography: even classical PSI/comparison protocols can be implemented as TCMEs — sometimes more efficiently.
Statelessness is engineered: memory leaks, side channels, and re-training contamination are the real attack surface, not the model logic.
Trust composition: TCME trust assumption = trust(model) ∧ trust(hardware) ∧ trust(policy enforcement).
Use cases sketched: private record matching, negotiation, triage, search over private corpora, semantic compliance checks.

Connections

Conceptual Contribution

Claim: Capable ML models, operated under explicit information-flow and statelessness constraints, can act as trusted third parties for private-inference problems that classical cryptography cannot scale to. This expands the realm of feasible privacy-preserving computation beyond MPC’s current envelope.
Mechanism: Define Trusted Capable Model Environments (TCMEs): model + explicit I/O constraints + explicit IFC + explicit statelessness. Demonstrate via use cases that TCMEs solve both novel privacy problems (semantic matching) and re-instantiate classical ones (PSI) at scales MPC cannot reach.
Concepts introduced/used: Trusted Capable Model Environment, Trusted Third Party, Information Flow Control, Private Inference, Statelessness (Privacy), Multi-Party Computation
Stance: position / architectural proposal
Relates to: Direct companion to NDAI Agreements — both treat TEE+AI or model+constraints as a substrate for previously infeasible commitment / privacy primitives. Provides the technical substrate that Privacy Reasoning in Ambiguous Contexts reasons about behaviourally and that Infrastructure for AI Agents would expose as governance infrastructure. Complementary to Defeating Prompt Injections by Design’s CaMeL: both treat the agent as a constrained reasoner whose outputs are gated by information-flow policy.

NDAI Agreements

Reference: Stephenson, Miller, Sun, Annem & Parikh (2025). NDAI Agreements. arXiv:2502.07924 (UIUC; Cornell Tech; et al.). URL.

Summary

The “NDAI agreement” — non-disclosure AI agreement — is a mechanism in which a TEE combined with an AI agent jointly stands in for a trusted human intermediary, resolving the classical disclosure–appropriation paradox of information markets first identified by Arrow (1962) and Nelson (1959). An inventor cannot reveal an idea to a potential investor without risking misappropriation; without revealing it, no efficient bargain can be struck. The result is well-known: under-disclosure, under-investment, under-licensing.

Stephenson et al. show formally — via a buyer/seller bargaining game — that delegating the disclosure-and-payment decision to a tamper-proof program running inside a TEE eliminates the hold-up problem, achieving full disclosure and an efficient ex post transfer. When the invention’s value exceeds the value a TEE can fully secure (e.g. because some leakage is unavoidable), partial disclosure still strictly improves welfare over the no-disclosure equilibrium. They then model agent error — payments or disclosures going wrong — and prove that simple safeguards (budget caps, acceptance thresholds) preserve most of the efficiency gains.

The substantive economic claim is that TEE + AI behave as an “ironclad NDA”: a credible commitment device for the disclosure problem that was previously unattainable with paper contracts (because expropriation is unverifiable) or with cryptography alone (because invention value is unbounded and the seller’s information is a complex unstructured artefact). The result links the Mechanism Design / Hold-Up Problem tradition to AI-agent infrastructure, and gives a sharp theoretical case for the economic value of trusted model environments and confidential-compute hardware as agent-economy substrates.

Key Ideas

Formalises the Arrow–Nelson information paradox / hold-up problem in a bargaining model between seller (inventor) and buyer (investor).
TEEs + AI agents delegate disclosure and payment to tamper-proof programs that neither party can subvert; this implements an ex-ante commitment device unavailable under classical contracts.
Full-disclosure efficient equilibrium under the NDAI when the invention’s value lies within what the TEE can secure.
Partial disclosure dominates no-disclosure even when full security is impossible: high-value inventions still get partially revealed in welfare-improving ways.
Models agent imperfection: errors in payment or disclosure can occur; budget caps and acceptance thresholds bound the damage and preserve most welfare.
Frames TEEs+AI as an “ironclad NDA”: a cryptographically/hardware-enforced commitment that traditional NDAs cannot match.
Policy implications for R&D Commercialisation, Technology Transfer, and inter-firm collaboration; bridges economic theory to confidential-compute hardware.

Connections

Conceptual Contribution

Claim: A trusted execution environment hosting an AI agent can serve as a credible commitment device that solves the classical Arrow–Nelson disclosure problem of information markets — achieving full disclosure and efficient transfer where paper NDAs and pure cryptography both fail.
Mechanism: A bargaining model in which TEEs+AI mediate the disclosure-and-payment decision; closed-form characterisation of equilibria for full and partial disclosure; sensitivity analysis to agent error with policy-instrument bounds (budget caps, acceptance thresholds).
Concepts introduced/used: NDAI Agreement, Trusted Execution Environment, Hold-Up Problem, Arrow Information Paradox, Mechanism Design, Commitment Device, Disclosure Game
Stance: formal economic theory with technical implications
Relates to: Companion to Trusted Machine Learning Models Unlock Private Inference — both argue that capability + trusted execution can replace previously infeasible cryptographic primitives. The economic counterpart to the engineering catalogue in Infrastructure for AI Agents; a building block for the markets imagined in Virtual Agent Economies and the information-asymmetry resolution explored in Language Models Can Reduce Asymmetry in Information Markets. Sits in the lineage of Vickrey / mechanism-design tradition.

Language Models Can Reduce Asymmetry in Information Markets

Reference: Rahaman, Weiss, Wüthrich, Bengio, Li, Pal & Schölkopf (2024). Language Models Can Reduce Asymmetry in Information Markets. arXiv:2403.14443 (Mila; Max-Planck; AWS AI Labs). URL.

Summary

The paper attacks the buyer’s inspection paradox in information markets — the same Arrow / Nelson disclosure paradox addressed contractually by NDAI Agreements. Buyers need to access information to assess its value; sellers must restrict access to prevent appropriation; in equilibrium, useful information often goes untraded. Rahaman et al. propose a mechanism-design solution using LLM agents with two abilities that humans lack: (i) the capacity to evaluate the quality of privileged information against a query, and (ii) the ability to forget — to be cryptographically or architecturally constrained to discard information when not retained.

They build an open-source simulated marketplace where LLM-powered buyer-agents and seller-agents transact information on behalf of external participants. The seller grants the buyer-agent temporary, evaluable access to proprietary information; if the agent judges the information non-essential, duplicative, or available more cheaply elsewhere, it can discard it without paying. The combination of evaluation + forgetting creates a credible commitment device: vendors can reveal information for valuation without losing it, and buyers can inspect without obligation.

Experiments yield three findings: (a) current LLMs exhibit systematic biases — anchoring, recency, and over-confidence — that produce irrational marketplace behaviour, but well-known debiasing techniques substantially mitigate them; (b) demand for informational goods responds to price in legible, economically intuitive ways; (c) both inspection access and higher budgets improve buyer outcome quality. The paper anticipates and complements the TCME / NDAI proposals that arrived a year later: it provides the agent-architectural version of the “trusted intermediary” thesis that Shumailov et al. and Stephenson et al. then formalise cryptographically/economically.

Key Ideas

Buyer’s inspection paradox / Arrow Information Paradox: must access information to value it; must restrict access to prevent theft.
Dual agent capability — evaluate + forget: LLM agents can judge quality of privileged information and be made to discard it.
Open-source marketplace simulation: buyer-agents and seller-agents transact on behalf of external principals.
Temporary-access commitment device: vendors safely reveal information for valuation because the agent’s forgetting is enforced.
Biases identified: anchoring, recency, over-confidence in LLM-driven market behaviour; standard debiasing helps.
Price elasticity of information: demand responds to price in legible ways — informational goods can be priced like other goods.
Quality–budget–inspection relationship: inspection access and budget jointly determine outcome quality.

Connections

Conceptual Contribution

Claim: LLM agents with the dual capability of evaluating privileged information and being made to forget it can resolve the Arrow / buyer’s-inspection paradox by acting as credible, forgetting trusted intermediaries — turning previously untradeable information into a market good.
Mechanism: Open-source simulated marketplace; LLM buyer-agents and seller-agents act on behalf of external participants; sellers grant temporary inspect-and-evaluate access; agents must discard non-retained information; experiments probe bias, price elasticity, and budget/inspection effects.
Concepts introduced/used: Information Markets, Buyer’s Inspection Paradox, Arrow Information Paradox, Agent Amnesia, Temporary Disclosure, Mechanism Design, Information Asymmetry
Stance: empirical / mechanism-design with system implementation
Relates to: Architectural precursor to NDAI Agreements (TEE+economic-theory version) and Trusted Machine Learning Models Unlock Private Inference (TCME / cryptographic version) — all three converge on “capable model + constraint = trusted intermediary”. Provides micro-foundations for the markets imagined in Virtual Agent Economies and Mechanism Design for Large Language Models.

Trusted Execution Environment

Hardware-isolated execution context (Intel SGX, AMD SEV, Apple Secure Enclave, etc.) that provides attested confidentiality and integrity guarantees to code running inside. Used in NDAI Agreements and Trusted Machine Learning Models Unlock Private Inference as a substrate for credible commitment between mutually distrustful parties.

In this vault

Arrow Information Paradox

The classical paradox of information markets, identified by Kenneth Arrow (1962) and Richard Nelson (1959): a buyer cannot assess an idea’s value without seeing it, yet revealing it removes the seller’s leverage to be paid. Resolved in the agentic setting by TEE-backed agreements and capable-model intermediaries that act as trusted third parties.