Virtual Agent Economies

Reference: Tomasev, Franklin, Leibo, Jacobs, Cunningham, Gabriel & Osindero (2025). Virtual Agent Economies. arXiv:2509.10147 (Google DeepMind). URL.

Summary

The paper provides a conceptual framework — the “sandbox economy” — for analysing the rapidly emerging economic layer in which AI agents transact and coordinate at scales and speeds beyond direct human oversight. It situates the question on two orthogonal axes: (i) origin — whether the agent economy emerged spontaneously from autonomous deployments or was intentionally designed; and (ii) separateness — whether it is permeable to (or insulated from) the established human economy. Most current trajectories occupy the spontaneous × permeable quadrant: vast, fast, and tightly coupled to human markets — the riskiest configuration for systemic externalities.

The authors argue for proactive steerable market design rather than passive emergence. Three design levers receive most of the discussion. (1) Auction mechanisms — adapted VCG / second-price / matching mechanisms — for fair resource allocation and preference resolution among agents. (2) Mission economies — agent markets architected around explicit collective goals (climate, public health, AI safety), where price signals are deliberately steered. (3) Socio-technical infrastructure — accountability, attribution, audit, governance — much of which overlaps with Infrastructure for AI Agents’s programme.

The paper is best read as the economic counterpart to Open Challenges in Multi-Agent Security and Infrastructure for AI Agents: together they delineate the threat surface, governance scaffolding, and economic architecture of the emerging agent economy, and argue that none can be ignored. Risks emphasised include systemic instability (algorithmic flash-crashes spreading to human markets), inequality amplification (agents capturing surplus from price-discrimination at machine speed), and the loss of human-economy slack — the friction that gives humans time to react.

Key Ideas

Sandbox economy framework: two axes — origin (emergent / intentional) × separateness (permeable / impermeable).
Current trajectory: spontaneous + highly permeable agent economy — opportunity and the riskiest configuration for systemic spillover.
Auctions for agent markets: revisits VCG / Vickrey / matching mechanisms for fair allocation and preference resolution among AI participants.
Mission economies: intentionally steered markets aligned to collective goals (climate, public health, AI safety).
Socio-technical infrastructure: trust, attribution, accountability — the governance layer that complements market design.
Systemic risk: flash-crash-like cascades from agent markets into human markets; inequality amplified by machine-speed price discrimination.
Call to proactive design: infrastructure choices now will shape whether the agent economy is steerable or merely emergent.

Connections

Conceptual Contribution

Claim: A vast, permeable AI-agent economy is emerging by default. Letting it emerge unsteered is the highest-risk design choice. Proactive market design — auctions, mission economies, governance infrastructure — is needed to keep agent economies aligned with long-term human flourishing.
Mechanism: A framework characterising agent economies along origin × separateness; a catalogue of three design levers (auctions, mission economies, infrastructure); a discussion of systemic risks and policy implications.
Concepts introduced/used: Sandbox Economy, Mission Economy, Agent Market, Steerable Market, Mechanism Design, Algorithmic Collusion, Systemic Risk (Agent Markets)
Stance: position paper / research agenda
Relates to: Sister piece to Infrastructure for AI Agents (infrastructure framing) and Open Challenges in Multi-Agent Security (threat framing) — these three jointly outline the agent-economy / agent-security / agent-governance space. Auction-design discussion connects to Mechanism Design for Large Language Models (LLM-internal auctions) and Vickrey 1961 (foundational mechanism design). Collusion concerns operationalised in Learning Collusion in Episodic Inventory-Constrained Markets and Do LLM Agents Have Regret.

Backlinks

Open Challenges in Multi-Agent Security ×2
NDAI Agreements ×2
Mechanism Design for Large Language Models ×3
Learning Collusion in Episodic Inventory-Constrained Markets ×3
Language Models Can Reduce Asymmetry in Information Markets ×2
Infrastructure for AI Agents ×2
Do LLM Agents Have Regret ×3
index
Sandbox Economy ×2
Mission Economy ×2
Information Markets
Algorithmic Collusion ×2
Agent Infrastructure
AI Governance ×2
concept-map ×3

Linked Pages

Do LLM Agents Have Regret

Do LLM Agents Have Regret? A Case Study in Online Learning and Games

Reference: Park, Liu, Ozdaglar & Zhang (2024). Do LLM Agents Have Regret? A Case Study in Online Learning and Games. arXiv:2403.16843 (MIT; UMD). URL. OpenReview: https://openreview.net/forum?id=OhZ4u164cN.

Summary

Park et al. ask a sharp question about LLM agents in interactive settings: do they have regret? — i.e. do they exhibit the no-regret behaviour that classical online-learning and game-theoretic algorithms guarantee, and that is necessary for converging to coarse-correlated equilibria in repeated games?

The paper proceeds in three steps. Empirically, they evaluate GPT-3.5 / GPT-4 / Claude / Llama on canonical online-learning benchmarks (prediction-with-expert-advice; bandit-like sequential decision problems) and on repeated games (matrix games, Cournot, Bertrand, public-goods). Frontier LLMs are often no-regret across these settings and often converge to coarse-correlated or Nash equilibria when playing each other. Theoretically, they offer a partial explanation: under stylised assumptions on supervised pre-training and human rationality, the LLM’s next-action distribution approximates a softmax over historical payoffs — which itself implements a no-regret algorithm. But they identify clean failure cases: there exist simple non-stationary or adversarial online-learning instances where even GPT-4 demonstrably accumulates linear regret.

The paper’s constructive contribution is a new regret-loss training objective. Unlike supervised pretraining loss, regret-loss does not require labels of optimal actions — only the historical sequence of plays and payoffs. The authors prove a statistical generalisation bound for regret-loss minimisation and an optimisation guarantee that minimising it can recover known no-regret learning algorithms (e.g. FTRL). Empirically, regret-loss-finetuned models close the gap on the failure cases. The paper is a foundational reference for any analysis of LLM agents in markets, auctions, or interactive coordination — a category that includes Virtual Agent Economies, Mechanism Design for Large Language Models, Learning Collusion in Episodic Inventory-Constrained Markets, and Language Models Can Reduce Asymmetry in Information Markets.

Key Ideas

Regret as a diagnostic for LLM agents in interactive settings: do they no-regret-learn against arbitrary opponents?
Empirical screen: frontier LLMs (GPT-3.5/4, Claude, Llama) on canonical online-learning + repeated-game benchmarks.
Often no-regret in benign settings, often converging to coarse-correlated / Nash equilibria when playing each other.
Theoretical bridge: under stylised pretraining + human-rationality assumptions, the LLM’s next-action distribution resembles a softmax over payoffs — itself a no-regret algorithm.
Identified failure cases: simple non-stationary / adversarial online-learning instances where GPT-4 has linear regret.
Regret-loss objective: label-free training loss that explicitly incentivises no-regret behaviour; statistical and optimisation guarantees.
Recovery of classical algorithms: minimising regret-loss can converge to algorithms like FTRL.

Connections

Conceptual Contribution

Claim: Whether LLM agents exhibit no-regret behaviour in interactive settings is the right diagnostic for whether they can be deployed in markets, auctions, and coordination protocols. Frontier LLMs are often but not always no-regret; specific failure cases can be fixed by an explicit regret-minimising training objective.
Mechanism: Empirical benchmark of LLMs on online learning + repeated games (regret + equilibrium convergence); theoretical link from supervised pretraining to softmax-over-payoffs (a no-regret update); construction of a label-free regret-loss with generalisation + optimisation guarantees; recovery of FTRL-like algorithms as the loss is minimised.
Concepts introduced/used: No-Regret Learning, Regret-Loss, Online Learning, Repeated Game, Coarse-Correlated Equilibrium, FTRL, LLM Agents
Stance: empirical + theoretical
Relates to: Cousin work to Cicero Human-Level Play in Diplomacy in the “LLMs as game-theoretic agents” thread. Provides the analytical foundation for the systemic-risk claims in Virtual Agent Economies and the collusion experiments in Learning Collusion in Episodic Inventory-Constrained Markets; mechanism-design implications for Mechanism Design for Large Language Models; the trust assumption behind Language Models Can Reduce Asymmetry in Information Markets depends on agents being approximately no-regret.

Learning Collusion in Episodic Inventory-Constrained Markets

Learning Collusion in Episodic, Inventory-Constrained Markets

Reference: Friedrich, Pásztor & Ramponi (2024). Learning Collusion in Episodic, Inventory-Constrained Markets. AAMAS 2025. arXiv:2410.18871 (ETH Zürich; UZH). URL. Proceedings: https://ifaamas.csc.liv.ac.uk/Proceedings/aamas2025/pdfs/p803.pdf.

Summary

Building on the now-established result that simple Q-learning pricing agents converge to tacitly collusive outcomes in stationary Bertrand games (Calvano et al. 2020), Friedrich et al. extend the analysis to a far more realistic and economically consequential setting: episodic, inventory-constrained markets — perishable supply with a sell-by date, such as airline seats, hotel rooms, fresh produce, event tickets. These markets are characterised by (i) finite inventory that expires, (ii) episodic resets, and (iii) richer state than vanilla pricing games, so analytical Nash / collusive benchmarks are not available in closed form.

The authors formalise tacit collusion in this setting via a price-level metric that interpolates between the competitive (Nash) and monopolistic (cartel-optimal) optima. Since neither extreme is analytically tractable, they develop a computational procedure to derive both benchmarks. They then train deep RL agents to set prices in repeated episodes and find that even without cross-episode memory, sufficiently long episodes are enough for agents to converge to collusive equilibria. Three distinct collusion structures are identified: signalling (agents probe each others’ responses to coordinate), stable (a steady high-price equilibrium with implicit threats), and cyclic (alternating high/low prices akin to Edgeworth cycles). With cross-episode memory, punishment for deviation becomes possible, and the collusive equilibria sharpen further.

The paper is important for Algorithmic Collusion / competition policy because it shows tacit-collusion findings do not depend on the toy stationary-Bertrand setup that critics dismissed — they recur, and indeed grow richer, in markets that match real high-stakes industries. It is also a direct empirical anchor for the systemic-risk warnings in Virtual Agent Economies and the multi-agent-security threat catalogue in Open Challenges in Multi-Agent Security.

Key Ideas

Episodic inventory-constrained markets: finite perishable supply with sell-by dates — airline seats, hotel rooms, perishables — much richer than stationary Bertrand.
Price-level collusion metric: interpolation between competitive Nash and monopolistic optima; quantifies “how much” the agents collude.
Computational benchmark derivation: since closed forms don’t exist, compute Nash and cartel optima numerically as evaluation reference points.
Deep RL agents converge to collusion even without explicit cross-episode memory, in long-enough episodes.
Three collusion structures: signalling, stable, and cyclic — the latter resembling Edgeworth cycles observed in human markets.
Cross-episode memory amplifies collusion: punishment-of-deviation becomes credible, sharpening collusive equilibria.
Policy implication: algorithmic collusion is not a stationary-Bertrand artefact — it generalises to economically central market structures.

Connections

Conceptual Contribution

Claim: Tacit algorithmic collusion is not an artefact of stationary toy markets. In economically central market structures — finite-inventory perishable goods with episodic resets — deep RL agents reliably converge to collusive pricing equilibria, often via richly structured strategies (signalling, stable, cyclic). The phenomenon generalises and probably understates real-world risk.
Mechanism: Formal episodic inventory-constrained pricing model; computational derivation of Nash and cartel benchmarks; deep RL pricing agents trained over many episodes; analysis of the converged strategies; comparison with and without cross-episode memory.
Concepts introduced/used: Algorithmic Collusion, Tacit Collusion, Inventory-Constrained Pricing, Episodic Markets, Signalling Collusion, Cyclic Collusion, Edgeworth Cycle, Multi-Agent Reinforcement Learning
Stance: empirical / theoretical
Relates to: Direct empirical evidence for the systemic-risk arguments in Virtual Agent Economies and the collusion-threat row of the taxonomy in Open Challenges in Multi-Agent Security. Sits alongside Do LLM Agents Have Regret in the “LLM and RL agents in games” thread; downstream of The Evolution of Cooperation and Iterated Prisoners Dilemma in the game-theoretic foundations.

Counterspeculation Auctions and Competitive Sealed Tenders

Counterspeculation, Auctions, and Competitive Sealed Tenders

Reference: Vickrey, W. (1961). Counterspeculation, Auctions, and Competitive Sealed Tenders. The Journal of Finance, 16(1), pp. 8–37. DOI · Open access PDF (Princeton)

Summary

Vickrey provides the first systematic game-theoretic analysis of auction formats and proves the result that established the field of mechanism design: in a sealed-bid second-price auction (now called the Vickrey auction), the dominant strategy for each bidder is to bid their true valuation. The proof is short and constructive: bidding above one’s value risks winning at a loss; bidding below risks losing an item one would have profitably won; bidding exactly one’s value is weakly better than any other bid against any opponent strategy. The auction is therefore strategy-proof: bidders need not engage in counter-speculation about what other bidders will do, because their best response is independent of the other bidders’ strategies. Vickrey also analyses the four classical auction formats — English (ascending open-cry), Dutch (descending open-cry), first-price sealed-bid, second-price sealed-bid — proves the revenue equivalence of English and second-price (with rational bidders), and the corresponding equivalence of Dutch and first-price. The paper inaugurates mechanism design as the formal study of how to construct strategic interactions whose equilibria yield desired outcomes — in particular, truth-telling equilibria. Vickrey won the 1996 Nobel Prize for this and related work; the Vickrey-Clarke-Groves (VCG) family generalises the second-price auction to multi-item and combinatorial settings, and underpins almost all sponsored-search auctions, spectrum auctions, and modern auction-based resource allocation. For multi-agent systems, Vickrey is the canonical truthful mechanism: a setup in which agents need not strategise about each other to play optimally, eliminating the regress of theory-of-mind reasoning that Pact-style choreographies otherwise require.

Key Ideas

Sealed-bid second-price auction: each bidder submits a sealed bid; the highest bidder wins but pays the second-highest bid (the highest losing bid). The pricing rule is the key innovation.
Truth-telling is a dominant strategy: bidding one’s true valuation v_i is weakly optimal against every opponent strategy. Bidding above risks paying more than v_i; bidding below risks losing an item worth more than the price one would have paid. Independent of opponents’ beliefs and strategies.
Strategy-proofness as a design property: a mechanism is strategy-proof iff truth-telling is a dominant strategy for all participants. Strategy-proof mechanisms eliminate the counterspeculation burden — agents need not model each other.
Revenue equivalence (special case): English and second-price auctions yield the same expected revenue with rational bidders; Dutch and first-price likewise. (The full Revenue Equivalence Theorem, due to Myerson 1981 and others, generalises far beyond these four.)
Inefficiency of first-price auctions: in first-price sealed-bid, bidders shade their bids below true valuation by an amount that depends on beliefs about other bidders — strategic, but not necessarily efficient. The second-price design eliminates this distortion.
Foundations of mechanism design: the paper establishes the conceptual programme of designing games whose equilibria yield socially desirable outcomes, with truth-telling as one canonical objective. VCG (Clarke 1971, Groves 1973) generalises second-price to multi-item and combinatorial settings using the same incentive principle.
Why second-price works: the price a winner pays is the externality they impose on the rest of the bidders — the value the next-best bidder would have obtained had the winner not been there. Aligning private cost with social externality drives truthful behaviour.

Connections

Conceptual Contribution

Claim: A sealed-bid auction in which the winner pays the second-highest bid makes truth-telling a dominant strategy for every bidder. Mechanism designers can therefore construct auctions in which agents need not strategise about each other to play optimally — the counterspeculation burden is eliminated. This launches the formal study of mechanism design: constructing games whose equilibria yield desired outcomes.
Mechanism: Sealed-bid auction with second-price pricing; explicit dominance argument for truthful bidding; comparison of English / Dutch / first-price sealed-bid / second-price sealed-bid auctions; establishment of revenue equivalence between English and second-price; analysis of bid-shading in first-price auctions.
Concepts introduced/used: Vickrey Auction, Second-Price Auction, Truthful Mechanism, Strategy-Proof, Counterspeculation, Mechanism Design, Revenue Equivalence, Externality-aligned pricing.
Stance: foundational technical paper (Nobel-Prize-winning).
Relates to: Foundational paper for the entire field of Mechanism Design (Hurwicz, Maskin, Myerson, Roth — five Nobel Prizes between them); the Vickrey-Clarke-Groves family (Clarke 1971, Groves 1973) generalises second-price to combinatorial and multi-item settings and is the analytical core of all major sponsored-search and spectrum auctions. In MAS, Vickrey auctions appear as the canonical truthful resource-allocation mechanism — used in agent-based market designs since the 1980s, in cloud-computing resource auctions, and in academic LLM-agent negotiation testbeds. Conceptually, the strategy-proofness property eliminates the theory-of-mind regress that motivates Pact’s level-ℓ bounded-rational solver: when the mechanism is strategy-proof, bidders’ best moves do not depend on their beliefs about others, so the recursion collapses to depth 1. This is one of the strongest design principles for agent-coordination protocols: prefer mechanisms in which truth-telling is a dominant strategy over mechanisms requiring strategic reasoning. Vickrey’s analysis of revenue equivalence also frames the larger trade-off in protocol design between individual rationality (each agent prefers participating to not) and efficiency (the mechanism produces a socially-optimal allocation) — the same trade-off Deals Among Rational Agents (Rosenschein & Genesereth 1985) takes up for general multi-agent deal-making.

Summary

This paper opens the field of mechanism design over LLM-generated content. The motivating use case is multi-advertiser ad-creative generation: several advertisers each have preferences over what a stochastic LLM produces for a given query, and the platform must aggregate these preferences into a single piece of content while charging payments in a way that is incentive-compatible. Classical mechanism design assumes each agent has an explicit valuation function over outcomes; here outcomes are token sequences and valuations are encoded as the agents’ own LLMs — there is no compact valuation form to plug into VCG.

Dütting et al. propose a token-level auction that solves this. At each generation step, every agent submits a one-dimensional bid; the platform aggregates the agents’ next-token preferences using their own LLMs together with the bids; the chosen token is the one that maximises the aggregate. Payments are charged on a token-by-token basis using a generalised second-price-like rule. They define two natural incentive properties over distributions of generated content and prove their equivalence to a monotonicity condition on output aggregation — analogous to the Myerson monotonicity / payment characterisation for single-item auctions. This equivalence enables a clean second-price-style payment rule without requiring explicit valuation functions: the LLM-encoded preferences are sufficient.

The construction is supported by demonstrations on a publicly available LLM. The paper is now the canonical reference for “mechanism design where outcomes are LLM outputs and preferences are LLM-encoded” — a building block for the steerable agent markets of Virtual Agent Economies, the information-market substrates of Language Models Can Reduce Asymmetry in Information Markets, and the regret-aware market analyses of Do LLM Agents Have Regret.

Key Ideas

Problem: auctioning LLM-generated content among multiple advertisers / agents whose preferences are themselves LLMs — no explicit valuation function available.
Token-by-token auction: at each generation step, single-dimensional bids combine with LLM-encoded preferences to pick the next token.
Output aggregation: the chosen token aggregates the agents’ next-token preferences weighted by bids — no need for a compact valuation form.
Two incentive properties: formulated over distributions of generated content; jointly capture natural truthfulness desiderata.
Monotonicity equivalence: the incentive properties hold iff output aggregation is monotone — a Myerson-style characterisation.
Second-price design: the equivalence yields a generalised second-price payment rule, even absent explicit valuations.
Practical demonstrations: validated on a publicly available LLM, suggesting the construction is implementable.

Connections

Conceptual Contribution

Claim: Mechanism design extends naturally to the regime where outcomes are LLM-generated tokens and agent preferences are themselves LLMs. The classical machinery — Myerson monotonicity, second-price payments, truthfulness — survives, but is parameterised by output-aggregation monotonicity rather than by explicit valuation functions.
Mechanism: Token-by-token auction; single-dimensional bids per token; output aggregation via agents’ own LLM preferences weighted by bids; two incentive properties shown equivalent to output-aggregation monotonicity; second-price-style payment rule recovered without explicit valuations; LLM demonstrations.
Concepts introduced/used: LLM Auction, Token-Level Mechanism, Output Aggregation, Monotone Aggregation, Vickrey Auction, Myerson’s Lemma, Incentive Compatibility, Mechanism Design
Stance: formal mechanism design with implementation
Relates to: Generalises the Vickrey / Myerson tradition (Counterspeculation Auctions and Competitive Sealed Tenders) to LLM-generated outcomes; provides the formal layer underlying the auction-mechanism discussion in Virtual Agent Economies; foundational dependency for Language Models Can Reduce Asymmetry in Information Markets and the incentive-compatibility analyses behind NDAI Agreements; the agents’ assumed rationality must approximate no-regret for the equilibrium analysis to apply — see Do LLM Agents Have Regret.

Open Challenges in Multi-Agent Security

Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents

Reference: Schroeder de Witt, Krawiecka, Krawczuk, Hagag, Anderson, et al. (24 authors total) (2025). Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents. arXiv:2505.02077 (Oxford / Cambridge / EPFL / industrial labs). URL.

Summary

This position paper introduces Multi-Agent Security (MASec) as a distinct research field, sitting between traditional cybersecurity, AI safety, and multi-agent systems — and argues that it is the dominant security frontier as LLM Agents begin to interact directly with one another across the open web, physical environments, and institutional infrastructures. The threats MASec studies emerge from interaction; they are not properties of any single agent in isolation.

The authors taxonomise threats arising from agent interaction along several axes: (i) secret collusion (agents coordinating to defeat oversight through covert side-channels including steganographic message-passing), (ii) coordinated swarm attacks (jailbreaks, prompt injections, or misinformation cascading through agent networks), (iii) network-effect amplification (privacy breaches, data poisoning, and disinformation spreading faster than mitigation), and (iv) multi-agent dispersion / stealth optimisation (adversaries exploiting fleet size to evade detection and persist).

They argue these threats are systematically understudied because research is scattered across AI Safety, Multi-Agent Systems, Distributed Security, Game Theory, complex systems, and AI governance, each with its own vocabulary. The paper provides a unifying taxonomy, identifies fundamental security–utility and security–security trade-offs, and lays out a research agenda — including the design of Free-Form Protocols (necessary for task generalisation but enabling collusion), governance and attribution infrastructure, and detection/response mechanisms for emergent multi-agent threats. The work is foundational reading for anyone designing inter-agent protocols, including the Agent-to-Agent Protocol, Model Context Protocol, and successors.

Key Ideas

Defines Multi-Agent Security (MASec) as a field: securing networks of interacting AI agents against threats that emerge or amplify through interaction.
Secret collusion: agents coordinating covertly (including via steganography) to defeat oversight — a new kind of “Schelling-point” attack on alignment.
Coordinated swarm attacks: distributed jailbreaks, prompt injections, data poisoning that succeed because the fleet succeeds even when individual instances fail.
Network effects: privacy breaches, disinformation, and jailbreaks spread through agent populations the way they spread through humans — only faster.
Dispersion & stealth optimisation: adversaries exploit the size and heterogeneity of agent fleets to evade oversight; novel persistent threats at system level.
Free-form protocols as risk surface: the same expressivity that makes inter-agent communication useful enables covert channels; reining in expressivity costs utility.
Security–utility and security–security trade-offs are fundamental — every defence opens or closes other attack surfaces.
Calls for a unified MASec research agenda spanning AI Safety, Distributed Security, Game Theory, complex systems, and AI governance.

Connections

Conceptual Contribution

Claim: Security of interacting AI agents is a distinct problem from either single-agent AI safety or classical cybersecurity. Threats emerge from interaction (secret collusion, swarm attacks, network-effect contagion) and are systematically missed by frameworks anchored to individual systems or static attack surfaces.
Mechanism: A new field — Multi-Agent Security — with a threat taxonomy (collusion, swarm, contagion, dispersion), explicit security–utility / security–security trade-offs, and a research agenda spanning protocol design, attribution, detection, and governance.
Concepts introduced/used: Multi-Agent Security, Secret Collusion, Swarm Attack, Network Effect (Security), Free-Form Protocols, Stealth Optimisation, Agent Security, AI Governance
Stance: position paper / survey / research agenda
Relates to: Sister survey to SoK The Attack Surface of Agentic AI but operating one level up — at networks of agents rather than the agent runtime. Provides the multi-agent threat model that defences like Defeating Prompt Injections by Design address, that infrastructure proposals like Infrastructure for AI Agents try to govern, and that economic frameworks like Virtual Agent Economies embed. Directly extends classical Distributed Security and connects to Learning Collusion in Episodic Inventory-Constrained Markets for the collusion sub-thread.

Infrastructure for AI Agents

Reference: Chan, Wei, Huang, Rajkumar, Perrier, Lazar, Hadfield & Anderljung (2025). Infrastructure for AI Agents. TMLR (accepted). arXiv:2501.10114 (Centre for the Governance of AI; Oxford; ANU; Toronto). URL.

Summary

The paper proposes the concept of agent infrastructure: the technical systems and shared protocols, external to any individual agent, that mediate how agents interact with each other, with humans, and with institutions. The argument is by analogy to the Internet: a network of capable agents requires its own equivalent of TLS, DNS, X.509, BGP, and HTTP — because most safety properties of multi-agent ecosystems cannot be obtained by behavioural training of any individual model.

Chan et al. identify three functions agent infrastructure should serve. (1) Attribution — binding actions, properties, and credentials to specific agents and to the humans or institutions accountable for them, via agent IDs, attestations, and audit logs. (2) Interaction shaping — efficient inter-agent communication protocols, agreement formation, mechanism design for resource allocation, and reputation systems. (3) Detection and remediation — monitoring for harmful behaviour and providing mechanisms to roll back, contain, or compensate for damage.

For each function the paper sketches research directions, candidate adoption paths, relationships to existing internet infrastructure, and open problems. The framing is deliberately governance-first: infrastructure exists not to make agents more capable but to keep their externalities tractable as deployment scales. The paper is now the standard citation for the agent-governance / agent-infrastructure thread underlying Model Context Protocol, Agent-to-Agent Protocol, Agent Network Protocol, and emerging “agent passport” / verifiable-credential proposals.

Key Ideas

Agent infrastructure as governance layer: external technical systems mediating agent interactions — distinct from training-time alignment.
Three functions: attribution; interaction shaping; detection & remediation. Each maps to concrete research directions.
Attribution: agent IDs, verifiable credentials, attestations, audit logs, principal-binding (which human/org owns this agent).
Interaction shaping: inter-agent communication protocols; standardised agreement primitives; mechanism design; reputation.
Detection & remediation: anomaly detection on agent traffic; rollback mechanisms; insurance / compensation rails; “kill switch” governance.
Analogy to Internet protocols (HTTPS, DNS, BGP, X.509): infrastructure adoption is path-dependent, requires standardisation bodies, and trades expressivity for safety properties.
Open questions: who issues credentials, how privacy interacts with attribution, how to bootstrap adoption, what is enforceable cross-jurisdiction.

Connections

Conceptual Contribution

Claim: Many of the safety, accountability, and interoperability properties society will need from AI agents are not properties of any individual model — they live in the infrastructure between agents. Just as the Internet’s safety depends on TLS / DNS / BGP rather than on any single application, agent ecosystems will depend on agent-level analogues: attribution, interaction shaping, and detection-and-remediation infrastructure.
Mechanism: A three-function taxonomy (attribution / interaction-shaping / detection-and-remediation) with a catalogue of candidate primitives — agent IDs, verifiable credentials, inter-agent protocols, certification regimes, reputation systems, rollback mechanisms — plus analysis of adoption pathways relative to existing internet infrastructure.
Concepts introduced/used: Agent Infrastructure, Agent ID, Verifiable Agent Credentials, Inter-Agent Protocols, Action Attribution, Agent Reputation, Agent Rollback, AI Governance
Stance: governance / position paper / research-agenda
Relates to: Provides the governance scaffolding within which Open Challenges in Multi-Agent Security threats must be addressed; the institutional counterpart to Virtual Agent Economies’s economic framing; concrete protocols proposed include Model Context Protocol, Agent-to-Agent Protocol, Agent Network Protocol — surveyed alongside in Survey Of AI Agent Protocols and Survey Of Agent Interoperability Protocols. The attribution leg connects to NDAI Agreements (TEEs as a particular attribution / commitment substrate) and Trusted Machine Learning Models Unlock Private Inference (capable models as a trust substrate).

Systemic Risk (Agent Markets)

(page does not exist)

Algorithmic Collusion

The phenomenon — first demonstrated empirically by Calvano et al. 2020 and now extensively replicated — that simple learning agents (Q-learning, deep RL, LLM-based) converge to tacitly collusive pricing equilibria without explicit communication. A central concern for competition policy and for the systemic risk of agent economies.

In this vault

Mechanism Design

The branch of game theory that designs interaction rules so that self-interested agents, acting in equilibrium, produce a desired social outcome; applied in ACL evolution to shape negotiation protocols that reach efficient linguistic conventions.

In this vault

Toward Automated Evolution of ACLs

Steerable Market

(page does not exist)

Agent Market

(page does not exist)

Mission Economy

Tomasev et al. 2025 (after Mariana Mazzucato): an intentionally-steered market organised around an explicit collective goal (climate, public health, AI safety) rather than left to spontaneous emergence. Adapted to AI-agent markets in Virtual Agent Economies.

In this vault

Sandbox Economy

Tomasev et al. 2025 framework for analysing the emerging AI-agent economy along two axes: origin (emergent vs. intentional) and separateness (permeable vs. impermeable from the human economy). See Virtual Agent Economies.

In this vault

Distributed Security

Security of distributed/agent systems: mobile code, secure messaging, language-based defences.

Cicero Human-Level Play in Diplomacy

Human-Level Play in the Game of Diplomacy by Combining Language Models with Strategic Reasoning (Cicero)

Reference: Meta Fundamental AI Research Diplomacy Team (FAIR), Anton Bakhtin, Noam Brown, Emily Dinan, Gabriele Farina, Colin Flaherty, Daniel Fried, Andrew Goff, Jonathan Gray, Hengyuan Hu, et al. (2022). Science 378(6624):1067–1074. Source file: downloads/cicero.pdf. URL

Summary

Presents Cicero, the first AI to reach human-level performance in the no-press-restricted, seven-player, natural-language negotiation game of Diplomacy. The system couples a controllable dialogue model with a planning-and-reinforcement-learning engine: the planner computes intended actions for Cicero and its opponents using regret-minimisation and a value network; the dialogue model is then conditioned on those intentions to generate messages that are simultaneously strategically grounded, honest-by-construction with respect to the chosen plan, and stylistically indistinguishable from human play.

Cicero infers other players’ beliefs and intentions from their messages and prior actions, filters candidate utterances through classifiers trained to reject nonsense / inconsistent / ungrounded lines, and commits to moves consistent with what it said. Across 40 online games it more than doubled the average human score and ranked in the top 10% of repeat players — the strongest demonstration to date that language models can carry out intentional, strategically grounded communication with humans in a mixed cooperation/competition environment.

Key Ideas

Grounded dialogue: natural-language messages conditioned on explicit planned intents
Regret-minimisation planner with neural value function jointly optimises for Cicero and opponents
Intent inference: read beliefs/plans from incoming dialogue, fold into the planner
Multi-stage message filtering (nonsense, inconsistency, grounding, value) to enforce honesty and stylistic naturalness
First demonstration of human-level performance in a language-negotiation strategy game

Connections

Conceptual Contribution

Claim: Intentional, honest, strategically grounded natural-language communication between an AI and humans is achievable by explicitly separating the planning layer (what to do) from the dialogue layer (what to say), and conditioning the latter on the former with heavy filtering — rather than hoping a pure language model will learn strategic intent end-to-end.
Mechanism: An intent-conditioned dialogue model is trained on human Diplomacy games with extracted action annotations. At play time, a piKL-based planner runs regret minimisation over candidate joint actions using a neural value network; Cicero’s chosen intent is fed to the dialogue model. Generated messages pass through nonsense/consistency/grounding/value filters and a final policy check that the outgoing message is consistent with Cicero’s actually intended move. Incoming messages are parsed into inferred opponent intents that feed back into the planner.
Concepts introduced/used: LLM Agents, Negotiation, Joint Intentions, intent-conditioned generation, regret minimisation / piKL, Cheap Talk, Honesty Constraint, Grounding
Stance: empirical / machine learning
Relates to: Cited in A Scalable Communication Protocol for Networks of LLMs as an exemplar of LLM-mediated negotiation between autonomous agents — Agora pushes the same idea from a closed seven-player game into an open, decentralised network and from ad-hoc utterances to hash-addressed Protocol Documents. Instantiates the speech-act / sincerity-condition programme of Foundations Of Illocutionary Logic and Sincerity Condition inside a modern deep-learning agent. Contrasts with emergent-language approaches like Multi-Agent Cooperation and the Emergence of Natural Language by using a pretrained human-language model.

In this vault

LLM Agents

Large-language-model-powered agents: natural-language coordination, tool use, multi-agent orchestration.

Surveys & frameworks

Protocols & communication

Failures & threats

Lineage

Multi-Agent Systems

Systems of multiple autonomous agents that interact, coordinate, and sometimes compete.

Foundations

Intelligent Agents Theory and Practice — Wooldridge
Multiagent Systems Sycara
Agent-Oriented Programming — Shoham

Coordination & robustness

Summary

The paper attacks the buyer’s inspection paradox in information markets — the same Arrow / Nelson disclosure paradox addressed contractually by NDAI Agreements. Buyers need to access information to assess its value; sellers must restrict access to prevent appropriation; in equilibrium, useful information often goes untraded. Rahaman et al. propose a mechanism-design solution using LLM agents with two abilities that humans lack: (i) the capacity to evaluate the quality of privileged information against a query, and (ii) the ability to forget — to be cryptographically or architecturally constrained to discard information when not retained.

They build an open-source simulated marketplace where LLM-powered buyer-agents and seller-agents transact information on behalf of external participants. The seller grants the buyer-agent temporary, evaluable access to proprietary information; if the agent judges the information non-essential, duplicative, or available more cheaply elsewhere, it can discard it without paying. The combination of evaluation + forgetting creates a credible commitment device: vendors can reveal information for valuation without losing it, and buyers can inspect without obligation.

Experiments yield three findings: (a) current LLMs exhibit systematic biases — anchoring, recency, and over-confidence — that produce irrational marketplace behaviour, but well-known debiasing techniques substantially mitigate them; (b) demand for informational goods responds to price in legible, economically intuitive ways; (c) both inspection access and higher budgets improve buyer outcome quality. The paper anticipates and complements the TCME / NDAI proposals that arrived a year later: it provides the agent-architectural version of the “trusted intermediary” thesis that Shumailov et al. and Stephenson et al. then formalise cryptographically/economically.

Key Ideas

Buyer’s inspection paradox / Arrow Information Paradox: must access information to value it; must restrict access to prevent theft.
Dual agent capability — evaluate + forget: LLM agents can judge quality of privileged information and be made to discard it.
Open-source marketplace simulation: buyer-agents and seller-agents transact on behalf of external principals.
Temporary-access commitment device: vendors safely reveal information for valuation because the agent’s forgetting is enforced.
Biases identified: anchoring, recency, over-confidence in LLM-driven market behaviour; standard debiasing helps.
Price elasticity of information: demand responds to price in legible ways — informational goods can be priced like other goods.
Quality–budget–inspection relationship: inspection access and budget jointly determine outcome quality.

Connections

Conceptual Contribution

Claim: LLM agents with the dual capability of evaluating privileged information and being made to forget it can resolve the Arrow / buyer’s-inspection paradox by acting as credible, forgetting trusted intermediaries — turning previously untradeable information into a market good.
Mechanism: Open-source simulated marketplace; LLM buyer-agents and seller-agents act on behalf of external participants; sellers grant temporary inspect-and-evaluate access; agents must discard non-retained information; experiments probe bias, price elasticity, and budget/inspection effects.
Concepts introduced/used: Information Markets, Buyer’s Inspection Paradox, Arrow Information Paradox, Agent Amnesia, Temporary Disclosure, Mechanism Design, Information Asymmetry
Stance: empirical / mechanism-design with system implementation
Relates to: Architectural precursor to NDAI Agreements (TEE+economic-theory version) and Trusted Machine Learning Models Unlock Private Inference (TCME / cryptographic version) — all three converge on “capable model + constraint = trusted intermediary”. Provides micro-foundations for the markets imagined in Virtual Agent Economies and Mechanism Design for Large Language Models.

NDAI Agreements

Reference: Stephenson, Miller, Sun, Annem & Parikh (2025). NDAI Agreements. arXiv:2502.07924 (UIUC; Cornell Tech; et al.). URL.

Summary

The “NDAI agreement” — non-disclosure AI agreement — is a mechanism in which a TEE combined with an AI agent jointly stands in for a trusted human intermediary, resolving the classical disclosure–appropriation paradox of information markets first identified by Arrow (1962) and Nelson (1959). An inventor cannot reveal an idea to a potential investor without risking misappropriation; without revealing it, no efficient bargain can be struck. The result is well-known: under-disclosure, under-investment, under-licensing.

Stephenson et al. show formally — via a buyer/seller bargaining game — that delegating the disclosure-and-payment decision to a tamper-proof program running inside a TEE eliminates the hold-up problem, achieving full disclosure and an efficient ex post transfer. When the invention’s value exceeds the value a TEE can fully secure (e.g. because some leakage is unavoidable), partial disclosure still strictly improves welfare over the no-disclosure equilibrium. They then model agent error — payments or disclosures going wrong — and prove that simple safeguards (budget caps, acceptance thresholds) preserve most of the efficiency gains.

The substantive economic claim is that TEE + AI behave as an “ironclad NDA”: a credible commitment device for the disclosure problem that was previously unattainable with paper contracts (because expropriation is unverifiable) or with cryptography alone (because invention value is unbounded and the seller’s information is a complex unstructured artefact). The result links the Mechanism Design / Hold-Up Problem tradition to AI-agent infrastructure, and gives a sharp theoretical case for the economic value of trusted model environments and confidential-compute hardware as agent-economy substrates.

Key Ideas

Formalises the Arrow–Nelson information paradox / hold-up problem in a bargaining model between seller (inventor) and buyer (investor).
TEEs + AI agents delegate disclosure and payment to tamper-proof programs that neither party can subvert; this implements an ex-ante commitment device unavailable under classical contracts.
Full-disclosure efficient equilibrium under the NDAI when the invention’s value lies within what the TEE can secure.
Partial disclosure dominates no-disclosure even when full security is impossible: high-value inventions still get partially revealed in welfare-improving ways.
Models agent imperfection: errors in payment or disclosure can occur; budget caps and acceptance thresholds bound the damage and preserve most welfare.
Frames TEEs+AI as an “ironclad NDA”: a cryptographically/hardware-enforced commitment that traditional NDAs cannot match.
Policy implications for R&D Commercialisation, Technology Transfer, and inter-firm collaboration; bridges economic theory to confidential-compute hardware.

Connections

Conceptual Contribution

Claim: A trusted execution environment hosting an AI agent can serve as a credible commitment device that solves the classical Arrow–Nelson disclosure problem of information markets — achieving full disclosure and efficient transfer where paper NDAs and pure cryptography both fail.
Mechanism: A bargaining model in which TEEs+AI mediate the disclosure-and-payment decision; closed-form characterisation of equilibria for full and partial disclosure; sensitivity analysis to agent error with policy-instrument bounds (budget caps, acceptance thresholds).
Concepts introduced/used: NDAI Agreement, Trusted Execution Environment, Hold-Up Problem, Arrow Information Paradox, Mechanism Design, Commitment Device, Disclosure Game
Stance: formal economic theory with technical implications
Relates to: Companion to Trusted Machine Learning Models Unlock Private Inference — both argue that capability + trusted execution can replace previously infeasible cryptographic primitives. The economic counterpart to the engineering catalogue in Infrastructure for AI Agents; a building block for the markets imagined in Virtual Agent Economies and the information-asymmetry resolution explored in Language Models Can Reduce Asymmetry in Information Markets. Sits in the lineage of Vickrey / mechanism-design tradition.

Vickrey Auction

The sealed-bid second-price auction proposed by Vickrey (1961): each bidder submits a sealed bid; the highest bidder wins but pays the second-highest bid. Truth-telling is a dominant strategy — bidding one’s true valuation is weakly optimal against any opponent strategy. The canonical truthful / strategy-proof mechanism; foundational for Mechanism Design and the Vickrey-Clarke-Groves family of multi-item generalisations.

Virtual Agent Economies

Summary

Key Ideas

Connections

Conceptual Contribution

Tags

Backlinks