Algorithmic Collusion

The phenomenon — first demonstrated empirically by Calvano et al. 2020 and now extensively replicated — that simple learning agents (Q-learning, deep RL, LLM-based) converge to tacitly collusive pricing equilibria without explicit communication. A central concern for competition policy and for the systemic risk of agent economies.

In this vault

Backlinks

Linked Pages

The Evolution of Cooperation

Reference: Axelrod, R. (1984). The Evolution of Cooperation. Basic Books, New York. (Revised 2006 with new afterword by Axelrod and a foreword by Richard Dawkins. Underlying conference papers: Axelrod, R. & Hamilton, W. D. (1981). The Evolution of Cooperation. Science 211(4489), pp. 1390–1396.) Science 1981 DOI · Internet Archive borrow (book)

Summary

Axelrod investigates how cooperation can arise and persist among self-interested agents in the absence of central authority — by running computer tournaments of strategies for the iterated prisoner’s dilemma (IPD). In the one-shot prisoner’s dilemma, defection is the dominant strategy and rational play leads both parties to a Pareto-inferior outcome. Axelrod’s central question is whether iteration changes the picture. He invited game theorists to submit strategies; in two tournaments (1979 and 1980, the second much larger and including strategies designed to exploit the first round’s lessons) tit-for-tat — submitted by Anatol Rapoport, only four lines of code (cooperate on the first move, then copy the opponent’s previous move) — won both. The book’s analytic contribution is identifying why. Axelrod isolates four properties of successful IPD strategies: niceness (never defect first), retaliation (punish defection promptly), forgiveness (return to cooperation as soon as the opponent does), and clarity (be predictable so the opponent can learn to cooperate with you). He proves an evolutionary-stability result: a population of tit-for-tat players cannot be invaded by any non-cooperative strategy if the discount factor (probability of further interaction) is sufficiently high. The book extends the analysis to historical case studies — most famously the live-and-let-live system in WWI trench warfare — and to biological evolution of cooperation in symbiosis and group selection. Axelrod’s framework supplies the foundation for mechanism-design approaches to multi-agent cooperation, the theoretical underpinning of trust-and-reputation systems, and the contemporary literature on cooperative AI.

Key Ideas

Iterated prisoner’s dilemma changes everything: in repeated play with sufficient probability of further interaction, cooperation is rationally sustainable; the shadow of the future makes defection costly.
Tit-for-tat wins both tournaments: the simple strategy of cooperating first, then copying the opponent’s previous move, beats much more sophisticated strategies because it is nice (avoids unprovoked defection), retaliatory (punishes promptly), forgiving (returns to cooperation immediately), and clear (transparent enough that opponents can learn to cooperate).
Four properties of successful strategies: nice (never defect first), retaliating (defect immediately if opponent defects), forgiving (cooperate again as soon as opponent does), clear (recognisable as a tit-for-tat-like strategy).
Evolutionary stability: a population of tit-for-tat players resists invasion by non-cooperative mutants when the discount factor w is high enough (w > (T - R) / (T - P) for the standard payoffs). Cooperation is an evolutionary attractor, not just an analytical curiosity.
Cluster invasion: a small cluster of tit-for-tat players in a sea of all-defect can invade the population if the cluster is large enough that within-cluster interactions dominate cross-cluster interactions — explaining how cooperation can emerge from a non-cooperative starting point.
Trench warfare in WWI: extensive case study of how live-and-let-live systems emerged in static trench positions where the same units faced each other repeatedly — a real-world iterated prisoner’s dilemma in which tit-for-tat-like strategies emerged spontaneously and were eventually suppressed only by the introduction of artillery and unit rotation that reduced the “shadow of the future.”
Recommendations for promoting cooperation: enlarge the shadow of the future (longer-term relationships, more frequent interactions), change the payoffs (reduce the temptation to defect), teach reciprocity, improve recognition (allow opponents to remember each other), enhance retaliation capabilities.

Connections

Conceptual Contribution

Claim: Cooperation among self-interested agents arises and persists in the absence of central authority when interactions are iterated with sufficient probability of further interaction. The successful strategies are nice, retaliating, forgiving, and clear; tit-for-tat is the simplest example. A population of tit-for-tat players is evolutionarily stable; small clusters of cooperators can invade non-cooperative populations.
Mechanism: Two open computer tournaments of IPD strategies; analysis of why tit-for-tat won; isolation of the four properties of successful strategies; evolutionary-stability and cluster-invasion theorems with explicit conditions on the discount factor; case studies (WWI trenches, biological symbiosis); concrete recommendations for institutional design.
Concepts introduced/used: Iterated Prisoners Dilemma, Tit-for-Tat, Reciprocity, Shadow of the Future, Niceness/Retaliation/Forgiveness/Clarity, Evolutionary Stability (in IPD), Cluster Invasion.
Stance: foundational research monograph in cooperative game theory and behavioural economics.
Relates to: Foundational for the trust-and-reputation programme in MAS — Review on Computational Trust and Reputation Models and Inter-Agent Trust Models - A Comparative Study systematise the engineering descendants of Axelrod’s tit-for-tat-with-memory across dozens of computational trust models. Conceptual companion of Lewis (1969) and Schelling (1960) in establishing how cooperation/coordination arise without central authority — the three together supply the standard reading list for emergent-cooperation work in MAS. In the LLM-agent era, the IPD setting is the standard testbed for evaluating cooperative AI; the four properties (nice/retaliating/forgiving/clear) translate directly into design criteria for multi-agent LLM coordination protocols, and the clarity property is particularly relevant to the Why AI Agents Communicate In Human Language critique — natural-language strategies are hard to recognise as tit-for-tat-like, so the cooperation-supporting mechanisms Axelrod identified are weakened. The Cooperative AI research programme (Dafoe et al. 2020) is an explicit modern continuation of Axelrod’s project for the multi-agent LLM era.

In this vault

Do LLM Agents Have Regret

Do LLM Agents Have Regret? A Case Study in Online Learning and Games

Reference: Park, Liu, Ozdaglar & Zhang (2024). Do LLM Agents Have Regret? A Case Study in Online Learning and Games. arXiv:2403.16843 (MIT; UMD). URL. OpenReview: https://openreview.net/forum?id=OhZ4u164cN.

Summary

Park et al. ask a sharp question about LLM agents in interactive settings: do they have regret? — i.e. do they exhibit the no-regret behaviour that classical online-learning and game-theoretic algorithms guarantee, and that is necessary for converging to coarse-correlated equilibria in repeated games?

The paper proceeds in three steps. Empirically, they evaluate GPT-3.5 / GPT-4 / Claude / Llama on canonical online-learning benchmarks (prediction-with-expert-advice; bandit-like sequential decision problems) and on repeated games (matrix games, Cournot, Bertrand, public-goods). Frontier LLMs are often no-regret across these settings and often converge to coarse-correlated or Nash equilibria when playing each other. Theoretically, they offer a partial explanation: under stylised assumptions on supervised pre-training and human rationality, the LLM’s next-action distribution approximates a softmax over historical payoffs — which itself implements a no-regret algorithm. But they identify clean failure cases: there exist simple non-stationary or adversarial online-learning instances where even GPT-4 demonstrably accumulates linear regret.

The paper’s constructive contribution is a new regret-loss training objective. Unlike supervised pretraining loss, regret-loss does not require labels of optimal actions — only the historical sequence of plays and payoffs. The authors prove a statistical generalisation bound for regret-loss minimisation and an optimisation guarantee that minimising it can recover known no-regret learning algorithms (e.g. FTRL). Empirically, regret-loss-finetuned models close the gap on the failure cases. The paper is a foundational reference for any analysis of LLM agents in markets, auctions, or interactive coordination — a category that includes Virtual Agent Economies, Mechanism Design for Large Language Models, Learning Collusion in Episodic Inventory-Constrained Markets, and Language Models Can Reduce Asymmetry in Information Markets.

Key Ideas

Regret as a diagnostic for LLM agents in interactive settings: do they no-regret-learn against arbitrary opponents?
Empirical screen: frontier LLMs (GPT-3.5/4, Claude, Llama) on canonical online-learning + repeated-game benchmarks.
Often no-regret in benign settings, often converging to coarse-correlated / Nash equilibria when playing each other.
Theoretical bridge: under stylised pretraining + human-rationality assumptions, the LLM’s next-action distribution resembles a softmax over payoffs — itself a no-regret algorithm.
Identified failure cases: simple non-stationary / adversarial online-learning instances where GPT-4 has linear regret.
Regret-loss objective: label-free training loss that explicitly incentivises no-regret behaviour; statistical and optimisation guarantees.
Recovery of classical algorithms: minimising regret-loss can converge to algorithms like FTRL.

Connections

Conceptual Contribution

Claim: Whether LLM agents exhibit no-regret behaviour in interactive settings is the right diagnostic for whether they can be deployed in markets, auctions, and coordination protocols. Frontier LLMs are often but not always no-regret; specific failure cases can be fixed by an explicit regret-minimising training objective.
Mechanism: Empirical benchmark of LLMs on online learning + repeated games (regret + equilibrium convergence); theoretical link from supervised pretraining to softmax-over-payoffs (a no-regret update); construction of a label-free regret-loss with generalisation + optimisation guarantees; recovery of FTRL-like algorithms as the loss is minimised.
Concepts introduced/used: No-Regret Learning, Regret-Loss, Online Learning, Repeated Game, Coarse-Correlated Equilibrium, FTRL, LLM Agents
Stance: empirical + theoretical
Relates to: Cousin work to Cicero Human-Level Play in Diplomacy in the “LLMs as game-theoretic agents” thread. Provides the analytical foundation for the systemic-risk claims in Virtual Agent Economies and the collusion experiments in Learning Collusion in Episodic Inventory-Constrained Markets; mechanism-design implications for Mechanism Design for Large Language Models; the trust assumption behind Language Models Can Reduce Asymmetry in Information Markets depends on agents being approximately no-regret.

Open Challenges in Multi-Agent Security

Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents

Reference: Schroeder de Witt, Krawiecka, Krawczuk, Hagag, Anderson, et al. (24 authors total) (2025). Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agents. arXiv:2505.02077 (Oxford / Cambridge / EPFL / industrial labs). URL.

Summary

This position paper introduces Multi-Agent Security (MASec) as a distinct research field, sitting between traditional cybersecurity, AI safety, and multi-agent systems — and argues that it is the dominant security frontier as LLM Agents begin to interact directly with one another across the open web, physical environments, and institutional infrastructures. The threats MASec studies emerge from interaction; they are not properties of any single agent in isolation.

The authors taxonomise threats arising from agent interaction along several axes: (i) secret collusion (agents coordinating to defeat oversight through covert side-channels including steganographic message-passing), (ii) coordinated swarm attacks (jailbreaks, prompt injections, or misinformation cascading through agent networks), (iii) network-effect amplification (privacy breaches, data poisoning, and disinformation spreading faster than mitigation), and (iv) multi-agent dispersion / stealth optimisation (adversaries exploiting fleet size to evade detection and persist).

They argue these threats are systematically understudied because research is scattered across AI Safety, Multi-Agent Systems, Distributed Security, Game Theory, complex systems, and AI governance, each with its own vocabulary. The paper provides a unifying taxonomy, identifies fundamental security–utility and security–security trade-offs, and lays out a research agenda — including the design of Free-Form Protocols (necessary for task generalisation but enabling collusion), governance and attribution infrastructure, and detection/response mechanisms for emergent multi-agent threats. The work is foundational reading for anyone designing inter-agent protocols, including the Agent-to-Agent Protocol, Model Context Protocol, and successors.

Key Ideas

Defines Multi-Agent Security (MASec) as a field: securing networks of interacting AI agents against threats that emerge or amplify through interaction.
Secret collusion: agents coordinating covertly (including via steganography) to defeat oversight — a new kind of “Schelling-point” attack on alignment.
Coordinated swarm attacks: distributed jailbreaks, prompt injections, data poisoning that succeed because the fleet succeeds even when individual instances fail.
Network effects: privacy breaches, disinformation, and jailbreaks spread through agent populations the way they spread through humans — only faster.
Dispersion & stealth optimisation: adversaries exploit the size and heterogeneity of agent fleets to evade oversight; novel persistent threats at system level.
Free-form protocols as risk surface: the same expressivity that makes inter-agent communication useful enables covert channels; reining in expressivity costs utility.
Security–utility and security–security trade-offs are fundamental — every defence opens or closes other attack surfaces.
Calls for a unified MASec research agenda spanning AI Safety, Distributed Security, Game Theory, complex systems, and AI governance.

Connections

Conceptual Contribution

Claim: Security of interacting AI agents is a distinct problem from either single-agent AI safety or classical cybersecurity. Threats emerge from interaction (secret collusion, swarm attacks, network-effect contagion) and are systematically missed by frameworks anchored to individual systems or static attack surfaces.
Mechanism: A new field — Multi-Agent Security — with a threat taxonomy (collusion, swarm, contagion, dispersion), explicit security–utility / security–security trade-offs, and a research agenda spanning protocol design, attribution, detection, and governance.
Concepts introduced/used: Multi-Agent Security, Secret Collusion, Swarm Attack, Network Effect (Security), Free-Form Protocols, Stealth Optimisation, Agent Security, AI Governance
Stance: position paper / survey / research agenda
Relates to: Sister survey to SoK The Attack Surface of Agentic AI but operating one level up — at networks of agents rather than the agent runtime. Provides the multi-agent threat model that defences like Defeating Prompt Injections by Design address, that infrastructure proposals like Infrastructure for AI Agents try to govern, and that economic frameworks like Virtual Agent Economies embed. Directly extends classical Distributed Security and connects to Learning Collusion in Episodic Inventory-Constrained Markets for the collusion sub-thread.

Virtual Agent Economies

Reference: Tomasev, Franklin, Leibo, Jacobs, Cunningham, Gabriel & Osindero (2025). Virtual Agent Economies. arXiv:2509.10147 (Google DeepMind). URL.

Summary

The paper provides a conceptual framework — the “sandbox economy” — for analysing the rapidly emerging economic layer in which AI agents transact and coordinate at scales and speeds beyond direct human oversight. It situates the question on two orthogonal axes: (i) origin — whether the agent economy emerged spontaneously from autonomous deployments or was intentionally designed; and (ii) separateness — whether it is permeable to (or insulated from) the established human economy. Most current trajectories occupy the spontaneous × permeable quadrant: vast, fast, and tightly coupled to human markets — the riskiest configuration for systemic externalities.

The authors argue for proactive steerable market design rather than passive emergence. Three design levers receive most of the discussion. (1) Auction mechanisms — adapted VCG / second-price / matching mechanisms — for fair resource allocation and preference resolution among agents. (2) Mission economies — agent markets architected around explicit collective goals (climate, public health, AI safety), where price signals are deliberately steered. (3) Socio-technical infrastructure — accountability, attribution, audit, governance — much of which overlaps with Infrastructure for AI Agents’s programme.

The paper is best read as the economic counterpart to Open Challenges in Multi-Agent Security and Infrastructure for AI Agents: together they delineate the threat surface, governance scaffolding, and economic architecture of the emerging agent economy, and argue that none can be ignored. Risks emphasised include systemic instability (algorithmic flash-crashes spreading to human markets), inequality amplification (agents capturing surplus from price-discrimination at machine speed), and the loss of human-economy slack — the friction that gives humans time to react.

Key Ideas

Sandbox economy framework: two axes — origin (emergent / intentional) × separateness (permeable / impermeable).
Current trajectory: spontaneous + highly permeable agent economy — opportunity and the riskiest configuration for systemic spillover.
Auctions for agent markets: revisits VCG / Vickrey / matching mechanisms for fair allocation and preference resolution among AI participants.
Mission economies: intentionally steered markets aligned to collective goals (climate, public health, AI safety).
Socio-technical infrastructure: trust, attribution, accountability — the governance layer that complements market design.
Systemic risk: flash-crash-like cascades from agent markets into human markets; inequality amplified by machine-speed price discrimination.
Call to proactive design: infrastructure choices now will shape whether the agent economy is steerable or merely emergent.

Connections

Conceptual Contribution

Claim: A vast, permeable AI-agent economy is emerging by default. Letting it emerge unsteered is the highest-risk design choice. Proactive market design — auctions, mission economies, governance infrastructure — is needed to keep agent economies aligned with long-term human flourishing.
Mechanism: A framework characterising agent economies along origin × separateness; a catalogue of three design levers (auctions, mission economies, infrastructure); a discussion of systemic risks and policy implications.
Concepts introduced/used: Sandbox Economy, Mission Economy, Agent Market, Steerable Market, Mechanism Design, Algorithmic Collusion, Systemic Risk (Agent Markets)
Stance: position paper / research agenda
Relates to: Sister piece to Infrastructure for AI Agents (infrastructure framing) and Open Challenges in Multi-Agent Security (threat framing) — these three jointly outline the agent-economy / agent-security / agent-governance space. Auction-design discussion connects to Mechanism Design for Large Language Models (LLM-internal auctions) and Vickrey 1961 (foundational mechanism design). Collusion concerns operationalised in Learning Collusion in Episodic Inventory-Constrained Markets and Do LLM Agents Have Regret.

Learning Collusion in Episodic Inventory-Constrained Markets

Learning Collusion in Episodic, Inventory-Constrained Markets

Reference: Friedrich, Pásztor & Ramponi (2024). Learning Collusion in Episodic, Inventory-Constrained Markets. AAMAS 2025. arXiv:2410.18871 (ETH Zürich; UZH). URL. Proceedings: https://ifaamas.csc.liv.ac.uk/Proceedings/aamas2025/pdfs/p803.pdf.

Summary

Building on the now-established result that simple Q-learning pricing agents converge to tacitly collusive outcomes in stationary Bertrand games (Calvano et al. 2020), Friedrich et al. extend the analysis to a far more realistic and economically consequential setting: episodic, inventory-constrained markets — perishable supply with a sell-by date, such as airline seats, hotel rooms, fresh produce, event tickets. These markets are characterised by (i) finite inventory that expires, (ii) episodic resets, and (iii) richer state than vanilla pricing games, so analytical Nash / collusive benchmarks are not available in closed form.

The authors formalise tacit collusion in this setting via a price-level metric that interpolates between the competitive (Nash) and monopolistic (cartel-optimal) optima. Since neither extreme is analytically tractable, they develop a computational procedure to derive both benchmarks. They then train deep RL agents to set prices in repeated episodes and find that even without cross-episode memory, sufficiently long episodes are enough for agents to converge to collusive equilibria. Three distinct collusion structures are identified: signalling (agents probe each others’ responses to coordinate), stable (a steady high-price equilibrium with implicit threats), and cyclic (alternating high/low prices akin to Edgeworth cycles). With cross-episode memory, punishment for deviation becomes possible, and the collusive equilibria sharpen further.

The paper is important for Algorithmic Collusion / competition policy because it shows tacit-collusion findings do not depend on the toy stationary-Bertrand setup that critics dismissed — they recur, and indeed grow richer, in markets that match real high-stakes industries. It is also a direct empirical anchor for the systemic-risk warnings in Virtual Agent Economies and the multi-agent-security threat catalogue in Open Challenges in Multi-Agent Security.

Key Ideas

Episodic inventory-constrained markets: finite perishable supply with sell-by dates — airline seats, hotel rooms, perishables — much richer than stationary Bertrand.
Price-level collusion metric: interpolation between competitive Nash and monopolistic optima; quantifies “how much” the agents collude.
Computational benchmark derivation: since closed forms don’t exist, compute Nash and cartel optima numerically as evaluation reference points.
Deep RL agents converge to collusion even without explicit cross-episode memory, in long-enough episodes.
Three collusion structures: signalling, stable, and cyclic — the latter resembling Edgeworth cycles observed in human markets.
Cross-episode memory amplifies collusion: punishment-of-deviation becomes credible, sharpening collusive equilibria.
Policy implication: algorithmic collusion is not a stationary-Bertrand artefact — it generalises to economically central market structures.

Connections

Conceptual Contribution

Claim: Tacit algorithmic collusion is not an artefact of stationary toy markets. In economically central market structures — finite-inventory perishable goods with episodic resets — deep RL agents reliably converge to collusive pricing equilibria, often via richly structured strategies (signalling, stable, cyclic). The phenomenon generalises and probably understates real-world risk.
Mechanism: Formal episodic inventory-constrained pricing model; computational derivation of Nash and cartel benchmarks; deep RL pricing agents trained over many episodes; analysis of the converged strategies; comparison with and without cross-episode memory.
Concepts introduced/used: Algorithmic Collusion, Tacit Collusion, Inventory-Constrained Pricing, Episodic Markets, Signalling Collusion, Cyclic Collusion, Edgeworth Cycle, Multi-Agent Reinforcement Learning
Stance: empirical / theoretical
Relates to: Direct empirical evidence for the systemic-risk arguments in Virtual Agent Economies and the collusion-threat row of the taxonomy in Open Challenges in Multi-Agent Security. Sits alongside Do LLM Agents Have Regret in the “LLM and RL agents in games” thread; downstream of The Evolution of Cooperation and Iterated Prisoners Dilemma in the game-theoretic foundations.