Discovering Agents

Reference: Kenton, Z., Kumar, R., Farquhar, S., Richens, J., MacDermott, M. & Everitt, T. (2022). Discovering Agents. arXiv:2208.08345 (DeepMind; University of Oxford; Imperial College London); later published in Artificial Intelligence, vol. 322 (2023), article 103963. URL.

Summary

How can we recognise agents? Causal models of agents are used to analyse the safety properties of machine learning systems, but the causal model is usually just assumed by the modeller, and modelling failures propagate into mistaken safety conclusions. Kenton et al. take the central feature of agency, following Dennett, to be that agents are systems whose outputs are moved by reasons: an agent chooses an action because it expects that action to precipitate an outcome it finds desirable. This distinguishes agents from systems whose output is merely accidentally optimal for some outcome — their example is a rock that happens to be the right size to block a pipe. Systems moved by reasons are systems that would act differently if they “knew” the world worked differently. The paper’s proposal, and the first formal causal definition of agents, is therefore: agents are systems that would adapt their policy if their actions influenced the world in a different way. Counterfactual adaptation is offered as a test for goal-directed influence.

The machinery is the mechanised SCM (Def. 1), a variant of Hammond et al.’s mechanised causal games: an SCM whose endogenous variables are partitioned into object-level variables V (white nodes) and mechanism variables Ṽ (black nodes), with |V| = |Ṽ|, each object-level variable having exactly one mechanism parent specifying its relationship to its object-level parents. Edges divide into object-level E^obj, mechanism E^mech, and functional E^func (from Ṽ to the V it controls). Mechanism variables generalise Correa–Bareinboim regime indicators and Dawid’s parameter variables; the object-level subgraph must be acyclic, but cycles in the mechanism subgraph are permitted (following Bongers et al.’s cyclic SCM formalism). Connecting mechanism variables by causal links deliberately violates the independent causal mechanisms (ICM) assumption. Onto this the authors add an edge-labelling (Def. 3): a mechanism edge W̃ → Ṽ is terminal iff (1) Ṽ still responds to W̃ after the effects of W on its children are removed by structural mechanism interventions (Def. 2 — interventions on a mechanism variable making V conditionally independent of its object-level parents), and (2) Ṽ stops responding to W̃ once the effects of V on its children are cut. Terminal edges pick out the variables an agent cares about for their own sake, separating intrinsic from merely instrumental value.

Three algorithms and three theorems connect the causal and game-theoretic pictures. Algorithm 1 (Edge-labelled mechanised SCM discovery) takes a set of interventional distributions and applies leave-one-out causal discovery (Lemma 1: correct even for cyclic SCMs) to the combined object-level and mechanism variables, then infers edge labels via structural mechanism interventions on object-level children. Algorithm 2 (Agency Identification) turns the edge-labelled mechanised causal graph into a game graph: decision nodes are those whose mechanisms have incoming terminal edges, utility nodes are those whose mechanisms have outgoing terminal edges, everything else is a chance node; decisions and utilities in the same connected component of the terminal-edge graph get the same colour, i.e. belong to the same agent. Algorithm 3 (Mechanism Identification) inverts this, drawing mechanism edges via Koller–Milch s-reachability and terminal edges wherever a directed path runs from a decision to one of its agent’s utilities without passing through another of that agent’s utilities. Under five assumptions — each weakly connected component of the decision-utility subgraph is an agent subgraph with at least one decision and one utility (A1); agents play a subgame-perfect equilibrium under any set of mechanism interventions (A2); agents have a preferred ordering over equally-optimal decision rules, ruling out unmotivated switches (A3); only decision mechanisms have ingoing terminal edges, a weakened ICM assumption (A4); interventions on Ṽ can instantiate any deterministic function of V’s parents (A5) — Theorem 1 shows Algorithm 1 followed by Algorithm 2 recovers the true game graph; Theorem 2 shows Algorithm 2 is a left inverse of Algorithm 3; Theorem 3 shows Algorithm 3 is a left inverse of Algorithm 2 on the restricted class of mechanised causal graphs producible from a mechanised causal game and in which every node with an incoming mechanism edge also has an incoming terminal edge. The two representations are thus (modulo that restriction) equivalent descriptions of the same system — one physical, one decision-theoretic.

The payoff is diagnostic. Worked examples — a mouse gridworld; a content-recommender optimising a model of a human; an actor-critic RL setup; modified-action MDPs; a zero-agent causal Bayesian network — show the algorithms correcting prior modelling mistakes. Notably, the recommender example reveals a path H̃₂ → Ũ → D̃ invisible in the game graph, meaning the system is adapting to how the human’s opinion is updated, casting doubt on an earlier incentive analysis that had deemed the design safe; graphical incentive analysis can be trusted only when non-decision mechanisms lack ingoing arrows. Section 4.6 exhibits systems that break Assumption 1: two agents sharing a utility (an Assistance Game / CIRL setup, where Algorithm 2 wrongly merges them into one agent) and non-descendant utilities. The authors are explicit that variable types are relative to the choice of variables modelled — decision-hood and utility-hood are frame-dependent, and coarse-graining may change whether agents are discovered at all. Whether a thermostat or a trained RL policy counts as an agent depends on whether one includes its creation process: a frozen learnt policy will not adapt, so it is not an agent by this definition, but the RL training process is. Finally, the result constrains ICM: P(X | pa^X) does not obey the ICM principle if it is an agent’s decision rule, or is strategically relevant to one.

Key Ideas

Agents are systems that would adapt their policy if their actions influenced the world in a different way — the first formal causal definition of agency, and a test for goal-directed influence rather than a claim about actual behaviour.
Moved by reasons vs accidentally optimal: a rock perfectly sized to block a pipe is optimal for reducing water flow but would not adapt if the pipe were wider; a firm would adapt its price if consumers were differently price-sensitive (and it were made aware).
Mechanised SCM (Def. 1): endogenous variables split into object-level V and mechanism variables Ṽ (one mechanism parent each, parameterising V’s dependence on its object-level parents); edges are object-level, mechanism, or functional. Object-level subgraph acyclic; mechanism subgraph may cycle.
Structural mechanism intervention (Def. 2): an intervention on Ṽ making V conditionally independent of its object-level parents — a way to cut a variable’s dependence on its parents that an agent can be aware of.
Terminal mechanism edges (Def. 3): the edge-label that distinguishes what an agent values intrinsically from what it values instrumentally. W̃ → Ṽ is terminal iff Ṽ still responds when W’s downstream effects are cut, and stops responding when V’s downstream effects are cut.
Decisions = mechanisms with incoming terminal edges; utilities = mechanisms with outgoing terminal edges; everything else is chance. Agents are the connected components of the terminal-edge graph.
Three algorithms: Alg. 1 Edge-labelled mechanised SCM discovery (interventional distributions → edge-labelled mechanised causal graph, via leave-one-out causal discovery); Alg. 2 Agency Identification (mechanised causal graph → game graph); Alg. 3 Mechanism Identification (game graph → mechanised causal graph, via s-reachability).
Theorem 1: under Assumptions 1–5, Alg. 2 ∘ Alg. 1 applied to a mechanised causal game recovers its true game graph. Theorem 2: a₂(a₃(G)) = G. Theorem 3: a₃(a₂(C)) = C on a restricted class of C. Game graph and mechanised causal graph are two representations of one system.
Pre-policy vs post-policy interventions correspond exactly to mechanism vs object-level interventions: agents can adapt to the former, not the latter. Awareness of the intervention is what makes adaptation possible.
Diagnostic payoff: the recommender-system example uncovers a mechanism path (H̃₂ → Ũ → D̃) showing goal-directed influence on the human’s opinion updating that the game graph hides — graphical incentive analysis is reliable only when non-decision mechanisms lack ingoing arrows.
Agency depends on the frame and on the creation process: decision/utility/chance labels are relative to which variables are modelled; a frozen learnt RL policy is not an agent, but the RL training process is; a thermostat is an agent only with its designers.
ICM is constrained, not assumed: P(X | pa^X) violates the independent-causal-mechanisms principle when it is an agent’s decision rule, or is strategically relevant to one. Assumption 4 is a deliberately weakened ICM.
Stated failures: Assumption 1 breaks for agents sharing a utility (Assistance Games / CIRL — Alg. 2 merges them into one agent) and for non-descendant utilities (a decision that doesn’t cause its own utility). Approach improves on Orseau et al.’s compression-based definition, which mis-handles small action spaces and accidentally-optimal systems.

Connections

Conceptual Contribution

Claim: A system is an agent iff it would adapt its policy if its actions influenced the world in a different way. Agency is a counterfactual, causal property — detectable by experiment rather than ascribed by a modeller — and it is what distinguishes systems whose outputs are moved by reasons from systems that are merely accidentally optimal. Because the property is counterfactual, it also serves as a test for goal-directed (as against incidental) influence on a variable.
Mechanism: Introduce mechanised SCMs (Def. 1), splitting endogenous variables into object-level V and mechanism variables Ṽ that parameterise each V’s structural equation, with mechanism edges allowed (violating ICM) and cycles permitted in the mechanism subgraph. Define structural mechanism interventions (Def. 2) so that an agent can be made aware of a change to how its actions influence the world, and edge-label the mechanism edges as terminal or non-terminal (Def. 3) by whether the response survives cutting downstream effects. Then: Algorithm 1 recovers the edge-labelled mechanised causal graph from interventional distributions by leave-one-out causal discovery over object-level and mechanism variables (Lemma 1, Lemma 2); Algorithm 2 reads decisions off ingoing terminal edges, utilities off outgoing terminal edges, and agent identity off connected components of the terminal-edge graph; Algorithm 3 goes back from game graph to mechanised graph via Koller–Milch s-reachability (Lemma 3). Theorems 1–3 establish correctness under Assumptions 1–5 (agent subgraphs, subgame-perfect equilibrium, preferred ordering over equally-optimal decision rules, terminal edges only into decisions, and rich enough mechanism interventions) and mutual left-inverse-hood of Algorithms 2 and 3.
Concepts introduced/used: Mechanised SCM, Causal Game, Influence Diagram, Structural Causal Model, Causal Discovery, Intervention, Counterfactual, s-reachability, Independent Causal Mechanisms, Subgame Perfect Equilibrium, Goal-Directedness, Intentional Stance
Stance: formal theory / algorithmic definition, with correctness theorems and worked diagnostic examples (no empirical study)
Relates to: A rival formal answer to the question posed by Interpreting Systems as Solving POMDPs — where Biehl and Virgo ask whether a system’s dynamics admit a consistent interpretation as belief-updating and optimising, Kenton et al. ask whether the system’s policy counterfactually adapts under interventions on the mechanisms of its environment; the former makes agency a property of the pair (system, interpretation), the latter makes it a property discoverable from interventional data. The authors present their definition as an alternative to, or elaboration of, the Intentional Stance of True Believers - The Intentional Strategy and Why It Works “couched in the language of causality and counterfactuals” — retaining Dennett’s moved by reasons while replacing predictive usefulness with an experimental test. It sits on the causal machinery of Causality - Pearl and Causal Diagrams for Empirical Research - Pearl, extends the influence-diagram tradition of Koller and Milch, and is motivated by the incentive- and safety-analysis programme it partially corrects. Contrast with the compression-based definition of Orseau et al., which the authors argue mis-classifies accidentally-optimal systems.

Backlinks

Linked Pages

Causal Diagrams for Empirical Research - Pearl

Causal Diagrams for Empirical Research

Reference: Pearl, J. (1995). Causal Diagrams for Empirical Research. Biometrika 82(4): 669–688. December 1995. With discussion and rejoinder. URL (UCLA CogSys Lab) · DOI

Summary

Pearl’s Causal Diagrams is the paper in which the modern graphical apparatus of causal inference takes recognisable shape: directed acyclic graphs (DAGs) over observed and latent variables as the carrier of non-parametric causal assumptions, the do-operator as the formal representation of intervention, and the backdoor criterion as the first widely-usable graphical test for whether a causal effect is identifiable from observational data given a set of measured covariates. Where the early structural-equations and path-analysis tradition (Wright 1921, Haavelmo 1943) had given causal interpretations to specific parametric models, and where the do-then-randomise tradition (Rubin 1974) had given a counterfactual semantics to causal effects without graphs, Pearl unifies both: the DAG carries the structural assumptions, the do-operator gives the counterfactual semantics, and a small algebra of graphical operations decides identifiability.

The mechanism is precise. A causal model is a DAG G over variables V together with structural equations or, more generally, a non-parametric structural model. The do-operator do(X = x) represents the intervention that sets X to x regardless of X’s natural causes — graphically, by severing the incoming edges to X in G. A causal effect P(Y | do(X = x)) is identifiable from observational data over V iff it can be written as an expression in the observational distribution P(V). The backdoor criterion gives a sufficient condition: a set Z of measured covariates is admissible for adjustment iff Z blocks every backdoor path from X to Y (every path with an arrow into X) and contains no descendants of X. When Z is admissible, P(Y | do(X = x)) = ∑_z P(Y | X = x, Z = z) P(Z = z) — the backdoor adjustment formula. The paper also introduces the front-door criterion for cases where backdoor adjustment fails but a mediator can be exploited.

For the CBCL / CIVeX programme, Pearl 1995 is the foundation. CIVeX certificates carry a DAG of causal assumptions and a proof that the queried effect is identifiable from the available data under those assumptions; this is exactly what Pearl establishes is expressible and checkable. The paper is also a methodological complement to the PCC architecture in CBCL: where PCC ships behavioural proofs, CIVeX ships causal-identification proofs, and both are checked against a stated specification rather than re-derived. The completion of the graphical identifiability story — characterising exactly which causal effects are identifiable in the presence of unobserved confounders — is Shpitser & Pearl 2006’s complete-ID algorithm; Pearl 1995 is the place the story starts to look like an algorithm rather than a case-by-case analysis.

Key Ideas

Causal DAG as non-parametric structural model: a DAG G over V together with structural equations (or a non-parametric Markov-compatible distribution). The DAG carries the causal assumptions; the distribution is what is observed.
do-operator: do(X = x) represents intervention — setting X to x by external manipulation, severing the natural mechanisms that determine X. Graphically: delete incoming edges to X in G; semantically: replace X’s structural equation with X := x. The post-intervention distribution P(V | do(X = x)) is the central object of causal inference.
Identifiability: a causal query P(Y | do(X = x)) is identifiable from observational data over V iff it can be written as an expression in P(V). Identifiability is a graphical property — it depends only on the DAG, not on the specific distribution.
Backdoor criterion: Z ⊆ V is admissible for adjustment iff (i) no node in Z is a descendant of X in G, and (ii) Z blocks every backdoor path from X to Y (every path with an arrow into X). When Z is admissible, P(Y | do(X = x)) = ∑_z P(Y | X = x, Z = z) P(Z = z).
Front-door criterion: when no admissible Z for backdoor adjustment exists, identification may still be possible via a mediator M on the directed path from X to Y. The front-door formula uses M to identify the effect even in the presence of unobserved confounders.
Markov compatibility / d-separation: G constrains P(V) via conditional-independence statements encoded by d-separation — a purely graphical criterion. The empirical content of a causal DAG is exactly the d-separations it implies.
Bridge to interventions: causal effects can be defined purely structurally (no counterfactual ontology needed) by reference to the post-intervention distribution. Counterfactuals re-enter when finer-grained queries (mediation, attribution) are asked, handled by structural causal models in Pearl’s later work.

Connections

Conceptual Contribution

Claim: Causal effects can be represented graphically (as DAGs over observed and latent variables) and identified from observational data by a calculus of graphical operations. The do-operator gives the formal semantics of intervention; the backdoor and front-door criteria give algorithmic identifiability conditions.
Mechanism: Encode causal assumptions as a DAG G over V. Define interventions via the do-operator, semantically the surgical replacement of X’s structural equation by X := x and graphically the severing of incoming edges. State identifiability as the existence of an expression in P(V) for P(Y | do(X = x)). Give the backdoor criterion as a sufficient graphical condition (block all backdoor paths, no descendants of X) together with the backdoor adjustment formula. Give the front-door criterion for the harder case of unmeasured confounding. The result is a non-parametric, graph-driven, algorithmic theory of causal identification.
Concepts introduced/used: do-Calculus, Causal DAG, Structural Causal Model, Intervention, Backdoor Criterion, Front-door Criterion, d-Separation, Identifiability, Markov Compatibility.
Stance: foundational technical paper.
Relates to: Pearl’s compact statement of the apparatus developed at book length in Causality (2000/2009); the algorithmic completion is Shpitser & Pearl 2006. The complementary tradition is Rubin’s potential-outcomes framework (1974) — the same causal effects in counterfactual notation; later work (Pearl–Bareinboim, Imbens–Rubin) establishes formal mappings between the two. For CBCL and CIVeX, this is the substrate: a CIVeX certificate carries a DAG and a proof of identifiability under stated assumptions, checked by the host rather than re-derived. The pairing with PCC is exact: PCC ships behavioural proofs over machine states; CIVeX ships causal-identification proofs over the DAG. The d-separation / Markov-compatibility connection grounds the bridge to probabilistic graphical models more broadly.

Causality - Pearl

Causality: Models, Reasoning, and Inference

Reference: Pearl, J. (2000, 2nd ed. 2009). Causality: Models, Reasoning, and Inference. Cambridge University Press. 384 pp (1st ed.) / 484 pp (2nd ed.). URL (ILLC Amsterdam, 2nd ed. PDF) · Cambridge UP page · Internet Archive

Summary

Causality is the book-length statement of Pearl’s graphical-causal-inference programme — the work that fixes a usable vocabulary for causation in the empirical sciences, replacing decades of philosophical regress with a constructive theory. The book extends the apparatus of Causal Diagrams for Empirical Research (1995) along three axes. (1) From interventional to counterfactual causal queries: the structural causal model (SCM) — a system of structural equations with exogenous error terms — supports queries about what would have happened if particular variables had taken different values, not only about intervention distributions. (2) From backdoor / front-door criteria to do-calculus: a complete deductive system (three rules: insertion/deletion of observations, action/observation exchange, and insertion/deletion of actions) for transforming do-expressions. Together with Shpitser & Pearl 2006’s completeness result, do-calculus is complete for non-parametric identifiability. (3) From identifiability to applied causal inference: chapters on confounding, mediation, instrumental variables, the actual-cause problem, and the foundations of legal-causal reasoning.

The technical core in three layers. Layer 1 (association): ordinary probabilistic conditional P(Y | X). Layer 2 (intervention): post-do distribution P(Y | do(X = x)), derived from the SCM by surgical replacement of X’s equation. Layer 3 (counterfactuals): queries like P(Y_{X=x} | X = x', Y = y') — “given that we observed X = x' and Y = y', what would Y have been if we had set X = x?” — derived by the abduction–action–prediction procedure: update exogenous-error distributions on the evidence, intervene on X, predict Y. Pearl calls this stratification the Ladder of Causation; the book argues that the climb from layer 1 to layer 3 is a matter of added assumptions (structural equations, then exogenous-error distributions), not of finer-grained statistical methodology.

For CBCL and CIVeX, the book is the textbook reference. CIVeX certificates carry a DAG, a stated set of structural assumptions, and a do-calculus derivation establishing identifiability of the target causal query. The host’s verifier checks the derivation against the do-calculus rules. The completeness of do-calculus (Shpitser & Pearl 2006) means that the verifier never has to apologise for being too weak: if the effect is identifiable at all, do-calculus derives it. The book also supplies the vocabulary for stating what CIVeX certificates are claiming: distinctions between association, intervention, and counterfactual; between identifiability and estimability; between confounding, mediation, and selection — all of which are needed to write precise threat models for causal claims in multi-agent settings.

Key Ideas

Structural Causal Model (SCM): a system of structural equations Vᵢ := fᵢ(PAᵢ, Uᵢ) over endogenous variables V with exogenous errors U, plus a distribution over U. The SCM is the formal carrier of all three causal layers.
Ladder of Causation: three layers of causal queries, each requiring strictly more assumptions. Layer 1 (seeing) — P(Y | X). Layer 2 (doing) — P(Y | do(X = x)). Layer 3 (imagining) — counterfactuals P(Y_{X=x} | E) for arbitrary evidence E.
do-calculus: three syntactic transformation rules over do-expressions: (R1) insertion/deletion of observations under d-separation; (R2) action/observation exchange under modified d-separation; (R3) insertion/deletion of actions under absence-of-causal-paths. Combined with probability axioms, the calculus is complete for non-parametric identifiability (Shpitser–Pearl 2006).
Identifiability under unobserved confounders: many effects are identifiable even when key confounders are unmeasured. The do-calculus discovers when, and the Shpitser–Pearl ID algorithm gives a complete procedure.
Counterfactual queries by abduction–action–prediction: (i) Update P(U) on observed evidence (abduction). (ii) Intervene on the SCM (action). (iii) Compute the distribution of the target variable (prediction). The procedure yields counterfactuals from the SCM.
Mediation analysis: natural direct, natural indirect, and total effects decompose causal influence along specific paths. Identification conditions are characterised by the mediation formula and the more general path-specific effects.
Instrumental variables: a classical econometric tool given a clean graphical characterisation — an instrument is a variable affecting X but not Y directly or via unmeasured confounders. The IV estimand is recoverable when the graphical conditions hold.
Actual cause / responsibility: chapter 10 develops a formal theory of actual causation — what actually caused the outcome in this token case? — built on counterfactuals and contingency. Foundational for legal and AI-ethics applications.
Transportability: under what conditions can a causal effect estimated in one population be transferred to another? Bareinboim–Pearl give graphical criteria.

Connections

Conceptual Contribution

Claim: Causal reasoning admits a precise, computable, non-parametric foundation built on structural causal models (SCMs), the do-operator for intervention, and the do-calculus for transforming causal queries. The three-layer Ladder of Causation (association / intervention / counterfactual) stratifies the assumptions required for each kind of causal claim.
Mechanism: Define SCMs as systems of structural equations with exogenous errors. Define interventions surgically (replace X’s equation by X := x). Define counterfactuals via abduction–action–prediction on the SCM. Give the do-calculus (three transformation rules) and show it is sound; later work (Shpitser–Pearl 2006) shows it is also complete for identification. Extend to mediation, instruments, actual causation, transportability across populations.
Concepts introduced/used: Structural Causal Model, Ladder of Causation, do-Calculus, Counterfactual, Mediation Analysis, Instrumental Variable, Actual Cause, Transportability, Identifiability.
Stance: foundational technical monograph / textbook.
Relates to: The book that the paper Causal Diagrams (1995) prefigures and that Shpitser & Pearl 2006 completes algorithmically. The complementary framework is Rubin’s potential outcomes (1974): the same causal effects in counterfactual notation, with explicit formal mappings between the two languages (Pearl–Bareinboim 2009). For CBCL / CIVeX, the book is the textbook reference: CIVeX certificates carry DAGs, structural assumptions, and do-calculus derivations of identifiability; the host’s verifier checks the derivation against the calculus. The PCC pairing is exact (Necula): PCC ships behavioural proofs over machine states; CIVeX ships causal-identification proofs over the DAG; both rely on small, generic verifiers that check rather than re-derive.

True Believers - The Intentional Strategy and Why It Works

True Believers: The Intentional Strategy and Why It Works

Reference: Dennett, D. C. (1981). In A. F. Heath (Ed.), Scientific Explanations, Oxford University Press. Reprinted in The Intentional Stance (1987, MIT Press, ch. 2) and in Mind Design II (1997). Source file: dennett_true_believers.pdf. URL

Summary

Dennett’s canonical defence of the intentional stance: a predictive strategy in which one treats a system as a rational agent with beliefs and desires, works out what a rational agent so situated ought to do, and predicts it will do that. The stance is contrasted with the physical stance (predict from physical laws and microstructure) and the design stance (predict from presumed function). Belief is a perfectly objective phenomenon, but one discernible only from the point of view of an adopter of this strategy; a system is a true believer just in case its behaviour is reliably predictable by intentional ascription.

Dennett argues this vindicates a “mild realism” (or “interpretationism”) about belief against both hard-core realism (beliefs as sentence-like tokens in the head) and eliminativism. The remarkable empirical fact is that the intentional strategy works — on people, animals, chess programs, even thermostats (in a degenerate way) — because evolution and engineering produce systems whose behaviour tracks their interests given accurate beliefs. This pattern of success is what there is to being a believer; no further fact about inner sentential structure is required. The paper grounds the philosophical licence to ascribe mental states to artificial agents and thus underwrites the mentalistic vocabulary of agent theory.

Key Ideas

Three stances: physical, design, intentional.
Intentional strategy: assume rationality, compute ought-to-do, predict.
True believer = system reliably predicted by the intentional strategy.
Belief is objective but stance-relative (mild realism / interpretationism).
Natural selection explains why the strategy works on organisms.
Thermostats as degenerate (but continuous) intentional systems.
Licence for mentalistic ascription to artificial agents.

Connections

Conceptual Contribution

Claim: Beliefs and desires are objective but stance-relative — a system genuinely has them just in case adopting the intentional strategy (assume rationality with appropriate beliefs/desires and predict accordingly) reliably predicts its behaviour.
Mechanism: Dennett distinguishes physical, design, and intentional stances; defines the intentional strategy operationally (attribute beliefs the system ought to have given its perceptual and epistemic history, attribute desires it ought to have given its biological/functional role, predict the rational action); argues success of the strategy on people, animals, and artefacts is an empirical fact explained by evolutionary/design pressures, and that this success is all there is to being a believer — no further requirement of language-of-thought tokens.
Concepts introduced/used: Intentional Stance, True Believer, Mild Realism, Design Stance, Physical Stance, Rationality Assumption
Stance: philosophical theory
Relates to: Justifies the ascription of BDI mental states to artificial systems in Modeling Rational Agents within a BDI-Architecture, Intelligent Agents Theory and Practice and Agent-Oriented Programming; cohabits with Two Faces of Intention as a pillar of agent-theoretic philosophy; foreshadows functional-role arguments in Society of Mind.

In this vault

Interpreting Systems as Solving POMDPs

Interpreting systems as solving POMDPs: a step towards a formal understanding of agency

Reference: Biehl, M. & Virgo, N. (2022). Interpreting systems as solving POMDPs: a step towards a formal understanding of agency. Proceedings of the 3rd International Workshop on Active Inference (IWAI 2022); published in Active Inference. IWAI 2022, Communications in Computer and Information Science, vol. 1721, Springer, Cham. arXiv:2209.01619 (Cross Labs / Cross Compass, Tokyo; Earth-Life Science Institute, Tokyo Institute of Technology). URL.

Summary

When is a physical system entitled to be described as having beliefs and goals? Biehl and Virgo answer with a consistency criterion rather than a predictive one. Their starting point is Dennett’s Intentional Stance, which licenses agent-talk when it out-predicts the Physical Stance. The authors explicitly depart from this comparison-of-stances approach: assuming we already know what the system does at the physical level, they ask whether we can consistently ascribe meaning to its physical states such that those states appear to implement belief updating and decision making. This builds directly on Virgo, Biehl & McGregor’s earlier notion of interpreting a dynamical system as a Bayesian reasoner — something that takes inputs and makes inferences about a hidden variable, but that has neither goals nor actions. The present paper’s contribution is to extend consistent interpretation to cover goals and actions, and thereby to reach agents.

Three deliberate conceptual commitments make this precise. The system is taken to be a stochastic Moore machine (M, I, O, μ, ω) — internal states, inputs, outputs, a stochastic machine kernel, and a deterministic expose function; the authors note this is far from the most general system but is general enough to cover the digital computers running most present-day artificial agents. Beliefs are probability distributions updated in accordance with Bayes’ law. Goals and rationality come from POMDPs, which supply both a formal notion of goal (expected cumulative discounted reward maximisation) and of rational behaviour (the optimal policy). The technical machinery is a consistent Bayesian influenced filtering interpretation (Def. 3): a hidden state space H, an action space A, an interpretation map ψ : M → PH sending machine states to beliefs over hidden states, an action function α : M → A, and a model kernel κ : H × A → P(H × I). A stochastic Moore machine is then interpretable as a solution to a POMDP (Def. 5) if its machine part admits such an interpretation and its expose function emits the optimal action for the belief its state parameterises: ω(m) = π*(ψ(m)). An agent, on this proposal, is the pair of a system and such an interpretation.

Two features do the real work. First, machine states need only parameterise beliefs rather than be belief states — ψ need not be injective, so interpretations come with a built-in notion of coarse-graining, and the definition covers more than the canonical belief-state machines. Proposition 1 shows this is not a loosening in disguise: for any input that is subjectively possible under the interpretation, the parameterised distributions ψ(m) update exactly as the Bayesian belief-state update requires, so V̄*(m) := V*(ψ(m)) is an optimal value function on the machine’s own state space with optimal policy ω(m) = π*(ψ(m)). Such machines are optimal in the same sense as machines whose states are correctly-updated beliefs. Second, consistency is not correctness: whether a consistent interpretation exists depends only on the machine being interpreted, not on any property of its environment. An agent may hold a wrong model — its conclusions will then be wrong while its reasoning stays consistent — and its actions need only be optimal according to its own model, possibly suboptimal in the true environment. The authors flag this as a clear point of difference from the original Free Energy Principle literature, where the model is derived from the actual environment’s dynamics and so cannot differ from it. The hidden states and the rewards need not have any physical realisation at all; they exist only within the interpretation, and the machine never observes the rewards. Appendix C establishes that machines meeting Def. 5 exist by constructing one from the known optimal solution to Sondik’s POMDP.

The authors are careful about scope. The commitments are adopted because they permit precision, not because they are claimed to be realistic or the only reasonable choices; POMDPs are not the only possible formalisation of having a goal; they do not argue that all agents are rational or possess beliefs and goals. The framework requires exact Bayesian updating, so approximate inference — which the FEP does accommodate — falls outside it. The conclusion is offered as one worked example of the kind of agent definition sought, not as a settled general answer.

Key Ideas

Agency as consistent interpretation, not predictive advantage: ask whether the physical dynamics are consistent with an ascription of beliefs and goals, rather than whether the intentional stance out-predicts the physical stance.
An agent is a pair: a system together with an interpretation of it as a POMDP solution — agency is not an intrinsic property of the system alone.
Three conceptual commitments: systems are stochastic Moore machines; beliefs are Bayes-updated probability distributions; goals and rationality are supplied by POMDPs (expected discounted reward, optimal policy).
Interpretation map ψ : M → PH: sends internal machine states to beliefs over an interpretation-internal hidden state space; extended here with an action function α and a model kernel κ (the “influenced filtering” of Def. 3).
Definition 5: a stochastic Moore machine solves a POMDP when it has a consistent Bayesian influenced filtering interpretation and ω(m) = π*(ψ(m)) — it outputs the optimal policy’s action for the belief its state parameterises.
Parameterisation, not identity: machine states need only parameterise beliefs; ψ may be non-injective, so interpretation carries an automatic notion of coarse-graining of the internal state.
Proposition 1: for subjectively possible inputs, ψ(m) updates exactly per the Bayesian belief-state update; hence V*(ψ(m)) is an optimal value function on the machine’s state space — these machines are optimal in the same sense as belief-state machines.
Consistency ≠ correctness: existence of an interpretation depends only on the machine, not the environment. An agent can have a wrong model and act optimally only relative to it — a deliberate contrast with the FEP, where the model derives from actual environment dynamics.
Hidden states and rewards are interpretation-only: they need exist only mathematically, not physically; the machine never observes rewards. Even an output-less machine can have “actions” confined to the interpretation.
Existence is demonstrated, not assumed: Appendix C builds a qualifying stochastic Moore machine from the known optimal solution to Sondik’s POMDP.
Stated limits: exact Bayesian updating only (approximate inference excluded); POMDPs are not the only formalisation of goals; the commitments are chosen for precision rather than realism.

Connections

Conceptual Contribution

Goal-Directedness

The capacity of a system to pursue distal targets despite perturbations — treated in Levin’s framework as a spectrum correlated with the cognitive “light cone” of a self, and a defining feature of agency at any scale.

In this vault

Subgame Perfect Equilibrium

(page does not exist)

Independent Causal Mechanisms

(page does not exist)

s-reachability

(page does not exist)

Counterfactual

A statement about what would have happened had things been otherwise; central to causal reasoning and to McCarthy’s programme of representing actions formally.

In this vault

Some Philosophical Problems from the Standpoint of Artificial Intelligence

Intervention

Rung-2 operation in Pearl’s Ladder of Causation (the do(X=x) operator): force X to take value x and break its incoming edges in the Causal DAG. Distinct from observation; required for causal as opposed to associational claims.

In this vault

Causal Discovery

(page does not exist)

Structural Causal Model

Pearl’s framework for causal reasoning: a set of structural equations + a Causal DAG over endogenous and exogenous variables, supporting observational, interventional, and counterfactual queries via the three-rung ladder of causation.

In this vault

Influence Diagram

(page does not exist)

Causal Game

(page does not exist)

Mechanised SCM

(page does not exist)

A Value for n-Person Games

Reference: Shapley, L. S. (1953). A Value for n-Person Games. In H. W. Kuhn & A. W. Tucker (eds.), Contributions to the Theory of Games, Volume II, Annals of Mathematics Studies 28, pp. 307–317. Princeton University Press. DOI · Open access PDF (RAND P-295)

Summary

Shapley asks a deceptively simple question: given a cooperative game in characteristic-function form — a set of players N and a function v assigning each coalition S ⊆ N the total payoff v(S) it can guarantee itself — how much is a seat in the game worth to each player before it is played? He seeks a value: a single number φ_i(v) for each player i representing a fair, a-priori assessment of what that player can expect. Rather than derive the value from a bargaining process, Shapley pins it down axiomatically, imposing three conditions any reasonable notion of fairness should satisfy and then proving that they leave exactly one possibility.

The axioms are: (1) Efficiency / carrier — the values distribute the whole worth of the game, Σ_i φ_i(v) = v(N), and dummy players outside every carrier receive nothing; (2) Symmetry — the value is invariant under relabelling players, so two players who make identical marginal contributions to every coalition receive equal value (equals are treated equally); (3) Additivity — for two independent games, φ(v + w) = φ(v) + φ(w). Shapley’s theorem is that a unique function satisfies all three, and it has the closed form

φ_i(v) = Σ_{S ⊆ N\{i}} [ |S|! (n − |S| − 1)! / n! ] · ( v(S ∪ {i}) − v(S) ).

The formula has a vivid interpretation: imagine the players joining the grand coalition one at a time in a random order, every one of the n! orders equally likely; each player’s value is their expected marginal contribution — the extra worth they bring to the coalition of those who arrived before them, averaged over all arrival orders. This makes fairness operational: each player is credited exactly for what, on average, they add. The Shapley value became the canonical solution concept for fair division, cost allocation (the Shapley–Shubik power index, airport cost-sharing), and — decades later — machine-learning attribution (SHAP feature importances, Data Shapley for data valuation) and fair credit assignment among cooperating agents.

Key Ideas

Characteristic-function game: a game reduced to (N, v) with v(∅) = 0, where v(S) is the worth a coalition S can secure; individual strategies are abstracted away.
Value as an a-priori assessment: φ_i(v) measures a player’s expected worth before play, not the outcome of a specific bargain — a normative allocation rule, not a positive prediction.
Three fairness axioms uniquely determine the value: efficiency (full distribution + dummies get zero), symmetry (equal treatment of interchangeable players), and additivity (linearity across combined games).
Marginal-contribution formula: the value equals the average of a player’s marginal contribution v(S ∪ {i}) − v(S) taken over all n! orderings of the players.
Null / dummy player: a player who adds nothing to any coalition receives zero — a corollary of the axioms, encoding “no reward without contribution”.
Symmetry ⇒ equal pay for equal work: players who are substitutes in every coalition get identical value; this is the axiom that makes the value a fairness concept.
Foundational reach: underpins cost-sharing, voting power (Shapley–Shubik), the Shapley–Folkman results’ economic cousins, and modern SHAP / Data-Shapley attribution.

Connections

Conceptual Contribution

Potential Outcomes Framework - Rubin

Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies

Reference: Rubin, D. B. (1974). Estimating Causal Effects of Treatments in Randomized and Nonrandomized Studies. Journal of Educational Psychology 66(5): 688–701. October 1974. APA PsycNet record (DOI 10.1037/h0037350) · ERIC index entry · See also Rubin’s later book with Imbens, Causal Inference for Statistics, Social, and Biomedical Sciences (Cambridge UP 2015) for the canonical textbook statement.

Summary

Rubin’s 1974 paper is the founding statement of the potential-outcomes framework (a.k.a. the Rubin causal model) — the alternative to Pearl’s graphical apparatus that has dominated statistics, econometrics, and biomedical inference. The central move: rather than thinking of causation in terms of mechanisms or structural equations, define a causal effect counterfactually. For each unit i and each treatment level t, posit a potential outcome Yᵢ(t) — the outcome unit i would have exhibited under treatment t. The unit-level causal effect of treatment t versus control c is Yᵢ(t) − Yᵢ(c). The fundamental problem of causal inference is that only one of Yᵢ(t) and Yᵢ(c) is ever observed for any single unit — the other is counterfactual. Causal inference is then statistical inference about the unobserved counterfactual.

The framework’s payoff comes from clarifying the role of randomization. Under random assignment to treatment, observed and counterfactual outcomes are exchangeable: the distribution of Yᵢ(t) in the treated group equals the distribution that would have obtained in the control group had they been treated. The average treatment effect (ATE) E[Y(t) − Y(c)] is then estimable as the simple difference in group means. The framework also clarifies what observational data lets us do: under stated ignorability (a.k.a. conditional exchangeability, no unmeasured confounders) and positivity assumptions, the ATE is identifiable by covariate adjustment — the same calculation Pearl’s backdoor adjustment formula yields, derived from the counterfactual side. Subsequent work (Rosenbaum–Rubin propensity scores 1983, matching, Imbens–Angrist instrumental-variables 1994) builds out the estimation toolbox.

The framework is now the dominant language of causal inference in applied statistics. For CBCL / CIVeX, knowing both languages matters: a CIVeX certificate may be written in either graphical or potential-outcomes notation, and the verifier may need to translate between them. The formal equivalence is established by Pearl–Bareinboim 2009 and Pearl 2010: any non-parametric causal effect expressible in one framework is expressible in the other under translation; the languages differ in ergonomics (graphs make assumptions visually inspectable; potential outcomes make assumptions algebraically inspectable) rather than expressive power. The paper is the place to cite to acknowledge the other half of the causal toolkit.

Key Ideas

Potential outcomes: for each unit i and treatment level t, posit Yᵢ(t) — the outcome unit i would exhibit under t. Both Yᵢ(treated) and Yᵢ(control) are well-defined, but only one is observed for any given unit.
Unit-level causal effect: Yᵢ(t) − Yᵢ(c). Defined for each unit, even though typically unobservable.
Fundamental problem of causal inference: only one potential outcome per unit is observable. Causal inference is statistical inference about the unobserved counterfactual.
SUTVA (Stable Unit-Treatment Value Assumption): implicit in defining Yᵢ(t) as a function of unit and treatment only — (i) no interference between units, and (ii) no multiple versions of treatment. SUTVA violations (peer effects, network spillovers, treatment heterogeneity) require explicit extension.
Ignorability / Conditional Exchangeability: (Y(t), Y(c)) ⫫ T | X — given covariates X, treatment assignment is independent of potential outcomes. The non-graphical statement of “no unmeasured confounders”.
Positivity / Common Support: 0 < P(T = t | X = x) < 1 for all x in the support. Required for ATE identification — must have some chance of either treatment at every covariate level.
Randomization: random assignment of T independent of (Y(t), Y(c)) makes ignorability hold without conditioning. The methodological justification for randomized experiments.
Average Treatment Effect (ATE): E[Y(t) − Y(c)]. The standard target estimand. Identifiable under ignorability + positivity by standardization (covariate adjustment), which yields the same formula as Pearl’s backdoor adjustment.
Successors: propensity score (Rosenbaum–Rubin 1983), matching, instrumental variables (Imbens–Angrist 1994), the Local Average Treatment Effect (LATE), doubly-robust estimation, the entire modern econometric/biostatistical causal-inference toolbox.

Connections

Conceptual Contribution

Claim: Causal effects can be defined and estimated by positing potential outcomes Yᵢ(t) for each unit-treatment pair and treating causal inference as statistical inference about unobserved counterfactuals. Random assignment guarantees identifiability of average effects without further assumptions; observational data require ignorability (no unmeasured confounders) and positivity (common support).
Mechanism: For each unit i and treatment level t, define Yᵢ(t) as the outcome i would exhibit under t. The fundamental problem: only one Yᵢ(t) is observed per unit. Define the unit-level effect Yᵢ(t) − Yᵢ(c) and population estimands (ATE, ATT). Identify the ATE under random assignment without further assumption; under observational data with measured covariates X, identify under ignorability (Y(t), Y(c)) ⫫ T | X and positivity. Estimate by direct standardization, propensity-score adjustment, matching, or instrumental variables when assumptions fail.
Concepts introduced/used: Potential Outcomes, Rubin Causal Model, Counterfactual, SUTVA, Ignorability, Positivity, Average Treatment Effect, Propensity Score (later), Instrumental Variable (later), Fundamental Problem of Causal Inference.
Stance: foundational technical paper.
Relates to: The complementary framework to Pearl 1995 / Causality 2000 — the same causal effects in different notation. Pearl–Bareinboim 2009 and Pearl 2010 establish formal mappings between the two; ignorability + positivity in the Rubin language correspond to the existence of an admissible covariate set in the Pearl backdoor sense. The languages differ in ergonomics: graphs make assumptions visually inspectable; potential outcomes make assumptions algebraically inspectable and integrate naturally with regression-based estimation. The 1974 paper is the seed; the canonical textbook statement is Imbens–Rubin 2015. For CBCL / CIVeX, the framework provides the other half of the causal toolkit: a CIVeX certificate may state its identification claim in either potential-outcomes or graphical language, and the verifier may need to translate. Knowing both is part of the technical literacy required for serious causal-claim verification. Shpitser–Pearl 2006’s completeness on the graphical side has no equally clean counterpart on the potential-outcomes side — the practical computational ground favours graphs.

Intelligence Without Representation

Reference

Brooks, Rodney A. (1991). “Intelligence without representation.” Artificial Intelligence 47: 139–159. (Received September 1987.)
PDF: people.csail.mit.edu/brooks/papers/representation.pdf
Local: brooks1991_representation.pdf

Summary

Brooks challenges the dominant symbol-manipulation paradigm of classical AI, arguing that “representation is the wrong unit of abstraction in building the bulkiest parts of intelligent systems.” Drawing on an evolutionary argument — 3.5 billion years were spent evolving sensorimotor and survival competence, and only a few thousand years on “expert” symbolic knowledge — he claims the hard part of intelligence is mobility, vision, and acting in dynamic environments, not the formal problem-solving that dominated early AI. The “blocks-world” style of AI succeeds only because researchers have already factored out perception and motor action, relegating them to input/output black boxes.

In place of centralized symbolic architectures, Brooks proposes building autonomous mobile robots he calls “Creatures,” decomposed by activity rather than by function. Each activity-producing layer connects sensing to action directly and runs in parallel; higher layers are incrementally added on top of lower, already-working layers without a central symbolic world model. This is the subsumption architecture. The slogan is “the world is its own best model”: rather than maintaining an internal representation the robot re-perceives what it needs.

The paper’s philosophical payload is a rejection of the sense–model–plan–act pipeline and of the assumption that intelligence can be decomposed into modules trading symbolic messages. Brooks argues this methodological commitment — that perception delivers a human-style Merkwelt that central reasoning can operate on — is a form of self-deception, since each species and each robot inhabits its own Merkwelt. Intelligence is to be built bottom-up, situated and embodied, tested continuously against the real world.

Key Ideas

Situatedness and embodiment as design primitives: Creatures must cope with a dynamic environment and have some purpose in being.
Decomposition by activity, not function: layers of task-achieving behavior (avoid, wander, explore) replace perception → modeling → planning → action pipelines.
Subsumption architecture: higher competence layers inhibit/subsume lower ones while lower continue running.
No central representation: “the world is its own best model”; no shared symbolic world model between layers.
Evolutionary time argument: sensorimotor intelligence is the hard, ancient core; symbolic expertise is a thin late veneer.
Critique of abstraction: abstracting away perception and action is how AI self-deceives about having solved problems.
Hypothesis (H): representation is the wrong primitive for building intelligent systems.

Connections

Subsumption Architecture — this paper is the manifesto.
Intelligent Agents Theory and Practice — Wooldridge & Jennings use Brooks as the exemplar of reactive agents, contrasted with deliberative BDI agents.
The Society of Mind — shares the anti-monolithic, many-small-agents intuition, though Minsky retains representations.
Multi-Agent Systems — reactive/behavior-based approaches feed directly into swarm and MAS work.
Actor Model — independent parallel activity producers without a central controller.
A Universal Modular Actor Formalism for Artificial Intelligence — kindred rejection of centralized control.
Agent Architecture — subsumption is a canonical architecture contrasted with BDI, layered, hybrid.
Contrast with Knowledge Representation, Ontologies, Advice Taker — papers that double down on representation.

Conceptual Contribution

Principle of Rationality

Newell’s bridge law for the knowledge level: an agent will take an action whose preconditions it knows to hold and whose effects it wants. Gives knowledge-level descriptions predictive force.

In this vault

Rationality Assumption

Dennett: to take the intentional stance is to assume the system will act rationally given its beliefs and desires. Without this, predictions don’t get off the ground.

In this vault

The Knowledge Level

Reference: Allen Newell (1982). “The Knowledge Level.” Artificial Intelligence 18(1): 87-127. (AAAI Presidential Address, Stanford, 19 Aug 1980; also in AI Magazine 2(2), 1981.) URL

Summary

Newell’s AAAI-80 presidential address proposes that above the symbol level (data structures manipulated by programs) there exists a distinct knowledge level whose medium is knowledge and whose laws are rationality. At the knowledge level an agent is characterised by what it knows and what goals it has; its behaviour is predicted by the principle of rationality — “if an agent knows that one of its actions will lead to a goal, it will select that action” — without commitment to any particular representation or processing mechanism.

The paper motivates the proposal by diagnosing three tensions in early-1980s AI: the mystification of “representation,” the residue of the 1960s-70s theorem-proving controversy (Planner/Microplanner/Conniver’s “uniform procedures will not work” reaction against resolution), and the diversity of answers in the SIGART Special Issue on Knowledge Representation (Brachman and Smith’s survey showing no consensus on quantification, quotation, self-description, or evaluation). Newell’s resolution: treat knowledge as a competence-level notion analogous to physicists’ treatment of temperature — a level with its own laws, reduced to but not replaced by the symbol level. Representations are then understood as symbol-level implementations of knowledge, and we gain a principled way to talk about what an AI system knows without confusing it with how that knowledge is stored.

The Knowledge Level is foundational for agent-oriented programming (Shoham’s “mental state” framing), for the BDI tradition, for KQML’s and FIPA’s claim that ACL semantics should be defined at the level of beliefs/intentions rather than byte-strings, and indirectly for McCarthy’s Elephant 2000, whose speech-act semantics presupposes that one can talk coherently about what a program knows and is committed to, independently of implementation.

Key Ideas

Computer systems are describable at multiple levels (device, circuit, logic, register-transfer, program/symbol); the knowledge level sits above the symbol level.
Medium of the knowledge level: knowledge (distinct from the data structures that represent it).
Law of behaviour at the knowledge level: the principle of rationality — actions selected to achieve goals given what is known.
The knowledge level is a competence-level account: it predicts behaviour without specifying mechanism, analogously to economic rationality.
Representation is the symbol-level realisation of knowledge; mystifying representation conflates the levels.
Diagnostic of the theorem-proving backlash: “uniform procedures will not work” was a symbol-level reaction to a knowledge-level question.

Connections

Agent-Oriented Programming — Shoham explicitly situates AOP at the knowledge level; mental states are Newell-level predicates over agents.
Elephant 2000 - A Programming Language Based on Speech Acts — Elephant commits to knowledge-level program variables (propositions, commitments, past speech acts), making knowledge-level description the native semantics of programs.
Ascribing Mental Qualities to Machines — McCarthy’s ascriptivism is the philosophical cousin of Newell’s levels argument.
Knowledge Representation
Agent Communication Languages — ACL semantics in the KQML/FIPA tradition are knowledge-level (beliefs/intentions) rather than symbol-level (bit patterns).
KIF — intended as a lingua franca at the symbol level that preserves knowledge-level content across heterogeneous systems.
Common Sense Reasoning

Conceptual Contribution

Claim: AI systems have a level of description above the symbol level — the knowledge level — whose medium is knowledge and whose law is rationality; this level is reducible to but not replaceable by the symbol level, and recognising it clarifies decades of representation debates.
Mechanism: Levels-of-description methodology imported from computer architecture; principle of rationality as the behaviour law; knowledge characterised competence-theoretically (what follows from what an agent knows and wants, not how); representations reconceived as symbol-level implementations of knowledge-level content; diagnostic application to the theorem-proving controversy and the SIGART Special Issue.
Concepts introduced/used: Knowledge Level, Symbol Level, Principle of Rationality, Competence vs Performance, Knowledge Representation, Levels of Description.
Stance: foundational
Relates to: The conceptual ground for agent-oriented and BDI programming; provides the “what an agent knows” vocabulary that KQML, Agent Communication Languages, and Elephant 2000 - A Programming Language Based on Speech Acts rely on; complements McCarthy’s ascriptivist argument in Ascribing Mental Qualities to Machines.

Ascribing Mental Qualities to Machines

Reference: John McCarthy (1979). Stanford CS Department; reprinted in Formalizing Common Sense: Papers by John McCarthy (V. Lifschitz ed., Ablex, 1990). Source file: mccarthy-ascribing.pdf. URL

Summary

McCarthy argues that ascribing beliefs, desires, intentions, knowledge, and other mental qualities to machines (including simple ones like thermostats) is legitimate when the ascription conveys the same information about the machine that it would about a person, and useful when it helps us predict, repair, or improve the machine’s behaviour. He proposes two new definitional tools — definitions relative to an approximate theory, and second-order structural definitions — so that mental-state vocabulary can be applied to physical systems conservatively rather than by reduction to physics.

This is the philosophical foundation for the entire mentalistic (BDI) tradition in agent research that ACL Rethinking Principles pushes back against. Singh cites McCarthy as the origin of the view that agent communication can be specified by ascribing beliefs and intentions; McCarthy himself is already careful to warn that such ascriptions must be conservative and may be merely convenient shorthand for structural facts.

Key Ideas

Ascribing mental qualities is warranted when it expresses information about the machine concisely — not because the machine “really” has a mind.
Ascription is most useful precisely when the structure of the system is incompletely known; for fully-listed programs one could in principle simulate instead.
Mental qualities should be ascribed separately (belief without desire, desire without consciousness, etc.) rather than bundled into a monolithic concept of mind; different machines warrant different subsets.
Novel definitional tools: (1) definitions relative to an approximate theory — a mental predicate is defined only within a coarser theory that abstracts away from physical detail; (2) second-order structural definitions — mental qualities defined by the existence of certain structural roles in the system.
Epistemological vs metaphysical adequacy of representations: AI needs representations that capture what is knowable about a situation, not just what is ultimately true.
The approach is intended as conservative (liberal in what it admits having some mental qualities, strict in how each is attributed).

Connections

Conceptual Contribution

Claim: It is legitimate and often necessary to ascribe mental qualities (belief, desire, intention, knowledge) to machines, provided the ascription is conservative, quality-by-quality, and grounded either in an approximate theory that abstracts away from physical detail or in a second-order structural definition that names the role the quality plays.
Mechanism: Philosophical / definitional. Distinguishes epistemological from metaphysical adequacy of representations, separates mental qualities rather than bundling them, and introduces approximate-theory and structural-definition techniques for defining predicates like believes(M, P) over machines.
Concepts introduced/used: Intentional Stance, Mentalistic Semantics, Approximate Theory, Structural Definitions, Epistemological Adequacy, Common Sense Reasoning
Stance: foundational / philosophical
Relates to: Source of the mental-ascription tradition that Intention Is Choice with Commitment formalises and that Semantics and Conversations for an ACL applies to KQML. ACL Rethinking Principles argues this tradition overreaches when used as a compliance criterion for ACLs; McCarthy’s own hedges about conservativism actually support that pushback.

In this vault

Weak Agency

Wooldridge’s baseline notion: an agent is autonomous, social, reactive, and proactive — no mental-state commitment required.

In this vault

Instrumental Convergence

(page does not exist)

Nash Equilibrium

A strategy profile from which no player can gain by unilaterally deviating. In multi-agent systems with communication failures, the set of Nash equilibria reached by best-response dynamics can include arbitrarily poor outcomes — the core negative result of the Gopalakrishnan et al. resilience analysis.

In this vault

Strategic Relevance

(page does not exist)

Identification of Joint Interventional Distributions - Shpitser Pearl

Identification of Joint Interventional Distributions in Recursive Semi-Markovian Causal Models

Reference: Shpitser, I. & Pearl, J. (2006). Identification of Joint Interventional Distributions in Recursive Semi-Markovian Causal Models. In Proceedings of the 21st National Conference on Artificial Intelligence (AAAI-06), Boston, July 2006, pp. 1219–1226. URL (AAAI PDF) · Companion: What Counterfactuals Can Be Tested, UAI-07

Summary

Shpitser and Pearl give the complete identification algorithm for non-parametric causal effects in recursive semi-Markovian causal models — the result that finishes the project Pearl 1995 began. The headline: given a DAG with possibly hidden variables, and a target intervention distribution P(Y | do(X = x)), there is a sound and complete algorithm — ID — that either returns a closed-form expression in the observational distribution P(V) or proves that no such expression exists (the effect is unidentifiable). Where Pearl 1995 gave sufficient conditions (backdoor, front-door) leaving a grey zone of “we haven’t found an adjustment, but maybe one exists,” Shpitser–Pearl close the grey zone: the ID algorithm finds an expression if and only if one exists. The do-calculus of Causality (2000) turns out to be exactly the right deductive system — the ID algorithm is essentially a complete proof-search procedure for do-calculus derivations.

The mechanism is technical but compact. The algorithm operates on a semi-Markovian causal DAG (a DAG with hidden / latent variables represented by bidirected edges in the ADMG — acyclic directed mixed graph). It recursively decomposes the target distribution by c-component factorisation — a generalisation of chain-rule decomposition that respects the latent-variable structure — and at each step either reduces to an identified sub-problem or detects a hedge, a particular graphical pattern (the canonical obstruction to identifiability) that proves unidentifiability. The companion paper What Counterfactuals Can Be Tested (UAI-07) extends the framework to counterfactual queries (layer-3 in the Pearl ladder), characterising which counterfactuals are identifiable from interventional data.

For CBCL / CIVeX, the Shpitser–Pearl ID algorithm is the operational substrate. A CIVeX certificate states a target causal query, the assumed DAG, and the ID derivation that identifies the query in terms of observational data. The verifier — running a small implementation of ID — checks the derivation against the algorithm. If the certificate type-checks, the causal claim is provably identifiable under the stated assumptions; if it fails, the certificate is rejected. Completeness means the verifier never needs to ask the prover for a different formulation: if any identification exists, ID finds it. The classical pair with PCC is exact: PCC + ID together give the picture of “ship the causal proof, check the proof” for causal-inference artefacts. Shpitser–Pearl is therefore one of the most direct technical citations the CBCL / CIVeX discussion section needs.

Key Ideas

Recursive semi-Markovian causal model: an acyclic directed graph over observed and hidden variables, equivalently an acyclic directed mixed graph (ADMG) over observed variables with bidirected edges representing shared hidden parents. The standard formal substrate for causal identifiability when some confounders are unmeasured.
Joint interventional distribution: P(Y | do(X = x)) for Y a (possibly multivariate) target and X a (possibly multivariate) intervention set. The paper handles the joint multivariate case rather than just single-cause single-effect.
ID algorithm: a recursive procedure that either returns a closed-form expression for P(Y | do(X = x)) in terms of P(V) (the observational distribution over observed variables) or detects unidentifiability via a hedge obstruction.
Soundness and completeness: the ID algorithm is sound (every expression returned is correct) and complete (if any expression for P(Y | do(X = x)) exists, ID returns one). Together with the soundness/completeness of do-calculus, this fully resolves non-parametric identifiability.
c-component factorisation: the structural decomposition that drives ID. Each c-component is a maximal set of observed variables connected by paths of bidirected edges; the joint distribution factorises along c-components in a way that respects the latent-variable structure.
Hedge as canonical obstruction: a hedge is a particular graphical pattern (a pair of c-forests in a tree-like configuration) whose presence proves unidentifiability. Every unidentifiable case witnesses a hedge.
Completeness of do-calculus: a corollary — Pearl’s three-rule do-calculus is complete for non-parametric identification. The ID algorithm is, in effect, an automated do-calculus prover.
Companion: counterfactual identification: What Counterfactuals Can Be Tested (UAI-07) extends the framework to layer-3 (counterfactual) queries, characterising which counterfactuals can be identified from interventional distributions.

Connections

Conceptual Contribution

Claim: For non-parametric causal identification in recursive semi-Markovian models, there is a sound and complete algorithm: given a target intervention distribution and a DAG with hidden variables, the algorithm either expresses the target in observational quantities or proves unidentifiability via the hedge graphical pattern. As a corollary, Pearl’s do-calculus is complete for identification.
Mechanism: The ID algorithm operates on a semi-Markovian DAG (or equivalently an ADMG over observed variables). At each recursive step, it performs c-component factorisation to decompose the target distribution along the latent-variable-induced equivalence classes; it then either reduces the problem to identified subproblems or recognises a hedge obstruction and outputs unidentifiable. Soundness follows from each step being a do-calculus transformation; completeness follows from hedges being the only obstruction.
Concepts introduced/used: ID Algorithm, c-Component, Hedge (Causal Inference), Semi-Markovian Causal Model, Acyclic Directed Mixed Graph, Identifiability, Completeness of do-Calculus.
Stance: foundational technical paper.
Relates to: The algorithmic completion of Pearl 1995 / Causality 2000: where 1995/2000 gave sufficient graphical conditions and the deductive do-calculus, Shpitser–Pearl 2006 gives the complete decision procedure. The companion What Counterfactuals Can Be Tested (UAI-07) extends to layer-3 queries. The complementary framework is Rubin’s potential outcomes; the algorithmic completeness on the graphical side has no equally clean counterpart on the potential-outcomes side, which is part of why graphical methods have won the practical computational ground. For CBCL / CIVeX, this is the operational substrate: a CIVeX certificate carries a DAG, a target query, and an ID derivation; the verifier runs a small ID implementation to check the derivation. Pairs with PCC in the obvious way: PCC + ID = “ship the causal proof, check the proof” for causal-inference artefacts. The CBCL/CIVeX discussion section’s claim that causal certificates are decidably checkable leans on Shpitser–Pearl completeness; without it, the architecture would have to apologise for occasionally being too weak.

Theory of Agency

The philosophical and formal account of what it is to be an agent — beliefs, goals, intentions, autonomy, social standing — and the first of the five persistent issues in ACL research.

Discovering Agents

Summary

Key Ideas

Connections

Conceptual Contribution

Tags

Backlinks