Interpreting Dynamical Systems as Bayesian Reasoners

Reference: Virgo, N., Biehl, M. & McGregor, S. (2021). Interpreting Dynamical Systems as Bayesian Reasoners. Proceedings of the 2nd International Workshop on Active Inference (IWAI 2021); published in Machine Learning and Principles and Practice of Knowledge Discovery in Databases, Communications in Computer and Information Science, vol. 1524, Springer, Cham. arXiv:2112.13523 (Earth-Life Science Institute, Tokyo Institute of Technology; Araya Inc., Tokyo; University of Sussex). URL.

Summary

When is a physical system entitled to be described as having beliefs about the world? Active inference takes as a central premise that the internal states of a physical system parametrise probability measures over states of the external world, and that these measures are the system’s beliefs — its Bayesian priors and posteriors. Virgo, Biehl and McGregor ask what would license that premise. Their question is deliberately narrower than agency: given a physical system influenced by its surroundings, under what circumstances can it be said to be performing inference, such that its internal states could be said to contain knowledge or beliefs about the outside world? Goals and actions are set aside; the system here is a reasoner, not yet an agent. The paper’s stated contribution is to make the parametrisation idea formal, and thereby to clarify the precise relationship between the interpretation level and the underlying physical level.

The approach has, in the authors’ words, “something in common with Dennett’s intentional stance”: whether a system performs inference is treated as a matter of interpretation, but interpretations are tied strongly to the underlying physical dynamics, so that whether a given interpretation is consistent becomes mathematically well-defined and has a definite answer. The machinery is category-theoretic — measurable spaces, Markov kernels, and string diagrams. A machine (Def. 1) is a state space Y, an input space S, and an update kernel γ : Y × S → P(Y). An interpretation map is a Markov kernel ψ_H : Y → P(H) sending each machine state to a belief over a hidden space H; unlike the kernels of the causal model, its job is to map states to beliefs rather than to model causal influence. A consistent Bayesian filtering interpretation (Def. 2) adds a model κ : H → P(H × S) — the reasoner’s own account of how the hidden world evolves and emits sensor values — subject to a consistency equation: sampling the reasoner’s joint belief about the next hidden state and next input and then feeding that input to the machine must agree with feeding in the sampled input and then reading off the machine’s resulting posterior belief. A Bayesian inference interpretation (Def. 3) is the special case where the hidden state does not change, so H is an unknown parameter, the model reduces to φ : H → P(S), and inputs are treated as i.i.d. samples; the authors relate this case to the category-theoretic notion of a conjugate prior, generalising earlier work on the Dirichlet case. A reasoner is a machine together with a chosen interpretation.

Two consequences are given prominence. First, an interpretation depends only on the machine’s internal dynamics — on Y, S and γ — and not at all on the external environment. A reasoner’s beliefs may therefore be consistent but incorrect; the authors note one can construct examples where the world “as the system sees it” has a different causal structure from the world as it really is, since a reasoner under an inference interpretation models its inputs as i.i.d. while their true generative structure may be an arbitrary Bayesian network. Second, interpretation is genuinely underdetermined: a given machine may admit multiple non-equivalent consistent interpretations, and the “correct” one cannot be determined from the system’s dynamics alone — the authors leave open that broader context (a designer’s intent, or selection by evolution) might distinguish them. Consistency is nonetheless a real constraint: every machine admits a trivial interpretation with respect to a parameterless model, and in the finite case non-trivial interpretations can exist only if some transitions are impossible; also in the finite case, consistency turns out to depend only on which states are reachable from which under a given input, not on the actual transition probabilities. Appendix C supplies three worked examples, including a three-state non-deterministic machine and a countably infinite machine interpreted as inferring the bias of a coin via the standard conjugate prior.

The discussion is explicit that the contribution is conceptual and that the framework is a beginning. Analogous consistency equations should exist for approximate Bayesian filtering, for smoothing and prediction, and probably even for non-Bayesian normative theories of what a system ought to represent. The most-signalled extension is to allow the machine’s state to influence the external world — adding deliberate actions combined with beliefs and possibly goals — at which point “a machine with such an interpretation might deserve the term agent instead of reasoner.” That extension is what Interpreting Systems as Solving POMDPs carries out. On the Free Energy Principle, the authors find that the ingredients of an interpretation map appear present in the Bayesian mechanics literature, but say it is not currently clear to them whether the FEP can be phrased as a consistency equation obeyed by that kernel; any such equation would differ from Defs. 2 and 3 because the FEP concerns approximate rather than exact inference and continuous time. They identify the FEP’s stationarity assumption on the true agent-environment dynamics as the key point of divergence — it is used to derive something corresponding to a model, whereas here the reasoner’s model and the ground-truth environment are separate things, so no stationarity assumption is needed. They see this conceptual separation as an advantage.

Key Ideas

Beliefs as parametrisation, made formal: active inference assumes internal states parametrise measures over external states; this paper asks when that reading is licensed, giving a mathematically definite answer for the filtering and inference cases.
Reasoners before agents: the scope is perception and representation only — no goals, no actions, no outputs. The system is a reasoner, and the authors flag agents as the next step.
A machine (Def. 1): two measurable spaces Y (states) and S (inputs) plus an update kernel γ : Y × S → P(Y); typically imagined inside a larger causal Bayesian network with hidden world states X_0, X_1, ….
Interpretation map ψ_H : Y → P(H): a Markov kernel read functionally — it sends a state y to a belief ψ_H(y) over hypotheses in H, playing a semantic role quite unlike the causal kernels of the graphical model.
Consistent Bayesian filtering interpretation (Def. 2): an H, an interpretation map ψ_H, and a model κ : H → P(H × S) satisfying a consistency equation — belief-then-input must agree with input-then-posterior. Bayes’ rule, in string-diagram form.
Consistent Bayesian inference interpretation (Def. 3): the special case of a static hidden state; H becomes an unknown parameter, the model is φ : H → P(S), inputs are modelled as i.i.d. Closely related to conjugate priors as characterised category-theoretically; generalises earlier Dirichlet-specific work.
A reasoner is a pair: a machine together with a particular choice of interpretation — being a reasoner is not an intrinsic property of the machine.
Interpretation ignores the environment: consistency depends only on Y, S, γ. Hence beliefs can be consistent but incorrect, and the world as the system models it may have a different causal structure from the world as it is (i.i.d. inputs vs. an arbitrary generative network).
Underdetermination: a machine may have several non-equivalent consistent interpretations, indistinguishable by inspecting internal dynamics alone; the choice of interpretation involves an ineliminable element of choice. External context (designer intent, evolutionary selection) is left open as a possible tiebreaker.
Consistency still bites: every machine has a trivial interpretation (parameterless model); in the finite case non-trivial interpretations require some transitions to be impossible, and consistency depends only on reachability, not on the transition probabilities themselves.
Category theory as the medium: measurable spaces, Markov kernels, and string diagrams from categorical probability, chosen to separate the physical machine cleanly from the semantic level of priors and posteriors.
Contrast with the FEP: the FEP needs a stationarity assumption on the true dynamics in order to derive a model; here the model is part of the interpretation and independent of ground truth, so no stationarity is needed. Whether the FEP admits a consistency equation at all is left open.
Not lenses: related categorical work on agency centres on lenses; the authors observe their interpretations lack the bidirectional structure of lenses, and conjecture bidirectionality would be needed for Bayesian smoothing rather than filtering.

Connections

Conceptual Contribution

Claim: A dynamical system may be said to hold beliefs about a hidden external world when its internal dynamics admit a consistent interpretation as performing Bayesian filtering (or, in the static-hidden-state case, Bayesian inference) — that is, when its states can be mapped to probability measures over hypotheses in a way that updates according to Bayes’ rule under the system’s own model. Whether such an interpretation exists is a well-posed mathematical question about the machine alone; whether the beliefs so ascribed are true of the actual environment is a separate question the framework deliberately does not ask.
Mechanism: Define a machine as (Y, S, γ : Y × S → P(Y)) (Def. 1). Add an interpretation map ψ_H : Y → P(H) sending states to beliefs over an interpretation-internal hidden space, and a model κ : H → P(H × S) describing how the reasoner takes the hidden world to evolve and emit sensor values. Derive the induced kernels ψ_{S,H',H}, ψ_{S,H'} and ψ_S by composition, then impose the consistency equation of Def. 2 in string-diagram form: sampling the belief over next hidden state and next input, then updating the machine, equals feeding in the sampled input and reading the posterior. Def. 3 specialises to a time-invariant hidden state with φ : H → P(S), recovering Bayes’ theorem in the discrete case and connecting to the categorical account of conjugate priors. Appendix B characterises which machines admit non-trivial interpretations; Appendix C exhibits three concrete examples.
Concepts introduced/used: Interpretation Map, Bayesian Filtering, Bayesian Inference, Belief, Markov Kernel, String Diagrams, Conjugate Prior, Category Theory, Intentional Stance, Free Energy Principle
Stance: formal theory / conceptual definition (category-theoretic; worked examples, no empirical study)
Relates to: The direct predecessor of Interpreting Systems as Solving POMDPs, which supplies exactly the extension flagged in this paper’s discussion — letting the machine influence the world, adding actions and goals, and thereby promoting the reasoner to an agent. Formalises the ascription question posed philosophically in True Believers - The Intentional Strategy and Why It Works, keeping Dennett’s move of treating belief-ascription as interpretation while replacing predictive advantage with a consistency condition on dynamics. Provides a mathematical footing for the parametrisation premise of Active Inference and the Free Energy Principle, while separating the reasoner’s model from the true environment and so dispensing with the FEP’s stationarity assumption. Its “consistent but incorrect beliefs” and divergent causal structure connect to Causality - Pearl; its licensing of mentalistic vocabulary for machines is kin to Ascribing Mental Qualities to Machines and The Knowledge Level, and a formal foil to Intelligence Without Representation.

Backlinks

Linked Pages

Intelligence Without Representation

Reference

Brooks, Rodney A. (1991). “Intelligence without representation.” Artificial Intelligence 47: 139–159. (Received September 1987.)
PDF: people.csail.mit.edu/brooks/papers/representation.pdf
Local: brooks1991_representation.pdf

Summary

Brooks challenges the dominant symbol-manipulation paradigm of classical AI, arguing that “representation is the wrong unit of abstraction in building the bulkiest parts of intelligent systems.” Drawing on an evolutionary argument — 3.5 billion years were spent evolving sensorimotor and survival competence, and only a few thousand years on “expert” symbolic knowledge — he claims the hard part of intelligence is mobility, vision, and acting in dynamic environments, not the formal problem-solving that dominated early AI. The “blocks-world” style of AI succeeds only because researchers have already factored out perception and motor action, relegating them to input/output black boxes.

In place of centralized symbolic architectures, Brooks proposes building autonomous mobile robots he calls “Creatures,” decomposed by activity rather than by function. Each activity-producing layer connects sensing to action directly and runs in parallel; higher layers are incrementally added on top of lower, already-working layers without a central symbolic world model. This is the subsumption architecture. The slogan is “the world is its own best model”: rather than maintaining an internal representation the robot re-perceives what it needs.

The paper’s philosophical payload is a rejection of the sense–model–plan–act pipeline and of the assumption that intelligence can be decomposed into modules trading symbolic messages. Brooks argues this methodological commitment — that perception delivers a human-style Merkwelt that central reasoning can operate on — is a form of self-deception, since each species and each robot inhabits its own Merkwelt. Intelligence is to be built bottom-up, situated and embodied, tested continuously against the real world.

Key Ideas

Situatedness and embodiment as design primitives: Creatures must cope with a dynamic environment and have some purpose in being.
Decomposition by activity, not function: layers of task-achieving behavior (avoid, wander, explore) replace perception → modeling → planning → action pipelines.
Subsumption architecture: higher competence layers inhibit/subsume lower ones while lower continue running.
No central representation: “the world is its own best model”; no shared symbolic world model between layers.
Evolutionary time argument: sensorimotor intelligence is the hard, ancient core; symbolic expertise is a thin late veneer.
Critique of abstraction: abstracting away perception and action is how AI self-deceives about having solved problems.
Hypothesis (H): representation is the wrong primitive for building intelligent systems.

Connections

Subsumption Architecture — this paper is the manifesto.
Intelligent Agents Theory and Practice — Wooldridge & Jennings use Brooks as the exemplar of reactive agents, contrasted with deliberative BDI agents.
The Society of Mind — shares the anti-monolithic, many-small-agents intuition, though Minsky retains representations.
Multi-Agent Systems — reactive/behavior-based approaches feed directly into swarm and MAS work.
Actor Model — independent parallel activity producers without a central controller.
A Universal Modular Actor Formalism for Artificial Intelligence — kindred rejection of centralized control.
Agent Architecture — subsumption is a canonical architecture contrasted with BDI, layered, hybrid.
Contrast with Knowledge Representation, Ontologies, Advice Taker — papers that double down on representation.

Conceptual Contribution

The Knowledge Level

Reference: Allen Newell (1982). “The Knowledge Level.” Artificial Intelligence 18(1): 87-127. (AAAI Presidential Address, Stanford, 19 Aug 1980; also in AI Magazine 2(2), 1981.) URL

Summary

Newell’s AAAI-80 presidential address proposes that above the symbol level (data structures manipulated by programs) there exists a distinct knowledge level whose medium is knowledge and whose laws are rationality. At the knowledge level an agent is characterised by what it knows and what goals it has; its behaviour is predicted by the principle of rationality — “if an agent knows that one of its actions will lead to a goal, it will select that action” — without commitment to any particular representation or processing mechanism.

The paper motivates the proposal by diagnosing three tensions in early-1980s AI: the mystification of “representation,” the residue of the 1960s-70s theorem-proving controversy (Planner/Microplanner/Conniver’s “uniform procedures will not work” reaction against resolution), and the diversity of answers in the SIGART Special Issue on Knowledge Representation (Brachman and Smith’s survey showing no consensus on quantification, quotation, self-description, or evaluation). Newell’s resolution: treat knowledge as a competence-level notion analogous to physicists’ treatment of temperature — a level with its own laws, reduced to but not replaced by the symbol level. Representations are then understood as symbol-level implementations of knowledge, and we gain a principled way to talk about what an AI system knows without confusing it with how that knowledge is stored.

The Knowledge Level is foundational for agent-oriented programming (Shoham’s “mental state” framing), for the BDI tradition, for KQML’s and FIPA’s claim that ACL semantics should be defined at the level of beliefs/intentions rather than byte-strings, and indirectly for McCarthy’s Elephant 2000, whose speech-act semantics presupposes that one can talk coherently about what a program knows and is committed to, independently of implementation.

Key Ideas

Computer systems are describable at multiple levels (device, circuit, logic, register-transfer, program/symbol); the knowledge level sits above the symbol level.
Medium of the knowledge level: knowledge (distinct from the data structures that represent it).
Law of behaviour at the knowledge level: the principle of rationality — actions selected to achieve goals given what is known.
The knowledge level is a competence-level account: it predicts behaviour without specifying mechanism, analogously to economic rationality.
Representation is the symbol-level realisation of knowledge; mystifying representation conflates the levels.
Diagnostic of the theorem-proving backlash: “uniform procedures will not work” was a symbol-level reaction to a knowledge-level question.

Connections

Agent-Oriented Programming — Shoham explicitly situates AOP at the knowledge level; mental states are Newell-level predicates over agents.
Elephant 2000 - A Programming Language Based on Speech Acts — Elephant commits to knowledge-level program variables (propositions, commitments, past speech acts), making knowledge-level description the native semantics of programs.
Ascribing Mental Qualities to Machines — McCarthy’s ascriptivism is the philosophical cousin of Newell’s levels argument.
Knowledge Representation
Agent Communication Languages — ACL semantics in the KQML/FIPA tradition are knowledge-level (beliefs/intentions) rather than symbol-level (bit patterns).
KIF — intended as a lingua franca at the symbol level that preserves knowledge-level content across heterogeneous systems.
Common Sense Reasoning

Conceptual Contribution

Claim: AI systems have a level of description above the symbol level — the knowledge level — whose medium is knowledge and whose law is rationality; this level is reducible to but not replaceable by the symbol level, and recognising it clarifies decades of representation debates.
Mechanism: Levels-of-description methodology imported from computer architecture; principle of rationality as the behaviour law; knowledge characterised competence-theoretically (what follows from what an agent knows and wants, not how); representations reconceived as symbol-level implementations of knowledge-level content; diagnostic application to the theorem-proving controversy and the SIGART Special Issue.
Concepts introduced/used: Knowledge Level, Symbol Level, Principle of Rationality, Competence vs Performance, Knowledge Representation, Levels of Description.
Stance: foundational
Relates to: The conceptual ground for agent-oriented and BDI programming; provides the “what an agent knows” vocabulary that KQML, Agent Communication Languages, and Elephant 2000 - A Programming Language Based on Speech Acts rely on; complements McCarthy’s ascriptivist argument in Ascribing Mental Qualities to Machines.

Ascribing Mental Qualities to Machines

Reference: John McCarthy (1979). Stanford CS Department; reprinted in Formalizing Common Sense: Papers by John McCarthy (V. Lifschitz ed., Ablex, 1990). Source file: mccarthy-ascribing.pdf. URL

Summary

McCarthy argues that ascribing beliefs, desires, intentions, knowledge, and other mental qualities to machines (including simple ones like thermostats) is legitimate when the ascription conveys the same information about the machine that it would about a person, and useful when it helps us predict, repair, or improve the machine’s behaviour. He proposes two new definitional tools — definitions relative to an approximate theory, and second-order structural definitions — so that mental-state vocabulary can be applied to physical systems conservatively rather than by reduction to physics.

This is the philosophical foundation for the entire mentalistic (BDI) tradition in agent research that ACL Rethinking Principles pushes back against. Singh cites McCarthy as the origin of the view that agent communication can be specified by ascribing beliefs and intentions; McCarthy himself is already careful to warn that such ascriptions must be conservative and may be merely convenient shorthand for structural facts.

Key Ideas

Ascribing mental qualities is warranted when it expresses information about the machine concisely — not because the machine “really” has a mind.
Ascription is most useful precisely when the structure of the system is incompletely known; for fully-listed programs one could in principle simulate instead.
Mental qualities should be ascribed separately (belief without desire, desire without consciousness, etc.) rather than bundled into a monolithic concept of mind; different machines warrant different subsets.
Novel definitional tools: (1) definitions relative to an approximate theory — a mental predicate is defined only within a coarser theory that abstracts away from physical detail; (2) second-order structural definitions — mental qualities defined by the existence of certain structural roles in the system.
Epistemological vs metaphysical adequacy of representations: AI needs representations that capture what is knowable about a situation, not just what is ultimately true.
The approach is intended as conservative (liberal in what it admits having some mental qualities, strict in how each is attributed).

Connections

Conceptual Contribution

Claim: It is legitimate and often necessary to ascribe mental qualities (belief, desire, intention, knowledge) to machines, provided the ascription is conservative, quality-by-quality, and grounded either in an approximate theory that abstracts away from physical detail or in a second-order structural definition that names the role the quality plays.
Mechanism: Philosophical / definitional. Distinguishes epistemological from metaphysical adequacy of representations, separates mental qualities rather than bundling them, and introduces approximate-theory and structural-definition techniques for defining predicates like believes(M, P) over machines.
Concepts introduced/used: Intentional Stance, Mentalistic Semantics, Approximate Theory, Structural Definitions, Epistemological Adequacy, Common Sense Reasoning
Stance: foundational / philosophical
Relates to: Source of the mental-ascription tradition that Intention Is Choice with Commitment formalises and that Semantics and Conversations for an ACL applies to KQML. ACL Rethinking Principles argues this tradition overreaches when used as a compliance criterion for ACLs; McCarthy’s own hedges about conservativism actually support that pushback.

Causality: Models, Reasoning, and Inference

Reference: Pearl, J. (2000, 2nd ed. 2009). Causality: Models, Reasoning, and Inference. Cambridge University Press. 384 pp (1st ed.) / 484 pp (2nd ed.). URL (ILLC Amsterdam, 2nd ed. PDF) · Cambridge UP page · Internet Archive

Summary

Causality is the book-length statement of Pearl’s graphical-causal-inference programme — the work that fixes a usable vocabulary for causation in the empirical sciences, replacing decades of philosophical regress with a constructive theory. The book extends the apparatus of Causal Diagrams for Empirical Research (1995) along three axes. (1) From interventional to counterfactual causal queries: the structural causal model (SCM) — a system of structural equations with exogenous error terms — supports queries about what would have happened if particular variables had taken different values, not only about intervention distributions. (2) From backdoor / front-door criteria to do-calculus: a complete deductive system (three rules: insertion/deletion of observations, action/observation exchange, and insertion/deletion of actions) for transforming do-expressions. Together with Shpitser & Pearl 2006’s completeness result, do-calculus is complete for non-parametric identifiability. (3) From identifiability to applied causal inference: chapters on confounding, mediation, instrumental variables, the actual-cause problem, and the foundations of legal-causal reasoning.

The technical core in three layers. Layer 1 (association): ordinary probabilistic conditional P(Y | X). Layer 2 (intervention): post-do distribution P(Y | do(X = x)), derived from the SCM by surgical replacement of X’s equation. Layer 3 (counterfactuals): queries like P(Y_{X=x} | X = x', Y = y') — “given that we observed X = x' and Y = y', what would Y have been if we had set X = x?” — derived by the abduction–action–prediction procedure: update exogenous-error distributions on the evidence, intervene on X, predict Y. Pearl calls this stratification the Ladder of Causation; the book argues that the climb from layer 1 to layer 3 is a matter of added assumptions (structural equations, then exogenous-error distributions), not of finer-grained statistical methodology.

For CBCL and CIVeX, the book is the textbook reference. CIVeX certificates carry a DAG, a stated set of structural assumptions, and a do-calculus derivation establishing identifiability of the target causal query. The host’s verifier checks the derivation against the do-calculus rules. The completeness of do-calculus (Shpitser & Pearl 2006) means that the verifier never has to apologise for being too weak: if the effect is identifiable at all, do-calculus derives it. The book also supplies the vocabulary for stating what CIVeX certificates are claiming: distinctions between association, intervention, and counterfactual; between identifiability and estimability; between confounding, mediation, and selection — all of which are needed to write precise threat models for causal claims in multi-agent settings.

Key Ideas

Structural Causal Model (SCM): a system of structural equations Vᵢ := fᵢ(PAᵢ, Uᵢ) over endogenous variables V with exogenous errors U, plus a distribution over U. The SCM is the formal carrier of all three causal layers.
Ladder of Causation: three layers of causal queries, each requiring strictly more assumptions. Layer 1 (seeing) — P(Y | X). Layer 2 (doing) — P(Y | do(X = x)). Layer 3 (imagining) — counterfactuals P(Y_{X=x} | E) for arbitrary evidence E.
do-calculus: three syntactic transformation rules over do-expressions: (R1) insertion/deletion of observations under d-separation; (R2) action/observation exchange under modified d-separation; (R3) insertion/deletion of actions under absence-of-causal-paths. Combined with probability axioms, the calculus is complete for non-parametric identifiability (Shpitser–Pearl 2006).
Identifiability under unobserved confounders: many effects are identifiable even when key confounders are unmeasured. The do-calculus discovers when, and the Shpitser–Pearl ID algorithm gives a complete procedure.
Counterfactual queries by abduction–action–prediction: (i) Update P(U) on observed evidence (abduction). (ii) Intervene on the SCM (action). (iii) Compute the distribution of the target variable (prediction). The procedure yields counterfactuals from the SCM.
Mediation analysis: natural direct, natural indirect, and total effects decompose causal influence along specific paths. Identification conditions are characterised by the mediation formula and the more general path-specific effects.
Instrumental variables: a classical econometric tool given a clean graphical characterisation — an instrument is a variable affecting X but not Y directly or via unmeasured confounders. The IV estimand is recoverable when the graphical conditions hold.
Actual cause / responsibility: chapter 10 develops a formal theory of actual causation — what actually caused the outcome in this token case? — built on counterfactuals and contingency. Foundational for legal and AI-ethics applications.
Transportability: under what conditions can a causal effect estimated in one population be transferred to another? Bareinboim–Pearl give graphical criteria.

Connections

Conceptual Contribution

Claim: Causal reasoning admits a precise, computable, non-parametric foundation built on structural causal models (SCMs), the do-operator for intervention, and the do-calculus for transforming causal queries. The three-layer Ladder of Causation (association / intervention / counterfactual) stratifies the assumptions required for each kind of causal claim.
Mechanism: Define SCMs as systems of structural equations with exogenous errors. Define interventions surgically (replace X’s equation by X := x). Define counterfactuals via abduction–action–prediction on the SCM. Give the do-calculus (three transformation rules) and show it is sound; later work (Shpitser–Pearl 2006) shows it is also complete for identification. Extend to mediation, instruments, actual causation, transportability across populations.
Concepts introduced/used: Structural Causal Model, Ladder of Causation, do-Calculus, Counterfactual, Mediation Analysis, Instrumental Variable, Actual Cause, Transportability, Identifiability.
Stance: foundational technical monograph / textbook.
Relates to: The book that the paper Causal Diagrams (1995) prefigures and that Shpitser & Pearl 2006 completes algorithmically. The complementary framework is Rubin’s potential outcomes (1974): the same causal effects in counterfactual notation, with explicit formal mappings between the two languages (Pearl–Bareinboim 2009). For CBCL / CIVeX, the book is the textbook reference: CIVeX certificates carry DAGs, structural assumptions, and do-calculus derivations of identifiability; the host’s verifier checks the derivation against the calculus. The PCC pairing is exact (Necula): PCC ships behavioural proofs over machine states; CIVeX ships causal-identification proofs over the DAG; both rely on small, generic verifiers that check rather than re-derive.

Free Energy Principle

(page does not exist)

Active Inference

(page does not exist)

True Believers - The Intentional Strategy and Why It Works

True Believers: The Intentional Strategy and Why It Works

Reference: Dennett, D. C. (1981). In A. F. Heath (Ed.), Scientific Explanations, Oxford University Press. Reprinted in The Intentional Stance (1987, MIT Press, ch. 2) and in Mind Design II (1997). Source file: dennett_true_believers.pdf. URL

Summary

Dennett’s canonical defence of the intentional stance: a predictive strategy in which one treats a system as a rational agent with beliefs and desires, works out what a rational agent so situated ought to do, and predicts it will do that. The stance is contrasted with the physical stance (predict from physical laws and microstructure) and the design stance (predict from presumed function). Belief is a perfectly objective phenomenon, but one discernible only from the point of view of an adopter of this strategy; a system is a true believer just in case its behaviour is reliably predictable by intentional ascription.

Dennett argues this vindicates a “mild realism” (or “interpretationism”) about belief against both hard-core realism (beliefs as sentence-like tokens in the head) and eliminativism. The remarkable empirical fact is that the intentional strategy works — on people, animals, chess programs, even thermostats (in a degenerate way) — because evolution and engineering produce systems whose behaviour tracks their interests given accurate beliefs. This pattern of success is what there is to being a believer; no further fact about inner sentential structure is required. The paper grounds the philosophical licence to ascribe mental states to artificial agents and thus underwrites the mentalistic vocabulary of agent theory.

Key Ideas

Three stances: physical, design, intentional.
Intentional strategy: assume rationality, compute ought-to-do, predict.
True believer = system reliably predicted by the intentional strategy.
Belief is objective but stance-relative (mild realism / interpretationism).
Natural selection explains why the strategy works on organisms.
Thermostats as degenerate (but continuous) intentional systems.
Licence for mentalistic ascription to artificial agents.

Connections

Conceptual Contribution

Claim: Beliefs and desires are objective but stance-relative — a system genuinely has them just in case adopting the intentional strategy (assume rationality with appropriate beliefs/desires and predict accordingly) reliably predicts its behaviour.
Mechanism: Dennett distinguishes physical, design, and intentional stances; defines the intentional strategy operationally (attribute beliefs the system ought to have given its perceptual and epistemic history, attribute desires it ought to have given its biological/functional role, predict the rational action); argues success of the strategy on people, animals, and artefacts is an empirical fact explained by evolutionary/design pressures, and that this success is all there is to being a believer — no further requirement of language-of-thought tokens.
Concepts introduced/used: Intentional Stance, True Believer, Mild Realism, Design Stance, Physical Stance, Rationality Assumption
Stance: philosophical theory
Relates to: Justifies the ascription of BDI mental states to artificial systems in Modeling Rational Agents within a BDI-Architecture, Intelligent Agents Theory and Practice and Agent-Oriented Programming; cohabits with Two Faces of Intention as a pillar of agent-theoretic philosophy; foreshadows functional-role arguments in Society of Mind.

Interpreting systems as solving POMDPs: a step towards a formal understanding of agency

Reference: Biehl, M. & Virgo, N. (2022). Interpreting systems as solving POMDPs: a step towards a formal understanding of agency. Proceedings of the 3rd International Workshop on Active Inference (IWAI 2022); published in Active Inference. IWAI 2022, Communications in Computer and Information Science, vol. 1721, Springer, Cham. arXiv:2209.01619 (Cross Labs / Cross Compass, Tokyo; Earth-Life Science Institute, Tokyo Institute of Technology). URL.

Summary

When is a physical system entitled to be described as having beliefs and goals? Biehl and Virgo answer with a consistency criterion rather than a predictive one. Their starting point is Dennett’s Intentional Stance, which licenses agent-talk when it out-predicts the Physical Stance. The authors explicitly depart from this comparison-of-stances approach: assuming we already know what the system does at the physical level, they ask whether we can consistently ascribe meaning to its physical states such that those states appear to implement belief updating and decision making. This builds directly on Virgo, Biehl & McGregor’s earlier notion of interpreting a dynamical system as a Bayesian reasoner — something that takes inputs and makes inferences about a hidden variable, but that has neither goals nor actions. The present paper’s contribution is to extend consistent interpretation to cover goals and actions, and thereby to reach agents.

Three deliberate conceptual commitments make this precise. The system is taken to be a stochastic Moore machine (M, I, O, μ, ω) — internal states, inputs, outputs, a stochastic machine kernel, and a deterministic expose function; the authors note this is far from the most general system but is general enough to cover the digital computers running most present-day artificial agents. Beliefs are probability distributions updated in accordance with Bayes’ law. Goals and rationality come from POMDPs, which supply both a formal notion of goal (expected cumulative discounted reward maximisation) and of rational behaviour (the optimal policy). The technical machinery is a consistent Bayesian influenced filtering interpretation (Def. 3): a hidden state space H, an action space A, an interpretation map ψ : M → PH sending machine states to beliefs over hidden states, an action function α : M → A, and a model kernel κ : H × A → P(H × I). A stochastic Moore machine is then interpretable as a solution to a POMDP (Def. 5) if its machine part admits such an interpretation and its expose function emits the optimal action for the belief its state parameterises: ω(m) = π*(ψ(m)). An agent, on this proposal, is the pair of a system and such an interpretation.

Two features do the real work. First, machine states need only parameterise beliefs rather than be belief states — ψ need not be injective, so interpretations come with a built-in notion of coarse-graining, and the definition covers more than the canonical belief-state machines. Proposition 1 shows this is not a loosening in disguise: for any input that is subjectively possible under the interpretation, the parameterised distributions ψ(m) update exactly as the Bayesian belief-state update requires, so V̄*(m) := V*(ψ(m)) is an optimal value function on the machine’s own state space with optimal policy ω(m) = π*(ψ(m)). Such machines are optimal in the same sense as machines whose states are correctly-updated beliefs. Second, consistency is not correctness: whether a consistent interpretation exists depends only on the machine being interpreted, not on any property of its environment. An agent may hold a wrong model — its conclusions will then be wrong while its reasoning stays consistent — and its actions need only be optimal according to its own model, possibly suboptimal in the true environment. The authors flag this as a clear point of difference from the original Free Energy Principle literature, where the model is derived from the actual environment’s dynamics and so cannot differ from it. The hidden states and the rewards need not have any physical realisation at all; they exist only within the interpretation, and the machine never observes the rewards. Appendix C establishes that machines meeting Def. 5 exist by constructing one from the known optimal solution to Sondik’s POMDP.

The authors are careful about scope. The commitments are adopted because they permit precision, not because they are claimed to be realistic or the only reasonable choices; POMDPs are not the only possible formalisation of having a goal; they do not argue that all agents are rational or possess beliefs and goals. The framework requires exact Bayesian updating, so approximate inference — which the FEP does accommodate — falls outside it. The conclusion is offered as one worked example of the kind of agent definition sought, not as a settled general answer.

Key Ideas

Agency as consistent interpretation, not predictive advantage: ask whether the physical dynamics are consistent with an ascription of beliefs and goals, rather than whether the intentional stance out-predicts the physical stance.
An agent is a pair: a system together with an interpretation of it as a POMDP solution — agency is not an intrinsic property of the system alone.
Three conceptual commitments: systems are stochastic Moore machines; beliefs are Bayes-updated probability distributions; goals and rationality are supplied by POMDPs (expected discounted reward, optimal policy).
Interpretation map ψ : M → PH: sends internal machine states to beliefs over an interpretation-internal hidden state space; extended here with an action function α and a model kernel κ (the “influenced filtering” of Def. 3).
Definition 5: a stochastic Moore machine solves a POMDP when it has a consistent Bayesian influenced filtering interpretation and ω(m) = π*(ψ(m)) — it outputs the optimal policy’s action for the belief its state parameterises.
Parameterisation, not identity: machine states need only parameterise beliefs; ψ may be non-injective, so interpretation carries an automatic notion of coarse-graining of the internal state.
Proposition 1: for subjectively possible inputs, ψ(m) updates exactly per the Bayesian belief-state update; hence V*(ψ(m)) is an optimal value function on the machine’s state space — these machines are optimal in the same sense as belief-state machines.
Consistency ≠ correctness: existence of an interpretation depends only on the machine, not the environment. An agent can have a wrong model and act optimally only relative to it — a deliberate contrast with the FEP, where the model derives from actual environment dynamics.
Hidden states and rewards are interpretation-only: they need exist only mathematically, not physically; the machine never observes rewards. Even an output-less machine can have “actions” confined to the interpretation.
Existence is demonstrated, not assumed: Appendix C builds a qualifying stochastic Moore machine from the known optimal solution to Sondik’s POMDP.
Stated limits: exact Bayesian updating only (approximate inference excluded); POMDPs are not the only formalisation of goals; the commitments are chosen for precision rather than realism.

Connections

Conceptual Contribution

Intentional Stance

Dennett’s strategy of predicting a system’s behaviour by treating it as a rational agent with beliefs, desires and goals. Wooldridge and Jennings argue it is a legitimate abstraction for sufficiently complex software, licensing mentalistic design of agents.

In this vault

Category Theory

(page does not exist)

Conjugate Prior

(page does not exist)

String Diagrams

(page does not exist)

Markov Kernel

(page does not exist)

Belief

In agent theory, the propositional content an agent treats as (probably) true. The B in BDI / Belief-Desire-Intention; the operand of the modal operator believes in BAN Logic and the BDI-logic family (Cohen & Levesque, Rao & Georgeff). Distinguished from knowledge (which classically requires truth) and from desire / goal. The unit of Belief Revision under the AGM postulates. In Mentalistic Semantics the meaning of an ACL message is given in terms of the belief updates it induces in sender and receiver — the locus of Singh’s critique that beliefs are unverifiable from message traces.

In this vault

Bayesian Inference

(page does not exist)

Bayesian Filtering

(page does not exist)

Interpretation Map

(page does not exist)

Bayesian Mechanics

(page does not exist)

Causal Diagrams for Empirical Research - Pearl

Causal Diagrams for Empirical Research

Reference: Pearl, J. (1995). Causal Diagrams for Empirical Research. Biometrika 82(4): 669–688. December 1995. With discussion and rejoinder. URL (UCLA CogSys Lab) · DOI

Summary

Pearl’s Causal Diagrams is the paper in which the modern graphical apparatus of causal inference takes recognisable shape: directed acyclic graphs (DAGs) over observed and latent variables as the carrier of non-parametric causal assumptions, the do-operator as the formal representation of intervention, and the backdoor criterion as the first widely-usable graphical test for whether a causal effect is identifiable from observational data given a set of measured covariates. Where the early structural-equations and path-analysis tradition (Wright 1921, Haavelmo 1943) had given causal interpretations to specific parametric models, and where the do-then-randomise tradition (Rubin 1974) had given a counterfactual semantics to causal effects without graphs, Pearl unifies both: the DAG carries the structural assumptions, the do-operator gives the counterfactual semantics, and a small algebra of graphical operations decides identifiability.

The mechanism is precise. A causal model is a DAG G over variables V together with structural equations or, more generally, a non-parametric structural model. The do-operator do(X = x) represents the intervention that sets X to x regardless of X’s natural causes — graphically, by severing the incoming edges to X in G. A causal effect P(Y | do(X = x)) is identifiable from observational data over V iff it can be written as an expression in the observational distribution P(V). The backdoor criterion gives a sufficient condition: a set Z of measured covariates is admissible for adjustment iff Z blocks every backdoor path from X to Y (every path with an arrow into X) and contains no descendants of X. When Z is admissible, P(Y | do(X = x)) = ∑_z P(Y | X = x, Z = z) P(Z = z) — the backdoor adjustment formula. The paper also introduces the front-door criterion for cases where backdoor adjustment fails but a mediator can be exploited.

For the CBCL / CIVeX programme, Pearl 1995 is the foundation. CIVeX certificates carry a DAG of causal assumptions and a proof that the queried effect is identifiable from the available data under those assumptions; this is exactly what Pearl establishes is expressible and checkable. The paper is also a methodological complement to the PCC architecture in CBCL: where PCC ships behavioural proofs, CIVeX ships causal-identification proofs, and both are checked against a stated specification rather than re-derived. The completion of the graphical identifiability story — characterising exactly which causal effects are identifiable in the presence of unobserved confounders — is Shpitser & Pearl 2006’s complete-ID algorithm; Pearl 1995 is the place the story starts to look like an algorithm rather than a case-by-case analysis.

Key Ideas

Causal DAG as non-parametric structural model: a DAG G over V together with structural equations (or a non-parametric Markov-compatible distribution). The DAG carries the causal assumptions; the distribution is what is observed.
do-operator: do(X = x) represents intervention — setting X to x by external manipulation, severing the natural mechanisms that determine X. Graphically: delete incoming edges to X in G; semantically: replace X’s structural equation with X := x. The post-intervention distribution P(V | do(X = x)) is the central object of causal inference.
Identifiability: a causal query P(Y | do(X = x)) is identifiable from observational data over V iff it can be written as an expression in P(V). Identifiability is a graphical property — it depends only on the DAG, not on the specific distribution.
Backdoor criterion: Z ⊆ V is admissible for adjustment iff (i) no node in Z is a descendant of X in G, and (ii) Z blocks every backdoor path from X to Y (every path with an arrow into X). When Z is admissible, P(Y | do(X = x)) = ∑_z P(Y | X = x, Z = z) P(Z = z).
Front-door criterion: when no admissible Z for backdoor adjustment exists, identification may still be possible via a mediator M on the directed path from X to Y. The front-door formula uses M to identify the effect even in the presence of unobserved confounders.
Markov compatibility / d-separation: G constrains P(V) via conditional-independence statements encoded by d-separation — a purely graphical criterion. The empirical content of a causal DAG is exactly the d-separations it implies.
Bridge to interventions: causal effects can be defined purely structurally (no counterfactual ontology needed) by reference to the post-intervention distribution. Counterfactuals re-enter when finer-grained queries (mediation, attribution) are asked, handled by structural causal models in Pearl’s later work.

Connections

Conceptual Contribution

Claim: Causal effects can be represented graphically (as DAGs over observed and latent variables) and identified from observational data by a calculus of graphical operations. The do-operator gives the formal semantics of intervention; the backdoor and front-door criteria give algorithmic identifiability conditions.
Mechanism: Encode causal assumptions as a DAG G over V. Define interventions via the do-operator, semantically the surgical replacement of X’s structural equation by X := x and graphically the severing of incoming edges. State identifiability as the existence of an expression in P(V) for P(Y | do(X = x)). Give the backdoor criterion as a sufficient graphical condition (block all backdoor paths, no descendants of X) together with the backdoor adjustment formula. Give the front-door criterion for the harder case of unmeasured confounding. The result is a non-parametric, graph-driven, algorithmic theory of causal identification.
Concepts introduced/used: do-Calculus, Causal DAG, Structural Causal Model, Intervention, Backdoor Criterion, Front-door Criterion, d-Separation, Identifiability, Markov Compatibility.
Stance: foundational technical paper.
Relates to: Pearl’s compact statement of the apparatus developed at book length in Causality (2000/2009); the algorithmic completion is Shpitser & Pearl 2006. The complementary tradition is Rubin’s potential-outcomes framework (1974) — the same causal effects in counterfactual notation; later work (Pearl–Bareinboim, Imbens–Rubin) establishes formal mappings between the two. For CBCL and CIVeX, this is the substrate: a CIVeX certificate carries a DAG and a proof of identifiability under stated assumptions, checked by the host rather than re-derived. The pairing with PCC is exact: PCC ships behavioural proofs over machine states; CIVeX ships causal-identification proofs over the DAG. The d-separation / Markov-compatibility connection grounds the bridge to probabilistic graphical models more broadly.

Markov Processes

Stochastic processes with the memoryless property: future states depend only on the current state; Chomsky’s finite-state grammars are Markov sources, which he argued were inadequate for natural language.

In this vault

Three Models for the Description of Language

Belief State

(page does not exist)

Theory of Agency

The philosophical and formal account of what it is to be an agent — beliefs, goals, intentions, autonomy, social standing — and the first of the five persistent issues in ACL research.

In this vault

Physical Stance

Dennett: predicting a system’s behaviour from its physical composition and the laws of physics. The first of his three stances (physical, design, intentional).

Interpreting Dynamical Systems as Bayesian Reasoners

Summary

Key Ideas

Connections

Conceptual Contribution

Tags

Backlinks