Active inference and epistemic value

Reference: Friston, K., Rigoli, F., Ognibene, D., Mathys, C., Fitzgerald, T. & Pezzulo, G. (2015). Active inference and epistemic value. Cognitive Neuroscience, 6(4), pp. 187–214 (Discussion Paper; the issue runs to p. 224 with commentaries). Wellcome Trust Centre for Neuroimaging, UCL; Centre for Robotics Research, King’s College London; Translational Neuromodeling Unit, University of Zürich and ETH Zürich; Institute of Cognitive Sciences and Technologies, CNR, Rome. DOI: 10.1080/17588928.2015.1020053. PMID 25689102. URL.

Summary

Where does exploration come from? Most normative accounts of choice bolt it on: a softmax temperature, an exploration bonus, an ad hoc curiosity term added to expected reward. Friston and colleagues argue that no such addition is needed, because exploration is already contained in the imperative to minimise surprise. Their move is to make agents entertain prior beliefs about their own policies — specifically, the belief that they will pursue policies minimising the expected free energy of future outcomes, ln P(ũ|γ) = γ · Q(π), where Q(π) is the negative expected free energy, or quality, of a policy and γ is its precision. Because the agent must believe it will act Bayes-optimally in order to act at all, and because expected free energy is evaluated over an extended future, exploration becomes “a necessary and emergent aspect of optimal behavior” rather than a separate drive. Goals do not enter as a reward function but as prior preferences over outcomes: preferred outcomes are simply the outcomes an agent expects, a priori, to encounter.

The paper’s pivot is an exact decomposition. Under the generative model of future states, the quality of a policy at time τ splits (Eq. 3) into extrinsic value, E_Q(o_τ|π)[ln P(o_τ|m)] — the expected utility of an outcome under the agent’s prior preferences C — and epistemic value, E_Q(o_τ|π)[ D[Q(s_τ|o_τ,π) ‖ Q(s_τ|π)] ], the expected reduction in uncertainty about hidden states afforded by the outcomes a policy would deliver. Rearranged (Eq. 4), epistemic value is exactly the predictive mutual information D[Q(s_τ,o_τ|π) ‖ Q(s_τ|π)Q(o_τ|π)] between hidden states and outcomes, connecting it to the Infomax principle; it is also the expected Bayesian surprise under counterfactual outcomes, which is to say salience. Because a KL divergence cannot be negative and is zero exactly when new observations fail to inform the posterior, epistemic value silently self-extinguishes: policies are drawn to informative “signs” until nothing remains to learn, whereupon extrinsic value alone selects among them. This is the exploration–exploitation resolution — not a schedule over a bonus, but a term that vanishes on its own. A further rearrangement (Eq. 6) recasts quality as minus the expected ambiguity of the likelihood, −E_Q(s_τ|π)[H[P(o_τ|s_τ)]], minus a predicted divergence D[Q(o_τ|π) ‖ P(o_τ|m)] between predicted and preferred outcomes: informative observations and preferred observations, in one currency.

That common currency is what lets the paper nest its rivals inside itself. Since utility becomes a log prior probability, reward and information are both measured in nats. When outcomes unambiguously specify hidden states (o_τ = s_τ), expected free energy reduces to risk-sensitive or KL control (Eq. 5), whose epistemic term degenerates into mere novelty — the entropy of posterior predictive beliefs over future states — and dropping the entropy of outcomes given their causes further reduces it to expected utility and classical reinforcement learning (Eq. 12). Expected utility is thus optimal when, and only when, distinct hidden states generate distinct outcomes; the epistemic value of informative observation is precisely what these schemes cannot see. Simulations use a three-arm T-maze: the agent starts at the centre, a cue in the lower arm discloses whether reward sits in the upper-right or upper-left arm, only two moves are allowed, and baited arms cannot be left. Optimal behaviour therefore requires moving away from the goal to secure it later. Only expected free energy minimisation attains near-optimal performance (about 90%); expected utility and KL control fall below the 3/8 chance level when utility is zero, since the ambiguous upper arms hold no utilitarian attraction. Notably, the epistemic detour only survives once utility exceeds roughly 0.6 nats — the cue is worth one bit, ln 2 ≈ 0.6931 nats, and preference must outbid information to lure the agent from unambiguous outcomes. A hierarchical variant, in which the agent must first learn which of four mazes it inhabits, shows learning as inference and a Bayes-optimal progression from exploration to exploitation, reaching near-optimal performance after about four trials while an expected utility agent never learns which maze it is in. Throughout, the precision γ — a gamma-distributed hyperparameter, not a fixed temperature — behaves like dopamine: its variational updates reproduce the transfer of phasic responses from unconditioned to conditioned stimuli, with the informative cue reducing the later response to reward.

Key Ideas

Exploration is not bolted on: agents hold prior beliefs that they will minimise the expected free energy of future outcomes; because this is evaluated over an extended horizon, epistemic (explorative) behaviour emerges as a consequence of Bayes-optimal inference rather than an added bonus.
Preferences as priors: goals are prior beliefs about outcomes (C), not a reward function. “Preferred outcomes are simply outcomes one expects, a priori, to be realized” — so utility becomes a log probability and shares a currency (nats/bits) with information.
The central decomposition (Eq. 3): Q_τ(π) = E_Q(o_τ|π)[ln P(o_τ|m)] (extrinsic value) + E_Q(o_τ|π)[D[Q(s_τ|o_τ,π) ‖ Q(s_τ|π)]] (epistemic value). Negative expected free energy = extrinsic + epistemic value.
Epistemic value has four faces: expected information gain; predictive mutual information between hidden states and outcomes (Eq. 4, hence Infomax); expected Bayesian surprise under counterfactual outcomes; salience.
Self-extinguishing exploration: a KL divergence is non-negative and smallest when the posterior predictive is uninformed by new observations. Epistemic value is maximised until no further information gain is available; thereafter extrinsic value dominates policy selection. No ad hoc schedule required.
Ambiguity and risk (Eq. 6): quality also equals −E_Q(s_τ|π)[H[P(o_τ|s_τ)]] (predicted uncertainty — ambiguity of the likelihood) − D[Q(o_τ|π) ‖ P(o_τ|m)] (predicted divergence between predicted and preferred outcomes).
Established formalisms as special cases: risk-sensitive/KL control obtains when outcomes unambiguously specify hidden states (o_τ = s_τ, Eq. 5), where epistemic value collapses to novelty (entropy of posterior predictive beliefs over states); expected utility and classical RL obtain by further ignoring the entropy of outcomes given their causes (Eq. 12). The nesting explains the prevalence of classical schemes, which assume hidden states are known.
Value of information: following Howard, information has no epistemic value per se — only relative to policy selection. Expected free energy contextualises value of information within extrinsic value and makes it tractable via approximate Bayesian inference.
Softmax precision is inferred, not stipulated: the inverse temperature γ becomes the precision of (confidence in) posterior beliefs about policies, endowed with a gamma prior Γ(α,β) and updated as γ̂ = α/(β − Q · π̂). Ad hoc softmax parameters thereby acquire Bayes-optimal values.
Precision updates resemble dopamine: changes in expected precision are identical to changes in expected value, offering an inferential reading of reward prediction error; simulated dopaminergic discharges reproduce transfer of responses from unconditioned to conditioned stimuli, with an informative cue attenuating the subsequent response to reward. The framework predicts a multifaceted dopaminergic sensitivity to novelty, salience and expected reward via one precision parameter.
T-maze simulation: four locations × two reward contexts = eight hidden states, sixteen outcomes (four locations × four stimuli: cue-left, cue-right, reward, no reward); reward delivered with probability a = 0.9 under the correct context; two moves only; baited arms are absorbing. Optimal play is the epistemic detour to the cue arm. Across 128 trials at six utility levels (c = 0 to c = 2), only expected free energy minimisation reaches near-optimal (~90%) performance; expected utility and KL control drop below the 3/8 chance level at zero utility.
Utility must outbid information: the epistemic detour appears only when utility exceeds about 0.6 nats, because resolving the hidden context is worth exactly one bit (ln 2 ≈ 0.6931 nats).
Learning as inference: adding a hierarchical level over which maze the agent occupies (32 hidden states) makes the transition from exploration to exploitation an emergent, Bayes-optimal progression — associated speculatively with the goal-directed-to-habitual transition. Performance becomes near optimal after ~4 trials; an expected utility agent fails to learn the maze at all. Suppressing epistemic value within a trial has deleterious consequences for between-trial learning.
The scheme is a process theory: variational message passing over expected states, policies and precision, tentatively mapped to prefrontal cortex/hippocampus (perception), striatum (action selection), and VTA/substantia nigra (precision).

Connections

Conceptual Contribution

Claim: Choice behaviour can be derived from a single imperative — minimise the expected free energy of future outcomes — and the negative expected free energy (the quality) of a policy decomposes exactly into extrinsic value (expected utility under prior preferences) and epistemic value (expected information gain about hidden states). Because the epistemic term is a KL divergence that vanishes precisely when observations can no longer inform beliefs, the exploration–exploitation dilemma is resolved normatively: agents explore until there is nothing left to learn, then exploit. Exploration is thus a consequence of inference, not an addendum to it.
Mechanism: Cast behaviour as inference over a POMDP-style generative model (A, B, C, D, γ) — likelihood A, controlled transitions B(u), prior preferences over outcomes C (a softmax of utility, hence a log probability), initial states D, and a gamma-distributed precision γ. Give the agent the self-consistent prior belief that its control states minimise expected free energy: P(ũ|γ) = σ(γ · Q(π)). Expand Q_τ(π) (Eq. 3) into extrinsic plus epistemic value; show (Eq. 4) that epistemic value equals the predictive mutual information D[Q(s_τ,o_τ|π) ‖ Q(s_τ|π)Q(o_τ|π)], and (Eq. 6) that quality equals minus predicted uncertainty (ambiguity) minus predicted divergence (risk). Recover risk-sensitive/KL control by setting o_τ = s_τ (Eq. 5) and expected utility by further discarding outcome entropy (Eq. 12). Solve by variational Bayes under a mean-field factorisation over hidden states, policies and precision, yielding three coupled updates — perceptual inference ŝ_t = σ(ln A · o_t + ln(B(a_{t−1})ŝ_{t−1})), a softmax over policy value π̂ = σ(γ̂ · Q), and precision γ̂ = α/(β − Q · π̂). Validate on a three-arm T-maze where the optimal policy is an epistemic detour away from reward, and on a hierarchical variant in which the maze itself must be learned.
Concepts introduced/used: Expected Free Energy, Epistemic Value, Extrinsic Value, Bayesian Surprise, Salience, Infomax Principle, Mutual Information, KL Control, Expected Utility, Value of Information, Precision, POMDP, Variational Message Passing, Active Inference, Free Energy Principle
Stance: formal theory plus simulation (normative derivation, process theory, and simulated behavioural/neurophysiological validation; no new empirical data)
Relates to: Supplies the decision-theoretic layer of the Free Energy Principle whose existential/ergodic grounding is argued in Life as We Know It, extending free energy minimisation from perception to policy selection over future outcomes. Shares the POMDP scaffolding of Interpreting Systems as Solving POMDPs but inverts its stance: where Biehl & Virgo take goals from an externally specified reward function and treat the agent’s model as independent of the true environment, here goals are prior beliefs over outcomes and no reward function exists — a difference that note itself flags. Both, however, rest on Belief State dynamics and Bayesian Filtering over hidden states, and the perceptual update here is exactly a Bayesian filter; Interpreting Dynamical Systems as Bayesian Reasoners supplies the belief-ascription side of the same picture. Subsumes Expected Utility and KL Control as limiting cases and gives principled Bayesian footing to the ad hoc softmax temperature of classical choice models and to the exploration bonuses of Artificial Curiosity.

Backlinks

Linked Pages

Artificial Curiosity

(page does not exist)

KL Control

(page does not exist)

Expected Utility

(page does not exist)

Interpreting Dynamical Systems as Bayesian Reasoners

Reference: Virgo, N., Biehl, M. & McGregor, S. (2021). Interpreting Dynamical Systems as Bayesian Reasoners. Proceedings of the 2nd International Workshop on Active Inference (IWAI 2021); published in Machine Learning and Principles and Practice of Knowledge Discovery in Databases, Communications in Computer and Information Science, vol. 1524, Springer, Cham. arXiv:2112.13523 (Earth-Life Science Institute, Tokyo Institute of Technology; Araya Inc., Tokyo; University of Sussex). URL.

Summary

When is a physical system entitled to be described as having beliefs about the world? Active inference takes as a central premise that the internal states of a physical system parametrise probability measures over states of the external world, and that these measures are the system’s beliefs — its Bayesian priors and posteriors. Virgo, Biehl and McGregor ask what would license that premise. Their question is deliberately narrower than agency: given a physical system influenced by its surroundings, under what circumstances can it be said to be performing inference, such that its internal states could be said to contain knowledge or beliefs about the outside world? Goals and actions are set aside; the system here is a reasoner, not yet an agent. The paper’s stated contribution is to make the parametrisation idea formal, and thereby to clarify the precise relationship between the interpretation level and the underlying physical level.

The approach has, in the authors’ words, “something in common with Dennett’s intentional stance”: whether a system performs inference is treated as a matter of interpretation, but interpretations are tied strongly to the underlying physical dynamics, so that whether a given interpretation is consistent becomes mathematically well-defined and has a definite answer. The machinery is category-theoretic — measurable spaces, Markov kernels, and string diagrams. A machine (Def. 1) is a state space Y, an input space S, and an update kernel γ : Y × S → P(Y). An interpretation map is a Markov kernel ψ_H : Y → P(H) sending each machine state to a belief over a hidden space H; unlike the kernels of the causal model, its job is to map states to beliefs rather than to model causal influence. A consistent Bayesian filtering interpretation (Def. 2) adds a model κ : H → P(H × S) — the reasoner’s own account of how the hidden world evolves and emits sensor values — subject to a consistency equation: sampling the reasoner’s joint belief about the next hidden state and next input and then feeding that input to the machine must agree with feeding in the sampled input and then reading off the machine’s resulting posterior belief. A Bayesian inference interpretation (Def. 3) is the special case where the hidden state does not change, so H is an unknown parameter, the model reduces to φ : H → P(S), and inputs are treated as i.i.d. samples; the authors relate this case to the category-theoretic notion of a conjugate prior, generalising earlier work on the Dirichlet case. A reasoner is a machine together with a chosen interpretation.

Two consequences are given prominence. First, an interpretation depends only on the machine’s internal dynamics — on Y, S and γ — and not at all on the external environment. A reasoner’s beliefs may therefore be consistent but incorrect; the authors note one can construct examples where the world “as the system sees it” has a different causal structure from the world as it really is, since a reasoner under an inference interpretation models its inputs as i.i.d. while their true generative structure may be an arbitrary Bayesian network. Second, interpretation is genuinely underdetermined: a given machine may admit multiple non-equivalent consistent interpretations, and the “correct” one cannot be determined from the system’s dynamics alone — the authors leave open that broader context (a designer’s intent, or selection by evolution) might distinguish them. Consistency is nonetheless a real constraint: every machine admits a trivial interpretation with respect to a parameterless model, and in the finite case non-trivial interpretations can exist only if some transitions are impossible; also in the finite case, consistency turns out to depend only on which states are reachable from which under a given input, not on the actual transition probabilities. Appendix C supplies three worked examples, including a three-state non-deterministic machine and a countably infinite machine interpreted as inferring the bias of a coin via the standard conjugate prior.

The discussion is explicit that the contribution is conceptual and that the framework is a beginning. Analogous consistency equations should exist for approximate Bayesian filtering, for smoothing and prediction, and probably even for non-Bayesian normative theories of what a system ought to represent. The most-signalled extension is to allow the machine’s state to influence the external world — adding deliberate actions combined with beliefs and possibly goals — at which point “a machine with such an interpretation might deserve the term agent instead of reasoner.” That extension is what Interpreting Systems as Solving POMDPs carries out. On the Free Energy Principle, the authors find that the ingredients of an interpretation map appear present in the Bayesian mechanics literature, but say it is not currently clear to them whether the FEP can be phrased as a consistency equation obeyed by that kernel; any such equation would differ from Defs. 2 and 3 because the FEP concerns approximate rather than exact inference and continuous time. They identify the FEP’s stationarity assumption on the true agent-environment dynamics as the key point of divergence — it is used to derive something corresponding to a model, whereas here the reasoner’s model and the ground-truth environment are separate things, so no stationarity assumption is needed. They see this conceptual separation as an advantage.

Key Ideas

Beliefs as parametrisation, made formal: active inference assumes internal states parametrise measures over external states; this paper asks when that reading is licensed, giving a mathematically definite answer for the filtering and inference cases.
Reasoners before agents: the scope is perception and representation only — no goals, no actions, no outputs. The system is a reasoner, and the authors flag agents as the next step.
A machine (Def. 1): two measurable spaces Y (states) and S (inputs) plus an update kernel γ : Y × S → P(Y); typically imagined inside a larger causal Bayesian network with hidden world states X_0, X_1, ….
Interpretation map ψ_H : Y → P(H): a Markov kernel read functionally — it sends a state y to a belief ψ_H(y) over hypotheses in H, playing a semantic role quite unlike the causal kernels of the graphical model.
Consistent Bayesian filtering interpretation (Def. 2): an H, an interpretation map ψ_H, and a model κ : H → P(H × S) satisfying a consistency equation — belief-then-input must agree with input-then-posterior. Bayes’ rule, in string-diagram form.
Consistent Bayesian inference interpretation (Def. 3): the special case of a static hidden state; H becomes an unknown parameter, the model is φ : H → P(S), inputs are modelled as i.i.d. Closely related to conjugate priors as characterised category-theoretically; generalises earlier Dirichlet-specific work.
A reasoner is a pair: a machine together with a particular choice of interpretation — being a reasoner is not an intrinsic property of the machine.
Interpretation ignores the environment: consistency depends only on Y, S, γ. Hence beliefs can be consistent but incorrect, and the world as the system models it may have a different causal structure from the world as it is (i.i.d. inputs vs. an arbitrary generative network).
Underdetermination: a machine may have several non-equivalent consistent interpretations, indistinguishable by inspecting internal dynamics alone; the choice of interpretation involves an ineliminable element of choice. External context (designer intent, evolutionary selection) is left open as a possible tiebreaker.
Consistency still bites: every machine has a trivial interpretation (parameterless model); in the finite case non-trivial interpretations require some transitions to be impossible, and consistency depends only on reachability, not on the transition probabilities themselves.
Category theory as the medium: measurable spaces, Markov kernels, and string diagrams from categorical probability, chosen to separate the physical machine cleanly from the semantic level of priors and posteriors.
Contrast with the FEP: the FEP needs a stationarity assumption on the true dynamics in order to derive a model; here the model is part of the interpretation and independent of ground truth, so no stationarity is needed. Whether the FEP admits a consistency equation at all is left open.
Not lenses: related categorical work on agency centres on lenses; the authors observe their interpretations lack the bidirectional structure of lenses, and conjecture bidirectionality would be needed for Bayesian smoothing rather than filtering.

Connections

Conceptual Contribution

Bayesian Filtering

(page does not exist)

Belief State

(page does not exist)

Interpreting Systems as Solving POMDPs

Interpreting systems as solving POMDPs: a step towards a formal understanding of agency

Reference: Biehl, M. & Virgo, N. (2022). Interpreting systems as solving POMDPs: a step towards a formal understanding of agency. Proceedings of the 3rd International Workshop on Active Inference (IWAI 2022); published in Active Inference. IWAI 2022, Communications in Computer and Information Science, vol. 1721, Springer, Cham. arXiv:2209.01619 (Cross Labs / Cross Compass, Tokyo; Earth-Life Science Institute, Tokyo Institute of Technology). URL.

Summary

When is a physical system entitled to be described as having beliefs and goals? Biehl and Virgo answer with a consistency criterion rather than a predictive one. Their starting point is Dennett’s Intentional Stance, which licenses agent-talk when it out-predicts the Physical Stance. The authors explicitly depart from this comparison-of-stances approach: assuming we already know what the system does at the physical level, they ask whether we can consistently ascribe meaning to its physical states such that those states appear to implement belief updating and decision making. This builds directly on Virgo, Biehl & McGregor’s earlier notion of interpreting a dynamical system as a Bayesian reasoner — something that takes inputs and makes inferences about a hidden variable, but that has neither goals nor actions. The present paper’s contribution is to extend consistent interpretation to cover goals and actions, and thereby to reach agents.

Three deliberate conceptual commitments make this precise. The system is taken to be a stochastic Moore machine (M, I, O, μ, ω) — internal states, inputs, outputs, a stochastic machine kernel, and a deterministic expose function; the authors note this is far from the most general system but is general enough to cover the digital computers running most present-day artificial agents. Beliefs are probability distributions updated in accordance with Bayes’ law. Goals and rationality come from POMDPs, which supply both a formal notion of goal (expected cumulative discounted reward maximisation) and of rational behaviour (the optimal policy). The technical machinery is a consistent Bayesian influenced filtering interpretation (Def. 3): a hidden state space H, an action space A, an interpretation map ψ : M → PH sending machine states to beliefs over hidden states, an action function α : M → A, and a model kernel κ : H × A → P(H × I). A stochastic Moore machine is then interpretable as a solution to a POMDP (Def. 5) if its machine part admits such an interpretation and its expose function emits the optimal action for the belief its state parameterises: ω(m) = π*(ψ(m)). An agent, on this proposal, is the pair of a system and such an interpretation.

Two features do the real work. First, machine states need only parameterise beliefs rather than be belief states — ψ need not be injective, so interpretations come with a built-in notion of coarse-graining, and the definition covers more than the canonical belief-state machines. Proposition 1 shows this is not a loosening in disguise: for any input that is subjectively possible under the interpretation, the parameterised distributions ψ(m) update exactly as the Bayesian belief-state update requires, so V̄*(m) := V*(ψ(m)) is an optimal value function on the machine’s own state space with optimal policy ω(m) = π*(ψ(m)). Such machines are optimal in the same sense as machines whose states are correctly-updated beliefs. Second, consistency is not correctness: whether a consistent interpretation exists depends only on the machine being interpreted, not on any property of its environment. An agent may hold a wrong model — its conclusions will then be wrong while its reasoning stays consistent — and its actions need only be optimal according to its own model, possibly suboptimal in the true environment. The authors flag this as a clear point of difference from the original Free Energy Principle literature, where the model is derived from the actual environment’s dynamics and so cannot differ from it. The hidden states and the rewards need not have any physical realisation at all; they exist only within the interpretation, and the machine never observes the rewards. Appendix C establishes that machines meeting Def. 5 exist by constructing one from the known optimal solution to Sondik’s POMDP.

The authors are careful about scope. The commitments are adopted because they permit precision, not because they are claimed to be realistic or the only reasonable choices; POMDPs are not the only possible formalisation of having a goal; they do not argue that all agents are rational or possess beliefs and goals. The framework requires exact Bayesian updating, so approximate inference — which the FEP does accommodate — falls outside it. The conclusion is offered as one worked example of the kind of agent definition sought, not as a settled general answer.

Key Ideas

Agency as consistent interpretation, not predictive advantage: ask whether the physical dynamics are consistent with an ascription of beliefs and goals, rather than whether the intentional stance out-predicts the physical stance.
An agent is a pair: a system together with an interpretation of it as a POMDP solution — agency is not an intrinsic property of the system alone.
Three conceptual commitments: systems are stochastic Moore machines; beliefs are Bayes-updated probability distributions; goals and rationality are supplied by POMDPs (expected discounted reward, optimal policy).
Interpretation map ψ : M → PH: sends internal machine states to beliefs over an interpretation-internal hidden state space; extended here with an action function α and a model kernel κ (the “influenced filtering” of Def. 3).
Definition 5: a stochastic Moore machine solves a POMDP when it has a consistent Bayesian influenced filtering interpretation and ω(m) = π*(ψ(m)) — it outputs the optimal policy’s action for the belief its state parameterises.
Parameterisation, not identity: machine states need only parameterise beliefs; ψ may be non-injective, so interpretation carries an automatic notion of coarse-graining of the internal state.
Proposition 1: for subjectively possible inputs, ψ(m) updates exactly per the Bayesian belief-state update; hence V*(ψ(m)) is an optimal value function on the machine’s state space — these machines are optimal in the same sense as belief-state machines.
Consistency ≠ correctness: existence of an interpretation depends only on the machine, not the environment. An agent can have a wrong model and act optimally only relative to it — a deliberate contrast with the FEP, where the model derives from actual environment dynamics.
Hidden states and rewards are interpretation-only: they need exist only mathematically, not physically; the machine never observes rewards. Even an output-less machine can have “actions” confined to the interpretation.
Existence is demonstrated, not assumed: Appendix C builds a qualifying stochastic Moore machine from the known optimal solution to Sondik’s POMDP.
Stated limits: exact Bayesian updating only (approximate inference excluded); POMDPs are not the only formalisation of goals; the commitments are chosen for precision rather than realism.

Connections

Conceptual Contribution

POMDP

(page does not exist)

Life as We Know It

Life as we know it

Reference: Friston, K. (2013). Life as we know it. Journal of the Royal Society Interface, 10(86):20130475. (The Wellcome Trust Centre for Neuroimaging, Institute of Neurology, University College London.) DOI: 10.1098/rsif.2013.0475. PMID 23825119; PMCID PMC3730701. URL.

Summary

This is the anchor statement of the Free Energy Principle: the paper offers what Friston calls a heuristic proof — accompanied by simulations of a primordial soup — that life, understood as biological self-organisation, is “an inevitable and emergent property of any (ergodic) random dynamical system that possesses a Markov Blanket.” The setting is a Random Dynamical System: states flow under a deterministic vector field perturbed by random fluctuations, described by a stochastic differential equation. The ergodic assumption is what buys the interpretive leverage — over sufficient time the system’s trajectory visits states in proportion to an invariant ergodic density, so how long a system spends in a state can be read as how probable that state is for that system. Under ergodicity the long-term average of surprise (the negative log probability of a state) is the entropy of the ergodic density, and a system that persists is, definitionally, one whose states remain in a small, characteristic set.

The argument then turns on a statistical rather than a thermodynamic boundary. If coupling among an ensemble of subsystems is mediated by short-range forces, the states of remote subsystems must be conditionally independent; these independencies induce a Markov blanket separating internal from external states. The blanket itself splits by the direction of influence, giving the fourfold partition the paper is famous for: external states ψ (states of the world that cause sensory states and depend on action), sensory states s (a probabilistic mapping from external states and action), active states a (which depend on sensory and internal states), and internal states λ (which cause action and depend on sensory states). Internal and external states are thereby d-separated: they can influence each other only through the blanket, and so each is condemned to know the other only by inference. Applying a Helmholtz Decomposition to the flow of the ergodic density (whose evolution obeys the Fokker-Planck Equation) separates it into a gradient descent on the Gibbs energy G = −ln p(ψ,s,a,λ) plus a divergence-free solenoidal component. Read against that decomposition, the flow of internal states appears to perform a gradient descent on a variational free energy — a functional of an arbitrary variational density q(ψ|λ) parametrised by the internal states, and a quantity that upper-bounds surprise. Because this is exactly the quantity optimised in variational Bayesian inference, the internal states appear to encode posterior beliefs about external causes, while the active states appear to change the world so that sensations conform to those beliefs. Internal states plus their blanket therefore “appear to engage in active Bayesian inference”: to model, and act on, their world so as to preserve their functional and structural integrity — homoeostasis and a simple form of autopoiesis. Friston is consistently careful to use as-if language: the inference is an appearance forced by the geometry of the dynamics, not an explicit representational computation posited inside the system.

The primordial soup simulation illustrates the emergence rather than proving it. An ensemble of 128 elemental subsystems is given two coupled sets of dynamics: Newtonian motion under a strong inverse-square repulsive force and a weaker attractive force that depends on the subsystems’ electrochemical states; and electrochemical dynamics chosen to have a Lorenz attractor, coupled only to spatial neighbours through an adjacency matrix. Heterogeneity is introduced by randomly selecting a third of the subsystems and preventing them from electrochemically influencing others (zeroing the corresponding column of the adjacency matrix), making them “functionally closed.” From random initial conditions the ensemble self-organises: a cluster of internal subsystems forms, enclosed by a membrane-like layer of sensory and active subsystems — a Markov blanket that was nowhere specified in the equations of motion. The internal electrochemical states come to predict the external structural states, which is precisely the appearance of a generative model.

Key Ideas

The thesis: any ergodic random dynamical system possessing a Markov blanket will appear to minimise a variational free energy functional of its blanket states, hence appear to engage in active Bayesian inference. Life is presented as an inevitable consequence of that setup, not an additional assumption.
Ergodicity as the interpretive bridge: if time-averages converge, state occupancy is probability. A system that persists occupies a small set of states, so persistence = low long-term average surprise = low entropy of the ergodic density.
Short-range forces ⇒ conditional independence ⇒ a Markov blanket. The boundary is statistical, derived from the sparsity of coupling, not stipulated as a physical membrane.
The fourfold partition: external ψ, sensory s, active a, internal λ. Sensory states are caused by external states; active states are caused by internal states; internal and external states are d-separated by the blanket {s, a} and can reach each other only through it.
Helmholtz decomposition of the flow: the flow of the ergodic density splits into a gradient descent on Gibbs energy G(ψ,s,a,λ) = −ln p(ψ,s,a,λ) and a solenoidal (divergence-free) circulation on its isocontours.
Free energy bounds surprise: F(s,a,λ) is a functional of a variational density q(ψ|λ) parametrised by internal states; by Gibbs’ inequality the gap between free energy and surprise is a non-negative divergence, so minimising F both bounds surprise and drives q toward the true posterior over external causes.
Active Inference has two limbs: internal states minimise free energy by changing beliefs (perception); active states minimise it by changing the world so that sensory states conform to those beliefs (action). Together they place an upper bound on the dispersion of the system’s states.
The model is not chosen — it is the environment’s own density. The generative model is implicit in p(ψ,s,a,λ), the invariant density of the very system being described. There is no free hand to posit a mistaken model.
“Appear to” is load-bearing. Friston does not claim systems do Bayesian inference; he claims their dynamics are indistinguishable from a system that does. The inferential vocabulary is a description that the mathematics licenses, not a mechanism it postulates.
Primordial soup simulation: 128 subsystems, Newtonian mechanics (inverse-square repulsion, weaker electrochemically-dependent attraction) plus Lorenz-attractor electrochemical dynamics coupled through local adjacency; a randomly chosen third made functionally closed. A Markov blanket — internal core, sensory/active membrane — self-organises out of random initial conditions.
Autopoiesis without teleology: homoeostasis and structural integrity fall out as consequences of the dynamics, rather than being written in as objectives.

Connections

Conceptual Contribution

Claim: Any ergodic random dynamical system that possesses a Markov blanket will appear to minimise a variational free energy functional of its blanket states — an upper bound on the surprise of those states — and therefore will appear to engage in active Bayesian inference: to model, and act upon, its world so as to preserve its structural and functional integrity. Biological self-organisation, on this account, is not an extra ingredient added to physics but an inevitable appearance of it wherever a statistical boundary exists.
Mechanism: Model the ensemble as a stochastic differential equation with random fluctuations and assume ergodicity, so that the system has an invariant density whose evolution obeys the Fokker–Planck equation. Observe that short-range coupling makes remote subsystems conditionally independent, inducing a Markov blanket; partition states into external ψ, sensory s, active a and internal λ, with {s, a} d-separating λ from ψ. Apply a Helmholtz decomposition to the flow, splitting it into a gradient descent on Gibbs energy G = −ln p(ψ,s,a,λ) and a solenoidal component. Recast the internal-state flow as gradient descent on a free energy F(s,a,λ) defined over a variational density q(ψ|λ) parametrised by internal states; Gibbs’ inequality makes F an upper bound on surprise, whose minimisation drives q toward the posterior p(ψ|s,a). Perception is then the internal-state limb of this descent and action the active-state limb. Illustrate with 128 coupled subsystems (Newtonian motion + Lorenz electrochemical dynamics, a third of them functionally closed), in which a Markov blanket with a membrane-like sensory/active boundary emerges spontaneously.
Concepts introduced/used: Markov Blanket, Free Energy Principle, Active Inference, Variational Free Energy, Surprisal, Ergodicity, Random Dynamical System, Fokker-Planck Equation, Helmholtz Decomposition, Generative Model, Autopoiesis, Bayesian Filtering
Stance: formal theory / heuristic proof with illustrative simulation (no empirical study; the simulation demonstrates emergence, it does not establish the general claim)
Relates to: Provides the canonical formulation against which Interpreting Systems as Solving POMDPs positions itself. Biehl & Virgo require only that an ascription of beliefs and goals be consistent with a system’s dynamics — an agent may hold an outright wrong model of its environment and still qualify, because the interpretation’s hidden states and rewards need have no physical realisation and existence of the interpretation depends on the machine alone. Here the opposite holds: the generative model is not chosen by an interpreter but derived from the actual environment’s dynamics — it simply is the invariant density p(ψ,s,a,λ) of the coupled system — so it cannot come apart from the world it models, and the question of a wrong model does not arise. Consistency versus correctness is exactly the axis of difference; a further one is that Biehl & Virgo demand exact Bayesian updating whereas the FEP is built around approximate (variational) inference. The Markov blanket construction inherits its conditional-independence machinery from Pearl (Causality - Pearl, Causal Diagrams for Empirical Research - Pearl). The account of persistence and boundary formation answers, in dynamical-systems terms, the self-maintenance questions posed by Theory of Self-Reproducing Automata, and it offers a reading of agent-talk that neither asserts explicit internal representation — cf. Intelligence Without Representation — nor merely licenses it pragmatically as True Believers - The Intentional Strategy and Why It Works does; the inferential description is instead forced by the geometry of the flow.

Free Energy Principle

(page does not exist)

Active Inference

(page does not exist)

Variational Message Passing

(page does not exist)

Precision

(page does not exist)

Value of Information

(page does not exist)

Mutual Information

I(X;Y) = H(X) - H(X|Y) = H(Y) - H(Y|X) — the reduction in uncertainty about X given knowledge of Y (and symmetrically). One of Shannon’s foundational quantities (A Mathematical Theory of Communication); the operational basis of Channel Capacity (C = max_{p(x)} I(X;Y)). Used in MAS as a measure of causal communication influence in emergent-language metrics (On the Pitfalls of Measuring Emergent Communication).

In this vault

Infomax Principle

(page does not exist)

Salience

(page does not exist)

Bayesian Surprise

(page does not exist)

Extrinsic Value

(page does not exist)

Epistemic Value

(page does not exist)

Expected Free Energy

(page does not exist)

Bayesian Brain

(page does not exist)

Planning as Inference

(page does not exist)

Reward Prediction Error

(page does not exist)

Bounded Rationality

Simon’s (1957) departure from classical game theory’s perfectly rational utility maximiser: real agents face computational and informational limits and adopt satisficing strategies rather than optimising. In multi-agent settings bounded rationality is often operationalised by capping recursion depth in Theory of Mind (level-ℓ reasoning), bounding planning horizons, or imposing noisy best-response models. The decision-policy solver in Pact - A Choreographic Language for Agentic Ecosystems is bounded-rational in this sense: agents reason about counterparts to a finite depth, with naïve priors at the bottom.

In this vault

Exploration-Exploitation Dilemma

(page does not exist)

Variational Free Energy

(page does not exist)

Markov Processes

Stochastic processes with the memoryless property: future states depend only on the current state; Chomsky’s finite-state grammars are Markov sources, which he argued were inadequate for natural language.

In this vault

Three Models for the Description of Language

Active inference and epistemic value

Summary

Key Ideas

Connections

Conceptual Contribution

Tags

Backlinks