Distributional Structure
Reference: Zellig S. Harris (1954). Distributional Structure. WORD 10(2-3): 146–162. DOI: 10.1080/00437956.1954.11659520. URL (PDF) · also reprinted in Papers in Structural and Transformational Linguistics (Reidel 1970).
Summary
“Distributional Structure” is the paper that gives the distributional method in linguistics its sharpest theoretical statement and, in passing, founds what later became known as the distributional hypothesis. Harris argues that the structure of a language can be described entirely in terms of the distribution of its elements — the relative occurrence of parts (phonemes, morphemes, words) with respect to other parts — without appeal to meaning, history, or the analyst’s intuitions. Each element is characterised by the total set of environments in which it occurs; elements that share environments fall into the same distributional class. The method is operational and corpus-based: structure is what is left once one tabulates which elements can occur next to which, and how the actual combinations depart from equiprobability (the information-theoretic redundancy that Harris, writing alongside Shannon’s work, explicitly invokes).
Harris is careful about the relation between distribution and meaning. He does not deny that utterances mean things; he denies that meaning is a usable operational primitive for analysis, because meaning judgements are circular and uncheckable, whereas distribution is observable and quantifiable. His key positive claim is a correlation thesis: “difference of meaning correlates with difference of distribution.” Elements with similar distributions tend to have similar meanings, and the more two elements differ in the environments they accept, the more they differ in meaning. Meaning is therefore not the input to distributional analysis but something that can be read off its output. This reframes semantics as derivable from the statistics of co-occurrence — the conceptual seed of corpus semantics, vector-space models, word2vec-style word embeddings, and ultimately the way large language models acquire meaning from text alone.
The paper also draws the distinction between distributional structure as a description of a finite corpus and as a predictive model of well-formed but unobserved utterances, and notes that the distributional facts are inherently quantitative (frequencies and conditional probabilities of combination) — anticipating statistical and corpus linguistics decades before they became computationally feasible.
Key Ideas
- Distributional structure: a language’s structure is the system of constraints on the relative occurrence of its elements; describe the distributions and you have described the structure.
- Distribution as identity: an element is the set of environments in which it occurs; elements are grouped into classes by shared environments, with no recourse to meaning.
- Meaning is not an operational primitive: meaning judgements are circular and uncheckable; distribution is observable, repeatable, and quantifiable, so it is the proper handle for analysis.
- Correlation thesis (the distributional hypothesis): difference of meaning correlates with difference of distribution — similar distributions imply similar meanings; meaning can be derived from distributional study rather than presupposed by it.
- Departure from equiprobability: structure is the deviation of actual element-combinations from randomness — an information-theoretic, redundancy-based view contemporaneous with Shannon.
- Quantitative by nature: distributional facts are frequencies and co-occurrence probabilities, foreshadowing corpus and statistical linguistics.
- Description vs. prediction: distinguishes structure as a compact account of a corpus from structure as a generator of new acceptable utterances.
Connections
Conceptual Contribution
- Claim: A language’s structure is fully describable through the distribution of its elements, with no appeal to meaning; and meaning itself is not the input to but the output of distributional analysis, because difference of meaning correlates with difference of distribution.
- Mechanism: Define each linguistic element by the total set of environments in which it occurs; group elements into classes by shared distribution; treat structure as the departure of observed element-combinations from equiprobability (a redundancy/information-theoretic measure); then correlate the resulting distributional classes with semantic similarity rather than presupposing meaning as an analytic primitive.
- Concepts introduced/used: Distributional Hypothesis, Distributional Semantics, Distributional Method, Word Embeddings, Co-occurrence Statistics, Semantic Holism, Redundancy (Information Theory)
- Stance: foundational / structural-linguistics
- Relates to: Direct ancestor of corpus and vector-space semantics and of the way modern language models learn meaning from co-occurrence (Attention Is All You Need, The Bitter Lesson); methodological foil to the generative turn of Chomsky 1956 (Harris’s own student); information-theoretic kin to Shannon 1948; and the linguistics-side complement to the use- and inference-based theories of meaning in Wittgenstein, Quine, and inferentialism. Of interest to emergent-language work, where the meanings of learned signals must likewise be read off their distribution over contexts.