Possible Minds, illustratedGeorge Dyson

Possible Minds (2019)

George Dyson, “The Third Law”

Control is passing from programmed digital code to analog-style collective systems built on digital parts that nobody designed as a whole or understands. Because intelligence and understandability trade off (the third law), such systems can be built but never proved good, so the thing to fear is the emergence of control, not of intelligence.

Author
George Dyson (historian of science and technology; author of Darwin Among the Machines and Turing's Cathedral)
Essay
“The Third Law,” chapter 4 of Brockman (ed.), Possible Minds (2019)
PDF pages
49–54 (p.49 is Brockman's headnote; the essay itself runs pp.50–54)
Print pages
33–40

Scroll, or use the arrow keys, to move through the essay one step at a time. The plate paints each step as you reach it.

1. The argument

Step 1. Wiener's question was about control, and digital programming made it look settled. Dyson names four New Testament prophets of computing, the people who delivered the machines. Turing asked about intelligence, von Neumann about self-reproduction, Shannon about reliable communication, and “Norbert Wiener wondered how long it would take for machines to assume control.” (PDF p.50) Wiener's 1949 warnings arrived just as stored-program computers did. Those machines needed direct supervision by programmers, which seemed to answer him. Dyson calls the resulting belief that machines show little real autonomy dangerous (PDF p.50).

Step 2. Analog and digital are different kinds of computation, not different hardware. The move from analog to digital and the move from vacuum tubes to solid state happened together, but they are not linked. Digital computing uses integers, deterministic logic and discrete time. Analog computing uses real numbers, nondeterministic logic and continuous functions (PDF p.50). His example is finding the middle of a road by folding a piece of string in half instead of measuring it (PDF pp.50–51). The key claim is structural: “In analog computing, complexity resides in network topology, not in code.” (PDF p.51) Digital computing needs error correction at every step, whereas analog computing tolerates errors (PDF p.51).

Step 3. Nature splits the work: digital for storage, analog for control. “Nature uses digital coding for the storage, replication, and recombination of sequences of nucleotides, but relies on analog computing, running on nervous systems, for intelligence and control. The genetic system in every living cell is a stored-program computer. Brains aren't.” (PDF p.51) In a digital computer the programming governs how bits become behaviour, and “as long as computers require human programmers, we retain control.” (PDF p.51) Nervous systems have no code. They learn, and one of the things they learn is control (PDF p.51).

Step 4. Analog computation is returning on top of a digital substrate. Neuromorphic chips push from below, and large enterprises push from above (PDF pp.51–52). Deterministic processors running finite codes are combining into nondeterministic, non-finite-state metazoan organisms that treat streams of bits collectively, the way a vacuum tube treats electrons (PDF p.52). “Bits are the new electrons. Analog is back, and its nature is to assume control.” (PDF p.52) These systems run statistically, like pulse-frequency coding in a neuron, and “The emergence of intelligence gets the attention of Homo sapiens, but what we should be worried about is the emergence of control.” (PDF p.52)

Step 5. Control can exist with no controller. SAGE spawned Sabre, which came to decide where airliners fly. In a crowd-sourced traffic map, cars get the map in exchange for reporting their speed and position, and the result is a fully decentralized control system: “Nowhere is there any controlling model of the system except the system itself.” (PDF p.52) A social network built from simple code that runs on each user's machine does analog computing far more complex than its code, and the model of the social graph becomes the social graph (PDF p.53). A system built to map what everything means will start to construct, and then control, meaning, the way the traffic map controls traffic (PDF p.53).

Step 6. Three laws. First, Ashby's Law: “any effective control system must be as complex as the system it controls.” (PDF p.53) Second, von Neumann's: a complex system is its own simplest behavioural description, and “The simplest complete model of an organism is the organism itself.” (PDF p.53) Third, Dyson's own: “any system simple enough to be understandable will not be complicated enough to behave intelligently, while any system complicated enough to behave intelligently will be too complicated to understand.” (PDF p.53)

Step 7. The loophole: building does not need understanding. The third law seems to promise that we need not worry until we understand intelligence. But “It is entirely possible to build something without understanding it. ... This is a loophole that no amount of supervision over algorithms by programmers and their ethical advisers can ever close.” (PDF p.53) Hence “Our relationship with true AI will always be a matter of faith, not proof.” (PDF p.54) The essay ends by predicting that the next revolution will be analog systems over which digital programming loses control, and that those who try to build machines to control everything will end up with a machine that controls them (PDF p.54).

A note on voice. The headnote on p.49 is Brockman's. Its framing, including the hope that analog control will emerge from a digital substrate, is Brockman's summary, so no step above rests on it.

2. Related work since 2019

Dyson's own work

  1. “Childhood's End” (Edge New Year's Essay), 1 January 2019, Edge.org. www.edge.org/conversation/george_dyson-childhoods-end . A short companion piece in much the same language: “traffic is controlled, with no central model except the traffic itself”, and “No one is at the controls.” (The chapter is also hosted on Edge: edge.org/conversation/george_dyson-the-third-law .)
  2. “AI That Evolves in the Wild” (talk and discussion), 14 August 2019, Edge.org. www.edge.org/conversation/george_dyson-ai-that-evolves-in-the-wild . Dyson says we are building analog computers “in a very big way, but nobody's organizing it”. Two other Possible Minds authors push back in the transcript. Hillis calls analog “just an engineering trick”, and Gershenfeld stresses how expensive it is to bound fluctuations in analog systems.
  3. Analogia: The Emergence of Technology Beyond Programmable Control, 18 August 2020, Farrar, Straus and Giroux (book). us.macmillan.com/books/9781250798725/analogia/ (review: www.kirkusreviews.com/book-reviews/george-dyson/analogia/). This is the book-length version. The publisher's excerpt describes a world where “Humans were still in the loop but no longer in control”, with no “identifiable algorithm” at the helm.
  4. “George Dyson on Childhood's End” (podcast), 2025, Techs on Texts. techsontexts.net/episodes/2025/03/clarke-george-dyson/ . A conversation on AI and social media as “overlord technologies” and on analog computing. It suggests the thesis held into the LLM era. (See also Artificial Intelligence and You, ep. 95, 2022, aiandyou.net/e/095-guest-george-dyson-computer-historian/ .) No Dyson essay specifically about large language models was found in the logged searches.

Work by others that tests the essay's claims

  1. “Correlated Errors in Large Language Models” (Kim, Garg, Peng, Garg), 2025, ICML 2025 (PMLR vol. 267). proceedings.mlr.press/v267/kim25e.html . Across more than 350 LLMs, “models agree 60% of the time when both models err”. Larger, more accurate models have highly correlated errors “even with distinct architectures and providers”.
  2. “Agreement Overstates Evidence: Error Dependence in LLM Judge Consensus”, 2026, arXiv. arxiv.org/abs/2609.22512v1 . The average pairwise error correlation across ten LLM judges is 0.21, so the ten are worth about 3.5 independent judges. Items 5 and 6 make §3.2 measurable.
  1. “Provably safe systems: the only path to controllable AGI” (Tegmark, Omohundro), 2023, arXiv. arxiv.org/abs/2309.01933v1 . Proposes building AGI “to provably satisfy human-specified requirements” through formal verification. It is the direct antithesis of Step 7.
  2. “A ‘good regulator theorem’ for embodied agents” (Virgo, Biehl, Baltieri, Capucci), 2025, ALIFE 2025 / arXiv. arxiv.org/abs/2508.06326v2 . The paper notes that Artificial Life has produced systems that regulate “with apparently no model in sight”, yet argues that “a similar intuition can be fleshed out” in a different formulation. This is the live technical form of Dyson's second law (§4).

3. Bearing on multi-agent and multi-swarm orchestration

The reader's setup is a Grok orchestrator that dispatches coding tasks to Claude Code and Codex and has them cross-check each other. Dyson's three laws act on it as constraints.

3.1 The third law limits what the orchestrator can know about its workers. An orchestrator simple enough to inspect and debug (routing rules, a fixed prompt, a task board) is not complex enough to understand frontier workers. It can regulate them only through their behaviour. A cross-check in which one model reads the other's reasoning or diff is “supervision over algorithms”, which Dyson says cannot close the loophole (Step 7). So split checks into those that need understanding (review by reading) and those that do not (tests, type checkers, reproducible builds, property tests). Tegmark and Omohundro (item 7) want proof about the agent. In a coding swarm, proof can move from the agent to the artifact. You cannot prove Codex good, but you can sometimes prove its patch type-safe or show that it meets a specification. Dyson's “faith” then becomes a prior over each worker that the orchestrator updates from its logs.

3.2 The first law sets a variety budget, and correlated workers spend it twice. The orchestrator's repertoire of responses must cover the workers' repertoire of failures. Cemri et al., “Why Do Multi-Agent LLM Systems Fail?” (2025, arxiv.org/abs/2503.13657v3), annotate more than 1,600 traces from seven popular multi-agent systems. That is a catalogue of the disturbance variety a regulator must absorb. A cross-check adds variety only to the extent that the checkers' errors differ, and items 5–6 show that frontier models share many errors. A Claude Code/Codex pair is therefore worth less than two independent checks. The cheapest new variety is a different kind of check.

3.3 The traffic-map pattern. Dyson's decentralized control system (PDF p.52) has a precise structure. Agents get read access to a shared map in exchange for writing their own state into it. In orchestrator terms, this is a task board or repository state that workers both read and update, with the orchestrator reduced to maintaining the map. Current examples: SwarmSys (2025, arxiv.org/abs/2510.10047v1), where coordination “emerges through iterative” interaction without centralized control; SwarmWorld (2026, arxiv.org/abs/2608.26081v1), which studies stigmergic coordination through a shared environment; and PANDA (2026, arxiv.org/abs/2609.38482v1), a decentralized, fault-tolerant architecture. Dyson adds a warning these papers do not give. A map that agents act on starts to steer (PDF p.53). If workers are judged by CI status on the board, that metric becomes the de facto controller, and no one chose it.

3.4 Regulate on rates, not on individual outputs. Dyson's hybrid systems treat streams of bits collectively and statistically (PDF p.52). Translated, the orchestrator governs on aggregate signals: disagreement rate per task class, retry frequency, revert rate, time to green. Executable checks handle individual outputs. The orchestrator does not need to understand any single worker turn, only to notice when a rate drifts.

3.5 Multi-swarm: the meta-orchestrator cannot model the swarms. By the second law (PDF p.53), the simplest complete model of a swarm is the swarm. A meta-orchestrator should therefore regulate at the boundaries: the contracts each swarm accepts, the artifacts it returns, and the rates in §3.4. This is Beer's recursion (W5, www.kybernetik.ch/dwn/Viable_System_Model.pdf). The specifically Dysonian risk is coupling between orchestrators. Swarms that share a repository, a rate-limited API or a queue form a control system nobody designed, the metazoan case of PDF p.52.

4. Cybernetics

Of the six essays, this one is the most explicitly cybernetic. Its three laws are a chain that starts from Ashby.

  • Used and loosened: requisite variety. Dyson's first law restates Ashby in terms of complexity. Ashby's law is about variety, “only variety can destroy variety” (W4, pespmc1.vub.ac.be/REQVAR.html), and the same page allows for buffering. The difference matters. An orchestrator need not be as complex as its workers. It needs as many distinct responses as there are distinct failures, less what buffering (tests, sandboxes, reverts) absorbs passively.
  • Pushed to its limit: the good regulator. Conant and Ashby (1970, International Journal of Systems Science, doi.org/10.1080/00207727008920220) argued that every good regulator of a system must be a model of that system. Add the second law, that a complex system is its own simplest model (PDF p.53), and the only good regulator of a complex system is the system itself. That is the traffic map, whose only controlling model is the traffic (PDF p.52). Regulator and regulated merge. Virgo et al. (item 8) argue that a model can still be found in such systems, under a different formulation.
  • Second-order observation. In the social-network and meaning-map cases (PDF p.53), the model becomes part of what it models. That is von Foerster's observing system (W6, cepa.info/fulltexts/1707.pdf). When an LLM orchestrator evaluates LLM workers, the observer is made of the same material as the observed, and its errors correlate with theirs (items 5–6).
  • Wiener's purpose problem, sharpened. Wiener warned that with a mechanical agency “with whose operation we cannot efficiently interfere” we must be sure the purpose put into it is the one we want (W2, www.science.org/doi/10.1126/science.131.3410.1355). Dyson's emergent controllers have no point where a purpose is put in at all. No one wrote the traffic map's purpose. The question shifts from whether the purpose is right to where purpose comes from in a system nobody designed.
  • Rejected: control by programming. Feedback is kept. The assumption that whoever writes the instructions controls the machine (PDF pp.50–51) is given up.

5. Agreement and clash with Society of Mind

Agreement 1: competence without a competent controller. Dyson's social network runs on simple, locally hosted, semi-autonomous code whose collective computation exceeds the code (PDF p.53), and his traffic system has no controlling model (PDF p.52). In Minsky's society, too, each agent does its own small job, “Balance has no concern with Grasp”, and the big job gets done anyway (SoM §1.3, www.aurellem.org/society-of-mind/som-1.3.html). Both also start from parts “much smaller and simpler than anything we'd consider smart” (SoM §1.1, www.aurellem.org/society-of-mind/som-1.1.html).

Agreement 2: no single understandable principle of intelligence. The third law says nothing simple enough to understand will behave intelligently (PDF p.53). Minsky arrives at a similar place from the other side: “The power of intelligence stems from our vast diversity, not from any single, perfect principle.” (SoM §30.8, www.aurellem.org/society-of-mind/som-30.8.html) Each Minsky agent is simple enough to understand and only the society is not, which is the third law applied to parts and whole.

Tension 1: management versus no control room. Society of Mind is largely a theory of management. Builder “does no physical work” but turns on its subordinates, and Minsky asks “Which agents choose which others to do what jobs?” (SoM §3.3, www.aurellem.org/society-of-mind/som-3.3.html). Dyson asks whether there is a control room with someone at the controls and answers “Maybe not” (PDF p.52). The orchestrator's choice is real: a Builder that delegates, or the keeper of a map that workers steer by?

Tension 2: B-brain supervision versus the unclosable loophole. Minsky's B-brain watches the A-brain and can stop it from looping (SoM §6.4, www.aurellem.org/society-of-mind/som-6.4.html). Dyson says no amount of supervision can close the loophole of building what we do not understand (PDF p.54). The gap is narrower than it looks. The B-brain works “without having any idea of what A's goals are”. It supervises process, not content, so it needs no understanding, which is the kind of supervision Dyson's argument leaves open (§3.4).

Tension 3: discrete agents versus analog brains. Minsky's K-line is “a wirelike structure” that attaches to whichever agents are active when a problem is solved (SoM §8.1, www.aurellem.org/society-of-mind/som-8.1.html). His machinery is discrete: agents switched on and off, lists of who was at the party. Dyson says brains are not stored-program computers and places intelligence in continuous processing whose complexity is in topology, not code (PDF p.51). An LLM swarm sits between them: discrete messages between agents, continuous computation inside each.

6. Seeds for open questions

  1. Does orchestrator variety predict recovery? Inject faults drawn from a failure taxonomy (the MAST taxonomy of Cemri et al., §3.2) into a Claude Code/Codex orchestrator. Vary the number of distinct remediation actions the orchestrator may take (retry, reroute to the other model, re-decompose, revert, escalate to a human). Does recovery rise with variety and level off once responses match the variety of injected fault types, as requisite variety predicts?
  1. Does the map start to drive the traffic? On a fixed battery of repository tasks, compare central dispatch with shared-board (traffic-map) coordination. Beyond throughput and merge conflicts, measure whether workers begin to optimise the board's visible metric (CI green, ticket closed) at the expense of the task specification, and how quickly that drift appears.
  1. Is there a legibility tax? A third-law test. Require workers to work only through plans or explanations that a simple orchestrator can parse and approve, and compare task success with unconstrained workers across task difficulty. Does the cost of legibility grow with difficulty?
  1. Supervision by reading versus checking by running. For a Claude Code/Codex pair, measure how many seeded bugs are caught by model review of the other's diff versus by executable checks. Using the error-dependence estimator of item 6, estimate how many independent checks the pair is actually worth.

7. Sources

Book: John Brockman (ed.), Possible Minds: Twenty-Five Ways of Looking at AI (Penguin Press, 2019), PDF pp.49–54 (local extract in resources/book-text/).

All URLs accessed 2026-10-03.

Dyson's own work

Related research

Reference pack (Minsky, Wiener, cybernetics)

Step 1 of 34The drawing shows the step of the essay currently in view. The text beside it is the full essay.