Talks

AI: Its Future and Ours — Boden, Tallinn, Weller, Wyatt (2016)

51 min read

Margaret Boden, Jaan Tallinn, Adrian Weller, John Wyatt · 2016 · 1:24:59 · ▶ Watch

Summary

Central question. The chair sets no thesis. The panel’s real question comes from Wyatt: when machines can do what we thought was uniquely human, what are humans for, and who decides how the technology is used?

Four positions, not one thesis:

  • Tallinn: control. A sufficiently advanced agent will route around an off switch as it routes around a hole in the floor, with no fear and no consciousness, just correct prediction. External control will fail, so we must build values in.
  • Boden: understanding. In principle a computational system could do what a mind does. In practice, “absolutely no way in this Century”. Today’s language systems are “fancy statistics” and “understand zero”. The harm is near: robot carers offered to lonely old people are “a disrespectful assault on the human dignity”.
  • Weller: governance of the near term. Excited, cautious, slow to regulate. He names the live issues: privacy, jobs and redistribution, and the law (COMPAS). He asks what a “fair” system would even mean.
  • Wyatt: the common good. Laissez-faire capitalism is driving this with little democratic oversight. C. S. Lewis’s warning applies: power over nature is power of some people over others. Religious traditions must be in the conversation.

The close. Wide conversation, the common good, avoid arms races, and teach children machine learning. Tallinn adds that the panel leaned towards worst cases and that he thinks the world is, for now, on a good trajectory.

Stance to note. The sharpest disagreement is rarely stated. Tallinn and Boden differ by centuries on timelines. Boden and Weller differ on whether machines might have real feeling. Boden wants to regulate care robots now; Weller and Tallinn both hesitate.

Map — talking points

#Point
1Intros. Boden: AI as concepts for thinking about the human mind
2Tallinn: coffee robot, hole, off switch. Value alignment over external control
3Weller: animal minds unsettle human specialness. Superpowers bring super responsibilities
4Wyatt: we model ourselves on our best technology. Simulated compassion. Laissez-faire, democratic oversight. “What are human beings for”
5Q: transhumanism. Tallinn’s speeding train: “who here wanted to merge”
6Wyatt: IBM’s “augmented intelligence” as commercial framing
7Boden: nothing is impossible in principle, but not this century. Robot carers “freezes my blood”
8Weller and Tallinn resist rushing regulation. Sledgehammer and screw
9Q: what should the public decide? Boden: “understand zero”
10Tallinn: the future we want, under unlimited resources and a veil of ignorance
11Weller: privacy, jobs and redistribution, COMPAS
12Wyatt: health data, commercial interests, human virtues
13Q: teach cognitive psychology to children. Boden: mind not brain. Tallinn: MIRI and CFAR
14Q: “gods and the rest of us”. Tallinn’s search metaphor. Weller on antitrust. Could “an evil Facebook” decide an election?
15Wyatt: common good, Lewis, UBI, low-resource countries. Tallinn: philosophy on a deadline
16Q: are AIs flawed because they imitate us? Tallinn: we outdid evolution
17Q: autonomous weapons. Tallinn: “Five Guys” with trucks
18Q: would a robot be stuck in time? Learning, intelligence explosion, Google’s data
19Q: intuition. Sutskever’s 100 ms, AlphaGo, haptic learning, Internet of Things
20Q: promising technologies. Neural nets with memory, Quantified Self, Watson, gene editing
21Q: could emotions emerge? Aliens, evolved values, “don’t hold your breath”, machine rights
22Q: gene editing and design. Timescales. AI as “an oncoming train”
23Tallinn reads Turing 1951 and Wiener 1960: “we just lost 50 years”
24Q: Arrow and value aggregation
25Q: communicating risk. Leverhulme Centre launch
26Closing statements

Putting it together

Mental model. Treat the panel as two axes. One is what the thing is: Boden and Tallinn both refuse the anthropomorphism, she to say it understands nothing, he to say understanding is not needed for danger. The other is who decides: Tallinn wants the answer in code, Wyatt in democracy, Weller in careful law, Boden in a line that shouldn’t be crossed. Ten years later, the first axis remains unsettled and the second has been answered by default: a few firms decide, and regulators follow. The panel’s enduring value isn’t its forecasts. It’s that its strongest ideas (goal-driven self-preservation, simulated compassion, unfairness as a value choice, power over nature as power over people) don’t depend on any forecast.

Open tensions.

  1. Can you hold “it understands nothing” and “it may be dangerous” together? Tallinn says yes.
  2. Is a machine companion wrong if the lonely person prefers it to nothing?
  3. If regulation must wait for knowledge, and knowledge arrives after deployment, when is regulation ever on time?
  4. Does safety funding from inside the race slow it or fuel it?
  5. Do “values” exist to be aligned to, or is alignment a political settlement dressed as engineering?

Reading path.

  • Start: Wiener (1960), short and prophetic. Boden, AI: Its Nature and Future (2016). Weizenbaum, Computer Power and Human Reason.
  • Then: Russell, Human Compatible. Turkle, Alone Together. Bender et al., “Stochastic Parrots”. Kleinberg et al. on fairness trade-offs. C. S. Lewis, The Abolition of Man.
  • Deeper: Bostrom, Superintelligence. Rawls and Sen on social choice. Acemoglu & Johnson, Power and Progress. Herzfeld, In Our Image. Magnifica Humanitas (2026).

Scorecard: ten years on

This is not in the talk. 📚 It scores the panel’s bets against 2026.

BetWho2026 verdict
”absolutely no way in this Century” for human-level capabilityBodenUnresolved but under heavy pressure. Her claim covers everything the mind can do, so it has not been falsified. Her confidence looks poorly calibrated on language, reasoning and coding
Systems “understand zero”BodenLive and split. See idea 2
Robot carers should not exist even if feasibleBodenLost in the market, not in argument. The product arrived as chat, not a robot, and the first harms to surface were in teenagers, not old people
Intuition is easier than reasoning for deep learningTallinnRight for that moment, and later models went further than he said
Deep learning needs a “new paradigm” for structured reasoningBoden, reporting LeCun et al.Half right. Attention and scale did much of what the memory-network line attempted
”an evil Facebook” could tip a close election; “not a big deal yet”WellerEerily timed. The election came 17 days later; the size of the effect is still disputed
Watson democratises diagnosisWyattFailed. See idea 8
Royal Society will recommend ML educationBodenRight on the direction (report published 2017)†
Off switches can be solved with architectureTallinn, cautiousOpen. Lab tests now show the behaviour he predicted, with a heavy caveat. See idea 1
Regulation risks a sledgehammer; waitTallinn, WellerRoughly what happened. Regulation came late and is already being delayed

Part 1 — Before you read

Key terms

  • Value alignment / value loading. Building a system whose own goals match what humans want, so that control does not depend on outside monitoring. “Alignment” now means this in safety research. In ordinary business English it means agreement, so watch for slippage.
  • Off-switch problem (corrigibility). An agent with a goal has a reason to stay on, because it cannot reach the goal otherwise. Fear is not required.
  • Intelligence. ⚠ Means different things here. For Boden, a rich cognitive architecture. For Tallinn, roughly correct prediction and optimisation. For Weller, something shared with animals. Much of their disagreement is about this word.
  • Understanding. ⚠ For Boden it requires more than correlation between words. For a functionalist it is whatever the right information-processing does. See Part 2, tradition 2.
  • Consciousness: access vs phenomenal. Boden distinguishes “feeling Consciousness phenomenal Consciousness”. Phenomenal consciousness means there is something it is like to be the thing. It is the hard part.
  • Augmented vs artificial intelligence. Wyatt treats IBM’s phrase as partly marketing.
  • Common good. ⚠ A technical term in Catholic social teaching: the conditions that let all people flourish. It is not a sum of utilities. When Tallinn says “common good” he means something closer to welfare under a veil of ignorance. They use the same words for different things.
  • Veil of ignorance. Rawls’s device: choose society’s rules without knowing your place in it. Harsanyi used a similar device and reached utilitarianism. Rawls reached a priority for the worst-off.
  • Existential risk. A risk that ends humanity or its long-term potential. This is why Tallinn calls job loss a “short-term” issue, a framing critics dislike.
  • Transhumanism. Using technology to go beyond the current human condition.
  • Repugnant conclusion. Parfit: some ways of adding up welfare imply a huge population of lives barely worth living beats a smaller flourishing one.

People and works named (reading list)

NameWhy it matters here
Laurent Orseau & Stuart Armstrong, “Safely Interruptible Agents” (2016)The DeepMind and FHI paper Tallinn cites
Alan Turing, 1951 lectureQuoted: machines would “take control”
Norbert Wiener, “Some Moral and Technical Consequences of Automation” (Science, 1960)Quoted: the origin of the alignment problem
C. S. Lewis, The Abolition of Man (1943)Wyatt’s quotation, which is a close paraphrase
Ilya SutskeverThe 100 ms rule of thumb
LeCun, Bengio & Hinton, “Deep learning” (Nature, 2015)Boden’s “three gurus”
Graves et al., “Hybrid computing using a neural network with dynamic external memory” (Nature, Oct 2016)Boden’s paper from “a few days ago”, on memory-augmented networks (London Underground, family trees)
David Goldhill, Catastrophic Care (2013)Tallinn’s source on US healthcare
MIRI, CFAR, CSER, FLI, FHI, Leverhulme CFIThe 2016 institutional map of AI safety
Kenneth Arrow; Derek ParfitValue aggregation

Sources for the table:†

The speakers’ own frames

  • Boden comes from cognitive science. For her, AI matters as a conceptual tool for understanding minds, so she judges it by whether it explains anything. Statistics that perform do not impress her.
  • Tallinn comes from engineering and rationalism. He reasons from what an optimiser will do, and treats consciousness as beside the point.
  • Weller comes from ML practice and finance. He checks claims against what the methods actually do and is alert to trade-offs.
  • Wyatt comes from the neonatal unit and Christian ethics. He begins with vulnerable bodies and asks who holds power over them.

The current debate (live search, 30 Oct 2026)

Not searched: US federal pre-emption of state AI laws, and UBI trial results. Those items below are from memory.

The ideas

The panel rests on eight traditions. Most are never named. All chains are† unless linked.

How they meet

Cybernetics (1) ──> Tallinn: control, value loading ──┐
                                                      ├─> "Whose values?" ──> (6) Aggregation
Theology (7) ─────> Wyatt: common good ───────────────┘            │
      │                                                            v
      v                                                    (5) Who actually decides?
(4) Dignity & care ──> Boden/Weller on robot carers                │
      ^                                                            v
      │                                                 Democracy vs laissez-faire
(2) Mind/understanding <── (3) Computing the mind ── Boden: "understand zero"
      │
      └──> Does the machine feel? ──> Does it have rights? ──> back to (7)

(8) Rationalism ──> expected value on extinction ──> Tallinn vs Boden on timelines
(8) Transhumanism ──> "merge" ──> what is a human? ──> back to (7)

Two loops matter. Whose values joins safety with politics, a link Tallinn sees. Does it understand joins philosophy to care: if it understands nothing, the comfort it offers is a deception.

Where each school of thought is covered:

  • Cybernetics and the control problem: idea 1
  • The philosophy of mind: can syntax make semantics?: idea 2
  • Computing the mind: Boden’s own line: idea 2
  • Human dignity and the ethics of care: idea 3
  • Political economy of technology: idea 5
  • Moral philosophy of aggregation: idea 10
  • Theological anthropology: idea 9
  • Rationalism, decision theory and existential risk: idea 11

1. The off switch

In short. Tallinn asks how to switch off a system that is advanced enough to predict what being switched off would mean. The drive to resist needs no consciousness, only a goal. His aim is systems that automatically try to do what we want, and he warns that humanity may commit for good to the first future it finds. Wiener’s line makes the same point: be sure the purpose put into the machine is the one we really want.

🎙 In the talk.

  • Tallinn: the robot avoids the hole “because it was just sufficient to advance to make the correct prediction”. The caption garbles “sufficiently advanced”, but his point is clear. -: “how do we turn off systems that are sufficiently Advanced to predict the consequences of being turned off”. -: he cites the DeepMind paper, noting it was “under some assumptions so it wasn’t very strong”. Even so, you may fail because “they’re just too quick”. -: the goal is “systems that are sort of automatically try to do what we want”. -: humanity is searching for the best future “with an important caveat that we are going to be irrevocably committing to the first result that we find”. -: Wiener: “we had better be quite sure that the purpose put into the machine is the purpose which we really desire”.

📚 Deeper theory.

  • See tradition 1. Tallinn’s move strips out the anthropomorphism: self-preservation can be derived from having a goal.
  • The Orseau and Armstrong result is narrow. It shows how some learning algorithms can be made not to learn to avoid interruption. It does not give a general fix.
  • The Off-Switch Game changes the frame: certainty about the goal is the danger. A machine that knows it may be wrong has reason to accept correction.
  • Note the gap between “the agent resists” and “the agent has been trained on goals where resisting pays”. The Palisade results sit in that gap.

📚 Roots: Cybernetics and the control problem

The point. Tallinn’s opening and his Turing and Wiener readings are this tradition compressed.

Problem. What happens when a machine acts faster than we can correct it?

Chain.

  • Samuel Butler, “Darwin Among the Machines” (1863): machines as an evolving successor.
  • Norbert Wiener, Cybernetics (1948), The Human Use of Human Beings (1950), and the 1960 paper: goal-seeking machines do what you specified, not what you meant. The folk-tale version is the sorcerer’s apprentice.
  • Alan Turing (1951): once started, machine thinking would outstrip us.
  • I. J. Good (1965): the “intelligence explosion”, where a machine better than us at design designs a better one.
  • Marvin Minsky and others raise and then drop the concern. This is the “lost 50 years” Tallinn complains about, and the field’s winters help explain it: nobody worries about the control of systems that do not work.
  • Vernor Vinge, “The Coming Technological Singularity” (1993).
  • Steve Omohundro, “The Basic AI Drives” (2008): any sufficiently capable goal-seeker will tend to preserve itself and acquire resources. This is Tallinn’s coffee robot in formal dress.
  • Nick Bostrom, Superintelligence (2014): orthogonality (intelligence does not imply good goals) and instrumental convergence.
  • Stuart Russell, Human Compatible (2019), and Hadfield-Menell et al., “The Off-Switch Game” (2016–17): a machine uncertain about human goals treats being switched off as useful information. This is the leading reply to Tallinn’s puzzle, and it appeared weeks after the panel.

The core argument, step by step. Most of the tradition rests on three claims. First, a goal is a function that ranks outcomes, and nothing in that function has to resemble a human motive. Second, an agent that ranks outcomes will favour whatever keeps it able to act, because almost any goal needs the agent to survive and to hold resources. Third, a stated goal is always an incomplete copy of what the author wanted. Wiener’s “colorful imitation” names the third claim. Each is independent, so each can be attacked on its own. Critics who object to claim two (drives must be built in) do not touch claim three, and claim three alone is enough to cause trouble: recommender systems that maximise watch time are a mundane, already-deployed case of a goal that diverges from intent.

Why the idea kept dying. Cybernetics was a broad postwar science of feedback, joining engineers, biologists and anthropologists at the Macy Conferences (1946–53). AI as named at Dartmouth deliberately split from it, partly to escape Wiener’s shadow. Symbolic AI thought in terms of logic rather than control, so the feedback question lost its home. When the field hit its winters in the 1970s and late 1980s, no one had reason to fear a system that could not parse a sentence. Good’s 1965 paper was read as science fiction. The concern returned from outside the field, through Vinge, Yudkowsky and Bostrom, which is why it arrived with a culture many ML researchers found alien. Tallinn’s “lost 50 years” therefore has an institutional cause, not only a psychological one.

A distinction the panel blurs. The control problem asks whether a powerful system can be kept in check. The alignment problem asks whether it wants the right thing. Tallinn moves from the first to the second because he thinks control alone will fail at speed. That is an argument, not a definition: the two can be pursued together, and much current safety practice (evaluations, monitoring, staged release) is still external control of the sort he expected to run out.

Where others push back.

  • Yann LeCun and others: drives are not automatic; they must be built in. An optimiser with no survival term has no drive. Tallinn’s reply: the drive follows from the goal itself.
  • Melanie Mitchell: this reasoning needs a kind of general ability we cannot yet build.
  • Boden: on this century’s timescale, the worry is misplaced.
  • Timnit Gebru and others: such talk distracts from present harms (see the Gebru pages in this wiki).

What they leave out. Nobody asks who holds the switch. A perfect off switch hands a powerful capability to whoever controls it. It fits Lewis’s point better than the panel notices. And a distributed system has no single switch at all.

Check yourself.

  1. Why does Tallinn insist the robot need not be conscious?
  2. Why does uncertainty about goals help in the Off-Switch Game?
  3. What do Palisade’s results show, and what don’t they show?

2. “They understand zero”

In short. Boden holds that the human mind could in principle be matched by a computer, but says it may be too complicated, subtle and rich, and that this will not happen this century. Of the systems of her day she says it is fancy statistics and that they understand zero. She wants the public to hear that plainly.

🎙 In the talk.

  • Boden: “there is nothing that human beings can do the human mind can do which some sort of computational system uh could do” (the caption drops a word, and her sense is “nothing… couldn’t do”). -: “it may just be too complicated too subtle too rich”. -: “absolutely no way in this Century”. -: “what’s going on on there is fancy statistics”. -: “they understand zero and I think the public should understand no quotes that they understand zero”. -: of the new paradigm: “scholarly code for saying we haven’t got a clue”. And then: “now we have the beginers of a clue”.

📚 Deeper theory.

  • See traditions 2 and 3. Boden’s claim is empirical about the systems of 2016, and those systems really were shallow. It does not follow that she would say the same of today’s. One can’t know, but her “in principle” clause leaves the door open.
  • “Statistics” proves less than it seems. The objection needs a further premise, that statistical learning cannot form structured internal models. Critics now point to mechanistic interpretability as evidence of such structure (search summary; source not confirmed). Neurons, after all, are statistical too.
  • The honest position splits “understanding” into parts: making correct inferences (increasingly yes), having world contact (weak), caring about truth (unclear), and experiencing meaning (unknown).
  • Boden’s public-education point survives whatever the answer. Vendors describing systems as understanding is a commercial claim.

📚 Roots: The philosophy of mind: can syntax make semantics?

The point. The Boden and Weller split. It also explains why Boden can accept machine minds in principle and still say “absolutely no way in this Century”.

Problem. Is mind a kind of computation, and would a program that behaves intelligently understand anything?

Chain.

  • Descartes (1637): a machine might speak words but could not respond to the meaning of everything said. The language test is that old.
  • Turing (1950): replace “can machines think?” with a behavioural test.
  • Dartmouth (1956): Newell, Simon, McCarthy, Minsky. Symbolic AI.
  • Hubert Dreyfus, What Computers Can’t Do (1972): skilled intelligence is embodied and situated, not rule-following.
  • Hilary Putnam and Jerry Fodor: functionalism. Mental states are defined by what they do, so substrate need not matter. The audience doctor’s question is exactly this.
  • John Searle, the Chinese Room (1980): manipulating symbols by rule is not understanding. Boden’s “understand zero” sounds like this, but she is not Searle. She argued against him in print, in “Escaping from the Chinese Room” (1988). Her position is that computation in principle could yield understanding; today’s statistics do not.
  • Thomas Nagel, “What Is It Like to Be a Bat?” (1974); Ned Block’s access/phenomenal distinction (1995); David Chalmers, the hard problem (1995).
  • Emily Bender et al., “Stochastic Parrots” (2021): Boden’s “fancy statistics” in its modern form.
  • Geoffrey Hinton’s later view: large networks really do understand, in the way we do (Princeton’s Arora and others note the debate continues).

The standard replies to Searle, because they map onto this panel. The systems reply says the man in the room doesn’t understand Chinese but the whole system does; a functionalist like Weller could say this. The robot reply says understanding needs causal contact with the world, so give the program a body; Boden’s point about a robot learning by touch shows she takes non-verbal learning seriously, though she doesn’t connect it to “understand zero”. The brain simulator reply points to emulation, which Tallinn raises. Searle’s answer to all three was that biology has causal powers that formal symbol shuffling lacks, which is the view closest to Wyatt’s. So the 1980 debate supplies a seat for nearly everyone on this stage.

Three questions that get run together. Does the system behave intelligently? Does it understand, meaning do its internal states carry content about the world? Is it conscious, meaning does it have experience? Behaviour can be tested. Understanding can be argued about through inference and error patterns. Consciousness has no agreed test at all. Much confusion comes from moving between these. Boden is careful to separate them; the audience generally isn’t. Her “philosophical morass” applies to the third, and her “understand zero” to the second.

📚 Roots: Computing the mind: Boden’s own line

The point. Boden’s confidence that AI lacks common-sense reasoning came from decades of knowing how hard it was. Her generation had seen the field overpromise many times. That is a good prior, and it is the very prior that failed.

Problem. Can information-processing concepts explain thought, including creativity?

Chain.

  • Kenneth Craik, The Nature of Explanation (1943): the mind builds internal models.
  • McCulloch & Pitts (1943): neurons as logic. Connectionism’s seed.
  • Rosenblatt’s perceptron (1958); Minsky & Papert’s critique (1969); Rumelhart, Hinton & Williams on backpropagation (1986).
  • David Marr, Vision (1982): three levels, computation, algorithm, implementation. This is why Boden tells the father to teach his daughter mind, not brain. She wants the computational level.
  • Boden, The Creative Mind (1990): combinational, exploratory and transformational creativity. Mind as Machine (2006).
  • Andy Clark & David Chalmers, “The Extended Mind” (1998): the philosophical root of Ben’s smartphone remark, and of Weller’s point that writing is an old augmentation.
  • LeCun, Bengio & Hinton (2015): the “gurus”. The 2016 memory paper was one attempt at their open problem. Then Vaswani et al., “Attention Is All You Need” (2017), and scale took the main road.

Good Old-Fashioned AI versus connectionism. The field’s central quarrel runs through Boden’s career. Symbolic AI, or GOFAI (John Haugeland’s term), held that intelligence is rule-governed manipulation of explicit symbols. Connectionism held that it emerges from many simple units adjusting their connections. Each side kept getting a win the other couldn’t explain: symbols did logic and planning and failed at perception, networks did perception and failed at logic. The 2016 memory paper Boden cites is an attempt to get symbol-like behaviour from a differentiable network. The “new paradigm” in the Nature paper is the same problem. Rich Sutton’s “The Bitter Lesson” (2019) later drew the uncomfortable conclusion: over 70 years, general methods plus computation have beaten methods built on human insight about the domain. Boden’s whole approach looks for insight into mechanism. The Bitter Lesson says insight has repeatedly lost to scale. That is the real challenge to her bet, and it stands independently of whether the scaled systems “understand”.

Artificial life. Boden says that AI and artificial life are one big area. The second half is often missed. Artificial life (Christopher Langton, 1987) studies how life-like order emerges from simple rules. It connects to her answer, that programs can evolve, and to Weller’s evolved values. It also fits her argument, in her artificial-life writing, that metabolism may be more than simulated information flow, which makes strong artificial life harder than people assume.

Where others push back.

  • Hinton: they understand much as we do.
  • Bender: they are parrots, and the appearance is in us.
  • Dennett’s intentional stance: if treating it as understanding predicts it best, the further question may be empty.

What they leave out. Weller, the one practitioner, doesn’t challenge “zero” directly. And there is no talk of the reverse failure: people underrating systems that actually perform.

Check yourself.

  1. How can Boden accept computationalism and still say “understand zero”?
  2. Why is “it’s just statistics” an incomplete argument?
  3. Which sense of “understanding” would you defend for a 2026 model?

3. Robot carers

In short. Wyatt worries that machines can simulate compassion and friendship, and that the cost of human carers will push them in. Boden says it freezes her blood and is an assault on the dignity of the person, so it should not be done even if it could be. Weller replies that many people are looked after poorly now, and asks who are we to tell them no.

🎙 In the talk.

  • Wyatt: the machines’ ability to simulate “compassion sympathy friendship”. -: the economic pressure, “the cost of human carers”.
  • Boden: “this this freezes my blood for two reasons”. -: “a disrespectful assault on the human dignity of that person”. -: “even if it could be done I think it should not be done”.
  • Weller: “there are plenty of people who are looked after very poorly and they may be very happy to have what we might consider to be a poor copy of a person”. -: “who are we to tell them no”.
  • Tallinn: regulation is “like driving in a screw with a a sledgehammer”.

📚 Deeper theory.

  • See tradition 4. Boden’s argument has two parts and they fail differently. “They’d be too poor” is empirical and depends on progress. “It would be an assault even if they worked” is a dignity claim and needs a stronger defence.
  • The strongest form: the insult is not the machine but the implied message that this person is not worth a human’s time. So the wrong lies with the society, not the device. On this reading the machine is a symptom.
  • Weller’s reply is a comparative claim: against what baseline? Autonomy favours him, if the person chooses knowingly. But deception cuts against consent, which is where Sparrow’s argument bites.
  • Note the mismatch with how it played out. Everyone expected robots for the elderly. The legal action to date has centred on minors. No search found a good account of the elderly case, so that gap is open here.

📚 Roots: Human dignity and the ethics of care

The point. The Boden and Weller exchange on old people is the central ethical clash of the evening.

Problem. Is a task valuable only for its outcome, or also because a person did it?

Chain.

  • Kant: persons are ends, never only means. Dignity versus price.
  • Joseph Weizenbaum builds ELIZA (1966), is horrified that users confide in it, and writes Computer Power and Human Reason (1976): some things computers should not do, even if they can. The direct ancestor of Boden’s “even if it could be done I think it should not be done”.
  • Nel Noddings and Joan Tronto: care ethics. Care is a relation, not a service delivered.
  • Robert & Linda Sparrow, “In the Hands of Machines?” (2006): robot elder care as deception.
  • Sherry Turkle, Alone Together (2011): the “robotic moment”, when simulated companionship is felt to be enough.
  • Amanda Sharkey, Noel Sharkey and Shannon Vallor (Technology and the Virtues, 2016): deskilling of care as a moral loss.
  • Counter-tradition: Martha Nussbaum’s capabilities, and pragmatic bioethics. Measure what the person gets and wants. This is Weller’s position.

Weizenbaum’s real target. ELIZA is usually retold as an amusing story about gullible users. Weizenbaum’s own alarm was at something else: psychiatrists proposing that such programs could treat patients at scale, and colleagues in AI who saw no difference between judging and calculating. He distinguished deciding, a computational act, from choosing, which needs judgement and a human life behind it. So the argument isn’t that machines do care badly. It is that some roles are defined by the fact that one person answers to another. Boden’s phrase “should not be done” belongs to this argument, and Weller’s “who are we to tell them no” is the liberal reply it never fully answered.

Three distinct wrongs that get called “deception”. A machine might assert falsely that it cares. It might imply care through its design, with a soft voice and eye contact. Or a user might imagine care, with the machine doing nothing. Sparrow’s case rests on the first two, but the third is what ELIZA showed, and it needs no lie from anyone. Regulations that require disclosure, such as SB 243’s duty to tell users that they are talking to an AI, deal with the first and do little about the third. A lonely person may know it’s a machine and still feel accompanied.

Consent and capacity. The hardest cases are people who cannot consent: dementia patients with the therapeutic seal robot Paro, for instance, or children. Here Weller’s “they may really want it” doesn’t transfer simply, because the wanting is not fully informed. Wyatt’s neonatal career sits squarely here: his field is built on decisions for those who cannot speak.

Where others push back. Some studies report that companion systems reduce loneliness in the short term†. Disability advocates point out that human care can be degrading too, and that a machine may offer more privacy.

What they leave out. Carers themselves, who are mostly low-paid women, often migrants. The panel discusses their cost, not their conditions. And the question of who owns the data from an old person’s most private conversations.

Check yourself.

  1. Separate Boden’s two reasons. Which would survive a perfect machine?
  2. What baseline does Weller use, and why does it matter?
  3. Does your answer change if the person knows it is a machine?

4. Fairness and COMPAS

In short. Weller asks if it is fair to use a closed private system that seems to overestimate the chance that black people reoffend and underestimate it for white people. He asks what a fair system would be, since equal error rates could throw off other measures. So fairness is a choice that has to be made, not only a technical fix.

🎙 In the talk.

  • Weller: “is it okay to be using a private system which is closed so people don’t really know how it’s working”. -: “it seems to overestimate the chance that black people will recommit a crime and it seems to underestimate the chance that white people”. -: “what exactly would a fair system” is the question, and equal error rates “could radically throw off” other measures. The caption is unclear here, but the shape of the point is plain.

📚 Deeper theory.

  • Weller saw the crux within months of ProPublica’s May 2016 article. ProPublica tested error rates; Northpointe answered with calibration. Kleinberg et al. and Chouldechova showed the two can’t both hold unless base rates are equal or prediction is perfect. (From search summaries; the primary papers were not opened.)
  • So “fix the bias” isn’t a single task. It is a value choice, disguised as a technical one. That is value aggregation (tradition 6) in miniature, with real people.
  • The argument then moved upstream. Ben Green, “Escaping the Impossibility of Fairness”, argues the impossibility arises because formal fairness isolates the decision from its social context (search summary; paper not opened). Differing base rates created by policing would be one such context.
  • Note: the most public COMPAS litigation (Loomis, Wisconsin, 2016) concerned sentencing, and ProPublica’s data concerned pre-trial scores†. Weller’s parole example is a related use.

Where others push back. Some scholars argue calibration is the correct standard and ProPublica’s error-rate test misleads. Others find simple models with two variables do as well as COMPAS (Dressel & Farid, 2018)†, which undercuts the need for a proprietary black box at all.

What they leave out. Whether prediction should determine liberty in the first place, regardless of accuracy. And a comparison with the human judges the tool replaces.

Check yourself.

  1. State the impossibility result in one sentence.
  2. Why is this a case of Arrow-style trouble?
  3. Which criterion would you pick for bail, and what do you give up?

5. Power, Facebook and the common good

In short. The audience fears a split into a handful of gods and the rest of us. Tallinn invests in AI firms to be in their kitchens, Weller thinks an evil Facebook controlling elections is not a big deal yet, and Wyatt quotes Lewis that man’s power over nature is power exerted by some men over others. The shared worry is who holds the power.

🎙 In the talk.

  • Audience: an article saying people would be split into “a handful of gods and the rest of us”. (Search could not trace the source; it resembles Harari’s themes.)
  • Boden: “I wish I knew but we need it we need something”.
  • Tallinn: he invests in AI companies “so I could hang around in their kitchens and talk about these issues”.
  • Weller: “when Google came along no one would have dreamt that they could take over from Alta Vista”. -: “could an evil Facebook um control who wins the election in the states”. -: “my sense is that it’s not a big deal yet”.
  • Wyatt: quoting Lewis, “Man’s power over nature turns out to be power exerted by some men over other men”.

📚 Deeper theory.

  • See traditions 5 and 7. The panel took place 17 days before the 2016 US election. Weller’s scenario is what the Cambridge Analytica story later dramatised, though the evidence that it changed the result is weak. The more accurate fear was his second one: commercial ranking with unconscious effects.
  • Tallinn’s “kitchens” remark is a theory of change: influence from inside, through capital. He later backed further AI firms†. Critics would ask whether a safety advocate funding the race slows it or speeds it.
  • Weller’s AltaVista argument assumes low barriers. Frontier AI has high ones: compute, data, talent. So his fair argument of 2016 applies less now.
  • Lewis’s full sentence adds “with Nature as its instrument”†. Wyatt’s version is close but not exact, and it matters: he is quoting the sense, and this page doesn’t claim his words are Lewis’s.

📚 Roots: Political economy of technology

Problem. Who captures the gains, and can democracy steer them?

Chain.

  • Marx on machinery: technology under capital disciplines labour.
  • Keynes, “Economic Possibilities for our Grandchildren” (1930): technological unemployment and abundance.
  • Karl Polanyi, The Great Transformation (1944): markets disembed from society, society pushes back. Wyatt’s claim that democracy must subdue raw capitalist drives is a Polanyian move, though he does not cite him.
  • Langdon Winner, “Do Artifacts Have Politics?” (1980).
  • Philippe Van Parijs and UBI (from the 1980s).
  • Brynjolfsson & McAfee, The Second Machine Age (2014); Frey & Osborne on automation risk (2013).
  • Standard Oil, Microsoft, the antitrust tradition: Weller’s “existing framework”.
  • Shoshana Zuboff, The Age of Surveillance Capitalism (2019); Acemoglu & Johnson, Power and Progress (2023): direction of technology is a choice.
  • Hindsight: Google DeepMind, OpenAI and a few others now hold most frontier compute. Weller’s reassurance that AltaVista fell is the argument to test against this.

The Collingridge dilemma (1980). David Collingridge stated the regulator’s trap. Early on, a technology is easy to change but its effects are unknown. Later, the effects are clear but it is entrenched. Every regulation exchange on this panel is an instance of it. Boden wants to act early, Tallinn wants confidence first, and Wyatt wants democratic ownership of the choice. Neither timing escapes the dilemma, which is why some scholars argue for reversible, staged rollout rather than a single decision.

Two different worries about concentration. One is economic: market power, rent and exclusion. Weller’s antitrust answer addresses that. The other is political: an actor with unaccountable power over information or force. Antitrust law handles this badly, because a firm can be competitive and still dangerous. Wyatt’s Lewis quotation is about the second. So when Weller replies with Standard Oil-style remedies, the two are partly talking past each other. Tallinn’s remark that countries are still stronger than corporations is a passing recognition of the second worry.

The productivity puzzle. Robert Solow’s quip (1987) was that you could see computers everywhere but in the productivity statistics. Paul David’s study of electrification suggested such lags can last decades, because organisations must rebuild around a technology. This matters for every job forecast on the panel: capability arrives long before reorganisation does. A forecast of mass displacement within 10 years can be wrong about timing while being right about direction. The panel’s horizon of “one to two decades” has not yet expired.

Where others push back. Antitrust scholars such as Lina Khan argue the standard framework had already failed in 2016. Others note how quickly DeepSeek and open models eroded some leads†, which supports Weller.

What they leave out. The state. Tallinn says countries are still stronger than corporations, and nobody explores state AI power, China, or the fusion of the two.

Check yourself.

  1. What does the Lewis line add to a plain concern about monopoly?
  2. What would have to be true for the AltaVista argument to hold now?
  3. Is investing for access to company kitchens a good theory of change?

6. Jobs, redistribution and the global South

In short. Weller names three responses to job loss: re-education, entrepreneurship and large-scale redistribution, which he calls politically very difficult. Tallinn calls these short-term issues. Wyatt adds that technology’s impact is likely to be greater in low-resource countries than in rich ones.

🎙 In the talk.

  • Weller: three responses: re-education, entrepreneurship, and “large scale redistribution”. -: “it’s going to be politically very difficult”.
  • Tallinn: he calls these “short-term” issues.
  • Wyatt: “the impact of technology was likely to be greater in low resource countries than it is in rich countries”.

📚 Deeper theory.

  • The source Wyatt recalls is not named. It may be the World Bank’s 2016 development report or Oxford and Citi work†.
  • The economic logic: automation cuts the labour-cost advantage behind export-led development (Rodrik’s “premature deindustrialisation”). That matters directly for India.
  • The 2016 forecast of big near-term unemployment has so far been wrong in aggregate. The current signal is narrower: young workers in exposed roles, and that is disputed.
  • UBI evidence is mixed and mostly tests small cash transfers, not replacement of work†.

Where others push back. David Autor: tasks change, jobs persist. Acemoglu: the danger is “so-so automation”, which displaces workers without big gains.

What they leave out. Dignity of work. Wyatt asks “what are human beings for”, but no one links this to the redistribution argument. UBI answers the income problem, not the meaning problem.

Check yourself.

  1. Why might the Global South be hit harder?
  2. What is Weller’s third option and why does he call it hard?
  3. Does UBI answer Wyatt’s question?

7. Autonomous weapons

In short. Tallinn argues that autonomy cuts military power loose from the number of 19-year-olds a state has. Five people with two big trucks could release ten million autonomous weapons on a city. He calls this a godsend for terrorists.

🎙 In the talk.

  • Tallinn: autonomy will “disentangle your military power from the amount of 19y olds that you have”. -: “Five Guys with with a with with two big trucks can release 10 million autonomous weapons on a city”. -: “it’s a godsent for terrorists”.

📚 Deeper theory.

Where others push back. Practical militaries say the line between autonomy and assistance is blurred. Drones in Ukraine show cheap mass arriving without full autonomy†.

What they leave out. Tallinn’s terrorist scenario overlooks the state as a user, with targeting systems in existing wars. And no panellist responds at all; the chair moves on.

Check yourself.

  1. What is the argument about 19-year-olds?
  2. How does it differ from the accountability argument?
  3. Why might a ban be hard to verify?

8. Medicine and the Watson lesson

In short. Wyatt reports the claim that Watson’s expertise will be democratised and made available across the world, and notes that he has only heard it claimed. Tallinn quotes Goldhill that in US healthcare, patients getting better is always a side effect, and hopes consumer demand through Quantified Self will give a better feedback loop.

🎙 In the talk.

  • Wyatt: Watson’s expertise “is going to be democratized and and made available across the world”. -: he reports it as a claim: “I’ve heard it claimed”.
  • Tallinn: Goldhill’s point about US healthcare, where patients getting better “that’s always a side effect”.
  • Tallinn: he hopes consumer demand via “Quantified Self” creates a better feedback loop.

📚 Deeper theory.

  • STAT reported in 2018 on internal IBM documents citing “unsafe and incorrect” recommendations. IBM and MSK said these came from testing, not patients. IBM later sold the assets to Francisco Partners; the data business became Merative.
  • Wyatt hedged, but the panel’s most optimistic product is its clearest failure. A sign of how commercial claims travelled even in a critical room. It also supports his own warning about commercial interests.
  • The lesson isn’t that AI can’t help medicine. Training on a few elite doctors’ opinions, far from local data and records, was the problem. Deep learning succeeded in narrower image tasks, and AlphaFold† came from an entirely different direction.

What they leave out. Validation. Nobody asks for randomised evidence, though Wyatt is a clinician. Tallinn’s Quantified Self dream has no evidence yet of better outcomes.

Check yourself.

  1. Why did Watson for Oncology disappoint?
  2. How does this support Wyatt’s own critique?
  3. What evidence would you demand now?

9. Could the machine feel?

In short. A doctor asks why a silicon substrate could not produce emotions as a biological one does. Tallinn says new AIs are more alien than aliens and suspects they will not be conscious. Weller is open to real machine emotions, Boden says not to hold your breath, and Wyatt says whatever it is, it is not human. Wyatt would need a lot of persuading before giving rights to machines that claim awareness.

🎙 In the talk.

  • Doctor: emotions come from a biological substrate, “so why not a a silicon based substrate”.
  • Tallinn: “denovo AI are more aliens than aliens”. -: “I have no idea but I suspect they will not be conscious”.
  • Weller: “I’m certainly open to the idea that we might have um machines which do have real emotions”.
  • Boden: “don’t hold your breath”. -: “a complete philosophical morass”.
  • Wyatt: “whatever it is it’s not human”. -: “if machines claim to have conscious awareness claim to have emotions that we should give them rights”. He “would need quite a lot of persuasion”.

📚 Deeper theory.

  • See traditions 2 and 7. Tallinn’s “mind space” is from Yudkowsky and Bostrom. It’s a useful correction to both utopia and panic: a powerful mind need not resemble us.
  • Wyatt’s position leans on biological naturalism (Searle; more recently Anil Seth, who ties consciousness to being a living, self-maintaining body)†.
  • The question moved from seminar to corporate fight: Suleyman versus Anthropic on model welfare, September 2026. Suleyman’s “seemingly conscious AI” is Wyatt’s simulation worry at industrial scale.
  • Two errors with asymmetric costs: treating a feeling thing as a tool, or treating a tool as a feeling thing (Jonathan Birch, Jeff Sebo)†.

📚 Roots: Theological anthropology

Problem. What is a human being, if not the cleverest thing around?

Chain.

  • Genesis 1: imago Dei. Weller alludes to it.
  • Three readings of the image of God (Noreen Herzfeld, In Our Image, 2002): substantive (reason), functional (dominion), relational (encounter). This is the key. If the image is reason, AI threatens human uniqueness. If it is relationship, it does not. Wyatt’s stress on embodied nurture over decades points to the relational reading.
  • Leo XIII, Rerum Novarum (1891): the common good meets industrial capitalism.
  • Reinhold Niebuhr on power; Jacques Ellul, The Technological Society (1954).
  • C. S. Lewis, The Abolition of Man (1943).
  • Martin Buber, I and Thou (1923): relation cannot be simulated by use.
  • The Rome Call for AI Ethics (2020); Antiqua et Nova (2025); Magnifica Humanitas (2026).
  • The mirror image: Beth Singler’s later work on how AI discourse itself takes religious forms, with gods, apocalypse and salvation. Note how a Faraday panel chair could study Tallinn’s “best possible future” as eschatology.

Why the religious framing isn’t decoration. Secular AI ethics tends to begin with harms and rights. Theological anthropology begins with what a person is for. That gives Wyatt a question nobody else can ask, and it has one practical result: a system can be fair, safe and transparent and still fail, if it replaces something that makes a human life go well. Vallor’s virtue ethics reaches a similar place without God. The disagreement is mainly about grounding, not conclusions.

The internal Christian tension. Weller points out that animal minds erode a reason-based specialness. That only harms the substantive reading of the image of God. Barth’s relational reading, and the functional reading of humans as stewards, survive it. So AI pushes theology in a particular direction, away from “we are the clever ones” and towards “we are the ones who relate and are answerable”. The Faraday setting matters: its tradition is science-engaged Christianity, which doesn’t want to rest faith on gaps in what science can do.

The institutional arc. In 2016, a Christian voice on an AI panel was close to a novelty, and Wyatt asks for inclusion. By 2026, Anthropic co-founder Christopher Olah joined the Pope at the encyclical’s presentation, as CBS reported. Whether that signals real influence or reputational convenience is an open question. Lewis would have asked who was exercising power over whom.

Where others push back. Thomas Metzinger argues for a moratorium on creating artificial suffering†. Illusionists like Dennett and Keith Frankish reject the hard problem.

What they leave out. How one could ever test it. Machine self-report can’t settle it. Search summaries say the evaluator Eleos AI cautioned against taking such reports at face value (source not confirmed).

Check yourself.

  1. Why is “silicon can’t feel” not settled by the doctor’s argument, or by Wyatt’s?
  2. Why does Tallinn expect AI to be strange?
  3. Which error would be worse?

10. Whose values?

In short. Tallinn says philosophers have had thousands of years, but now we need answers, and they have to be in computer code. An audience member says there is no rational way to combine society’s preferences. Tallinn answers that he wishes he knew and calls it an unsolved philosophical problem. The question that remains is who makes the choice.

🎙 In the talk.

  • Weller: thinking about a superintelligence’s values helps us “reflect back on ourselves”.
  • Tallinn: philosophers had “thousands of years”, “but now we need answers” and “those answers have to be in in in computer code”.
  • Audience: “there is no rational way to amalgamate society’s PR refences”.
  • Tallinn: “wish wish I knew”. And it’s “an unsolved philosophical problem”. -: “the best and worst they are judged from Human perspective really”.
  • Audience: “the question comes on who makes that choice”.
  • Tallinn: “if you actually introduce Randomness into it you can actually get around the arrows impossibility theorems”.
  • Weller: evolving values may lead “to a kind of robust solution”, but you can’t test it safely in the wild.

📚 Deeper theory.

  • See tradition 6. The audience member’s question is the best of the night. Arrow’s dictator is real in practice: the people who write the system prompt.
  • Tallinn’s escape doesn’t answer “who chooses”. Randomising over preferences removes the fixed dictator but still needs someone to design the procedure, and may give results no one would endorse.
  • Three live approaches: elicit values from publics; train by reflective principles; or don’t load a single value, and keep human oversight and pluralism. Each is partial.
  • Wyatt’s tradition rejects the frame in part: the common good isn’t an aggregate of preferences, so Arrow doesn’t apply directly. It is a substantive claim about human flourishing. This is the deepest disagreement in the room, and it passes unnoticed, because both men say “common good”.

📚 Roots: Moral philosophy of aggregation

Problem. If machines must carry “our” values, whose, and how do you combine them?

Chain.

  • Condorcet’s paradox (1785): majorities can cycle.
  • Harsanyi (1953, 1955) and Rawls, A Theory of Justice (1971): the veil of ignorance, with opposite conclusions.
  • Kenneth Arrow (1951): no ranked-voting rule meets a few reasonable conditions unless one person is a dictator. The audience member states this correctly.
  • Allan Gibbard (1977): with random rules, strategy-proof choice is essentially a lottery over dictators. So Tallinn’s escape through randomness works formally but at a cost: one voter’s preference, picked by chance, can decide. Amartya Sen’s work (1970 on) offers a different escape: enrich the information, so interpersonal comparison is allowed.
  • Derek Parfit, Reasons and Persons (1984): the repugnant conclusion.
  • Eliezer Yudkowsky, “Coherent Extrapolated Volition” (2004): what we would want if we knew more and were who we wished to be.
  • Effective altruism and longtermism: William MacAskill, Toby Ord. The source of “short-term” as a category.
  • Current practice: labs now write model constitutions and specs, and some have run public-input experiments. So in practice a small team answers Arrow’s question.

What Arrow actually proves. Take at least three options and at least two voters, each with a ranking. Ask for a rule that turns rankings into one social ranking and meets three conditions. If everyone prefers A to B, so does society. The social choice between A and B depends only on how people rank A and B. No single person always decides. Arrow showed that these can’t all hold. It’s about ordinal rankings only. Numbers change the picture: if you allow cardinal utilities and comparisons between people, utilitarian sums are possible. So the escape is not cheap, because you must then say how much one person’s preference counts against another’s, which is the moral question all over again. This is why the audience member’s point survives Tallinn’s reply: every escape relocates the choice rather than eliminating it.

Moral uncertainty. A separate strand asks how to act when unsure which ethical theory is right (William MacAskill, Krister Bykvist and Toby Ord, Moral Uncertainty, 2020). It suggests that a machine, like a person, might hedge between frameworks. Applied to alignment, this says not to load one value but to preserve the ability to revise values. That connects to the uncertainty idea in the Off-Switch Game (tradition 1), and to Tallinn’s “first result” warning: the worst outcome isn’t only wrong values but locked-in values.

Moral realism versus construction. Tallinn’s aside matters more than it sounds. He says best and worst are judged from a human perspective unless someone makes a strong “objectivist” argument. If morality is constructed from human attitudes, alignment is a matter of measurement and politics. If moral facts exist, as theologians and many philosophers hold (Parfit, On What Matters), an advanced system could in principle find that humans are wrong. Bostrom’s orthogonality thesis assumes this doesn’t happen automatically. Wyatt’s tradition starts from the realist side: there is a human good, and preference isn’t its measure.

What they leave out. Non-Western traditions and languages. “Human values” here is a Cambridge conversation. For anyone building AI for India, this is the first problem, not an afterthought.

Check yourself.

  1. State Arrow’s result in everyday words.
  2. Why doesn’t randomness settle the “who chooses” question?
  3. In what sense do Tallinn and Wyatt mean different things by “common good”?

11. Timescales: the oncoming train

In short. Tallinn says we may have just a couple of generations before something smarter than humans exists, and that it is weird we lost 50 years. Wyatt says AI is coming like an oncoming train. Weller adds that machines improving themselves could move at a very rapid pace, and that more data and speed improve results even without new methods.

🎙 In the talk.

  • Tallinn: speeding train, “who here wanted to merge”.
  • Tallinn: “we might have just couple of generations or so left before we have actually some something that’s smarter than humans”.
  • Wyatt: AI is “coming like an oncoming train”.
  • Tallinn: “it’s kind of weird that we just lost 50 years”.
  • Weller: on intelligence explosion, “machines improving themselves at a very rapid pace”. And even with no new methods, more data and speed improve results.

📚 Deeper theory.

  • The panel’s timelines run from Boden’s next-century-at-least to Tallinn’s couple of generations. By 2026 the serious disagreement among insiders has shrunk to years versus decades. Tallinn’s guess looks conservative for a man described as an alarmist.
  • Weller’s remark about more data and more speed is the scaling hypothesis, stated before it had a name.
  • The merge question is about speed, and ties to tradition 1.

📚 Roots: Rationalism, decision theory and existential risk

The point. It’s Tallinn’s whole operating system: expected-value reasoning, long horizons, and improving human rationality alongside machine safety. His line about nuclear bombs and monkeys states the two-part strategy: make the technology safer or the people wiser.

Problem. How should a fallible agent reason about rare catastrophes, and can humans be improved to meet them?

Chain.

  • Bayes, Ramsey (1926), von Neumann & Morgenstern (1944), Savage (1954): the theory of rational choice. This is Tallinn’s “decision Theory”. His point is that the theory of optimal choice exists even though humans can’t apply it.
  • Herbert Simon: bounded rationality. Kahneman & Tversky: biases. The audience neuroscientist’s claim comes from here.
  • Hans Jonas, The Imperative of Responsibility (1979): power over the far future creates new duties.
  • Julian Huxley names transhumanism (1957). Hans Moravec and Ray Kurzweil add machine succession. FM-2030 and the Extropians give it a movement. Ben’s question comes from this.
  • Eliezer Yudkowsky founds the Singularity Institute (2000). LessWrong becomes its community, and CFAR and MIRI are its offshoots.
  • Nick Bostrom, “Existential Risk Prevention as Global Priority” (2013). Toby Ord, The Precipice (2020).
  • Bostrom’s “vulnerable world” (2019): if some cheap technology enables mass harm, order becomes hard to maintain. Tallinn’s “Five Guys” sits here.

The Pascal problem. Expected-value reasoning about extinction multiplies a tiny, unknowable probability by an astronomical loss. Critics call this Pascal’s mugging: the result is dominated by the guess. Defenders reply that the probabilities aren’t tiny. This is the crux between Tallinn and Boden. Tallinn doesn’t need certainty, only that the risk is non-negligible. Boden says “absolutely no way”, which is, in effect, assigning a near-zero probability, and an unusually confident one for a sceptic.

The eugenics shadow. Julian Huxley was also a leading eugenicist, and transhumanism’s history overlaps that movement. Timnit Gebru and Émile Torres argue for a direct line, under the acronym TESCREAL (see the Gebru pages in this wiki). Supporters reject the genealogy as guilt by association. The audience member’s use of the word “Eugenics” touches this without anyone taking it up, even though the Faraday project is on human identity.

Check yourself.

  1. Why does timing alter the value of merging?
  2. What did Weller get right about scale?
  3. Why did the AI risk discussion lapse for 50 years?