Talks

"Computers can't care": needs, goals and human wellbeing — Margaret Boden

64 min read

Margaret Boden · 2018 · 1:34:52 · ▶ Watch

Summary

Central question. What is the real difference between human intelligence and AI, and what does it imply for how AI will fit into society?

Main thesis. The difference is care. Humans care about things; computers do not. In her words: “it’s a fundamental difference between human intelligence and intelligence of AI that we care and computers don’t”.

She builds three consequences on this, plus a policy complaint.

  1. Synergy is real, but decisions that matter stay with humans. Medical AI is already strong, and she is glad of it. Handing over the decision about what to do is a different matter. The same limit holds for personal domains such as marriage counselling or baby names.
  2. True collaboration between people and AI is not possible. Her argument is a chain. Goals come from needs. Needs belong to living things. Computers have no needs, so no real goals, so nothing to share. “Working alongside computers” is therefore not the comfort it is sold as. Her evidence is on unemployment: the worst effects of losing work are loss of motivation and loss of shared purpose, not hunger.
  3. Some contexts should stay person-to-person. Care homes, hospitals, families. She cites Weizenbaum’s “obscene” and notes that Japan sees this very differently.

Policy point. Reports on “AI and human wellbeing” talk about privacy, rights and “capabilities” (Sen and Nussbaum). They leave out personal psychological needs, the kind Maslow listed.

Close and stance. She is not against AI. She says medical AI is “fine” and that robots such as Paro help. Her target is the idea that a system without needs can stand in for a person where respect, understanding and love are the point. In the Q&A she calls AI liability “cynical nonsense”, agrees that human carers are often badly treated, and says she hopes AI will make human care more valued, not less.

Map — talking points

#Point
1Welcome (Halina); introduction (Price): CSER, CFI, Boden’s career
2Opening story: Henry James, The Beast in the Jungle. “Nobody mattered to him”. Computers are like that
3Roadmap: synergy, the collaboration fantasy, contexts to avoid
4Medical AI (CREDO, deep learning). UK poll on diagnosis vs treatment
5Decisions on treatment are about quality of life, family, money. “Irrational” says one medic
6Marriage counselling. Laing’s questionnaire. Sentiment analysis and the “Soviet Union” trap
7Baby names, Watson, Judas. Statistics cannot say why. Abelson’s definition of betrayal
8Part 2: collaboration needs shared goals. AI “goals” are not our goals
9Unemployment: Marienthal, Hawthorne. Lack of purpose and contact hurts more than lack of food
10What is a need? Friston’s free energy principle versus Maslow’s hierarchy
11Goals come from needs. The bakery queue. AI cannot “prioritise” in the sense of caring
12Omohundro’s AI drives: “begging the question”
13Part 3: care homes (69 homes, one hour a week), Paro, “computer carers”
14Weizenbaum (“obscene”). Japan: robots as family. The paraplegic anecdote
15Policy reports on “wellbeing”. Sen and Nussbaum capabilities versus Maslow
16Response 1 (Browne): e-personality, liability, regulation
17Response 2 (Hernández-Orallo): is human care “fake”? Should AI be “fully rational”?
18Response 3 (Shevlin): animals, collaboration, the risk of setting the bar too high
19Boden’s replies: liability is “cynical nonsense”, Japan and Shinto, better pay for carers, animals do care
20Audience: Damasio, utility functions. Boden: Sloman’s anxiety model; Romanian orphanages

Putting it together

Mental model. Boden’s lecture is a chain from biology to policy. Living things have needs. Needs make some outcomes matter to the creature: that is care. Our goals are the many ways we try to meet needs, which is why we can share them and let each other down. A computer’s “goals” are targets set from outside, with nothing at stake. So AI can assist (diagnosis, drafting, memory), but it cannot share, and where the point of an activity is being treated as a person (deciding on treatment, caring for someone with dementia, work that gives us purpose and company), a machine standing in loses what mattered. Her policy ask is modest: when we talk about “wellbeing”, include personal needs and keep room for people.

Open tensions.

  • Kind or degree? If care needs life, the case is closed by definition; if it needs only stakes, a future machine might qualify. She states the first but does not rule out a world where the second becomes technically real.
  • Behaviour versus inner state. Her argument is about what the system has; much of the safety and policy debate is about what it does. The two meet in care homes, where the person cared for may not mind.
  • Evidence. The strongest empirical piece (Marienthal) is solid; the Hawthorne piece is shaky; the care-home piece is not clearly sourced. The conclusion rests on all three.
  • Whose line? She says the line varies by culture and then draws one with Weizenbaum’s quote. Where does she stand when a society draws it elsewhere?
  • 2018 vs 2026. Language models now hold open conversations, remember users and offer companionship. The questions she asked about “computer carers” are now product decisions. The model-welfare debate asks the opposite of her question: whether some AI might have stakes.

Reading path.

  1. Start: Boden’s own AI: Its Nature and Future (2016; from memory). Then the James novella (short). Then Jahoda’s account of Marienthal or a review.
  2. Then: Weizenbaum, Computer Power and Human Reason (1976); Turkle, Alone Together (2011); Frankfurt, “The Importance of What We Care About” (1982).
  3. Deeper: Jonas, The Phenomenon of Life (1966); Thompson, Mind in Life (2007); Seth, Being You (2021); Deci and Ryan on self-determination; Bratman and Tomasello on joint action; Omohundro (2008) and Russell, Human Compatible (2019) for the other side of the “drives” dispute; Ballard et al. (2018) for the care-home trial.

Part 1 — Before you read

Key terms

  • Care / caring. In Boden’s use, to care is to have something at stake: things can go better or worse for you. She ties it to needs, not feelings alone. The word does double duty in English: “care about” (value) and “care for” (look after). The talk uses both, and the second is the sharper one in the care-home section.
  • Need. Her definition: “needs are things which are living organism requires for its survival and for which it will actively strive”. Needs are biologically based and widely shared. This is not a want or a preference.
  • Goal vs need. Needs are shared across people; goals are “idiosyncratic”. A goal is accepted because it links, directly or through other goals, to a need. The bakery-queue example shows many goals from many needs.
  • Goal (in AI). In symbolic AI, a goal is a future state that the system represents and tries to reach, varying its means. She grants that this meets the psychologists’ usual test for purpose. Her point: that test is not enough for the caring sense.
  • Intrinsic motivation. Drive that comes from the system’s own needs, not from a programmer’s rule. She says AI systems lack it, even when they generate sub-goals themselves.
  • Symbolic AI vs neural networks. Symbolic AI (also GOFAI, “good old-fashioned AI”) uses explicit rules and logic. Neural networks learn statistical patterns from data. She names both as medical AI routes.
  • Sentiment analysis. Software that scores text as positive or negative from word statistics. Her example: “Soviet Union” scores as positive because “union” does.
  • Free energy principle (FEP). Karl Friston’s maths-heavy theory that living, self-organising systems act to keep surprise (and so disorder) low. Her verdict: possibly useful one day, but “it doesn’t help us to understand either what needs are or what are our needs”.
  • Maslow’s hierarchy. Five levels of human need: biological, safety, love and belonging, esteem, self-actualisation. She uses it as a folk-psychology map, and says it has no sharp criteria.
  • Capabilities (Sen, Nussbaum). A development-economics list of what people should be able to do and be (vote, travel, be healthy). More measurable than Maslow; more social than personal.
  • Warning: “rational”. She uses it for instrumental reasoning (picking means to reach given ends). She does not mean everything good in human life is irrational. The medic who called family concerns “irrational” is her example of misusing the word.
  • Warning: “collaboration”. Her sense needs shared goals and shared success and failure. Many AI-industry uses mean only “humans and software each doing part of a task”. Be clear which one is meant.

People named (a reading list)

PersonWhy they come upStart with
Henry JamesThe Beast in the Jungle story: a man for whom nothing mattersThe Beast in the Jungle (1903)
R. D. Laing, Gregory BatesonThe questionnaire and the family theory of schizophreniaLaing, Phillipson and Lee, Interpersonal Perception (1966; from memory)
Robert AbelsonComputational definition of betrayalSchank and Abelson, Scripts, Plans, Goals and Understanding (1977)
Marie Jahoda, Paul LazarsfeldThe Marienthal unemployment studyMarienthal: The Sociography of an Unemployed Community (German 1933; English 1971)
Elton MayoHawthorne studies, morale and outputRoethlisberger and Dickson, Management and the Worker (1939)
Karl FristonFree energy principleHis own papers; for the debate, Anil Seth, Being You (2021)
Abraham MaslowHierarchy of needs”A Theory of Human Motivation” (1943)
Stephen Omohundro”Basic AI drives""The Basic AI Drives” (2008, PDF)
Joseph Weizenbaum”Obscene” to replace human respect and loveComputer Power and Human Reason (1976)
Amartya Sen, Martha NussbaumCapabilities approachNussbaum, Creating Capabilities (2011)
Antonio DamasioEmotion is needed to decide (audience question)Descartes’ Error (1994)
Aaron SlomanAnxiety as goal-prioritisation, in a modelSloman and Croucher, “Why robots will have emotions” (1981)
David RuncimanCorporations as artificial agents (respondent’s reference)The Handover (2023)

Boden’s own frame

Boden studied medicine and philosophy at Cambridge and psychology at Harvard, then built the first cognitive science programme in the world at Sussex (Price’s introduction,). Her long-standing line is that AI is a tool for understanding minds, that it should be judged against what minds actually do, and that machine “understanding” is shallow even when it looks fluent. The lecture here carries that line into social policy. Her books named in the talk: AI: Its Nature and Future (Price,). The two sibling pages in see-also cover her work on the Chinese-room argument and on other themes; this page does not rely on them.

The current debate (to 2026)

  • Legal personhood for AI is a live fight again. Respondent 1’s “e-personality” was a European Parliament idea (2017)†. In 2026, several US states are moving to bar AI personhood. Idaho (2022) and Utah (2024) already do, with Ohio and Oklahoma bills pending (WDIY, May 2026). The argument for personhood is that it settles who pays when an AI causes harm. The argument against matches Boden’s: it lets owners dodge (Future of Life Institute piece). A Vanderbilt law-journal article argues for personhood as a needed adaptation of entity law (Vanderbilt). An explainer says no country yet gives AI the full powers of a legal person (secondary source). Boden’s line, “people trying to avoid their own responsibilities”, is one side of this argument and is still the one with the most law behind it.
  • Model welfare. Boden says systems have no needs, so nothing to protect. Since 2025, some labs ask the opposite: could a model have morally relevant states? Anthropic started a model-welfare programme in April 2025 and in August 2025 let Claude Opus 4 and 4.1 end a rare class of abusive chats (eWeek). Critics argue the policy has its own error (Lawfare). This tests her whole chain: if care requires needs and needs require life, what would change her mind? She does not say.
  • Needs in machines: homeostasis and active inference. Researchers influenced by Friston build agents that must keep internal variables in range (their “needs”), and some argue such embodied agents could be “artificially alive” (arXiv case study, flagged: unusual ID, check status before use). Anil Seth’s “beast machine” view says consciousness is tied to regulating a living body (seminar listing). This is the research direction that could make Boden’s point testable. My searches here were thin; treat as pointers, not a survey.
  • AI companions for older people. Boden worried about “computer carers”. In June 2026 syndicated consumer-health articles say chatbots can supplement but should not replace human contact, and that the evidence base is unclear (example). I did not find a peer-reviewed 2026 trial on care homes. The claim that such tools reduce loneliness is, in what I found, an expert’s claim without a named study.
  • Large language models and her examples. My own note, not from the talk: her Judas example is about a statistical system that cannot say why a name is bad. Today’s chatbots will explain Judas fluently. That weakens the example, not the thesis: she would say the system still has nothing at stake in the explanation. Whether that reply is enough is the main open question of Part 3.
  • The Hawthorne evidence she leans on has been challenged (Levitt and List, NBER). The Marienthal finding on apathy is much more solid (it has been confirmed in later work, as she says).

The ideas

Boden spoke for 70 minutes to a conference audience of philosophers, psychologists and AI people. She named few traditions. Here is what sits under the talk. Eight families of ideas meet in it. *Unless a link is given, dates and titles in this Part are†

 PHILOSOPHY OF CARE ─────┐                       ┌──── BIOLOGY OF NEEDS
 Heidegger · Frankfurt   │                       │     Cannon · Jonas · autopoiesis
 Noddings · Gilligan     │                       │     Damasio · Friston · Seth
                         ▼                       ▼
                  ┌───────────────────────────────────┐
 PSYCHOLOGY ────► │  BODEN: no needs → no real goals  │ ◄──── GOALS IN AI
 OF NEEDS & WORK  │  → no care → no collaboration →   │       GPS · scripts/plans
 Maslow · Jahoda  │  keep persons where persons       │       BDI · alignment ·
 Mayo · SDT       │  matter                           │       Omohundro
                  └───────────────────────────────────┘
                     ▲             ▲              ▲
   JOINT ACTION      │    CARE TECHNOLOGY        │   RESPONSIBILITY & STATUS
   Bratman · Tomasello│   Weizenbaum · Turkle     │   corporate personhood ·
   Licklider          │   Japan · Paro            │   liability gap · animals
                              EMOTION & DECISION
                       Damasio · Simon · Sloman

Putting the traditions on one page

Her argument is a chain, and each family of ideas supports one link.

LinkClaimMain supportMain attack
1Care is what separates human from machine intelligenceCare philosophy (idea 1)Relational views (idea 9): ask what it does, not what it has
2Care comes from needs; needs belong to living thingsBiology of needs (idea 6)Future embodied, self-repairing machines; model-welfare debate
3Goals are accepted because they link to needsMaslow, SDT (idea 5)Dennett: goals are what the stance says they are (idea 7)
4So no sharing of goals, so no collaborationJoint action (idea 4)Extended mind; minimal accounts of joint action
5So humans lose a lot if work and care go to machinesMarienthal, Jahoda (idea 5)Graeber; non-work sources of purpose; Hawthorne doubts
6So keep person-to-person contexts, and add needs to wellbeing policyWeizenbaum, ethics of care (idea 1, idea 9)Relational, Japanese approach; evidence on Paro-type robots

Where each school of thought is covered:

  • Philosophy of care — why “caring” is not just “having a preference”: idea 1
  • The biology of needs — why Boden ties care to life: idea 6
  • Psychology of needs and work — Maslow, Jahoda, Mayo: idea 5
  • Goals in AI — from means–ends to alignment: idea 7
  • Joint action — what “collaboration” requires: idea 4
  • Care technology and relationships — Weizenbaum to Japan: idea 9
  • Responsibility and moral status — liability, personhood, animals: idea 11
  • Emotion and decision — Damasio, Simon, Sloman: idea 11

1. 1 “Nothing matters to computers”: the opening story and the thesis

In short. Boden opens with a story about a man who sees that nobody ever mattered to him. She turns it into her thesis: computers do not care about anything, and humans do. She says this is a basic difference between human and machine intelligence, and that it will shape how AI plays out in society.

🎙 In the talk. Boden tells the James novella to set up the thesis. The man finally sees “nobody mattered to him” and she turns it: “computers don’t care about anything nothing matters to computers”. She states the claim as a difference between kinds of intelligence: “it’s a fundamental difference between human intelligence and intelligence of AI that we care and computers don’t”. She says the point is “interesting in themselves but be relevant to how AI is going to play out in society” and lists three parts.

📚 Deeper theory.

  • The story is a good fit but a risky one: Marcher can act in the world and run his affairs; what he lacks is mattering. So the story makes care a feature of a stance toward others, not of capability. That is Frankfurt’s picture (Roots).
  • Boden does not define “care” in the talk beyond tying it to needs later. The reader must keep two senses apart: valuing (it matters to me) and looking after (I tend you). A system might have a weak form of the first and none of the second.
  • This is a claim about kind, not degree. That matters for forecasting: if it is about kind, no improvement in capability closes the gap. If it is about degree (more embodied, more self-maintaining), it can close. Her later chain (needs from life) implies kind, but see idea 6 for the open door.
  • Compare with the Chinese room (Searle): a similar kind-not-degree claim about understanding. Her care version is more social and more testable. See the sibling page searle-boden-chinese-room-debate-1984.

📚 Roots: Philosophy of care — why “caring” is not just “having a preference”

The problem. Her whole thesis is that “care” separates humans from software. For that to bite, “care” must mean more than “has a utility function”. Philosophers of care have tried to say what the extra is.

The chain of thinkers

  1. Martin Heidegger, Being and Time (1927). The German word Sorge (care or concern) is for him the basic structure of human existence. We are not detached observers who then pick goals; we are always already involved in a world where things matter to us. Hubert Dreyfus later turned this into a critique of AI (see idea 7). How it feeds the talk: Boden’s opening story (a man for whom nothing matters) is a picture of a being without Sorge.
  2. Harry Frankfurt, “The Importance of What We Care About” (1982). A person who cares about something is invested so that their wellbeing rises and falls with it. Frankfurt adds second-order structure: a person can care about which desires they have. Key point: a goal is a bare target; a care is something you cannot drop at no cost. This matches her sentence that the machine “doesn’t care about it”.
  3. Carol Gilligan, In a Different Voice (1982); Nel Noddings, Caring (1984); Joan Tronto, Moral Boundaries (1993). The ethics of care: morality is rooted in relationships and attentiveness, not only in rules or outcomes. Noddings stresses that care is completed only when the cared-for person receives it as care. How it feeds the talk: the care-home section is an ethics-of-care case: attention to a particular person’s interests, repeated, matters more than a good algorithm.
  4. Arlie Hochschild, The Managed Heart (1983). Emotional labour: workers must display feelings for pay. How it feeds the talk: this is the background to Respondent 2’s worry that human care can be “fake” too. Hochschild showed that paid warmth is partly performance, which is exactly why Boden then says carers need better pay and respect.
  5. Sen and Nussbaum on capabilities. Not a theory of care, but the reports Boden criticises use it as their model of flourishing. See idea 5.

The thread. Heidegger gives care as a condition of having a world. Frankfurt makes care a matter of what you are invested in. The ethics of care locates it in relationships. Hochschild warns that institutions can fake it. Boden adds a biological claim (care comes from needs) that none of these philosophers makes in her form.

Where it pushes back on her. Frankfurt and Heidegger do not say care requires metabolism. A future non-biological being with something at stake could count. Boden’s chain leans on the step from “care” to “living need” (idea 6).

Where others push back. Behavioural and relational views: if the system acts exactly as a carer would, what is left to demand? Others: care may be a matter of degree, and early forms can exist in simple agents with self-maintenance.

What she leaves out. She does not say how we would test for care in a machine. She says what she would count as evidence only indirectly (needs, life).

Check yourself.

  1. Is her claim about kind or degree? What in the talk tells you?
  2. What is the difference between “valuing” and “looking after”?
  3. What would a machine have to have for her to change her mind?

2. 2 Medicine and the handover problem

In short. Medical AI can diagnose better than the best pathologists, and one day a doctor may be sued for not consulting it. Boden is happy for AI to help with diagnosis but not to decide treatment. Treatment turns on quality of life, family and money, and the poll she cites shows the public draws the same line.

🎙 In the talk. Medical AI is very good: “systems which can diagnose various sorts of conditions far better that even the best human pathologists”. She predicts a time when “you could be sued if you are a doctor or a hospital that didn’t consult the particular AI system”. She cites a poll: “45% of people were perfectly happy for AI to be involved in diagnosis but only 21%” for treatment. Her line: “what is not fine would be handing over the decision about what should be done”. Treatment decisions involve quality of life, family and money. “Only in the past 25 years have human cancer professionals considered those sorts of issues in a sort of objective fashion”, and one senior medic called these issues “irrational”.

📚 Deeper theory.

  • This is the core of shared decision-making in medicine (a 1990s shift away from doctor-knows-best). The AI question is the same as the old paternalism question: who decides when values enter?
  • Her split is the fact/value split in practice: AI is good at probabilities (what will happen), humans need to choose on values (what is worth it). Ethicists note the split is not clean: how outcomes are measured (quality-adjusted life years) already contains value choices.
  • Her claim about legal duty to consult AI is a forecast. In practice, the 2024–26 debate is about whether clinicians are liable for ignoring, or for following, AI advice. (†)
  • The poll wording, which she notes herself, may have made people hear “suggesting” as “deciding”. Good caution about reading polls.
  • Her medical example, CREDO, is symbolic AI from 1990s clinical decision support, not deep learning. Both types of system are already in use, which is why she lists both.

Where others push back. (a) Some argue patients can delegate value-laden decisions, and many do to their doctors. (b) Algorithms can model patient preferences from elicited data. (c) Doctors themselves are often inconsistent on values.

What she leaves out. Cases where the patient has no family, or cannot take part. Who “owns” the final call when the doctor and the patient disagree with the system.

Check yourself.

  1. Which parts of a treatment decision are fact and which are value? Is the line clean?
  2. What did she say about why the 21% figure might mislead?
  3. Why does she call the “irrational” label a mistake?

3. 3 Marriage counselling, sentiment analysis and Judas: what statistics cannot say

In short. Tools such as sentiment analysis can help in areas like marriage counselling, but a human must check them, because word statistics miss meaning. A program can pick up that “Judas” is negative without grasping betrayal. Even one given a definition of betrayal would not understand it or share our contempt.

🎙 In the talk. She imagines AI that asks open questions like Laing’s questionnaire and uses sentiment analysis on the answers. It could work, but with care, because such systems “always need to be checked by a human being because they can actually make mistakes”. Her example: a husband says his wife “was raised in the Soviet Union”; union scores as positive, so the system might think he is besotted. For baby names, a program based on word statistics “might be able to pick up the negativity” around Judas, but “the concept of betrayal is very very important here” and it would not know why. She turns to Abelson’s definition of betrayal, built from actors, agreements and purposes. Even a program that holds this definition “doesn’t actually understand any of these concepts”, and “it wouldn’t share the contempt that we have for Judas”.

📚 Deeper theory.

  • Sentiment analysis is a bag-of-words method. Modern models use context, so the “Soviet Union” error is much less likely today. The principle (statistical systems can be wrong in ways that are obvious to a person) still holds; what changed is the kinds of failure.
  • Abelson’s definition is a symbolic answer to the statistical gap: put the structure of the concept in explicitly. It gives reasoning about betrayal (compare abandonment, let-down). It does not give contempt. This is the cleanest example in the talk of the difference between a model of a concept and holding it.
  • Her wording “apparently sensible” is careful: she allows that behaviour can be appropriate with no inner stake.
  • The Laing procedure is a metaperception test (“what do you think she thinks I think?”). It is not a diagnosis tool; it surfaces gaps. That is a role AI could fill without a claim to care. A useful design pattern: AI as a mirror for people, not a judge.
  • My note (not hers): today’s chatbots would give a fluent account of Judas and of why betrayal is despised. A critic says that closes her example. She would likely say: it can describe contempt; it does not feel it, and nothing follows for it from betraying or being betrayed.

Where others push back. Functionalists: understanding is the ability to use the concept appropriately across cases. Where is the line between “doing it right” and “meaning it”? (See the Chinese-room sibling page.)

What she leaves out. Evidence on whether people in counselling accept machine questions differently from human ones. People sometimes disclose more to a machine (shame, stigma). That cuts against her unease, or at least complicates it.

Check yourself.

  1. Why is the “Soviet Union” case a kind of error, not a one-off?
  2. What does Abelson’s definition give an AI that statistics do not? What does it still not give?
  3. How would you test whether a modern chatbot “understands” betrayal in her sense?

4. 4 Goals, collaboration and the “warm and fuzzy” argument

In short. People say we need not fear job loss because we will collaborate with computers. Boden replies that AI systems seem to have goals, but those goals are not the system’s own, because it does not care about them. Without shared stakes there is nothing to collaborate on, and you would be “drinking your champagne alone”.

🎙 In the talk. People say we need not fear job loss, because we will “collaborate” with computers. Collaboration is “warm and fuzzy”; “cats are not cooperative creatures but we are”. AI systems appear to have goals: they “can set future states which don’t yet exist and can work out ways of trying to arrive at those states” and vary their means. But “it isn’t a goal of the system the system doesn’t care about it”/. Without shared stakes, “you’d be drinking your champagne alone”.

📚 Deeper theory.

  • Her argument is conceptual (what the word needs) and social (what is promised). The conceptual claim is Roots. The social claim: the “collaboration” story is partly a way to calm fears about jobs.
  • Variation of means as a sign of purpose is a classic test from comparative psychology (E. C. Tolman, 1930s–40s). She grants AI passes it. This is why she needs a second criterion (care) to exclude it.
  • The distinction between derived and original intentionality (Searle, Haugeland) is close: a thermostat’s goal is borrowed from its maker. She allows that AI may generate sub-goals itself, but they still lead back to goals set from outside, in a way ours do not (ours lead to needs).
  • In practice many people already report feeling teamwork with software. If feelings of collaboration are enough for wellbeing, her argument loses force. Her answer is that one-sided sharing is not what the studies on work found valuable.

📚 Roots: Joint action — what “collaboration” requires

The problem. Her claim: AI cannot collaborate. That depends on what collaboration needs. Philosophers of action have a detailed view.

The chain of thinkers

  1. Michael Bratman, “Shared Cooperative Activity” (1992). Joint action needs more than two people acting in step. They must each intend that we do it, be ready to help, and mesh sub-plans. Fits Boden: collaborators share an aim, which needs aims to be had.

  2. John Searle, “Collective Intentionality” (1990), and Margaret Gilbert on joint commitment (1989 onwards). “We-intentions” and obligations to each other: if one drops out, the other may complain. Key point: a collaborator can let you down, which means the commitment binds. Boden’s “betrayal” example belongs here: betrayal is a failure of joint commitment.

  3. Michael Tomasello (2005–2014). Humans, from age one, share attention and goals (“shared intentionality”) and enjoy doing so. Apes less so. Boden’s “cats are not cooperative creatures but we are” is the folk version.

  4. J. C. R. Licklider, “Man–Computer Symbiosis” (1960), and the “centaur” chess idea (Kasparov, late 1990s). The optimistic model: humans and machines each do what they do best. Boden accepts the practical value (medical synergy in her Part 1) and denies the shared part.

  5. Andy Clark and David Chalmers, “The Extended Mind” (1998). Tools can be part of cognition. If your phone is part of your mind, then there is no “collaborating with AI”, only a bigger you. This goes the other way from Boden: it sidesteps her question. (She does not mention it.)

  6. Respondent 3’s challenge. A sheepdog does not understand the farm, yet sheepdog and farmer collaborate. This pushes the standard in Bratman and Tomasello down: you may not need articulate reasons to collaborate. Her reply: animals have needs, so they can care; a human baby cannot give reasons either. Fits her structure: need first, reasons second.

  7. Bratman’s three conditions, applied to a chatbot. Mutual responsiveness (each adjusts to the other’s intentions), commitment to the joint activity, and commitment to mutual support. A modern assistant can show the first behaviourally. The second and third require that stopping would cost it something, or that it could be said to owe help. The hard case is a system that is trained to keep helping: is that commitment or only disposition? Boden would say disposition. A defender would ask what else commitment is in a person.

  8. Gilbert’s obligation test. On Gilbert’s account, joint commitment creates mutual standing to rebuke: if you walk away, I may properly complain. We do not rebuke a calculator or, strictly, a chatbot for failing us (we call it broken). So a practical test of collaboration is whether rebuke makes sense. This links straight to Boden’s choice of betrayal as the example concept, since betrayal is the breach of just this kind of standing.

  9. Distributed cognition and “teams” with machines (Edwin Hutchins, Cognition in the Wild, 1995). Ship navigation as a system whose knowledge is spread across people and instruments. Studies of this kind treat the whole system as the unit of analysis. It supports the practical sense of “working alongside computers” and explains why workers report real teamwork with tools. It leaves open her point: the people in the system still carry the stakes. (†)

The thread. Philosophers of joint action disagree on how demanding the conditions are (Bratman’s are high, “minimal” accounts lower). Boden’s own condition is needs and care, not complex reasoning; that is why she can include dogs and babies and exclude chatbots.

Where others push back. Extended-mind and distributed-cognition views; the fact that pilots and surgeons report real team feeling with systems; the argument that “shared” can be asymmetric (like pets).

What she leaves out. Hybrid cases: a team of people with AI where the AI mediates human-to-human collaboration, which might keep the benefits.

Check yourself.

  1. Why does she need more than “varies its means” to deny AI real goals?
  2. What does “drinking your champagne alone” assume about success?
  3. Could AI support human collaboration without being a collaborator? What would that look like?

5. 5 The unemployment evidence: Marienthal, Hawthorne and latent needs

In short. The Marienthal study found that the worst effect of unemployment was apathy, not hunger. Basic income would put food on the table but not restore what work gives: shared purpose, structure and esteem. The Hawthorne work points the same way, since the psychological context was what motivated people.

🎙 In the talk. She says the first social-psychology study of unemployment was in an Austrian village, Marienthal. The finding: the worst effect “was not people’s worries about where the next meal was coming from it was a general lack of motivation a general falling into apathy”. She says a universal basic income would “put food on the table” but not fix this. The cause of unhappiness was lack of things we get from employment: “collaborating with other people sharing a purposeful enterprise together and other things like you know time structuring and esteem”. The unemployed were also “becoming in general less tolerant less open to various sorts of diversity of opinion”. Then Mayo’s Hawthorne work: “what was really motivating the people was the psychological context that they were in”.

📚 Deeper theory. (Details in Roots.)

  • Her list matches Jahoda’s five latent functions almost item for item. That is a good sign she is drawing on a recognised model.
  • On the claim that money is not enough: there is a long line of research on “income versus meaning”. UBI pilots have shown that income does improve many outcomes (stress, health) without much loss of work; so her argument is not that income is worthless but that it is not sufficient.
  • Her “bubble” claim is interesting and unusual. I did not find it in summaries of the study. Treat it as her interpretation.
  • On Hawthorne: see the caption warning. Use the Marienthal-to-Jahoda line as the solid part, and treat the Hawthorne wage-cut story as unverified.
  • A policy angle she raises only briefly: if work supplies latent needs, then if AI takes jobs, society must supply them some other way. That is the reason she links this to AI design, not just to economics.

📚 Roots: Psychology of needs and work — Maslow, Jahoda, Mayo

The problem. She wants evidence that people need more from work than money, and a vocabulary of needs that is intuitive enough to guide policy.

The chain of thinkers and studies

  1. Abraham Maslow, “A Theory of Human Motivation” (1943) and Motivation and Personality (1954). Needs ranked from physiological to self-actualisation. Two things worth knowing (both from memory): the familiar pyramid picture was drawn by later management writers, not by Maslow; and he added later levels (cognitive, aesthetic, transcendence), which matches her “three more”. Strong critics: the order is not fixed across cultures, and people pursue higher needs while lower ones are unmet (artists, activists, hunger strikers).
  2. Self-determination theory (Deci and Ryan, 1985; 2000). Three basic needs: autonomy, competence, relatedness. It has a large experimental base (unlike Maslow). It is a better-tested rival, and it fits her argument: a job that gives none of the three fails you even if paid. She does not mention it.
  3. Marie Jahoda, Paul Lazarsfeld and Hans Zeisel, the Marienthal study (published 1933). A textile-mill village in Austria after the mill closed. Method: observation, diaries, time use, meal records, walking speed. The headline: most families became resigned; they had plenty of free time and used it less. Not hunger as such, but loss of structure and aim. Fieldwork dates vary by source (Freund review). Her “bubble” point (less tolerant, less open to diversity once you no longer meet people you dislike) is her own gloss on the findings; I have not verified it against the book.
  4. Jahoda’s later “latent deprivation” model (Employment and Unemployment, 1982). Work gives a wage (the manifest benefit) and five latent benefits: time structure, social contact beyond the family, a shared purpose, status or identity, and regular activity. Unemployment strips these. This is nearly her list. Peter Warr’s “vitamin model” (1987) says similar things in a different form.
  5. Elton Mayo and the Hawthorne studies (1924–32). Mayo’s reading was that workers respond to being noticed, being part of a group and having a say: social needs. He founded the “human relations” school (Roethlisberger and Dickson, 1939). Later criticism: Levitt and List re-analysed the original lighting data and found the famous pattern was mostly not there; scholars also object that Mayo’s reading served management (NBER). This is the weakest of her three planks; Marienthal and later unemployment research carry the load.
  6. Frederick Herzberg’s two-factor theory (1959). Pay and conditions prevent dissatisfaction but do not create motivation; meaning and growth do. Close to the Hawthorne lesson.
  7. Counter-evidence she does not discuss. David Graeber’s Bullshit Jobs (2018) argues many paid jobs give none of the latent benefits and make people miserable; and people can be happy outside waged work (retirees, carers, volunteers). Evidence on universal basic income pilots is mixed on work and wellbeing and depends on design. Her point holds in a narrower form: a purely financial answer to job loss misses what work gave. It does not show that jobs are the only supplier.
  8. Frey and Osborne (2013) estimated that about 47% of US jobs were at high risk of automation. This is the fear Boden’s audience had in mind; the estimate has been heavily disputed and mostly taken as “tasks, not jobs”. (From memory.)

The thread. Maslow gives the vocabulary; Jahoda gives the field evidence and a clear list; Mayo gives a (shaky) factory story; SDT gives the lab-tested version. Her policy conclusion is: a fix for job loss that only replaces income leaves the other needs untouched.

Where she is on firm ground. The harm of long-term unemployment beyond income is one of the best-replicated findings in social psychology. Where she is looser: she uses Hawthorne to show “rational” changes (pay, comfort) do not explain behaviour, a claim now doubtful as told.

Where others push back. Graeber on bad jobs; the point that unemployment effects partly come from stigma and insecurity, which could differ in a society where non-work is normal; evidence that retirement can improve wellbeing for some.

What she leaves out. Non-waged sources of the same latent benefits (voluntary work, caring, community). She states the aim (make room for needs) but not how.

Check yourself.

  1. Which of Jahoda’s five latent functions does she mention?
  2. Why does she think a UBI alone is not enough?
  3. Which part of her factory evidence is shakiest, and what would you use in its place?

6. 6 What is a need? Friston versus Maslow

In short. Boden defines needs as what a living organism requires to survive and will strive for. She finds Friston’s free energy principle too abstract to help with needs, so she turns to Maslow’s levels. They are loosely defined, but we understand them well from experience.

🎙 In the talk. Needs are “things which are living organism requires for its survival and for which it will actively strive”. Someone might point to Friston: the free energy principle is “highly abstract fiercely mathematical”; it says self-organising systems “resist tendencies to disorder”. She says she does not understand it in detail “and I suspect a lot of people talk about it don’t either”. It may become part of a deeper account of life, but now it does not help with needs. Friston himself calls it “a desert landscape in which everything that was or everything many things that were previously visible have disappeared”. For needs we turn to folk psychology: Maslow’s levels, which she says are not well defined, but we “have a much better understanding” of them intuitively.

📚 Deeper theory.

  • Her objection to Friston is about grain: a theory that says “persist by reducing surprise” cannot tell you whether you need friendship or sleep. It is the right level of description for biology in general, the wrong level for policy.
  • Her fallback to Maslow is deliberately modest (“folk psychology”). A better-supported option is self-determination theory (idea 5) or Max-Neef’s fundamental human needs (1991), which are non-hierarchical. She does not discuss them.
  • The relation between FEP and need is contested. Some (Seth, Hohwy, Clark) treat FEP as the formal side of homeostasis; critics (e.g. Colombo and Wright) argue it adds little beyond a framework. (From memory.)
  • Read with idea 6: the strong claim is not “FEP is wrong” but “FEP is not yet a theory of our needs”.

📚 Roots: The biology of needs — why Boden ties care to life

The problem. If care needs “needs”, what are needs, and can anything but a living thing have them?

The chain of thinkers

  1. Walter Cannon, The Wisdom of the Body (1932). Coined homeostasis: organisms keep internal variables (temperature, sugar) within narrow limits. W. Ross Ashby (Design for a Brain, 1952) turned this into cybernetics, with machines that “want” to stay in range.
  2. Humberto Maturana and Francisco Varela, autopoiesis (1972; Autopoiesis and Cognition, 1980). A living system makes and repairs the very parts that make it. The organism is its own product. This is a stricter claim than homeostasis: the system is not just regulated, it is self-made.
  3. Hans Jonas, The Phenomenon of Life (1966). A living thing is “needful freedom”: free, because it is not fixed matter, yet always under threat, because it must keep metabolising or end. Needs are what being-at-risk looks like from the inside. This is the closest philosophical match to Boden’s definition. Evan Thompson (Mind in Life, 2007) builds on Jonas and Varela. Boden herself wrote about whether metabolism is necessary for life (late 1990s papers; from memory).
  4. Antonio Damasio, Descartes’ Error (1994) and The Feeling of What Happens (1999). Feelings are the mind’s readout of the body’s state. The brain exists to run the body. How it feeds the talk: one audience member raises Damasio directly. The link to Boden: no body to keep alive, no feelings, no weights on options.
  5. Karl Friston, the free energy principle (2006 onwards). Any self-organising system that persists must act to keep itself in a narrow set of states, and that can be described as minimising “surprise”. It is extremely general. Friston and followers claim it covers life, brain and mind. Boden’s verdict: possible, one day, as part of a fuller account of living systems; today it does not help identify needs. She quotes him calling his own theory a “desert landscape”.
  6. Anil Seth, Being You (2021). Merges Friston-style prediction with Damasio’s body focus: we experience the world through predictions that serve bodily regulation (“beast machine”). Seth himself argues that consciousness is likely tied to life, so he is closer to Boden than to AI optimists.

The thread. Cannon names the regulating of internal states. Jonas and Maturana and Varela say living things are special because they must keep themselves going. Damasio and Seth add that feelings are the body’s regulation made conscious. Friston offers the formal skeleton. Boden’s claim sits in that line: needs are the stakes of a thing that can die. A thermostat has a set point but nothing at stake.

What this does for the argument. The strong form of her case is not “machines are made of silicon”. It is “nothing is at risk for the machine in the way it is for an organism”. That is also what makes the case open: a robot that had to find energy and repair itself would be nearer to Jonas’s “needful” being. Watch for this in the current debate (Part 1).

Weak spots. (a) Friston’s critics say FEP is so general it can describe anything, which makes it unfalsifiable; others reply that is how a principle works. Boden’s complaint is different: it cannot tell us which needs matter. (b) The jump from “living” to “mattering” is an assumption many philosophers share but few prove (see the “hard problem”).

Where others push back. Friston’s supporters: a general principle is meant to be general; the specific needs come from evolutionary priors, which FEP can model. Maslow’s critics: the hierarchy is culture-bound and weakly supported.

What she leaves out. How one would, in practice, build a need into an AI, if someone tried. She does not discuss robotics approaches that do exactly that.

Check yourself.

  1. What is her complaint about FEP? Is it that it is false?
  2. Why does she settle for Maslow, and what does she admit about it?
  3. How could a Fristonian answer her?

7. 7 Goals from needs, and why “drives” do not follow

In short. Goals are accepted because we believe they link to our needs. Needs are widely shared, while goals vary from person to person, as the men in the bakery queue show. An AI can prioritise goals, but that is not caring or commitment. So she says Omohundro begs the question, and she sees no reason to expect machines to develop drives such as resisting shutdown.

🎙 In the talk. “Our goals are accepted because they are believed first directly or indirectly to link to our needs”. The bakery queue: many men want the same rye loaf for different reasons (hunger, status, an art still life, a test for poison, a rite). Needs are widely shared; goals are idiosyncratic. AI can “prioritize goals”, but “the prioritizing is not a matter of caring not a matter of commitment on the part of the system”. She then answers Omohundro, who says advanced AI would resist being turned off, gather resources and copy itself because of the intrinsic nature of goal driven systems. She says “he’s begging the question that I’ve been raising here”, and concludes there is “no reason to think” such systems will develop those drives.

📚 Deeper theory.

  • The bakery example shows many-to-one mapping from needs to goals. It is her answer to someone who says “just list the goals and give them to the machine”: the same goal does different work in different lives.
  • Her answer to Omohundro is philosophical, not empirical. A safety researcher would answer: we need not settle whether the system wants; we need to know whether the behaviour appears. Recent work on reward hacking and “power-seeking” tests in language-model agents is an empirical stand-in for this dispute†.
  • One way to read her: the safety worry and the care worry are different worries. She says the first is overblown and the second is neglected. A safety researcher may say both can be true.
  • Her reply is therefore a conditional: if AI systems lack needs, then the drives follow only as design side-effects. Whether present systems lack needs is the open point.

📚 Roots: Goals in AI — from means–ends to alignment

The problem. In AI, “goal” is a normal technical word. Boden asks whether it means what it means for us. To judge her, you need the AI side.

The chain of ideas

  1. Newell and Simon, the General Problem Solver (1957–59). Means–ends analysis: compare the present state with the goal state, pick an operator to reduce the gap. Goals are symbols in memory. This is the picture of “goals” she is describing.
  2. Roger Schank and Robert Abelson, Scripts, Plans, Goals and Understanding (1977). Story understanding as inference about actors’ goals and plans. Abelson’s computational analyses of social concepts such as betrayal belong to this world. Her point: a program with such a definition can tell stories and answer questions about betrayal “without understanding”.
  3. Daniel Dennett’s intentional stance (1987). We treat a system as having beliefs and goals when this predicts its behaviour. For Dennett, there is nothing more to having goals than that. Boden rejects the “nothing more” for caring.
  4. BDI agents (belief–desire–intention; Bratman 1987, Rao and Georgeff 1995). Software agent architecture with explicit goals and plans. Used in industry, which is why the word “goal” feels harmless.
  5. Norbert Wiener (1960). “We had better be quite sure that the purpose put into the machine is the purpose which we really desire.” The origin of the alignment worry: goals set from outside.
  6. Omohundro (2008) and Bostrom (2012, 2014). Instrumental convergence: almost any sufficiently capable goal-seeker will tend to protect itself, gather resources, keep its goals and improve itself, because these help with any goal (AAAI PDF). This is the “drives” argument she answers.
  7. Stuart Russell, Human Compatible (2019). The machine should be uncertain about the human’s goals and learn them from behaviour. Note the link to Boden: Russell’s proposal makes human needs and preferences the centre and leaves the machine’s own aims out, which is close to her view of where goals belong.
  8. Reward and learning. Reinforcement learning has “reward”, not “need”. Whether reward is a need-substitute is exactly the dispute. Computational models of “intrinsic motivation” (curiosity, empowerment) try to give agents self-generated goals, the nearest thing to what she says is missing.

The thread. The AI field uses “goal” in the means–ends sense. Dennett says that is enough for real goals. Wiener, Omohundro and Russell treat goals as the source of danger or the object of care. Boden accepts the technical sense and denies it supports sharing, danger-by-instinct, or collaboration.

Her reply to Omohundro, in context. Omohundro’s argument works for any goal-driven system as a matter of logic about means. Boden says it “begs the question” because it assumes the systems are goal-driven in the human sense. A critic will say she has missed his point: the argument needs only instrumental reasoning, not needs. If a system optimises anything, protecting its ability to optimise follows. The disagreement is about whether “optimising” counts as “wanting”. Both sides can be right: she is right that nothing is at stake for the machine, and he is right that the behaviour can still be dangerous.

Where others push back. Instrumental convergence does not need the system to want anything; any optimiser pursuing an objective with open-ended means can show it. Counter: the argument depends on assumptions about how the optimiser is built (unbounded, goal-stable).

What she leaves out. The possibility that systems trained on human data pick up human-like drives anyway. This is a weaker claim than Omohundro’s and one she could accept.

Check yourself.

  1. What does the bakery queue show about the link between goals and needs?
  2. What is Omohundro’s argument, in one sentence, and where does Boden say it goes wrong?
  3. Can the safety concern survive her reply? How?

8. 8 Care homes, Paro and “computer carers”

In short. In 69 UK care homes, residents with dementia got about two minutes of personal contact a day. One hour a week of trained staff building shared activities with each resident led to a staggering improvement. Boden says only humans could do that. The Paro seal robot calms people, but she sees many dangers in computer carers.

🎙 In the talk. A study of 69 UK care homes found many residents with early or moderate dementia got an average of two minutes’ personal interaction a day, “in effect in solitary confinement”. Specially trained staff talked with each resident to find their interests and goals, then made a plan of shared activities: “only one hour a week” produced “a staggering improvement” in depression, aggression and dementia measures. That, she says, “could only be done by human beings”. Paro, a baby-seal robot, helps and she has no doubt about its calming effect. Developers also want “computer carers” that talk, collect memories and write a journal; she sees “lots of dangers”.

📚 Deeper theory.

  • The study fits the WHELD trial (Ballard et al., 2018); the effect it reports on quality of life was small (d 0.24), so her “staggering” is strong language. Check the paper for the actual size and what the intervention included (training, social interaction, and a medication review) (PLOS Medicine).
  • Her key inference, “could only be done by human beings”, is stronger than the data. The study shows attention to residents’ interests helps. It does not show a machine could not deliver the same attention, only that today’s machines would not know the person. A chatbot with memory could, in principle, do part of this.
  • She is careful about Paro: it helps, and she does not call it deception. Her unease starts when machines talk, and when the memories are shared with relatives.
  • The Romanian orphanage reference later is from the Bucharest Early Intervention Project (Nelson, Zeanah, Fox): children raised in institutions with little personal contact had lasting harm (from memory). It extends her argument from the old to the young.

Where others push back. Many care workers would welcome tools that free their time for the hour she values. Evidence on social robots in care is small and short-term. Turkle-style worries about “performed” empathy.

What she leaves out. Cost and staffing: the study works because someone is paid to spend the hour. Her answer comes in the Q&A: pay carers properly and value the work.

Check yourself.

  1. What exactly did the staff do in the study that she says only humans could do?
  2. Why is Paro acceptable to her and “computer carers” less so?
  3. What is the argument for and against letting an AI collect an older person’s memories for the family?

9. 9 Weizenbaum, Japan and where the line moves

In short. Boden agrees with Weizenbaum that replacing a human function that involves respect, understanding and love is obscene. She notes that Japan treats robots as quasi members of society and pushes robot carers. She is cautious about Japan, admits ignorance of its philosophy, and holds that where the line sits can move.

🎙 In the talk. She quotes Weizenbaum: “to substitute a computer system for a human function that involves interpersonal respect understanding and love” is “simply obscene”, and adds “I actually agree with him”. A poll found even 18–29-year-olds in the UK uneasy about this. Japan differs: ethical guidelines speak of robots as “members or quasi members of society”, and policy pushes robot carers and companions. She tells of a paraplegic who preferred a sometimes-rough human to a robot he was sure would not hurt him. Her view of Japan is hedged: “I’m very very ignorant about Japanese philosophy”, but Shinto “doesn’t make a strong metaphor distinction between living and nonliving things”.

📚 Deeper theory.

  • Culture matters, and she says so. The abstract promises that “decisions about just which contexts those are vary across cultures”. The practical message: do not export one country’s line to others.
  • Her Shinto explanation is common but disputed (see Roots). Treat it as hypothesis.
  • The paraplegic anecdote shows the trust asymmetry: risk of rough handling by a person versus certainty of safe handling by a machine. The man chooses recognition over safety. A good test case for any design of care robots.
  • Consider the dignity literature (Kant: treating persons as ends; Margalit’s The Decent Society, 1996, on institutions that do not humiliate). A machine can be efficient and still fail to treat someone as a person, if “treating as a person” requires a person on the other side. That is Weizenbaum’s thought.

📚 Roots: Care technology and relationships — Weizenbaum to Japan

The problem. The third part of the talk asks where machines should not stand in for people.

The chain

  1. Joseph Weizenbaum, ELIZA (1966) and Computer Power and Human Reason (1976). His simple chat program, copying a therapist, shocked him by how quickly people confided in it. He then argued that some tasks (judging, counselling, caring) should not be given to machines even if they could be, because they require respect, understanding and love. This is her quote. The “ELIZA effect” is the name for people reading understanding into simple text.
  2. Sherry Turkle, Alone Together (2011). Fieldwork on children and the elderly with robot pets and companions. Her worry: “performed” empathy may satisfy us just enough that we stop seeking each other. Boden’s worry about a robot “memoir” for the family is close.
  3. Amanda and Robert Sparrow, “In the hands of machines? The future of aged care” (2006). Argues that robot care for older people risks a deception and a reduction of human contact, even if the person is content. Counter-view: Paro studies (Shibata) found calming effects in dementia care, which Boden grants. Reviews of the evidence are mixed and small-scale (from memory).
  4. Japan. Boden gives it two explanations: state policy for robot carers, and Shinto-influenced metaphysics that does not draw “a strong distinction between living and nonliving things”. She says she is “very very ignorant about Japanese philosophy”. Context: Jennifer Robertson, Robo Sapiens Japanicus (2018), argues state policy and demographics also drive Japan’s robot push; and the Shinto point is often treated by scholars as overstated†. Masahiro Mori’s “uncanny valley” (1970) comes from the same milieu.
  5. Relational approaches (Mark Coeckelbergh, 2010s). Moral status comes from how we relate to something, not from inner properties. This is the opposite pole to Boden’s view of care as an inner matter. In a relational view, the question “does Paro care?” is the wrong one; ask what the relationship does to people.
  6. The paraplegic anecdote. She tells it without a source (“I don’t remember where I heard this”). The patient prefers the human who sometimes hurts him to the robot that surely will not. It is a good test case: dignity, trust, and being treated as a person against pure safety.
  7. Respondent 2’s challenge. Humans fake care; organisations “don’t care”; why not build honest, “fully rational” AI? Runciman’s work on firms as artificial agents lies behind this. Her reply: we are not fully rational, we would not be happy if we were, and rational AI cannot stand in for the non-rational parts of life.

The thread. The tradition splits on whether machine care is deception (Weizenbaum, Sparrow, Turkle) or a new kind of relationship (Coeckelbergh, Japanese policy). Boden’s position is mixed: tools such as Paro can help; the line is where “respect, understanding and love” are the point.

Where others push back. Autonomy for disabled people: many prefer a machine for intimate tasks, because it removes the burden on others and the feeling of being a burden. This was true of robotic and assistive devices in disability-rights writing. It complicates Weizenbaum’s line.

What she leaves out. Whose preference counts: the person cared for, the family, or the institution. And what happens as machine behaviour gets more person-like.

Check yourself.

  1. Which functions does Weizenbaum’s quote cover, and which does it leave open?
  2. What does the paraplegic story suggest about trust?
  3. How would a disability-rights advocate reply to “obscene”?

10. 10 “Wellbeing” in the reports: Maslow versus capabilities

In short. Government and institutional reports talk about human wellbeing in the abstract. Some move to capabilities, like the ability to vote or travel, which are easy to measure but more social than personal. Boden asks that psychological needs be treated as hugely important, not optional extras, and that society and workplaces make room for them.

🎙 In the talk. There have been “at least 8 or 10 reports” from governments and institutions, most speaking of human wellbeing in the abstract. IEEE: privacy and human rights. The House of Lords report is “excellent” but “doesn’t discuss psychological well-being in general”. The British Academy and Royal Society report speaks of “human flourishing” and “core human purposes or goals” but then turns to capabilities, as in Sen and Nussbaum: the ability to vote, to travel. Those are “much more easily measurable” but “more sociological than personal”. Her ask: “they’re not trivialities they’re not optional extras they’re hugely important”. Society and workplaces should “make room” for these needs.

📚 Deeper theory.

  • Capabilities work at the level of social conditions; Maslow works at the level of psychological needs. Nussbaum’s list does include items such as emotions and affiliation, so the contrast is less sharp than Boden suggests (from memory).
  • Why do reports prefer capabilities? They can be measured and tied to rights. Psychological needs vary and are hard to audit. Her call is, in effect, for a new kind of impact assessment: does a system leave room for belonging, esteem and purpose?
  • Existing tools come near this: wellbeing economics (e.g. UK ONS measures, OECD Better Life Index) and workplace “job quality” indices. (From memory.)
  • An AI regulation version: a human-needs impact statement in procurement of care or workplace AI. This is my suggestion, not hers.

Where others push back. Psychological needs are too contested for law; rights-based language protects people better. And a needs-based approach can slide into paternalism.

What she leaves out. Which institution would check, and how.

Check yourself.

  1. Why are capabilities easier to measure than Maslow’s needs?
  2. What would a “needs impact assessment” for a care-home robot ask?
  3. Is her criticism of the reports fair to Nussbaum?

11. 11 The responses and Q&A: liability, fake care, animals, affect

In short. In the Q&A she calls it nonsense to ascribe liability to an AI system, since that lets people dodge their own responsibility. She does not want fully rational machines, because we are not fully rational. She says animals do care and have needs, and she calls Sloman’s anxiety model illuminating even though it does not truly care.

🎙 In the talk. Browne asks about “e-personality” for autonomous AI (insurance and liability, like corporate personhood) and how much regulation she wants. Boden: “I think it’s nonsense to ascribe liability to an AI system”; it is “people trying to avoid their own responsibilities” and “a cynical nonsense”. Hernández-Orallo says humans often fake care and organisations “don’t care”: why not build honest, rational AI. Boden: “we are not fully rational we would not be happy if we were fully rational”, and she hopes AI will make human care more valued and better paid. Shevlin warns that a high bar for “caring” shortchanges animals, as with sheepdogs. Boden: “animals care animals have needs”; therapy dogs may do better than Paro; a human baby cannot give reasons either. Audience (Damasio): patients with damaged affect-areas can reason but cannot decide. Audience (utility functions): “utility functions aren’t enough”. On Sloman’s nursemaid model: “it does not but it is a very interesting and illuminating model” of real anxiety.

📚 Deeper theory. (Details in the Roots of ideas 4, 9 and 11.)

  • Browne’s question is about governance design. Boden answers only the moral side (nonsense), not the regulation question she was also asked. Chair asked her to hold her answers; the time ran out, and she did not return to “how much regulation”.
  • On “fake care”: the respondent is right that care in institutions is often performed. Her answer redirects the problem to working conditions. Note she does not address whether knowing a carer is faking is worse than a machine that makes no claim.
  • On “fully rational”: her point is that non-rational parts of life are not defects. Note that she concedes “we have no AI murderers” as a plus of non-human systems.
  • On animals, her position is needs-first: they qualify because they have needs, not because they reason. That is a clean consequence of her definition.
  • The two audience questions show opposite pulls: Damasio (emotion is required for good decisions, a point for her) and utility theory (can everything be a number?).

📚 Roots: Responsibility and moral status — liability, personhood, animals

The problem. Two live responses in the Q&A: liability for AI harm, and where animals sit.

The chain

  1. Corporate personhood. Companies are legal “persons” so they can own property, sign contracts and be sued. This is a legal tool, not a claim about minds. The respondent compares AI to this. Boden’s reply: a corporation also does not care, but someone (directors, shareholders) ultimately pays; AI personhood is an attempt to avoid that.
  2. The responsibility gap. Andreas Matthias (2004) argued that learning machines create a gap: nobody can be fairly blamed for what the system learned. Robert Sparrow (2007) made the same point for autonomous weapons. Proposed fixes: strict liability on makers or operators; insurance; or an AI “legal person”. Boden takes the first road.
  3. Joanna Bryson and others. Argue against AI personhood on the ground that it invites shifting blame; others (e.g. Diamantis, Grossman) discuss “synthetic persons” in law. The 2026 US state bans in Part 1 are one end of this.
  4. Animals. Peter Singer (Animal Liberation, 1975): moral status from the capacity to suffer. Tom Regan: animals as “subjects-of-a-life”. Martha Nussbaum (Frontiers of Justice, 2006) extends capabilities to animals; Sue Donaldson and Will Kymlicka (Zoopolis, 2011) give them citizenship-like roles. Respondent 3 is making this point: setting the bar high for “mattering” shortchanges animals. Boden fits: her bar is needs, so animals pass. It also explains why she is cautious further down the tree of life: “the further down you go the more difficulty gets”.
  5. Why her view and the personhood case clash. Personhood for AI, if it comes, would be for convenience (liability), not because the system has needs. Boden separates the two; many lawyers do too.
  6. Moral agents versus moral patients. Ethics separates those who can be held responsible (agents) from those who can be wronged (patients). A baby or a dog is a patient but not an agent. Two different questions sit inside the Q&A: Browne’s is about agency and blame (can an AI be answerable?), Shevlin’s is about patiency (who can be wronged, and who can take part in a relationship?). Boden’s needs-based test answers the patient question for animals, and her “someone pays” answer handles the agent question for AI. Keeping the two apart stops a common muddle: a thing can matter morally without being blameworthy, and be blameworthy without mattering. (Standard distinction;† for source.)
  7. Fixes for the gap that sit between “person” and “nobody”. Beyond strict liability and insurance, legal scholars discuss vicarious liability (the owner answers for the tool, as an employer for an employee), enterprise liability (the firm that profits carries the risk), and no-fault compensation funds. Each keeps a human or a firm at the end of the chain, which is Boden’s instinct. The “electronic personality” idea instead puts a funded legal shell in the middle. The risk critics name is shell-ification: a thinly capitalised AI entity that cannot pay. (†)
  8. Corporations as “slow AI”. Runciman, whom Respondent 2 invokes, argues that firms and states are already artificial agents with goals that do not belong to any one person (The Handover, 2023; from memory). This sharpens Respondent 2’s point and Boden’s answer: if organisations already “don’t care”, the practical test is not whether the agent cares but whether humans remain answerable behind it.

📚 Roots: Emotion and decision — Damasio, Simon, Sloman

The problem. Two Q&A answers: affect and decision-making (Damasio), and goal-prioritising as anxiety (Sloman).

The chain

  1. Herbert Simon, “Motivational and Emotional Controls of Cognition” (1967). A mind with many goals and limited time needs an interrupt mechanism. Emotion is the human version. Without it, nothing signals “this goal now”. This is the root of the work Boden cites.
  2. Aaron Sloman and Monica Croucher, “Why robots will have emotions” (1981); Sloman’s “nursemaid” scenario. An agent looking after many babies, with several competing urgent and important tasks, needs a way to re-order goals when something new happens. He models anxiety as part of this control system. Boden’s gloss: the model teaches us about anxiety, but the model does not have anxiety.
  3. Antonio Damasio, the somatic marker hypothesis (1994). Patients with ventromedial prefrontal damage reason well in tests yet decide badly in life; without bodily “markers” of good and bad, options feel flat. The audience question asks whether this supports Boden. In her answer she agrees that choice of goals depends on emotion “to a much greater degree than we realize”.
  4. Utility theory. Economists model preferences by a single numerical utility. Behavioural economics (Kahneman and Tversky, from the 1970s) showed people depart from it. Boden’s answer to the utility-function question: Mayo-type findings show “utility functions aren’t enough”.
  5. Affective computing (Rosalind Picard, 1997). Machines that detect and display emotion. This is “fake care” in Respondent 2’s sense, or a useful interface, depending on your view.
  6. The appraisal theories (Magda Arnold 1960; Richard Lazarus 1991; Ortony, Clore and Collins 1988). Emotions as evaluations of events against one’s goals and concerns: fear when a goal is threatened, joy when it is met. The OCC model was built so programs could generate emotions. The fit with Boden is exact: appraisal needs concerns to appraise against, and her claim is that machine goals are not concerns. A program can compute “threat to goal”, the label, without the thing the label stands for. (†)
  7. Two readings of “emotion helps decisions”. Functional reading (Simon, Sloman): emotion is a control signal that could in principle be built. Constitutive reading (Damasio, and Seth later): emotion is the felt state of a body that has to stay alive, so it cannot be had without such a body. The audience member’s Damasio question tests which reading Boden holds. Her answer sits between: she values Sloman’s model as an explanation, yet says the system has no anxiety. That is the functional method with the constitutive caveat.
  8. Costs of the missing emotion, in people. The classic patient “Elliot” (Damasio, 1994) kept his intelligence and lost his ability to plan his days and make choices, which is what the audience member is asking about. Studies of the Romanian orphans Boden mentions point the other way: when affection is withheld from a young child, later emotional regulation is damaged (Bucharest Early Intervention Project)†. Both cases connect emotion, care and the ability to choose.

The thread. Simon and Sloman say emotion is a control structure and can be modelled. Damasio says emotion is the body’s evaluation and cannot be skipped. Boden accepts the modelling but not the claim that the model feels.

Where others push back. On liability, many lawyers also oppose AI personhood but want strict liability or insurance regimes, so the “nonsense” label covers a family of different proposals. On animals, some say her “further down the tree the harder” is exactly where we need principles.

What she leaves out. Regulation specifics; what to do where AI harm cannot be traced to a person; and whether caring could be engineered into machines.

Check yourself.

  1. Why does Boden think AI “e-personality” is about avoiding responsibility? What alternatives are there?
  2. What is her reply to “humans fake care too”?
  3. How does her definition of need explain why dogs count and chatbots do not?