Talks

The Future of AI in Law — Frank Pasquale

48 min read

Frank Pasquale · 2023 · 55:36 · ▶ Watch

Summary

Central question. As law adopts AI, which parts of legal work should be computed and which should stay in human, narrative, accountable judgment?

Thesis. Pasquale does not say AI has no place in law. He says three tests must be met first. A system must be validated (does it work, and can anyone check?), unbiased in its data (is the data already tainted?), and narratively intelligible (can a person tell the story of why this happened to this person?). He calls the standing temptation “law being made in the image of computation”.

How he builds it:

  1. Computational thinking. Law is a humanity as well as an engineering problem. “Law as code” works for some things, such as parts of the tax code. Law’s strength is that it adapts to human affairs.
  2. Digitization of judgment. Hiring, content moderation, ICU triage and law all face machine prediction. In some areas an inexplicable but better predictor may be fine. In law, which imposes benefits and burdens, we need explanations.
  3. Criminal justice case. Slobogin argues for more algorithms because human judges are noisy and biased. Pasquale answers with the dirty-data problem, feedback loops, the lack of any real test of risk tools, and secrecy (Loomis).
  4. Automated force. The spring gun, security robots, acoustic drones, killer robots and asylum software. Cheaper, more precise force may be used more widely.
  5. Binaries and data. Core AI work is classifying into two bins. That is easy for dogs and bagels and impossible for “terrorist” or “innocent”. Red-light cameras are the best case for automation, yet even there New York puts a person in the loop.
  6. Affective computing. Reading emotion from faces is attributing, not detecting. It carries bias, and it slides toward remote diagnosis. The Goldwater episode shows why remote diagnosis is a professional taboo.

Close. AI can add much to law when it is transparent and explainable and when judges understand it. The weak human baseline is no reason “to just sweep them aside”. The aim is to augment professional judgment.

Stance. A critical friend, not a rejectionist. He says he is “fine” with facial screening for depression if a doctor uses it to supplement judgment, and he finds red-light cameras “not that objectionable”. He sides with those pushing back against precise automated force.

Map — talking points

#Point
1Introductions: the host, then a long bio of Pasquale
2Plan of the talk. What is computational thinking? Law as engineering vs law as humanity. “Is law computable?“
3Rouvroy’s “algorithmic governmentality”. Colleague’s challenge: “isn’t all thinking algorithmic?” What is the non-algorithmic complement?
4The “AI ultimatum”: Nourbakhsh quote. Robots and drones share our spaces. The right to shoot down a drone
5Narrow AI and the “digitization of judgment”. Hiring, white-font resumes, duct tape on a stop sign, suicide alerts, ICU prediction
6Lipson: “explaining Shakespeare to a dog”. When do we need explainability? Why law differs
7Dreyfus, Collins, Smith: judgment vs calculation. What computers shouldn’t do
8Criminal justice: Slobogin (for algorithms) vs Ferguson (against). Hungry judges, LSU football
9Risk-needs scoring. Dirty data (Richardson, Schultz, Crawford). Contagion across departments. “As much art as science”
10Dehumanization of decision-making. O’Neil, Criado Perez, Broussard, transparency books
11”A risk assessment of risk assessment”. Self-fulfilling prophecy. Credit scores
12Narrative intelligibility (Charles Taylor). Wisconsin v. Loomis. Rule of law
13Automated force. Spring guns. Knightscope. Ruha Benjamin on cheap force. Acoustic drones. Fiction as foresight. Joh and Stop Killer Robots
14Asylum software (IBM). Border cameras
15Binary classification. Dogs vs bagels vs terrorists. The dataset problem
16Red-light cameras. Confrontation clause. Human double-check in New York. Bot-vs-bot enforcement
17Private security embedded in stores. Affective computing. Bias (Rhue). Privacy. Attribution vs inference. Legibility
18Manipulation. Remote mental-health diagnosis. The Goldwater episode
19Conclusion: when AI helps, and the questions to ask. Q&A starts but is not in the recording

Putting it together

Mental model. Pasquale treats law as a human practice whose value lies in reasons a person can contest. Computation can serve that practice where the rules are clear and the stakes low (red-light cameras), or where it supports an accountable human (a doctor, a judge who understands the tool). It fails the practice when it hides its reasons (Loomis), launders biased history as data (“dirty data”), cannot be tested (the risk assessment of risk assessment), turns contested social labels into binaries (“terrorist”), or reads inner states and acts on the guess (affective computing). His three tests are validation, unbiased data and narrative intelligibility. His design principle is to augment professional judgment, not replace it.

Open tensions

  1. The weak baseline. If human judges are noisy and biased, “augment” may protect a poor status quo. He asks for better baselines but gives no mechanism.
  2. Explanation vs accuracy. He accepts inexplicable predictors “in some cases” and rejects them in law. Where the line falls in borderline fields is unsettled.
  3. Human in the loop. The red-light-camera human may be a rubber stamp. Accountability needs time, authority and training, not just a signature.
  4. Date stamp. The talk predates the large-language-model wave in law. Its binary-classification frame fits risk scores better than systems that draft briefs and may invent the cases (see Current debate).
  5. Access to justice. The case for cheap automation rests on people who now have no lawyer. The talk speaks mostly to adjudication and policing.

Reading path

  • Start: this page’s summary, then New Laws of Robotics (2020) for his full argument. A sibling page on that book: pasquale-new-laws-of-robotics.
  • Then: pasquale-machines-judging-humans for the same themes (automated judgment of people) in a longer academic talk, and pasquale-ai-changing-medical-practice for the medical parallel he mentions here.
  • Deeper: Slobogin, Just Algorithms against Ferguson, The Rise of Big Data Policing; Richardson, Schultz and Crawford, “Dirty Data, Bad Predictions”; Kleinberg et al., “Human Decisions and Machine Predictions”; Hart, The Concept of Law, ch. 7; Weizenbaum, Computer Power and Human Reason; Barrett et al. (2019) on facial emotion; Scott, Seeing Like a State.

Part 1 — Before you read

Key terms

  • Computational thinking. Two meanings in the talk. Thinking like a computer (step-by-step, rule-based). Or being able to tell computers what we want. His worry is the first sneaking into law.
  • Law as code / code as law / computational law. Efforts to write legal rules in machine-executable form. “Law as engineering” is David Howarth’s phrase (a 2011 book; year from memory)†.
  • Algorithmic governmentality (Rouvroy and Berns). Governing people by predicting their behaviour from data, in advance, without addressing them as subjects who can argue back.
  • Digitization of judgment. Pasquale’s phrase for machine prediction entering every field where a person once weighed a case: hiring, medicine, policing, sentencing.
  • Narrow AI. Systems that predict or classify within a narrow set of tasks. He says this, not robots, is the real advance.
  • Explainability / intelligibility. Different things. Explainable AI tries to give a technical account of a model’s output. Pasquale’s narrative intelligibility asks for a human-readable story of what happened to a person and why. Watch for this difference in other authors.
  • Risk assessment instrument (RAI). A points-based or statistical tool that scores a defendant’s chance of re-offending. COMPAS is the best-known US example.
  • Dirty data. Training data produced by policing that was itself biased or unlawful. Richardson, Schultz and Crawford’s term.
  • Affective computing. Systems that claim to read emotion from face, voice or text.
  • Attribution vs detection. A system that “reads” your emotion is, he says, attributing an emotion to you. The machine’s label becomes a fact about you.
  • Legibility. From James C. Scott’s idea of states making people readable to rule them. He calls the metaphor “a power play”.
  • The weak baseline. The reply that human decision-makers are already poor, so a mediocre algorithm is an improvement. He takes it seriously and answers it.

People named (a reading list)

PersonWhy they come upStart with
Antoinette Rouvroy and Thomas Berns”Algorithmic governmentality”Their 2013 essay (from memory)
Illah NourbakhshThe “AI ultimatum” quoteRobot Futures (2013)
A. Michael Froomkin and Zak ColangeloRight to shoot down a drone”Self-Defense Against Robots and Drones” (2015)
Hod Lipson”Explaining Shakespeare to a dog”His Columbia robotics lab; press interviews
Hubert DreyfusWhat computers can’t doWhat Computers Can’t Do (1972)
Harry Collins (and Robert Evans)What computers shouldn’t do / expertiseRethinking Expertise (2007); Artifictional Intelligence (2018)
Brian Cantwell SmithReckoning vs judgmentThe Promise of Artificial Intelligence (2019)
Christopher SloboginThe case for risk algorithmsJust Algorithms (2021)
Andrew Guthrie FergusonThe case against big-data policingThe Rise of Big Data Policing (2017)
Rashida Richardson, Jason Schultz, Kate CrawfordDirty data”Dirty Data, Bad Predictions” (2019)
Cathy O’NeilSelf-reinforcing biasWeapons of Math Destruction (2016)
Caroline Criado PerezGaps in data about womenInvisible Women (2019)
Meredith BroussardSkeptic on self-driving carsArtificial Unintelligence (2018)
Charles TaylorNarrative, agency and language; secular and religious humanismA Catholic Modernity? (1999)
Ruha BenjaminPrecise control can widen controlRace After Technology (2019)
Elizabeth JohRobots can’t read intentHer work on policing and robots
Ian McEwanFiction as foresightMachines Like Me (2019)
Lauren RhueEmotion software reads Black men as angrierHer 2018 study of emotion APIs†
Rob HorningChallenges the “legibility” metaphorHis essays from Real Life magazine

Pasquale’s own frame

He is a professor of law who has built a career on opaque power. The Black Box Society (2015) argues that search, reputation and finance run on secret algorithms. New Laws of Robotics (2020) argues for AI that complements professionals instead of replacing them, and revisits Joseph Weizenbaum’s question about what work we should not give to computers (as the introducer says). The introducer calls this “anticipatory social research”. This talk applies that frame to law itself, and it is aimed at a Catholic-thought audience, which is why he reaches for Charles Taylor.

The four “new laws” of his book†: AI should complement, not substitute for, professionals; AI should not counterfeit humanity; AI should not fuel arms races; AI should always indicate its creators’ and controllers’ identity, so that a human is accountable. The talk does not list them.

The current debate (late 2023 to 2026)

The talk is from early 2023, just after ChatGPT. Much has changed.

  • Hallucinated citations became a courtroom problem. In Mata v. Avianca (S.D.N.Y., June 2023), lawyers were sanctioned after filing a brief with cases ChatGPT invented (Wikipedia). Damien Charlotin keeps a public database of such decisions; it had about 116 cases in May 2025 (Simon Willison), and later snapshots cite hundreds more (Stanford CIS). Counts differ by snapshot date. This is a validation failure, one of Pasquale’s three tests.
  • Legal-research tools tested. A Stanford study found that Lexis+ AI and Westlaw’s AI research tool still hallucinated, at roughly 17% and 33% of queries, against 43% for GPT-4, despite “hallucination-free” marketing (arXiv 2405.20362). A later vendor-benchmark report claims general chatbots now match the legal tools, so the comparison is contested (LLRX summary).
  • Evidence rules. The federal Advisory Committee voted 8-1 in May 2025 to publish a draft Rule of Evidence 707, which would test machine-generated evidence under the Rule 702 reliability standard when no expert sponsors it. The comment period ran to February 2026. I could not confirm what happened after (Proskauer; AAJ). ABA Formal Opinion 512 (July 2024) on lawyers’ use of generative AI:†
  • The EU wrote part of this talk into law. The EU AI Act lists AI that assists judicial authorities in researching and applying the law as high-risk (Annex III, point 8; recital 61, Commission AI Act Service Desk). It also bans predicting an individual’s criminality solely from profiling, and emotion inference in workplaces and schools (Article 5; secondary summary). Application dates for Annex III are disputed in the sources I found, with one citing a deferral to December 2027 that I could not confirm. Check the Official Journal before relying on a date.
  • Hungry judges, again. The 2011 study (Danziger, Levav and Avnaim-Pesso, on 1,112 Israeli parole rulings) was challenged for non-random case ordering, and Glöckner (2016) showed by simulation that a rational judge who avoids starting long cases before a break can produce the same pattern, so the effect is probably overestimated (Glöckner paper; discussion). That supports Pasquale’s wariness more than his hedge suggests.
  • Autonomous weapons. The UN General Assembly adopted resolution 79/62 on 2 December 2024 by 166 to 3, and a further resolution in December 2025 (164 to 6, per the UN Project listing). Neither is a treaty (Human Rights Watch; UN Project).
  • Emotion recognition. Microsoft said in 2022 that it would retire emotion-inference features from its face service†. The EU ban above is the larger change.
  • Risk assessment tools. State-level use, and the status of Loomis-style challenges, since 2023: not checked in this session.
  • What has not changed. The weak-baseline argument, the opacity of vendor tools, and the pull of speed and scale remain the live fault lines.

The ideas

Pasquale spoke for 50 minutes to a general audience, in the manner of a survey. He names few thinkers, but eight traditions run through the talk. This section is from memory unless a link is given.

 LEGAL THEORY                        PHILOSOPHY OF JUDGMENT
 formalism·realism·open texture      Aristotle·Kant·Arendt·Dreyfus
 rules vs standards·law-as-code      Smith·Collins/Evans·Polanyi
          \                              /
           v                            v
        +-------------------------------------+
        |  PASQUALE: what in law can be       |
        |  computed, and what must stay a     |
        |  human, narrative, accountable act? |
        +-------------------------------------+
          ^        ^          ^           ^
 ALGORITHMIC     DUE PROCESS   FORCE &     AFFECT & LEGIBILITY
 GOVERNANCE &    & LEGITIMACY  ROBOTS      Ekman·Barrett·Scott
 FAIRNESS        Weber·Fuller  Weber·IHL   Picard·Goldwater rule
 Harcourt·ProPublica Mathews·Citron·Taylor  Stop Killer Robots
 Kleinberg·O'Neil    Loomis·GDPR
                              PROFESSIONS & COMPLEMENTARITY
                              Weizenbaum·Susskind·Autor·Licklider

Where each school of thought is covered:

  • Legal theory — can law be computed?: idea 1
  • Philosophy of judgment — calculation is not judgment: idea 2
  • Algorithmic governance, bias and fairness: idea 3
  • Due process, legitimacy and the Catholic thread: idea 5
  • Force, robots and the law of self-defence: idea 4
  • Affect, legibility and remote diagnosis: idea 6
  • Professions and complementarity — augment, don’t replace: idea 2
  • Two short threads: idea 3

1. Computational thinking and law’s resilient fragility

In short. Pasquale asks whether computational thinking means thinking like a computer or being able to talk to machines and make them do what we want. The second can help law. The danger is law “being made in the image of computation”. Some parts, such as tax, may be fully computable, but law is resilient because it adapts to human affairs and free will. He wants to ask what non-computational thought can add to algorithms.

🎙 In the talk. He starts from a question: “does computational thinking mean thinking like a computer”, or does it mean being able to talk to machines and get them to do what we want? The second, he says, can bring a lot to law. The danger is that we end up with “law being made in the image of computation”. He sets himself against those who call law a form of engineering (David Howarth, “law as code”). Some parts, such as the tax system, may be fully computable. But law is “resilient” because it adapts to human affairs and human free will, and he wants to ask what non-computational methods of thought and practice law has.

A colleague challenges him: “isn’t all thinking algorithmic”? Is he advocating irrationalism? His answer is a programme: ask what non-algorithmic thought is, so it can complement algorithms. He frames the urgency as the “AI ultimatum”: Nourbakhsh predicted that “we will need to read what they write”, and by early 2023 half of what he reads on Twitter and Mastodon is people’s ChatGPT output.

📚 Deeper theory

  • Hart’s open texture (Roots) is the strongest support. Law’s rules have a clear core and a penumbra where choice is unavoidable. A system that handles the core perfectly still fails at the penumbra, and the penumbra is where litigation lives.
  • The colleague’s challenge has a precise answer. “Algorithmic” in the strict sense (a finite, rule-governed procedure) and “reasoned” are not the same. Aristotle’s phronesis and Kant’s reflective judgment (idea 2) are reasoning that does not reduce to a procedure. So he is not an irrationalist; he holds that some rationality is not computation.
  • “Resilient” law is partly an argument from legitimacy. Law’s ability to bend (equity) keeps it acceptable when rules give unjust results. Strictly coded law is brittle: a loophole or an absurd result is not fixed until someone rewrites the code.
  • Lessig’s “code is law” (1999) is the mirror image. If software already regulates, then putting law in code is a continuation of an existing trend. Pasquale’s worry is about whose values enter, and whether anyone can contest them.
  • The “AI ultimatum” is a form of Dreyfus-style argument turned inside out: not “machines can’t”, but “machines will, and law must decide terms”.

📚 Roots: Legal theory — can law be computed?

The problem. Law looks like rules applied to facts, so it looks computable. Does it work that way?

  1. Leibniz (17th century), calculemus. The dream of settling disputes by calculation: “let us calculate”. The oldest ancestor of law-as-code.
  2. Legal formalism and its realist critics. Classical formalism (late 19th century) treated law as a logical system. Oliver Wendell Holmes Jr. answered in 1881 that “the life of the law has not been logic: it has been experience”. Jerome Frank, in Law and the Modern Mind (1930), pressed the realist point that judges are people, and the line about “what the judge ate for breakfast” is attached to this tradition (attribution from memory)†. Link: the hungry-judges study is a modern, quantified version of realism. Realists used it to say law is not mechanical. Slobogin uses it to say so replace the human. Same fact, opposite conclusion.
  3. H. L. A. Hart, The Concept of Law (1961). Rules have a clear core and an open texture at the edges (“no vehicles in the park: what of a toy car?). Hard cases are exactly where a rule gives out and judgment must act. This is the best statement of why Pasquale thinks law is resilient”.
  4. Ronald Dworkin, Law’s Empire (1986). Law is an interpretive practice that seeks the best moral account of the record. That needs narrative coherence (“law as integrity”), which links to the Taylor thread.
  5. Rules vs standards (Duncan Kennedy 1976; Louis Kaplow 1992). Rules are cheap to apply and costly to write; standards are the reverse. Machines suit rules. Many legal questions use standards (“reasonable”, “undue hardship”), and a standard is a request for judgment.
  6. Lawrence Lessig, Code (1999). “Code is law”: software architecture regulates as law does. Pasquale reverses the direction, worrying that law becomes code.
  7. Computational law. Michael Genesereth and Stanford CodeX; Harry Surden on “computable contracts” (2012); Deakin and Markou (eds), Is Law Computable? (2020), whose foreword Pasquale wrote.

Feeds the talk. The first third. Some law is very computable (tax rules, speeding fines), and some law is open-textured on purpose. His red-light-camera concession marks the computable end.

Where others push back.

  • Computational-law advocates say precision helps ordinary people: contracts and regulations that are clear enough to compile are clearer to read. They do not claim to replace judges.
  • Some jurists argue that all legal reasoning is already rule-bound and so formalisable in principle; the dispute is empirical.
  • Susskind-style critics say the “resilience” of law is partly a story told by a profession that benefits from it.

What he leaves out. What he means by non-algorithmic thought in detail. He promises to “get into some of the ways”, but the remainder of the talk is mostly critique of specific systems. The positive account (equity, narrative, judgment) appears in pieces: the Taylor passage, the human double-check, the close.

Check yourself

  • Why might a rule-based system do well on the core of a legal rule and badly on its edge? Which jurist gives the standard account?
  • What is the difference between “not algorithmic” and “irrational”? Give an Aristotelian example.
  • Is the “AI ultimatum” an argument for or against adopting AI? Explain.

2. The digitization of judgment and the case for explanation

In short. He calls the wider trend “the digitization of judgment”, with examples from hiring, image classifiers, suicide prevention and intensive care. He can accept an inexplicable but better predictor in some settings, but when law imposes benefits and burdens, we need explainability. He argues for a counter-narrative to the idea that AI will take over knowledge work. The question is not only what computers can’t do, but what they shouldn’t do. The key pair is judgment versus calculation.

🎙 In the talk. He calls the broader phenomenon “what I call the digitization of judgment”: law is one of many professions touched. He gives four quick examples. Hiring: instead of an HR person who talks to candidates, a natural-language system ranks 10,000 resumes by similarity to the 100 or 250 firm “stars”. This invites gaming, and people put prestigious university names in white font so that only the machine reads them. Image classifiers: duct tape on a stop sign made a car read a 45 mph sign. Suicide prevention: Facebook algorithms trigger police calls to people’s homes. Intensive care: software gives the percentage chance a patient lives.

He quotes Hod Lipson that explaining AI may be “like explaining Shakespeare to a dog”, and replies that he can accept an inexplicable but better predictor in some settings, but “when law imposes benefits and burdens we need explainability”. He is building a counter-narrative to books saying AI will take over knowledge work. From Dreyfus on what computers can’t do, he moves to what “computers shouldn’t do”: even if a chatbot could be a therapist or social worker, the question is which human values are at stake. The key pair is judgment versus calculation.

📚 Deeper theory

  • Why law gets a higher bar (Fuller, due process; idea 5). A diagnosis can be useful without an explanation if the patient gains. A sentence or a benefits denial is a reason-giving act by the state. The justification is part of the product.
  • Two senses of explanation. Technical interpretability (what features drove this output) and a justification a person can contest. A saliency map may satisfy the first and fail the second. Burrell’s three kinds of opacity (idea 5) show the first helps with only one.
  • The Lipson quote concedes a lot. If models are beyond human understanding, then the question becomes institutional: should any decision with legal force rest on a system its operators cannot explain? Pasquale says no; others say the accuracy gain can justify it. This is the same split as in medicine.
  • Dreyfus to “shouldn’t”. Dreyfus’s argument was about capacity and has been partly overtaken by machine learning (it does not need explicit rules). That is why Pasquale moves to a normative claim, as Weizenbaum did (Roots). Capacity arguments age badly; value arguments do not.
  • Brittleness and Goodhart (idea 3). The white-font resume and the duct-taped sign show that automated filters create new attack surfaces. Any system of ranking will be gamed once it is known.

📚 Roots: Philosophy of judgment — calculation is not judgment

The problem. What is the thing a judge does that a calculation does not?

  1. Aristotle, Nicomachean Ethics. Phronesis (practical wisdom) is the skill of acting well in particulars that no rule covers. In Book V he describes epieikeia (equity) as correcting a law where its generality fails. Equity is the oldest argument that law needs human judgment.
  2. Kant, Critique of Judgment (1790). Determinant judgment applies a known rule to a case. Reflective judgment finds the rule when only the case is given. Machines do the first well. The second is the gap Pasquale gestures at.
  3. Hannah Arendt (lectures on Kant’s political philosophy; Eichmann in Jerusalem, 1963). Judging is a public, imaginative act, and thoughtless rule-following is a route to harm.
  4. Michael Polanyi, The Tacit Dimension (1966). “We know more than we can tell.” The reason rule-based systems fail on experts.
  5. Hubert Dreyfus, What Computers Can’t Do (1972) and Mind over Machine (1986). Named in the talk (as “Hubert Ryan” in the captions). Expertise is skilled, embodied pattern recognition, not rule following.
  6. Brian Cantwell Smith, The Promise of Artificial Intelligence (2019). Separates reckoning (calculation) from judgment (deliberate commitment to a world). This fits the pairing Pasquale describes, “judgment… calculation”, though the captions are garbled there.
  7. Harry Collins and Robert Evans on expertise. “Interactional expertise” comes from being immersed in a community. Collins argues machines lack the social embedding that expertise needs. (The “Collins and myself” attribution is the likeliest reading of a garbled caption.)

Pasquale’s twist. The Dreyfus tradition asks what computers can’t do. He adds what they shouldn’t do: a normative claim, not a technical one. That is how he escapes the “just wait for better AI” reply.

📚 Roots: Professions and complementarity — augment, don’t replace

The problem. Will AI replace professionals or work with them? What should the design principle be?

  1. Joseph Weizenbaum, Computer Power and Human Reason (1976). After building ELIZA (1966), he argued that some tasks, such as those needing empathy or wisdom, should not be handed to computers even if they could be, because the question is moral, not technical. The introducer cites him.
  2. J. C. R. Licklider, “Man-Computer Symbiosis” (1960); Douglas Engelbart, Augmenting Human Intellect (1962). The older “augmentation” tradition that Pasquale reuses.
  3. Richard and Daniel Susskind, The Future of the Professions (2015). The main opposing view: professional work will be unbundled and much of it automated, because the public wants access more than it wants guilds. He is writing against this narrative.
  4. David Autor, “Why Are There Still So Many Jobs?” (2015). Automation complements some tasks and substitutes for others. Erik Brynjolfsson, “The Turing Trap” (2022), warns against designing AI to mimic and replace human labour when complementarity creates more value (both from memory)†.
  5. Access to justice. The strongest pro-automation argument is that most people cannot afford a lawyer at all, so even a crude tool beats nothing. Pasquale’s talk is about adjudication and enforcement, not the access gap; the two need to be kept apart.

Where others push back.

  • Humans are also black boxes. A judge’s stated reasons are often post-hoc rationalisations (the realist point). Why demand more from software than from a person?
  • Explainability trades off with accuracy in some tasks, and a worse but explainable model may harm more people.
  • Scale and access. An automated screening tool can serve people who would otherwise get nothing.

What he leaves out. A test for when the inexplicable-but-better predictor is acceptable. He grants “some cases” and does not say which. His later criterion is law’s coercive power, but borderline cases, such as triage or fraud flags, need a rule of thumb.

Check yourself

  • Why does Pasquale say law needs explanation when medicine might not?
  • Name two kinds of explanation and say which one a defendant needs.
  • Why does a “shouldn’t” argument last longer than a “can’t” argument?

3. Criminal justice: the algorithm, the dirty data and the missing test

In short. He sets Ferguson, who is critical of big-data policing, against Slobogin, who wants more computation in criminal justice. The pro-algorithm case rests on studies showing that human judges are inconsistent. Pasquale replies that the data is dirty, that cleaning it is as much art as science, and that bias can reinforce itself. His sharpest point is to run a risk assessment of the risk assessment. A tool may look accurate because prison itself causes the re-offending.

🎙 In the talk. He sets two authors against each other. Ferguson is critical of big-data policing. Slobogin argues “more computation more quantitative analysis to improve our criminal justice system”. Both agree that policing discriminates. The pro-algorithm case draws on studies of human judges: the Israeli parole study found “the closer you got to lunch the harsher the Judgment was”, called “a remarkable finding”, though a Harvard law professor doubts it is robust. An LSU football study found harsher sentencing after losses. The human baseline, Slobogin says, is “a holistic and often Unwritten but holistic assessment by a judge”, and a simple points system (criminal history, age, friends in prison) can beat it.

His reply starts with Richardson, Schultz and Crawford: “that data is dirty because it’s coming from departments that have a civil rights issue”. It is hard to know how far dirty data spread, for instance if one department’s data feeds another’s model. Cleaning it is “as much art as science”, and so as contestable as law. Next, other books show self-reinforcing bias: patrol a neighbourhood more, find more crime, then patrol it more. Then the sharpest point: do a “risk assessment of risk assessment”. If the high-risk group serves longer terms and re-offends more, the tool seems to work, but prison itself may cause the re-offending, so the tool may be creating its own proof. Credit scores work the same way. Finally, Wisconsin v. Loomis: “we still don’t know how the risk assessment was done”, which runs against the rule of law’s need for intelligibility.

📚 Deeper theory

  • The hungry-judges dispute is a case study in “science that travelled too well”. The study’s 2011 pattern drew critiques about how cases are ordered within a session, and a 2016 simulation showed a rational scheduling explanation (see Current debate). Pasquale’s remark that he sees the study cited everywhere, but has seen little rebalancing, is a fair complaint about how findings spread.
  • The fairness impossibility results (Roots). Because several fairness definitions cannot hold together when base rates differ, “make it unbiased” has no unique target. Choosing the metric is a political act. This supports Pasquale’s claim that the data-cleaning question is “as much art as science”.
  • Counterfactual validation is the real problem. To know whether a risk tool works you need outcomes without the intervention. Detained people cannot re-offend outside; released people are selected by the tool. Methods exist (natural experiments, random judge assignment as in Kleinberg et al., Roots), but they need data courts rarely collect.
  • Self-fulfilling loops (Roots). Hacking’s looping effect and Merton’s prophecy explain the credit-score and prison cases. The spiral is not a bug in one model; it is a feature of any classification that changes what it classifies.
  • The best counter-arguments. Meehl and Kahneman: simple rules reduce noise. Kleinberg et al.: a carefully built tool can improve on judges on both crime and jail measures in simulation. Slobogin: transparent, open-source points tools, validated locally, differ from COMPAS. Pasquale’s reply is narrower than a blanket no; it is that the conditions (clean data, tested effect, intelligibility) are rarely met.
  • Loomis and proprietary tools (idea 5). A court that lets trade secrecy block scrutiny of a sentencing input gives up Fuller’s publicity principle.

📚 Roots: Algorithmic governance, bias and fairness

The problem. Prediction tools are used to allocate liberty and money. Do they treat people fairly, and can anyone tell?

  1. Foucault, governmentality (1970s lectures); Deleuze, “Postscript on the Societies of Control” (1990). Power works through statistical management of populations, and in control societies through continuous modulation. Rouvroy and Berns’s “algorithmic governmentality” (2013) applies this to data.
  2. Paul Meehl, Clinical vs. Statistical Prediction (1954). Simple models often beat expert clinicians. The founding result behind Slobogin’s case.
  3. Bernard Harcourt, Against Prediction (2007). Statistical profiling in policing concentrates attention on one group, produces more arrests of that group, and distorts the data, so it may increase total crime. A direct ancestor of the feedback-loop argument.
  4. ProPublica, “Machine Bias” (May 2016) on COMPAS: higher false-positive rates for Black defendants. The vendor’s reply: calibration was equal across groups. Kleinberg, Mullainathan and Raghavan (2016) and Chouldechova (2017) then proved that, when base rates differ, several fairness criteria cannot all hold at once. So “unbiased” has no single meaning, and each choice is a value judgment. Dressel and Farid (2018) found COMPAS no more accurate than untrained online volunteers.
  5. Lum and Isaac, “To Predict and Serve?” (2016); Ensign et al. (2018) on runaway feedback loops. Predictive-policing models trained on arrest data send police back to the same blocks.
  6. Counter-evidence for the pro-algorithm side. Kleinberg et al., “Human Decisions and Machine Predictions” (QJE, 2018): an algorithm for New York bail decisions could, in their simulation, cut crime or jail populations without worsening racial disparity. Daniel Kahneman, Olivier Sibony and Cass Sunstein, Noise (2021), argue that unwanted variability in judgments is a large, neglected cost. This is the best form of the “weak baseline” argument. (All†)
  7. The critical books he cites. Cathy O’Neil (2016); Caroline Criado Perez (2019); Meredith Broussard (2018). Also in the same line: Virginia Eubanks, Automating Inequality (2018); Safiya Noble, Algorithms of Oppression (2018); Ruha Benjamin, Race After Technology (2019).
  8. Pasquale’s own earlier work. The Black Box Society (2015) on secret scoring; Danielle Citron and Pasquale, “The Scored Society” (2014)†.

Feeds the talk. The whole criminal-justice section. It also explains why he refuses to rest on bias alone: the fairness literature shows bias can be reduced on one metric and increased on another, so he adds validation and intelligibility.

📚 Roots: Two short threads

Adversarial examples and brittleness. The duct-tape stop sign comes from research on physical-world attacks on image classifiers (Eykholt et al., 2018)†. Models can be right on average and wrong in a way no person would be. White-font resumes are the same in spirit: an automated filter invites gaming (Goodhart’s law: a measure that becomes a target stops measuring).

Self-fulfilling prediction. Merton’s “self-fulfilling prophecy” (1948) and Ian Hacking’s “looping effects of human kinds” (1995): classifying people changes them, so the classification then confirms itself. Prison as a criminogenic setting and the credit-score spiral are textbook cases.

Where others push back.

  • Slobogin and similar scholars say the choice is not “algorithm vs perfect justice” but “algorithm vs the actual system”, and the actual system is arbitrary and racially skewed. Pasquale acknowledges this (“not doing terrible” is how he puts the opposing view) and ends by saying the answer is to augment, not sweep aside.
  • Defenders of structured tools say transparent checklists can be audited far more easily than a judge’s gut.
  • Data scientists say bias in data can sometimes be corrected statistically, and that risk-adjustment is routine in health law (he mentions this himself).

What he leaves out. The hungry-judges evidence cuts both ways. If that study is weak, one prop of the pro-algorithm case weakens, and that deserved more than a passing hedge. He also does not engage with the New York bail simulation work or with open, locally validated tools; and a reader should ask what a good version of Slobogin’s proposal would need to satisfy.

Check yourself

  • Why can a risk tool look validated when it is only creating its own evidence? Use the prison example.
  • Why does the existence of several fairness metrics make “remove the bias” an incomplete instruction?
  • State the strongest pro-algorithm argument and Pasquale’s best reply.

4. Automated force: from spring guns to security robots

In short. He moves from prediction to force. The spring gun, a trap set to shoot intruders, is generally not allowed, and he calls that the wisdom of the common law. Its modern descendant is the security robot. He expects precise robotic use of force. Reformers hope for less prison, but Ruha Benjamin asks whether cheaper force will spread more widely. He is on the side of those pushing back, and he recommends fiction because regulators are a few steps behind.

🎙 In the talk. He moves from prediction to the physical world. The spring gun is a trap set in a house to shoot intruders. The cases say in general it is not allowed, with narrow exceptions if you can show the person meant serious harm, “that’s the wisdom of the common law”. Its modern descendant is the security robot. A Knightscope robot, deployed by an animal charity in San Francisco, was used against homeless people, who then covered it in barbecue sauce. (See the correction in the caption box.) He expects “robotized and very precise use of force”.

Two angles. First, reformers say tracking technology could empty prisons into house arrest. Ruha Benjamin asks whether precise control is a step forward when “by cheapening force they could make it much more widespread”. Second, he imagines drones with long-range acoustic devices that hurt but do not permanently harm. That may reduce injuries to police and protesters, but “if it becomes too precise and Too Perfect in your ability to control reality”, the question is like that of genetic engineering: control of social space. He is “on the side of the people who are sort of pushing back”. He recommends fiction (a film on a protective robot; McEwan’s Machines Like Me) because it sees steps ahead while “Regulators are just a few steps behind”. Elizabeth Joh answers the claim that a robot can tell a burglar from a sheltering stranger: “even now these robots are far from being able to make those types of determinations”, which is the heart of the campaign against killer robots. Next, asylum: IBM pitched software to European authorities to screen asylum applicants, and the stock image already carries a stereotype.

📚 Deeper theory

  • The spring-gun rule is a rule about delegated force. The law holds you responsible for force applied in your name without your judgment in the moment (Roots). A robot is a smarter spring gun, so the question is whether intelligence changes responsibility. Pasquale’s fourth “law of robotics” (attribution to a responsible human)† is the principled answer.
  • Weber’s monopoly (Roots). If private actors field robot guards and acoustic drones, the line between public and private force blurs. The SF case is a private charity acting on a public sidewalk; the city said the robot needed a permit.
  • The cheap-force argument. In the ethics of war, lowering the cost of force lowers the threshold for using it. Benjamin’s concern about house arrest is that gentler tools could widen the net: more people controlled, for longer.
  • Precision is not the same as legitimacy. Distinction and proportionality (international humanitarian law) need contextual judgment: is this a child, a surrendering person? Joh’s point and the Stop Killer Robots campaign rest on that.
  • Regulatory lag. The “pacing problem”: technology changes faster than law. Fiction as foresight is one response; “anticipatory social research” in the introducer’s words is another.
  • Asylum software. Asylum claims are the paradigm case of individual assessment. A classifier trained on past cases imports past decisions, so the baseline problem he raises is real: who set it, and with what error costs? (The “baseline” question is his, and he says it “needs to be elaborated more”.)

📚 Roots: Force, robots and the law of self-defence

The problem. Who may use force, and what happens when it can be automated, cheapened and made precise?

  1. Weber, “Politics as a Vocation” (1919). The state claims a monopoly on the legitimate use of physical force. Privately deployed or autonomous force strains that.
  2. The spring-gun cases. English and American courts condemned booby traps early (Bird v. Holbrook, 1825), and Katko v. Briney (Iowa, 1971) is the usual US citation (both from memory)†. The principle: you may not set up a device that applies deadly force without a human making the call in the moment. This is the “wisdom of the common law”.
  3. International humanitarian law. Distinction (combatant vs civilian), proportionality, and precaution, in Additional Protocol I (1977), plus the Martens Clause. Critics of lethal autonomous weapons (Noel Sharkey, Peter Asaro, the Campaign to Stop Killer Robots, founded 2013) argue a machine cannot reliably judge these. The “reasonable person” test Pasquale mentions is the legal form of the same worry.
  4. Elizabeth Joh on police robots. The legal and ethical limits of robot policing. She makes the point that robots cannot read intent.
  5. Froomkin and Colangelo. Treat drones as a test for property and self-defence law: when may you shoot one down?
  6. Cheap violence and “e-carceration”. Michelle Alexander’s The New Jim Crow (2010) and her later writing on electronic monitoring as a new form of control; Ruha Benjamin worries that cheaper control means more control.
  7. Precision as a moral hazard. A recurring theme in the ethics of weapons: if force becomes cheaper and cleaner, the threshold to use it falls. Pasquale links it to genetic engineering.
  8. Fiction as foresight. Machines Like Me (2019) and films about protective machines. Pasquale uses fiction as an early-warning system where regulation lags.

Where others push back.

  • Proponents argue that human guards and police also err, with worse bias, and sometimes kill. A well-tested robot with non-lethal tools may reduce harm.
  • On the electronic-monitoring point, many reformers think house arrest is far better than prison, and that the risk of “net-widening” can be managed by law.
  • On killer robots, some military ethicists (Ronald Arkin is the usual name)† argue machines could follow rules of engagement more consistently than stressed soldiers.

What he leaves out. The acoustic-drone and house-arrest passages are speculative, and he gives no evidence on how fast adoption is happening. He does not say how law should treat defensive home robots, which is where the spring-gun analogy is tightest.

Check yourself

  • What principle makes a spring gun unlawful, and how does a security robot differ from it?
  • Why might making force “more precise” make it more common?
  • What is the “pacing problem”, and how does fiction help?

5. Binaries, datasets and the limits of automating law

In short. The core of the technology is sorting things into two bins, like dog or bagel. That is hard to do for categories like terrorist, or a child versus a threat, and a broad enough dataset is hard to build. Law could be automated where rules are clear, as with red-light cameras. Even then, scholars ask about due process: is the computer your accuser? New York sends the image to a person, and he asks whether that check is meaningful or feather bedding.

🎙 In the talk. The core of the technology is classification into two bins, “dog or bagel”. With enough labelled examples, a supervised learner will do it, perhaps in ways humans cannot explain. But try “terrorist” against not, or a child against a threat, and “that’s a lot harder to program for”. You could assemble 400 dogs and 400 bagels with mechanical-turk labellers, not a dataset for asylum. His point: people focus on algorithms and learning types and forget “the fundamental problem of like the nature of the distinctions being very binary”, plus the difficulty of building a broad enough dataset.

Where could law be automated? Red-light cameras: “the law can be very clear about that”. He imagines automatic debiting, and people opting into facial recognition for a discount. Even so, scholars worry about due process and confrontation: “is the computer your accuser”. New York sends the image to a person who double-checks, and that person is the accuser. He asks whether this is meaningful or “feather bedding” (the caption reads “feather betting”). He adds that some civil-rights advocates prefer cameras to officers because they trust the camera more. He guesses people will use bots to contest tickets, as with high-frequency trading.

📚 Deeper theory

  • Why the dataset is the product. A classifier learns the boundary in its labels. Dog and bagel are visual kinds; “terrorist” is a social judgment made by institutions with their own biases, so the label is the bias. Philosophers call these interactive or human kinds (Hacking, idea 3): classifying them changes them.
  • Construct validity. Whether a measure captures what it claims to. “Aggression”, “risk” and “suspicion” are constructs, not objects. Without construct validity, more data only measures the proxy better.
  • Why red-light cameras are the easy case. A bright-line rule, an objective event, low stakes, and strict liability (intent irrelevant). That combination makes rule-following the whole task. Few legal tasks have it.
  • The accuser question is about legitimacy, not accuracy. Confrontation (Roots) is meant to let the accused test the witness. A machine cannot be cross-examined on its motives, but it can be audited. The New York rule treats a human signature as the accountable act, and whether that is real review or ritual depends on how much time the reviewer has (the automation-bias literature suggests people often defer to the machine)†.
  • Automation bias and the human in the loop. A human who simply confirms the machine adds a name, not a check. This is Pasquale’s own “feather bedding” worry in more formal terms.
  • Opt-in discounts. A familiar pattern from insurance telematics: voluntary at first, then penalising those who decline. He predicts the first step and signals discomfort with the last.

📚 Roots: Due process, legitimacy and the Catholic thread

The problem. Beyond accuracy, what makes a decision legitimate for the person it is imposed on?

  1. Max Weber on legitimacy and “formal rationality” (Economy and Society, 1922). Modern bureaucracies prize calculable rules. Weber also feared the “iron cage”. He contrasted formal law with kadi justice (case-by-case discretion), and a risk score is oddly a way to get calculability and individualisation at once.
  2. Lon Fuller, The Morality of Law (1964). Law has an “inner morality”: rules must be public, clear, stable and applied as announced. A secret scoring system fails it.
  3. Procedural due process. Mathews v. Eldridge (1976) balances the private interest, the risk of error, and the cost of added procedure. Danielle Citron, “Technological Due Process” (2008), argues automated systems should meet that standard†.
  4. Confrontation. The Sixth Amendment right to confront your accuser, as in Crawford v. Washington (2004)†. This is the frame of: “is the computer your accuser”.
  5. State v. Loomis, 881 N.W.2d 749 (Wis. 2016). The Wisconsin Supreme Court let a sentencing judge consider COMPAS output, with cautions, although the method was proprietary†. Pasquale’s point is that this secrecy defeats intelligibility.
  6. Explanation rights. GDPR Article 22 limits solely automated decisions; scholars dispute whether it gives a “right to explanation”. Jenna Burrell, “How the Machine ‘Thinks’” (2016), separates three sources of opacity: secrecy, technical illiteracy, and the scale and complexity of the model itself. Only the third is addressed by better explainability tools.
  7. Charles Taylor and the narrative self. Taylor, Sources of the Self (1989) and A Catholic Modernity? (1999). Agency is bound up with the stories and the language by which we understand ourselves. Alasdair MacIntyre’s After Virtue (1981) argues similarly that a life has narrative unity. The talk uses Taylor to say that a decision you cannot narrate is one you cannot own or contest.
  8. Catholic thought on AI. The Rome Call for AI Ethics (Feb 2020, signed by the Pontifical Academy for Life) speaks of “algor-ethics”; and the Vatican note Antiqua et Nova (Jan 2025) discusses AI and human intelligence (both from memory)†. Pasquale does not cite either, but the audience is the kind that would.

Feeds the talk. The step from “the risk tool might be biased” to “the risk tool is not an acceptable reason for what happens to you”.

Where others push back.

  • Efficiency advocates say that if the rule is clear, automation lowers cost and arbitrariness, and the human check is waste.
  • Civil-rights advocates note that officers’ discretion has been the source of bias, so the camera has real merit (he raises this himself).
  • On datasets, modern foundation models learn from far broader data than a labelled bagel set, so “never broad enough” may be too strong. The deeper point, that the label is a social judgment, survives.

What he leaves out. He does not distinguish classification from the generative models just then in the news; the “binary” frame fits risk scoring well and chatbots poorly. He does not say what a good human review looks like in practice.

Check yourself

  • Why is “terrorist” a different kind of label from “dog”?
  • List the features that make a red-light camera an easy case for automation.
  • When does a human double-check add accountability, and when is it theatre?

6. Affective computing, attribution and remote diagnosis

In short. Affective computing needs a reliable database of images, yet human expressions change and real people will not fit eight categories. Lauren Rhue found bias in how Black men’s expressions are read. Pasquale’s central point is that these systems are attributing emotion, which he calls something of a power play. He asks where evoking emotion becomes manipulation. Remote diagnosis of mental-health risk is worse. A doctor using a screen to supplement judgment is fine, but a system with no relationship is not.

🎙 In the talk. He imagines a store chain with an automated kiosk that records customers, detects “violence”, and dispatches a manager or the police. This depends on affective computing, which needs “a reliable database of images”, yet human expressions evolve. The Microsoft system offers eight categories, “surprise” among them, and real people will not fit. Bias: Lauren Rhue found “black men’s facial expressions are associated with the emotions associated with threatening behaviors more often than white men even when smiling”. There is a privacy dimension and a conceptual one (Bard): is emotion biometric data, or belief?

His central point: “what they’re really doing is attributing emotion”. He debated this at Cornell Tech, challenged the metaphor of legibility, and credits Rob Horning for the insight that “I do think it’s something of a power play”. Then: “at what point does it become creation or evocation of emotion and at what point is it manipulation”. The slide to diagnosis is worse: an AI that flags someone as a mental-health risk, in a world that links mental illness and mass shootings. In a doctor’s hands, as a screen “to supplement my judgment I’m fine with it”. With no relationship, it is not. The Goldwater episode shows why: remote diagnosis of a public figure was condemned as unprofessional, and he notes psychiatrists who speculated about Trump got “a lot less Traction” for the same reason.

📚 Deeper theory

  • The science is shaky (Roots). The Ekman six-emotion model is the basis of many products, and Barrett’s review found weak support for reading specific emotions from faces across people and settings. A system trained on posed or labelled faces learns the labeller’s guess.
  • Attribution is performative. Once an institution acts on the label, it shapes what happens to the person (Roots, idea 3). A “violent” tag can cause the confrontation it predicts.
  • Legibility as power (Scott; Roots). Making people readable benefits whoever reads. The question “how legible should humans be to machines” assumes legibility is a neutral property of persons; Horning’s reply is that it is a demand made on them.
  • Manipulation. Affect-sensing systems can feed back into persuasion: the same data that “reads” a mood can be used to move it. This is an old worry in advertising ethics, now cheaper.
  • Why the clinic matters (Foucault, Goldwater rule). A diagnosis made inside a relationship is open to reply: the patient can correct, consent, refuse. Remote or automated diagnosis removes the reply. The professional rule is a rule about standing to judge.
  • Regulation now. The EU AI Act bans emotion inference in workplaces and schools, with medical and safety exceptions (see Current debate). That is an institutional version of the distinction he draws between a doctor’s tool and a store’s.

📚 Roots: Affect, legibility and remote diagnosis

The problem. Can a machine read inner states from outward signs, and what happens if institutions act as if it can?

  1. Paul Ekman and the basic emotions (1970s). Six or so universal facial expressions. Many emotion-detection products rest on this model. A Microsoft system with eight categories is the kind of product he shows.
  2. Rosalind Picard, Affective Computing (1997). The founding statement of the field.
  3. Lisa Feldman Barrett and colleagues (2019, Psychological Science in the Public Interest). A major review found that facial movements do not reliably map to specific emotions across people and contexts. Kate Crawford’s Atlas of AI (2021) makes the same point for policy audiences. (†)
  4. Lauren Rhue’s bias study. Emotion software assigned angrier or more negative scores to Black men’s faces than to white men’s, even when smiling. This is a double failure: the model is weak and the error is not evenly spread.
  5. James C. Scott, Seeing Like a State (1998). States make society legible (surnames, maps, censuses) so that it can be taxed and ruled, and the simplifications destroy local knowledge. Pasquale’s challenge to “how legible should humans be to machines” is in this line.
  6. Attribution. If a system’s label is acted on (not allowed into a shop, flagged for police), the label becomes the person’s situation. This is a self-fulfilling mechanism, as in the risk-assessment section.
  7. The Goldwater rule and clinical relationships. Diagnosis in professional psychiatry rests on examination and consent. The American Psychiatric Association’s rule against diagnosing public figures at a distance came from the 1964 Fact magazine episode. Pasquale extends it to machines.
  8. Foucault, The Birth of the Clinic (1963). The medical gaze turns a person into a case. Remote machine diagnosis is an extreme version.

Where others push back.

  • Some researchers say narrow tasks (pain, drowsiness, depression screens) can work and help. He agrees in the clinical case.
  • Vendors argue the system reports a probability of a state, not a certainty, and humans decide.
  • Some argue the Goldwater rule is too broad: experts have knowledge the public needs, and a duty to warn. This is a live dispute inside psychiatry.

What he leaves out. Language models, which now write emotionally persuasive text, make the creation-or-manipulation line he draws more urgent, but that was outside the 2023 scene. He also leaves open how a store should handle real violence.

Check yourself

  • What is the difference between detecting and attributing an emotion?
  • Why do he and Scott treat “legibility” as an exercise of power?
  • Why is a doctor’s depression screen different from a store’s aggression detector?