Rationality
From AI to Zombies
Eliezer Yudkowsky
Eliezer Yudkowsky is a Research Fellow at the Machine Intelligence Research Institute.
Written by Eliezer Yudkowsky
Published in 2015
Machine Intelligence Research Institute
Berkeley 94704
United States of America
intelligence.org
Released under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported license.
CC BY-NC-SA 3.0
isbn-10: 1-939311-15-2
isbn-13: 978-1-939311-15-3
(epub)
The Machine Intelligence Research Institute gratefully acknowledges the generous support of all those involved in the publication of this book, and the donors who support MIRI’s work in general.
Cover by Tim Woolley.
Contents
Preface
Biases: An Introduction
* * *
Book I: Map and Territory
* * *
A. Predictably Wrong
1. What Do I Mean By “Rationality”?
2. Feeling Rational
3. Why Truth? And . . .
4. . . . What’s a Bias, Again?
5. Availability
6. Burdensome Details
7. Planning Fallacy
8. Illusion of Transparency: Why No One Understands You
9. Expecting Short Inferential Distances
10. The Lens That Sees Its Own Flaws
B. Fake Beliefs
11. Making Beliefs Pay Rent (in Anticipated Experiences)
12. A Fable of Science and Politics
13. Belief in Belief
14. Bayesian Judo
15. Pretending to be Wise
16. Religion’s Claim to be Non-Disprovable
17. Professing and Cheering
18. Belief as Attire
19. Applause Lights
C. Noticing Confusion
20. Focus Your Uncertainty
21. What Is Evidence?
22. Scientific Evidence, Legal Evidence, Rational Evidence
23. How Much Evidence Does It Take?
24. Einstein’s Arrogance
25. Occam’s Razor
26. Your Strength as a Rationalist
27. Absence of Evidence Is Evidence of Absence
28. Conservation of Expected Evidence
29. Hindsight Devalues Science
D. Mysterious Answers
30. Fake Explanations
31. Guessing the Teacher’s Password
32. Science as Attire
33. Fake Causality
34. Semantic Stopsigns
35. Mysterious Answers to Mysterious Questions
36. The Futility of Emergence
37. Say Not “Complexity”
38. Positive Bias: Look into the Dark
39. Lawful Uncertainty
40. My Wild and Reckless Youth
41. Failing to Learn from History
42. Making History Available
43. Explain/Worship/Ignore?
44. “Science” as Curiosity-Stopper
45. Truly Part of You
Interlude: The Simple Truth
* * *
Book II: How to Actually Change Your Mind
* * *
Rationality: An Introduction
E. Overly Convenient Excuses
46. The Proper Use of Humility
47. The Third Alternative
48. Lotteries: A Waste of Hope
49. New Improved Lottery
50. But There’s Still a Chance, Right?
51. The Fallacy of Gray
52. Absolute Authority
53. How to Convince Me That 2 + 2 = 3
54. Infinite Certainty
55. 0 And 1 Are Not Probabilities
56. Your Rationality Is My Business
F. Politics and Rationality
57. Politics is the Mind-Killer
58. Policy Debates Should Not Appear One-Sided
59. The Scales of Justice, the Notebook of Rationality
60. Correspondence Bias
61. Are Your Enemies Innately Evil?
62. Reversed Stupidity Is Not Intelligence
63. Argument Screens Off Authority
64. Hug the Query
65. Rationality and the English Language
66. Human Evil and Muddled Thinking
G. Against Rationalization
68. Update Yourself Incrementally
69. One Argument Against An Army
70. The Bottom Line
71. What Evidence Filtered Evidence?
72. Rationalization
73. A Rational Argument
74. Avoiding Your Belief’s Real Weak Points
75. Motivated Stopping and Motivated Continuation
76. Fake Justification
77. Is That Your True Rejection?
78. Entangled Truths, Contagious Lies
79. Of Lies and Black Swan Blowups
80. Dark Side Epistemology
H. Against Doublethink
81. Singlethink
82. Doublethink (Choosing to be Biased)
83. No, Really, I’ve Deceived Myself
84. Belief in Self-Deception
85. Moore’s Paradox
86. Don’t Believe You’ll Self-Deceive
I. Seeing with Fresh Eyes
87. Anchoring and Adjustment
88. Priming and Contamination
89. Do We Believe Everything We’re Told?
90. Cached Thoughts
91. The “Outside the Box” Box
92. Original Seeing
93. Stranger than History
94. The Logical Fallacy of Generalization from Fictional Evidence
95. The Virtue of Narrowness
96. How to Seem (and Be) Deep
97. We Change Our Minds Less Often Than We Think
98. Hold Off On Proposing Solutions
99. The Genetic Fallacy
J. Death Spirals
100. The Affect Heuristic
101. Evaluability (and Cheap Holiday Shopping)
102. Unbounded Scales, Huge Jury Awards, and Futurism
103. The Halo Effect
104. Superhero Bias
105. Mere Messiahs
106. Affective Death Spirals
107. Resist the Happy Death Spiral
108. Uncritical Supercriticality
109. Evaporative Cooling of Group Beliefs
110. When None Dare Urge Restraint
111. The Robbers Cave Experiment
112. Every Cause Wants to Be a Cult
113. Guardians of the Truth
114. Guardians of the Gene Pool
115. Guardians of Ayn Rand
116. Two Cult Koans
117. Asch’s Conformity Experiment
118. On Expressing Your Concerns
119. Lonely Dissent
120. Cultish Countercultishness
K. Letting Go
121. The Importance of Saying “Oops”
122. The Crackpot Offer
123. Just Lose Hope Already
124. The Proper Use of Doubt
125. You Can Face Reality
126. The Meditation on Curiosity
127. No One Can Exempt You From Rationality’s Laws
128. Leave a Line of Retreat
129. Crisis of Faith
130. The Ritual
* * *
Book III: The Machine in the Ghost
* * *
Minds: An Introduction
Interlude: The Power of Intelligence
L. The Simple Math of Evolution
131. An Alien God
132. The Wonder of Evolution
133. Evolutions Are Stupid (But Work Anyway)
134. No Evolutions for Corporations or Nanodevices
135. Evolving to Extinction
136. The Tragedy of Group Selectionism
137. Fake Optimization Criteria
138. Adaptation-Executers, Not Fitness-Maximizers
139. Evolutionary Psychology
140. An Especially Elegant Evolutionary Psychology Experiment
141. Superstimuli and the Collapse of Western Civilization
142. Thou Art Godshatter
M. Fragile Purposes
143. Belief in Intelligence
144. Humans in Funny Suits
145. Optimization and the Intelligence Explosion
146. Ghosts in the Machine
147. Artificial Addition
148. Terminal Values and Instrumental Values
149. Leaky Generalizations
150. The Hidden Complexity of Wishes
151. Anthropomorphic Optimism
152. Lost Purposes
N. A Human’s Guide to Words
153. The Parable of the Dagger
154. The Parable of Hemlock
155. Words as Hidden Inferences
157. Similarity Clusters
158. Typicality and Asymmetrical Similarity
159. The Cluster Structure of Thingspace
160. Disguised Queries
161. Neural Categories
162. How An Algorithm Feels From Inside
163. Disputing Definitions
164. Feel the Meaning
165. The Argument from Common Usage
166. Empty Labels
167. Taboo Your Words
168. Replace the Symbol with the Substance
169. Fallacies of Compression
170. Categorizing Has Consequences
171. Sneaking in Connotations
172. Arguing “By Definition”
173. Where to Draw the Boundary?
174. Entropy, and Short Codes
175. Mutual Information, and Density in Thingspace
176. Superexponential Conceptspace, and Simple Words
177. Conditional Independence, and Naive Bayes
178. Words as Mental Paintbrush Handles
179. Variable Question Fallacies
180. 37 Ways That Words Can Be Wrong
Interlude: An Intuitive Explanation of Bayes’s Theorem
* * *
Book IV: Mere Reality
* * *
The World: An Introduction
O. Lawful Truth
181. Universal Fire
182. Universal Law
183. Is Reality Ugly?
184. Beautiful Probability
185. Outside the Laboratory
186. The Second Law of Thermodynamics, and Engines of Cognition
187. Perpetual Motion Beliefs
188. Searching for Bayes-Structure
P. Reductionism 101
189. Dissolving the Question
190. Wrong Questions
191. Righting a Wrong Question
192. Mind Projection Fallacy
193. Probability is in the Mind
194. The Quotation is Not the Referent
195. Qualitatively Confused
196. Think Like Reality
197. Chaotic Inversion
198. Reductionism
199. Explaining vs. Explaining Away
200. Fake Reductionism
201. Savannah Poets
Q. Joy in the Merely Real
202. Joy in the Merely Real
203. Joy in Discovery
204. Bind Yourself to Reality
205. If You Demand Magic, Magic Won’t Help
206. Mundane Magic
207. The Beauty of Settled Science
208. Amazing Breakthrough Day: April 1st
209. Is Humanism a Religion Substitute?
210. Scarcity
211. The Sacred Mundane
212. To Spread Science, Keep It Secret
213. Initiation Ceremony
R. Physicalism 201
214. Hand vs. Fingers
215. Angry Atoms
216. Heat vs. Motion
217. Brain Breakthrough! It’s Made of Neurons!
218. When Anthropomorphism Became Stupid
219. A Priori
220. Reductive Reference
221. Zombies! Zombies?
222. Zombie Responses
223. The Generalized Anti-Zombie Principle
224. GAZP vs. GLUT
225. Belief in the Implied Invisible
226. Zombies: The Movie
227. Excluding the Supernatural
228. Psychic Powers
S. Quantum Physics and Many Worlds
229. Quantum Explanations
230. Configurations and Amplitude
231. Joint Configurations
232. Distinct Configurations
233. Collapse Postulates
234. Decoherence is Simple
235. Decoherence is Falsifiable and Testable
236. Privileging the Hypothesis
237. Living in Many Worlds
238. Quantum Non-Realism
239. If Many-Worlds Had Come First
240. Where Philosophy Meets Science
241. Thou Art Physics
242. Many Worlds, One Best Guess
T. Science and Rationality
243. The Failures of Eld Science
244. The Dilemma: Science or Bayes?
245. Science Doesn’t Trust Your Rationality
246. When Science Can’t Help
247. Science Isn’t Strict Enough
248. Do Scientists Already Know This Stuff?
249. No Safe Defense, Not Even Science
250. Changing the Definition of Science
251. Faster Than Science
252. Einstein’s Speed
253. That Alien Message
254. My Childhood Role Model
255. Einstein’s Superpowers
256. Class Project
Interlude: A Technical Explanation of Technical Explanation
* * *
Book V: Mere Goodness
* * *
Ends: An Introduction
U. Fake Preferences
257. Not for the Sake of Happiness (Alone)
258. Fake Selfishness
259. Fake Morality
260. Fake Utility Functions
261. Detached Lever Fallacy
262. Dreams of AI Design
263. The Design Space of Minds-in-General
V. Value Theory
264. Where Recursive Justification Hits Bottom
265. My Kind of Reflection
266. No Universally Compelling Arguments
267. Created Already in Motion
268. Sorting Pebbles into Correct Heaps
269. 2-Place and 1-Place Words
270. What Would You Do Without Morality?
271. Changing Your Metaethics
272. Could Anything Be Right?
273. Morality as Fixed Computation
274. Magical Categories
275. The True Prisoner’s Dilemma
276. Sympathetic Minds
277. High Challenge
278. Serious Stories
279. Value is Fragile
280. The Gift We Give to Tomorrow
W. Quantified Humanism
281. Scope Insensitivity
282. One Life Against the World
283. The Allais Paradox
284. Zut Allais!
285. Feeling Moral
286. The “Intuitions” Behind “Utilitarianism”
287. Ends Don’t Justify Means (Among Humans)
288. Ethical Injunctions
289. Something to Protect
290. When (Not) to Use Probabilities
291.
Newcomb’s Problem and Regret of Rationality
Interlude: The Twelve Virtues of Rationality
* * *
Book VI: Becoming Stronger
* * *
Beginnings: An Introduction
X. Yudkowsky’s Coming of Age
292. My Childhood Death Spiral
293. My Best and Worst Mistake
294. Raised in Technophilia
295. A Prodigy of Refutation
296. The Sheer Folly of Callow Youth
297. That Tiny Note of Discord
298. Fighting a Rearguard Action Against the Truth
299. My Naturalistic Awakening
300. The Level Above Mine
301. The Magnitude of His Own Folly
302. Beyond the Reach of God
303. My Bayesian Enlightenment
Y. Challenging the Difficult
304. Tsuyoku Naritai! (I Want to Become Stronger)
305. Tsuyoku vs. the Egalitarian Instinct
306. Trying to Try
307. Use the Try Harder, Luke
308. On Doing the Impossible
309. Make an Extraordinary Effort
310. Shut Up and Do the Impossible!
311. Final Words
Z. The Craft and the Community
312. Raising the Sanity Waterline
313. A Sense That More Is Possible
314. Epistemic Viciousness
315. Schools Proliferating Without Evidence
316. Three Levels of Rationality Verification
317. Why Our Kind Can’t Cooperate
318. Tolerate Tolerance
319. Your Price for Joining
320. Can Humanism Match Religion’s Output?
321. Church vs. Taskforce
322. Rationality: Common Interest of Many Causes
323. Helpless Individuals
324. Money: The Unit of Caring
325. Purchase Fuzzies and Utilons Separately
326. Bystander Apathy
327. Collective Apathy and the Internet
328. Incremental Progress and the Valley