Dan Ariely's Weird Science: Why the Best Experiments Look Ridiculous | Brainfluence — Transcript

Full transcript

1. 0:00Welcome to Brainfluence.
2. 0:01I'm Roger Dooley.
3. 0:02A quick reminder for our audio listeners, all episodes,
4. 0:06plus many more Brainfluence Briefs are available on my YouTube channel.
5. 0:10Way back in 1969,
6. 0:11a psychologist named Robert Zajonc recruited 72 cockroaches
7. 0:16and had them run mazes.
8. 0:18They ran in front of an audience.
9. 0:20Here's the weird part.
10. 0:21The audience wasn't lab assistants or undergrad psych majors earning credit.
11. 0:26It was other cockroaches.
12. 0:28The watcher bugs were sealed in little clear plastic boxes
13. 0:31right next to the track.
14. 0:32The paper doesn't tell us whether or not they cheered.
15. 0:35Some roach runners got an easy job,
16. 0:37run straight down a path to get away from a bright light.
17. 0:40Others got a harder job, a maze with twists and turns.
18. 0:44And with that audience of fellow roaches watching,
19. 0:47Zajonc observed something strange.
20. 0:49The roaches got faster on the easy path and slower on the difficult maze.
21. 0:54By now you're probably wondering, who cares how fast a cockroach runs?
22. 0:58I guess some people do.
23. 0:59Anecdotally, I can tell you
24. 1:01that New York City roaches scatter three times faster
25. 1:04than their laid-back Texas relatives.
26. 1:06But this experiment was never about cockroach athletic performance.
27. 1:10And by the end of this episode, you'll see why the strangest-looking,
28. 1:14most bizarre experiments in behavioral science are usually the best ones.
29. 1:19and how the thinking behind them can transform
30. 1:22the way your team runs your marketing experiments.
31. 1:31So what was Zajonc actually studying?
32. 1:33It was something called social facilitation,
33. 1:35the question of whether being watched changes performance.
34. 1:39Here's the design problem.
35. 1:41If you test that on humans, things get complicated.
36. 1:43People care what the audience thinks.
37. 1:45They get self-conscious.
38. 1:47They show off, or maybe they choke.
39. 1:49It's hard to separate whether the presence of others changes performance
40. 1:53or worrying about others.
41. 1:54A cockroach solves that problem.
42. 1:57It can sense that other roaches are nearby.
43. 1:59but it almost certainly doesn't care what they think of its lap time.
44. 2:03If you remove self-consciousness,
45. 2:05and if the audience still changes performance,
46. 2:07you've isolated something real about the mechanism.
47. 2:10That's not a silly experiment.
48. 2:12That's a very precise one that happens to look silly.
49. 2:15Remember this study because
50. 2:17there's a twist coming at the end of the episode.
51. 2:19Fifty years later, someone re-ran it and the story changed.
52. 2:23I'll get to that in a little bit.
53. 2:25First, let me give you the frame that ties this whole episode together.
54. 2:28It comes from Dan Ariely, who I've had on this show a couple of times.
55. 2:32He recently wrote a short piece about how he comes
56. 2:35up with his experiments, and I really like his analogy.
57. 2:38An experiment is a painting, not a photograph.
58. 2:42A painter usually doesn't try to reproduce every detail of a scene.
59. 2:46They capture the essence of it.
60. 2:48A good experiment works the same way.
61. 2:51You start with a real-world problem.
62. 2:53People don't save enough for retirement.
63. 2:55People quit taking their medication.
64. 2:57People abandon projects halfway through.
65. 2:59Then you boil that problem down to a single principle.
66. 3:02Next, you build a tiny self-contained world that includes
67. 3:05that principle and almost nothing else.
68. 3:08Finally, you mess with it and watch what happens.
69. 3:11The question a good researcher is really asking is actually quite simple.
70. 3:15What's the essence of this problem?
71. 3:18That explains why so much behavioral research looks,
72. 3:21frankly, a little crazy from the outside.
73. 3:24The weirdness isn't sloppiness or weirdness for its own sake.
74. 3:27It's what's left after you strip away everything that isn't the essence.
75. 3:32When I interviewed Ariely years ago, I teased him about this.
76. 3:35Other researchers need multimillion-dollar brain scanners.
77. 3:38Dan often seems to need little more than paper,
78. 3:41a pencil, and a few grad students.
79. 3:43His answer stuck with me.
80. 3:45He said, I'm not really a good scientist.
81. 3:47I'm more of a behavioral hacker.
82. 3:49He starts with something in the real world that isn't working,
83. 3:52and asks how you could poke at it, even imperfectly,
84. 3:55to understand it better.
85. 3:56Keep that phrase in mind, behavioral hacker,
86. 3:59because I'm going to walk you through four experiments
87. 4:02that sound like pranks
88. 4:03and turn out to be very useful findings in behavioral science.
89. 4:06a scary bridge, a beer menu, a pile of Legos,
90. 4:10and an overpriced energy drink.
91. 4:12Let's start with the bridge.
92. 4:14British Columbia, 1974.
93. 4:16The Capilano Canyon has two ways across.
94. 4:19One is a narrow suspension bridge,
95. 4:21wobbly swaying two hundred thirty feet above rocks and rapids.
96. 4:25The other is a solid wooden bridge crossing a calm
97. 4:28stream just ten feet down.
98. 4:30Researchers Donald Dutton and Arthur Aron
99. 4:33stationed an attractive woman at the end of each bridge.
100. 4:37As men came across, she asked
101. 4:38them to write a short story about an ambiguous picture,
102. 4:41and then gave each one her phone number,
103. 4:44supposedly in case he had questions about the study.
104. 4:47Here's what happened.
105. 4:49Of the men who crossed the terrifying bridge, half called her.
106. 4:52From the safe bridge, only two out of sixteen.
107. 4:56and the stories written by the scary-bridge men contained
108. 4:59noticeably more romantic and sexual imagery.
109. 5:02Now, was this a study about bridges?
110. 5:04Of course not.
111. 5:05The essence being tested was something called misattribution of arousal.
112. 5:09When you're frightened, your body changes.
113. 5:12Your heart pounds, your palms sweat.
114. 5:14Your brain notices these changes and looks for an explanation.
115. 5:18And it doesn't always pick the right one.
116. 5:20It picks the most interesting one available.
117. 5:23And on that bridge, the most
118. 5:24interesting explanation in sight was an attractive stranger.
119. 5:28The bridge was never the point.
120. 5:30It was just a cheap, reliable way to spike a man's heart rate
121. 5:34that had nothing to do with romance.
122. 5:36Now, full disclosure,
123. 5:37the methodology of this study has been criticized for fifty years.
124. 5:41Men weren't randomly assigned to bridges, and you could argue
125. 5:44that cautious guys who chose the safe crossing might also
126. 5:48be more reluctant to cold call a woman.
127. 5:50The sample size was tiny.
128. 5:52But here's the good news.
129. 5:53Replications using roller coasters
130. 5:55and horror movies confirm the arousal-attraction transfer is real.
131. 5:59It's just more modest and shorter-lived than originally thought.
132. 6:03And notice one more piece of elegant design.
133. 6:05Why not just ask the men how attracted they felt?
134. 6:08Because that answer would be worthless.
135. 6:11If you've followed my work for any length of time,
136. 6:13you know this is one of the foundational truths of neuromarketing.
137. 6:16People can't or won't reliably tell you why they did what they did,
138. 6:21explain how they feel, or predict what they will do in the future.
139. 6:24The scientists didn't need a questionnaire.
140. 6:26They only needed a phone number and a count of who called.
141. 6:29This raises a fun question.
142. 6:31If people can't explain their own behavior,
143. 6:33what happens when you make them announce their choices out loud in public?
144. 6:37That question brings us to the beer.
145. 6:40This one is Ariely's own favorite,
146. 6:41and it's one with a direct payoff for marketers.
147. 6:44He and Jonathan Levav ran it at the Carolina Brewery in Chapel Hill.
148. 6:49Tables of diners were offered a free sample from a menu of four beers.
149. 6:53At some tables, everyone ordered the normal way, out loud,
150. 6:56one after another, while the rest of the table listened.
151. 6:59At other tables, everyone wrote their choice privately on a card.
152. 7:03The tables ordering out loud chose significantly more variety.
153. 7:07Nobody, it seems, wants to be the fourth person at the table to say,
154. 7:10I'll have the same thing.
155. 7:11But here's the twist, and this is the part I love.
156. 7:14The people who ordered out loud enjoyed their beer less.
157. 7:18All that variety-seeking produced regret.
158. 7:22And the happiest drinker at the table in those groups
159. 7:25was the person who ordered first,
160. 7:27the only one who chose without hearing anyone else's pick.
161. 7:30The essence here isn't beer.
162. 7:32It's what happens when a private
163. 7:34preference becomes part of a public performance.
164. 7:36People stop choosing what they want
165. 7:38and start choosing what their choice says about them.
166. 7:41Now, swap out the brewery table.
167. 7:43Make it a boardroom where opinions are given in order of seniority.
168. 7:47A Zoom call where the boss weighs in first.
169. 7:50A couple choosing options at a car dealership
170. 7:52while watching each other's reactions.
171. 7:54The same distortion kicks in.
172. 7:56The later you speak,
173. 7:57the less your stated preference reflects your actual preference.
174. 8:01One quick note, I'm not aware
175. 8:03of an authoritative replication of the beer study,
176. 8:05but the mechanism, public settings, distorting private preferences,
177. 8:09shows up all over the literature.
178. 8:11So far we've messed with people's hearts and their pride.
179. 8:15What happens when you mess with the meaning of their work?
180. 8:17That one needed Legos.
181. 8:19This was yet another Ariely experiment.
182. 8:22Students were paid to build 40-piece Lego Bionicle figures,
183. 8:26a little less money for each one they completed until they decided to stop.
184. 8:30In one condition, every finished Bionicle went on the desk,
185. 8:34where the builder could watch their completed work product grow.
186. 8:37In the other condition, Ariely has described this as the Sisyphus condition.
187. 8:42While the student built their second figure,
188. 8:45the experimenter destroyed the first one right in front of them
189. 8:48and dumped the pieces back into the box.
190. 8:50Let me be clear about what did not change.
191. 8:53The task, the pay, everything.
192. 8:55The only difference was watching your work get demolished.
193. 8:59The builders who could see their work accumulate made about ten
194. 9:02and a half Bionicles.
195. 9:03The Sisyphus group quit after about seven.
196. 9:06When Ariely told me about this study,
197. 9:08he mentioned a detail I think is the real finding.
198. 9:11Beforehand, people predicted
199. 9:13that meaningless condition would cost maybe one Bionicle of output.
200. 9:17The real gap was three times that.
201. 9:20We know meaning matters,
202. 9:22but we hugely underestimate how much it matters.
203. 9:25And there was something darker in the data.
204. 9:28In the normal condition,
205. 9:29the students who genuinely love Legos worked longest.
206. 9:32Passion predicted persistence, exactly as you'd expect.
207. 9:36In the Sisyphus condition, that relationship vanished entirely.
208. 9:40Loving the work no longer mattered.
209. 9:43As Dan put it to me, we have sucked the joy out of them.
210. 9:47I'm sure you don't need the business translation spelled out,
211. 9:50because you've seen it happen.
212. 9:52The reorg that made six months of a team's work useless.
213. 9:55The report you slaved over but nobody read.
214. 9:58The project cancelled the week before launch.
215. 10:00The leaders making these decisions didn't cut anyone's pay,
216. 10:04but they dismantled their Bionicles right in front of them.
217. 10:07And the best people, the ones who love the work,
218. 10:10are exactly the ones this behavior breaks.
219. 10:13When we first moved to Texas, we had some construction done in our house.
220. 10:17More than once.
221. 10:18We saw the workers building stone walls weren't following the blueprints.
222. 10:21Each time the project manager said, Tell them to keep working.
223. 10:25I'll be there in the morning to sort it out.
224. 10:27and in the morning he had the workers tear down their work and redo it.
225. 10:30I didn't have Ariely's research then,
226. 10:32but I couldn't imagine anything more demotivating than
227. 10:35being told to destroy a day's worth of one's own work
228. 10:38laying heavy stone blocks in Texas summer heat.
229. 10:41Okay, one study left, and I've saved it for last
230. 10:44because it's the one where a pure marketing variable acts like a drug.
231. 10:48Baba Shiv, Ziv Carmon,
232. 10:50and Ariely sold students a SoBe energy drink.
233. 10:53the kind that promises to up your mental focus.
234. 10:56Some students paid full price, others got it at a discount.
235. 11:00Then everyone sat down to solve word puzzles.
236. 11:03The full-price group solved 5.8 puzzles on average.
237. 11:06The discount group managed just 4.2.
238. 11:09Same drink, absolutely identical.
239. 11:11The only thing that changed was the price tag,
240. 11:14and the subjects' brains subconsciously adjusted the product's
241. 11:18actual potency to match.
242. 11:19The price wasn't a mere aspect of the product,
243. 11:22the price was acting like a pharmaceutical.
244. 11:24And of the four studies today,
245. 11:26this is the one with the strongest replication record.
246. 11:28The same team later showed
247. 11:30that full-price placebo painkillers relieved more pain than discounted ones.
248. 11:34Other labs have reproduced the marketing placebo effect using scarcity
249. 11:38and packaging instead of price.
250. 11:40Check my recent video about how the same wine
251. 11:43with a higher price tag tastes better.
252. 11:46This effect is real, it's robust,
253. 11:47and it's been verified from multiple angles.
254. 11:50I've been writing about this for years
255. 11:52because the implication is one most companies still refuse to believe.
256. 11:56When you discount a product, you may not just be trimming your margin,
257. 12:00you may be making the product objectively worse for the customer.
258. 12:04The experience itself can degrade.
259. 12:07That's the potential hidden risk from every promotion,
260. 12:11every coupon, every 20% off your first order.
261. 12:14Now sometimes discounting is the right move,
262. 12:16but you should know that it can change the product experience.
263. 12:20So let's wrap up.
264. 12:21Here's what I see when companies run experiments.
265. 12:24The methodology is usually fine.
266. 12:26The problem is nobody asked what the essence
267. 12:29was before hitting run.
268. 12:31You find out that the orange
269. 12:32button beat the blue button on the landing page.
270. 12:35Which is nice, but it transfers to nothing else.
271. 12:38Here are four things the cockroach and beer scientists do
272. 12:42that most growth teams don't.
273. 12:44One: Specify the principle before you pick the variable.
274. 12:47Don't test headline A versus headline B.
275. 12:50Test, does reducing perceived risk increase signups?
276. 12:54The first gives you a one-time winner.
277. 12:56The second gives you a principle you can use everywhere.
278. 12:59Two: Exaggerate the manipulation.
279. 13:02Ariely didn't gently dial down the meaning in the Lego study.
280. 13:06He tore the models apart in front of the builders.
281. 13:08If you can't detect an effect at full strength,
282. 13:11don't waste your time A/B testing a diluted version.
283. 13:14Test the maximum first, then dial back.
284. 13:17Three: Remove everything that isn't the essence.
285. 13:21The bridge study didn't need an app, a personality inventory,
286. 13:24and a follow-up survey.
287. 13:26It needed a phone number and a count of who called.
288. 13:28Every variable you add is noise you'll have to explain later.
289. 13:32Adding friction reduces participation.
290. 13:35Four: Flip the essence and see if the behavior flips.
291. 13:39This is the step almost nobody does.
292. 13:41If your explanation is right,
293. 13:43reversing the principle should reverse the result.
294. 13:45And that fourth rule brings us back to those cockroaches,
295. 13:48because I told you there would be a twist.
296. 13:50In 2020, 50 years after Zajonc's original experiment,
297. 13:54German researchers ran a preregistered replication
298. 13:57with a much bigger sample of roaches.
299. 13:59The audience effect on the hard maze held up.
300. 14:02Roaches still slowed down.
301. 14:03But the other part, the speed-up on the easy task?
302. 14:06Those roaches ran slower too.
303. 14:09The replicators also noted, a bit awkwardly,
304. 14:11that the famous original paper never ran a standard
305. 14:15statistical analysis on its widely publicized result.
306. 14:18Now, is that bad news?
307. 14:20For the textbooks, sure,
308. 14:21but that's science working exactly how it's supposed to,
309. 14:24and it was only possible because the original experimental design
310. 14:28was so simple, so stripped to its essence
311. 14:30that someone could rerun it half a century later
312. 14:33and find out exactly what survived.
313. 14:35There's a line from my conversations with Ariely
314. 14:37that you should keep in mind when a test of yours doesn't work.
315. 14:40He said, There's nothing quite as humiliating as having an idea,
316. 14:45testing it, and watching it fail.
317. 14:47And he meant that as praise for testing,
318. 14:50because the alternative, never finding out, is worse.
319. 14:53Our intuition is surprisingly unreliable,
320. 14:56and the whole point of these strange little experiments is to find out
321. 15:00where it fails us.
322. 15:01So here's your takeaway.
323. 15:02The best experiments look silly to outsiders precisely
324. 15:06because they aren't about their obvious subject.
325. 15:08Not roaches, not bridges, not beer.
326. 15:11Before you and your team plan your next test, ask one question.
327. 15:15What do we actually believe is happening here?
328. 15:17If nobody can state the essence in a sentence,
329. 15:20you don't have an experiment yet.
330. 15:22And when you do get a beautiful result, don't forget the roaches.
331. 15:25Rerun it before you build an important strategy on it.
332. 15:29Remember, keep it simple, find the essence,
333. 15:33and be nice to your cockroaches.
334. 15:35Thanks for tuning in.