YouTube2Text

Harvard Just Discovered AI Isn’t Actually Working — Transcript

by Dylan John · 2,671 words · 415 segments · language en · Watch on YouTube

Full transcript

  1. 0:00Dario promised a country full of
  2. 0:02geniuses in the data center. Elon Musk
  3. 0:04said AI may soon exceed
  4. 0:06>> some of human intelligence.
  5. 0:08>> But Harvard just released a new study on
  6. 0:09AI in the workplace. And as an AI
  7. 0:11engineer, I was truly horrified. I
  8. 0:15distilled every ounce of what I could
  9. 0:16find into this single video. So I want
  10. 0:19to look at how AI is slowing down work,
  11. 0:20[music] but not in the places where you
  12. 0:22might expect. Why you will lose your job
  13. 0:24due to AI, not to AI, and why the AI
  14. 0:27takeover hasn't happened at all how
  15. 0:29we've expected it to.
  16. 0:31>> Annual budgets blown through in weeks.
  17. 0:33>> US companies have cited AI as the number
  18. 0:36one reason for layoffs.
  19. 0:37>> Some employees say that dream is turning
  20. 0:40into a nightmare.
  21. 0:44>> I was in a meeting with my director a
  22. 0:45couple days ago. We were walking through
  23. 0:47AI spend across the entire company. We
  24. 0:49are spending over [music] $6 million a
  25. 0:52month on tokens. That is $200,000 a day.
  26. 0:55>> Companies are facing a choice nobody has
  27. 0:57ever had to [music] make before, tokens
  28. 0:59or humans.
  29. 1:00>> That amount of money could treat 6
  30. 1:02million cases of malaria. You could feed
  31. 1:041.2 million children for an entire
  32. 1:07school year, and you could almost pay
  33. 1:09Tim Cook. Don't worry, we will get to
  34. 1:11the Harvard study in a moment. But the
  35. 1:12question everyone is asking, is AI
  36. 1:14actually helping? We are spending all of
  37. 1:17this money. It has to be tied to ROI. It
  38. 1:19has to be helping our our business
  39. 1:21somehow, right? Well, let's back up a
  40. 1:22mere 365
  41. 1:25days. The conventional wisdom was that
  42. 1:27AI made coding obsolete.
  43. 1:29>> I have engineers within Anthropic who
  44. 1:31say I don't write any code anymore.
  45. 1:33>> Anthropic was sending out job offers
  46. 1:34with this role may not exist in 12
  47. 1:36months. Bubble talk was at its peak, and
  48. 1:38MIT found 95% of corporate AI pilots
  49. 1:42failed. News that I thought was going to
  50. 1:44crash the entire economy. But despite
  51. 1:46all of that, a research group named
  52. 1:48Meter dropped an absolute bombshell. 16
  53. 1:51developers, 246 tasks, a randomized
  54. 1:54controlled trial on software
  55. 1:56engineering. I remember that day vividly
  56. 1:57at work. Slack, email, Zoom, everyone
  57. 2:00was about it like Trump had just cured
  58. 2:02cancer. And I did it. The main takeaway
  59. 2:05was that software engineers were 19%
  60. 2:07slower utilizing AI. Managers blamed the
  61. 2:10individuals, engineers blamed the study,
  62. 2:12but engineers, engineers alone truly
  63. 2:14believed that AI was making them
  64. 2:16quicker. Call it ignorance, call it
  65. 2:17confidence, call it whatever the
  66. 2:19you want. But Meta heard people loud and
  67. 2:21clear. And to prove their point, they
  68. 2:23declared that they would replicate their
  69. 2:25atom bomb, 47 new devs, 10 from the
  70. 2:27original, $100 pay reduction. That last
  71. 2:31point not relevant, but really funny to
  72. 2:33me. Subscribe to the channel for more
  73. 2:35jokes that make me laugh. What they
  74. 2:38found was shocking. Nothing like this
  75. 2:39has ever happened in the course of human
  76. 2:41history, and it really shows how quickly
  77. 2:43AI has been advancing. [music]
  78. 2:44>> Problems are getting harder,
  79. 2:46but the progress is getting faster.
  80. 2:49>> But let's take a moment to look at what
  81. 2:51changed in that time because it was a
  82. 2:54lot. [music] From Claude 3.7 to GPT-5 in
  83. 2:56August to Opus 4.5 in November, we are
  84. 2:59talking huge leaps here, like going from
  85. 3:01the iPhone [music] 3G to the iPhone 18.
  86. 3:04It's like going from dial-up internet to
  87. 3:05fiber, running a 5K versus a marathon.
  88. 3:08But something seriously changed that
  89. 3:10December.
  90. 3:14December of last year was universally
  91. 3:15known as the moment Claude got good.
  92. 3:18OpenAI co-founder said, I've never felt
  93. 3:20this much behind as a programmer.
  94. 3:22Wikipedia said the winter holiday was
  95. 3:24when Claude code went viral. And not
  96. 3:27even a month later, Anthropic started
  97. 3:28telling everyone that AI now writes 100%
  98. 3:32of their code. My personal favorite
  99. 3:34viral trend was the era of the
  100. 3:36leaderboard.
  101. 3:37>> Believe it or not, Silicon Valley
  102. 3:38engineers are now being scored on how
  103. 3:40many AI tokens they use.
  104. 3:42>> Meta adding rankings for who spent the
  105. 3:44most tokens. Microsoft and Uber followed
  106. 3:47suit. Jensen Huang said he would be
  107. 3:49deeply
  108. 3:50alarmed if his $500,000 engineer did not
  109. 3:53consume at least $250,000
  110. 3:56of tokens.
  111. 3:57>> I will go ape something else.
  112. 3:59>> Yes.
  113. 3:59>> Right.
  114. 4:00>> A small three-bedroom house for middle
  115. 4:03America, resulting in the legendary
  116. 4:05meme, CEO of Oreos, I would advise that
  117. 4:08everyone eats 50 Oreos a day, maybe even
  118. 4:11100. Fast forward not even a few weeks
  119. 4:13and Amazon was begging engineers to not
  120. 4:15use AI just for the sake of using AI.
  121. 4:17What a golden era. But the second time
  122. 4:19Meter tried to replicate the study,
  123. 4:21engineers completely refused to do work
  124. 4:23without artificial intelligence. Meter
  125. 4:24admitted that the study was now entirely
  126. 4:27corrupted due to that fact. The
  127. 4:29developers most sped up by AI won't
  128. 4:31participate in the no AI condition.
  129. 4:33You're sampling a population that is
  130. 4:34already pre-filtered out the people with
  131. 4:36the strongest positive signal. That's
  132. 4:38called selection bias. Meter finally
  133. 4:40gave up and by the time they did
  134. 4:41concluded AI speeding up developers now
  135. 4:44seemed likely. The only caveat, no
  136. 4:46engineer will go without it, so who the
  137. 4:49really knows?
  138. 4:50Well, Harvard?
  139. 4:54Let's start with the first question.
  140. 4:55Does AI make coding faster? Meter had no
  141. 4:58idea, but Harvard did. Researchers at
  142. 5:00Harvard got inside 718 real companies
  143. 5:02accessing data inside an analytics
  144. 5:04platform. It monitored planning tasks in
  145. 5:06Jira, coding in GitHub, pull requests,
  146. 5:08deployment, [music] and even your
  147. 5:09calendar. And some Harvard distilled 300
  148. 5:12million different data points over five
  149. 5:14consecutive years into an aggregate
  150. 5:16[music] study. They concluded an
  151. 5:18unequivocal yes, coding is quicker.
  152. 5:21Companies with AI agents, Cloud Code,
  153. 5:22Cursor, Codex wrote 30% more lines,
  154. 5:25committed 20% more code, and opened pull
  155. 5:27requests 23% more often. But Harvard
  156. 5:30decided to take that problem one step
  157. 5:32further than Meter did. The authors of
  158. 5:34the study followed the code through the
  159. 5:36finish line. [music] Listen, I've worked
  160. 5:38in tech for almost 10 years and I could
  161. 5:39not I could not believe this. I've spent
  162. 5:42the last year as an engineering manager
  163. 5:44and I can say it's easier than ever to
  164. 5:46generate code. One prompt can create
  165. 5:48[music] thousands of lines in minutes.
  166. 5:50Tests on test specs, documentation, PRs.
  167. 5:53I wouldn't be able to keep up with the
  168. 5:54rate of change on my team if I worked
  169. 5:55every minute of every hour of every day.
  170. 5:57It's that bad or that easy depending on
  171. 6:00how you look at it. A senior engineer
  172. 6:02even described simple tasks as feeling
  173. 6:04100 times quicker with AI. You might be
  174. 6:07thinking, that's it? Video over. AI wins
  175. 6:09and takes our jobs again. But if you
  176. 6:12analyze how engineers spend their time
  177. 6:13in figure 10, the left is computed time.
  178. 6:15The right is surveyed. You can see
  179. 6:18engineers typically spend 42% of their
  180. 6:20time coding, 18.6% of their time doing
  181. 6:22code review, and a whopping [music]
  182. 6:2439.5%
  183. 6:26of their time managing meetings, syncs,
  184. 6:28documents, and so on. Man, that last
  185. 6:30number is way higher than I would
  186. 6:31expect. But what Harvard found was a
  187. 6:33zero change in code delivered. Completed
  188. 6:36Jira tasks didn't increase, finished
  189. 6:38projects didn't go up, and the size of
  190. 6:40tasks didn't change. But maybe you
  191. 6:42should fire that software engineer
  192. 6:43because where the did all of that extra
  193. 6:45time go? He being lazy, buddy? Want a
  194. 6:47pip? Want a pip? Where a lot of that
  195. 6:49time went was a lot of different places.
  196. 6:51To start, pull requests took 49% longer
  197. 6:54to approve. The amount of time something
  198. 6:55needed changing in code review increased
  199. 6:58twofold, and reviewers left 35% more
  200. 7:01comments than before AI. One senior
  201. 7:03engineer spent six months waiting for
  202. 7:05legal approval for something that should
  203. 7:07have probably taken a month. Coding
  204. 7:09agents can dramatically increase coding
  205. 7:11productivity, but Jira tickets often
  206. 7:13capture bottlenecks that AI cannot
  207. 7:14solve. Another engineer said the real
  208. 7:16bottleneck has been communication
  209. 7:18between business and product teams,
  210. 7:20review and QA. You can generate features
  211. 7:22much faster, but they still need to be
  212. 7:24validated and improved by a human. The
  213. 7:27point is time shifted. It swam up and
  214. 7:30downstream. All the time saved on
  215. 7:32implementation left more time to be
  216. 7:34spent in code review, training, or
  217. 7:36arguing with your product manager.
  218. 7:37There's another really problematic place
  219. 7:39you can be spending your time. Look at
  220. 7:41Amazon. In March, their website went
  221. 7:43down for six hours, preventing customers
  222. 7:45from completing transactions, viewing
  223. 7:47account details, or accessing product
  224. 7:50pages.
  225. 7:51>> This was a big headache for a lot of
  226. 7:53people trying to make a purchase on
  227. 7:54Amazon.
  228. 7:55>> Amazon lost roughly 6.3 million orders
  229. 7:58in a single day, amounting to hundreds
  230. 7:59of millions of dollars in lost revenue.
  231. 8:01And listen, I'm not the type of person
  232. 8:03to lose any sleep over Jeff Bezos not
  233. 8:05being a little richer, but Amazon root
  234. 8:07caused the problem to a gen AI assisted
  235. 8:10change and serious consequences were a
  236. 8:12result. First, Amazon initiated a 90-day
  237. 8:15code safety reset, introducing [music]
  238. 8:16intentional friction to the coding
  239. 8:19process. Keep that word friction in your
  240. 8:21mind, by the way. Second, senior
  241. 8:23engineers are now required to sign off
  242. 8:24on all all AI assisted code changes made
  243. 8:27by lower level engineers. And third,
  244. 8:30Amazon mandated documentation before
  245. 8:33deploying critical changes. More
  246. 8:35friction. Does anything I just said
  247. 8:37sound like speeding up and
  248. 8:39revolutionizing the workflows to you?
  249. 8:41>> It's going to reinvent every single
  250. 8:43customer experience we know.
  251. 8:44>> Amazon had to intentionally add friction
  252. 8:47to their process to prevent problems
  253. 8:49from AI. But let's expand our lens a bit
  254. 8:51bigger because AI is obviously being
  255. 8:53used in way more places besides just
  256. 8:55coding. My little brother's in medical
  257. 8:57school. I've been meaning to ask him if
  258. 8:59he uses one of these things, but this is
  259. 9:01an AI scribe. It's built to listen to
  260. 9:03consultations and use gen AI to draft
  261. 9:05structured clinical notes, patient
  262. 9:07information, and other admin materials.
  263. 9:09A mega study was done by JAMA showing AI
  264. 9:12scribe adoption was associated with
  265. 9:13decreases in total charting and
  266. 9:16documentation time of 13.4 and 16
  267. 9:19minutes. That's great, but physicians
  268. 9:21working with AI scribes spent 23.2% of
  269. 9:25their patient encounter finishing the
  270. 9:26chart compared to 16.2% [music]
  271. 9:28without one, meaning less time with the
  272. 9:30patient, more time with the AI. The main
  273. 9:34culprit authors cited was physicians
  274. 9:36needed to review, audit, and edit the
  275. 9:38tone of AI dictation, which I will say
  276. 9:40is not exactly the same as reviewing
  277. 9:42code. However, it rhymes. It rhymes. You
  278. 9:45see the same thing playing out in the
  279. 9:47legal field where some firms have been
  280. 9:49fined hundreds of thousands of dollars
  281. 9:51for AI hallucinations in reports.
  282. 9:53>> The courts turning up the heat on some
  283. 9:55attorneys using AI to write legal briefs
  284. 9:58and arguments.
  285. 9:59>> You see the same thing in accounting.
  286. 10:01>> Deloitte forced to refund the government
  287. 10:03partially after it was found a report it
  288. 10:05produced was riddled with fake AI
  289. 10:07references and citations.
  290. 10:09>> Well, let me know in
  291. 10:10>> the comments how you use AI at work. But
  292. 10:13I want to expand Harvard's research to
  293. 10:14the next level because if AI isn't
  294. 10:16actually working at improving
  295. 10:18efficiency, then what is it actually
  296. 10:19doing? American Express is cutting 5% of
  297. 10:22its workforce to prepare for AI. The
  298. 10:24cuts come in a strong year. Net income
  299. 10:26rose to 3.7 billion from 2.9 billion a
  300. 10:28year earlier. Earnings per share climbed
  301. 10:3031%. Amex preparing for AI. What the
  302. 10:34hell does that even mean? AI is here. It
  303. 10:37it it's been here. It's been here for
  304. 10:38two years. You're preparing now? Netflix
  305. 10:41laid off 1,000 employees despite being
  306. 10:43growing and being more profitable than
  307. 10:45ever before.
  308. 10:47>> The streamer is reportedly laying off
  309. 10:481,000 employees today. That's about 6%
  310. 10:51of its workforce.
  311. 10:52>> This year, they aim to grow 14%, raise
  312. 10:55operating margin, and Netflix is sitting
  313. 10:57on a teeny tiny 9 billion dollars in
  314. 11:00cash. 9 billion dollars in cash and they
  315. 11:02they still can't make a TV show for But
  316. 11:04Microsoft did something more bizarre
  317. 11:06than everything I have seen this year
  318. 11:08combined.
  319. 11:08>> Tech company now offering buyouts to
  320. 11:1160,000 workers.
  321. 11:12>> Microsoft offered 60,000 workers
  322. 11:14voluntary exit packages. That's more
  323. 11:16than three and a half times every
  324. 11:18Starbucks location in the United States
  325. 11:20combined. Two baseball stadiums worth of
  326. 11:22the Tampa Bay Rays fans. I'll pass it
  327. 11:24back to Harvard. We find that both
  328. 11:26technologies increase coding
  329. 11:28productivity with larger effects for AI
  330. 11:30agents. Despite these gains, we find
  331. 11:32little evidence that firms can increase
  332. 11:34software [music] output or reduce
  333. 11:36employment as a result. But what did
  334. 11:38Microsoft site? AI efficiency. Why keep
  335. 11:41100% when you can just make the 94% do
  336. 11:44the work of 100?
  337. 11:48It's some pretty toxic behavior for CEOs
  338. 11:50to keep their employees in constant fear
  339. 11:52of layoffs. Meta is the most notorious
  340. 11:55example. Every moment of every week,
  341. 11:57people are wondering if they're going to
  342. 11:58be the part of the next round of AI
  343. 12:00assisted layoffs.
  344. 12:01>> And from that point, we'll no longer be
  345. 12:03expected to work for Meta.
  346. 12:05>> The work culture has become more toxic
  347. 12:07than ever as a result. But to be
  348. 12:09pretending for a moment that AI is
  349. 12:11speeding up so many workflows, coding,
  350. 12:13automation, we don' need people anymore,
  351. 12:15is just plainly ridiculous. It's like
  352. 12:18saying you don' need as many people
  353. 12:19working at a McDonald's because cars
  354. 12:21drive around the drive-through a little
  355. 12:23bit quicker. You might lose your job due
  356. 12:25to AI, but not to AI, because the CEO is
  357. 12:29claiming that AI is improving efficiency
  358. 12:31the most also happen to be the ones
  359. 12:33laying off the quickest in the name of
  360. 12:35AI. We have not conducted a single
  361. 12:37AI-induced layoff at the company I work
  362. 12:40for. We are hiring more now than ever,
  363. 12:42and we have been no less impacted by the
  364. 12:44so-called AI transformation than anyone
  365. 12:46else. And ironically, we've also been
  366. 12:48the least vocal about it. Harvard put it
  367. 12:50very eloquently when they said the
  368. 12:52economics of AI adoption cannot be
  369. 12:54understood by studying automated tasks
  370. 12:56in isolation. Production processes
  371. 12:58consist of multiple complementary
  372. 12:59activities, and accelerating one stage
  373. 13:01often just shifts the constraint
  374. 13:04somewhere else. It's like rushing along
  375. 13:06a stream just to damn it up down the
  376. 13:08road, which for software engineers
  377. 13:09specifically means you get to automate a
  378. 13:11ton of your work for free. Things
  379. 13:13that took weeks take days, things that
  380. 13:15took days take hours, things that took
  381. 13:17hours take minutes. But the big if is if
  382. 13:20you still have a job and haven't been
  383. 13:22let go by a CEO pretending to replace
  384. 13:24it, in the grand scheme of things,
  385. 13:26someone working in an industry where AI
  386. 13:28does all of your work is a pretty sweet
  387. 13:30[music] deal. I call going into work now
  388. 13:33to write software the prompt factory
  389. 13:35because that's really what it is. You're
  390. 13:36basically on an assembly line in an
  391. 13:38office cubicle, you're tip-typing away
  392. 13:40to Cloud, waiting for something to
  393. 13:41respond, you're going through your loops
  394. 13:42of different agents and sending software
  395. 13:44on its way. Not to say expectations
  396. 13:46haven't just risen to mean you have to
  397. 13:48do 10 times more as a result because
  398. 13:50that has also definitely happened. But
  399. 13:52there's not a bunch of geniuses sitting
  400. 13:54in a data center, there's a bunch of
  401. 13:55software engineers in offices waiting
  402. 13:57for Cloud to respond. There's not the
  403. 14:00entire sum of human intelligence in AI,
  404. 14:02there's a bunch of data nerds
  405. 14:03transcribing their thoughts to code
  406. 14:05because in a way the work is automated,
  407. 14:08but the human is still required. All
  408. 14:10while CEOs foot the inexorbitant bill in
  409. 14:13the name of being AI first. I'd like to
  410. 14:15thank our channel members. Hit that join
  411. 14:17button down below for exclusive access
  412. 14:19to videos just like these. But if you'd
  413. 14:21like to see why replacing humans with AI
  414. 14:23completely backfired, click the video on
  415. 14:26screen now.

About this transcript

This page contains the full transcript of Harvard Just Discovered AI Isn’t Actually Working by Dylan John, generated from the public captions YouTube serves with the video. The transcript has 2,671 words across 415 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.