Harvard Just Discovered AI Isn’t Actually Working — Transcript
Full transcript
- 0:00Dario promised a country full of
- 0:02geniuses in the data center. Elon Musk
- 0:04said AI may soon exceed
- 0:06>> some of human intelligence.
- 0:08>> But Harvard just released a new study on
- 0:09AI in the workplace. And as an AI
- 0:11engineer, I was truly horrified. I
- 0:15distilled every ounce of what I could
- 0:16find into this single video. So I want
- 0:19to look at how AI is slowing down work,
- 0:20[music] but not in the places where you
- 0:22might expect. Why you will lose your job
- 0:24due to AI, not to AI, and why the AI
- 0:27takeover hasn't happened at all how
- 0:29we've expected it to.
- 0:31>> Annual budgets blown through in weeks.
- 0:33>> US companies have cited AI as the number
- 0:36one reason for layoffs.
- 0:37>> Some employees say that dream is turning
- 0:40into a nightmare.
- 0:44>> I was in a meeting with my director a
- 0:45couple days ago. We were walking through
- 0:47AI spend across the entire company. We
- 0:49are spending over [music] $6 million a
- 0:52month on tokens. That is $200,000 a day.
- 0:55>> Companies are facing a choice nobody has
- 0:57ever had to [music] make before, tokens
- 0:59or humans.
- 1:00>> That amount of money could treat 6
- 1:02million cases of malaria. You could feed
- 1:041.2 million children for an entire
- 1:07school year, and you could almost pay
- 1:09Tim Cook. Don't worry, we will get to
- 1:11the Harvard study in a moment. But the
- 1:12question everyone is asking, is AI
- 1:14actually helping? We are spending all of
- 1:17this money. It has to be tied to ROI. It
- 1:19has to be helping our our business
- 1:21somehow, right? Well, let's back up a
- 1:22mere 365
- 1:25days. The conventional wisdom was that
- 1:27AI made coding obsolete.
- 1:29>> I have engineers within Anthropic who
- 1:31say I don't write any code anymore.
- 1:33>> Anthropic was sending out job offers
- 1:34with this role may not exist in 12
- 1:36months. Bubble talk was at its peak, and
- 1:38MIT found 95% of corporate AI pilots
- 1:42failed. News that I thought was going to
- 1:44crash the entire economy. But despite
- 1:46all of that, a research group named
- 1:48Meter dropped an absolute bombshell. 16
- 1:51developers, 246 tasks, a randomized
- 1:54controlled trial on software
- 1:56engineering. I remember that day vividly
- 1:57at work. Slack, email, Zoom, everyone
- 2:00was about it like Trump had just cured
- 2:02cancer. And I did it. The main takeaway
- 2:05was that software engineers were 19%
- 2:07slower utilizing AI. Managers blamed the
- 2:10individuals, engineers blamed the study,
- 2:12but engineers, engineers alone truly
- 2:14believed that AI was making them
- 2:16quicker. Call it ignorance, call it
- 2:17confidence, call it whatever the
- 2:19you want. But Meta heard people loud and
- 2:21clear. And to prove their point, they
- 2:23declared that they would replicate their
- 2:25atom bomb, 47 new devs, 10 from the
- 2:27original, $100 pay reduction. That last
- 2:31point not relevant, but really funny to
- 2:33me. Subscribe to the channel for more
- 2:35jokes that make me laugh. What they
- 2:38found was shocking. Nothing like this
- 2:39has ever happened in the course of human
- 2:41history, and it really shows how quickly
- 2:43AI has been advancing. [music]
- 2:44>> Problems are getting harder,
- 2:46but the progress is getting faster.
- 2:49>> But let's take a moment to look at what
- 2:51changed in that time because it was a
- 2:54lot. [music] From Claude 3.7 to GPT-5 in
- 2:56August to Opus 4.5 in November, we are
- 2:59talking huge leaps here, like going from
- 3:01the iPhone [music] 3G to the iPhone 18.
- 3:04It's like going from dial-up internet to
- 3:05fiber, running a 5K versus a marathon.
- 3:08But something seriously changed that
- 3:10December.
- 3:14December of last year was universally
- 3:15known as the moment Claude got good.
- 3:18OpenAI co-founder said, I've never felt
- 3:20this much behind as a programmer.
- 3:22Wikipedia said the winter holiday was
- 3:24when Claude code went viral. And not
- 3:27even a month later, Anthropic started
- 3:28telling everyone that AI now writes 100%
- 3:32of their code. My personal favorite
- 3:34viral trend was the era of the
- 3:36leaderboard.
- 3:37>> Believe it or not, Silicon Valley
- 3:38engineers are now being scored on how
- 3:40many AI tokens they use.
- 3:42>> Meta adding rankings for who spent the
- 3:44most tokens. Microsoft and Uber followed
- 3:47suit. Jensen Huang said he would be
- 3:49deeply
- 3:50alarmed if his $500,000 engineer did not
- 3:53consume at least $250,000
- 3:56of tokens.
- 3:57>> I will go ape something else.
- 3:59>> Yes.
- 3:59>> Right.
- 4:00>> A small three-bedroom house for middle
- 4:03America, resulting in the legendary
- 4:05meme, CEO of Oreos, I would advise that
- 4:08everyone eats 50 Oreos a day, maybe even
- 4:11100. Fast forward not even a few weeks
- 4:13and Amazon was begging engineers to not
- 4:15use AI just for the sake of using AI.
- 4:17What a golden era. But the second time
- 4:19Meter tried to replicate the study,
- 4:21engineers completely refused to do work
- 4:23without artificial intelligence. Meter
- 4:24admitted that the study was now entirely
- 4:27corrupted due to that fact. The
- 4:29developers most sped up by AI won't
- 4:31participate in the no AI condition.
- 4:33You're sampling a population that is
- 4:34already pre-filtered out the people with
- 4:36the strongest positive signal. That's
- 4:38called selection bias. Meter finally
- 4:40gave up and by the time they did
- 4:41concluded AI speeding up developers now
- 4:44seemed likely. The only caveat, no
- 4:46engineer will go without it, so who the
- 4:49really knows?
- 4:50Well, Harvard?
- 4:54Let's start with the first question.
- 4:55Does AI make coding faster? Meter had no
- 4:58idea, but Harvard did. Researchers at
- 5:00Harvard got inside 718 real companies
- 5:02accessing data inside an analytics
- 5:04platform. It monitored planning tasks in
- 5:06Jira, coding in GitHub, pull requests,
- 5:08deployment, [music] and even your
- 5:09calendar. And some Harvard distilled 300
- 5:12million different data points over five
- 5:14consecutive years into an aggregate
- 5:16[music] study. They concluded an
- 5:18unequivocal yes, coding is quicker.
- 5:21Companies with AI agents, Cloud Code,
- 5:22Cursor, Codex wrote 30% more lines,
- 5:25committed 20% more code, and opened pull
- 5:27requests 23% more often. But Harvard
- 5:30decided to take that problem one step
- 5:32further than Meter did. The authors of
- 5:34the study followed the code through the
- 5:36finish line. [music] Listen, I've worked
- 5:38in tech for almost 10 years and I could
- 5:39not I could not believe this. I've spent
- 5:42the last year as an engineering manager
- 5:44and I can say it's easier than ever to
- 5:46generate code. One prompt can create
- 5:48[music] thousands of lines in minutes.
- 5:50Tests on test specs, documentation, PRs.
- 5:53I wouldn't be able to keep up with the
- 5:54rate of change on my team if I worked
- 5:55every minute of every hour of every day.
- 5:57It's that bad or that easy depending on
- 6:00how you look at it. A senior engineer
- 6:02even described simple tasks as feeling
- 6:04100 times quicker with AI. You might be
- 6:07thinking, that's it? Video over. AI wins
- 6:09and takes our jobs again. But if you
- 6:12analyze how engineers spend their time
- 6:13in figure 10, the left is computed time.
- 6:15The right is surveyed. You can see
- 6:18engineers typically spend 42% of their
- 6:20time coding, 18.6% of their time doing
- 6:22code review, and a whopping [music]
- 6:2439.5%
- 6:26of their time managing meetings, syncs,
- 6:28documents, and so on. Man, that last
- 6:30number is way higher than I would
- 6:31expect. But what Harvard found was a
- 6:33zero change in code delivered. Completed
- 6:36Jira tasks didn't increase, finished
- 6:38projects didn't go up, and the size of
- 6:40tasks didn't change. But maybe you
- 6:42should fire that software engineer
- 6:43because where the did all of that extra
- 6:45time go? He being lazy, buddy? Want a
- 6:47pip? Want a pip? Where a lot of that
- 6:49time went was a lot of different places.
- 6:51To start, pull requests took 49% longer
- 6:54to approve. The amount of time something
- 6:55needed changing in code review increased
- 6:58twofold, and reviewers left 35% more
- 7:01comments than before AI. One senior
- 7:03engineer spent six months waiting for
- 7:05legal approval for something that should
- 7:07have probably taken a month. Coding
- 7:09agents can dramatically increase coding
- 7:11productivity, but Jira tickets often
- 7:13capture bottlenecks that AI cannot
- 7:14solve. Another engineer said the real
- 7:16bottleneck has been communication
- 7:18between business and product teams,
- 7:20review and QA. You can generate features
- 7:22much faster, but they still need to be
- 7:24validated and improved by a human. The
- 7:27point is time shifted. It swam up and
- 7:30downstream. All the time saved on
- 7:32implementation left more time to be
- 7:34spent in code review, training, or
- 7:36arguing with your product manager.
- 7:37There's another really problematic place
- 7:39you can be spending your time. Look at
- 7:41Amazon. In March, their website went
- 7:43down for six hours, preventing customers
- 7:45from completing transactions, viewing
- 7:47account details, or accessing product
- 7:50pages.
- 7:51>> This was a big headache for a lot of
- 7:53people trying to make a purchase on
- 7:54Amazon.
- 7:55>> Amazon lost roughly 6.3 million orders
- 7:58in a single day, amounting to hundreds
- 7:59of millions of dollars in lost revenue.
- 8:01And listen, I'm not the type of person
- 8:03to lose any sleep over Jeff Bezos not
- 8:05being a little richer, but Amazon root
- 8:07caused the problem to a gen AI assisted
- 8:10change and serious consequences were a
- 8:12result. First, Amazon initiated a 90-day
- 8:15code safety reset, introducing [music]
- 8:16intentional friction to the coding
- 8:19process. Keep that word friction in your
- 8:21mind, by the way. Second, senior
- 8:23engineers are now required to sign off
- 8:24on all all AI assisted code changes made
- 8:27by lower level engineers. And third,
- 8:30Amazon mandated documentation before
- 8:33deploying critical changes. More
- 8:35friction. Does anything I just said
- 8:37sound like speeding up and
- 8:39revolutionizing the workflows to you?
- 8:41>> It's going to reinvent every single
- 8:43customer experience we know.
- 8:44>> Amazon had to intentionally add friction
- 8:47to their process to prevent problems
- 8:49from AI. But let's expand our lens a bit
- 8:51bigger because AI is obviously being
- 8:53used in way more places besides just
- 8:55coding. My little brother's in medical
- 8:57school. I've been meaning to ask him if
- 8:59he uses one of these things, but this is
- 9:01an AI scribe. It's built to listen to
- 9:03consultations and use gen AI to draft
- 9:05structured clinical notes, patient
- 9:07information, and other admin materials.
- 9:09A mega study was done by JAMA showing AI
- 9:12scribe adoption was associated with
- 9:13decreases in total charting and
- 9:16documentation time of 13.4 and 16
- 9:19minutes. That's great, but physicians
- 9:21working with AI scribes spent 23.2% of
- 9:25their patient encounter finishing the
- 9:26chart compared to 16.2% [music]
- 9:28without one, meaning less time with the
- 9:30patient, more time with the AI. The main
- 9:34culprit authors cited was physicians
- 9:36needed to review, audit, and edit the
- 9:38tone of AI dictation, which I will say
- 9:40is not exactly the same as reviewing
- 9:42code. However, it rhymes. It rhymes. You
- 9:45see the same thing playing out in the
- 9:47legal field where some firms have been
- 9:49fined hundreds of thousands of dollars
- 9:51for AI hallucinations in reports.
- 9:53>> The courts turning up the heat on some
- 9:55attorneys using AI to write legal briefs
- 9:58and arguments.
- 9:59>> You see the same thing in accounting.
- 10:01>> Deloitte forced to refund the government
- 10:03partially after it was found a report it
- 10:05produced was riddled with fake AI
- 10:07references and citations.
- 10:09>> Well, let me know in
- 10:10>> the comments how you use AI at work. But
- 10:13I want to expand Harvard's research to
- 10:14the next level because if AI isn't
- 10:16actually working at improving
- 10:18efficiency, then what is it actually
- 10:19doing? American Express is cutting 5% of
- 10:22its workforce to prepare for AI. The
- 10:24cuts come in a strong year. Net income
- 10:26rose to 3.7 billion from 2.9 billion a
- 10:28year earlier. Earnings per share climbed
- 10:3031%. Amex preparing for AI. What the
- 10:34hell does that even mean? AI is here. It
- 10:37it it's been here. It's been here for
- 10:38two years. You're preparing now? Netflix
- 10:41laid off 1,000 employees despite being
- 10:43growing and being more profitable than
- 10:45ever before.
- 10:47>> The streamer is reportedly laying off
- 10:481,000 employees today. That's about 6%
- 10:51of its workforce.
- 10:52>> This year, they aim to grow 14%, raise
- 10:55operating margin, and Netflix is sitting
- 10:57on a teeny tiny 9 billion dollars in
- 11:00cash. 9 billion dollars in cash and they
- 11:02they still can't make a TV show for But
- 11:04Microsoft did something more bizarre
- 11:06than everything I have seen this year
- 11:08combined.
- 11:08>> Tech company now offering buyouts to
- 11:1160,000 workers.
- 11:12>> Microsoft offered 60,000 workers
- 11:14voluntary exit packages. That's more
- 11:16than three and a half times every
- 11:18Starbucks location in the United States
- 11:20combined. Two baseball stadiums worth of
- 11:22the Tampa Bay Rays fans. I'll pass it
- 11:24back to Harvard. We find that both
- 11:26technologies increase coding
- 11:28productivity with larger effects for AI
- 11:30agents. Despite these gains, we find
- 11:32little evidence that firms can increase
- 11:34software [music] output or reduce
- 11:36employment as a result. But what did
- 11:38Microsoft site? AI efficiency. Why keep
- 11:41100% when you can just make the 94% do
- 11:44the work of 100?
- 11:48It's some pretty toxic behavior for CEOs
- 11:50to keep their employees in constant fear
- 11:52of layoffs. Meta is the most notorious
- 11:55example. Every moment of every week,
- 11:57people are wondering if they're going to
- 11:58be the part of the next round of AI
- 12:00assisted layoffs.
- 12:01>> And from that point, we'll no longer be
- 12:03expected to work for Meta.
- 12:05>> The work culture has become more toxic
- 12:07than ever as a result. But to be
- 12:09pretending for a moment that AI is
- 12:11speeding up so many workflows, coding,
- 12:13automation, we don' need people anymore,
- 12:15is just plainly ridiculous. It's like
- 12:18saying you don' need as many people
- 12:19working at a McDonald's because cars
- 12:21drive around the drive-through a little
- 12:23bit quicker. You might lose your job due
- 12:25to AI, but not to AI, because the CEO is
- 12:29claiming that AI is improving efficiency
- 12:31the most also happen to be the ones
- 12:33laying off the quickest in the name of
- 12:35AI. We have not conducted a single
- 12:37AI-induced layoff at the company I work
- 12:40for. We are hiring more now than ever,
- 12:42and we have been no less impacted by the
- 12:44so-called AI transformation than anyone
- 12:46else. And ironically, we've also been
- 12:48the least vocal about it. Harvard put it
- 12:50very eloquently when they said the
- 12:52economics of AI adoption cannot be
- 12:54understood by studying automated tasks
- 12:56in isolation. Production processes
- 12:58consist of multiple complementary
- 12:59activities, and accelerating one stage
- 13:01often just shifts the constraint
- 13:04somewhere else. It's like rushing along
- 13:06a stream just to damn it up down the
- 13:08road, which for software engineers
- 13:09specifically means you get to automate a
- 13:11ton of your work for free. Things
- 13:13that took weeks take days, things that
- 13:15took days take hours, things that took
- 13:17hours take minutes. But the big if is if
- 13:20you still have a job and haven't been
- 13:22let go by a CEO pretending to replace
- 13:24it, in the grand scheme of things,
- 13:26someone working in an industry where AI
- 13:28does all of your work is a pretty sweet
- 13:30[music] deal. I call going into work now
- 13:33to write software the prompt factory
- 13:35because that's really what it is. You're
- 13:36basically on an assembly line in an
- 13:38office cubicle, you're tip-typing away
- 13:40to Cloud, waiting for something to
- 13:41respond, you're going through your loops
- 13:42of different agents and sending software
- 13:44on its way. Not to say expectations
- 13:46haven't just risen to mean you have to
- 13:48do 10 times more as a result because
- 13:50that has also definitely happened. But
- 13:52there's not a bunch of geniuses sitting
- 13:54in a data center, there's a bunch of
- 13:55software engineers in offices waiting
- 13:57for Cloud to respond. There's not the
- 14:00entire sum of human intelligence in AI,
- 14:02there's a bunch of data nerds
- 14:03transcribing their thoughts to code
- 14:05because in a way the work is automated,
- 14:08but the human is still required. All
- 14:10while CEOs foot the inexorbitant bill in
- 14:13the name of being AI first. I'd like to
- 14:15thank our channel members. Hit that join
- 14:17button down below for exclusive access
- 14:19to videos just like these. But if you'd
- 14:21like to see why replacing humans with AI
- 14:23completely backfired, click the video on
- 14:26screen now.
About this transcript
This page contains the full transcript of Harvard Just Discovered AI Isn’t Actually Working by Dylan John, generated from the public captions YouTube serves with the video. The transcript has 2,671 words across 415 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.