Transformers, explained: Understand the model behind GPT, BERT, and T5 — Transcript
Full transcript
- 0:00the neat thing about working in machine
- 0:01learning is that every few years
- 0:03somebody invents something crazy that
- 0:04makes you totally reconsider what's
- 0:06possible
- 0:07like models that can play go or generate
- 0:10hyper-realistic faces
- 0:12and today the mind-blowing discovery
- 0:13that's rocking everyone's world is a
- 0:15type of neural network called a
- 0:16transformer
- 0:17transformers are models that can
- 0:18translate text write poems and op-eds
- 0:21and even generate computer code these
- 0:23can be used in biology to solve the
- 0:24protein folding problem
- 0:26transformers are like this magical
- 0:28machine learning hammer that seems to
- 0:29make every problem into a nail if you've
- 0:31heard of the trendy new ml models bert
- 0:33or gpt-3 or t5 all of these models are
- 0:37based on transformers
- 0:38so if you want to stay hip in machine
- 0:40learning and especially natural language
- 0:42processing you have to know about the
- 0:44transformer so in this video i'm going
- 0:46to tell you about what transformers are
- 0:47how they work and why they've been so
- 0:49impactful let's get to it so what is a
- 0:52transformer it's a type of neural
- 0:54network architecture to recap neural
- 0:56networks are a very effective type of
- 0:58model for analyzing complicated data
- 1:00types like images videos audio and text
- 1:02but there are different types of neural
- 1:04networks optimized for different types
- 1:05of data like if you're analyzing images
- 1:07you typically use a convolutional neural
- 1:09network which is designed to vaguely
- 1:11mimic the way that the human brain
- 1:13processes vision and since around 2012
- 1:16neural networks have been really good at
- 1:17solving vision tasks like identifying
- 1:19objects and photos but for a long time
- 1:21we didn't have anything comparably good
- 1:23for analyzing language whether for
- 1:25translation or text summarization or
- 1:27text generation
- 1:28and this is a problem because language
- 1:30is the primary way that humans
- 1:31communicate you see until transformers
- 1:33came around the way we use deep learning
- 1:35to understand text was with a type of
- 1:36model called a recurrent neural network
- 1:39or an rnn that looks something like this
- 1:42let's say you wanted to translate a
- 1:44sentence from english to french
- 1:46an rnn would take as input an english
- 1:48sentence and process the words one at a
- 1:50time and then sequentially spit out
- 1:52their french counterparts the key word
- 1:54here is sequential in language the order
- 1:56of words matters and you can't just
- 1:58shuffle them around
- 1:59for example the sentence jane went
- 2:02looking for trouble means something very
- 2:04different in the sentence trouble went
- 2:05looking for jane
- 2:07so any model that's going to deal with
- 2:08language has to capture word order and
- 2:10recurrent neural networks do this by
- 2:12looking at one word at a time
- 2:13sequentially but rnns had a lot of
- 2:15problems
- 2:16first they never really did well at
- 2:17handling large sequences of text like
- 2:20long paragraphs or essays by the time
- 2:22they were analyzing the end of a
- 2:23paragraph they'd forget what happened in
- 2:24the beginning
- 2:25and even worse rnns were pretty hard to
- 2:28train because they processed words
- 2:30sequentially they couldn't paralyze well
- 2:32which means that you couldn't just speed
- 2:33them up by throwing lots of gpus at them
- 2:36and when you have a model that's slow to
- 2:37train you can't train it on all that
- 2:38much data this is where the transformer
- 2:40changed everything they were a model
- 2:42developed in 2017 by researchers at
- 2:44google and the university of toronto and
- 2:46they were initially designed to do
- 2:47translation but unlike recurrent neural
- 2:50networks you could really efficiently
- 2:51paralyze transformers and that meant
- 2:53that with the right hardware you could
- 2:54train some really big models
- 2:56how big
- 2:57really big
- 2:59remember gpt3 that model that writes
- 3:00poetry and code and has conversations
- 3:03that was trained on almost 45 terabytes
- 3:05of text data including like almost the
- 3:07entire public web
- 3:09[Music]
- 3:10so if you remember anything about
- 3:11transformers let it be this combine a
- 3:13model that scales really well with a
- 3:15huge data set and the results will
- 3:17probably blow your mind so how do these
- 3:19things actually work from the diagram of
- 3:21the paper it should be pretty clear
- 3:24or maybe not actually it's simpler than
- 3:26you might think there are three main
- 3:28innovations that make this model work so
- 3:29well
- 3:30positional encodings and attention and
- 3:32specifically a type of attention called
- 3:34self-attention
- 3:36let's start by talking about the first
- 3:37one positional encodings
- 3:39let's say we're trying to translate text
- 3:40from english to french positional
- 3:42encodings is the idea that instead of
- 3:44looking at words sequentially you take
- 3:46each word in your sentence and before
- 3:47you feed it into the neural network you
- 3:49slap a number on it one two three
- 3:51depending on what number the word is in
- 3:53the sentence in other words you store
- 3:55information about word order in the data
- 3:57itself rather than in the structure of
- 3:58the network then as you train the
- 4:00network on lots of text data it learns
- 4:02how to interpret those positional
- 4:04encodings
- 4:05in this way the neural network learns
- 4:07the importance of word order from the
- 4:09data
- 4:10this is a high-level way to understand
- 4:12positional encodings but it's an
- 4:13innovation that really helped make
- 4:15transformers easier to train than rnns
- 4:18the next innovation in this paper is a
- 4:20concept called attention which you'll
- 4:21see used everywhere in machine learning
- 4:23these days in fact the title of the
- 4:25original transformer paper is attention
- 4:27is all you need
- 4:28so
- 4:29the agreement on the european economic
- 4:31area was signed in august 1992. did you
- 4:34know that
- 4:35that's the example sentence given in the
- 4:36original paper and remember the original
- 4:39transformer was designed for translation
- 4:41now imagine trying to translate that
- 4:43sentence to french one bad way to
- 4:45translate text is to try to translate
- 4:46each word one for one but in french some
- 4:49words are flipped like in the french
- 4:51translation european comes before
- 4:52economic
- 4:54plus french is a language that has
- 4:55gendered agreement between words so the
- 4:57word europe bien needs to be in the
- 5:00feminine form to match with la zone
- 5:02the attention mechanism is a neural
- 5:03network structure that allows a text
- 5:05model to look at every single word in
- 5:07the original sentence when making a
- 5:09decision about how to translate a word
- 5:10in the output sentence in fact here's a
- 5:13nice visualization from that paper that
- 5:14shows what words in the input sentence
- 5:16the model is attending to when it makes
- 5:18predictions about a word for the output
- 5:20sentence
- 5:21so when the model outputs the word
- 5:24european it's looking at the input words
- 5:26european and economic you can think of
- 5:29this diagram as a sort of heat map for
- 5:31attention
- 5:32and how does the model know which words
- 5:33it should be attending to
- 5:35it's something that's learned over time
- 5:37from data
- 5:38by seeing thousands of examples of
- 5:39french and english sentence pairs the
- 5:41model learns about gender and word order
- 5:43and plurality and all of that
- 5:44grammatical stuff so we talked about two
- 5:47key transformer innovations positional
- 5:49encoding and attention
- 5:51but actually attention had been invented
- 5:52before this paper the real innovation in
- 5:55transformers was something called
- 5:56self-attention a twist on traditional
- 5:59attention
- 6:00the type of attention we just talked
- 6:01about had to do with aligning words in
- 6:03english and french which is really
- 6:05important for translation but what if
- 6:06you're just trying to understand the
- 6:08underlying meaning in language so that
- 6:10you can build a network that can do any
- 6:12number of language tasks
- 6:14what's incredible about neural networks
- 6:16like transformers is that as they
- 6:18analyze tons of text data they begin to
- 6:20build up this internal representation or
- 6:22understanding of language automatically
- 6:25they may learn for example that the
- 6:27words programmer and software engineer
- 6:29and software developer are all
- 6:30synonymous and they might also naturally
- 6:33learn the rules of grammar and gender
- 6:34and tense and so on the better this
- 6:37internal representation of language the
- 6:38neural network learns the better it will
- 6:40be at any language task
- 6:42and it turns out that attention can be a
- 6:44very effective way to get a neural
- 6:45network to understand language if it's
- 6:47turned on the input text itself
- 6:50let me give you an example take these
- 6:52two sentences server can i have the
- 6:54check
- 6:55versus looks like i just crashed the
- 6:57server the word server here means two
- 6:59very different things and i know that
- 7:01because i'm looking at the context of
- 7:03the surrounding words
- 7:04self-attention allows a neural network
- 7:06to understand a word in the context of
- 7:08the words around it so when a model
- 7:10processes the word server in the first
- 7:12sentence it might be attending to the
- 7:14word check which helps it disambiguate
- 7:16from a human server versus a mental one
- 7:19in the second sentence the model might
- 7:21be attending to the word crashed to
- 7:22determine that this server is a machine
- 7:24self-attention can also help neural
- 7:26networks disambiguate words recognize
- 7:28parts of speech and even identify word
- 7:30tense
- 7:31this in a nutshell is the value of
- 7:32self-attention
- 7:34so to summarize transformers boil down
- 7:36to positional encodings attention and
- 7:39self-attention
- 7:41of course this is a 10 000 foot look at
- 7:43transformers but how are they actually
- 7:45useful one of the most popular
- 7:46transformer based models is called bert
- 7:49which was invented just around the time
- 7:50that i joined google in 2018.
- 7:52bert was trained on a massive text
- 7:54corpus and has become this sort of
- 7:56general pocket knife for nlp that can be
- 7:58adopted to a bunch of different tasks
- 8:01like text summarization question
- 8:02answering classification and finding
- 8:04similar sentences
- 8:06it's used in google search to help
- 8:08understand search queries and it powers
- 8:10a lot of google cloud's nlp tools like
- 8:12google cloud automl natural language
- 8:14burt also proved that you could build
- 8:16very good models on unlabeled data like
- 8:18text scraped from wikipedia or reddit
- 8:21this is called semi-supervised learning
- 8:23and it's a big trend in machine learning
- 8:24right now
- 8:27so if i've sold you on how cool
- 8:28transformers are you might want to start
- 8:30using them in your app no problem
- 8:32tensorflow hub is a great place to grab
- 8:34pre-trained transformer models like
- 8:36burnt you can download them for free in
- 8:38multiple language and drop them straight
- 8:40into your app
- 8:41you can also check out the popular
- 8:42transformers python library built by the
- 8:45company hugging face that's one of the
- 8:46community's favorite ways to train and
- 8:48use transformer models for more
- 8:50transformer tips check out my blog post
- 8:52linked below and thanks for watching
- 8:59[Music]
- 9:10you
About this transcript
This page contains the full transcript of Transformers, explained: Understand the model behind GPT, BERT, and T5 by Google Cloud Tech, generated from the public captions YouTube serves with the video. The transcript has 1,734 words across 292 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.
What you can do with it
Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.
Free YouTube transcript tool
YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.