YouTube2Text

Transformers, explained: Understand the model behind GPT, BERT, and T5 — Transcript

by Google Cloud Tech · 1,734 words · 292 segments · language en · Watch on YouTube

Full transcript

  1. 0:00the neat thing about working in machine
  2. 0:01learning is that every few years
  3. 0:03somebody invents something crazy that
  4. 0:04makes you totally reconsider what's
  5. 0:06possible
  6. 0:07like models that can play go or generate
  7. 0:10hyper-realistic faces
  8. 0:12and today the mind-blowing discovery
  9. 0:13that's rocking everyone's world is a
  10. 0:15type of neural network called a
  11. 0:16transformer
  12. 0:17transformers are models that can
  13. 0:18translate text write poems and op-eds
  14. 0:21and even generate computer code these
  15. 0:23can be used in biology to solve the
  16. 0:24protein folding problem
  17. 0:26transformers are like this magical
  18. 0:28machine learning hammer that seems to
  19. 0:29make every problem into a nail if you've
  20. 0:31heard of the trendy new ml models bert
  21. 0:33or gpt-3 or t5 all of these models are
  22. 0:37based on transformers
  23. 0:38so if you want to stay hip in machine
  24. 0:40learning and especially natural language
  25. 0:42processing you have to know about the
  26. 0:44transformer so in this video i'm going
  27. 0:46to tell you about what transformers are
  28. 0:47how they work and why they've been so
  29. 0:49impactful let's get to it so what is a
  30. 0:52transformer it's a type of neural
  31. 0:54network architecture to recap neural
  32. 0:56networks are a very effective type of
  33. 0:58model for analyzing complicated data
  34. 1:00types like images videos audio and text
  35. 1:02but there are different types of neural
  36. 1:04networks optimized for different types
  37. 1:05of data like if you're analyzing images
  38. 1:07you typically use a convolutional neural
  39. 1:09network which is designed to vaguely
  40. 1:11mimic the way that the human brain
  41. 1:13processes vision and since around 2012
  42. 1:16neural networks have been really good at
  43. 1:17solving vision tasks like identifying
  44. 1:19objects and photos but for a long time
  45. 1:21we didn't have anything comparably good
  46. 1:23for analyzing language whether for
  47. 1:25translation or text summarization or
  48. 1:27text generation
  49. 1:28and this is a problem because language
  50. 1:30is the primary way that humans
  51. 1:31communicate you see until transformers
  52. 1:33came around the way we use deep learning
  53. 1:35to understand text was with a type of
  54. 1:36model called a recurrent neural network
  55. 1:39or an rnn that looks something like this
  56. 1:42let's say you wanted to translate a
  57. 1:44sentence from english to french
  58. 1:46an rnn would take as input an english
  59. 1:48sentence and process the words one at a
  60. 1:50time and then sequentially spit out
  61. 1:52their french counterparts the key word
  62. 1:54here is sequential in language the order
  63. 1:56of words matters and you can't just
  64. 1:58shuffle them around
  65. 1:59for example the sentence jane went
  66. 2:02looking for trouble means something very
  67. 2:04different in the sentence trouble went
  68. 2:05looking for jane
  69. 2:07so any model that's going to deal with
  70. 2:08language has to capture word order and
  71. 2:10recurrent neural networks do this by
  72. 2:12looking at one word at a time
  73. 2:13sequentially but rnns had a lot of
  74. 2:15problems
  75. 2:16first they never really did well at
  76. 2:17handling large sequences of text like
  77. 2:20long paragraphs or essays by the time
  78. 2:22they were analyzing the end of a
  79. 2:23paragraph they'd forget what happened in
  80. 2:24the beginning
  81. 2:25and even worse rnns were pretty hard to
  82. 2:28train because they processed words
  83. 2:30sequentially they couldn't paralyze well
  84. 2:32which means that you couldn't just speed
  85. 2:33them up by throwing lots of gpus at them
  86. 2:36and when you have a model that's slow to
  87. 2:37train you can't train it on all that
  88. 2:38much data this is where the transformer
  89. 2:40changed everything they were a model
  90. 2:42developed in 2017 by researchers at
  91. 2:44google and the university of toronto and
  92. 2:46they were initially designed to do
  93. 2:47translation but unlike recurrent neural
  94. 2:50networks you could really efficiently
  95. 2:51paralyze transformers and that meant
  96. 2:53that with the right hardware you could
  97. 2:54train some really big models
  98. 2:56how big
  99. 2:57really big
  100. 2:59remember gpt3 that model that writes
  101. 3:00poetry and code and has conversations
  102. 3:03that was trained on almost 45 terabytes
  103. 3:05of text data including like almost the
  104. 3:07entire public web
  105. 3:09[Music]
  106. 3:10so if you remember anything about
  107. 3:11transformers let it be this combine a
  108. 3:13model that scales really well with a
  109. 3:15huge data set and the results will
  110. 3:17probably blow your mind so how do these
  111. 3:19things actually work from the diagram of
  112. 3:21the paper it should be pretty clear
  113. 3:24or maybe not actually it's simpler than
  114. 3:26you might think there are three main
  115. 3:28innovations that make this model work so
  116. 3:29well
  117. 3:30positional encodings and attention and
  118. 3:32specifically a type of attention called
  119. 3:34self-attention
  120. 3:36let's start by talking about the first
  121. 3:37one positional encodings
  122. 3:39let's say we're trying to translate text
  123. 3:40from english to french positional
  124. 3:42encodings is the idea that instead of
  125. 3:44looking at words sequentially you take
  126. 3:46each word in your sentence and before
  127. 3:47you feed it into the neural network you
  128. 3:49slap a number on it one two three
  129. 3:51depending on what number the word is in
  130. 3:53the sentence in other words you store
  131. 3:55information about word order in the data
  132. 3:57itself rather than in the structure of
  133. 3:58the network then as you train the
  134. 4:00network on lots of text data it learns
  135. 4:02how to interpret those positional
  136. 4:04encodings
  137. 4:05in this way the neural network learns
  138. 4:07the importance of word order from the
  139. 4:09data
  140. 4:10this is a high-level way to understand
  141. 4:12positional encodings but it's an
  142. 4:13innovation that really helped make
  143. 4:15transformers easier to train than rnns
  144. 4:18the next innovation in this paper is a
  145. 4:20concept called attention which you'll
  146. 4:21see used everywhere in machine learning
  147. 4:23these days in fact the title of the
  148. 4:25original transformer paper is attention
  149. 4:27is all you need
  150. 4:28so
  151. 4:29the agreement on the european economic
  152. 4:31area was signed in august 1992. did you
  153. 4:34know that
  154. 4:35that's the example sentence given in the
  155. 4:36original paper and remember the original
  156. 4:39transformer was designed for translation
  157. 4:41now imagine trying to translate that
  158. 4:43sentence to french one bad way to
  159. 4:45translate text is to try to translate
  160. 4:46each word one for one but in french some
  161. 4:49words are flipped like in the french
  162. 4:51translation european comes before
  163. 4:52economic
  164. 4:54plus french is a language that has
  165. 4:55gendered agreement between words so the
  166. 4:57word europe bien needs to be in the
  167. 5:00feminine form to match with la zone
  168. 5:02the attention mechanism is a neural
  169. 5:03network structure that allows a text
  170. 5:05model to look at every single word in
  171. 5:07the original sentence when making a
  172. 5:09decision about how to translate a word
  173. 5:10in the output sentence in fact here's a
  174. 5:13nice visualization from that paper that
  175. 5:14shows what words in the input sentence
  176. 5:16the model is attending to when it makes
  177. 5:18predictions about a word for the output
  178. 5:20sentence
  179. 5:21so when the model outputs the word
  180. 5:24european it's looking at the input words
  181. 5:26european and economic you can think of
  182. 5:29this diagram as a sort of heat map for
  183. 5:31attention
  184. 5:32and how does the model know which words
  185. 5:33it should be attending to
  186. 5:35it's something that's learned over time
  187. 5:37from data
  188. 5:38by seeing thousands of examples of
  189. 5:39french and english sentence pairs the
  190. 5:41model learns about gender and word order
  191. 5:43and plurality and all of that
  192. 5:44grammatical stuff so we talked about two
  193. 5:47key transformer innovations positional
  194. 5:49encoding and attention
  195. 5:51but actually attention had been invented
  196. 5:52before this paper the real innovation in
  197. 5:55transformers was something called
  198. 5:56self-attention a twist on traditional
  199. 5:59attention
  200. 6:00the type of attention we just talked
  201. 6:01about had to do with aligning words in
  202. 6:03english and french which is really
  203. 6:05important for translation but what if
  204. 6:06you're just trying to understand the
  205. 6:08underlying meaning in language so that
  206. 6:10you can build a network that can do any
  207. 6:12number of language tasks
  208. 6:14what's incredible about neural networks
  209. 6:16like transformers is that as they
  210. 6:18analyze tons of text data they begin to
  211. 6:20build up this internal representation or
  212. 6:22understanding of language automatically
  213. 6:25they may learn for example that the
  214. 6:27words programmer and software engineer
  215. 6:29and software developer are all
  216. 6:30synonymous and they might also naturally
  217. 6:33learn the rules of grammar and gender
  218. 6:34and tense and so on the better this
  219. 6:37internal representation of language the
  220. 6:38neural network learns the better it will
  221. 6:40be at any language task
  222. 6:42and it turns out that attention can be a
  223. 6:44very effective way to get a neural
  224. 6:45network to understand language if it's
  225. 6:47turned on the input text itself
  226. 6:50let me give you an example take these
  227. 6:52two sentences server can i have the
  228. 6:54check
  229. 6:55versus looks like i just crashed the
  230. 6:57server the word server here means two
  231. 6:59very different things and i know that
  232. 7:01because i'm looking at the context of
  233. 7:03the surrounding words
  234. 7:04self-attention allows a neural network
  235. 7:06to understand a word in the context of
  236. 7:08the words around it so when a model
  237. 7:10processes the word server in the first
  238. 7:12sentence it might be attending to the
  239. 7:14word check which helps it disambiguate
  240. 7:16from a human server versus a mental one
  241. 7:19in the second sentence the model might
  242. 7:21be attending to the word crashed to
  243. 7:22determine that this server is a machine
  244. 7:24self-attention can also help neural
  245. 7:26networks disambiguate words recognize
  246. 7:28parts of speech and even identify word
  247. 7:30tense
  248. 7:31this in a nutshell is the value of
  249. 7:32self-attention
  250. 7:34so to summarize transformers boil down
  251. 7:36to positional encodings attention and
  252. 7:39self-attention
  253. 7:41of course this is a 10 000 foot look at
  254. 7:43transformers but how are they actually
  255. 7:45useful one of the most popular
  256. 7:46transformer based models is called bert
  257. 7:49which was invented just around the time
  258. 7:50that i joined google in 2018.
  259. 7:52bert was trained on a massive text
  260. 7:54corpus and has become this sort of
  261. 7:56general pocket knife for nlp that can be
  262. 7:58adopted to a bunch of different tasks
  263. 8:01like text summarization question
  264. 8:02answering classification and finding
  265. 8:04similar sentences
  266. 8:06it's used in google search to help
  267. 8:08understand search queries and it powers
  268. 8:10a lot of google cloud's nlp tools like
  269. 8:12google cloud automl natural language
  270. 8:14burt also proved that you could build
  271. 8:16very good models on unlabeled data like
  272. 8:18text scraped from wikipedia or reddit
  273. 8:21this is called semi-supervised learning
  274. 8:23and it's a big trend in machine learning
  275. 8:24right now
  276. 8:27so if i've sold you on how cool
  277. 8:28transformers are you might want to start
  278. 8:30using them in your app no problem
  279. 8:32tensorflow hub is a great place to grab
  280. 8:34pre-trained transformer models like
  281. 8:36burnt you can download them for free in
  282. 8:38multiple language and drop them straight
  283. 8:40into your app
  284. 8:41you can also check out the popular
  285. 8:42transformers python library built by the
  286. 8:45company hugging face that's one of the
  287. 8:46community's favorite ways to train and
  288. 8:48use transformer models for more
  289. 8:50transformer tips check out my blog post
  290. 8:52linked below and thanks for watching
  291. 8:59[Music]
  292. 9:10you

About this transcript

This page contains the full transcript of Transformers, explained: Understand the model behind GPT, BERT, and T5 by Google Cloud Tech, generated from the public captions YouTube serves with the video. The transcript has 1,734 words across 292 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.