구글의 초대형 TPU 8i Max, 8tx 기습 출시, NVIDIA 대항 자체 칩 개발 가속 | 칩 두 개를 붙인 TPU가 노리는 것 | TPU Sync 활용 — Transcript

Full transcript

1. 0:00Hi, everyone. Hello. This is Patch.
2. 0:02Google has announced a new AI chip. This
3. 0:05time, they put two existing chips into a
4. 0:06single package.
5. 0:08They doubled both the processing power
6. 0:10and the speed of retrieving data from
7. 0:11memory.
8. 0:12But, you know, when an announcement like
9. 0:14this comes out, that's the biggest
10. 0:15question, right? I mean, does this mean
11. 0:17it's better than Nvidia? So, does it
12. 0:19mean we can run AI more cheaply? You're
13. 0:21curious about things like that, right?
14. 0:22Looking at this announcement, I think
15. 0:24there's one more reason for AI companies
16. 0:26to get a quote from Google when adding
17. 0:28servers in the future.
18. 0:29As Google increases chip performance,
19. 0:31they are also introducing software for
20. 0:33developers to actually switch over and
21. 0:35use. So today, I'm going to look at
22. 0:37three things. What Google changed this
23. 0:40time, how it can reduce AI operating
24. 0:42costs, and to what extent it can replace
25. 0:45Nvidia. What do we call Google's AI chip
26. 0:48? Right, we call it a TPU. Like a GPU,
27. 0:50it's a chip that handles the massive
28. 0:52calculations needed for AI. Google uses
29. 0:55TPUs when building and servicing its own
30. 0:57Gemini. They also supply these to
31. 0:59external customers. Now, let's take a
32. 1:01quick look at the process of running AI
33. 1:03here. When we ask an AI a question, the
34. 1:05answer doesn't appear all at once, it
35. 1:07comes out little by little.
36. 1:09While calculating the next content to
37. 1:10use, the AI constantly refers to the
38. 1:12numbers inside the model and the input
39. 1:14received earlier. At this point, it
40. 1:16needs to fetch the necessary data from
41. 1:18memory. The faster the computing chip,
42. 1:20the more important the speed of feeding
43. 1:22it data becomes. Because if the chip is
44. 1:24ready for the next calculation but the
45. 1:26necessary data hasn't arrived, it just
46. 1:28has to wait.
47. 1:29So, when reading long documents or when
48. 1:31many people use AI at the same time,
49. 1:34this burden can increase.
50. 1:36That's why the memory we attach right
51. 1:38next to the AI chip to supply this data
52. 1:40quickly is HBM.
53. 1:41It's the same HBM that often comes up
54. 1:43when talking about Korean memory
55. 1:44companies. But in memory, you just need
56. 1:46to distinguish between two things.
57. 1:48How much capacity it can hold, and how
58. 1:50much it can carry at once, which refers
59. 1:52to bandwidth.
60. 1:54In other words, using a water supply
61. 1:55analogy, it's the difference between the
62. 1:57size of the water tank and the thickness
63. 1:58of the pipe. Even if the water tank is
64. 2:00big, if the pipe is thin, the water
65. 2:02won't gush out, but just trickle, right
66. 2:03? The newly released TPU v5p
67. 2:06significantly increased the speed of
68. 2:08data supply. They put two existing v5e
69. 2:10AI chips into one package to double the
70. 2:13computing power. They also doubled the
71. 2:15HBM bandwidth. They’ve increased the
72. 2:17data retrieval capability in line with
73. 2:19the increased calculation workload. HBM
74. 2:21capacity has increased from the previous
75. 2:24288GB to 384GB. That’s an increase of
76. 2:27about 33%. Computing power and bandwidth
77. 2:29have doubled, while capacity has
78. 2:31increased by about 1.33 times. This
79. 2:32difference shows just how much Google
80. 2:34prioritized data supply speed in this
81. 2:36configuration.
82. 2:38According to Google’s explanation, you
83. 2:39can fit double the amount of computing
84. 2:41silicon into the space occupied by a
85. 2:42previous rack. For data centers, this
86. 2:45creates room to process more AI
87. 2:47workloads in the same footprint. Of
88. 2:49course, how much more is actually
89. 2:51processed depends on the model, power,
90. 2:52and cooling conditions. However, if you
91. 2:54interpret the spec that computing power
92. 2:56has doubled to mean that electricity
93. 2:58costs have been cut in half, you might
94. 2:59end up with some miscalculations.
95. 3:01Everything up to this point was about
96. 3:03calculations needed for the AI to
97. 3:04perform current tasks.
98. 3:05But when running AI, there are moments
99. 3:07where data occupies memory even when
100. 3:09it's not immediately being calculated.
101. 3:11For example, let's say you give an AI a
102. 3:14long report and ask, "Find out why
103. 3:15revenue decreased here."
104. 3:17After receiving the answer, you might
105. 3:19ask, "Now look at the overseas business
106. 3:20in more detail." You could ask something
107. 3:22like that. In that case, the content of
108. 3:24the report read earlier becomes
109. 3:25necessary again. AI can store the
110. 3:27intermediate calculation values of what
111. 3:29it has read to prepare for these
112. 3:31follow-up questions.
113. 3:32We call this "KV cache."
114. 3:35By reusing these calculation values, you
115. 3:37can reduce the need to recalculate the
116. 3:38beginning of the report from scratch.
117. 3:40This might seem trivial when one person
118. 3:42is using it, but if thousands or tens of
119. 3:44thousands of people are having it read
120. 3:45long documents, the calculation values
121. 3:47to be stored will become massive, right
122. 3:49? HBM must hold the data needed to run
123. 3:51the model. It must also store the
124. 3:53calculation values for users currently
125. 3:55receiving answers. So, there isn't
126. 3:56enough room to hold everything,
127. 3:58including data waiting for follow-up
128. 3:59questions. TPU Sync, unveiled by Google
129. 4:02, is software that moves and manages
130. 4:04these calculation values. It sends
131. 4:06content that won't be used for a while
132. 4:08to the server's general memory and
133. 4:09brings it back when needed. This way,
134. 4:11you can secure HBM space while still
135. 4:13being able to reuse previous
136. 4:15calculations. This creates two paths to
137. 4:18cost savings.
138. 4:20You can reduce the number of times you
139. 4:21recalculate the same content and take on
140. 4:23other requests with the secured HBM
141. 4:25space.
142. 4:25Google has made it possible to perform
143. 4:27other calculations even while data is
144. 4:29being moved. Google announced that in
145. 4:32internal tests reusing the beginning of
146. 4:34long inputs, the time to first token has
147. 4:37been reduced by up to 90%. This is a
148. 4:39comparison of results using TPU-Sync
149. 4:41versus not using it on
150. 4:42previous-generation TPUs. These effects
151. 4:44were observed under test conditions
152. 4:45where the same content could be reused
153. 4:47multiple times. What I find notable here
154. 4:49is the actual way AI is being used.
155. 4:51We’ve moved from getting a single answer
156. 4:54and ending the session to repeatedly
157. 4:55modifying and checking the same
158. 4:57materials.
159. 4:58As this happens more, how well we reuse
160. 5:00previously processed content will affect
161. 5:02service costs.
162. 5:04Nvidia also supports KV cache reuse and
163. 5:06off-chip memory utilization through
164. 5:08software like TensorRT-LLM. Both
165. 5:10companies are competing to reduce the
166. 5:12inefficiencies that arise when actually
167. 5:14operating AI. Now, we can start to
168. 5:17compare Google with Nvidia.
169. 5:19What customers want is to get answers of
170. 5:21the same quality at similar speeds while
171. 5:23paying less.
172. 5:24Only by meeting these conditions can we
173. 5:26determine which equipment is more
174. 5:27efficient. There are already test
175. 5:29results worth considering for Google
176. 5:31TPUs. The analysis firm SemiAnalysis
177. 5:33compared the previous-generation
178. 5:35Ironwood TPU with Nvidia hardware. They
179. 5:38ran the same open model in scenarios
180. 5:40involving long-form conversations. The
181. 5:42model used was GLM 5.2. In both cases,
182. 5:45the calculations were performed using
183. 5:48the 8-bit FP8 format. Under those
184. 5:50conditions, the TPU produced more
185. 5:52answers for the same cost compared to
186. 5:55equipment including the B200 and B300.
187. 5:57But the interesting part is when the
188. 5:59settings were changed on the Nvidia side
189. 6:00.
190. 6:01When running the model in the 4-bit FP4
191. 6:04format, Nvidia led in terms of cost
192. 6:06across most speed ranges.
193. 6:08Representing numbers with fewer bits
194. 6:10consumes less memory and reduces the
195. 6:12amount of data to be moved. The test did
196. 6:15show a slight difference in accuracy as
197. 6:17a trade-off.
198. 6:18In any case, the question of which side
199. 6:20is cheaper must always be tied to the
200. 6:21model name and settings.
201. 6:23Results will vary depending on the
202. 6:25required accuracy and how quickly
203. 6:27answers need to be generated for the
204. 6:29customer’s specific model.
205. 6:31There are no official measurements
206. 6:33directly comparing the new Trillium TPU
207. 6:34to Nvidia hardware in the announcement
208. 6:36materials. If it were me, I would test
209. 6:38separately for running AI translation
210. 6:40services versus coding services. For
211. 6:42translation, I’d check how many
212. 6:44sentences need human correction, and for
213. 6:46coding, I’d verify if the generated code
214. 6:48actually works. Because even if the cost
215. 6:50of generating an answer goes down, the
216. 6:52cost of reprocessing incorrect results
217. 6:54could increase.
218. 6:56So, looking at what we've checked so far
219. 6:58, TPUs are an option for comparing costs
220. 7:00against Nvidia under specific models and
221. 7:02operating conditions. From a customer's
222. 7:05perspective, there is now a reason to
223. 7:06actually host their services and verify
224. 7:08the price estimates. So then, once
225. 7:10they've compared the prices, the next
226. 7:12thing a company would ask is: "Will the
227. 7:14programs I'm using now run well?" That
228. 7:16would be the question.
229. 7:18AI companies have been developing models
230. 7:20and operating services on Nvidia GPUs.
231. 7:23Developers have tools they are familiar
232. 7:24with, and there are programs optimized
233. 7:26specifically for that equipment.
234. 7:28If the development team has to spend a
235. 7:30long time moving everything to a TPU,
236. 7:31the total cost could end up being higher
237. 7:33even if they save a little on server
238. 7:34costs.
239. 7:35This is exactly why Google has
240. 7:37strengthened its support for PyTorch.
241. 7:39PyTorch is software that is widely used
242. 7:41for AI development.
243. 7:42Google has widened the path so that
244. 7:44developers can effectively use the
245. 7:46models and tools they were already using
246. 7:47on TPUs as well. In other words, they
247. 7:49made it easy for customers to test parts
248. 7:51of their services.
249. 7:53I believe this aspect is just as
250. 7:55important as the chip's performance when
251. 7:56looking at the potential for Google TPUs
252. 7:58to replace Nvidia.
253. 8:00Because even if a chip is good, if it’s
254. 8:01difficult for developers to actually
255. 8:03test it, they won't be inclined to
256. 8:04purchase it. Only when they can verify
257. 8:06performance and even run operations
258. 8:08using existing tools will the next order
259. 8:10come in. Also, there are sectors where
260. 8:12security is critical, such as government
261. 8:13or financial companies. Those places
262. 8:15have different requirements for adoption
263. 8:17. They need to be able to protect
264. 8:19information even while the AI is reading
265. 8:21and calculating sensitive data. The TPU
266. 8:23v5p that Google announced this time was
267. 8:25aimed at these types of customers. It
268. 8:27combines existing v5p chips with AMD
269. 8:29server CPUs to configure hardware-level
270. 8:32protection for data being processed and
271. 8:34internal model information.
272. 8:36In this sense, Google is fulfilling the
273. 8:38conditions required for customers to buy
274. 8:41TPUs one by one.
275. 8:42They provide memory performance for
276. 8:44companies handling long inputs, and
277. 8:45support for existing tools for
278. 8:46development teams. And for those
279. 8:48handling sensitive data, they offer
280. 8:50security features. The fact that they
281. 8:52build these systems using existing chips
282. 8:53is also a major advantage for Google.
283. 8:56This is because it can significantly
284. 8:57reduce the time it takes to design new
285. 8:59semiconductors from scratch. There is
286. 9:01actually a company that uses various
287. 9:02types of AI chips together. It is
288. 9:04Anthropic, which builds Claude.
289. 9:06The company states that it trains and
290. 9:08services its models using Nvidia GPUs,
291. 9:10Google TPUs, and Amazon Trainium.
292. 9:13This year, they signed a contract with
293. 9:15Google and Broadcom for next-generation
294. 9:17TPU capacity starting in 2027.
295. 9:19They have also secured additional Nvidia
296. 9:21GPU capacity. They seem to be increasing
297. 9:23their required computing power by
298. 9:24utilizing various equipment tailored to
299. 9:26specific tasks. That is why I believe an
300. 9:28alternative to Nvidia might first emerge
301. 9:30through additional customer orders. As
302. 9:32the number of AI service users grows,
303. 9:34the company needs to secure more servers
304. 9:36.
305. 9:37If some tasks can be processed more
306. 9:39cheaply on TPUs, they could allocate a
307. 9:40portion of the new demand to Google.
308. 9:43While existing Nvidia equipment
309. 9:45continues to run, the TPUs will take on
310. 9:47a portion of the increasing workload.
311. 9:49As these choices accumulate, it will
312. 9:51become easier for customers to compare
313. 9:53supply conditions than when they relied
314. 9:55solely on Nvidia.
315. 9:57Following this Google announcement, what
316. 9:59we need to verify next is the cost and
317. 10:01quality when customers actually run
318. 10:03their models on the new system.
319. 10:05The real key is whether the increased
320. 10:07computing power, memory performance, and
321. 10:09improved software from Google actually
322. 10:10lead to lower operating costs for
323. 10:12customers.
324. 10:13And if the results are positive, I think
325. 10:15many more customers will compare them
326. 10:17with Nvidia to get competitive quotes.
327. 10:19That’s all for today.