YouTube2Text

AI内存的技术混战 | 伯恩斯坦研究报告 | AI内存 | HBM | CXL | DRAM | NAND | KV缓存 | 存储器产业 | AI算力底层硬件 | 存储级内存 | 人工智能推理 — Transcript

by 最佳拍档 · 3,817 words · 655 segments · language en · Watch on YouTube

Full transcript

  1. 0:00Hello everyone, this is Best Partner,
  2. 0:02I'm Da Fei. When most people talk about
  3. 0:04AI memory, the first reaction is HBM,
  4. 0:07as if HBM is all there is to AI memory,
  5. 0:09and as long as you stack HBM, all
  6. 0:11problems are solved. But you might not
  7. 0:13know that what really bottlenecks AI
  8. 0:15memory is not the training stage, but
  9. 0:17the decoding KV cache in inference. The
  10. 0:20expansion speed far exceeds most
  11. 0:22people's expectations; new industry
  12. 0:24terms pop up every day, such as HBF,
  13. 0:27ZNAND, CXL, CMX, ZM, HBC, and so on.
  14. 0:30The industry is busy creating both new
  15. 0:32technologies and abbreviations at a
  16. 0:33speed that many practitioners can't
  17. 0:35even keep up with. Recently, four
  18. 0:37analysts from Bernstein jointly
  19. 0:39authored the latest introductory report
  20. 0:41on AI memory. They categorized all
  21. 0:43these new technologies into two
  22. 0:45frameworks: one based on system
  23. 0:47architecture levels, and the other on
  24. 0:49hardware media levels. So, in today's
  25. 0:51video, we will follow this framework to
  26. 0:53clarify the full picture of AI memory.
  27. 0:55Let's start with the basics: different
  28. 0:57stages of AI tasks have completely
  29. 0:59different requirements for memory and
  30. 1:01cannot be generalized. Many people
  31. 1:03start by talking only about HBM, which
  32. 1:04is actually oversimplifying the problem
  33. 1:06. Although training is the most
  34. 1:08compute-intensive stage in the entire
  35. 1:10AI process, memory bandwidth is
  36. 1:11critical. Memory bandwidth directly
  37. 1:13determines the model size you can run
  38. 1:15and the training speed. The core status
  39. 1:17of HBM in training goes without saying,
  40. 1:20but training definitely cannot run on
  41. 1:22HBM alone. Think about it, massive
  42. 1:24amounts of training data must be
  43. 1:25prepared in advance and stored in
  44. 1:27slower, cheaper storage media. During
  45. 1:29the training process, data must be
  46. 1:30gradually scheduled through multiple
  47. 1:32layers of cache before it can finally
  48. 1:34enter the HBM. Moreover, training can
  49. 1:35run for months, requiring frequent
  50. 1:37checkpointing in between to prevent
  51. 1:38soft or hard hardware failures from
  52. 1:40causing a restart. Simply put, training
  53. 1:42requires the entire hierarchy of memory
  54. 1:44to function, from HBM to system RAM,
  55. 1:46local SSDs, and network storage;
  56. 1:48missing any link won't work. Inference,
  57. 1:50on the other hand, is even more
  58. 1:51interesting. It is divided into two
  59. 1:53completely different stages: prefill
  60. 1:55and decoding. The bottlenecks for the
  61. 1:56two are completely different. The
  62. 1:58prefill stage processes the user's
  63. 1:59input prompt to generate the first
  64. 2:01token. This stage is dominated by
  65. 2:02matrix operations with high GPU
  66. 2:04utilization; it's a typical
  67. 2:05compute-bound scenario where data
  68. 2:07movement time is much lower than
  69. 2:08compute time, making memory pressure
  70. 2:10relatively smaller and HBM's role more
  71. 2:11prominent. The core metric for this
  72. 2:13stage is the time to first token, or
  73. 2:15TTFT. In a typical chat scenario, under
  74. 2:170.5 seconds is ideal, one second is
  75. 2:19generally acceptable, and exceeding two
  76. 2:21seconds causes users to lose patience.
  77. 2:23If you are handling heavy tasks like
  78. 2:25processing hundreds of pages of
  79. 2:26documents, the tolerance is higher, and
  80. 2:28with a "thinking" prompt, users can
  81. 2:29still find it acceptable. However, the
  82. 2:31logic for the decoding stage is
  83. 2:33completely different. Tokens are
  84. 2:34generated one by one, which is
  85. 2:35academically called autoregressive
  86. 2:37generation, where each new token
  87. 2:38requires information from all previous
  88. 2:40tokens. To avoid redundant calculations
  89. 2:42, current large models will store
  90. 2:44previous key-value pairs, which is
  91. 2:46known as KV cache. The size of the KV
  92. 2:48cache grows linearly with the number of
  93. 2:50tokens. There is a key difference here:
  94. 2:52model weights are read-only and can be
  95. 2:54shared among all users, but the KV
  96. 2:56cache is unique to each user, so the
  97. 2:57more concurrent users there are, the
  98. 2:59higher the total consumption. Once both
  99. 3:01are combined, the decoding stage
  100. 3:03becomes a typical memory-bound scenario
  101. 3:05. What many people may not realize is
  102. 3:07that in large-scale deployments, the
  103. 3:09memory consumed by the KV cache can be
  104. 3:10even more than the model weights
  105. 3:12themselves. It directly determines how
  106. 3:14many users a server can serve
  107. 3:15simultaneously and how large a context
  108. 3:17window it can support; in other words,
  109. 3:19KV cache efficiency directly impacts an
  110. 3:21AI model's deployment scale and revenue
  111. 3:23ceiling. The core metric for the
  112. 3:25decoding stage is the time per output
  113. 3:27token, or TPOT. For text chat, anything
  114. 3:29under 50 milliseconds is generally
  115. 3:31acceptable. For high-demand scenarios
  116. 3:33like real-time voice or programming
  117. 3:34agents, this must be pushed under 10
  118. 3:36milliseconds. Beyond training and
  119. 3:37inference, the memory requirements for
  120. 3:39Retrieval-Augmented Generation, or RAG,
  121. 3:41follow a different logic. RAG connects
  122. 3:42the model to external databases, using
  123. 3:44specific databases or the latest
  124. 3:45information to supplement the model's
  125. 3:47knowledge, while also reducing
  126. 3:48hallucinations. The entire process is
  127. 3:50divided into database generation and
  128. 3:52searching. The generation stage
  129. 3:53involves processing massive amounts of
  130. 3:55unstructured data like PDFs, code, and
  131. 3:57web pages into a searchable vector
  132. 3:59database. First, you need
  133. 4:00large-capacity storage for raw data;
  134. 4:02SSDs can lower access latency much
  135. 4:04better than HDDs. Generating vectors
  136. 4:06and indexing relies mainly on the
  137. 4:08system, with little actual use of HBM.
  138. 4:10This work is done offline and is
  139. 4:12independent of the number of users or
  140. 4:14tokens; it is a one-time investment.
  141. 4:16During the retrieval phase, user
  142. 4:17queries are first converted into
  143. 4:19vectors. This is processed quickly by
  144. 4:20HBM, but most of the time is spent
  145. 4:22searching for the best matches in the
  146. 4:23vector database. This part relies on
  147. 4:25system-level retrieval. Once the
  148. 4:26results are merged into the user prompt
  149. 4:28, it follows the standard prefill and
  150. 4:30decoding inference process, with memory
  151. 4:31requirements consistent with what we
  152. 4:33discussed earlier. Moving up to agentic
  153. 4:34workflows, memory pressure reaches an
  154. 4:36even higher level. It is not a single
  155. 4:38prompt-to-output cycle, but an
  156. 4:39autonomous multi-step iteration where
  157. 4:41the system breaks down goals, calls
  158. 4:42tools, evaluates results, and performs
  159. 4:44rollbacks for corrections if errors
  160. 4:45occur. This process requires additional
  161. 4:47computation and memory to manage states
  162. 4:49, store intermediate results, and
  163. 4:50handle external tool interactions,
  164. 4:52generating significant demand for
  165. 4:53traditional CPUs and standard memory.
  166. 4:55More importantly, the output of one
  167. 4:57agent becomes the input for the next.
  168. 4:59When multiple agents are running, the
  169. 5:00requirements for prefill, decoding, and
  170. 5:02KV cache stack up, making memory
  171. 5:04pressure much greater than in
  172. 5:05single-round inference. After covering
  173. 5:07the differences in workload
  174. 5:08requirements, let's look at how memory
  175. 5:10is layered from a system architecture
  176. 5:12perspective. We have categorized memory
  177. 5:14in AI systems into a complete hierarchy
  178. 5:16from G0 to G4 based on architectural
  179. 5:17location and function. This
  180. 5:19classification is not an official
  181. 5:21standard; it integrates NVIDIA's SIDM
  182. 5:23definitions while incorporating new
  183. 5:25technologies like HBM, CXL, Storage,
  184. 5:27and Next into a comprehensive framework
  185. 5:29. After understanding this, you will
  186. 5:31see where each new technology fits in
  187. 5:33the system and what issues it solves.
  188. 5:35G0 memory is typically SRAM, featuring
  189. 5:37extremely low latency and very small
  190. 5:39capacity, used for immediate
  191. 5:41computational caching and produced on
  192. 5:43the same logic die as the accelerator.
  193. 5:46G1 is HBM, or High Bandwidth Memory,
  194. 5:48which uses CoWoS advanced packaging to
  195. 5:51stack DRAM chips together with the
  196. 5:53accelerator, offering high bandwidth
  197. 5:55and low latency. Model weights and
  198. 5:57frequently accessed "hot" data are
  199. 5:59stored at this level. G2 is system
  200. 6:01memory, the common DDR DRAM, which has
  201. 6:04larger capacity but higher latency than
  202. 6:06HBM and is typically located next to
  203. 6:08the CPU. Both NVIDIA and AMD GPUs can
  204. 6:11bypass the CPU to access this memory
  205. 6:13directly. G 2.5 is Compute Express Link
  206. 6:15(CXL) memory; this technology was
  207. 6:17initially driven by Intel and has now
  208. 6:19gained full industry support. Its core
  209. 6:21is to integrate physically separated
  210. 6:23memory into a logical shared pool to
  211. 6:25avoid resource waste. Through CXL, the
  212. 6:27system can access accelerator HBM;
  213. 6:29accelerators can also access
  214. 6:30CPU-managed system memory, and even
  215. 6:32PCIe-based memory expanders can be
  216. 6:34incorporated into the shared pool.
  217. 6:36Later, CXL expanded support for NAND.
  218. 6:38The prerequisite is that NAND products
  219. 6:40must be optimized to perform closer to
  220. 6:42DRAM characteristics, usually by
  221. 6:44pairing them with DRAM or SRAM as a
  222. 6:45cache to mask NAND's high latency.
  223. 6:47Samsung's CMM-H is a CXL module that
  224. 6:50blends DRAM and NAND. Next down is G
  225. 6:532.6, called StorageNext memory. This is
  226. 6:55an ecosystem project led by NVIDIA, and
  227. 6:57almost all mainstream storage vendors
  228. 6:59have participated. It has two core
  229. 7:01directions: one is shifting memory
  230. 7:03management from being CPU-centric to
  231. 7:05GPU-centric; the second is making NAND
  232. 7:07act more like DRAM, enabling it to
  233. 7:08enjoy NAND's low cost and high capacity
  234. 7:10while achieving near-DRAM latency and
  235. 7:12fine-grained data access. It is
  236. 7:14positioned between DDR DRAM and local
  237. 7:16SSDs, with products mostly being
  238. 7:18hybrids of DRAM and NAND. Many utilize
  239. 7:20Single-Level Cell (SLC) chips; Kioxia's
  240. 7:23FL6 series SSD is representative of
  241. 7:25this type of product, focusing on
  242. 7:27ultra-high IOPS, or input/output
  243. 7:29operations per second. Third is the
  244. 7:31local SSD using PCIe hardware standards
  245. 7:33and NVMe protocol standards, located at
  246. 7:35the edge of the motherboard within a
  247. 7:36single compute node, also known as a
  248. 7:38computational SSD. This layer extends
  249. 7:40capacity beyond DRAM, but cannot
  250. 7:41communicate across nodes; it is usually
  251. 7:43managed by the CPU, with some
  252. 7:44accelerators capable of direct access.
  253. 7:46G 3.5 is CMX memory and the STX
  254. 7:48architecture. CMX stands for Context
  255. 7:51Memory, previously called SMS (Serving
  256. 7:53Memory Storage), a tier added by NVIDIA
  257. 7:56specifically for KV caching. SSDs
  258. 7:58compliant with the CMX standard are
  259. 8:00deployed in data nodes and can be
  260. 8:02accessed by multiple compute nodes
  261. 8:04across the entire cluster, suitable for
  262. 8:06shared dynamic content in multi-agent
  263. 8:08scenarios. Compute nodes and data nodes
  264. 8:10can be connected via Ethernet, with
  265. 8:11forwarding handled by Spectrum-X switch
  266. 8:13chips in between, and both sides
  267. 8:15equipped with BlueField-4 DPUs. GPUs
  268. 8:17can access storage directly without
  269. 8:19requiring CPU intervention. STX is a
  270. 8:21reference architecture launched by
  271. 8:22NVIDIA to integrate solutions from
  272. 8:24various storage and ODM vendors,
  273. 8:26ensuring compatibility between products
  274. 8:27. The bottom layer is G4, shared
  275. 8:30storage, which can be SSDs or HDDs,
  276. 8:32connected via PCIe or network to store
  277. 8:34long-term, non-critical path data. As
  278. 8:37long as the network and software
  279. 8:38support it, GPUs can also access this
  280. 8:40layer of storage directly. With the
  281. 8:42architectural layers covered, let’s
  282. 8:43now look at storage innovation from the
  283. 8:45perspective of hardware media. This is
  284. 8:46another dimension of classification,
  285. 8:48based on the storage unit structure and
  286. 8:49operating principle, overlaid with
  287. 8:51various packaging and modular
  288. 8:52technologies. By combining these two
  289. 8:54dimensions, the industry can tailor
  290. 8:55solutions that meet different
  291. 8:57requirements. SRAM is the fastest
  292. 8:58within the entire hardware hierarchy.
  293. 9:00Its unit structure consists of six
  294. 9:02transistors, which occupies more
  295. 9:03silicon area than the 1T1C structure of
  296. 9:05DRAM—that is, one transistor plus one
  297. 9:07capacitor. Both are volatile memories;
  298. 9:09once the power is cut off, the data is
  299. 9:11lost. DRAM requires constant refreshing
  300. 9:12of its capacitors, while SRAM does not
  301. 9:14require refresh circuits. Most SRAM
  302. 9:15today is embedded inside logic chips;
  303. 9:17standalone packaged versions only exist
  304. 9:19in niche scenarios, corresponding to
  305. 9:20memory levels in the architecture used
  306. 9:22for various cache tiers. L1 cache is
  307. 9:24closest to the computing unit with the
  308. 9:26smallest capacity, while L2 and L3
  309. 9:27caches have larger capacities but
  310. 9:29slightly higher latency. AI has also
  311. 9:31spurred new uses for SRAM. For example,
  312. 9:33the LPU (Language Processing Unit) from
  313. 9:36Groq—the current version is the LPU
  314. 9:3820—uses on-chip SRAM to satisfy the
  315. 9:40most latency-sensitive parts of
  316. 9:42inference. The next-generation LPU40 is
  317. 9:44rumored to use hybrid bonding
  318. 9:45technology to stack SRAM chips onto
  319. 9:47logic processor chips, resulting in
  320. 9:49only a slight increase in latency but
  321. 9:50allowing for much larger capacity and
  322. 9:52better cost control. This usage roughly
  323. 9:54corresponds to the memory hierarchy in
  324. 9:56architecture. There is also embedded
  325. 9:58DRAM, which sits inside logic chips to
  326. 10:00compete with embedded SRAM for
  327. 10:01large-capacity cache scenarios, such as
  328. 10:03L4 cache. IBM’s POWER processors and
  329. 10:05Sony’s PlayStation use it, but the
  330. 10:07overall market share is currently small
  331. 10:09. DRAM is the absolute powerhouse of
  332. 10:11revenue and profit for the entire
  333. 10:12storage industry, larger in scale than
  334. 10:14NAND. HBM is essentially DRAM as well,
  335. 10:16utilizing the same wafer capacity as
  336. 10:18traditional DRAM. In CXL memory
  337. 10:19expansion, which corresponds to the G1
  338. 10:21and G2 tiers of the architecture, many
  339. 10:23NAND products are used, often paired
  340. 10:24with DRAM as a cache to prefetch data
  341. 10:26and mask NAND's high latency, though
  342. 10:28the volume is not large. The evolution
  343. 10:30of DRAM technology still relies mainly
  344. 10:31on Moore’s Law, which means EUV
  345. 10:33lithography is essential. Manufacturers
  346. 10:35without EUV will gradually lose
  347. 10:37competitiveness in the DRAM field in
  348. 10:38the long run. Specifically in China,
  349. 10:40CXMT, as a major domestic DRAM
  350. 10:42manufacturer, is developing 4 F² DRAM
  351. 10:44to compete with traditional 6 F² 1T1C
  352. 10:47DRAM units. The 4 F² design shrinks
  353. 10:49the unit area by moving peripheral
  354. 10:51circuits from beside the capacitor to
  355. 10:53underneath it, or even to another wafer
  356. 10:55. However, the improvement brought by
  357. 10:574F squared is a one-time thing, much
  358. 10:58like the FAT in logic chips. After that
  359. 11:00, further development will still rely
  360. 11:02on lateral scaling. Key players are
  361. 11:04also researching 3D DRAM, similar to
  362. 11:06the 3D NAND approach, by vertically
  363. 11:07stacking memory cells. The average cell
  364. 11:09area can be smaller than 4F squared,
  365. 11:11with potential for continuous scaling,
  366. 11:13but the technical difficulty of 3D DRAM
  367. 11:14is much higher, and it has a long way
  368. 11:16to go before commercialization.
  369. 11:18Manufacturers are innovating in
  370. 11:19different directions around HBM. There
  371. 11:21are many technical paths here, so let's
  372. 11:23go through them one by one. Samsung’s
  373. 11:25ZHBM aims to push packaging from 2.5 D
  374. 11:27to 3D. Current HBM stacks sit next to
  375. 11:29the XPU and connect via an interposer.
  376. 11:31ZHBM wants to place the HBM stack
  377. 11:33directly on top of the XPU to further
  378. 11:35shorten communication distances and
  379. 11:36increase bandwidth. However, the
  380. 11:38problem is also obvious: the underlying
  381. 11:40XPU generates a lot of heat, and it is
  382. 11:41a major question whether the DRAM chip
  383. 11:43on top can work stably at high
  384. 11:44temperatures. Samsung intends to use
  385. 11:46wafer-to-wafer hybrid bonding so there
  386. 11:48are no gaps between the chips, which
  387. 11:49should improve heat dissipation. But it
  388. 11:51is hard to say right now if the yield
  389. 11:53and cost of hybrid bonding can support
  390. 11:54large-scale mass production. NVHBM is a
  391. 11:57customized HBM led by NVIDIA. Its base
  392. 11:59die is designed by NVIDIA and produced
  393. 12:02by TSMC. Moving the memory controller
  394. 12:04from the XPU to the HBM base die can
  395. 12:05reduce the XPU's chip size, lower costs
  396. 12:07, or free up space. This allows for
  397. 12:09more compute units while also
  398. 12:10increasing bandwidth and reducing power
  399. 12:12consumption. NVIDIA is pushing NVHBM
  400. 12:14toward openness. On one hand, it
  401. 12:16standardizes the implementation,
  402. 12:17allowing multiple memory manufacturers
  403. 12:18to......participate in the supply. On
  404. 12:20the other hand, through NVLink Fusion,
  405. 12:22other manufacturers 'accelerators can
  406. 12:23also use it; Amazon's Annapurna Labs
  407. 12:25has already announced a partnership.
  408. 12:27Compared to standard HBM, NVHBM will
  409. 12:28compress the value space for memory
  410. 12:30manufacturers. The base die originally
  411. 12:32designed by memory makers is now
  412. 12:33designed by NVIDIA and manufactured by
  413. 12:35a foundry, so the differentiation space
  414. 12:37for memory makers will be constrained.
  415. 12:39Intel also has two DRAM-related
  416. 12:41technical paths: XBM and FeRAM. XBM
  417. 12:43stands for Cross-point memory.
  418. 12:45According to its patents, it uses
  419. 12:46back-end-of-line processes,
  420. 12:48specifically back-end lithography and
  421. 12:49thin-film transistors, to build DRAM
  422. 12:51cells. Traditional DRAM cells are made
  423. 12:53in the front-end process, with the
  424. 12:54back-end only used for metal
  425. 12:55interconnects. XBM places memory cells
  426. 12:57between the back-end metal layers, so
  427. 12:59DRAM cells and logic circuits can be
  428. 13:01integrated on the same chip, making it
  429. 13:03a potential candidate for in-memory
  430. 13:05computing. There aren't many publicly
  431. 13:06available details yet, so it is hard to
  432. 13:08judge the mass production prospects of
  433. 13:09this technology or who is best suited
  434. 13:11to produce it. The full name of-Z is
  435. 13:12Z-angle memory, an innovation at the
  436. 13:14packaging level that rotates stacked
  437. 13:15DRAM chips by 90 degrees to arrange
  438. 13:17them horizontally, with the vertical
  439. 13:19gaps between chips improving thermal
  440. 13:20efficiency. Intel's partner, SoftBank's
  441. 13:22subsidiary SMOR, stated that they still
  442. 13:24source DRAM chips from memory
  443. 13:25manufacturers, focusing mainly on
  444. 13:27unique packaging and substrate design.
  445. 13:29The physical connection between the
  446. 13:30chips and the substrate after rotation
  447. 13:32is highly difficult; achieving data
  448. 13:34transmission is the core challenge,
  449. 13:36with practical use expected by 2029.
  450. 13:37Qualcomm's High Bandwidth Compute (HBC)
  451. 13:39takes a cost-performance approach,
  452. 13:41forgoing CoWoS advanced packaging for
  453. 13:42traditional packaging; while
  454. 13:44performance is somewhat compromised, it
  455. 13:45saves the high costs of interposers and
  456. 13:472.5 D organic substrates, while using
  457. 13:49LPDDR to reduce power consumption. The
  458. 13:51first generation is expected to ship in
  459. 13:532027, with the second generation
  460. 13:54released in 2028. Beyond chips and
  461. 13:56packaging, DRAM module form factors are
  462. 13:58also innovating, with MRDIMM, CAMM2,
  463. 14:00and LPCAMM2 serving as representatives
  464. 14:01of this trend. MRDIMM shares the same
  465. 14:03physical dimensions as traditional
  466. 14:05server RDIMMs, simply adding more
  467. 14:06interface chips to the module to
  468. 14:08increase bandwidth. CAMM2 features more
  469. 14:10significant changes, with both the
  470. 14:11module's shape and dimensions differing
  471. 14:13from traditional standards. It supports
  472. 14:15LPDDR chips, doubles the data bus width
  473. 14:17from 64 to 128 bits, and has a lower
  474. 14:19profile, which reduces power
  475. 14:20consumption, boosts single-module
  476. 14:22bandwidth, and improves airflow and
  477. 14:24cooling inside servers. Adding a side
  478. 14:26note: the first-generation CAMM was an
  479. 14:28Nvidia-proprietary experimental version
  480. 14:30, while CAMM2 has been officially
  481. 14:31released by JEDEC as an industry
  482. 14:33standard; it is first being adopted by
  483. 14:35Intel's CPUs, and AMD's CPUs will
  484. 14:37follow with support. LPCAMM2 is the
  485. 14:39corresponding version for AI laptops,
  486. 14:40also standardized by JEDEC; the early
  487. 14:42LPCAMM prototype was led by Samsung,
  488. 14:44and these modules all correspond to the
  489. 14:46second level of the architecture,
  490. 14:47alongside Processing-in-Memory (PIM),
  491. 14:49sometimes called Computing-in-Memory (
  492. 14:51CIM). The concept is to integrate logic
  493. 14:53circuits and memory circuits on the
  494. 14:54same chip, performing calculations
  495. 14:56directly within the memory to solve the
  496. 14:58data bottleneck at its source. This
  497. 15:00approach is the complete opposite of
  498. 15:01the classic von Neumann architecture.
  499. 15:03The von Neumann system separates
  500. 15:05computing and storage, whereas PIM
  501. 15:07fuses the two together. Storage
  502. 15:08manufacturers have researched this for
  503. 15:10many years and proposed various
  504. 15:12implementation methods; for instance,
  505. 15:13Samsung added programmable compute
  506. 15:15units to HBM. However, actual adoption
  507. 15:17is very limited because it requires a
  508. 15:18complete redesign of the processor,
  509. 15:20software, and network systems to fully
  510. 15:22realize its potential. Furthermore, the
  511. 15:23current industry chain has established
  512. 15:25a structure of separate logic and
  513. 15:27memory manufacturing, each with massive
  514. 15:28scale and high efficiency, making the
  515. 15:30cost of a full-chain overhaul too high.
  516. 15:32Next is Storage Class Memory (SCM). It
  517. 15:34does not refer to a specific technology
  518. 15:36, but is a collective term for all
  519. 15:37technologies with performance between
  520. 15:39DRAM and NAND, roughly corresponding to
  521. 15:40the range from the bottom of the G2
  522. 15:42tier to the high end of the G3 tier in
  523. 15:43the architecture. One category includes
  524. 15:46emerging memory technologies with
  525. 15:48completely different principles, such
  526. 15:50as MRAM, PCM, and RRAM; the previously
  527. 15:53launched 3DXP belongs to PCM technology
  528. 15:55. The storage principles of these
  529. 15:57technologies are fundamentally
  530. 15:58different from DRAM and NAND, though
  531. 16:00their current mass production scale is
  532. 16:01not large. Another category is enhanced
  533. 16:04NAND, such as ZNS and XL-Flash. By
  534. 16:06pairing with DRAM or SRAM caches, or
  535. 16:08using SLC/MLC particles, the latency of
  536. 16:10NAND is brought closer to DRAM levels.
  537. 16:13Here, I will briefly explain the
  538. 16:14differences between NAND particles. SLC
  539. 16:16is Single-Level Cell, storing one bit
  540. 16:18per cell; it is the fastest with the
  541. 16:20longest lifespan, but has the smallest
  542. 16:21capacity and highest cost. MLC is
  543. 16:23Multi-Level Cell, storing two bits per
  544. 16:25cell, offering a balanced performance.
  545. 16:27TLC is Triple-Level Cell, storing three
  546. 16:29bits per cell, currently the mainstream
  547. 16:31, offering the best balance between
  548. 16:32capacity and cost. QLC is Quad-Level
  549. 16:34Cell, storing four bits per cell,
  550. 16:36providing the largest capacity and
  551. 16:37lowest cost, but with inferior speed
  552. 16:39and lifespan. Kioxia's XL-Flash uses
  553. 16:41SLC and MLC particles, and its FL6
  554. 16:43series SSD can currently achieve
  555. 16:45microsecond-level latency. The goal for
  556. 16:48next year is to reach the nanosecond
  557. 16:49level. Samsung's Z-NAND is made into
  558. 16:52Z-SSD products, focusing on ultra-low
  559. 16:54latency. Other manufacturers use pSLC
  560. 16:56mode, which treats TLC particles like
  561. 16:58SLC, sacrificing capacity for higher
  562. 17:00performance and lifespan. Since TLC is
  563. 17:03the most mass-produced and
  564. 17:04cost-effective process, using pSLC mode
  565. 17:06allows for enjoying economies of scale
  566. 17:08while obtaining performance close to
  567. 17:11SLC. Micron's XTR SSD is one such
  568. 17:13solution. Kioxia itself has a very
  569. 17:15clear product segmentation, with
  570. 17:17different NAND technologies
  571. 17:18corresponding to different AI inference
  572. 17:20scenarios: XL-Flash corresponds to the
  573. 17:22Storage-Next G 2.6 tier, TLC particles
  574. 17:24correspond to the CMx G 3.5 tier, and
  575. 17:26QLC particles correspond to G4-tier
  576. 17:28high-capacity storage. In the NAND
  577. 17:30field, there are two other noteworthy
  578. 17:31directions. One is High Bandwidth Flash
  579. 17:34(HBF), led by SanDisk and SK Hynix,
  580. 17:35aiming to use NAND to achieve bandwidth
  581. 17:38close to HBM while offering lower costs
  582. 17:40and higher capacity. If placed within
  583. 17:42the architectural hierarchy, it can be
  584. 17:43considered a Tier 1.5 level. NAND is
  585. 17:45inherently one to two tiers below DRAM,
  586. 17:47so placing it directly alongside HBM as
  587. 17:49a supplement involves a significant
  588. 17:50technical leap and a very high barrier
  589. 17:52to entry. The other is Z-NAND, a
  590. 17:54technology Samsung introduced for edge
  591. 17:56AI. It uses TLC particles to ensure
  592. 17:57large capacity, and the packaging is
  593. 17:59similar to HBM: first stacking NAND
  594. 18:01chips, then using advanced packaging to
  595. 18:03connect them tightly with the processor
  596. 18:05. The bottom two levels of the hardware
  597. 18:06hierarchy are HDDs and magnetic tape.
  598. 18:08HDDs are suitable for cold data storage
  599. 18:09—that is, data with large capacity,
  600. 18:11low cost, and low access frequency—
  601. 18:12corresponding to the bottom layer of
  602. 18:14the architecture. With the massive
  603. 18:15scale of KV caches now, non-critical
  604. 18:17data is gradually settling downwards,
  605. 18:18and demand for HDDs is also growing.
  606. 18:20Magnetic tape was originally used only
  607. 18:21for data archiving, sitting at the very
  608. 18:23bottom of the hardware hierarchy. But
  609. 18:24with the global explosion in data
  610. 18:26storage demand, the production capacity
  611. 18:28for both NAND and HDDs cannot keep up,
  612. 18:29so even the demand for tape has seen an
  613. 18:31increase. Finally, in this report,
  614. 18:33Bernstein provided ratings and target
  615. 18:35prices for major global storage
  616. 18:36companies. To clarify, in Bernstein's
  617. 18:38rating system, "Outperform" is defined
  618. 18:40as an expectation to beat the
  619. 18:41corresponding market index by more than
  620. 18:4315 percentage points over the next 12
  621. 18:45months. "Underperform" means lagging
  622. 18:47behind the index by more than 15
  623. 18:49percentage points. Specifically,
  624. 18:51Samsung Electronics is rated "
  625. 18:52Outperform" with a target price of
  626. 18:5440,000 KRW, SK Hynix is also "
  627. 18:56Outperform" with a target of 330,000
  628. 18:58KRW, and Micron is "Outperform" with a
  629. 19:00target of $ 130. Kioxia is rated "
  630. 19:02Underperform" with a target of 4,000
  631. 19:04JPY; SanDisk is "Outperform" with a
  632. 19:06target of $ 300; Seagate is "Outperform
  633. 19:08" with a target of $ 135; Western
  634. 19:10Digital is "Outperform" with a target
  635. 19:12of $ 77. A reminder here: the above are
  636. 19:14all Bernstein's forecast data, intended
  637. 19:16for industry research reference only,
  638. 19:17and do not constitute any investment
  639. 19:19advice. Overall, the technical battle
  640. 19:21in the NAND memory field is still
  641. 19:22ongoing, and new technical paths
  642. 19:24continue to emerge. From an
  643. 19:25architectural perspective, the entire
  644. 19:27storage system is shifting from
  645. 19:29CPU-centric to GPU-centric, with every
  646. 19:31layer being redefined. Looking at the
  647. 19:33hardware media, the boundaries between
  648. 19:34DRAM, NAND, and SRAM are becoming
  649. 19:36blurred, as they are all penetrating
  650. 19:38into the middle ground. For industry
  651. 19:43insiders, this is perhaps the most
  652. 19:45exciting time for the storage sector in
  653. 19:48the last 20 years; it is still too
  654. 19:50early to tell which path will
  655. 19:51ultimately emerge as the winner.

About this transcript

This page contains the full transcript of AI内存的技术混战 | 伯恩斯坦研究报告 | AI内存 | HBM | CXL | DRAM | NAND | KV缓存 | 存储器产业 | AI算力底层硬件 | 存储级内存 | 人工智能推理 by 最佳拍档, generated from the public captions YouTube serves with the video. The transcript has 3,817 words across 655 segments, with the original timestamps preserved so you can click any line to jump to that moment in the embedded player.

What you can do with it

Use the transcript to take notes, quote the speaker, build a study guide, generate a summary with ChatGPT or Claude via the YouTube Summary tool, or export it as a timed subtitle file with YouTube to SRT. You can also re-open it in the transcriber to choose a caption track available on the video.

Free YouTube transcript tool

YouTube2Text is a free YouTube transcript generator — no signup, no daily limit. Paste any YouTube link and get the full transcript instantly, with timestamps, click-to-jump, caption language selection, AI prompts for ChatGPT, Claude, and Gemini, and exports to TXT, SRT, VTT, or Markdown.