From GPU-Centric to Memory-Centric: A Shift in the AI Chip Paradigm
===================================================================

The bottleneck in AI systems is shifting from computational performance alone to the ability to store and move data. As HBM, NAND-based HBF, custom base dies, 3D packaging, and optical interconnects converge, memory-centric computing that co-designs accelerators and memory is gaining traction.

- Memory-centric computing is not about eliminating GPUs, but about co-designing processors and memory hierarchies to reduce the cost of moving data.
- Long contexts and more concurrent requests increase KV cache capacity requirements, making total memory capacity and memory hierarchy technologies as important as HBM bandwidth.
- HBF leverages the high density of NAND flash, but latency, write endurance, and error management issues make it more likely to complement HBM than replace it outright.
- Vertically stacking memory above or below a GPU shortens interconnect distances but creates new constraints involving thermal density, yield, power delivery, and cooling.
- Custom HBM expands memory companies' participation in design, but it does not eliminate memory price volatility or supply cycles themselves.

The competitive criteria for AI semiconductors are expanding beyond simple computation counts to how much data can be stored and moved using little power. In large-scale AI inference, GPUs may sit idle not because they cannot perform computations, but because they do not receive the required model weights and KV cache in time.
This change is often described as a “shift from GPU-centric to memory-centric computing.” However, this does not mean that GPUs are disappearing or have stopped advancing. More precisely, it is a move toward jointly designing GPUs, HBM, external memory, storage, networking, and software as a single data movement system.

What Is Memory-Centric Computing?
Memory-centric computing is an approach that takes memory location, bandwidth, capacity, and computing capabilities as the starting point of system design to reduce the cost of repeatedly moving data to processors.
The arithmetic throughput of AI chips has increased rapidly, but retrieving data from outside the chip remains relatively slow and consumes significant power. As a result, the following technologies are becoming important together.

HBM placed close to a GPU or AI accelerator
3D packaging that vertically connects memory and logic
Customized HBM that incorporates control or limited computing functions into the base die
Memory expansion and pooling using CXL
The HBF concept, which connects NAND flash through a wide interface
Optical interconnects intended to reduce transmission power between chips and servers
PIM and near-memory computing, which process data close to memory

In other words, memory-centric computing does not refer to a single memory product. It is a system principle that combines multiple tiers, from fast memory to high-capacity storage, to minimize data movement.

Why AI Inference Intensifies the Memory Bottleneck

Model Weights and the KV Cache Create Different Burdens
The memory required for AI inference can broadly be divided into model weights, activations generated during execution, and the KV cache. Model weights occupy a fixed amount of space even when there are no requests, while the KV cache grows according to input length, output length, and the number of concurrent requests.
To avoid recomputing previous tokens from scratch each time, transformer models store attention key and value information for each layer in the KV cache. In simplified terms, the KV cache size for a single request is proportional to the following factors.
The exact KV cache size must be verified from the model architecture and implementation specifications.
The KV cache stores keys and values separately. Therefore, as context grows longer or more users are processed simultaneously, the KV cache grows rapidly. Not every model uses millions of tokens, nor does cache size uniformly increase by hundreds of times, but it is clear that long-context inference and agentic tasks increase memory pressure.

Hallucinations and Memory Capacity Are Not the Same Problem
Longer context and retrieval-augmented generation can provide models with more supporting evidence. However, increasing memory alone does not eliminate hallucinations. The quality of search results, prompt design, the model’s reasoning capabilities, source verification, and evaluation frameworks are also necessary.

Software Can Reduce Physical Memory Demand
The KV cache bottleneck is not solved solely by adding hardware.

MQA and GQA reduce the number of KV heads that must be stored.
Quantization reduces the number of bytes used by weights and caches.
PagedAttention manages caches in pages, mitigating fragmentation and waste.
Continuous batching efficiently groups multiple requests.
Prefix caching reuses repeated system prompts or shared context.
Cache eviction and tiered offloading move less important information to slower memory.

Therefore, the actual amount of memory investment in a data center depends not only on model size but also on the efficiency of the inference engine.

The Memory Hierarchy of AI Data Centers
It is difficult for a single type of memory to satisfy speed, capacity, cost, and power efficiency requirements all at once. AI data centers are likely to evolve into the following hierarchy.

Tier
Representative Technologies
Strengths
Main Limitations
Expected Role

On-chip
SRAM, registers
Shortest latency
Very limited capacity and high area cost
Immediately needed data and intermediate computation results

Adjacent to accelerator
HBM
High bandwidth and relatively short latency
Capacity, packaging cost, and thermal constraints
Active weights and frequently used KV cache

Server memory
DDR, CXL-connected memory
Greater expandable capacity than HBM
Farther from accelerators and lower bandwidth
Cache expansion, model tiering, and memory pools

High-bandwidth flash
HBF concept
High NAND-based density and non-volatility
Read latency, write endurance, and control complexity
Less frequently used weights and KV cache tiers

Storage
NVMe SSD
High capacity and low cost per bit
Longer latency than memory
Checkpoints, datasets, and cold data

The key is not for either HBM or HBF to win, but to deploy multiple tiers according to how frequently data is used.

The Next-Generation Memory Competition Revealed by FMS 2026
FMS, held in Santa Clara, United States, in August 2026, addressed post-HBM AI memory and storage architectures as major agenda items. Discussions centered on taller stacks, high-capacity NAND-based tiers, customized memory, and data center connectivity technologies.
Samsung Electronics’ GHBM and the HBF discussed by SK hynix with Google and SanDisk were introduced as examples illustrating this direction. However, it is difficult to conclude that names such as GHBM, HBF, and THBM are all universal generation names finalized under identical JEDEC specifications. Technologies at the corporate announcement stage, joint development concepts, and standardized commercial products must be distinguished from one another.
Moreover, a higher generation name does not necessarily make actual AI services faster. Effective bandwidth, memory capacity, latency, power, cooling, package yield, and software support must all be compared together.

HBM’s Strengths and Capacity Limitations
HBM vertically stacks multiple DRAM dies and connects them using through-silicon vias and a wide interface. Its key advantage is that it can supply large amounts of data in parallel close to the accelerator, unlike conventional board-level memory.
However, HBM capacity cannot be increased indefinitely.

As the number of stacked layers increases, manufacturing yield and testing become more difficult.
The package area required to place the GPU and HBM together increases.
Heat generated by the accelerator and memory must be removed from a confined space.
Power delivery networks and interposer wiring also become more complex.
Using expensive HBM to store even infrequently accessed data reduces economic efficiency.

For this reason, tiering is becoming important, with HBM handling the most frequently used data and the rest being moved to CXL memory, flash, or SSDs.

Can HBF Replace HBM?
HBF stands for High Bandwidth Flash and is a concept that parallelizes and stacks NAND flash to create a high-bandwidth memory tier closer to accelerators than conventional SSDs. Because NAND offers higher density and non-volatility than DRAM, it has the potential to store more data in the same physical space.
However, NAND has different characteristics from DRAM.

It has longer read latency.
Data must be erased before it can be overwritten.
Endurance must be managed according to the number of writes.
Bad blocks, error correction, and wear leveling must be handled.
It is better suited to large sequential accesses than small, irregular accesses.

Therefore, even if HBF is commercialized, it is more likely to function as a high-capacity tier between HBM and SSDs than to directly replace HBM. Read-heavy model weights, infrequently reused caches, checkpoints, and retrieval data could be candidates. Its actual suitability will depend on interface standards, latency, endurance, controllers, and inference software support.

The 3D Packaging Challenges Raised by GHBM and THBM
Placing accelerators and HBM side by side within a package imposes limits on connection width and package area. To overcome these limitations, concepts have emerged that vertically stack logic and memory to shorten wiring distances.

A Structure That Places Memory Above the GPU
The concept introduced as GHBM or Z-axis HBM aims to reduce data movement distances by vertically integrating the GPU and HBM. Numerous short vertical connections are expected to provide high bandwidth and low I/O power.
The problem is heat. When memory is placed above the GPU, heat generated by the GPU may pass through the memory on its way to the cooling device. The expression “the memory melts” is closer to a metaphor emphasizing the thermal problem than a technical explanation. The actual risks are exceeding the allowable temperatures of the bonding interfaces and memory, increased leakage current, performance throttling, and reduced lifespan.

Memory Below, GPU Above
THBM, reportedly proposed by Professor Kim Jung-ho’s team at KAIST, is based on the idea of placing the GPU above the HBM stack so that the heat-generating logic sits closer to the cooling device. This may be advantageous for heat dissipation, but the following challenges remain.

Mechanical stability of the heavy logic die and memory stack
Power delivery and signal routing
Bonding yield for dies manufactured using different processes
The ability to replace and test defective dies
Package design that includes the cooling plate

The competitiveness of vertical stacking cannot be determined from a conceptual diagram alone. Measurements of thermal resistance, effective bandwidth, yield, and total cost of ownership are required.

Are Optical Interconnects a Way to Avoid Dependence on Memory?
Using memory and storage outside the GPU requires data to travel farther. As electrical signals travel longer distances at higher transmission speeds, they require more power for signal conditioning and retransmission. Optical interconnects are attracting attention as a candidate for improving bandwidth density and transmission power in high-speed connections among servers, racks, and clusters.
It is difficult to explain NVIDIA’s move to strengthen optical networking solely as a strategy to avoid dependence on a particular memory company. The more direct reason is that as accelerator deployments scale beyond a single server to racks and entire data centers, networking becomes a bottleneck for the overall AI system.
Optical links also do not replace HBM. HBM is likely to provide short latency immediately next to accelerators, while optical links connect more distant memory pools and multiple accelerators.

How Customized HBM Is Changing the Memory Industry
Conventional commodity DRAM has largely involved supplying products with the same specifications to multiple customers. HBM requires coordinated design of the GPU package, interposer, base die, power, and thermals, increasing the degree of joint design between customers and memory companies.
Customized HBM, or CHBM, can incorporate the following functions into the base die.

Memory control and interface optimization
Error correction and reliability management
Data compression and movement control
Security or virtualization functions
Limited data preprocessing and computation

This structure could elevate memory suppliers from simple component manufacturers to system co-designers. Because volume and specifications are discussed early in development, it may also increase the effects of long-term contracts and customer lock-in.
However, customization does not completely eliminate the memory cycle. The pace of AI investment, customer concentration, packaging capacity, yield, and commodity DRAM prices can still cause fluctuations in performance. Products optimized for a particular customer also carry the risk of being difficult to sell to other customers if demand changes.

Is It Accurate to Say That GPU Development Has Stopped?
It is difficult to conclude that GPU development has effectively stopped. Accelerators continue to advance through low-precision computation formats, sparsity processing, transformer-specific engines, chiplets, networking, and cooling.
What has changed is the evaluation criteria. The following metrics have become more important than the peak computing performance of a single chip.

Effective memory bandwidth achieved with actual models
Time to first token and token generation speed per user
Tokens processed per watt
Memory capacity and network bandwidth per rack
Total cost of serving a model

The spread of open-source models has not equalized all model capabilities. However, as multiple providers can now use similar models, it is true that hardware operating efficiency and service deployment capabilities have become more important.

Easily Overlooked Variables: Reliability, Security, and Programming Models
Discussions of next-generation memory focus on bandwidth and capacity, but other factors determine actual commercialization.

Data Accuracy and Lifespan
As memory stacking and density increase, heat, error rates, and lifespan become more important. In AI inference, silent data corruption may manifest as incorrect output rather than an immediate system failure. Error correction, data integrity checks, and fault isolation are as important as performance.

Security of Shared Memory
When multiple accelerators and customers share CXL memory pools or external caches, data isolation, encryption, access control, and residual data deletion are required. As memory capacity grows, the amount of model weights and user context that must be protected also increases.

Can Developers Use It?
Even if a new memory tier exists, its usefulness will be limited if compilers and inference engines cannot automatically determine data placement. A runtime is needed to manage which tensors and KV caches reside in HBM, HBF, CXL memory, or SSDs. Ultimately, hardware competition leads to competition in memory schedulers and system software.

Are AI Data Centers Memory Factories?
Calling a large-scale AI data center a “memory factory” is a useful metaphor for emphasizing the strategic importance of memory. Model weights, KV caches, training data, checkpoints, and search indexes are stored across multiple tiers.
However, there is insufficient evidence for generalizations claiming that a fixed percentage of data center costs is always spent on memory and power. Cost structures vary depending on the balance between training and inference, electricity prices, server depreciation, networking, cooling, utilization, and model efficiency.
A more accurate conclusion is as follows.

Future AI data centers will go beyond being places that install large numbers of GPUs and become systems that place data in the most appropriate memory tier and move it using the least possible power.

Implications for the Korean Semiconductor Industry
Korea has large-scale manufacturing capabilities in HBM and NAND flash, placing it in an important position in the shift toward memory-centric computing. However, memory production volume alone is unlikely to guarantee a long-term advantage.
The necessary strategies are as follows.

Secure the capability to jointly design HBM, HBF, and advanced packaging.
Increase the share of design capabilities in base dies, interface IP, and memory controllers.
Foster companies specializing in thermal management, power semiconductors, optical interconnects, and data center systems.
Build a software ecosystem that enables compilers and inference engines to use Korean-made memory efficiently.
Secure real-world use cases that connect hardware, AI models, and services in smartphones, automobiles, robots, and home appliances.
Manage the risks of dependence on specific customers or a single packaging supply chain.

A position in which “it is difficult to build AI without going through Korea” cannot be created by production volume alone. A difficult-to-replace industrial position emerges when standards, design assets, manufacturing, packaging, software, and end services are connected.

Outlook: Tiering and Co-Design, Not Replacement
Rather than viewing AI semiconductors as being completely replaced from a GPU-centric model by a memory-centric one, it is more accurate to see this as a process in which the boundary between processors and memory becomes blurred.

HBM handles data that must be accessed most quickly.
HBF and CXL memory provide greater capacity.
SSDs handle long-term storage and cold data.
Optical interconnects reduce the cost of connecting distant resources.
Customized base dies and PIM move some computation closer to the data.
Inference software determines the tier in which each piece of data should reside.

The winner in next-generation AI infrastructure is likely to be not the company that creates the single fastest GPU, but the ecosystem that optimizes computing, memory, connectivity, power, cooling, and software as one integrated system.

FAQ
Q. Does memory-centric computing mean replacing GPUs with memory?
A. No. GPUs and other accelerators continue to handle computation. Memory-centric computing is an approach that jointly designs accelerators, HBM, external memory, storage, and networks to reduce the time and power required for data movement.
Q. Why does the KV cache grow with long contexts?
A. Transformers store the key and value information of previous tokens for each layer to avoid repeating the same computation. The amount of storage is generally proportional to the number of tokens, the number of layers, the number of KV heads, and the size of the data format, so long contexts and many concurrent requests increase memory usage.
Q. Is HBF always better than HBM if it has greater capacity?
A. No. NAND-based HBF offers the advantages of high density and non-volatility, but it has higher latency than DRAM-based HBM, and write endurance and error management are more complex. A tiered configuration that places frequently used data in HBM and less frequently used data in HBF is more practical.
Q. What is the difference between HBF and a conventional NVMe SSD?
A. HBF is a concept intended to provide higher bandwidth than conventional SSDs by connecting NAND flash closer to the accelerator through a wider, more parallel interface. Specific performance and connection methods may vary depending on the product and the outcome of standardization.
Q. What are the advantages of vertically stacking a GPU and HBM?
A. Shorter connection distances and the ability to use many vertical interconnects can offer advantages in bandwidth and I/O power. On the other hand, thermal density, power delivery, bonding yield, testing, and cooling become more challenging.
Q. Can optical interconnects replace HBM?
A. Optical interconnects are well suited to connecting multiple accelerators or memory resources over long distances, but they do not directly replace the low latency of HBM located right next to an accelerator. The two technologies are likely to serve different distances and memory tiers.
Q. Can custom HBM eliminate memory price cycles?
A. Custom designs and long-term supply contracts can improve price stability and strengthen customer relationships compared with commodity memory. However, they do not eliminate the effects of fluctuations in AI investment, production capacity, yield, customer concentration, or commodity DRAM prices.
Q. Does supporting long contexts eliminate AI hallucinations?
A. No. Long contexts can provide more supporting evidence, but poor-quality retrieved material, limited model capabilities, poorly constructed instructions, and inadequate source verification can still lead to incorrect answers. Expanding memory is only one way to mitigate hallucinations.
Q. Are GHBM, HBF, and THBM already finalized industry standards?
A. Not all of these terms should be regarded as finalized standards at the same level. Some may be names used in corporate announcements or research proposals, so the specifications of standards bodies such as JEDEC, manufacturers' final specifications, and whether they have actually entered mass production must be verified separately.

Sources
- FMS: the Future of Memory and Storage: https://futurememorystorage.com/
- Efficient Memory Management for Large Language Model Serving with PagedAttention: https://arxiv.org/abs/2309.06180
- FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness: https://arxiv.org/abs/2205.14135
- Mastering LLM Techniques: Inference Optimization: https://developer.nvidia.com/blog/mastering-llm-techniques-inference-optimization/
- Compute Express Link Consortium: https://www.cxlconsortium.org/
- UCIe Consortium: https://www.uciexpress.org/

Images
- Central AI chip linked by data paths to RAM, stacked memory, servers, and storage: https://injoys.com/rails/active_storage/blobs/proxy/eyJfcmFpbHMiOnsiZGF0YSI6OTAxOCwicHVyIjoiYmxvYl9pZCJ9fQ==--5e9ff3355e5af64800fe3a3753eb07a1847f7fba/ai-db43622c.webp
- Exploded AI chip with stacked memory, processor layers, server racks, and glowing data links: https://injoys.com/rails/active_storage/blobs/proxy/eyJfcmFpbHMiOnsiZGF0YSI6OTAyNCwicHVyIjoiYmxvYl9pZCJ9fQ==--0b4dbc6fd04a7a2893bbf9865c8b7e32463f6e20/ai-29d9b4a3.webp

---
Category: Trends
Source: https://injoys.com/en/articles/memory-centric-ai-chips-hbm-hbf-packaging
License: cc_by
Translation-Status: reviewed