From GPU-Centric to Memory-Centric: A Shift in the AI Chip Paradigm

The competitive criteria for AI semiconductors are expanding beyond simple computation counts to how much data can be stored and moved using little power. In large-scale AI inference, GPUs may sit idle not because they cannot perform computations, but because they do not receive the required model weights and KV cache in time.

This change is often described as a “shift from GPU-centric to memory-centric computing.” However, this does not mean that GPUs are disappearing or have stopped advancing. More precisely, it is a move toward jointly designing GPUs, HBM, external memory, storage, networking, and software as a single data movement system.

What Is Memory-Centric Computing?

Memory-centric computing is an approach that takes memory location, bandwidth, capacity, and computing capabilities as the starting point of system design to reduce the cost of repeatedly moving data to processors.

The arithmetic throughput of AI chips has increased rapidly, but retrieving data from outside the chip remains relatively slow and consumes significant power. As a result, the following technologies are becoming important together.

• HBM placed close to a GPU or AI accelerator
• 3D packaging that vertically connects memory and logic
• Customized HBM that incorporates control or limited computing functions into the base die
• Memory expansion and pooling using CXL
• The HBF concept, which connects NAND flash through a wide interface
• Optical interconnects intended to reduce transmission power between chips and servers
• PIM and near-memory computing, which process data close to memory

In other words, memory-centric computing does not refer to a single memory product. It is a system principle that combines multiple tiers, from fast memory to high-capacity storage, to minimize data movement.

Why AI Inference Intensifies the Memory Bottleneck

Model Weights and the KV Cache Create Different Burdens

The memory required for AI inference can broadly be divided into model weights, activations generated during execution, and the KV cache. Model weights occupy a fixed amount of space even when there are no requests, while the KV cache grows according to input length, output length, and the number of concurrent requests.

To avoid recomputing previous tokens from scratch each time, transformer models store attention key and value information for each layer in the KV cache. In simplified terms, the KV cache size for a single request is proportional to the following factors.

The exact KV cache size must be verified from the model architecture and implementation specifications.

The KV cache stores keys and values separately. Therefore, as context grows longer or more users are processed simultaneously, the KV cache grows rapidly. Not every model uses millions of tokens, nor does cache size uniformly increase by hundreds of times, but it is clear that long-context inference and agentic tasks increase memory pressure.

Hallucinations and Memory Capacity Are Not the Same Problem

Longer context and retrieval-augmented generation can provide models with more supporting evidence. However, increasing memory alone does not eliminate hallucinations. The quality of search results, prompt design, the model’s reasoning capabilities, source verification, and evaluation frameworks are also necessary.

Software Can Reduce Physical Memory Demand

The KV cache bottleneck is not solved solely by adding hardware.

• MQA and GQA reduce the number of KV heads that must be stored.
• Quantization reduces the number of bytes used by weights and caches.
• PagedAttention manages caches in pages, mitigating fragmentation and waste.
• Continuous batching efficiently groups multiple requests.
• Prefix caching reuses repeated system prompts or shared context.
• Cache eviction and tiered offloading move less important information to slower memory.

Therefore, the actual amount of memory investment in a data center depends not only on model size but also on the efficiency of the inference engine.

The Memory Hierarchy of AI Data Centers

It is difficult for a single type of memory to satisfy speed, capacity, cost, and power efficiency requirements all at once. AI data centers are likely to evolve into the following hierarchy.

The key is not for either HBM or HBF to win, but to deploy multiple tiers according to how frequently data is used.

The Next-Generation Memory Competition Revealed by FMS 2026

FMS, held in Santa Clara, United States, in August 2026, addressed post-HBM AI memory and storage architectures as major agenda items. Discussions centered on taller stacks, high-capacity NAND-based tiers, customized memory, and data center connectivity technologies.

Samsung Electronics’ GHBM and the HBF discussed by SK hynix with Google and SanDisk were introduced as examples illustrating this direction. However, it is difficult to conclude that names such as GHBM, HBF, and THBM are all universal generation names finalized under identical JEDEC specifications. Technologies at the corporate announcement stage, joint development concepts, and standardized commercial products must be distinguished from one another.

Moreover, a higher generation name does not necessarily make actual AI services faster. Effective bandwidth, memory capacity, latency, power, cooling, package yield, and software support must all be compared together.

HBM’s Strengths and Capacity Limitations

HBM vertically stacks multiple DRAM dies and connects them using through-silicon vias and a wide interface. Its key advantage is that it can supply large amounts of data in parallel close to the accelerator, unlike conventional board-level memory.

However, HBM capacity cannot be increased indefinitely.

1. As the number of stacked layers increases, manufacturing yield and testing become more difficult.
2. The package area required to place the GPU and HBM together increases.
3. Heat generated by the accelerator and memory must be removed from a confined space.
4. Power delivery networks and interposer wiring also become more complex.
5. Using expensive HBM to store even infrequently accessed data reduces economic efficiency.

For this reason, tiering is becoming important, with HBM handling the most frequently used data and the rest being moved to CXL memory, flash, or SSDs.

Can HBF Replace HBM?

HBF stands for High Bandwidth Flash and is a concept that parallelizes and stacks NAND flash to create a high-bandwidth memory tier closer to accelerators than conventional SSDs. Because NAND offers higher density and non-volatility than DRAM, it has the potential to store more data in the same physical space.

However, NAND has different characteristics from DRAM.

• It has longer read latency.
• Data must be erased before it can be overwritten.
• Endurance must be managed according to the number of writes.
• Bad blocks, error correction, and wear leveling must be handled.
• It is better suited to large sequential accesses than small, irregular accesses.

Therefore, even if HBF is commercialized, it is more likely to function as a high-capacity tier between HBM and SSDs than to directly replace HBM. Read-heavy model weights, infrequently reused caches, checkpoints, and retrieval data could be candidates. Its actual suitability will depend on interface standards, latency, endurance, controllers, and inference software support.

The 3D Packaging Challenges Raised by GHBM and THBM

Placing accelerators and HBM side by side within a package imposes limits on connection width and package area. To overcome these limitations, concepts have emerged that vertically stack logic and memory to shorten wiring distances.

A Structure That Places Memory Above the GPU

The concept introduced as GHBM or Z-axis HBM aims to reduce data movement distances by vertically integrating the GPU and HBM. Numerous short vertical connections are expected to provide high bandwidth and low I/O power.

The problem is heat. When memory is placed above the GPU, heat generated by the GPU may pass through the memory on its way to the cooling device. The expression “the memory melts” is closer to a metaphor emphasizing the thermal problem than a technical explanation. The actual risks are exceeding the allowable temperatures of the bonding interfaces and memory, increased leakage current, performance throttling, and reduced lifespan.

Memory Below, GPU Above

THBM, reportedly proposed by Professor Kim Jung-ho’s team at KAIST, is based on the idea of placing the GPU above the HBM stack so that the heat-generating logic sits closer to the cooling device. This may be advantageous for heat dissipation, but the following challenges remain.

• Mechanical stability of the heavy logic die and memory stack
• Power delivery and signal routing
• Bonding yield for dies manufactured using different processes
• The ability to replace and test defective dies
• Package design that includes the cooling plate

The competitiveness of vertical stacking cannot be determined from a conceptual diagram alone. Measurements of thermal resistance, effective bandwidth, yield, and total cost of ownership are required.

Are Optical Interconnects a Way to Avoid Dependence on Memory?

Using memory and storage outside the GPU requires data to travel farther. As electrical signals travel longer distances at higher transmission speeds, they require more power for signal conditioning and retransmission. Optical interconnects are attracting attention as a candidate for improving bandwidth density and transmission power in high-speed connections among servers, racks, and clusters.

It is difficult to explain NVIDIA’s move to strengthen optical networking solely as a strategy to avoid dependence on a particular memory company. The more direct reason is that as accelerator deployments scale beyond a single server to racks and entire data centers, networking becomes a bottleneck for the overall AI system.

Optical links also do not replace HBM. HBM is likely to provide short latency immediately next to accelerators, while optical links connect more distant memory pools and multiple accelerators.

How Customized HBM Is Changing the Memory Industry

Conventional commodity DRAM has largely involved supplying products with the same specifications to multiple customers. HBM requires coordinated design of the GPU package, interposer, base die, power, and thermals, increasing the degree of joint design between customers and memory companies.

Customized HBM, or CHBM, can incorporate the following functions into the base die.

• Memory control and interface optimization
• Error correction and reliability management
• Data compression and movement control
• Security or virtualization functions
• Limited data preprocessing and computation

This structure could elevate memory suppliers from simple component manufacturers to system co-designers. Because volume and specifications are discussed early in development, it may also increase the effects of long-term contracts and customer lock-in.

However, customization does not completely eliminate the memory cycle. The pace of AI investment, customer concentration, packaging capacity, yield, and commodity DRAM prices can still cause fluctuations in performance. Products optimized for a particular customer also carry the risk of being difficult to sell to other customers if demand changes.

Is It Accurate to Say That GPU Development Has Stopped?

It is difficult to conclude that GPU development has effectively stopped. Accelerators continue to advance through low-precision computation formats, sparsity processing, transformer-specific engines, chiplets, networking, and cooling.

What has changed is the evaluation criteria. The following metrics have become more important than the peak computing performance of a single chip.

• Effective memory bandwidth achieved with actual models
• Time to first token and token generation speed per user
• Tokens processed per watt
• Memory capacity and network bandwidth per rack
• Total cost of serving a model

The spread of open-source models has not equalized all model capabilities. However, as multiple providers can now use similar models, it is true that hardware operating efficiency and service deployment capabilities have become more important.

Easily Overlooked Variables: Reliability, Security, and Programming Models

Discussions of next-generation memory focus on bandwidth and capacity, but other factors determine actual commercialization.

Data Accuracy and Lifespan

As memory stacking and density increase, heat, error rates, and lifespan become more important. In AI inference, silent data corruption may manifest as incorrect output rather than an immediate system failure. Error correction, data integrity checks, and fault isolation are as important as performance.

Security of Shared Memory

When multiple accelerators and customers share CXL memory pools or external caches, data isolation, encryption, access control, and residual data deletion are required. As memory capacity grows, the amount of model weights and user context that must be protected also increases.

Can Developers Use It?

Even if a new memory tier exists, its usefulness will be limited if compilers and inference engines cannot automatically determine data placement. A runtime is needed to manage which tensors and KV caches reside in HBM, HBF, CXL memory, or SSDs. Ultimately, hardware competition leads to competition in memory schedulers and system software.

Are AI Data Centers Memory Factories?

Calling a large-scale AI data center a “memory factory” is a useful metaphor for emphasizing the strategic importance of memory. Model weights, KV caches, training data, checkpoints, and search indexes are stored across multiple tiers.

However, there is insufficient evidence for generalizations claiming that a fixed percentage of data center costs is always spent on memory and power. Cost structures vary depending on the balance between training and inference, electricity prices, server depreciation, networking, cooling, utilization, and model efficiency.

A more accurate conclusion is as follows.

Future AI data centers will go beyond being places that install large numbers of GPUs and become systems that place data in the most appropriate memory tier and move it using the least possible power.

Implications for the Korean Semiconductor Industry

Korea has large-scale manufacturing capabilities in HBM and NAND flash, placing it in an important position in the shift toward memory-centric computing. However, memory production volume alone is unlikely to guarantee a long-term advantage.

The necessary strategies are as follows.

1. Secure the capability to jointly design HBM, HBF, and advanced packaging.
2. Increase the share of design capabilities in base dies, interface IP, and memory controllers.
3. Foster companies specializing in thermal management, power semiconductors, optical interconnects, and data center systems.
4. Build a software ecosystem that enables compilers and inference engines to use Korean-made memory efficiently.
5. Secure real-world use cases that connect hardware, AI models, and services in smartphones, automobiles, robots, and home appliances.
6. Manage the risks of dependence on specific customers or a single packaging supply chain.

A position in which “it is difficult to build AI without going through Korea” cannot be created by production volume alone. A difficult-to-replace industrial position emerges when standards, design assets, manufacturing, packaging, software, and end services are connected.

Outlook: Tiering and Co-Design, Not Replacement

Rather than viewing AI semiconductors as being completely replaced from a GPU-centric model by a memory-centric one, it is more accurate to see this as a process in which the boundary between processors and memory becomes blurred.

• HBM handles data that must be accessed most quickly.
• HBF and CXL memory provide greater capacity.
• SSDs handle long-term storage and cold data.
• Optical interconnects reduce the cost of connecting distant resources.
• Customized base dies and PIM move some computation closer to the data.
• Inference software determines the tier in which each piece of data should reside.

The winner in next-generation AI infrastructure is likely to be not the company that creates the single fastest GPU, but the ecosystem that optimizes computing, memory, connectivity, power, cooling, and software as one integrated system.