Cost per Token: The New Benchmark in AI Chip Competition

Competition among generative AI semiconductors is expanding from maximum compute capacity to the real-world economics of services. Token cost-effectiveness, an informal term used in the industry, refers to how many tokens can be usefully processed for a given cost or amount of power.

However, the cheapest tokens do not necessarily produce the most economical results. Actual competitiveness can be assessed only by measuring model accuracy, response latency, retry counts, and data center utilization together.

What Is Token Cost-Effectiveness?

A token is a unit of text processing that a language model uses when reading input and generating output. API providers typically price input and output tokens separately and may apply a separate rate to cached input.

Token cost-effectiveness is neither a formal accounting term nor a standard benchmark. In practice, the following metrics must be distinguished.

Conceptually, the total cost per token can be viewed as follows.

Total cost per token = depreciation + power and cooling + memory and networking + operating expenses + software costs ÷ effectively processed tokens

Here, effectively processed tokens means tokens that meet service quality standards, not simply the volume generated. Because different models use different tokenizers and produce answers of different lengths, directly comparing only the price per 1 million tokens can be misleading.

Why Inference Costs Are Changing Semiconductor Competition

Costs Recur Every Time a Service Is Used

Large-scale training is a phase in which enormous costs are concentrated, but it is not a completely one-time expense. Even after pretraining, fine-tuning, reinforcement learning, evaluation, and training of new versions are repeated. Nevertheless, inference costs incurred each time a user makes a request have a more direct impact on the business model because they increase almost in step with service growth.

Stanford AI Index 2025 analyzed that the cost of inference providing a given level of model performance has fallen rapidly. This resulted not only from semiconductor improvements but also from smaller models, quantization, inference engines, and increased competition. Lower unit costs make more features economically viable, but a Jevons effect may also emerge as increased usage offsets the decline in unit costs.

AI Agents Amplify Token Usage

A typical chatbot completes a question and answer relatively quickly. By contrast, an AI agent may repeat the following tasks.

• Planning and revision
• Reading long documents or code repositories
• Calling tools such as search, databases, and terminals
• Reviewing execution results and correcting errors
• Generating and evaluating multiple candidates

As the number of task steps grows, input context, intermediate reasoning, tool results, and retries accumulate. In the agent era, the total cost of successfully completing a task matters more than the price of a single model call.

Power and Facilities Create the Actual Supply Constraints

Even if AI accelerators are secured, they cannot operate without sufficient incoming power, cooling, high-bandwidth memory, networking, and data center space. Tokens per watt and tokens per rack therefore mean more than savings on electricity bills. They determine the volume of services that can be provided within limited power capacity.

Hardware Strategies for Lowering Cost per Token

1. Custom Accelerators for Specific Workloads

General-purpose GPUs support a wide range of models and operations, but that flexibility comes at a cost. Large cloud providers can analyze their repeatedly executed internal workloads, eliminate unnecessary functionality, and optimize data movement paths.

Ironwood, unveiled by Google in 2025, is a seventh-generation TPU designed primarily for inference. It is more accurate to understand it not as a chip fixed to a single specific Gemini model, but as part of a vertical integration strategy that jointly optimizes Google’s models, XLA compiler, and data centers.

The purpose of custom chips is not limited to reducing chip purchase costs. Control over supply schedules, improved power efficiency, optimization of internal workloads, and greater bargaining power with external suppliers all play a role.

2. Designing the Entire Rack as One Computer

Because large models do not fit in the memory of a single accelerator, they are distributed across multiple accelerators. Performance may then depend more on communication speed between accelerators, memory bandwidth, and software scheduling than on the computing power of an individual chip.

The NVIDIA Blackwell platform has emphasized a rack-scale approach integrating GPUs, CPUs, NVLink, networking, and software. AMD is also strengthening its Instinct accelerators, open software ecosystem, and rack-level systems. In this competition, the following factors matter more than single-chip benchmarks.

• The number of requests that can be processed concurrently in one rack
• The cost of moving weights and KV cache between accelerators
• The ability to use remaining equipment when a failure occurs
• Sustained performance maintained within power delivery and cooling limits
• Development time required to deploy a model in practice

Even if a rack is expensive, sufficiently high utilization and throughput can lower the cost per token. Conversely, even inexpensive chips lose their economic advantage if immature software or communication bottlenecks cause long idle times.

3. Reducing Communication Boundaries at Wafer Scale

Cerebras developed the WSE family, which uses wafer-sized processors instead of multiple conventional individual dies. This approach places large computing resources and memory bandwidth on a single wafer to reduce some of the chip-to-chip communication found in conventional clusters.

A wafer-scale architecture can target low latency and high throughput, but it does not eliminate all communication. Connecting multiple systems or operating large models still requires external memory, networking, fault recovery, and compilers. Tokens-per-second records for a particular model also cannot be directly compared when precision, batch size, context length, and output conditions differ.

4. Prioritizing Memory and Data Movement Optimization

In inference, moving model weights and KV cache can easily become a bottleneck rather than the computation itself. The decode phase, which generates tokens one at a time, is particularly affected by memory bandwidth.

This is why hardware providers emphasize high-bandwidth memory, chiplets, high-speed interconnects, and larger caches. Even with high compute performance figures, actual token throughput will not increase if the required data cannot be supplied in time.

5. Co-Designing Hardware and Inference Software

Token cost-effectiveness is not determined by semiconductors alone. Even on the same hardware, throughput can vary greatly depending on the following techniques.

• Quantization: Reduces memory use and computation by lowering the precision of weights and operations.
• Continuous batching: Efficiently groups requests that arrive at different times.
• Prefix caching: Reuses calculations for recurring system prompts and context.
• Speculative decoding: A small model generates token candidates, and a large model verifies them.
• Prefill-decode disaggregation: Assigns input processing and output generation to different resources.
• Sparse model routing: Activates only some of the required experts in a Mixture-of-Experts model.

Custom chips are effective only when compilers and models are prepared together. CUDA is an important defensive moat for NVIDIA because an ecosystem encompassing libraries, development tools, optimization expertise, and personnel has already accumulated around it.

Why Dependence on NVIDIA Will Not Disappear Quickly

Companies adopting custom chips or AMD accelerators may also continue using NVIDIA systems. This is less a contradiction in strategy than a form of workload distribution.

NVIDIA’s strengths are as follows.

1. CUDA and broad support for AI libraries
2. General-purpose flexibility for rapidly testing new models
3. Maturity spanning servers, networking, and management software
4. Accumulated experience among developers and operations personnel
5. Broad availability from cloud providers and server manufacturers

By contrast, custom ASICs are more likely to gain an advantage in stable, repetitive, large-scale workloads. The market is therefore likely to evolve toward a structure in which general-purpose GPUs and specialized accelerators divide roles, rather than one type of chip replacing everything else.

Conditions That Must Be Controlled When Comparing Token Cost-Effectiveness

Maximum performance or cost reduction rates announced by vendors cannot be directly compared unless they come from independent benchmarks under identical conditions. At a minimum, the following conditions must be aligned.

Results that specify the model, scenario, and quality conditions, such as MLCommons’ MLPerf Inference, are relatively useful. However, even public benchmarks cannot fully reproduce the model configurations, regional power costs, and traffic patterns of actual companies.

Beyond Simple Token Pricing to Cost per Successful Task

Tokens are easy to measure, but they do not directly represent the value of the final output. Even if Model A has a lower token price than Model B, the cost of completing a task may actually be higher if it produces longer answers or fails frequently.

The following calculation is more appropriate for agent services.

Cost per successful task = total model, tool, and infrastructure costs ÷ number of tasks that pass quality standards

This includes not only token costs but also search APIs, code execution, databases, human review, failed attempts, and costs caused by delays. This perspective is often missing from discussions focused on raw performance or token cost-effectiveness.

How to Distinguish Public Claims from Unverified Figures

Product roadmaps and semiconductor market forecasts change frequently. Among some names and figures included in the provided materials, items that could not be sufficiently verified through official product documentation or reproducible benchmarks were not treated as established facts. Examples include Frozen V2 and Claude Fable 5, which are reportedly dedicated to specific models, Cerebras speeds on a particular undisclosed model, fixed costs per 1 million tokens by chip, and future market share figures.

The conditions behind expressions such as the following must also be checked.

• Up to 10× efficiency: The comparison target, precision, and system scope must be checked.
• 50% reduction in token costs: Conditions for the model, region, utilization, and depreciation are required.
• Hundreds of tokens per second: It must be determined whether this refers to single-user speed or total throughput.
• Moving away from NVIDIA: It must be determined whether this means complete replacement or migration of some internal workloads.

Investment or infrastructure procurement decisions should not rely solely on presentation materials. Independent benchmarks, actual supply schedules, the scope of software support, and total cost of ownership must also be reviewed.

The Market’s Conclusion

AI semiconductor competition is moving from a stage focused solely on peak speed to one centered on how cheaply tokens and tasks that meet quality standards can be produced in terms of total cost.

Custom ASICs pursue high efficiency in repetitive, large-scale workloads, while general-purpose GPUs respond with flexible model support and mature ecosystems. Rack-scale design, memory bandwidth, networking, power, and inference software have become as important as the chips themselves.

Ultimately, the metric that separates winners is not simply tokens per second or the price per 1 million tokens. Total cost per successful task, reflecting quality, latency, utilization, and power constraints under actual traffic, is a more accurate standard.