Injoys
Competition among generative AI semiconductors is expanding from maximum compute capacity to the real-world economics of services. Token cost-effectiveness, an informal term used in the industry, refers to how many tokens can be usefully processed for a given cost or amount of power.
However, the cheapest tokens do not necessarily produce the most economical results. Actual competitiveness can be assessed only by measuring model accuracy, response latency, retry counts, and data center utilization together.
What Is Token Cost-Effectiveness?
A token is a unit of text processing that a language model uses when reading input and generating output. API providers typically price input and output tokens separately and may apply a separate rate to cached input.
Token cost-effectiveness is neither a formal accounting term nor a standard benchmark. In practice, the following metrics must be distinguished.
| Metric | Meaning | Easily overlooked factors |
|---|---|---|
| Tokens per dollar | Number of tokens processed for the same cost | Quality, latency, differences between input and output pricing |
| Tokens per second | Generation processing speed | Number of concurrent users and time to first token |
| Tokens per watt | Power efficiency | Cooling and power conversion losses |
| Throughput per server | Total throughput of a server or rack | Low utilization and wait times |
| Cost per successful task | Total cost of completing a target | Retries, tool calls, failed paths |
Conceptually, the total cost per token can be viewed as follows.
Total cost per token = depreciation + power and cooling + memory and networking + operating expenses + software costs ÷ effectively processed tokens
Here, effectively processed tokens means tokens that meet service quality standards, not simply the volume generated. Because different models use different tokenizers and produce answers of different lengths, directly comparing only the price per 1 million tokens can be misleading.
Why Inference Costs Are Changing Semiconductor Competition
Costs Recur Every Time a Service Is Used
Large-scale training is a phase in which enormous costs are concentrated, but it is not a completely one-time expense. Even after pretraining, fine-tuning, reinforcement learning, evaluation, and training of new versions are repeated. Nevertheless, inference costs incurred each time a user makes a request have a more direct impact on the business model because they increase almost in step with service growth.
Stanford AI Index 2025 analyzed that the cost of inference providing a given level of model performance has fallen rapidly. This resulted not only from semiconductor improvements but also from smaller models, quantization, inference engines, and increased competition. Lower unit costs make more features economically viable, but a Jevons effect may also emerge as increased usage offsets the decline in unit costs.
AI Agents Amplify Token Usage
A typical chatbot completes a question and answer relatively quickly. By contrast, an AI agent may repeat the following tasks.
- Planning and revision
- Reading long documents or code repositories
- Calling tools such as search, databases, and terminals
- Reviewing execution results and correcting errors
- Generating and evaluating multiple candidates
As the number of task steps grows, input context, intermediate reasoning, tool results, and retries accumulate. In the agent era, the total cost of successfully completing a task matters more than the price of a single model call.
Power and Facilities Create the Actual Supply Constraints
Even if AI accelerators are secured, they cannot operate without sufficient incoming power, cooling, high-bandwidth memory, networking, and data center space. Tokens per watt and tokens per rack therefore mean more than savings on electricity bills. They determine the volume of services that can be provided within limited power capacity.
Hardware Strategies for Lowering Cost per Token
1. Custom Accelerators for Specific Workloads
General-purpose GPUs support a wide range of models and operations, but that flexibility comes at a cost. Large cloud providers can analyze their repeatedly executed internal workloads, eliminate unnecessary functionality, and optimize data movement paths.
| Company | Publicly disclosed accelerator family | Main direction | Caveats when interpreting |
|---|---|---|---|
| TPU, Ironwood | Co-design of models, compilers, and cloud infrastructure | The same efficiency cannot be assumed to be reproducible outside Google’s internal environment | |
| AWS | Trainium, Inferentia | AWS-specific chips and Neuron software tailored to training and inference | Supported operations and model porting costs must be checked |
| Microsoft | Maia 100 | Custom accelerator for Azure AI workloads | Commercial workload costs are difficult to calculate from public specifications alone |
| Meta | MTIA | Optimization of large-scale internal inference workloads such as recommendation and ranking | Not a general-purpose LLM chip that replaces all generative AI models |
Ironwood, unveiled by Google in 2025, is a seventh-generation TPU designed primarily for inference. It is more accurate to understand it not as a chip fixed to a single specific Gemini model, but as part of a vertical integration strategy that jointly optimizes Google’s models, XLA compiler, and data centers.
The purpose of custom chips is not limited to reducing chip purchase costs. Control over supply schedules, improved power efficiency, optimization of internal workloads, and greater bargaining power with external suppliers all play a role.
2. Designing the Entire Rack as One Computer
Because large models do not fit in the memory of a single accelerator, they are distributed across multiple accelerators. Performance may then depend more on communication speed between accelerators, memory bandwidth, and software scheduling than on the computing power of an individual chip.
The NVIDIA Blackwell platform has emphasized a rack-scale approach integrating GPUs, CPUs, NVLink, networking, and software. AMD is also strengthening its Instinct accelerators, open software ecosystem, and rack-level systems. In this competition, the following factors matter more than single-chip benchmarks.
- The number of requests that can be processed concurrently in one rack
- The cost of moving weights and KV cache between accelerators
- The ability to use remaining equipment when a failure occurs
- Sustained performance maintained within power delivery and cooling limits
- Development time required to deploy a model in practice
Even if a rack is expensive, sufficiently high utilization and throughput can lower the cost per token. Conversely, even inexpensive chips lose their economic advantage if immature software or communication bottlenecks cause long idle times.
3. Reducing Communication Boundaries at Wafer Scale
Cerebras developed the WSE family, which uses wafer-sized processors instead of multiple conventional individual dies. This approach places large computing resources and memory bandwidth on a single wafer to reduce some of the chip-to-chip communication found in conventional clusters.
A wafer-scale architecture can target low latency and high throughput, but it does not eliminate all communication. Connecting multiple systems or operating large models still requires external memory, networking, fault recovery, and compilers. Tokens-per-second records for a particular model also cannot be directly compared when precision, batch size, context length, and output conditions differ.
4. Prioritizing Memory and Data Movement Optimization
In inference, moving model weights and KV cache can easily become a bottleneck rather than the computation itself. The decode phase, which generates tokens one at a time, is particularly affected by memory bandwidth.
This is why hardware providers emphasize high-bandwidth memory, chiplets, high-speed interconnects, and larger caches. Even with high compute performance figures, actual token throughput will not increase if the required data cannot be supplied in time.
5. Co-Designing Hardware and Inference Software
Token cost-effectiveness is not determined by semiconductors alone. Even on the same hardware, throughput can vary greatly depending on the following techniques.
- Quantization: Reduces memory use and computation by lowering the precision of weights and operations.
- Continuous batching: Efficiently groups requests that arrive at different times.
- Prefix caching: Reuses calculations for recurring system prompts and context.
- Speculative decoding: A small model generates token candidates, and a large model verifies them.
- Prefill-decode disaggregation: Assigns input processing and output generation to different resources.
- Sparse model routing: Activates only some of the required experts in a Mixture-of-Experts model.
Custom chips are effective only when compilers and models are prepared together. CUDA is an important defensive moat for NVIDIA because an ecosystem encompassing libraries, development tools, optimization expertise, and personnel has already accumulated around it.
Why Dependence on NVIDIA Will Not Disappear Quickly
Companies adopting custom chips or AMD accelerators may also continue using NVIDIA systems. This is less a contradiction in strategy than a form of workload distribution.
NVIDIA’s strengths are as follows.
- CUDA and broad support for AI libraries
- General-purpose flexibility for rapidly testing new models
- Maturity spanning servers, networking, and management software
- Accumulated experience among developers and operations personnel
- Broad availability from cloud providers and server manufacturers
By contrast, custom ASICs are more likely to gain an advantage in stable, repetitive, large-scale workloads. The market is therefore likely to evolve toward a structure in which general-purpose GPUs and specialized accelerators divide roles, rather than one type of chip replacing everything else.
