AI Inference Costs: Why Banks Pay More Despite Cheaper Models, and What Helps

AI model prices have been falling faster than almost any other technology metric for three years. Yet at many banks, the bill for AI in customer-facing work keeps rising. The reason is inference, the day-to-day running of the models: every answer from an assistant, every summary of an advisory conversation and every step taken by an AI agent consumes compute. This article explains why AI inference costs are rising for banks even as individual tokens get cheaper, what is shifting in chips and data centres, and which architecture keeps the cost per customer conversation under control without handing over customer data.

Why AI inference costs are rising even as tokens get cheaper

The price per unit of performance is falling sharply, but volume is growing faster, driven above all by AI agents. Epoch AI has analysed language model prices since 2023: the cost of reaching a given level of performance falls by around 47 percent per quarter on average, roughly a factor of 13 per year (Epoch AI, September 2026). Epoch flags a caveat itself: the calculation assumes users always switch to the cheapest model at that level. In practice, companies save less.

Volume, meanwhile, keeps growing. Gartner expects inference costs per agentic workflow to rise more than fivefold by 2028. An agentic reasoning model costs at least five times as much as a basic chatbot interaction, and agents need five to thirty times more tokens per task (VentureBeat on Gartner, October 2026). Gartner calls this the "inference paradox": the unit price falls while the bill grows.

Across total compute, Deloitte estimates that inference will account for around two thirds of all AI compute in 2026, up from about a third in 2023 and half in 2025 (Deloitte TMT Predictions 2026). For a bank, this means the number that matters is no longer the price per million tokens but the price per resolved request.

A hypothetical savings bank sorts its AI requests

Illustration: AI-generated, hypothetical example.

Hypothetical example: The head of IT at a mid-sized savings bank receives next year's projection. The AI assistant in customer service is due to move from the pilot group to all advisers, along with an agent that prepares account requests. If every request goes to a large cloud model, as it did in the pilot, the bill would multiply. She and her architect review the pilot requests. The result: a large share are routine cases such as questions about blocking a card, standing orders or account statements. A small model in the bank's own or a regional data centre is enough for these, and the personal data never leaves it. Only complex cases without direct personal data should continue to go to a large model. The bank and the people are fictitious.

What large banks are doing today: routing and smaller models

The banks that have scaled AI furthest are deliberately putting a switch in front of their models. Bank of America routes requests through its own routing layer, "Orchestra". A small classifier model decides which model can handle a request most cost-effectively. CTO and CIO Hari Gopalkrishnan said in October 2026: "We didn't want to be beholden to any given model", adding that simpler classifier models handle easy decisions that don't need a large language model (CIO Dive, October 2026).

Capital One takes the same route with distilled, smaller models for its Chat Concierge. According to Prem Natarajan, response latency fell fivefold and dealer engagement rose by 50 percent (also CIO Dive).

BNP Paribas runs its own LLM-as-a-service platform in its data centres with open-source and Mistral models (BNP Paribas) and renewed its Mistral partnership in May 2026, including for on-premise KYC applications (BNP Paribas, May 2026). Its own infrastructure stays in place after a new cloud agreement with Google, too. CIO Marc Camus stressed that the agreement "doesn't mean that we will get rid of our on-premise infrastructure"; highly confidential data stays outside the public cloud, and model and infrastructure are chosen per application (Mobile World Live, September 2026).

In a VentureBeat survey of 109 companies, only 31 percent deliberately send routine tasks to cheaper models, and 21 percent have no automatic stop when agents consume too many tokens (VentureBeat, October 2026). How to choose models by task in the first place is covered in our article AI model costs: which model for what.

More choice in chips, but not always freely available

There are now real alternatives to Nvidia for inference, and that changes the negotiating position of banks and their data centre partners. According to Silicon Analysts estimates, Nvidia held more than 90 percent of the training market in 2025 but only 60 to 75 percent of inference (Quartz, July 2026).

Three developments are relevant for banks. At CES 2026, AMD presented the MI440X, an accelerator for inference in on-premise data centres (Computerworld, January 2026). Google describes its "Ironwood" TPU as the first TPU designed specifically for inference; it is, however, only available to rent in Google Cloud (Quartz). Intel has announced "Crescent Island", an air-cooled inference GPU in its GPU D-series with up to 480 GB of memory at 350 watts that fits into existing data centres (Intel, October 2026); samples are available and launch is expected in 2027 (ServeTheHome).

For a bank's architecture, this means more competition can lower the cost per answer. Some alternatives, however, tie the bank to a single cloud provider. Under DORA, that is a concentration risk that must be documented and backed by an exit option, as our article on AI provider concentration risk shows.

Local, in your own data centre or in the cloud?

For banks, the real choice is not "laptop or cloud" but "own or regional data centre or public cloud". Deloitte expects almost all AI computing in 2026 to run in data centres and enterprise servers rather than on PCs or smartphones; the AI chips in devices mainly handle simple one-off tasks (Deloitte). At the same time, Deloitte sees the market for on-premise enterprise AI systems growing to more than 50 billion US dollars in 2026. A server with eight GPUs costs 300,000 to 500,000 US dollars, a full rack three to five million.

Lenovo's Arthur Hu puts the economics simply: "if you know what your workload size and predictability is, you don't need to pay the additional premium" of flexible cloud capacity. He names privacy and sovereignty as additional drivers (Computerworld). That describes many banking applications well: service questions, conversation summaries and classifications are predictable and contain personal data.

The local route has bottlenecks of its own. In May 2026, Apple CEO Tim Cook spoke of Mac mini and Mac Studio shortages lasting several months, driven in part by demand for local AI; memory is also tight because the same manufacturers are serving data centre demand (Tom's Hardware, May 2026). What it takes to get to production is covered in our article on self-hosting LLMs.

Bubble or rebuild? Three scenarios instead of a forecast

Nobody can say with confidence today whether investment in AI data centres is a bubble. What matters more for banks is that their architecture works in several scenarios. Three developments are plausible:

1. Price declines win out. More competition in chips and models brings the cost per answer down faster than volume grows. The winners are those who can switch between models quickly.

2. Volume grows faster. Agents with many reasoning steps drive consumption, as Gartner expects. The winners are those who consistently send routine cases to small models and set budgets per agent.

3. Scarcity raises prices for a while. Memory, power or chips become the bottleneck, and providers raise prices or tighten quotas. The winners are those who run part of the load predictably in their own or a regional data centre.

The answer is the same in all three cases: a switch in front of the models, measured costs per request and an exit option. How to measure the actual value against this is described in our article AI bubble or real value?

How banks control the cost per customer conversation

AI inference costs become manageable when every request has a route, a budget and a measuring point. Gartner's recommendations include cost observability, evaluation by value per outcome, model tiering with routing and stress tests against swings in token prices (VentureBeat on Gartner). For a bank, this translates into five steps:

• Measure cost per request: not per token, but per resolved customer question, per conversation summary, per agent task.

• Sort requests by difficulty and data type: routine cases with personal data go to small models in the bank's own or a regional data centre; complex cases without personal data, or pseudonymised, go to large models.

• Set budgets and stops per agent: a cap on tokens and steps, followed by a handover to a person.

• Keep context small: an agent receives only the data it needs for the task. This reduces tokens and meets the GDPR principle of data minimisation at the same time.

• Prepare for switching: connect models through a common interface so that replacing one does not require a new integration.

How such a switch works together with a planner, domain agents and a reviewer is described in our article on the orchestration layer in banks.

Data protection is the second reason to run inference close to the data

For banks, where a model runs is not only a cost question but also a question of data protection and supervision. The more personal data an assistant processes, the more important it is that the data does not leave the bank or its contractually bound data centre partner. ACCELERAID therefore follows GDPR by design: the platform runs as a private cloud in an ISO 27001-certified data centre in Frankfurt, on-premise at the bank on request, and is SOC 2 Type I certified (ACCELERAID). There is no cross-client training; a bank's data trains only its own models. The AI Context Layer checks consent and purpose before every request to a model and passes on only the context the task requires; the overarching orchestration and control plane is currently in development. Less context also means fewer tokens, so data protection and cost control point in the same direction.

Five takeaways

1. AI inference costs are rising at banks even as tokens get cheaper: Epoch AI measures a price decline of roughly 13 times per year per performance level, while Gartner expects costs per agentic workflow to rise more than fivefold by 2028.

2. According to Deloitte, inference accounts for around two thirds of AI compute in 2026; the decisive metric for banks is the cost per resolved request.

3. Bank of America, Capital One and BNP Paribas show the way: routing in front of the models, smaller models for routine work and infrastructure chosen per application.

4. Choice in chips is growing, but some alternatives are only available from one cloud provider; under DORA, that is a concentration risk.

5. Inference close to the data lowers costs for predictable workloads and protects personal data; agent budgets, small context and an exit option make the architecture robust.

Illustration: AI-generated. AI-assisted content: We use AI technologies and automated agents in the creation of our articles, including from Microsoft, Google, OpenAI, Anthropic and other providers. Topics, editorial direction and final approval remain with our team.