LLM cost accounting · defect note

OpenAI prompt_tokens vs Anthropic input_tokens

They look like the same field with different names. They are not. One counts cached tokens, the other does not, and every adapter that treats them as interchangeable is wrong — the only question is in which direction.

The difference in one line

OpenAI's prompt_tokens is inclusive. Cached tokens are already inside it; prompt_tokens_details.cached_tokens tells you how many of them were cache hits, as a breakdown of a total you already have.

Anthropic's input_tokens is exclusive. Cached tokens are not inside it; cache_read_input_tokens and cache_creation_input_tokens are separate quantities that must be added.

# OpenAI — 100 tokens of input, 75 of them cached
"usage": {
  "prompt_tokens": 100,                      # already includes the 75
  "prompt_tokens_details": { "cached_tokens": 75 },
  "completion_tokens": 40
}

# Anthropic — the same request
"usage": {
  "input_tokens": 25,                        # excludes the 75
  "cache_read_input_tokens": 75,
  "cache_creation_input_tokens": 0,
  "output_tokens": 40
}

Both objects describe an identical request. One says 100, the other says 25. Both are correct.

Where it goes wrong

The bug is never in the provider SDK. It appears one layer up, in the function that flattens several providers into a single usage shape — the thing every framework, gateway and observability exporter has.

# Under-counting: Anthropic through an OpenAI-shaped normaliser
prompt_tokens = usage.input_tokens          # 25, not 100
# every cost built on this is low by the cache read

# Over-counting: OpenAI semantics applied to an Anthropic payload
prompt_tokens = usage.input_tokens + usage.cache_read_input_tokens \
              + usage.prompt_tokens         # double-counts the cache

Both ship. In Meta's Llama Stack — now ogx — the Anthropic-compatible endpoint copied OpenAI's inclusive prompt tokens into Anthropic's exclusive input tokens and added the cache read on top. A 100-token request with 75 cached totalled 175. Same root cause as the under-count, opposite sign, equally silent.

The conventions, side by side

• OpenAI — prompt_tokens inclusive; breakdown in prompt_tokens_details.cached_tokens.
• Anthropic — input_tokens exclusive; add cache_read_input_tokens and cache_creation_input_tokens.
• Anthropic via Bedrock — same exclusive semantics, different envelope and different field names again. I fixed the Bedrock path in LiveKit agents and in Pipecat separately from the direct API path, because they are genuinely different code.
• Anthropic via an OpenAI-compatible gateway — the worst case. The OpenAI schema has nowhere to put the cache fields, so they are frequently dropped at the translation boundary and the loss is invisible on both sides.
• Gemini — adds a third axis: thinking tokens, which can appear in the total without appearing in the cost. That exact bug is open in Langfuse as #16321.

How to check yours in five minutes

1. Log one raw usage object per provider, unmodified, straight from the SDK.
2. For the Anthropic one, confirm input_tokens is smaller than the prompt you actually sent. If it is, your cache fields are live and must be added somewhere.
3. Grep for every site that reads input_tokens or prompt_tokens. Each one either handles both conventions or is wrong for one provider.
4. Assert it: send a known request twice, so the second is a cache hit, and check that your recorded total is the same both times. It should be — the cache changes the price, not the token count. If your total drops on the second call, you are dropping the cache read.

That last test is the one worth keeping in CI. It fails on exactly this defect class and on nothing else.

Related: Anthropic cache tokens are not in total_tokens · Why your Claude bill is higher than your dashboard

← Back to the audit