Cached Tokens

Token Factory automatically reuses previously computed prompt prefixes. Tokens served from this cache are billed at the model’s cached input price, and every response reports how many tokens were cached. Caching is implicit: you don’t need to change your requests, set cache keys, or opt in. It is also best-effort. Cache hits, hit rates, and cache duration are not guaranteed.

How it works

When you send a request, the inference engine checks whether the beginning of your prompt (the prefix) matches a prompt it processed recently. If it does, it reuses the stored computation for that prefix instead of recomputing it. The tokens served this way are cached tokens. Prompts with a long, stable prefix benefit most, for example:

a repeated system prompt or set of instructions;
tool and function definitions that are the same across calls;
few-shot examples or reference documents placed at the start of the prompt;
multi-turn conversations and agent loops, where each request repeats the earlier history.

Billing

Every Serverless text-generation request is billed under up to three token types:

Token type	What it covers
Input	Prompt tokens that were not served from the cache
Cached input	Prompt tokens served from the cache
Output	Tokens generated by the model

Cached input tokens are a subset of the prompt tokens, so each prompt token is billed once:

prompt tokens = input (uncached) tokens + cached input tokens

For example, a request with 3,108 prompt tokens, of which 3,072 are cached, is billed for 36 input tokens, 3,072 cached input tokens, and the output tokens. Your invoice and consumption report show cached input as a separate line item. If a request has no cache hit, no cached input item is created and all prompt tokens are billed at the regular input price.

Pricing

The cached input price is set per model. See the pricing page for current input, cached input, and output prices for each model. The cached input price applies only to tokens that were actually served from the cache. How many tokens are served from the cache depends on your workload and on factors outside your control, so the amount you save is not guaranteed and can vary from request to request. Models that are not listed, or that don’t yet have a cached input price, bill all prompt tokens at the regular input price.

Check cache usage in API responses

The number of cached tokens is returned in the usage object of every response, including streaming responses. prompt_tokens (or input_tokens) is always the total number of input tokens, cached tokens included.

Chat Completions and Completions

{
  "usage": {
    "prompt_tokens": 3108,
    "completion_tokens": 62,
    "total_tokens": 3170,
    "prompt_tokens_details": {
      "cached_tokens": 3072
    }
  }
}

Responses API

{
  "usage": {
    "input_tokens": 3108,
    "output_tokens": 62,
    "total_tokens": 3170,
    "input_tokens_details": {
      "cached_tokens": 3072
    }
  }
}

Streaming

Not all models return usage information in streaming mode by default. To make sure usage, including cached_tokens, is returned when streaming, set stream_options.include_usage to true. Without it, a streaming response may not show how many tokens were cached, even though the tokens are still billed.

For Chat Completions and Completions, the final chunk of the stream then includes the usage object. The Responses API returns usage in the final response.completed event.

from openai import OpenAI

client = OpenAI(
    base_url="https://api.tokenfactory.nebius.com/v1/",
    api_key="<NEBIUS_API_KEY>",
)

response = client.chat.completions.create(
    model="<model-name>",
    messages=[
        {"role": "system", "content": "<long, stable system prompt>"},
        {"role": "user", "content": "<your question>"},
    ],
)

usage = response.usage
cached = usage.prompt_tokens_details.cached_tokens
print(f"{cached} of {usage.prompt_tokens} prompt tokens were cached")

Check a charge

If you think a request was billed incorrectly, compare the usage object in the response with the line items in your consumption report, and contact support with the request ID. Billing is based on the cached token count reported for each request. For streaming requests, make sure stream_options.include_usage is set, so that the response contains usage to compare against.

Get more cache hits

Put static content first (system prompt, tool definitions, examples, documents) and dynamic content last (the user’s latest message).
Keep the prefix identical between requests: the same text, the same order, and the same tool definitions. Even a small change early in the prompt can prevent a cache hit for everything after it.
In multi-turn conversations, append new messages to the end of the history instead of rewriting earlier messages.
Send requests that share a prefix close together in time, because cache entries are evicted over time.
Use the same model for requests that share a prefix. The cache is not shared between models.

Best-effort caching and no guarantees

Caching is best-effort. Nebius doesn’t provide a service level agreement (SLA) for caching, and doesn’t guarantee a cache hit, a minimum cache-hit rate, a specific discount or cost saving, or how long a prefix stays cached. The cached input price is the price per token for tokens that were served from the cache. It is not a promise that any share of your tokens will be cached. In particular:

A repeated request can return cached_tokens: 0. This is a valid result, not an error. Possible reasons include cache eviction, the request being served by a different replica, or a model or service update.
cached_tokens does not have to match the length of the repeated part of your prompt. The engine caches in internal blocks, so the value is usually rounded down to a block boundary.
The first request with a new prefix can still show some cached tokens, for example when it shares a chat template or prefix with other requests.
Changing the model, chat template, or tokenization parameters changes the effective prefix and can reduce cache hits.

Data, privacy, and cache scope

What is cached

The cache holds temporary computed state that the model derives from your prompt. It doesn’t hold prompt or response text, can’t be read back into text, and isn’t exposed through any API. Frequently reused prefixes stay cached longer, and less-used ones are removed. How long a prefix stays cached isn’t guaranteed. An unused prefix may be removed after only a few minutes, and entries can be evicted at any time under load.

Zero data retention (ZDR)

Caching works the same way for accounts with zero data retention (ZDR) enabled. Prompts and responses are still not stored or logged, so ZDR commitments stay in place. Don’t rely on caching as an access control or isolation boundary. Don’t put secrets in the beginning of a prompt expecting them to be isolated from other requests. If you use prompt_cache_key, note that it only helps route requests to the server that holds the cache. It isn’t an access key or a security boundary.

What Nebius Token Factory records

Nebius Token Factory records only per-request token counts (input, cached input, and output) and the resulting cache-hit share. These records are used for billing and for resolving billing disputes, in the same way as input and output token counts today. They don’t contain prompt fragments, token IDs, prefix hashes, cache keys, or session identifiers. Cached tokens don’t introduce new data collection. For details on how Nebius Token Factory handles your data, see the Privacy policy and your agreement with Nebius Token Factory.

OpenRouter and other routers

Requests that arrive through OpenRouter are served by Nebius Token Factory as public endpoints traffic, so the same cache scope applies. There is no separate cache isolation for each downstream OpenRouter customer.

Limitations

Cached input pricing applies to Serverless text-generation requests: Chat Completions, Completions, and the Responses API.
Embeddings, rerank, and image APIs don’t report or bill cached tokens.
Batch inference is not covered.
You can’t configure cache keys, cache lifetime (TTL), or explicitly cache a prompt.

Get Started

Inference

Observability

Post-training

Data Lab

Sandboxes

Teams & Access Management

Other Capabilities

Integrations

Platform Notices

How it works

Billing

Pricing

Check cache usage in API responses

Chat Completions and Completions

Responses API

Streaming

Check a charge

Get more cache hits

Best-effort caching and no guarantees

Data, privacy, and cache scope

What is cached

Zero data retention (ZDR)

What Nebius Token Factory records

OpenRouter and other routers

Limitations

​How it works

​Billing

​Pricing

​Check cache usage in API responses

​Chat Completions and Completions

​Responses API

​Streaming

​Check a charge

​Get more cache hits

​Best-effort caching and no guarantees

​Data, privacy, and cache scope

​What is cached

​Zero data retention (ZDR)

​What Nebius Token Factory records

​OpenRouter and other routers

​Limitations

How it works

Billing

Pricing

Check cache usage in API responses

Chat Completions and Completions

Responses API

Streaming

Check a charge

Get more cache hits

Best-effort caching and no guarantees

Data, privacy, and cache scope

What is cached

Zero data retention (ZDR)

What Nebius Token Factory records

OpenRouter and other routers

Limitations