Prompt caching in OpenAI, Anthropic, and Gemini: architecture and trade-offs

Prompt caching reuses computation for an identical beginning of a request: system instructions, tool schemas, few-shot examples, or a large shared context. It is not a semantic cache of completed answers. The model still generates a new result for the variable part, so caching must not replace checks for quality, freshness, or permissions.

A useful design has three zones: a long-lived shared prefix, versioned team or tenant context, and a dynamic tail with the user request and fresh data. Put stable material earlier and variable material later. A timestamp, random ID, unstable JSON-field order, or personal data near the beginning can break the match and turn expected savings into repeated cache writes.

• Cache only a reusable prefix that is large enough for the rules of the selected model.
• Version instructions, tools, schemas, and corpus revisions instead of mutating them invisibly.
• Measure reads, writes, misses, uncached input, latency, and quality separately.
• Do not mix tenants or access levels to improve hit rate.
• Verify current model, region, retention, and pricing rules before rollout.

OpenAI documents automatic caching for eligible exact prefixes and exposes cached-token telemetry; newer models also offer cache keys, cache points, and mode controls. Anthropic supports top-level automatic caching and explicit cache breakpoints through cache_control, with separate cache-creation and cache-read token accounting. Gemini provides implicit caching and, in the compatible Generate Content API, explicit cached content with managed TTL; availability depends on API and model.

Do not hide these differences behind a generic cache=true flag. A provider adapter should describe capabilities: implicit or explicit mode, minimum prefix, allowed breakpoints, TTL, write/read billing, telemetry, region, and data-retention constraints. If a capability is unknown, run a normal request and record cache status as unsupported or unknown rather than inventing a hit.

Assemble the request deterministically: immutable policy and system instructions, stably sorted tool definitions, schemas, validated examples, then shared documents, and only afterward user-specific input. Serialization should be byte-stable within the provider contract. Even semantically identical JSON with a different field order may fail to match as an exact prefix.

Derive cache identity from provider, model family, prompt version, tool-schema version, policy version, locale, tenant or access scope, and corpus revision. Do not put secret text into logs or cache keys; use an opaque digest of controlled identifiers. A change in model, permissions, system policy, or a source that requires immediate revocation creates a new version; old cache traffic stops and the object expires naturally or is deleted through an available API.

Model one logical cohort: cache-write tokens and storage, cache-read tokens, uncached input, output, number of reuses, time to first token, and cost per successful task. Break-even depends on the exact model pricing, TTL, and observed reuse count. Do not transfer the discount percentage from one provider or model to another contract, and do not assume a hit merely because the text looks identical.

A low hit rate often indicates a poor workload shape rather than a weak service: short prompts, rare repetition, fragmented cache keys, frequent changes near the prefix, or a parallel burst before the first write completes. Compare controlled variants on the same traffic slice. If canonicalization adds more complexity, write spend, or revocation delay than it saves, a normal uncached request is the better design.

Cache reuse does not authorize weaker access control. A shared prefix may contain only data allowed for every request within its scope. Tenant-specific documents, personal data, and tool results with different ACLs require separate identities or must remain after the safe boundary. A cache key is a routing hint or resource identifier, not an authorization mechanism; the server still has to authorize the request and construct the allowed context.

Explicit caching can create persistent application state for the TTL. Verify data residency, zero-data-retention compatibility, encryption, deletion, and incident-response behavior in current documentation and contracts. For legal hold or urgent revocation, document the path: stop new reads, change version or scope, delete the cache object where supported, and verify that later traces no longer use the retired revision.

A trace should record provider, model, cache mode, a safe prefix fingerprint, version, breakpoint, requested TTL, read/write/uncached token counts, latency, outcome, and miss reason without storing the raw private prompt. Normalize provider fields into shared categories, but retain the original usage payload in a protected audit layer for billing-semantics verification.

The dashboard should show eligible requests, hit rate among eligible requests, cached-token share, write amplification, cost per accepted outcome, and latency for hits and misses. Alert on a sharp drop after deploy, an unexpected cross-scope fingerprint, writes with no subsequent reads, or use of a retired version. Hit rate alone is not a quality KPI: an unchanged but wrong system prompt can also be cached perfectly.

First measure prefix repetition without changing behavior. Then enable canonical rendering and compare exact fingerprints, while excluding sensitive cohorts from caching. The canary should cover one provider/model and a low-risk workload; a quality eval must confirm that reordering blocks did not alter instruction precedence, tool behavior, or groundedness.

The release gate should require zero cross-tenant defects, correct usage attribution, acceptable write amplification, and non-regression in task quality. Rollback disables explicit breakpoints or cache-resource references, restores the previous renderer, and routes traffic to normal requests. Old cache objects are not considered deleted without provider evidence; their expiry or deletion is tracked separately from application rollback.