Evaluating Token Caching Strategies to Cut OpenAI and Claude API Bills

When building an AI micro-SaaS, LLM API costs are your largest operational expense.

If your product processes long context windows—such as analyzing 40-page financial PDFs, querying large technical codebases, or executing multi-turn conversational agents—your input token consumption can quickly spiral out of control.

Every time a user asks a simple follow-up question, your backend resends the entire 30,000-token document back to the API. At standard Claude 3.5 Sonnet or GPT-4o input rates, that single document resend costs real money.

In late 2024 and 2025, both Anthropic and OpenAI introduced Prompt Caching.

Prompt caching allows you to store frequently used context on the provider’s servers. On subsequent calls, cached tokens are processed at an up to 90% discount with dramatically lower latency.

Here is how prompt caching works and how to structure your prompts to maximize caching hit rates.

How Prompt Caching Works Under the Hood

When you send a prompt to an LLM provider:

1. The model must compute the mathematical attention states for every single input token (KV Cache computation).
2. With Prompt Caching, the provider saves these computed attention states in memory for a short window (typically 5 to 60 minutes).
3. If a subsequent API request arrives containing the identical token prefix, the provider skips recalculating the attention states and simply reads the cached memory.

Look at the dramatic price difference on Anthropic’s Claude 3.5 Sonnet:

• Standard Input Tokens: $3.00 per 1M tokens
• Cache Write Tokens: $3.75 per 1M tokens (first call)
• Cache Read Tokens: $0.30 per 1M tokens (90% discount!)

If your application makes 10 requests against the same 20,000-token document, prompt caching slashes your input cost on calls 2 through 10 from $0.60 down to $0.06.

The Golden Rule of Cache Invalidation: Put Static Content First

Prompt caching uses prefix matching. The cache is valid only up to the very first token that changes.

If you make the mistake of placing dynamic, changing data at the beginning of your prompt:

<!-- BROKEN CACHING: Dynamic data breaks the cache immediately -->
Current Timestamp: 2025-10-14 14:22:05
User ID: 94821
System Instructions: [5,000 tokens of static rules]
Document: [20,000 tokens of reference text]

Because the timestamp and User ID change on every request, the cache breaks on Token 1. Your caching hit rate will be 0%.

To maximize caching hits, arrange your prompts strictly from most static to most dynamic:

<!-- OPTIMAL CACHING: Static prefix is preserved across calls -->
1. System Prompt & Behavioral Instructions (Static)
2. Large Reference Documentation or PDF Content (Static)
3. Historical Conversation Turns (Appended sequentially)
4. Dynamic User Metadata & Current Query (Dynamic - at the very end)

Explicit Cache Breakpoints in Anthropic Claude

Anthropic allows you to set explicit cache control breakpoints using the cache_control parameter in their SDK:

const response = await anthropic.messages.create({
  model: "claude-3-5-sonnet-20241022",
  max_tokens: 1000,
  system: [
    {
      type: "text",
      text: largeStaticDocumentation,
      cache_control: { type: "ephemeral" } // Cache this large block!
    }
  ],
  messages: [
    {
      role: "user",
      content: userQuery // Dynamic query at the end
    }
  ]
});

Monitoring Cache Performance in Production

Always log the cache telemetry returned in the API response headers:

• cache_creation_input_tokens: Tokens written to cache.
• cache_read_input_tokens: Tokens served directly from cache.

If your cache_read_input_tokens is consistently zero, inspect your prompt assembly logic to ensure you aren’t injecting variable whitespace, dynamic dates, or non-deterministic serialization into your static headers.

Related Operational Guides

To optimize your AI architecture and reduce cloud infrastructure bills, review:

• How to build a custom RAG knowledge base without vendor lock-in
• Prompt injection defense for small B2B apps using LLM APIs
• Automating inbound lead qualification using webhooks and small LLMs