Response caching
Response caching stores an entire completion and replays it verbatim the next time an identical request comes in — no provider call, no tokens billed, no latency beyond a cache lookup. It's opt-in, configured per Mizan API key, and separate from Prompt Caching, which still makes a real (discounted) call to the provider.
What it is
When response caching is enabled for a key, every incoming request — streaming or not — is hashed into a cache key — the model, the full request body, and the streaming flag, canonicalized so key order in the JSON body doesn't change the hash. If an earlier request with the same structural content was served successfully within the configured TTL, Mizan replays that stored response directly. The provider is never called, so the request costs nothing and returns as fast as Mizan can look up and return the cached payload — for a streaming request, the exact same chunks are replayed back in one burst instead of being generated live.
vs. Prompt Caching
Prompt Caching caches a stable prefix — a system prompt, tool definitions — at the provider, so the provider still generates a fresh completion but bills the reused portion at a discount. Response caching is a different mechanism entirely: it caches the entire response to an entire request, at Mizan's layer, before the provider is ever reached. A cache hit here means zero provider calls and zero cost — not a cheaper call, no call at all. The two are independent and can both be active on the same key at once.
Enabling it
Response caching is off by default and configured per key, not per request — there is no request-level header to turn it on. Enable it and set a TTL (10 seconds to 30 days, default 5 minutes) from the Response Caching settings page, per Mizan API key. Once enabled, caching applies automatically to every eligible request made with that key — nothing to change in your application code.
The determinism trade-off
Enabling response caching means an identical repeat request returns the exact same cached response until that cache entry expires — this holds even at temperature > 0, where you'd normally expect a different sampled output on every call. This is expected, correct behavior for a response cache, not a bug — but it's a real footgun if your app relies on repeat calls actually varying.
For example: a "regenerate" button that resends the same prompt at temperature: 0.9 hoping for a different answer, or a best-of-N sampling loop that fires the same request several times to pick the best result — both will silently get the same response back on every call after the first, for as long as the TTL holds. If your workload depends on varied output from identical inputs, either keep response caching off for that key, or vary something in the request (e.g. a no-op field) so it hashes differently — or send x-mizan-cache-bypass: true on the calls that need a fresh sample.
v1 scope & limits
Response caching currently covers a deliberately narrow slice of traffic:
- Streaming is cached too. A request sent with
stream: truegets the same treatment as a regular call — a hit replays the exact same chunks back in one burst instead of generating them live. One nuance: a streaming cache entry is scoped to the endpoint that created it (/v1/chat/completionsvs./v1/messages), since the stored bytes are already formatted for that endpoint's SSE shape — an identical request sent through the other endpoint writes its own separate entry. Non-streaming cache entries don't have this restriction; they're reformatted fresh for whichever endpoint asks. - TTL bounds. Configurable from 10 seconds to 30 days, default 5 minutes.
- 256KB size cap. Request or response bodies larger than 256KB are never cached — this guards against unbounded cache growth from large multimodal payloads (images, audio, documents). Oversized calls still work, they just always go to the provider.
Cache headers
Every response from a key with caching enabled carries an x-mizan-cache header so you can tell what happened without inspecting cost or latency:
| Header value | Meaning |
|---|---|
| HIT | Served from cache. No provider call was made; the call cost nothing. |
| MISS | Caching is enabled for this key, but no matching entry was found. The request went to the provider as normal and the response was cached for next time. |
| (absent) | Response caching isn't enabled or doesn't apply to this call — e.g. caching is off for the key, or the request body is over the 256KB cap. |
To force a fresh provider call for one request without turning caching off entirely, send x-mizan-cache-bypass: true. It skips the cache lookup for that call only — the fresh response is still written back to cache normally, so subsequent identical requests can hit it.
Cache scoping
Cache entries are strictly scoped to the individual Mizan API key that created them. They are never shared across different keys — not even between keys on the same project, account, or organization. Unlike Prompt Caching, which is a server-side timing optimization on the provider side, a response-cache entry is a full response body — real content — so it is never pooled or shared cross-tenant. Two keys sending byte-identical requests will each build and read their own independent cache.
BYOK and caching
Response caching works for BYOK keys too — a cache hit still skips the call to your own provider account, saving your own provider bill, even though Mizan never billed you for either the original call or the cached replay. This is a deliberate contrast with Prompt Caching: that feature excludes BYOK calls from its Analytics savings tracking, since the discount happens entirely inside your provider bill and off Mizan's ledger. Response-cache savings tracking does include BYOK hits, because Mizan itself performed the cache lookup and can see the avoided call either way.
Savings
The running total saved by response caching — plus how many requests hit vs. missed — shows up under Analytics, in the "Response cache savings" section, alongside Prompt Caching's "Caching savings" section.
Best-fit use cases
Response caching pays off most on workloads where exact-repeat calls are expected and an identical, idempotent output is actually what you want:
- Agent or workflow retries — re-running a failed step from the top without re-paying for the steps that already succeeded and would produce the same output again.
- Eval and test suites that call the same prompts repeatedly across runs.
- Any workload where the same input should reliably produce the same output, and repeat calls are a matter of when, not if.