Caching

-H "x-vatan-cache: 300"   # allow a five minute cache on this request

The response carries x-vatan-cache: HIT or MISS. A hit costs nothing and is still recorded, so the saving is visible rather than the traffic simply vanishing from your dashboard.

The cache key covers everything that changes the answer, temperature included. An identical request inside the window returns an identical response. Streamed requests are never cached: replaying stored chunks would fabricate timings no model produced.

Your hit rate is measured over requests that asked to be cacheable, not over all traffic. Dividing by everything would report a rate near zero for a cache working perfectly.

This is the one place the gateway keeps a model's answer. Everywhere else the request log is metadata only, so it is worth being exact about the exception rather than leaving you to infer it from the fact that a cache obviously has to hold something.

The answer is stored in full, because serving it back is the entire point. The request is stored only as a SHA-256 of your organisation, the model and the normalised body, so the prompt itself is never written down: a hash is how a later request is recognised as the same one, and it cannot be read back into the text that produced it.

It lives for exactly the number of seconds you asked for and is unreadable after that. The cache page in the dashboard shows what is held and purges all of it on one button. Purging is all-or-nothing on purpose: choosing individual entries would mean showing you a list of prompts, which is precisely what not storing them is meant to avoid.

Returning a stored answer to somebody expecting a fresh one is a correctness problem dressed as a saving, and there is no way for them to tell it happened. You know which of your requests can tolerate that; we do not.