The last lesson let you manage prompts safely. But production has a real problem: an app may fetch a prompt before every single LLM call ("give me the production one"). Hitting Postgres each time is slow and crushes the database. This lesson is about how PromptService uses Redis to make prompt serving fast and steady, focusing on two mechanisms: the classic read-through cache, and an exceptionally elegant "swap the namespace, don't delete keys one by one" epoch invalidation strategy.
Along the way we'll see Lesson 37's dependencies' "second half": referenced child prompts are resolved and inlined into one complete prompt at serving time—and it's precisely this fact, "one prompt may span multiple names", that forces the clever invalidation design.
📋 Analogy
Prompt serving is like a library's popular-books fast-access shelf. A reader (the app) comes to borrow a book (prompt); the librarian first checks the fast shelf (Redis): if it's there, hand it over instantly; if not, run to the main stacks (Postgres) to fetch one and drop a copy on the fast shelf, so the next person is quick.
Then the hard part: a book gets a new edition—how do you stop everyone from still borrowing the old one? Combing the whole shelf to pull old editions one by one is too hard—besides, one book may have chapters of other books tucked inside (Lesson 37's dependencies), so one change ripples everywhere. The library's trick: hang a new "edition tag" on the entire fast shelf. Books under the old tag are no longer recognized (the rule only honors the current tag); they needn't be cleaned, just left to expire; and every new borrow goes through the shelf with the new tag. Swapping a tag is far faster and steadier than pulling books one by one.
The read-through cache: ask Redis first, fall back to the DB
The core method is PromptService.getPrompt, a textbook read-through cache: check Redis first, return on a hit; only on a miss go back to Postgres (findPrompt), and after fetching, write it into Redis (with a TTL), so the next identical request hits the cache. Whether the cache is on is controlled by LANGFUSE_CACHE_PROMPT_ENABLED, and how long each entry lives by LANGFUSE_CACHE_PROMPT_TTL_SECONDS.
read-through + backfill: getPrompt first getCachedPrompt, returns on a hit; on a miss runs findPrompt (DB) then cachePrompt (redis.set(key, value, "EX", ttlSeconds)). Toggle LANGFUSE_CACHE_PROMPT_ENABLED, duration LANGFUSE_CACHE_PROMPT_TTL_SECONDS. Source: PromptService/index.ts:49-86,166-173.
The cache turns "hit the DB every time" into "almost never" (latency/hit-rate illustrative; mechanism from PromptService): a hit returns from Redis in about 1ms, only a miss queries Postgres and backfills at about 25ms — roughly 25x. A high steady-state hit rate (≈98% here) means the vast majority of prompt fetches never touch the database. Two safeguards: TTL 3600s as a backstop, and rotating the project epoch on a prompt change to invalidate old keys instantly (the concrete payoff of the epoch figure above).
public asyncgetPrompt(params): Promise<PromptResult | null> {
if (this.cacheEnabled) {
const cachedPrompt = await this.getCachedPrompt(params); // ① ask Redis firstif (cachedPrompt) return cachedPrompt; // hit → return instantly, no DB
}
const dbPrompt = await this.findPrompt(params); // ② miss → query Postgresif (this.cacheEnabled && dbPrompt)
await this.cachePrompt({ ...params, prompt: dbPrompt }); // ③ backfill Redis (with TTL)return dbPrompt;
}
// cachePrompt: redis.set(key, value, "EX", this.ttlSeconds) — every entry self-expires
case
which path
latency
touches Postgres?
cache hit (vast majority)
Redis returns directly
milliseconds
no
cache miss (cold/just invalidated)
query DB → backfill Redis → return
one DB round-trip
yes (only this one)
TTL expiry
next read becomes a miss, auto-backfills
same, single request
yes (single)
In a phrase: the database only bears "the first request", then Redis catches them all. That's the value of read-through—amortizing a "read on every call" high-frequency access into "one DB read per TTL period".
Resolution: inline dependencies into one complete prompt
Lesson 37 said prompts can reference other prompts. But what the app ultimately gets must be a complete, self-contained prompt—the runtime can't be left to assemble the referenced snippets on the fly. So serving includes a resolve step: buildAndResolvePromptGraph walks the dependency graph, finds each referenced child, and inlines it into the parent, producing an "expanded" finished product. Two safety gates are essential: a nesting depth cap (MAX_PROMPT_NESTING_DEPTH) and cycle detection (a seen set), preventing A-depends-on-B-depends-on-A from hanging the service.
resolution = expand composition into a product: buildAndResolvePromptGraph walks Lesson 37's dependency graph, inlining children into the parent to get resolvedPrompt; MAX_PROMPT_NESTING_DEPTH caps depth, a seen set blocks cycles. What's cached and returned is this expanded product. Source: PromptService/index.ts:132-144,236+.
packages/shared/src/server/services/PromptService/index.tsresolve + two gates
// resolve: return the product with dependencies inlinedpublic asyncresolvePrompt(prompt) {
const promptGraph = await this.buildAndResolvePromptGraph(...);
return { ...prompt, prompt: promptGraph.resolvedPrompt }; // children inlined
}
// recursively inline along the graph, two safety gates:if (level >= MAX_PROMPT_NESTING_DEPTH) throw …; // gate1: stop if nesting too deepconst seen = new Set<string>(); // gate2: track visited, catch A→B→A instantly
why resolve at serving time
Dependencies are stored in the DB as "reference relations" (pointing to a child by label/version). But a child's label moves and its content changes—so only at the moment of fetching can you pull the current child and inline it, getting the product correct as of now.
gate 1: depth cap
MAX_PROMPT_NESTING_DEPTH limits dependency nesting levels. It prevents someone from making the chain so deep that one resolution pulls dozens of prompts and drags down the service.
gate 2: cycle detection
The seen set records visited prompts. The instant a cycle appears—A depends on B, B depends on A—it's caught and errors out; otherwise the recursion spins forever.
Epoch invalidation: swap the namespace, don't delete keys one by one
The hardest part of caching is always invalidation: when a prompt changes (new version, label move), the old cache must be voided. The difficulty—a resolved prompt may span several names (via dependencies), so you can't even say "which keys to delete". Langfuse's solution is elegant to the point of slyness: embed a project-scoped epoch token in the cache key; to invalidate, delete no keys, just rotate that epoch token to a new random value—so all old keys are instantly "orphaned" (new reads/writes use the new epoch in a new namespace), and the old ones simply expire by TTL.
O(1) invalidation, no scan-and-delete: invalidateCache just SETs a new random token (48 bits of entropy) on the project-scoped epoch key. Keys under the old epoch are no longer hit by any read and die off by their own TTL; the epoch key itself has a 7-day TTL. getOrCreateEpoch uses SET NX for concurrent initialization. Source: PromptService/index.ts:179-260.
// invalidate: delete no keys, just rotate the project-scoped epoch token → all old keys orphanedpublic asyncinvalidateCache({ projectId }) {
await this.redis.set(this.getEpochKey({ projectId }), this.newEpochToken(),
"EX", this.epochTtlSeconds); // epoch key lives 7 days
}
// embed epoch in the key: change the epoch and the whole namespace turns the page
getCacheKey = `prompt:${projectId}:${epoch}:${promptName}:${version ?? label}`;
// epoch project-scoped (not prompt-scoped): a resolved prompt spans names, so any change invalidates all
getEpochKey = `prompt_cache_epoch:${projectId}`;
This design turns "invalidating a tangle of related caches" from an intractable precise-deletion problem into an O(1) token swap. The only cost is old keys briefly occupying a little memory until they expire—trading a touch of wasted memory for invalidation logic that is maximally simple and correct. It's a distributed-caching move well worth adding to your toolbox.
comparison
delete keys one by one (naive)
rotate epoch (Langfuse)
invalidation complexity
find all affected keys, O(N) with a scan
SET one new token, O(1)
dependency tracking
must reverse-lookup "which resolved products contain this child"
none needed—the whole namespace turns the page
miss risk
one miss serves a stale prompt
impossible to miss, all old keys orphaned
cost
complex, error-prone logic
old keys hold a little memory until expiry (auto-die)
🎯 Design trade-off
Why invalidate by "rotating the epoch" rather than dutifully deleting the old keys? Because precise deletion is nearly infeasible here. A served prompt is a resolved product that may inline several other prompts (Lesson 37 dependencies). When you change any one child, all resolved products that indirectly depend on it should be invalidated—but you can barely reverse-lookup "which keys are affected", let alone the fact that they're scattered across different names. "Find them one by one and delete" is slow and miss-prone, and one miss means someone reads a stale prompt. Rotating the epoch cuts the knot: don't search, don't delete, just swap to a fresh namespace; the old ones are all voided and expire naturally. Invalidation is now O(1) and never misses. The cost is a little wasted memory before old keys expire—a textbook "trade space for correctness and simplicity".
Why is the epoch "project-scoped" rather than "prompt-scoped"? A direct corollary of the above. If the epoch were isolated per prompt name, then on changing a child prompt you'd again have to figure out "which parents depend on it, and should I rotate their epochs too"—the dependency-tracking problem reborn. Putting the epoch at project granularity means "any prompt change in this project turns the page on the whole project's prompt cache". A bit coarse—occasionally invalidating caches that could have been kept—but it buys the certainty of never needing to track dependencies at all. Between "invalidate a little less" and "never read stale data", it firmly chooses the latter—exactly right for prompts, where correctness matters far more than cache hit rate.
🎯 Key points
Prompt serving must be fast: production may fetch a prompt on every LLM call; hitting Postgres directly is too slow and heavy, so PromptService puts a Redis cache in front.
Read-through cache: getPrompt checks Redis first, returns on a hit; on a miss falls back to Postgres then backfills Redis (SET EX ttl). Toggle LANGFUSE_CACHE_PROMPT_ENABLED, duration ..._TTL_SECONDS.
Resolve dependencies at serving time: buildAndResolvePromptGraph inlines referenced children into a self-contained product, with a depth cap + seen cycle check; what's cached/returned is this resolved product.
Epoch invalidation (the gem): the cache key embeds a project-scoped epoch token; to invalidate, delete no keys, just rotate the token to a new random value—all old keys orphaned, expiring by TTL. O(1), never misses, no scan.
Why epoch is project-scoped: a resolved prompt spans names, so precise deletion / prompt-scoped isolation would require dependency tracking; project-scoped epoch trades "a little over-invalidation" for "no tracking + never stale"—the right call for correctness-first data like prompts.
🧪 Self-test - think about the design
1. Langfuse's PromptService uses Redis as a read-through cache. What flow does 'read-through' specifically mean?
queries Postgres every time, then updates Redis on the side
getPrompt checks Redis first, returns on a hit (no DB); only on a miss queries Postgres and backfills the result into Redis (SET EX ttl), so the next identical request hits the cache—the vast majority are millisecond and don't load the DB
writes Redis first, then async to the DB
reads only Redis, never the DB
Show answer & explanation click to expand
Answer: B. Read-through: a hit takes the shortcut, only a miss penetrates to the authoritative DB and backfills. getPrompt first getCachedPrompt, returns on hit; on miss findPrompt(DB)→cachePrompt(redis.set EX ttlSeconds). Production may fetch a prompt every LLM call; this cache shields the DB, with only the first request after a cold/invalidated cache hitting the DB. Toggle LANGFUSE_CACHE_PROMPT_ENABLED, duration _TTL_SECONDS.
2. After a prompt change the cache must be invalidated. Langfuse doesn't delete old keys one by one but embeds a project-scoped epoch token in the cache key, rotating it to a new random value on invalidation. Why?
because a resolved prompt may inline several other prompts (dependencies), precisely reverse-looking-up 'which keys are affected' is nearly infeasible and one miss serves stale data; rotating the epoch swaps to a fresh namespace, old keys orphaned and expiring by TTL—O(1), never misses
to save Redis memory
because Redis doesn't support DEL
to make the cache never expire
Show answer & explanation click to expand
Answer: A. A served prompt is a resolved product that may inline several children (L37). Changing one child should invalidate all resolved products indirectly depending on it, but you can barely reverse-lookup the affected keys, scattered across names—deleting one by one is slow and miss-prone. invalidateCache just SETs a new token on the project-scoped epoch key; cache keys embed that epoch, so old keys are all orphaned and expire by TTL. Trade a little wasted memory for simple, correct invalidation.
3. Why is Langfuse's cache epoch 'project-scoped' (prompt_cache_epoch:{projectId}) rather than isolated per prompt name?
prompt-scoped epochs aren't supported by Redis
because a resolved prompt spans multiple names (dependencies); if the epoch were per prompt name, changing a child would again require tracking 'which parents depend on it'; a project-scoped epoch trades 'occasionally over-invalidating' for 'no dependency tracking at all + never stale'
to share cache across projects
project-scoped keys are shorter, saving memory
Show answer & explanation click to expand
Answer: B. The source comment states: the epoch is project-scoped because a resolved prompt can include transitive dependencies across multiple prompt names. Per-name isolation would revive the 'which parents are affected' dependency-tracking problem on a child change. Project granularity means 'any prompt change in the project turns the whole prompt cache's page'—coarser, occasionally over-invalidating, but buying certainty of no dependency tracking. Right for prompts, where correctness beats hit rate.
💭 Open questions (no single right answer - just think or try)
The epoch-invalidation move—'change the namespace, not delete keys'—is essentially 'trade space for correctness and simplicity'. Recall caching systems you've built or imagined: which invalidation scenarios gave you headaches (cascading invalidation, dependency tracking, thundering herd)? Could this epoch/version-namespace idea apply? How would you weigh its costs (memory, TTL choice, cold-start thundering herd)?