🪢 Langfuse 图解教程Langfuse Visual Guide 第七部分 · Prompt 与 PlaygroundPart 7 · Prompts & Playground 38 / 55
第七部分 · Prompt 与 PlaygroundPart 7 · Prompts & Playground

Prompt 服务与缓存Prompt serving & caching

🧭 读前Before you start
先读 第37课 的 prompt 管理,本课讲服务端如何缓存与下发 prompt。
Read Lesson 37 on prompt management; this lesson covers how the server caches and serves prompts.

上一课让你能安全地管 prompt。但生产里有个现实问题:应用可能每一次 LLM 调用都要先取一次 prompt(「给我 production 那版」)。如果每次都去 Postgres 查一遍,既慢、又把数据库压垮。这一课讲 PromptService 怎么用 Redis 把 prompt 服务做得又快又稳,重点是两个机制:经典的 read-through 缓存,和一个极其优雅的「换命名空间,而非逐个删 key」的 epoch 失效策略。 中间还会看到第 37 课依赖的「下半场」:被引用的子 prompt 在服务时会被解析内联成一个完整 prompt,而正是这个「一份 prompt 可能横跨多个名字」的事实,逼出了那个聪明的失效设计。

📋 生活类比
prompt 服务像图书馆的热门书快取架。读者(应用)来借书(prompt),管理员先看快取架(Redis)上有没有:有就秒递,没有才跑去大书库(Postgres)取一本、顺手在快取架放一份,下个人就快了。 难题来了:一本书出了新版,怎么让大家别再借到旧版?满架去翻、把旧版一本本抽走太难了——何况一本书里还可能夹着别本书的章节(第 37 课的依赖),牵一发动全身。图书馆的妙招是:给整个快取架挂一块新的「版本号牌」。旧号牌下的书没人再认(按规则只认当前号牌),它们不必清理、放着自然过期就行;而所有新借的,都从挂着新号牌的货架走。换块牌子,比逐本抽书,快太多也稳太多。

read-through 缓存:先问 Redis,没有再回库

核心方法是 PromptService.getPrompt,它实现教科书式的 read-through(读穿透)缓存:先查 Redis,命中就直接返回;未命中才回 Postgres 查(findPrompt),查到后顺手写进 Redis(带 TTL),下一次同样的请求就走缓存了。缓存开不开由 LANGFUSE_CACHE_PROMPT_ENABLED 控制,每条缓存活多久由 LANGFUSE_CACHE_PROMPT_TTL_SECONDS 控制。

read-through:缓存命中走捷径,未命中回库并回填 应用getPrompt(production) PromptService① 先查 Redis命中→返回 Redis 缓存毫秒级 · 带 TTL Postgres(权威)未命中才查 ① 命中 ② 未命中→查库 ③ 回填 Redis 绝大多数请求走 ①(毫秒、不碰库);只有缓存冷/失效后第一笔走 ②③——数据库压力被缓存挡在前面
读穿透 + 回填:getPrompt 先 getCachedPrompt,命中即返回;未命中走 findPrompt(DB)再 cachePrompt(redis.set(key, value, "EX", ttlSeconds))。开关 LANGFUSE_CACHE_PROMPT_ENABLED、时长 LANGFUSE_CACHE_PROMPT_TTL_SECONDS。源码:PromptService/index.ts:49-86,166-173。
示例:命中 vs 未命中的延迟 cache HITRedis ~1 ms · 秒回 cache MISS查 PG+回填 ~25 ms ≈ 25× 更快 稳态命中率 ≈ 98% TTL 3600s(1h,LANGFUSE_CACHE_PROMPT_TTL_SECONDS)· 改 prompt → 轮换项目级 epoch → 旧 key 瞬间失效,无需逐个删
缓存把「每次都查库」变成「几乎不查库」(延迟/命中率为示例,机制取自 PromptService):命中走 Redis 约 1ms 秒回,未命中才查 Postgres 再回填、约 25ms——相差约 25 倍。稳态命中率高(图中 ≈98%),意味着绝大多数取 prompt 的请求都不碰数据库。两道保险:TTL 3600s 兜底过期,改 prompt 时轮换项目级 epoch让旧 key 瞬间失效(上面 epoch 图的实际收益)。
packages/shared/src/server/services/PromptService/index.tsgetPrompt 读穿透
public async getPrompt(params): Promise<PromptResult | null> {
  if (this.cacheEnabled) {
    const cachedPrompt = await this.getCachedPrompt(params);   // ① 先问 Redis
    if (cachedPrompt) return cachedPrompt;                     // 命中 → 秒返回,不碰库
  }
  const dbPrompt = await this.findPrompt(params);              // ② 未命中 → 查 Postgres
  if (this.cacheEnabled && dbPrompt)
    await this.cachePrompt({ ...params, prompt: dbPrompt });    // ③ 回填 Redis(带 TTL)
  return dbPrompt;
}
// cachePrompt: redis.set(key, value, "EX", this.ttlSeconds)  —— 每条缓存自带过期
情形走哪条路延迟碰 Postgres 吗
缓存命中(绝大多数)Redis 直接返回毫秒级否
缓存未命中(冷启/刚失效)查 DB → 回填 Redis → 返回一次 DB 往返是(仅这一笔)
TTL 到期下次读变未命中,自动回填同上,单笔是(单笔)

一句话:数据库只在「第一笔」承压,之后全被 Redis 接住。这正是 read-through 的价值——把一个「每次调用都要读」的高频访问,摊薄成「每个 TTL 周期一次 DB 读」。

解析:把依赖内联成一个完整 prompt

第 37 课说 prompt 能引用别的 prompt。但应用最终拿到的,得是一个完整、自包含的 prompt——不能让运行时还去现凑那些被引用的片段。所以服务时有一步解析(resolve):buildAndResolvePromptGraph 沿着依赖图把每个被引用的子 prompt 找出来、内联进父 prompt,得到一个「展开后」的成品。两道安全闸不可少:嵌套深度上限(MAX_PROMPT_NESTING_DEPTH)和环检测(seen 集合),防止 A 依赖 B、B 又依赖 A 把服务拖死。

解析:沿依赖图内联,产出一个自包含的成品 父 prompt引用 子A(label)引用 子B(version) buildAndResolvePromptGraph沿图找子 prompt 内联闸1 深度上限闸2 seen 环检测 resolved prompt子片段已全部内联完整、自包含← 缓存/返回的就是它 关键事实:一个 resolved prompt 可能横跨好几个 prompt 名字——这正是下一节失效设计要项目级的根因
解析 = 把组合展开成成品:buildAndResolvePromptGraph 沿第 37 课的依赖图把子 prompt 内联进父,得到 resolvedPrompt;MAX_PROMPT_NESTING_DEPTH 限深、seen 集合防环。缓存与返回的都是这个展开后的成品。源码:PromptService/index.ts:132-144,236+。
packages/shared/src/server/services/PromptService/index.ts解析 + 两道闸
// 解析:返回把依赖内联后的成品
public async resolvePrompt(prompt) {
  const promptGraph = await this.buildAndResolvePromptGraph(...);
  return { ...prompt, prompt: promptGraph.resolvedPrompt };   // 子片段已内联
}
// 沿依赖图递归内联,两道安全闸:
if (level >= MAX_PROMPT_NESTING_DEPTH) throw …;   // 闸1:嵌套太深就停
const seen = new Set<string>();                    // 闸2:记录已访问,A→B→A 立刻识破

为什么服务时才解析

依赖在数据库里存的是「引用关系」(按 label/version 指向 child)。但 child 的 label 会移动、内容会变——所以只能在取用的那一刻才把当前的 child 拉出来内联,得到此刻正确的成品。

闸 1:深度上限

MAX_PROMPT_NESTING_DEPTH 限制依赖嵌套层数。防止有人把依赖链拉得过深,导致一次解析要拉几十个 prompt、拖垮服务。

闸 2:环检测

seen 集合记录已访问的 prompt。一旦出现 A 依赖 B、B 又依赖 A 的环,立刻识破并报错——否则递归会无限打转。

epoch 失效:换命名空间,而非逐个删 key

缓存最难的永远是失效:prompt 一改(新版、移 label),旧缓存就得作废。难点在于——一个 resolved prompt 可能横跨好几个名字(因为依赖),你根本说不清「该删哪些 key」。Langfuse 的解法漂亮得近乎狡黠:缓存 key 里嵌一个项目级的 epoch(纪元)令牌;失效时不删任何 key,只是把这个 epoch 令牌换成一个新随机值——于是所有旧 key 瞬间「失联」(新读写都用新 epoch、落在新命名空间),旧的放着自然按 TTL 过期。

失效 = 转 epoch 令牌(换命名空间),不删 key 缓存 key 形如:prompt : {projectId} : {epoch} : {promptName} : {version|label} 旧 epoch = e1(失联)prompt:proj:e1:greeting:productionprompt:proj:e1:refund:production没人再读 → 按 TTL 自然过期 新 epoch = e2(现役)prompt:proj:e2:greeting:production…新读写都落这里(先 miss 再回填)相当于一整片干净缓存 转 epoch epoch 是项目级的(不是 prompt 级)——因为 resolved prompt 跨多名字,任一变更都得让整片缓存翻篇;O(1),无需扫描/删除
O(1) 失效,无需扫库删 key:invalidateCache 只对项目级 epoch 键 SET 一个新随机令牌(48 位熵)。旧 epoch 的 key 不再被任何读命中,靠各自 TTL 自然消亡;epoch 键本身 TTL 7 天。getOrCreateEpoch 用 SET NX 处理并发初始化。源码:PromptService/index.ts:179-260。
packages/shared/src/server/services/PromptService/index.tsepoch 失效 + key
// 失效:不删 key,只把项目级 epoch 令牌转成新随机值 → 全部旧 key 失联
public async invalidateCache({ projectId }) {
  await this.redis.set(this.getEpochKey({ projectId }), this.newEpochToken(),
                       "EX", this.epochTtlSeconds);   // epoch 键活 7 天
}
// key 里嵌 epoch:epoch 一变,整片命名空间就翻篇
getCacheKey = `prompt:${projectId}:${epoch}:${promptName}:${version ?? label}`;
// epoch 项目级(非 prompt 级):因为 resolved prompt 跨多名字,任一变更都要整体失效
getEpochKey = `prompt_cache_epoch:${projectId}`;

这套设计把「失效一片关联复杂的缓存」从一个难解的精确删除问题,变成了一次O(1) 的令牌替换。代价仅仅是旧 key 在过期前还短暂占着点内存——用一点点内存的浪费,换来失效逻辑的极致简单与正确。这是分布式缓存里一个非常值得收进工具箱的招式。

对比逐个删 key(朴素做法)转 epoch(Langfuse)
失效复杂度需找出所有受影响 key,O(N) 且要扫描SET 一个新令牌,O(1)
依赖追踪必须反查「哪些 resolved 成品含此 child」完全不必——整片命名空间翻篇
漏删风险漏一个就有人读到陈旧 prompt不可能漏,旧 key 一律失联
代价逻辑复杂、易错旧 key 过期前占点内存(自动消亡)
🎯 设计取舍
为什么用「转 epoch」失效,而不是老老实实把旧 key 删掉? 因为精确删除在这里几乎不可行。一个被服务的 prompt 是解析后的成品,可能内联了好几个别的 prompt(第 37 课依赖)。当你改动其中任意一个 child,所有间接依赖它的 resolved 成品都该失效——但你很难反查「到底哪些 key 受了牵连」,更别说它们还散落在不同名字下。「逐个找出来删」既慢又容易漏,漏一个就意味着有人读到了陈旧的 prompt。转 epoch 把这个难题一刀斩断:不去找、不去删,直接换一片新命名空间,旧的全部作废、自然过期。失效从此是 O(1) 的、绝不漏的。代价是旧 key 过期前的一点内存浪费——这是教科书级的「用空间换正确性与简单性」。

为什么 epoch 是「项目级」而不是「prompt 级」? 正是上面那个原因的直接推论。如果 epoch 按单个 prompt 名字隔离,那改 child prompt 时,你又得去想「哪些 parent 依赖了它、要不要也转它们的 epoch」——依赖追踪的难题原地复活。把 epoch 放在项目这个粒度,意味着「这个项目里任何一个 prompt 有变动,整个项目的 prompt 缓存就翻篇」。粒度粗了点、偶尔会多失效一些本可保留的缓存,但换来的是彻底不必追踪依赖关系的确定性。在「失效少一点」和「永不读到陈旧数据」之间,它果断选了后者——对 prompt 这种正确性远比缓存命中率重要的数据,这个取舍无比正确。
🎯 本课要点
  • prompt 服务必须快:生产可能每次 LLM 调用都取 prompt,直接打 Postgres 太慢太重,故 PromptService 用 Redis 缓存挡在前面。
  • read-through 缓存:getPrompt 先查 Redis、命中即返回;未命中回 Postgres 再回填 Redis(SET EX ttl)。开关 LANGFUSE_CACHE_PROMPT_ENABLED、时长 ..._TTL_SECONDS。
  • 服务时解析依赖:buildAndResolvePromptGraph 把被引用的子 prompt 内联成自包含成品,带深度上限 + seen 环检测两道闸;缓存/返回的都是这个 resolved 成品。
  • epoch 失效(招式精髓):缓存 key 嵌项目级 epoch 令牌;失效时不删 key,只把令牌转成新随机值——旧 key 全部失联、按 TTL 自然过期。O(1)、绝不漏、无需扫描。
  • epoch 项目级的根因:resolved prompt 跨多名字,精确删除/prompt 级隔离都要追踪依赖;项目级 epoch 用「多失效一点」换「彻底不必追踪 + 永不读到陈旧」——对 prompt 这种正确性优先的数据是正确取舍。

🧪 自测 · 想一想为什么这么设计

1. Langfuse 的 PromptService 用 Redis 做 read-through 缓存。「read-through」具体指什么流程?
  1. 每次都查 Postgres,再顺便更新 Redis
  2. getPrompt 先查 Redis,命中即返回(不碰库);未命中才查 Postgres,并把结果回填进 Redis(SET EX ttl),于是下次同请求走缓存——绝大多数请求毫秒级、不压数据库
  3. 先写 Redis 再异步落库
  4. 只读 Redis,从不回库
看答案与解析 点击展开
答案:B。read-through:缓存命中走捷径,未命中才穿透到权威库并回填。getPrompt 先 getCachedPrompt,命中返回;未命中 findPrompt(DB)→cachePrompt(redis.set EX ttlSeconds)。生产可能每次 LLM 调用都取 prompt,这层缓存把数据库压力挡在前面,只有缓存冷/失效后第一笔才回库。开关 LANGFUSE_CACHE_PROMPT_ENABLED、时长 _TTL_SECONDS。
2. prompt 改动后要让缓存失效。Langfuse 不去逐个删旧 key,而是在缓存 key 里嵌一个项目级 epoch 令牌、失效时只把它转成新随机值。为什么这样做?
  1. 因为 resolved prompt 可能内联多个别的 prompt(依赖),精确反查「哪些 key 受牵连」几乎不可行、漏一个就读到陈旧;转 epoch 等于直接换一片干净命名空间,旧 key 失联按 TTL 过期——O(1)、绝不漏
  2. 为了节省 Redis 内存
  3. 因为 Redis 不支持 DEL
  4. 为了让缓存永不过期
看答案与解析 点击展开
答案:A。服务的 prompt 是解析后的成品,可能内联多个 child(第37课)。改一个 child,所有间接依赖它的 resolved 成品都该失效,但你很难反查受影响的 key、它们还散落在不同名字下,逐个删既慢又漏。invalidateCache 只对项目级 epoch 键 SET 新令牌,缓存 key 含该 epoch,于是旧 key 全部失联、按 TTL 自然过期。用一点内存浪费换失效的简单与正确。
3. 为什么 Langfuse 的缓存 epoch 是「项目级」(prompt_cache_epoch:{projectId}),而不是按单个 prompt 名字隔离?
  1. prompt 级 epoch 不被 Redis 支持
  2. 因为 resolved prompt 跨多个名字(依赖),若按 prompt 名隔离 epoch,改 child 时又得追踪「哪些 parent 依赖它」;项目级 epoch 用「偶尔多失效一点」换「彻底不必追踪依赖 + 永不读到陈旧」
  3. 为了跨项目共享缓存
  4. 项目级 key 更短更省内存
看答案与解析 点击展开
答案:B。源码注释明说:epoch 项目级是因为 resolved prompt 可含跨多个 prompt 名字的传递依赖。若按 prompt 名隔离,改 child 又要回到「哪些 parent 受影响」的依赖追踪难题。项目级粒度意味着「项目内任一 prompt 变动,整片 prompt 缓存翻篇」——粗一点、偶尔多失效,但换来彻底不必追踪依赖的确定性。对 prompt 这种正确性远比命中率重要的数据,这个取舍正确。
💭 发散思考(没有标准答案,动手或动脑想想)
  • epoch 失效这一招——「改命名空间而非删 key」——本质上是「用空间换正确性与简单性」。回想你做过或设想的缓存系统,哪些场景的失效逻辑曾让你头疼(关联失效、依赖追踪、惊群)?这种 epoch/版本号命名空间的思路能不能套上去?它的代价(内存、TTL 选择、冷启动惊群)你会怎么权衡?

The last lesson let you manage prompts safely. But production has a real problem: an app may fetch a prompt before every single LLM call ("give me the production one"). Hitting Postgres each time is slow and crushes the database. This lesson is about how PromptService uses Redis to make prompt serving fast and steady, focusing on two mechanisms: the classic read-through cache, and an exceptionally elegant "swap the namespace, don't delete keys one by one" epoch invalidation strategy. Along the way we'll see Lesson 37's dependencies' "second half": referenced child prompts are resolved and inlined into one complete prompt at serving time—and it's precisely this fact, "one prompt may span multiple names", that forces the clever invalidation design.

📋 Analogy
Prompt serving is like a library's popular-books fast-access shelf. A reader (the app) comes to borrow a book (prompt); the librarian first checks the fast shelf (Redis): if it's there, hand it over instantly; if not, run to the main stacks (Postgres) to fetch one and drop a copy on the fast shelf, so the next person is quick. Then the hard part: a book gets a new edition—how do you stop everyone from still borrowing the old one? Combing the whole shelf to pull old editions one by one is too hard—besides, one book may have chapters of other books tucked inside (Lesson 37's dependencies), so one change ripples everywhere. The library's trick: hang a new "edition tag" on the entire fast shelf. Books under the old tag are no longer recognized (the rule only honors the current tag); they needn't be cleaned, just left to expire; and every new borrow goes through the shelf with the new tag. Swapping a tag is far faster and steadier than pulling books one by one.

The read-through cache: ask Redis first, fall back to the DB

The core method is PromptService.getPrompt, a textbook read-through cache: check Redis first, return on a hit; only on a miss go back to Postgres (findPrompt), and after fetching, write it into Redis (with a TTL), so the next identical request hits the cache. Whether the cache is on is controlled by LANGFUSE_CACHE_PROMPT_ENABLED, and how long each entry lives by LANGFUSE_CACHE_PROMPT_TTL_SECONDS.

read-through: hit takes the shortcut, miss falls back and backfills appgetPrompt(production) PromptService① check Redis firsthit→return Redis cachemillisecond · with TTL Postgres (authority)queried only on a miss ① hit ② miss→DB ③ backfill Redis the vast majority of requests take ① (millisecond, no DB); only the first after a cold/invalidated cache takes ②③—DB load is shielded behind the cache
read-through + backfill: getPrompt first getCachedPrompt, returns on a hit; on a miss runs findPrompt (DB) then cachePrompt (redis.set(key, value, "EX", ttlSeconds)). Toggle LANGFUSE_CACHE_PROMPT_ENABLED, duration LANGFUSE_CACHE_PROMPT_TTL_SECONDS. Source: PromptService/index.ts:49-86,166-173.
Example: cache hit vs miss latency cache HITRedis ~1 ms · instant cache MISSPG+backfill ~25 ms ≈ 25× faster hit rate ≈ 98% TTL 3600s (1h, LANGFUSE_CACHE_PROMPT_TTL_SECONDS) · change a prompt → rotate the project epoch → old keys invalidate instantly, no per-key delete
The cache turns "hit the DB every time" into "almost never" (latency/hit-rate illustrative; mechanism from PromptService): a hit returns from Redis in about 1ms, only a miss queries Postgres and backfills at about 25ms — roughly 25x. A high steady-state hit rate (≈98% here) means the vast majority of prompt fetches never touch the database. Two safeguards: TTL 3600s as a backstop, and rotating the project epoch on a prompt change to invalidate old keys instantly (the concrete payoff of the epoch figure above).
packages/shared/src/server/services/PromptService/index.tsgetPrompt read-through
public async getPrompt(params): Promise<PromptResult | null> {
  if (this.cacheEnabled) {
    const cachedPrompt = await this.getCachedPrompt(params);   // ① ask Redis first
    if (cachedPrompt) return cachedPrompt;                     // hit → return instantly, no DB
  }
  const dbPrompt = await this.findPrompt(params);              // ② miss → query Postgres
  if (this.cacheEnabled && dbPrompt)
    await this.cachePrompt({ ...params, prompt: dbPrompt });    // ③ backfill Redis (with TTL)
  return dbPrompt;
}
// cachePrompt: redis.set(key, value, "EX", this.ttlSeconds)  — every entry self-expires
casewhich pathlatencytouches Postgres?
cache hit (vast majority)Redis returns directlymillisecondsno
cache miss (cold/just invalidated)query DB → backfill Redis → returnone DB round-tripyes (only this one)
TTL expirynext read becomes a miss, auto-backfillssame, single requestyes (single)

In a phrase: the database only bears "the first request", then Redis catches them all. That's the value of read-through—amortizing a "read on every call" high-frequency access into "one DB read per TTL period".

Resolution: inline dependencies into one complete prompt

Lesson 37 said prompts can reference other prompts. But what the app ultimately gets must be a complete, self-contained prompt—the runtime can't be left to assemble the referenced snippets on the fly. So serving includes a resolve step: buildAndResolvePromptGraph walks the dependency graph, finds each referenced child, and inlines it into the parent, producing an "expanded" finished product. Two safety gates are essential: a nesting depth cap (MAX_PROMPT_NESTING_DEPTH) and cycle detection (a seen set), preventing A-depends-on-B-depends-on-A from hanging the service.

resolve: inline along the graph, produce a self-contained product parent promptrefs child A (label)refs child B (version) buildAndResolvePromptGraphwalk graph, inline childrengate1 depth capgate2 seen cycle check resolved promptall child snippets inlinedcomplete, self-contained← this is what's cached/returned key fact: a resolved prompt may span several prompt names—the very reason the next section's invalidation is project-scoped
resolution = expand composition into a product: buildAndResolvePromptGraph walks Lesson 37's dependency graph, inlining children into the parent to get resolvedPrompt; MAX_PROMPT_NESTING_DEPTH caps depth, a seen set blocks cycles. What's cached and returned is this expanded product. Source: PromptService/index.ts:132-144,236+.
packages/shared/src/server/services/PromptService/index.tsresolve + two gates
// resolve: return the product with dependencies inlined
public async resolvePrompt(prompt) {
  const promptGraph = await this.buildAndResolvePromptGraph(...);
  return { ...prompt, prompt: promptGraph.resolvedPrompt };   // children inlined
}
// recursively inline along the graph, two safety gates:
if (level >= MAX_PROMPT_NESTING_DEPTH) throw …;   // gate1: stop if nesting too deep
const seen = new Set<string>();                    // gate2: track visited, catch A→B→A instantly

why resolve at serving time

Dependencies are stored in the DB as "reference relations" (pointing to a child by label/version). But a child's label moves and its content changes—so only at the moment of fetching can you pull the current child and inline it, getting the product correct as of now.

gate 1: depth cap

MAX_PROMPT_NESTING_DEPTH limits dependency nesting levels. It prevents someone from making the chain so deep that one resolution pulls dozens of prompts and drags down the service.

gate 2: cycle detection

The seen set records visited prompts. The instant a cycle appears—A depends on B, B depends on A—it's caught and errors out; otherwise the recursion spins forever.

Epoch invalidation: swap the namespace, don't delete keys one by one

The hardest part of caching is always invalidation: when a prompt changes (new version, label move), the old cache must be voided. The difficulty—a resolved prompt may span several names (via dependencies), so you can't even say "which keys to delete". Langfuse's solution is elegant to the point of slyness: embed a project-scoped epoch token in the cache key; to invalidate, delete no keys, just rotate that epoch token to a new random value—so all old keys are instantly "orphaned" (new reads/writes use the new epoch in a new namespace), and the old ones simply expire by TTL.

invalidate = rotate the epoch token (swap namespace), delete no keys cache key looks like: prompt : {projectId} : {epoch} : {promptName} : {version|label} old epoch = e1 (orphaned)prompt:proj:e1:greeting:productionprompt:proj:e1:refund:productionno one reads them → expire by TTL new epoch = e2 (active)prompt:proj:e2:greeting:production…new reads/writes land here (miss then backfill)effectively a clean cache rotate epoch the epoch is project-scoped (not prompt-scoped)—because a resolved prompt spans names, any change must turn the whole page; O(1), no scan/delete
O(1) invalidation, no scan-and-delete: invalidateCache just SETs a new random token (48 bits of entropy) on the project-scoped epoch key. Keys under the old epoch are no longer hit by any read and die off by their own TTL; the epoch key itself has a 7-day TTL. getOrCreateEpoch uses SET NX for concurrent initialization. Source: PromptService/index.ts:179-260.
packages/shared/src/server/services/PromptService/index.tsepoch invalidation + key
// invalidate: delete no keys, just rotate the project-scoped epoch token → all old keys orphaned
public async invalidateCache({ projectId }) {
  await this.redis.set(this.getEpochKey({ projectId }), this.newEpochToken(),
                       "EX", this.epochTtlSeconds);   // epoch key lives 7 days
}
// embed epoch in the key: change the epoch and the whole namespace turns the page
getCacheKey = `prompt:${projectId}:${epoch}:${promptName}:${version ?? label}`;
// epoch project-scoped (not prompt-scoped): a resolved prompt spans names, so any change invalidates all
getEpochKey = `prompt_cache_epoch:${projectId}`;

This design turns "invalidating a tangle of related caches" from an intractable precise-deletion problem into an O(1) token swap. The only cost is old keys briefly occupying a little memory until they expire—trading a touch of wasted memory for invalidation logic that is maximally simple and correct. It's a distributed-caching move well worth adding to your toolbox.

comparisondelete keys one by one (naive)rotate epoch (Langfuse)
invalidation complexityfind all affected keys, O(N) with a scanSET one new token, O(1)
dependency trackingmust reverse-lookup "which resolved products contain this child"none needed—the whole namespace turns the page
miss riskone miss serves a stale promptimpossible to miss, all old keys orphaned
costcomplex, error-prone logicold keys hold a little memory until expiry (auto-die)
🎯 Design trade-off
Why invalidate by "rotating the epoch" rather than dutifully deleting the old keys? Because precise deletion is nearly infeasible here. A served prompt is a resolved product that may inline several other prompts (Lesson 37 dependencies). When you change any one child, all resolved products that indirectly depend on it should be invalidated—but you can barely reverse-lookup "which keys are affected", let alone the fact that they're scattered across different names. "Find them one by one and delete" is slow and miss-prone, and one miss means someone reads a stale prompt. Rotating the epoch cuts the knot: don't search, don't delete, just swap to a fresh namespace; the old ones are all voided and expire naturally. Invalidation is now O(1) and never misses. The cost is a little wasted memory before old keys expire—a textbook "trade space for correctness and simplicity".

Why is the epoch "project-scoped" rather than "prompt-scoped"? A direct corollary of the above. If the epoch were isolated per prompt name, then on changing a child prompt you'd again have to figure out "which parents depend on it, and should I rotate their epochs too"—the dependency-tracking problem reborn. Putting the epoch at project granularity means "any prompt change in this project turns the page on the whole project's prompt cache". A bit coarse—occasionally invalidating caches that could have been kept—but it buys the certainty of never needing to track dependencies at all. Between "invalidate a little less" and "never read stale data", it firmly chooses the latter—exactly right for prompts, where correctness matters far more than cache hit rate.
🎯 Key points
  • Prompt serving must be fast: production may fetch a prompt on every LLM call; hitting Postgres directly is too slow and heavy, so PromptService puts a Redis cache in front.
  • Read-through cache: getPrompt checks Redis first, returns on a hit; on a miss falls back to Postgres then backfills Redis (SET EX ttl). Toggle LANGFUSE_CACHE_PROMPT_ENABLED, duration ..._TTL_SECONDS.
  • Resolve dependencies at serving time: buildAndResolvePromptGraph inlines referenced children into a self-contained product, with a depth cap + seen cycle check; what's cached/returned is this resolved product.
  • Epoch invalidation (the gem): the cache key embeds a project-scoped epoch token; to invalidate, delete no keys, just rotate the token to a new random value—all old keys orphaned, expiring by TTL. O(1), never misses, no scan.
  • Why epoch is project-scoped: a resolved prompt spans names, so precise deletion / prompt-scoped isolation would require dependency tracking; project-scoped epoch trades "a little over-invalidation" for "no tracking + never stale"—the right call for correctness-first data like prompts.

🧪 Self-test - think about the design

1. Langfuse's PromptService uses Redis as a read-through cache. What flow does 'read-through' specifically mean?
  1. queries Postgres every time, then updates Redis on the side
  2. getPrompt checks Redis first, returns on a hit (no DB); only on a miss queries Postgres and backfills the result into Redis (SET EX ttl), so the next identical request hits the cache—the vast majority are millisecond and don't load the DB
  3. writes Redis first, then async to the DB
  4. reads only Redis, never the DB
Show answer & explanation click to expand
Answer: B. Read-through: a hit takes the shortcut, only a miss penetrates to the authoritative DB and backfills. getPrompt first getCachedPrompt, returns on hit; on miss findPrompt(DB)→cachePrompt(redis.set EX ttlSeconds). Production may fetch a prompt every LLM call; this cache shields the DB, with only the first request after a cold/invalidated cache hitting the DB. Toggle LANGFUSE_CACHE_PROMPT_ENABLED, duration _TTL_SECONDS.
2. After a prompt change the cache must be invalidated. Langfuse doesn't delete old keys one by one but embeds a project-scoped epoch token in the cache key, rotating it to a new random value on invalidation. Why?
  1. because a resolved prompt may inline several other prompts (dependencies), precisely reverse-looking-up 'which keys are affected' is nearly infeasible and one miss serves stale data; rotating the epoch swaps to a fresh namespace, old keys orphaned and expiring by TTL—O(1), never misses
  2. to save Redis memory
  3. because Redis doesn't support DEL
  4. to make the cache never expire
Show answer & explanation click to expand
Answer: A. A served prompt is a resolved product that may inline several children (L37). Changing one child should invalidate all resolved products indirectly depending on it, but you can barely reverse-lookup the affected keys, scattered across names—deleting one by one is slow and miss-prone. invalidateCache just SETs a new token on the project-scoped epoch key; cache keys embed that epoch, so old keys are all orphaned and expire by TTL. Trade a little wasted memory for simple, correct invalidation.
3. Why is Langfuse's cache epoch 'project-scoped' (prompt_cache_epoch:{projectId}) rather than isolated per prompt name?
  1. prompt-scoped epochs aren't supported by Redis
  2. because a resolved prompt spans multiple names (dependencies); if the epoch were per prompt name, changing a child would again require tracking 'which parents depend on it'; a project-scoped epoch trades 'occasionally over-invalidating' for 'no dependency tracking at all + never stale'
  3. to share cache across projects
  4. project-scoped keys are shorter, saving memory
Show answer & explanation click to expand
Answer: B. The source comment states: the epoch is project-scoped because a resolved prompt can include transitive dependencies across multiple prompt names. Per-name isolation would revive the 'which parents are affected' dependency-tracking problem on a child change. Project granularity means 'any prompt change in the project turns the whole prompt cache's page'—coarser, occasionally over-invalidating, but buying certainty of no dependency tracking. Right for prompts, where correctness beats hit rate.
💭 Open questions (no single right answer - just think or try)
  • The epoch-invalidation move—'change the namespace, not delete keys'—is essentially 'trade space for correctness and simplicity'. Recall caching systems you've built or imagined: which invalidation scenarios gave you headaches (cascading invalidation, dependency tracking, thundering herd)? Could this epoch/version-namespace idea apply? How would you weigh its costs (memory, TTL choice, cold-start thundering herd)?