🪢 Langfuse 图解教程Langfuse Visual Guide 第五部分 · 评估与评分Part 5 · Evaluation & Scoring 30 / 55
第五部分 · 评估与评分Part 5 · Evaluation & Scoring

eval 执行流水线The eval execution pipeline

🧭 读前Before you start
先读 第29课 的 LLM 裁判与 第14课 的队列,本课讲 eval 执行流水线。
Read Lesson 29 (LLM judge) and Lesson 14 (the queue); this lesson covers the eval execution pipeline.

上一课讲了「一次评判怎么跑」,但留了个大问题:谁来触发它?什么时候?评哪些 trace? 你不可能给每条 trace 手动点「评估」。这一课讲调度流水线——一个 trace 事件进来,系统怎么自动算出「该用哪些评估器评它」、给每个开一张待办工单(JobExecution),排进执行队列,最后跑出分。 我们会看到 createEvalJobs 的扇出与匹配、去重 / 抽样 / 延迟三个闸门、JobExecution 的状态生命周期(PENDING→COMPLETED/ERROR/CANCELLED/DELAYED),以及一个微妙但关键的无限循环防护——因为评估本身也产生 trace。

📋 生活类比
想象一条工厂流水线的质检调度。每当一件产品下线(一个 trace 事件到达),调度员(createEvalJobs)翻出所有启用中的质检规程(ACTIVE 评估器),逐条问「这件产品符合这条规程的适用范围吗?」(按 filter 匹配 trace)。 符合的,就开一张质检工单(JobExecution,初始 PENDING)放进排队筐(执行队列)——但有三道闸:这张工单已经开过就不重复开(去重)、按抽检比例决定抽不抽(采样)、还能让工单等一会儿再做(延迟,等产品彻底定型)。 若产品在后续工序里被撤下生产线(第二个事件让 trace 不再匹配),还没做的工单就作废(CANCELLED)。最妙的一道防呆:质检过程本身也会产出一件「产品」(评估也生成 trace),必须拦住它,否则质检又触发质检,无限套娃。

从一个事件到一队工单:两级队列

评估是事件驱动的。当一条 trace 写入(TraceUpsert 事件,第 14 课那条摄取队列的下游),worker 的「创建器」就跑 createEvalJobs。它不直接评,而是把工作拆成两级队列:第一级负责「该评谁」(创建工单),第二级负责「真正评」(执行)。这种分离是这套系统能扛量、能重试、能采样的关键。

两级队列:创建工单 → 执行评估 trace 写入TraceUpsert 事件 ① 创建器createEvalJobs扇出到所有 ACTIVE 评估器 逐个评估器filter 匹配 trace?闸1 去重闸2 采样闸3 延迟 ② JobExecutionstatus = PENDING落 Postgres + 入执行队列 执行队列EvaluationExecution(可延迟) ③ 执行消费端跑 L29 裁判流水线写分 → ingestion 队列标 COMPLETED + 记 scoreId 第一级只决定「该评谁、开不开工单」(轻、快、纯 Postgres);第二级才真正调 LLM(慢、可能失败、要重试) 两级分离 = 把「调度」与「干活」解耦,各自独立扩容、重试、限流
创建与执行分离:worker/src/queues/evalQueue.ts 里,TraceUpsert(:25)/CreateEvalQueue(:98)触发 createEvalJobs;它产出 PENDING 工单并入 EvaluationExecution 队列(:126 消费)。大项目还可分流到 EvaluationExecutionSecondaryQueue(:151)隔离。
示例:一个 eval 任务的状态机 入队 → PENDING在队列里等 COMPLETEDscore 已写回 ERROR裁判/解析失败 DELAYED延后,稍后回 PENDING CANCELLED配置被删/停用 worker 处理 ✓ worker 处理 ✗
任务的一生就是几个状态(状态枚举取自 prisma 的 JobExecutionStatus;值为示例):一个 JobExecution 从 PENDING(入队)开始,worker 处理后落到 COMPLETED(score 写回)或 ERROR(失败);DELAYED 是延后稍后再回 PENDING,CANCELLED 是配置被删/停用。把「评估有没有跑、跑成没成」变成可查询、可重试的明确状态。
扇出取出所有启用的评估器
createEvalJobs 先查这个项目下所有 jobType=EVAL、status=ACTIVE、未被 blockedAt 的 JobConfiguration。一条 trace 可能同时命中多个评估器(如「有用性」+「毒性」),所以是一对多的扇出。若一个都没有,还会缓存「本项目无评估器」,省掉后续每条 trace 的空查。
匹配这条 trace 符合评估器的目标吗
每个评估器都带一个 filter(如「环境=production 且 name=chat」)。系统优先在内存里用已缓存的 trace 判断(InMemoryFilterService.evaluateFilter,:422);只有当 filter 需要查库才回 ClickHouse。匹配上才继续,否则跳过。
建单开一张 PENDING 工单并入队
过了三道闸(下一节)后,创建一行 JobExecution(status=PENDING) 落 Postgres,并把它的 id 推进执行队列,带上 config.delay 毫秒延迟。工单落库的意义:它是有状态、可追溯的——日后能查「这条 trace 被哪些评估器评过、结果如何」。

谁会触发这套创建器?不止「线上来了条新 trace」一种。createEvalJobs 有三个入口(evalQueue.ts),sourceEventType 字段记下它从哪来:

线上 trace(trace-upsert)

最常见:用户的真实调用写入 trace,evalJobTraceCreatorQueueProcessor 接住。这也是那条需要 langfuse- 防循环的入口。

数据集运行(dataset-run-item-upsert)

跑实验时,数据集里每个条目过一遍模型也会触发评估(第 35 课)。evalJobDatasetCreatorQueueProcessor 接住,带 datasetItemId——这就是为什么变量映射能取 dataset_item 的列。

回填(CreateEvalQueue)

新建/改了评估器后,想对历史 trace 也补评,由这条队列驱动;timeScope 控制一个评估器是只评新数据还是也评历史。

三道闸:去重、采样、延迟

匹配上了,也不一定立刻开工单。createEvalJobs 在创建前串了三道闸——每一道都解决一个真实的生产问题:

createEvalJobs:建单前串三道闸 (评估器, trace) 匹配上 ① 去重 已有工单? 是 → 跳过 ② 采样 random > sampling 中 → 跳过 ③ 延迟 带 delay 入队 等数据到齐 JobExecution PENDING 跳过(不建单) 反向闸:trace 后来不再匹配 filter → 未完成工单标 CANCELLED(排除 COMPLETED)
闸门它问什么不通过会怎样解决的问题
① 去重这条 (评估器, trace) 已经有工单了吗?跳过,不再建同一 trace 的多个事件(创建、更新)重复触发,不该评好几遍
② 采样Math.random() > config.sampling?本次抽不中,跳过评估要花 LLM 钱;只抽 10% 的流量评,成本可控
③ 延迟(不拦截,而是推迟)工单带 config.delay 毫秒入队等整条 trace 的所有 observation 都到齐了再评,避免评到半截数据
worker/src/features/evaluation/evalService.ts三道闸 + 建单 + 入队
// 闸1 去重:批量查过的已有工单里,已存在就不再建
if (existingJob.length > 0) { logger.debug("already exists"); continue; }

// 闸2 采样:sampling∈[0,1] 是抽中概率,抽不中就跳过
if (Number(config.sampling) !== 1) {
  if (Math.random() > Number(config.sampling)) { continue; }   // sampled out
}

// 建单:一行 PENDING 的 JobExecution 落 Postgres
await prisma.jobExecution.create({ data: {
  id: jobExecutionId, projectId, jobConfigurationId: config.id,
  jobInputTraceId: event.traceId, jobTemplateId: config.evalTemplateId,
  status: "PENDING", startTime: new Date() } });

// 闸3 延迟:推进执行队列时带上 config.delay 毫秒
await EvalExecutionQueue.getInstance({ shardingKey })?.add(
  QueueName.EvaluationExecution,
  { payload: { projectId, jobExecutionId, delay: config.delay } },
  { delay: config.delay });   // 毫秒

还有一道「反向闸」:如果某条 trace 后来不再匹配评估器的 filter(第二个 trace 事件改了它的属性,把它「踢出」目标集合),而它之前已有工单,系统会把那张还没跑完的工单标成 CANCELLED(用 updateMany 且排除 COMPLETED,避免误改已完成的)。评估的目标集合是动态的——选中了能取消,这让调度对「数据后到、属性后改」很有韧性。

工单的一生:状态机与无限循环防护

JobExecution 是一台状态机。它出生于 PENDING,有三种终局(COMPLETED/ERROR/CANCELLED);中间还可能因限流而 DELAYED 后重试。下图是它的完整生命周期:

JobExecution 状态生命周期 PENDING建单时 执行中跑 L29 裁判 COMPLETED ✓记 jobOutputScoreId DELAYED限流,~120min 后重试 ERROR ✗不可重试/预算耗尽 CANCELLED被后续事件取消选中 重试回执行 三种终局 + 一个可重试中间态 DELAYED(初态 PENDING);DELAYED→执行的环让限流不致命,而是「稍后再试」
状态枚举来自真实 schema:JobExecutionStatus = { COMPLETED, ERROR, PENDING, CANCELLED, DELAYED }(schema.prisma:1080-1086)。完成时 evalCompletion.ts:82-92 把分写进 ingestion 队列、再把工单标 COMPLETED 并填 jobOutputScoreId 与 executionTraceId——工单从此双向链接到「它产出的分」和「裁判自己的 trace」。
worker/src/features/evaluation/evalService.ts:240 · evalCompletion.ts:82防循环 + 收尾
// 无限循环防护:来自 trace-upsert 且环境以 "langfuse" 开头的内部 trace,直接不建评估工单
// 否则:用户 trace → 评估 → 评估自己的 trace → 又触发评估 → 无限套娃
if (sourceEventType === "trace-upsert" &&
    event.traceEnvironment?.startsWith("langfuse")) {
  return;   // 跳过:这是评估自身产生的 trace
}

// 收尾:分写进 ingestion 队列,工单标 COMPLETED 并双向链接产出物
await deps.enqueueScoreIngestion({ projectId, scoreId, eventId });
await deps.updateJobExecution({ id: jobExecutionId, projectId, data: {
  status: "COMPLETED", endTime: new Date(),
  jobOutputScoreId,                       // ← 链到「评出的那条分」
  executionTraceId: result.executionTraceId } });  // ← 链到「裁判自己的 trace」
🎯 设计取舍
为什么要拆成「创建」和「执行」两级队列,而不是一步评完? 因为这两件事的性质截然不同。创建(匹配、去重、采样)是轻、快、纯数据库的活,失败了重跑代价小;执行(调 LLM)是慢、花钱、会被限流、可能超时的活。 混在一起,慢的会拖死快的、贵的没法单独采样、失败的没法独立重试。拆开后:第一级可以高吞吐地决定「该评谁」,第二级可以带延迟、采样、独立重试、甚至分流到二级队列地慢慢消化。这正是第 14 课「队列解耦」思想在评估域的复刻。

为什么 JobExecution 必须落 Postgres,而不只是一条队列消息? 因为它要有状态、可追溯、可去重、可取消。队列消息是「阅后即焚」的,但工单要回答「这条 trace 被哪些评估器评过、跑成功没、产出哪条分」——这些都需要一行持久、可更新的记录。jobOutputScoreId 和 executionTraceId 两个外链,让你能从「一条分」反查「哪个工单、哪个评估器、哪次裁判调用产生了它」,全链路可审计。

最微妙的一处:无限循环防护。 评估本身要调 LLM,而 Langfuse 会观测每一次 LLM 调用——于是裁判的调用也变成一条 trace。若不拦截,这条「评估产生的 trace」又会触发评估,套娃到天荒地老。Langfuse 的解法是给内部 trace 打上 langfuse- 环境前缀,并在创建器入口直接挡掉。一个能观测一切的系统,必须小心别把自己卷进去——这是自指系统的经典陷阱。
🎯 本课要点
  • 评估是事件驱动的:trace 写入(TraceUpsert)触发 createEvalJobs,把一条 trace 扇出到项目内所有 ACTIVE 评估器,一对多创建工单。
  • 两级队列:第一级「创建器」轻快地决定「该评谁」(匹配+建单),第二级「执行队列」带延迟/采样/重试地真正调 LLM——解耦让系统能扛量、能重试、能限流。
  • 三道闸:去重(同 trace 多事件不重复评)、采样(config.sampling 概率控成本)、延迟(config.delay 等数据到齐);外加反向的「取消」(后续事件让 trace 不再匹配 → CANCELLED)。
  • filter 匹配优先在内存做:用缓存的 trace + InMemoryFilterService.evaluateFilter,只有复杂过滤才回 ClickHouse 查——省一次次的库查。
  • JobExecution 是落库的状态机:PENDING→COMPLETED/ERROR/CANCELLED,限流则 DELAYED 后重试;完成时双向链接 jobOutputScoreId 与 executionTraceId,全链路可审计。
  • 无限循环防护:评估自身的 LLM 调用也会成为 trace;用 langfuse- 环境前缀在创建器入口挡掉,避免「评估触发评估」的自指套娃。

🧪 自测 · 想一想为什么这么设计

1. Langfuse 的评估调度把工作拆成「创建」和「执行」两级队列。这种分离最核心的好处是?
  1. 让评估结果更准确
  2. 减少数据库表的数量
  3. 省掉 LLM 调用
  4. 两件事性质不同:创建是轻快的纯数据库匹配,执行是慢、花钱、会限流的 LLM 调用——分开后各自独立扩容、采样、重试,慢的不拖死快的
看答案与解析 点击展开
答案:D。创建(匹配、去重、采样)轻快、失败重跑代价小;执行(调 LLM)慢、贵、会被限流、可能超时。混在一起则慢拖快、贵的没法单独采样、失败没法独立重试。两级队列让第一级高吞吐决定「该评谁」,第二级带延迟/采样/重试/分流地慢慢消化——正是第 14 课「队列解耦」在评估域的复刻。
2. 评估本身要调 LLM,而 Langfuse 会把每次 LLM 调用记成一条 trace。createEvalJobs 里有一段「如果 trace 环境以 langfuse 开头就直接 return」的代码。它防的是什么?
  1. 加快内部 trace 的写入
  2. 防止重复计费
  3. 无限循环:用户 trace 触发评估 → 评估的 LLM 调用又成一条 trace → 又触发评估 → 无限套娃;给内部 trace 打 langfuse- 前缀并在入口挡掉即可斩断
  4. 防止内部 trace 被用户看到
看答案与解析 点击展开
答案:C。这是自指系统的经典陷阱:一个能观测一切的系统,会把自己的评估调用也观测成 trace,从而触发对自己的评估。Langfuse 让内部 trace 用 LangfuseInternalTraceEnvironment(langfuse- 前缀),并在 createEvalJobs 入口对 trace-upsert 来源的这类 trace 直接 return,斩断递归。fetchLLMCompletion 还强制内部 trace 必须带该前缀,形成双重保险。
3. 一行 JobExecution 落在 Postgres 里,而不只是一条「阅后即焚」的队列消息。为什么评估系统需要这样一个有状态、可更新的工单?
  1. Postgres 比队列快
  2. 队列不能存 JSON
  3. 为了不用 Redis
  4. 它要支持去重、取消、追溯:能回答「这条 trace 被哪些评估器评过、成功没、产出哪条分」,并通过 jobOutputScoreId/executionTraceId 双向链到产出的分与裁判自己的 trace
看答案与解析 点击展开
答案:D。队列消息消费即逝,无法回答「评过没、结果如何」。JobExecution 是落库的状态机(PENDING→COMPLETED/ERROR/CANCELLED/DELAYED),支持:去重(已有工单就不重复建)、取消(trace 被后续事件踢出目标集合则标 CANCELLED)、追溯(jobOutputScoreId 链到评出的分、executionTraceId 链到裁判的 trace),全链路可审计。
💭 发散思考(没有标准答案,动手或动脑想想)
  • 评估系统给了三道闸:去重、采样、延迟。假设你的生产流量每天一百万条 trace,每条评估要调一次 GPT-4 花约 ¥0.05。你会怎么配置这三道闸(采样率、延迟)来平衡「成本」与「质量信号的及时性/代表性」?延迟设太短、采样设太低分别会带来什么问题?

Last lesson covered "how one verdict runs", but left a big question: who triggers it? when? which traces get evaluated? You can't click "evaluate" on every trace by hand. This lesson is the scheduling pipeline—how a trace event automatically fans out into "which evaluators should judge it", opens a work ticket (JobExecution) for each, queues it, and finally produces a score. We'll see createEvalJobs' fan-out and matching, the three gates of dedup / sampling / delay, JobExecution's status lifecycle (PENDING→COMPLETED/ERROR/CANCELLED/DELAYED), and a subtle but crucial infinite-loop safeguard—because evaluation itself produces traces.

📋 Analogy
Picture the QC dispatch on a factory line. Each time a product rolls off (a trace event arrives), the dispatcher (createEvalJobs) pulls out all active QC procedures (ACTIVE evaluators) and asks of each "does this product fall within this procedure's scope?" (match the trace by filter). For matches, it opens a QC ticket (JobExecution, initially PENDING) into a queue basket (the execution queue)—but through three gates: don't re-open a ticket already opened (dedup), decide by sampling rate whether to inspect at all (sampling), and let the ticket wait a bit before doing it (delay, until the product fully sets). If the product is later pulled off the line (a second event makes the trace no longer match), an un-run ticket is voided (CANCELLED). The neatest fail-safe: the QC process itself yields a "product" (evaluation also generates a trace), and you must stop it, or QC triggers QC, nesting forever.

From one event to a queue of tickets: two-stage queues

Evaluation is event-driven. When a trace is written (a TraceUpsert event, downstream of Lesson 14's ingestion queue), the worker's "creator" runs createEvalJobs. It doesn't evaluate directly—it splits the work into two queue stages: the first decides "whom to evaluate" (create tickets), the second does "the actual judging" (execution). This separation is the key to scaling, retrying, and sampling.

Two-stage queues: create tickets → execute evaluation trace writtenTraceUpsert event ① creatorcreateEvalJobsfan out to all ACTIVE evaluators per evaluatorfilter matches trace?gate1 dedupgate2 samplinggate3 delay ② JobExecutionstatus = PENDINGPostgres row + into exec queue execution queueEvaluationExecution (delayable) ③ exec consumerruns the L29 judge pipelinewrite score → ingestion queuemark COMPLETED + record scoreId stage one only decides "whom to evaluate, open a ticket?" (light, fast, pure Postgres); stage two calls the LLM (slow, may fail, must retry) two-stage split = decouple "scheduling" from "doing the work", each scaling/retrying/rate-limiting independently
Creation and execution split: in worker/src/queues/evalQueue.ts, TraceUpsert (:25)/CreateEvalQueue (:98) trigger createEvalJobs; it emits PENDING tickets into the EvaluationExecution queue (consumed at :126). Large projects can also offload to EvaluationExecutionSecondaryQueue (:151) for isolation.
Example: an eval job's state machine enqueue → PENDINGwaiting in queue COMPLETEDscore written ERRORjudge/parse failed DELAYEDscheduled, back to PENDING CANCELLEDconfig removed/disabled worker processes ✓ worker processes ✗
A job's life is a few states (status enum from prisma JobExecutionStatus; values illustrative): a JobExecution starts PENDING (enqueued); after the worker runs it lands on COMPLETED (score written) or ERROR (failed); DELAYED reschedules back to PENDING, and CANCELLED means the config was removed/disabled. It turns "did the eval run, did it succeed" into an explicit, queryable, retryable state.
fan-outpull all active evaluators
createEvalJobs first queries all JobConfigurations in this project with jobType=EVAL, status=ACTIVE, not blockedAt. One trace may hit several evaluators at once (e.g. "helpfulness" + "toxicity"), so it's a one-to-many fan-out. If there are none, it caches "no evaluators for this project" to skip the empty query on every future trace.
matchdoes this trace fit the evaluator's target
Each evaluator carries a filter (e.g. "environment=production and name=chat"). The system prefers to judge in memory using the cached trace (InMemoryFilterService.evaluateFilter, :422); only when the filter needs a DB lookup does it go back to ClickHouse. Continue only on a match, else skip.
createopen a PENDING ticket and enqueue
After the three gates (next section), it creates a JobExecution(status=PENDING) row in Postgres and pushes its id into the execution queue with a config.delay ms delay. Why persist the ticket: it is stateful and auditable—later you can ask "which evaluators evaluated this trace, and with what outcome".

Who triggers this creator? Not only "a new trace arrived live". createEvalJobs has three entry points (evalQueue.ts), and a sourceEventType field records where it came from:

live trace (trace-upsert)

The common case: a user's real call writes a trace, caught by evalJobTraceCreatorQueueProcessor. This is also the entry needing the langfuse- loop guard.

dataset run (dataset-run-item-upsert)

When running experiments, each dataset item passing through the model also triggers evaluation (Lesson 35). Caught by evalJobDatasetCreatorQueueProcessor with a datasetItemId—which is why variable mapping can read dataset_item columns.

backfill (CreateEvalQueue)

After creating/changing an evaluator, to also score historic traces, this queue drives it; timeScope controls whether an evaluator scores only new data or history too.

Three gates: dedup, sampling, delay

A match doesn't mean a ticket opens immediately. Before creating, createEvalJobs chains three gates—each solving a real production problem:

createEvalJobs: three gates before opening a ticket (evaluator, trace) matched 1 dedup job exists? yes → skip 2 sampling random > sampling out → skip 3 delay enqueue +delay let data settle JobExecution PENDING skip (no ticket) reverse gate: a trace that no longer matches the filter → unfinished job marked CANCELLED (excludes COMPLETED)
gatewhat it asksif it failsproblem solved
① dedupdoes a ticket for this (evaluator, trace) already exist?skip, don't createmultiple events on the same trace (create, update) shouldn't trigger several evals
② samplingMath.random() > config.sampling?not sampled in, skipevaluation costs LLM money; sampling only 10% keeps cost in check
③ delay(doesn't block, defers)ticket enqueued with config.delay mswait until all of a trace's observations have arrived, to avoid judging half the data
worker/src/features/evaluation/evalService.tsthree gates + create + enqueue
// gate1 dedup: from the batch-queried existing tickets, skip if one exists
if (existingJob.length > 0) { logger.debug("already exists"); continue; }

// gate2 sampling: sampling∈[0,1] is the keep probability; skip if not sampled
if (Number(config.sampling) !== 1) {
  if (Math.random() > Number(config.sampling)) { continue; }   // sampled out
}

// create: one PENDING JobExecution row in Postgres
await prisma.jobExecution.create({ data: {
  id: jobExecutionId, projectId, jobConfigurationId: config.id,
  jobInputTraceId: event.traceId, jobTemplateId: config.evalTemplateId,
  status: "PENDING", startTime: new Date() } });

// gate3 delay: push to the execution queue with config.delay ms
await EvalExecutionQueue.getInstance({ shardingKey })?.add(
  QueueName.EvaluationExecution,
  { payload: { projectId, jobExecutionId, delay: config.delay } },
  { delay: config.delay });   // milliseconds

There's also a "reverse gate": if a trace later no longer matches the evaluator's filter (a second trace event changed its attributes, kicking it out of the target set) and it already had a ticket, the system marks that un-run ticket CANCELLED (via updateMany excluding COMPLETED, to avoid touching finished ones). The evaluation target set is dynamic—selected can become deselected, making scheduling resilient to "data arriving late, attributes changing late".

A ticket's life: the state machine and the infinite-loop safeguard

JobExecution is a state machine. It is born PENDING, with three possible endings (COMPLETED/ERROR/CANCELLED); in between it may go DELAYED on rate-limit and retry. Below is its full lifecycle:

JobExecution status lifecycle PENDINGon create executingruns the L29 judge COMPLETED ✓records jobOutputScoreId DELAYEDrate-limit, retry ~120min ERROR ✗non-retryable/budget out CANCELLEDdeselected by later event retry back to exec three endings + one retryable middle state DELAYED (initial PENDING); the DELAYED→exec loop makes rate-limits non-fatal, just "try later"
The status enum is from the real schema: JobExecutionStatus = { COMPLETED, ERROR, PENDING, CANCELLED, DELAYED } (schema.prisma:1080-1086). On success evalCompletion.ts:82-92 writes the score into the ingestion queue, then marks the ticket COMPLETED and fills jobOutputScoreId and executionTraceId—the ticket is now bidirectionally linked to "the score it produced" and "the judge's own trace".
worker/src/features/evaluation/evalService.ts:240 · evalCompletion.ts:82loop guard + finish
// loop guard: internal traces from trace-upsert whose env starts with "langfuse" → create no eval ticket
// otherwise: user trace → eval → eval's own trace → triggers eval again → nesting forever
if (sourceEventType === "trace-upsert" &&
    event.traceEnvironment?.startsWith("langfuse")) {
  return;   // skip: this is a trace produced by evaluation itself
}

// finish: score into the ingestion queue, ticket marked COMPLETED and bidirectionally linked
await deps.enqueueScoreIngestion({ projectId, scoreId, eventId });
await deps.updateJobExecution({ id: jobExecutionId, projectId, data: {
  status: "COMPLETED", endTime: new Date(),
  jobOutputScoreId,                       // ← links to "the score produced"
  executionTraceId: result.executionTraceId } });  // ← links to "the judge's own trace"
🎯 Design trade-off
Why split into "creation" and "execution" queues rather than evaluate in one step? Because the two are fundamentally different. Creation (matching, dedup, sampling) is light, fast, pure-database work, cheap to re-run on failure; execution (calling the LLM) is slow, costly, rate-limited, may time out. Mixed together, the slow drags down the fast, the expensive can't be sampled separately, and failures can't be retried independently. Split: stage one decides "whom to evaluate" at high throughput, stage two digests slowly with delay, sampling, independent retry, even offload to a secondary queue. This is Lesson 14's "queue decoupling" replayed in the eval domain.

Why must a JobExecution live in Postgres, not just be a queue message? Because it must be stateful, auditable, dedup-able, cancellable. A queue message is "read once, gone", but a ticket must answer "which evaluators evaluated this trace, did it succeed, which score did it produce"—all needing a durable, updatable row. The two foreign links jobOutputScoreId and executionTraceId let you trace from "a score" back to "which ticket, which evaluator, which judge call produced it"—fully auditable end to end.

The subtlest part: the infinite-loop safeguard. Evaluation calls the LLM, and Langfuse observes every LLM call—so the judge's call becomes a trace too. Unstopped, that "eval-produced trace" would trigger evaluation again, nesting endlessly. Langfuse's fix: tag internal traces with a langfuse- environment prefix and block them right at the creator's entry. A system that can observe everything must be careful not to observe itself into a loop—the classic trap of self-referential systems.
🎯 Key points
  • Evaluation is event-driven: a trace write (TraceUpsert) triggers createEvalJobs, fanning one trace out to all ACTIVE evaluators in the project, creating tickets one-to-many.
  • Two-stage queues: stage one "creator" lightly decides "whom to evaluate" (match + create ticket), stage two "execution queue" actually calls the LLM with delay/sampling/retry—decoupling lets the system scale, retry, and rate-limit.
  • Three gates: dedup (multiple events on one trace don't re-evaluate), sampling (config.sampling probability to control cost), delay (config.delay to wait for data); plus a reverse "cancel" (a later event makes the trace no longer match → CANCELLED).
  • Filter matching prefers memory: cached trace + InMemoryFilterService.evaluateFilter, going back to ClickHouse only for complex filters—saving repeated DB queries.
  • JobExecution is a persisted state machine: PENDING→COMPLETED/ERROR/CANCELLED, or DELAYED then retry on rate-limit; on completion it bidirectionally links jobOutputScoreId and executionTraceId—auditable end to end.
  • Infinite-loop safeguard: evaluation's own LLM call also becomes a trace; a langfuse- environment prefix blocks it at the creator's entry, avoiding "eval triggers eval" self-referential nesting.

🧪 Self-test - think about the design

1. Langfuse's eval scheduling splits work into 'creation' and 'execution' two-stage queues. The core benefit of this separation is?
  1. makes eval results more accurate
  2. reduces the number of DB tables
  3. avoids the LLM call
  4. the two differ in nature: creation is light pure-DB matching, execution is slow, costly, rate-limited LLM calls — split apart they scale/sample/retry independently, and the slow doesn't drag down the fast
Show answer & explanation click to expand
Answer: D. Creation (match, dedup, sample) is light and cheap to re-run; execution (LLM call) is slow, costly, rate-limited, may time out. Mixed, the slow drags the fast, the costly can't be sampled alone, failures can't retry independently. Two-stage queues let stage one decide 'whom' at high throughput and stage two digest slowly with delay/sampling/retry/offload — Lesson 14's queue decoupling in the eval domain.
2. Evaluation calls the LLM, and Langfuse records each LLM call as a trace. createEvalJobs has code that 'returns immediately if the trace environment starts with langfuse'. What does it prevent?
  1. speeds up writing internal traces
  2. prevents double billing
  3. an infinite loop: a user trace triggers eval → the eval's LLM call becomes a trace → triggers eval again → nesting forever; tagging internal traces with a langfuse- prefix and blocking them at the entry breaks the cycle
  4. hides internal traces from users
Show answer & explanation click to expand
Answer: C. A classic self-reference trap: a system that observes everything observes its own eval call as a trace, triggering eval on itself. Langfuse marks internal traces with LangfuseInternalTraceEnvironment (langfuse- prefix) and returns early at createEvalJobs for such trace-upsert-sourced traces, cutting the recursion. fetchLLMCompletion also enforces that internal traces carry the prefix — a dual safeguard.
3. A JobExecution row lives in Postgres, not just a 'read-once' queue message. Why does the eval system need such a stateful, updatable ticket?
  1. Postgres is faster than a queue
  2. queues can't store JSON
  3. to avoid using Redis
  4. it must support dedup, cancellation, and audit: answering 'which evaluators evaluated this trace, did it succeed, which score did it produce', and via jobOutputScoreId/executionTraceId bidirectionally linking to the produced score and the judge's own trace
Show answer & explanation click to expand
Answer: D. A queue message is gone on consume, unable to answer 'evaluated yet, what outcome'. JobExecution is a persisted state machine (PENDING→COMPLETED/ERROR/CANCELLED/DELAYED) supporting: dedup (don't recreate an existing ticket), cancel (mark CANCELLED if a later event kicks the trace out of the target set), and audit (jobOutputScoreId links to the score, executionTraceId links to the judge's trace) — auditable end to end.
💭 Open questions (no single right answer - just think or try)
  • The eval system offers three gates: dedup, sampling, delay. Suppose your production handles a million traces/day, each eval calling GPT-4 at ~$0.007. How would you configure these gates (sample rate, delay) to balance 'cost' against 'timeliness/representativeness of the quality signal'? What problems arise from setting delay too short or sampling too low?