上一课讲了「一次评判怎么跑」,但留了个大问题:谁来触发它?什么时候?评哪些 trace? 你不可能给每条 trace 手动点「评估」。这一课讲调度流水线——一个 trace 事件进来,系统怎么自动算出「该用哪些评估器评它」、给每个开一张待办工单(JobExecution),排进执行队列,最后跑出分。
我们会看到 createEvalJobs 的扇出与匹配、去重 / 抽样 / 延迟三个闸门、JobExecution 的状态生命周期(PENDING→COMPLETED/ERROR/CANCELLED/DELAYED),以及一个微妙但关键的无限循环防护——因为评估本身也产生 trace。
createEvalJobs)翻出所有启用中的质检规程(ACTIVE 评估器),逐条问「这件产品符合这条规程的适用范围吗?」(按 filter 匹配 trace)。
符合的,就开一张质检工单(JobExecution,初始 PENDING)放进排队筐(执行队列)——但有三道闸:这张工单已经开过就不重复开(去重)、按抽检比例决定抽不抽(采样)、还能让工单等一会儿再做(延迟,等产品彻底定型)。
若产品在后续工序里被撤下生产线(第二个事件让 trace 不再匹配),还没做的工单就作废(CANCELLED)。最妙的一道防呆:质检过程本身也会产出一件「产品」(评估也生成 trace),必须拦住它,否则质检又触发质检,无限套娃。
评估是事件驱动的。当一条 trace 写入(TraceUpsert 事件,第 14 课那条摄取队列的下游),worker 的「创建器」就跑 createEvalJobs。它不直接评,而是把工作拆成两级队列:第一级负责「该评谁」(创建工单),第二级负责「真正评」(执行)。这种分离是这套系统能扛量、能重试、能采样的关键。
worker/src/queues/evalQueue.ts 里,TraceUpsert(:25)/CreateEvalQueue(:98)触发 createEvalJobs;它产出 PENDING 工单并入 EvaluationExecution 队列(:126 消费)。大项目还可分流到 EvaluationExecutionSecondaryQueue(:151)隔离。JobExecutionStatus;值为示例):一个 JobExecution 从 PENDING(入队)开始,worker 处理后落到 COMPLETED(score 写回)或 ERROR(失败);DELAYED 是延后稍后再回 PENDING,CANCELLED 是配置被删/停用。把「评估有没有跑、跑成没成」变成可查询、可重试的明确状态。createEvalJobs 先查这个项目下所有 jobType=EVAL、status=ACTIVE、未被 blockedAt 的 JobConfiguration。一条 trace 可能同时命中多个评估器(如「有用性」+「毒性」),所以是一对多的扇出。若一个都没有,还会缓存「本项目无评估器」,省掉后续每条 trace 的空查。InMemoryFilterService.evaluateFilter,:422);只有当 filter 需要查库才回 ClickHouse。匹配上才继续,否则跳过。JobExecution(status=PENDING) 落 Postgres,并把它的 id 推进执行队列,带上 config.delay 毫秒延迟。工单落库的意义:它是有状态、可追溯的——日后能查「这条 trace 被哪些评估器评过、结果如何」。谁会触发这套创建器?不止「线上来了条新 trace」一种。createEvalJobs 有三个入口(evalQueue.ts),sourceEventType 字段记下它从哪来:
最常见:用户的真实调用写入 trace,evalJobTraceCreatorQueueProcessor 接住。这也是那条需要 langfuse- 防循环的入口。
跑实验时,数据集里每个条目过一遍模型也会触发评估(第 35 课)。evalJobDatasetCreatorQueueProcessor 接住,带 datasetItemId——这就是为什么变量映射能取 dataset_item 的列。
新建/改了评估器后,想对历史 trace 也补评,由这条队列驱动;timeScope 控制一个评估器是只评新数据还是也评历史。
匹配上了,也不一定立刻开工单。createEvalJobs 在创建前串了三道闸——每一道都解决一个真实的生产问题:
| 闸门 | 它问什么 | 不通过会怎样 | 解决的问题 |
|---|---|---|---|
| ① 去重 | 这条 (评估器, trace) 已经有工单了吗? | 跳过,不再建 | 同一 trace 的多个事件(创建、更新)重复触发,不该评好几遍 |
| ② 采样 | Math.random() > config.sampling? | 本次抽不中,跳过 | 评估要花 LLM 钱;只抽 10% 的流量评,成本可控 |
| ③ 延迟 | (不拦截,而是推迟) | 工单带 config.delay 毫秒入队 | 等整条 trace 的所有 observation 都到齐了再评,避免评到半截数据 |
// 闸1 去重:批量查过的已有工单里,已存在就不再建 if (existingJob.length > 0) { logger.debug("already exists"); continue; } // 闸2 采样:sampling∈[0,1] 是抽中概率,抽不中就跳过 if (Number(config.sampling) !== 1) { if (Math.random() > Number(config.sampling)) { continue; } // sampled out } // 建单:一行 PENDING 的 JobExecution 落 Postgres await prisma.jobExecution.create({ data: { id: jobExecutionId, projectId, jobConfigurationId: config.id, jobInputTraceId: event.traceId, jobTemplateId: config.evalTemplateId, status: "PENDING", startTime: new Date() } }); // 闸3 延迟:推进执行队列时带上 config.delay 毫秒 await EvalExecutionQueue.getInstance({ shardingKey })?.add( QueueName.EvaluationExecution, { payload: { projectId, jobExecutionId, delay: config.delay } }, { delay: config.delay }); // 毫秒
还有一道「反向闸」:如果某条 trace 后来不再匹配评估器的 filter(第二个 trace 事件改了它的属性,把它「踢出」目标集合),而它之前已有工单,系统会把那张还没跑完的工单标成 CANCELLED(用 updateMany 且排除 COMPLETED,避免误改已完成的)。评估的目标集合是动态的——选中了能取消,这让调度对「数据后到、属性后改」很有韧性。
JobExecution 是一台状态机。它出生于 PENDING,有三种终局(COMPLETED/ERROR/CANCELLED);中间还可能因限流而 DELAYED 后重试。下图是它的完整生命周期:
JobExecutionStatus = { COMPLETED, ERROR, PENDING, CANCELLED, DELAYED }(schema.prisma:1080-1086)。完成时 evalCompletion.ts:82-92 把分写进 ingestion 队列、再把工单标 COMPLETED 并填 jobOutputScoreId 与 executionTraceId——工单从此双向链接到「它产出的分」和「裁判自己的 trace」。// 无限循环防护:来自 trace-upsert 且环境以 "langfuse" 开头的内部 trace,直接不建评估工单 // 否则:用户 trace → 评估 → 评估自己的 trace → 又触发评估 → 无限套娃 if (sourceEventType === "trace-upsert" && event.traceEnvironment?.startsWith("langfuse")) { return; // 跳过:这是评估自身产生的 trace } // 收尾:分写进 ingestion 队列,工单标 COMPLETED 并双向链接产出物 await deps.enqueueScoreIngestion({ projectId, scoreId, eventId }); await deps.updateJobExecution({ id: jobExecutionId, projectId, data: { status: "COMPLETED", endTime: new Date(), jobOutputScoreId, // ← 链到「评出的那条分」 executionTraceId: result.executionTraceId } }); // ← 链到「裁判自己的 trace」
jobOutputScoreId 和 executionTraceId 两个外链,让你能从「一条分」反查「哪个工单、哪个评估器、哪次裁判调用产生了它」,全链路可审计。langfuse- 环境前缀,并在创建器入口直接挡掉。一个能观测一切的系统,必须小心别把自己卷进去——这是自指系统的经典陷阱。
createEvalJobs,把一条 trace 扇出到项目内所有 ACTIVE 评估器,一对多创建工单。config.sampling 概率控成本)、延迟(config.delay 等数据到齐);外加反向的「取消」(后续事件让 trace 不再匹配 → CANCELLED)。InMemoryFilterService.evaluateFilter,只有复杂过滤才回 ClickHouse 查——省一次次的库查。jobOutputScoreId 与 executionTraceId,全链路可审计。langfuse- 环境前缀在创建器入口挡掉,避免「评估触发评估」的自指套娃。
Last lesson covered "how one verdict runs", but left a big question: who triggers it? when? which traces get evaluated? You can't click "evaluate" on every trace by hand. This lesson is the scheduling pipeline—how a trace event automatically fans out into "which evaluators should judge it", opens a work ticket (JobExecution) for each, queues it, and finally produces a score.
We'll see createEvalJobs' fan-out and matching, the three gates of dedup / sampling / delay, JobExecution's status lifecycle (PENDING→COMPLETED/ERROR/CANCELLED/DELAYED), and a subtle but crucial infinite-loop safeguard—because evaluation itself produces traces.
createEvalJobs) pulls out all active QC procedures (ACTIVE evaluators) and asks of each "does this product fall within this procedure's scope?" (match the trace by filter).
For matches, it opens a QC ticket (JobExecution, initially PENDING) into a queue basket (the execution queue)—but through three gates: don't re-open a ticket already opened (dedup), decide by sampling rate whether to inspect at all (sampling), and let the ticket wait a bit before doing it (delay, until the product fully sets).
If the product is later pulled off the line (a second event makes the trace no longer match), an un-run ticket is voided (CANCELLED). The neatest fail-safe: the QC process itself yields a "product" (evaluation also generates a trace), and you must stop it, or QC triggers QC, nesting forever.
Evaluation is event-driven. When a trace is written (a TraceUpsert event, downstream of Lesson 14's ingestion queue), the worker's "creator" runs createEvalJobs. It doesn't evaluate directly—it splits the work into two queue stages: the first decides "whom to evaluate" (create tickets), the second does "the actual judging" (execution). This separation is the key to scaling, retrying, and sampling.
worker/src/queues/evalQueue.ts, TraceUpsert (:25)/CreateEvalQueue (:98) trigger createEvalJobs; it emits PENDING tickets into the EvaluationExecution queue (consumed at :126). Large projects can also offload to EvaluationExecutionSecondaryQueue (:151) for isolation.JobExecutionStatus; values illustrative): a JobExecution starts PENDING (enqueued); after the worker runs it lands on COMPLETED (score written) or ERROR (failed); DELAYED reschedules back to PENDING, and CANCELLED means the config was removed/disabled. It turns "did the eval run, did it succeed" into an explicit, queryable, retryable state.createEvalJobs first queries all JobConfigurations in this project with jobType=EVAL, status=ACTIVE, not blockedAt. One trace may hit several evaluators at once (e.g. "helpfulness" + "toxicity"), so it's a one-to-many fan-out. If there are none, it caches "no evaluators for this project" to skip the empty query on every future trace.InMemoryFilterService.evaluateFilter, :422); only when the filter needs a DB lookup does it go back to ClickHouse. Continue only on a match, else skip.JobExecution(status=PENDING) row in Postgres and pushes its id into the execution queue with a config.delay ms delay. Why persist the ticket: it is stateful and auditable—later you can ask "which evaluators evaluated this trace, and with what outcome".Who triggers this creator? Not only "a new trace arrived live". createEvalJobs has three entry points (evalQueue.ts), and a sourceEventType field records where it came from:
The common case: a user's real call writes a trace, caught by evalJobTraceCreatorQueueProcessor. This is also the entry needing the langfuse- loop guard.
When running experiments, each dataset item passing through the model also triggers evaluation (Lesson 35). Caught by evalJobDatasetCreatorQueueProcessor with a datasetItemId—which is why variable mapping can read dataset_item columns.
After creating/changing an evaluator, to also score historic traces, this queue drives it; timeScope controls whether an evaluator scores only new data or history too.
A match doesn't mean a ticket opens immediately. Before creating, createEvalJobs chains three gates—each solving a real production problem:
| gate | what it asks | if it fails | problem solved |
|---|---|---|---|
| ① dedup | does a ticket for this (evaluator, trace) already exist? | skip, don't create | multiple events on the same trace (create, update) shouldn't trigger several evals |
| ② sampling | Math.random() > config.sampling? | not sampled in, skip | evaluation costs LLM money; sampling only 10% keeps cost in check |
| ③ delay | (doesn't block, defers) | ticket enqueued with config.delay ms | wait until all of a trace's observations have arrived, to avoid judging half the data |
// gate1 dedup: from the batch-queried existing tickets, skip if one exists if (existingJob.length > 0) { logger.debug("already exists"); continue; } // gate2 sampling: sampling∈[0,1] is the keep probability; skip if not sampled if (Number(config.sampling) !== 1) { if (Math.random() > Number(config.sampling)) { continue; } // sampled out } // create: one PENDING JobExecution row in Postgres await prisma.jobExecution.create({ data: { id: jobExecutionId, projectId, jobConfigurationId: config.id, jobInputTraceId: event.traceId, jobTemplateId: config.evalTemplateId, status: "PENDING", startTime: new Date() } }); // gate3 delay: push to the execution queue with config.delay ms await EvalExecutionQueue.getInstance({ shardingKey })?.add( QueueName.EvaluationExecution, { payload: { projectId, jobExecutionId, delay: config.delay } }, { delay: config.delay }); // milliseconds
There's also a "reverse gate": if a trace later no longer matches the evaluator's filter (a second trace event changed its attributes, kicking it out of the target set) and it already had a ticket, the system marks that un-run ticket CANCELLED (via updateMany excluding COMPLETED, to avoid touching finished ones). The evaluation target set is dynamic—selected can become deselected, making scheduling resilient to "data arriving late, attributes changing late".
JobExecution is a state machine. It is born PENDING, with three possible endings (COMPLETED/ERROR/CANCELLED); in between it may go DELAYED on rate-limit and retry. Below is its full lifecycle:
JobExecutionStatus = { COMPLETED, ERROR, PENDING, CANCELLED, DELAYED } (schema.prisma:1080-1086). On success evalCompletion.ts:82-92 writes the score into the ingestion queue, then marks the ticket COMPLETED and fills jobOutputScoreId and executionTraceId—the ticket is now bidirectionally linked to "the score it produced" and "the judge's own trace".// loop guard: internal traces from trace-upsert whose env starts with "langfuse" → create no eval ticket // otherwise: user trace → eval → eval's own trace → triggers eval again → nesting forever if (sourceEventType === "trace-upsert" && event.traceEnvironment?.startsWith("langfuse")) { return; // skip: this is a trace produced by evaluation itself } // finish: score into the ingestion queue, ticket marked COMPLETED and bidirectionally linked await deps.enqueueScoreIngestion({ projectId, scoreId, eventId }); await deps.updateJobExecution({ id: jobExecutionId, projectId, data: { status: "COMPLETED", endTime: new Date(), jobOutputScoreId, // ← links to "the score produced" executionTraceId: result.executionTraceId } }); // ← links to "the judge's own trace"
jobOutputScoreId and executionTraceId let you trace from "a score" back to "which ticket, which evaluator, which judge call produced it"—fully auditable end to end.langfuse- environment prefix and block them right at the creator's entry. A system that can observe everything must be careful not to observe itself into a loop—the classic trap of self-referential systems.
createEvalJobs, fanning one trace out to all ACTIVE evaluators in the project, creating tickets one-to-many.config.sampling probability to control cost), delay (config.delay to wait for data); plus a reverse "cancel" (a later event makes the trace no longer match → CANCELLED).InMemoryFilterService.evaluateFilter, going back to ClickHouse only for complex filters—saving repeated DB queries.jobOutputScoreId and executionTraceId—auditable end to end.langfuse- environment prefix blocks it at the creator's entry, avoiding "eval triggers eval" self-referential nesting.