上一课说 score 是评估的原子单位,也说了它有三种来源。这一课讲最有意思的那种来源:EVAL——用一个 LLM 去给另一个 LLM 的输出评分,业界叫 LLM-as-a-judge(LLM 当裁判)。 人工标注准但慢且贵,规则匹配快但只能查表面。让一个「裁判 LLM」读你的 trace、按你给的标准评分,就能自动、可扩展地评估「这次回答到底好不好」。这一课拆开 Langfuse 的裁判流水线: 模板(评分标准)、变量映射(把 trace 的哪些部分递给裁判看)、结构化输出(裁判必须返回规整的分数 + 理由),以及最关键的——评出来的分怎么变回一个 score。
{{question}}/{{answer}})。
评委看完,必须在规定表格里填「分数 + 评语」(结构化输出,不能写成一段散文让人没法录入)。最后这张评分表被归档进成绩册(评出的分作为一个 source=EVAL 的 score,走回第 12 课那条摄取链路,落进 scores 表)。
整套流程的妙处:评委本身也是一次 LLM 调用,所以它自己也被 Langfuse 记成一条 trace——裁判也在被观测。
Langfuse 把 LLM-as-a-judge 拆成三件可复用的东西:模板(评什么、怎么评、用哪个模型)、评估器(把模板绑到哪些 trace、变量怎么取、分数叫什么名)、以及一次次的执行(真正去取数据、调裁判、产出分数)。下图是一条 trace 走完整条流水线的样子:
worker/src/features/evaluation/evalService.ts 的 runLLMAsJudgeEvaluation(:735)串起④⑤,createEvalJobs(:180,下一课)负责①②③ 的调度。input/output 填进一个裁判模板,发给一个(通常更便宜的)裁判模型,并强制它只返回结构化 JSON(score + reasoning)。结构化是关键——只有规整的分才能以 source=EVAL 写回 scores 表、被聚合比较。理由(reasoning)则让你知道「为什么是这分」。{{变量}} 的 prompt(如「就有用性给 0–1 分并说明理由」)、一个输出定义(产出 NUMERIC/CATEGORICAL/BOOLEAN 哪种分、范围多少),外加用哪个 provider/model 当裁判。模板只描述「怎么评」,不绑定具体数据。{{answer}} 该取 trace 的哪一列),以及最终分数叫什么名(scoreName)。模板里写的是 {{question}} {{answer}} 这样的占位符——它们怎么变成真实内容?靠变量映射:每个模板变量声明它来自哪个 langfuseObject(trace / observation / dataset_item)的哪一列(selectedColumnId,如 input/output/metadata)。执行时,extractVariablesFromTracingData 按映射去真实的 trace 里把值一个个取出来。
{{question}} → 用户的问题{{answer}} → 模型的回答{{context}} → 检索到的资料
模板只认名字,不关心数据从哪来。
question ← trace.inputanswer ← trace.outputcontext ← observation[retriever].output
每条映射 = {templateVariable, langfuseObject, selectedColumnId}。
对每个变量找到映射,去真实 trace/observation/dataset_item 取那一列的值,得到 {var, value}。取不到就退化为空串并告警,不让整次评估崩。
// 1) 按映射从真实 trace 取每个变量的值(evalService.ts:1152) for (const variable of variables) { const mapping = variableMapping.find(m => m.templateVariable === variable); if (mapping.langfuseObject === "trace") { // 也可能是 observation / dataset_item const trace = await getTraceById({ traceId, projectId, timestamp }); results.push({ var: variable, value: parseDatabaseRowValue(trace, mapping) }); } } // 2) 把取到的值填进模板的 {{占位符}}(evalRuntime.ts:14) export function compileEvalPrompt(params) { const variableMap = Object.fromEntries( params.variables.map(({ var: key, value }) => [key, parseUnknownToString(value)])); return compileTemplateString(params.templatePrompt, variableMap); // {{answer}} → 实际回答 }
注意一个细节:提取阶段保留原始数据形状(给代码 eval 用,第 31 课),只有在 compileEvalPrompt 这个「喂给裁判 LLM」的边界上才用 parseUnknownToString 拍平成字符串——因为 LLM 的 prompt 终究是文本。这种「在边界处才转换、内部保留原样」的克制,是干净数据流的标志。
裁判 LLM 不能只回一段散文——那样没法录入。Langfuse 用结构化输出(structured output)强制它返回规整的 JSON:按模板的输出定义,要么 {score: 数, reasoning}(NUMERIC/BOOLEAN),要么 {matches: [...], reasoning}(CATEGORICAL)。拿到后 toNormalizedScores 把它归一成 score:value 是分、comment 是裁判的理由。
evalScoreEvent.ts:52 把结果包成 SCORE_CREATE 且 source: ScoreSourceEnum.EVAL,再走第 12 课的摄取链路。所以裁判产出的分天然与其它来源同表、同尺、可比。// 归一:value=分,comment=裁判的理由(evalService.ts: toNormalizedScores) const baseFields = { name: scoreName, comment: outputResult.reasoning }; if (outputResult.dataType === "NUMERIC") return [{ ...baseFields, dataType: "NUMERIC", value: outputResult.score }]; if (outputResult.dataType === "BOOLEAN") return [{ ...baseFields, dataType: "BOOLEAN", value: outputResult.score ? 1 : 0 }]; // 回流:包成与手写 score 完全相同的 SCORE_CREATE 事件(evalScoreEvent.ts) { type: eventTypes.SCORE_CREATE, body: { name: score.name, value: score.value, dataType: score.dataType, comment: score.comment, configId: score.configId, source: ScoreSourceEnum.EVAL, // ← 唯一的「来源」标记 executionTraceId: params.executionTraceId } }
| 输出定义 | 裁判返回 | 归一成 score | 产几条分 |
|---|---|---|---|
| NUMERIC | { score: 数, reasoning } | value = score | 1 条 |
| BOOLEAN | { score: 真/假, reasoning } | value = score ? 1 : 0 | 1 条 |
| CATEGORICAL | { matches: [标签…], reasoning } | 每个 match 一条,value = 该标签 | 可多条 |
三种输出定义对应模板里你选的分数类型;注意 CATEGORICAL 的 matches 是数组——一次评判可以打出多条分类分(如同时命中「事实正确」与「语气友好」两个标签)。无论哪种,reasoning 都被原样塞进每条 score 的 comment。源码:evalService.ts:953-997 的 toNormalizedScores。
{score, reasoning} 的 schema,让分数能进 scores 表(可聚合),又把「为什么给这个分」留在 comment 里(可追溯)——两全。SCORE_CREATE 事件、走第 12 课那条链路,
于是去重、合并、落 ClickHouse 的逻辑只写一遍、对所有来源一致。评估因此不是「另一个系统」,而是摄取管道的又一个上游。还有个优雅的递归:裁判自己的那次 LLM 调用,也被记成一条 Langfuse trace(环境标为 LLMJudge)——观测平台连自己的裁判都一并观测了。
{{占位符}} 各自声明来自 trace/observation/dataset_item 的哪一列;extractVariablesFromTracingData 执行时去真实数据里取值,compileEvalPrompt 在「喂给 LLM」的边界才拍平成字符串。{score, reasoning}(或分类的 {matches, reasoning});toNormalizedScores 归一为 value=分、comment=理由——可聚合又可追溯。SCORE_CREATE 事件走第 12 课链路。评估是摄取管道的又一个生产者,不是独立系统;裁判自身的调用也被记成 trace。Last lesson said a score is evaluation's atomic unit, and that it has three sources. This lesson covers the most interesting source: EVAL—using one LLM to score another LLM's output, known industry-wide as LLM-as-a-judge. Human labeling is accurate but slow and costly; rule matching is fast but only checks the surface. Letting a "judge LLM" read your trace and score it against your criteria gives you automatic, scalable assessment of "was this answer actually good". This lesson takes apart Langfuse's judge pipeline: the template (the rubric), the variable mapping (which parts of the trace to hand the judge), the structured output (the judge must return a tidy score + reasoning), and most crucially—how the verdict turns back into a score.
{{question}}/{{answer}}).
After reading, the judge must fill in a fixed form with "score + comment" (structured output—not a prose blob no one can record). Finally that scoring sheet is filed into the gradebook (the verdict becomes a source=EVAL score, flowing back through Lesson 12's ingestion path into the scores table).
The neat part: the judge is itself an LLM call, so it too is recorded as a Langfuse trace—the judge is also observed.
Langfuse splits LLM-as-a-judge into three reusable things: a template (what to rate, how, with which model), an evaluator (which traces to bind the template to, how variables are fetched, what the score is named), and repeated executions (actually fetching data, calling the judge, producing scores). Below is one trace's journey through the whole pipeline:
worker/src/features/evaluation/evalService.ts's runLLMAsJudgeEvaluation (:735) wires ④⑤; createEvalJobs (:180, next lesson) schedules ①②③.input/output are filled into a judge template and sent to a (usually cheaper) judge model, forced to return only structured JSON (score + reasoning). Structure is the point — only a regular score can be written back with source=EVAL and aggregated. The reasoning tells you why it scored that way.{{variables}} (e.g. "rate helpfulness 0–1 with reasoning"), an output definition (which score it produces—NUMERIC/CATEGORICAL/BOOLEAN, what range), plus which provider/model judges. A template only describes "how to rate", binding no specific data.{{answer}} reads), and what the final score is named (scoreName).The template holds placeholders like {{question}} {{answer}}—how do they become real content? Via the variable mapping: each template variable declares which langfuseObject (trace / observation / dataset_item) and which column (selectedColumnId, e.g. input/output/metadata) it comes from. At execution, extractVariablesFromTracingData pulls each value from the real trace per the mapping.
{{question}} → the user's question{{answer}} → the model's answer{{context}} → retrieved material
The template only knows names, not where data comes from.
question ← trace.inputanswer ← trace.outputcontext ← observation[retriever].output
Each mapping = {templateVariable, langfuseObject, selectedColumnId}.
For each variable find its mapping, fetch that column's value from the real trace/observation/dataset_item, yielding {var, value}. If missing, degrade to an empty string with a warning—never crash the whole eval.
// 1) Pull each variable's value from the real trace per its mapping (evalService.ts:1152) for (const variable of variables) { const mapping = variableMapping.find(m => m.templateVariable === variable); if (mapping.langfuseObject === "trace") { // could also be observation / dataset_item const trace = await getTraceById({ traceId, projectId, timestamp }); results.push({ var: variable, value: parseDatabaseRowValue(trace, mapping) }); } } // 2) Fill the values into the template's {{placeholders}} (evalRuntime.ts:14) export function compileEvalPrompt(params) { const variableMap = Object.fromEntries( params.variables.map(({ var: key, value }) => [key, parseUnknownToString(value)])); return compileTemplateString(params.templatePrompt, variableMap); // {{answer}} → actual answer }
Note a detail: extraction preserves the original data shape (for code eval, Lesson 31), and only at the compileEvalPrompt boundary—the "feed to the judge LLM" boundary—is it flattened to a string via parseUnknownToString, because a prompt is ultimately text. This restraint—"convert at the boundary, keep raw internally"—is a hallmark of clean data flow.
The judge LLM can't just return prose—that can't be recorded. Langfuse uses structured output to force tidy JSON: per the template's output definition, either {score: number, reasoning} (NUMERIC/BOOLEAN) or {matches: [...], reasoning} (CATEGORICAL). Then toNormalizedScores normalizes it into a score: value is the score, comment is the judge's reasoning.
evalScoreEvent.ts:52 wraps the result as a SCORE_CREATE with source: ScoreSourceEnum.EVAL, then runs Lesson 12's ingestion path. So judge-produced scores are inherently same-table, same-ruler, comparable with other sources.// normalize: value=score, comment=judge's reasoning (evalService.ts: toNormalizedScores) const baseFields = { name: scoreName, comment: outputResult.reasoning }; if (outputResult.dataType === "NUMERIC") return [{ ...baseFields, dataType: "NUMERIC", value: outputResult.score }]; if (outputResult.dataType === "BOOLEAN") return [{ ...baseFields, dataType: "BOOLEAN", value: outputResult.score ? 1 : 0 }]; // flow back: wrap as the exact same SCORE_CREATE event as a hand-written score (evalScoreEvent.ts) { type: eventTypes.SCORE_CREATE, body: { name: score.name, value: score.value, dataType: score.dataType, comment: score.comment, configId: score.configId, source: ScoreSourceEnum.EVAL, // ← the one "source" marker executionTraceId: params.executionTraceId } }
| output definition | judge returns | normalized to score | scores produced |
|---|---|---|---|
| NUMERIC | { score: number, reasoning } | value = score | 1 |
| BOOLEAN | { score: true/false, reasoning } | value = score ? 1 : 0 | 1 |
| CATEGORICAL | { matches: [labels…], reasoning } | one per match, value = that label | may be many |
The three output definitions match the score type you pick in the template; note CATEGORICAL's matches is an array—one verdict can emit several categorical scores (e.g. hitting both "factually correct" and "friendly tone"). Either way, reasoning is dropped verbatim into each score's comment. Source: evalService.ts:953-997 toNormalizedScores.
{score, reasoning} schema lets the score enter the scores table (aggregatable) while keeping "why this score" in comment (traceable)—both at once.SCORE_CREATE event through Lesson 12's path,
so dedup, merge, and ClickHouse persistence are written once and consistent for every source. Evaluation is therefore not "another system" but another upstream of the ingestion pipeline. And an elegant recursion: the judge's own LLM call is recorded as a Langfuse trace (environment tagged LLMJudge)—the observability platform observes even its own judge.
{{placeholder}} declares which column of trace/observation/dataset_item it comes from; extractVariablesFromTracingData fetches real values at run, and compileEvalPrompt flattens to a string only at the "feed-to-LLM" boundary.{score, reasoning} (or {matches, reasoning} for categorical); toNormalizedScores normalizes to value=score, comment=reasoning—aggregatable and traceable.SCORE_CREATE event through Lesson 12's path. Evaluation is another producer for the ingestion pipeline, not a separate system; the judge's own call is also recorded as a trace.