🪢 Langfuse 图解教程Langfuse Visual Guide 第五部分 · 评估与评分Part 5 · Evaluation & Scoring 29 / 55
第五部分 · 评估与评分Part 5 · Evaluation & Scoring

LLM 当裁判LLM-as-a-judge

🧭 读前Before you start
先读 第28课 的评分模型,本课讲用 LLM 当裁判自动评分。
Read Lesson 28 on the scoring model; this lesson covers using an LLM as a judge to score automatically.

上一课说 score 是评估的原子单位,也说了它有三种来源。这一课讲最有意思的那种来源:EVAL——用一个 LLM 去给另一个 LLM 的输出评分,业界叫 LLM-as-a-judge(LLM 当裁判)。 人工标注准但慢且贵,规则匹配快但只能查表面。让一个「裁判 LLM」读你的 trace、按你给的标准评分,就能自动、可扩展地评估「这次回答到底好不好」。这一课拆开 Langfuse 的裁判流水线: 模板(评分标准)、变量映射(把 trace 的哪些部分递给裁判看)、结构化输出(裁判必须返回规整的分数 + 理由),以及最关键的——评出来的分怎么变回一个 score。

📋 生活类比
想象一场作文比赛的评委。评分细则(模板 prompt)写着「请就『有用性』给 0–1 分,并说明理由」;但评委不会读整个学生档案,工作人员只把作文正文和题目递过去(变量映射:把 trace 的 input/output 填进模板的 {{question}}/{{answer}})。 评委看完,必须在规定表格里填「分数 + 评语」(结构化输出,不能写成一段散文让人没法录入)。最后这张评分表被归档进成绩册(评出的分作为一个 source=EVAL 的 score,走回第 12 课那条摄取链路,落进 scores 表)。 整套流程的妙处:评委本身也是一次 LLM 调用,所以它自己也被 Langfuse 记成一条 trace——裁判也在被观测。

裁判流水线:从一条 trace 到一个分数

Langfuse 把 LLM-as-a-judge 拆成三件可复用的东西:模板(评什么、怎么评、用哪个模型)、评估器(把模板绑到哪些 trace、变量怎么取、分数叫什么名)、以及一次次的执行(真正去取数据、调裁判、产出分数)。下图是一条 trace 走完整条流水线的样子:

裁判流水线:trace → 变量 → 裁判 LLM → score(EVAL) ① 模板 Templateprompt + 输出定义+ 模型/provider ② 评估器 Evaluator目标过滤 + 变量映射+ scoreName ③ 提取变量从 trace/observation/dataset 取列值 编译提示{{var}} 填实值 ④ 裁判 LLM结构化输出{ score, reasoning }自身也被记成 trace ⑤ score(source=EVAL)value + comment=理由 ⑥ 摄取链路(第12课)SCORE_CREATE → scores 表 配置一次(①②),就对每条命中的 trace 自动跑③④⑤⑥——评估是「持续的」,不是「一次性的」
三件套 + 六步:模板与评估器是配置(一次),提取变量/编译提示/调裁判/产出分数/回流摄取是执行(每条 trace 一次)。源码:worker/src/features/evaluation/evalService.ts 的 runLLMAsJudgeEvaluation(:735)串起④⑤,createEvalJobs(:180,下一课)负责①②③ 的调度。
示例:LLM 当裁判,返回结构化分数 裁判模板(prompt) 给「有用性」打 0–1 分: user 问:{trace.input}ai 答:{trace.output} 只返回 JSON: { "score":_, "reasoning":_ } 把被评 trace 的字段填进去 裁判 LLM便宜模型即可 结构化输出(必须规整) { "score": 0.8, "reasoning": "切题但略啰嗦…" } → score: source=EVAL 规整 → 才能进 scores 表聚合
裁判必须返回「规整的分」(值为示例):把被评 trace 的 input/output 填进一个裁判模板,发给一个(通常更便宜的)裁判模型,并强制它只返回结构化 JSON(score + reasoning)。结构化是关键——只有规整的分才能以 source=EVAL 写回 scores 表、被聚合比较。理由(reasoning)则让你知道「为什么是这分」。
①模板 Eval Template
可复用的「评分标准」:一段带 {{变量}} 的 prompt(如「就有用性给 0–1 分并说明理由」)、一个输出定义(产出 NUMERIC/CATEGORICAL/BOOLEAN 哪种分、范围多少),外加用哪个 provider/model 当裁判。模板只描述「怎么评」,不绑定具体数据。
②评估器 Evaluator(JobConfiguration)
把模板「落地」:用一个过滤条件圈定要评哪些 trace(如「只评 production 环境、name=chat 的」)、一份变量映射(模板里的 {{answer}} 该取 trace 的哪一列),以及最终分数叫什么名(scoreName)。
③–⑥执行 Execution(JobExecution)
每条命中的 trace 触发一次执行:提取变量 → 编译提示 → 调裁判 LLM 拿结构化结果 → 归一成 score → 经摄取链路写回。这条「执行」本身有状态生命周期(第 30 课细讲)。

变量映射:把 trace 的哪一部分递给裁判

模板里写的是 {{question}} {{answer}} 这样的占位符——它们怎么变成真实内容?靠变量映射:每个模板变量声明它来自哪个 langfuseObject(trace / observation / dataset_item)的哪一列(selectedColumnId,如 input/output/metadata)。执行时,extractVariablesFromTracingData 按映射去真实的 trace 里把值一个个取出来。

{{占位符}} 怎么变成真实内容:映射 → 提取 → 编译 ① 模板 {{question}} {{answer}} {{context}} ② 变量映射 question ← trace.input answer ← trace.output context ← obs[retriever] ③ 提取真实值 extractVariables… 去 trace 那一列取值 {var, value} ④ 编译填入 compileEvalPrompt → 喂给裁判 LLM 提取阶段保留原始数据形状(供代码 eval, L31) 仅在 compileEvalPrompt 喂 LLM 的边界,才 parseUnknownToString 拍平成字符串

模板变量(你写的占位符)

{{question}} → 用户的问题
{{answer}} → 模型的回答
{{context}} → 检索到的资料
模板只认名字,不关心数据从哪来。

映射(你配的对应关系)

question ← trace.input
answer ← trace.output
context ← observation[retriever].output
每条映射 = {templateVariable, langfuseObject, selectedColumnId}。

提取(执行时自动做)

对每个变量找到映射,去真实 trace/observation/dataset_item 取那一列的值,得到 {var, value}。取不到就退化为空串并告警,不让整次评估崩。

worker/src/features/evaluation/evalRuntime.ts · evalService.ts提取 + 编译
// 1) 按映射从真实 trace 取每个变量的值(evalService.ts:1152)
for (const variable of variables) {
  const mapping = variableMapping.find(m => m.templateVariable === variable);
  if (mapping.langfuseObject === "trace") {        // 也可能是 observation / dataset_item
    const trace = await getTraceById({ traceId, projectId, timestamp });
    results.push({ var: variable, value: parseDatabaseRowValue(trace, mapping) });
  }
}

// 2) 把取到的值填进模板的 {{占位符}}(evalRuntime.ts:14)
export function compileEvalPrompt(params) {
  const variableMap = Object.fromEntries(
    params.variables.map(({ var: key, value }) => [key, parseUnknownToString(value)]));
  return compileTemplateString(params.templatePrompt, variableMap);  // {{answer}} → 实际回答
}

注意一个细节:提取阶段保留原始数据形状(给代码 eval 用,第 31 课),只有在 compileEvalPrompt 这个「喂给裁判 LLM」的边界上才用 parseUnknownToString 拍平成字符串——因为 LLM 的 prompt 终究是文本。这种「在边界处才转换、内部保留原样」的克制,是干净数据流的标志。

结构化输出与回流:分数怎么变回一个 score

裁判 LLM 不能只回一段散文——那样没法录入。Langfuse 用结构化输出(structured output)强制它返回规整的 JSON:按模板的输出定义,要么 {score: 数, reasoning}(NUMERIC/BOOLEAN),要么 {matches: [...], reasoning}(CATEGORICAL)。拿到后 toNormalizedScores 把它归一成 score:value 是分、comment 是裁判的理由。

结构化输出 → 归一 → 回流摄取链路 裁判 LLM 的结构化输出{ score: 0.82, reasoning: "回答完整…" }schema 强校验,非自由散文 toNormalizedScoresvalue ← scorecomment ← reasoning SCORE_CREATE 事件source = EVALname/value/dataType/comment和你手写 score 同一种事件 scores 宽表三来源同表可比 关键:评估不是另开一条管道,而是「第 12 课摄取链路」的又一个生产者EVAL 分数和 API、ANNOTATION 分数走同一条 SCORE_CREATE 入口,享受同样的去重/合并/落库
评出来的分,和你亲手提交的分,是同一种事件。evalScoreEvent.ts:52 把结果包成 SCORE_CREATE 且 source: ScoreSourceEnum.EVAL,再走第 12 课的摄取链路。所以裁判产出的分天然与其它来源同表、同尺、可比。
worker/src/features/evaluation/evalService.ts:953 · evalScoreEvent.ts:52归一 + 回流
// 归一:value=分,comment=裁判的理由(evalService.ts: toNormalizedScores)
const baseFields = { name: scoreName, comment: outputResult.reasoning };
if (outputResult.dataType === "NUMERIC") return [{ ...baseFields, dataType: "NUMERIC", value: outputResult.score }];
if (outputResult.dataType === "BOOLEAN") return [{ ...baseFields, dataType: "BOOLEAN", value: outputResult.score ? 1 : 0 }];

// 回流:包成与手写 score 完全相同的 SCORE_CREATE 事件(evalScoreEvent.ts)
{ type: eventTypes.SCORE_CREATE,
  body: { name: score.name, value: score.value, dataType: score.dataType,
          comment: score.comment, configId: score.configId,
          source: ScoreSourceEnum.EVAL,        // ← 唯一的「来源」标记
          executionTraceId: params.executionTraceId } }
输出定义裁判返回归一成 score产几条分
NUMERIC{ score: 数, reasoning }value = score1 条
BOOLEAN{ score: 真/假, reasoning }value = score ? 1 : 01 条
CATEGORICAL{ matches: [标签…], reasoning }每个 match 一条,value = 该标签可多条

三种输出定义对应模板里你选的分数类型;注意 CATEGORICAL 的 matches 是数组——一次评判可以打出多条分类分(如同时命中「事实正确」与「语气友好」两个标签)。无论哪种,reasoning 都被原样塞进每条 score 的 comment。源码:evalService.ts:953-997 的 toNormalizedScores。

🎯 设计取舍
为什么裁判必须用「结构化输出」,而不是让它自由发挥? 因为评估的产物要能录入、聚合、比较。若裁判回一段「我觉得这个回答还不错,但开头有点啰嗦……」的散文,你既没法可靠地抽出一个分、也没法跨千条 trace 求平均。 强制 {score, reasoning} 的 schema,让分数能进 scores 表(可聚合),又把「为什么给这个分」留在 comment 里(可追溯)——两全。

更深一层:为什么评估结果要走回摄取链路,而不是直接写库? 这是 Langfuse 反复出现的信念——「一个入口,多个生产者」。无论分数来自 API、人工、还是裁判 LLM,都包成同一种 SCORE_CREATE 事件、走第 12 课那条链路, 于是去重、合并、落 ClickHouse 的逻辑只写一遍、对所有来源一致。评估因此不是「另一个系统」,而是摄取管道的又一个上游。还有个优雅的递归:裁判自己的那次 LLM 调用,也被记成一条 Langfuse trace(环境标为 LLMJudge)——观测平台连自己的裁判都一并观测了。
🎯 本课要点
  • LLM-as-a-judge = 用一个 LLM 给另一个 LLM 的输出评分:在人工(准但慢)与规则(快但浅)之间,提供自动、可扩展的质量评估。
  • 三件套:模板(prompt+输出定义+模型,描述「怎么评」)、评估器(过滤目标+变量映射+scoreName,描述「评谁」)、执行(每条 trace 实际跑一次)。
  • 变量映射:模板的 {{占位符}} 各自声明来自 trace/observation/dataset_item 的哪一列;extractVariablesFromTracingData 执行时去真实数据里取值,compileEvalPrompt 在「喂给 LLM」的边界才拍平成字符串。
  • 结构化输出:裁判被强制返回 {score, reasoning}(或分类的 {matches, reasoning});toNormalizedScores 归一为 value=分、comment=理由——可聚合又可追溯。
  • 回流摄取(source=EVAL):结果包成与手写 score 相同的 SCORE_CREATE 事件走第 12 课链路。评估是摄取管道的又一个生产者,不是独立系统;裁判自身的调用也被记成 trace。

🧪 自测 · 想一想为什么这么设计

1. Langfuse 的 LLM-as-a-judge 强制裁判 LLM 用「结构化输出」返回 {score, reasoning},而不是让它自由写一段评语。这么做的核心原因是?
  1. 让裁判模型必须用 GPT-4
  2. 结构化输出让 LLM 调用更便宜
  3. 可以不调用 LLM
  4. 评估的产物要能录入、聚合、比较:分必须能进 scores 表跨千条 trace 求平均,理由留在 comment 里可追溯——散文两样都做不到
看答案与解析 点击展开
答案:D。若裁判回一段散文,既没法可靠抽出一个数值分、也没法跨大量 trace 聚合。强制 {score, reasoning} 的 schema 让 value 进 scores 表(可聚合)、reasoning 进 comment(可追溯),两全。toNormalizedScores 正是据此把输出归一成 score(value=分,comment=理由)。
2. 裁判 LLM 评出的分,最终是怎么变成数据库里一条可查询的 score 的?
  1. 存进一张专门的 eval 结果表
  2. 直接 INSERT 进 ClickHouse,绕过摄取链路
  3. 包成 source=EVAL 的 SCORE_CREATE 事件,走第 12 课那条和手写 score 完全相同的摄取链路——评估是摄取管道的又一个生产者,不是独立系统
  4. 只存在内存里供仪表盘读取
看答案与解析 点击展开
答案:C。evalScoreEvent.ts 把归一后的结果包成 eventTypes.SCORE_CREATE、标 source=ScoreSourceEnum.EVAL,再走摄取链路。于是去重、合并、落 ClickHouse 的逻辑只写一遍、对 API/EVAL/ANNOTATION 三来源一致。这正是「一个入口,多个生产者」的体现,也让裁判分天然与其它来源同表可比。
3. 模板里写的是 {{question}}、{{answer}} 这样的占位符。执行时它们怎么变成真实内容?
  1. 用固定的示例数据
  2. 裁判 LLM 自己去数据库查
  3. 前端把内容硬编码进模板
  4. 靠变量映射:每个模板变量声明来自 trace/observation/dataset_item 的哪一列,extractVariablesFromTracingData 去真实数据取值,compileEvalPrompt 再填进占位符
看答案与解析 点击展开
答案:D。每条映射是 {templateVariable, langfuseObject, selectedColumnId}。执行时按映射从真实 trace/observation/dataset_item 取那一列的值,得到 {var, value};compileEvalPrompt 在「喂给 LLM」的边界用 parseUnknownToString 拍平成字符串再填进 {{占位符}}。提取阶段保留原始形状(给代码 eval 用),只在边界处转换。
💭 发散思考(没有标准答案,动手或动脑想想)
  • LLM-as-a-judge 用一个 LLM 评判另一个 LLM——但裁判自己也会犯错、也有偏好。你会怎么验证「裁判本身是否可靠」?(提示:Langfuse 把裁判的每次调用也记成一条 trace,且评出的分能和人工标注的分同表比较。)这对「用 AI 评 AI」的可信度建设意味着什么?

Last lesson said a score is evaluation's atomic unit, and that it has three sources. This lesson covers the most interesting source: EVAL—using one LLM to score another LLM's output, known industry-wide as LLM-as-a-judge. Human labeling is accurate but slow and costly; rule matching is fast but only checks the surface. Letting a "judge LLM" read your trace and score it against your criteria gives you automatic, scalable assessment of "was this answer actually good". This lesson takes apart Langfuse's judge pipeline: the template (the rubric), the variable mapping (which parts of the trace to hand the judge), the structured output (the judge must return a tidy score + reasoning), and most crucially—how the verdict turns back into a score.

📋 Analogy
Picture a judge at an essay contest. The scoring rubric (template prompt) says "rate 'helpfulness' 0–1 and explain why"; but the judge doesn't read the whole student file—staff hand over only the essay body and the prompt (variable mapping: fill the trace's input/output into the template's {{question}}/{{answer}}). After reading, the judge must fill in a fixed form with "score + comment" (structured output—not a prose blob no one can record). Finally that scoring sheet is filed into the gradebook (the verdict becomes a source=EVAL score, flowing back through Lesson 12's ingestion path into the scores table). The neat part: the judge is itself an LLM call, so it too is recorded as a Langfuse trace—the judge is also observed.

The judge pipeline: from one trace to one score

Langfuse splits LLM-as-a-judge into three reusable things: a template (what to rate, how, with which model), an evaluator (which traces to bind the template to, how variables are fetched, what the score is named), and repeated executions (actually fetching data, calling the judge, producing scores). Below is one trace's journey through the whole pipeline:

judge pipeline: trace → variables → judge LLM → score(EVAL) ① Templateprompt + output def+ model/provider ② Evaluatortarget filter + var mapping+ scoreName ③ extract varsfrom trace/observation/dataset columns compile prompt{{var}} → real value ④ judge LLMstructured output{ score, reasoning }itself recorded as a trace ⑤ score(source=EVAL)value + comment=reasoning ⑥ ingestion path (L12)SCORE_CREATE → scores table configure once (①②) and ③④⑤⑥ run automatically per matching trace—evaluation is "continuous", not "one-off"
Three pieces + six steps: template and evaluator are config (once); extract/compile/judge/produce/flow-back are execution (once per trace). Source: worker/src/features/evaluation/evalService.ts's runLLMAsJudgeEvaluation (:735) wires ④⑤; createEvalJobs (:180, next lesson) schedules ①②③.
Example: an LLM judge returns a structured score judge template (prompt) Rate helpfulness 0–1: user: {trace.input}ai: {trace.output} Return only JSON: { "score":_, "reasoning":_ } fill in the evaluated trace's fields judge LLMa cheap model is fine structured output (must be regular) { "score": 0.8, "reasoning": "on-topic but verbose…" } → score: source=EVAL regular → can enter the scores table
The judge must return a "regular score" (values illustrative): the evaluated trace's input/output are filled into a judge template and sent to a (usually cheaper) judge model, forced to return only structured JSON (score + reasoning). Structure is the point — only a regular score can be written back with source=EVAL and aggregated. The reasoning tells you why it scored that way.
①Eval Template
A reusable "rubric": a prompt with {{variables}} (e.g. "rate helpfulness 0–1 with reasoning"), an output definition (which score it produces—NUMERIC/CATEGORICAL/BOOLEAN, what range), plus which provider/model judges. A template only describes "how to rate", binding no specific data.
②Evaluator (JobConfiguration)
"Grounds" the template: a filter picking which traces to evaluate (e.g. "only production, name=chat"), a variable mapping (which column the template's {{answer}} reads), and what the final score is named (scoreName).
③–⑥Execution (JobExecution)
Each matching trace triggers one execution: extract variables → compile prompt → call judge LLM for a structured result → normalize into a score → write back via the ingestion path. This "execution" has its own status lifecycle (detailed in Lesson 30).

Variable mapping: which part of the trace to hand the judge

The template holds placeholders like {{question}} {{answer}}—how do they become real content? Via the variable mapping: each template variable declares which langfuseObject (trace / observation / dataset_item) and which column (selectedColumnId, e.g. input/output/metadata) it comes from. At execution, extractVariablesFromTracingData pulls each value from the real trace per the mapping.

how {{placeholders}} become real content: mapping → extract → compile 1 template {{question}} {{answer}} {{context}} 2 variable mapping question ← trace.input answer ← trace.output context ← obs[retriever] 3 extract values extractVariables… read that trace column {var, value} 4 compile compileEvalPrompt → to judge LLM extraction preserves the raw data shape (for code eval, L31) only at compileEvalPrompt (the LLM boundary) is it parseUnknownToString-flattened to text

Template variables (your placeholders)

{{question}} → the user's question
{{answer}} → the model's answer
{{context}} → retrieved material
The template only knows names, not where data comes from.

Mapping (the correspondence you configure)

question ← trace.input
answer ← trace.output
context ← observation[retriever].output
Each mapping = {templateVariable, langfuseObject, selectedColumnId}.

Extraction (done automatically at run)

For each variable find its mapping, fetch that column's value from the real trace/observation/dataset_item, yielding {var, value}. If missing, degrade to an empty string with a warning—never crash the whole eval.

worker/src/features/evaluation/evalRuntime.ts · evalService.tsextract + compile
// 1) Pull each variable's value from the real trace per its mapping (evalService.ts:1152)
for (const variable of variables) {
  const mapping = variableMapping.find(m => m.templateVariable === variable);
  if (mapping.langfuseObject === "trace") {        // could also be observation / dataset_item
    const trace = await getTraceById({ traceId, projectId, timestamp });
    results.push({ var: variable, value: parseDatabaseRowValue(trace, mapping) });
  }
}

// 2) Fill the values into the template's {{placeholders}} (evalRuntime.ts:14)
export function compileEvalPrompt(params) {
  const variableMap = Object.fromEntries(
    params.variables.map(({ var: key, value }) => [key, parseUnknownToString(value)]));
  return compileTemplateString(params.templatePrompt, variableMap);  // {{answer}} → actual answer
}

Note a detail: extraction preserves the original data shape (for code eval, Lesson 31), and only at the compileEvalPrompt boundary—the "feed to the judge LLM" boundary—is it flattened to a string via parseUnknownToString, because a prompt is ultimately text. This restraint—"convert at the boundary, keep raw internally"—is a hallmark of clean data flow.

Structured output and flow-back: how a verdict becomes a score

The judge LLM can't just return prose—that can't be recorded. Langfuse uses structured output to force tidy JSON: per the template's output definition, either {score: number, reasoning} (NUMERIC/BOOLEAN) or {matches: [...], reasoning} (CATEGORICAL). Then toNormalizedScores normalizes it into a score: value is the score, comment is the judge's reasoning.

structured output → normalize → flow back to ingestion judge LLM structured output{ score: 0.82, reasoning: "complete…" }schema-enforced, not free prose toNormalizedScoresvalue ← scorecomment ← reasoning SCORE_CREATE eventsource = EVALname/value/dataType/commentsame event as a hand-written score scores table3 sources, comparable Key: evaluation isn't a separate pipeline—it's another producer for Lesson 12's ingestion pathEVAL scores share the SCORE_CREATE entry with API & ANNOTATION—same dedup/merge/persist
A judged score and a hand-submitted score are the same event. evalScoreEvent.ts:52 wraps the result as a SCORE_CREATE with source: ScoreSourceEnum.EVAL, then runs Lesson 12's ingestion path. So judge-produced scores are inherently same-table, same-ruler, comparable with other sources.
worker/src/features/evaluation/evalService.ts:953 · evalScoreEvent.ts:52normalize + flow back
// normalize: value=score, comment=judge's reasoning (evalService.ts: toNormalizedScores)
const baseFields = { name: scoreName, comment: outputResult.reasoning };
if (outputResult.dataType === "NUMERIC") return [{ ...baseFields, dataType: "NUMERIC", value: outputResult.score }];
if (outputResult.dataType === "BOOLEAN") return [{ ...baseFields, dataType: "BOOLEAN", value: outputResult.score ? 1 : 0 }];

// flow back: wrap as the exact same SCORE_CREATE event as a hand-written score (evalScoreEvent.ts)
{ type: eventTypes.SCORE_CREATE,
  body: { name: score.name, value: score.value, dataType: score.dataType,
          comment: score.comment, configId: score.configId,
          source: ScoreSourceEnum.EVAL,        // ← the one "source" marker
          executionTraceId: params.executionTraceId } }
output definitionjudge returnsnormalized to scorescores produced
NUMERIC{ score: number, reasoning }value = score1
BOOLEAN{ score: true/false, reasoning }value = score ? 1 : 01
CATEGORICAL{ matches: [labels…], reasoning }one per match, value = that labelmay be many

The three output definitions match the score type you pick in the template; note CATEGORICAL's matches is an array—one verdict can emit several categorical scores (e.g. hitting both "factually correct" and "friendly tone"). Either way, reasoning is dropped verbatim into each score's comment. Source: evalService.ts:953-997 toNormalizedScores.

🎯 Design trade-off
Why must the judge use "structured output" instead of free-forming? Because the product of evaluation must be recordable, aggregatable, comparable. If the judge returns prose like "I think this answer is decent but the opening is wordy…", you can neither reliably extract a score nor average across a thousand traces. Forcing a {score, reasoning} schema lets the score enter the scores table (aggregatable) while keeping "why this score" in comment (traceable)—both at once.

Deeper: why do eval results flow back through the ingestion path rather than write the DB directly? This is a recurring Langfuse belief—"one entry, many producers". Whether a score comes from API, human, or a judge LLM, it's wrapped as the same SCORE_CREATE event through Lesson 12's path, so dedup, merge, and ClickHouse persistence are written once and consistent for every source. Evaluation is therefore not "another system" but another upstream of the ingestion pipeline. And an elegant recursion: the judge's own LLM call is recorded as a Langfuse trace (environment tagged LLMJudge)—the observability platform observes even its own judge.
🎯 Key points
  • LLM-as-a-judge = use one LLM to score another's output: between human (accurate but slow) and rules (fast but shallow), it offers automatic, scalable quality assessment.
  • Three pieces: template (prompt+output def+model, "how to rate"), evaluator (target filter+variable mapping+scoreName, "whom to rate"), execution (actually runs once per trace).
  • Variable mapping: each template {{placeholder}} declares which column of trace/observation/dataset_item it comes from; extractVariablesFromTracingData fetches real values at run, and compileEvalPrompt flattens to a string only at the "feed-to-LLM" boundary.
  • Structured output: the judge is forced to return {score, reasoning} (or {matches, reasoning} for categorical); toNormalizedScores normalizes to value=score, comment=reasoning—aggregatable and traceable.
  • Flow back (source=EVAL): the result is wrapped as the same SCORE_CREATE event through Lesson 12's path. Evaluation is another producer for the ingestion pipeline, not a separate system; the judge's own call is also recorded as a trace.

🧪 Self-test - think about the design

1. Langfuse's LLM-as-a-judge forces the judge LLM to return {score, reasoning} via 'structured output' rather than free prose. The core reason is?
  1. it forces the judge to use GPT-4
  2. structured output makes the LLM call cheaper
  3. it avoids calling the LLM
  4. the product of evaluation must be recordable, aggregatable, comparable: the score must enter the scores table to average across thousands of traces, with the reasoning kept in comment for traceability — prose does neither
Show answer & explanation click to expand
Answer: D. If the judge returns prose, you can't reliably extract a numeric score nor aggregate across many traces. Forcing a {score, reasoning} schema puts value into the scores table (aggregatable) and reasoning into comment (traceable) — both at once. toNormalizedScores normalizes the output into a score exactly this way (value=score, comment=reasoning).
2. How does the judge LLM's verdict ultimately become a queryable score in the database?
  1. stored in a dedicated eval-results table
  2. directly INSERTed into ClickHouse, bypassing ingestion
  3. wrapped as a source=EVAL SCORE_CREATE event through the exact same Lesson-12 ingestion path as a hand-written score — evaluation is another producer for the ingestion pipeline, not a separate system
  4. kept in memory only for dashboards
Show answer & explanation click to expand
Answer: C. evalScoreEvent.ts wraps the normalized result as eventTypes.SCORE_CREATE tagged source=ScoreSourceEnum.EVAL, then runs the ingestion path. So dedup/merge/ClickHouse-persist are written once and consistent across API/EVAL/ANNOTATION. This is 'one entry, many producers', and makes judge scores inherently comparable in the same table.
3. The template holds placeholders like {{question}}, {{answer}}. How do they become real content at execution?
  1. with fixed sample data
  2. the judge LLM queries the database itself
  3. the frontend hardcodes content into the template
  4. via variable mapping: each template variable declares which column of trace/observation/dataset_item it comes from; extractVariablesFromTracingData fetches real values, then compileEvalPrompt fills the placeholders
Show answer & explanation click to expand
Answer: D. Each mapping is {templateVariable, langfuseObject, selectedColumnId}. At run it pulls that column's value from the real trace/observation/dataset_item into {var, value}; compileEvalPrompt flattens to a string via parseUnknownToString at the 'feed-to-LLM' boundary before filling {{placeholders}}. Extraction keeps original shapes (for code eval), converting only at the boundary.
💭 Open questions (no single right answer - just think or try)
  • LLM-as-a-judge uses one LLM to judge another — but the judge itself errs and has biases. How would you validate 'whether the judge itself is reliable'? (Hint: Langfuse records each judge call as a trace, and judge scores sit in the same table as human-annotation scores for comparison.) What does this imply for building trust in 'AI judging AI'?