🪢 Langfuse 图解教程Langfuse Visual Guide 第五部分 · 评估与评分Part 5 · Evaluation & Scoring 28 / 55
第五部分 · 评估与评分Part 5 · Evaluation & Scoring

评分模型The scoring model

🧭 读前Before you start
先读 第3课 的 score 维度,本课深入评分模型的数据结构。
Read Lesson 3 on the score dimensions; this lesson dives into the scoring model's data structures.

前四部分让你能看清每一次 LLM 调用发生了什么。但「发生了什么」不等于「做得好不好」。这一部分讲评估——怎么给 trace 评分。而一切的原子单位,是 score(评分)。 这一课先把 score 的数据模型讲透:一个 score 长什么样、有哪三种数据类型(数值/分类/布尔)、score config 这个「评分的 schema」管什么,以及一个 score 可能来自哪三种来源。 理解了这套模型,后面五课(LLM 评判、代码 eval、人工标注、监控)就都只是「用不同方式生产 score」而已。

📋 生活类比
把 score 想成给一次回答贴的一张评价标签。标签上写着评什么(name,如「有用性」)、打多少(value)、以及这是哪种刻度(dataType:是 0–1 的分数?还是「好/中/差」的档位?还是「对/错」的勾叉?)。 而 score config 就像这类标签的印制规范:它规定「有用性」这张标签必须是 0–1 的数值、「情感」那张必须从 {正/中/负} 里选一个——于是同名标签永远同一把尺,才能横向比较、求平均。 至于标签是谁贴的(source):可能是你亲手贴(API)、可能是 AI 评判员贴(EVAL)、也可能是人工审核贴(ANNOTATION)——但只要刻度一致,三方贴的标签就活在同一个可比的空间里。

三种刻度:数值、分类、布尔

一个 score 的灵魂是它的 dataType——它决定 value 怎么解读、怎么校验。Langfuse 用 Zod 把三种类型定义得清清楚楚(外加一个旧的 TEXT):

score 的三种刻度(dataType) NUMERIC 数值value 是一个数可设 minValue / maxValue例:有用性 0–1、延迟分 1–5"helpfulness": 0.82可求平均、可排序 CATEGORICAL 分类一组「标签 ↔ 值」标签唯一、值唯一例:情感 {正/中/负}"sentiment": "positive"可计数、可分组 BOOLEAN 布尔特殊的分类固定两档:True=1 / False=0例:是否含敏感词"is_toxic": False (0)可算通过率
三种刻度,各有所长:NUMERIC(数,带可选 min/max,能求平均/排序)、CATEGORICAL(标签↔值的固定集合,标签与值都唯一,能计数/分组)、BOOLEAN(固定为 True=1 / False=0 的特殊分类,能算通过率)。源码:packages/shared/src/domain/score-configs.ts:18-80。
三种来源,一张 scores 表 API程序直接上报 EVALLLM 裁判/代码评估 ANNOTATION人工标注 scores 表(同一形状) name: "helpfulness" value: 0.9 // 或 stringValue(分类) dataType: NUMERIC | CATEGORICAL | BOOLEAN source: API | EVAL | ANNOTATION traceId / observationId · comment → 同名同尺度才可比:不管谁打的,都进同一空间聚合 这也是 source 字段的意义:留住「这分谁打的」
三源同表(来源枚举取自 domain/scores.ts;值为示例):score 有三种来源——API(程序上报)、EVAL(LLM 裁判或代码评估)、ANNOTATION(人工标注),但它们都落进同一张 scores 表、共享同一形状(name / value 或 stringValue / dataType / source)。于是「机器打的」「人审的」「API 报的」活在一个空间里,同名同尺度即可比。
packages/shared/src/domain/score-configs.ts三种可比刻度
// 数值:可选区间,无 categories
NumericConfigFields = { dataType: "NUMERIC", minValue?, maxValue? };

// 分类:一组 {label, value},标签唯一、值唯一(validateCategories)
CategoricalConfigFields = { dataType: "CATEGORICAL", categories: [{label, value}, …] };

// 布尔:其实是「恰好两档」的特殊分类,且锁死为 True=1 / False=0
BooleanConfigFields = { dataType: "BOOLEAN",
  categories: [{label:"True", value:1}, {label:"False", value:0}] };  // length===2 强校验

注意一个优雅之处:布尔不是独立的第四种类型,而是被实现成「恰好两档、且锁死为 True=1/False=0」的分类。这样底层只需处理「数值」和「分类」两套逻辑,布尔顺势复用分类的存储与聚合—— 少一套特例,多一分一致。这也呼应了第 8 课「宽事件」的思路:能用通用机制表达的,就别新造一个。

还有个贯穿始终的设计:value 永远是一个数,而 stringValue 只用来存「标签」。NUMERIC 的 value 就是那个数、stringValue 为空;CATEGORICAL/BOOLEAN 则把标签放进 stringValue、把映射后的数放进 value(如 "positive"→某数、"True"→1)。 正因为任何 score 都有一个数值 value,求平均、画趋势、算通过率才对所有类型一视同仁——这是「可比」的物理基础。

dataTypevalue / stringValue约束能做的聚合例子
NUMERICvalue=那个数;stringValue 空可选 minValue / maxValue均值、分布、趋势helpfulness 0.82
CATEGORICALstringValue=标签;value=映射数标签唯一、值唯一计数、占比、分组sentiment "positive"
BOOLEANstringValue="True"/"False";value∈{0,1}锁死两档 True=1 / False=0通过率is_toxic False (0)

提醒:scores.ts 里 dataType 其实列了 5 种——上面 3 种是 score config 支持的「可比刻度」,另有 TEXT(自由文本,长度 ≤500,旧式)和 CORRECTION(人工标注纠正专用,name 固定为 "output")。后两者不是可聚合的刻度,故本课聚焦前三。源码:scores.ts:46-52(5 种)、score-configs.ts:117-138(config 仅 4 种 discriminatedUnion)。

score config:一张「评分的 schema」

光有 dataType 还不够。如果同一个「有用性」分数,今天有人打 0–1、明天有人打 1–100,数据就乱成一锅。score config 就是解决这个的:它是某个 score name 的 schema—— 声明这个名字下的分数是什么 dataType、什么区间/有哪些类别。于是无论分数从哪来,都按同一把尺校验,才谈得上比较与聚合。

一份 config 把守一个 name:三来源、同一把尺 API(你亲手提交)source=API EVAL(AI/代码评判)source=EVAL ANNOTATION(人工)source=ANNOTATION score config(name 的 schema)"有用性" = NUMERIC [0,1]校验 value 是否合规 scores 表(第 8 课宽表)同名同尺 → 可比、可聚合 三种来源殊途同归:都按 config 校验、都写进同一张 scores 表——评判/人工/API 活在同一个可比空间
一份 config 守一个 name:score config 声明某 name 的 dataType 与约束(数值区间 / 类别集合),校验来自 API / EVAL / ANNOTATION 三种来源的分数,使它们落进同一张 scores 表、同一把尺。源码:scores.ts:4(ScoreSourceArray=["API","EVAL","ANNOTATION"])。
API你亲手提交
用 SDK 直接 score.create——比如你自己的离线评测脚本、或线上收集的用户点赞。这是第 6、12 课摄取链路里 SCORE_CREATE 事件的来路之一。
EVALAI 或代码评判
LLM-as-a-judge(第 29 课)或代码 eval(第 31 课)算出来的分。它们最终也走回同一条摄取链路写成 score——评估不是独立管道,而是 score 的生产者。
ANNOTATION人工标注
人类审核员在标注队列里(第 32 课)按 config 给 trace 评分。同样落进 scores 表,只是 source 标成 ANNOTATION。

一个 score 对象到底有哪些字段

把 ScoreSchema 拆开看,一条 score 大致分三组字段——「评什么·打多少」「谁打的·带什么」「挂在哪」。最值得记的是最后一组:一条 score 必须挂在 trace、session 或数据集运行 三者之一上(这三个引用字段恰有一个非空);observationId 不是平级的第四项,而是在 trace 内再定位到某一步(必须与 traceId 同在)。

评什么 · 打多少

name 评的是哪个维度(如「有用性」);value 永远是一个数;stringValue 标签(分类/布尔/文本才有);dataType 哪种刻度。

谁打的 · 带什么

source API / EVAL / ANNOTATION;authorUserId 人工标注者(可空);comment 备注(可空);configId 按哪份 config 校验;queueId 来自哪个标注队列。

挂在哪(三选一 + 可定位到步)

三选一:traceId 整条 trace、sessionId 整个会话、datasetRunId 某次数据集运行;另可用 observationId 在 traceId 之上再定位到某一步(需与 traceId 同在)。

这组「三选一(外加步内定位)的挂载点」很关键:它让评分既能贴在整条 trace(这次回答整体好不好)、也能用 observationId 贴到某一步(这步检索准不准)、还能贴在整个 session 或一次数据集运行(第 34、35 课的实验场景)。(特例:CORRECTION 类分数只能挂 trace/observation,不能挂 session/datasetRun。)同一个 score 模型,覆盖了从「单步」到「整批实验」的所有评估粒度。校验规则见 packages/shared/src/utils/scores.ts 的 applyScoreValidation(字段定义 domain/scores.ts)。

一条 score traceId = t-42 ✓ observationId = null sessionId = null datasetRunId = null 约束:trace/session/run 三选一 trace · 整条回答好不好 observation · 某一步准不准 session · 整个会话 数据集运行 · 一次实验(L34/35)
🎯 设计取舍
为什么要专门搞一个 score config,而不是让大家随手评分? 因为「能比较」是评估的全部价值。如果「有用性」这个名字下,A 团队打 0–1、B 团队打百分制、C 又用「好/差」, 那这些分数放在一起毫无意义——求平均是错的、画趋势是错的、做对比更是错的。score config 把「某个名字 = 某种刻度」固定下来、强制校验,于是同名分数永远可比, 你才能放心地算「这周有用性均分涨了没」「哪个模型的毒性通过率更高」。这和第 8 课「provided vs computed」、第 16 课「定价 schema」是同一种工程信念:把「数据该长什么样」显式建模、前置校验, 下游的一切分析才有坚实地基。而把三种来源统一到同一张 scores 表、同一套 config,则让「AI 评的」「人审的」「API 报的」分数活在一个空间里——这正是 Part 5 后续每一课的共同终点:生产可比的 score。
🎯 本课要点
  • score 是评估的原子单位:一个 (name, value, dataType, source) 挂在 trace 或其中某步 observation 上(也可挂 session / 数据集运行),把「发生了什么」变成「做得多好」。
  • 三种可比刻度:NUMERIC(数,带可选 min/max)、CATEGORICAL(标签↔值的固定集合,二者皆唯一)、BOOLEAN(锁死 True=1/False=0 的特殊分类)。
  • 布尔复用分类:不是独立第四类,而是「恰好两档」的分类——少一套特例,呼应第 8 课「能用通用机制就别新造」。
  • score config = 某 name 的 schema:声明 dataType 与约束(区间/类别),强制校验,使同名分数永远同一把尺、可比可聚合。
  • 三种 source:API(亲手提交)、EVAL(AI/代码评判)、ANNOTATION(人工)——殊途同归,都写进第 8 课的 scores 表。后续五课都只是「用不同方式生产 score」。

🧪 自测 · 想一想为什么这么设计

1. Langfuse score config 有三种可比刻度:NUMERIC、CATEGORICAL、BOOLEAN。但源码里 BOOLEAN 并不是独立的一套校验逻辑,而是被实现成什么?
  1. 一个「恰好两档、且锁死为 True=1 / False=0」的特殊 CATEGORICAL——底层只需处理数值与分类两套逻辑,布尔复用分类的存储与聚合
  2. 一个独立的布尔表
  3. 一个 NUMERIC 的别名
  4. 一段前端专用的校验
看答案与解析 点击展开
答案:A。BooleanConfigFields 强制 categories 长度为 2、且锁死 {True:1, False:0}(score-configs.ts)。这样就不必为布尔新造一套存储/聚合,它顺着分类的路走——少一套特例、多一分一致。这正呼应第 8 课「能用通用机制表达的,就别新造一个」。
2. 「有用性」这个分数,A 团队打 0–1、B 团队用百分制、C 用「好/差」。score config 要解决的核心问题是?
  1. 自动生成分数
  2. 可比性:config 是某个 name 的 schema,把「这个名字=这种刻度」固定并强制校验,于是同名分数永远同一把尺——求平均、画趋势、做对比才有意义
  3. 减少数据库存储
  4. 让评分速度更快
看答案与解析 点击展开
答案:B。「能比较」是评估的全部价值。若同名分数刻度各异,放一起求平均/画趋势/对比全是错的。score config 显式声明 dataType 与约束(区间/类别)并前置校验,使同名分数恒可比。这和第 8 课 provided/computed、第 16 课定价 schema 是同一种「把数据该长什么样显式建模」的工程信念。
3. score 有三种 source:API、EVAL、ANNOTATION(scores.ts 的 ScoreSourceArray)。关于这三者的关系,下列哪句最准确?
  1. 三种 source 各有独立的表和聚合逻辑
  2. EVAL 分数不经过摄取链路
  3. 三者殊途同归:AI 评判(EVAL)、人工标注(ANNOTATION)、亲手提交(API)最终都按同一 config 校验、写进同一张 scores 表——评估不是独立管道,而是 score 的生产者
  4. 只有 API 来源会被存储
看答案与解析 点击展开
答案:C。ScoreSourceArray=["API","EVAL","ANNOTATION"] 只是给同一张 scores 表的一行标注「这分谁打的」。LLM-as-a-judge、代码 eval、人工标注算出的分都走回同一条摄取链路、按同一 config 校验。这就是 Part 5 后续每课的共同终点:用不同方式生产可比的 score。
💭 发散思考(没有标准答案,动手或动脑想想)
  • Langfuse 把「AI 评的、人审的、API 报的」分数统一成一种 score、一张表、一套 config,只用 source 字段区分来源。在你做过或设想的系统里,如果要把「多种来源的质量信号」统一成一个可比的指标,你会怎么设计它的数据模型?需要一个像 score config 那样的「schema 守门人」吗,为什么?

The first four parts let you see what happened on every LLM call. But "what happened" is not the same as "how good was it". This part is about evaluation—how to put a number on a trace. And the atomic unit of all of it is the score. This lesson nails down the score's data model: what a score looks like, its three data types (numeric / categorical / boolean), what the score config ("the schema of a score") governs, and which three sources a score can come from. Once you have this model, the next five lessons (LLM-as-a-judge, code eval, human annotation, monitors) are all just "different ways of producing a score".

📋 Analogy
Think of a score as an evaluation sticker slapped on one answer. The sticker says what you're rating (name, e.g. "helpfulness"), the rating (value), and which kind of scale (dataType: a 0–1 number? a "good/ok/bad" tier? a "right/wrong" tick?). The score config is the printing standard for that kind of sticker: it mandates that "helpfulness" must be a 0–1 number, that "sentiment" must be one of {pos/neutral/neg}—so same-named stickers always use the same ruler, which is what makes them comparable and averageable. As for who applied the sticker (source): maybe you did (API), maybe an AI judge did (EVAL), maybe a human reviewer did (ANNOTATION)—but as long as the scale matches, all three live in the same comparable space.

Three scales: numeric, categorical, boolean

The soul of a score is its dataType—it decides how the value is read and validated. Langfuse pins down three types with Zod (plus a legacy TEXT):

A score's three scales (dataType) NUMERICvalue is a numberoptional minValue / maxValuee.g. helpfulness 0–1, latency 1–5"helpfulness": 0.82averageable, sortable CATEGORICALa set of label ↔ valuelabels unique, values uniquee.g. sentiment {pos/neu/neg}"sentiment": "positive"countable, groupable BOOLEANa special categorylocked: True=1 / False=0e.g. contains toxic words?"is_toxic": False (0)pass-rate computable
Three scales, each with a strength: NUMERIC (a number with optional min/max, averageable/sortable), CATEGORICAL (a fixed set of label↔value, both unique, countable/groupable), BOOLEAN (a special category locked to True=1 / False=0, pass-rate computable). Source: packages/shared/src/domain/score-configs.ts:18-80.
Three sources, one scores table APIreported by code EVALLLM judge / code eval ANNOTATIONhuman scores table (one shape) name: "helpfulness" value: 0.9 // or stringValue (categorical) dataType: NUMERIC | CATEGORICAL | BOOLEAN source: API | EVAL | ANNOTATION traceId / observationId · comment → comparable only at same name+scale: whoever scored, one space that is what source is for: keep "who scored this"
Three sources, one table (source enum from domain/scores.ts; values illustrative): a score has three sources — API (reported by code), EVAL (an LLM judge or code eval), ANNOTATION (human) — yet all land in one scores table sharing one shape (name / value or stringValue / dataType / source). So "machine-scored", "human-reviewed" and "API-reported" live in one space, comparable at the same name and scale.
packages/shared/src/domain/score-configs.tsthree comparable scales
// numeric: optional bounds, no categories
NumericConfigFields = { dataType: "NUMERIC", minValue?, maxValue? };

// categorical: a set of {label, value}; labels unique, values unique (validateCategories)
CategoricalConfigFields = { dataType: "CATEGORICAL", categories: [{label, value}, …] };

// boolean: really a "exactly two" special category, locked to True=1 / False=0
BooleanConfigFields = { dataType: "BOOLEAN",
  categories: [{label:"True", value:1}, {label:"False", value:0}] };  // length===2 enforced

Note one elegant move: boolean is not a separate fourth type—it is implemented as a "exactly two categories, locked to True=1/False=0" category. So the core only handles two logics, "numeric" and "categorical", and boolean reuses categorical storage and aggregation— one fewer special case, one more bit of consistency. This echoes Lesson 8's "wide event" idea: if a general mechanism can express it, don't invent a new one.

One more design that runs throughout: the value is always a number, while stringValue only holds the "label". A NUMERIC value is just that number with an empty stringValue; CATEGORICAL/BOOLEAN put the label in stringValue and the mapped number in value (e.g. "positive"→a number, "True"→1). Precisely because every score has a numeric value, averaging, trending, and pass-rate work uniformly across all types—the physical basis of "comparable".

dataTypevalue / stringValueconstraintsaggregationsexample
NUMERICvalue=the number; stringValue emptyoptional minValue / maxValuemean, distribution, trendhelpfulness 0.82
CATEGORICALstringValue=label; value=mapped numberlabels unique, values uniquecount, share, group-bysentiment "positive"
BOOLEANstringValue="True"/"False"; value∈{0,1}locked: True=1 / False=0pass-rateis_toxic False (0)

Reminder: scores.ts actually lists 5 dataTypes—the three above are the "comparable scales" a score config supports, plus TEXT (free text, length ≤500, legacy) and CORRECTION (for human-annotation corrections, name fixed to "output"). The latter two aren't aggregatable scales, so this lesson focuses on the first three. Source: scores.ts:46-52 (5 types), score-configs.ts:117-138 (config is a 4-way discriminatedUnion).

score config: a "schema for scoring"

A dataType alone isn't enough. If the same "helpfulness" score is rated 0–1 by one person today and 1–100 by another tomorrow, the data turns to mush. score config fixes exactly this: it is the schema of a score name— declaring what dataType that name uses, and what bounds / which categories. So no matter where a score comes from, it is validated against the same ruler, which is the precondition for comparison and aggregation.

One config guards one name: three sources, one ruler API (you submit it)source=API EVAL (AI/code judge)source=EVAL ANNOTATION (human)source=ANNOTATION score config (name's schema)"helpfulness" = NUMERIC [0,1]validates the value scores table (L8 wide table)same name+ruler → comparable three sources converge: all validated by config, all written to one scores table—judge/human/API in one comparable space
One config guards one name: a score config declares a name's dataType and constraints (numeric bounds / category set), validating scores from API / EVAL / ANNOTATION into one scores table, one ruler. Source: scores.ts:4 (ScoreSourceArray=["API","EVAL","ANNOTATION"]).
APIyou submit it
Use the SDK's score.create directly—e.g. your own offline eval script, or thumbs-up collected in production. This is one origin of the SCORE_CREATE event on the ingestion path of Lessons 6 & 12.
EVALAI or code judge
Scores computed by LLM-as-a-judge (Lesson 29) or code eval (Lesson 31). They too flow back through the same ingestion path to become scores—evaluation isn't a separate pipeline, it's a producer of scores.
ANNOTATIONhuman annotation
Human reviewers score traces against the config in an annotation queue (Lesson 32). They land in the same scores table, just with source set to ANNOTATION.

What fields a score object actually has

Unpack ScoreSchema and a score splits into three groups of fields—"what & how much", "who & with what", and "attached to what". The most worth remembering is the last: a score must attach to exactly one of trace / session / dataset-run; observationId is not a fourth peer but pins a step within a trace (it requires traceId).

what · how much

name which dimension (e.g. "helpfulness"); value always a number; stringValue the label (categorical/boolean/text only); dataType which scale.

who · with what

source API / EVAL / ANNOTATION; authorUserId the human annotator (nullable); comment a note (nullable); configId which config validates it; queueId which annotation queue it came from.

attached to (one of three)

one of traceId (whole trace), sessionId (whole session), datasetRunId (one dataset run); plus an optional observationId to pin a step within a trace (requires traceId).

This "one-of-three (plus in-trace step) attachment point" matters: it lets a score sit on the whole trace (was this answer good overall), on one observation within it via observationId (was this retrieval step accurate), or on a whole session or a dataset run (the experiment scenarios of Lessons 34–35). (Special case: CORRECTION scores may attach only to trace/observation, never session/datasetRun.) One score model covers every evaluation granularity. Validation rule: packages/shared/src/utils/scores.ts (applyScoreValidation); field defs in domain/scores.ts.

one score traceId = t-42 ✓ observationId = null sessionId = null datasetRunId = null rule: one of trace/session/run trace · whole answer good? observation · this step accurate? session · whole conversation dataset run · one experiment (L34/35)
🎯 Design trade-off
Why bother with a dedicated score config instead of letting everyone score freely? Because "being comparable" is the entire value of evaluation. If under the name "helpfulness" team A rates 0–1, team B uses a percentage, and C uses "good/bad", then those scores are meaningless together—averaging is wrong, trending is wrong, comparing is more wrong still. A score config fixes and enforces "a given name = a given scale", so same-named scores are forever comparable, and you can confidently ask "did average helpfulness rise this week" or "which model has the higher toxicity pass-rate". This is the same engineering belief as Lesson 8's "provided vs computed" and Lesson 16's "pricing schema": model "what the data should look like" explicitly and validate up front, and every downstream analysis stands on solid ground. Unifying three sources into one scores table and one config set lets "AI-judged", "human-reviewed", and "API-reported" scores live in one space—which is the shared destination of every remaining lesson in Part 5: producing comparable scores.
🎯 Key points
  • A score is evaluation's atomic unit: one (name, value, dataType, source) attached to a trace or one observation within it (or a session / dataset run), turning "what happened" into "how well".
  • Three comparable scales: NUMERIC (a number with optional min/max), CATEGORICAL (a fixed set of label↔value, both unique), BOOLEAN (a special category locked to True=1/False=0).
  • Boolean reuses categorical: not a separate fourth type but an "exactly two" category—one fewer special case, echoing Lesson 8's "don't invent when a general mechanism works".
  • score config = a name's schema: declares dataType and constraints (bounds/categories), enforces validation, so same-named scores always share one ruler—comparable and aggregatable.
  • Three sources: API (you submit), EVAL (AI/code judge), ANNOTATION (human)—all converge, all written to Lesson 8's scores table. The next five lessons are just "producing scores in different ways".

🧪 Self-test - think about the design

1. A Langfuse score config has three comparable scales: NUMERIC, CATEGORICAL, BOOLEAN. But in the source, BOOLEAN is not its own independent validation logic — what is it implemented as?
  1. a special CATEGORICAL with 'exactly two categories, locked to True=1 / False=0' — the core only handles numeric and categorical, and boolean reuses categorical storage and aggregation
  2. a separate boolean table
  3. an alias of NUMERIC
  4. a frontend-only validation
Show answer & explanation click to expand
Answer: A. BooleanConfigFields forces categories length 2, locked to {True:1, False:0} (score-configs.ts). So boolean needs no new storage/aggregation; it rides the categorical path — one fewer special case, more consistency. This echoes Lesson 8's 'if a general mechanism expresses it, don't invent a new one'.
2. For the score 'helpfulness', team A rates 0–1, team B uses a percentage, C uses 'good/bad'. What core problem does a score config solve?
  1. auto-generates scores
  2. comparability: a config is the schema of a name, fixing and enforcing 'this name = this scale', so same-named scores always use one ruler — only then are averaging, trending, and comparison meaningful
  3. reduces database storage
  4. makes scoring faster
Show answer & explanation click to expand
Answer: B. 'Being comparable' is the whole value of evaluation. If same-named scores use different scales, averaging/trending/comparing them is all wrong. A score config explicitly declares dataType and constraints (bounds/categories) and validates up front, keeping same-named scores forever comparable. Same belief as Lesson 8's provided/computed and Lesson 16's pricing schema: model 'what the data should look like' explicitly.
3. A score has three sources: API, EVAL, ANNOTATION (ScoreSourceArray in scores.ts). Which statement about them is most accurate?
  1. each source has its own table and aggregation logic
  2. EVAL scores bypass the ingestion path
  3. all three converge: AI judging (EVAL), human annotation (ANNOTATION), and direct submission (API) are all validated by the same config and written to the same scores table — evaluation isn't a separate pipeline but a producer of scores
  4. only the API source gets stored
Show answer & explanation click to expand
Answer: C. ScoreSourceArray=["API","EVAL","ANNOTATION"] merely labels a row in the same scores table with 'who scored this'. Scores from LLM-as-a-judge, code eval, and human annotation all flow back through the same ingestion path, validated by the same config. This is the shared destination of every remaining Part 5 lesson: producing comparable scores in different ways.
💭 Open questions (no single right answer - just think or try)
  • Langfuse unifies 'AI-judged, human-reviewed, API-reported' scores into one score type, one table, one config, distinguishing origin only by a source field. In a system you've built or imagined, if you had to unify 'quality signals from many sources' into one comparable metric, how would you design its data model? Would you want a 'schema gatekeeper' like a score config — and why?