🪢 Langfuse 图解教程Langfuse Visual Guide 第五部分 · 评估与评分Part 5 · Evaluation & Scoring 32 / 55
第五部分 · 评估与评分Part 5 · Evaluation & Scoring

人工标注Human annotation

🧭 读前Before you start
先读 第28课 的评分模型,本课讲人工标注如何产出 score。
Read Lesson 28 on the scoring model; this lesson covers how human annotation produces scores.

前三课让机器自动评分——LLM 当裁判、代码做确定性检查。但有些判断离不开人:建立可信的 ground truth(基准答案)、审核「AI 到底判得对不对」、处理 AI 拿不准的边界情形、给训练/对齐采人类偏好。这一课讲第 28 课埋下的第三种 score 来源——人工标注(source=ANNOTATION),以及把「请人评审」这件本质上无序又易冲突的事,组织成有序流程的工具:标注队列(annotation queue)。 重点有三:队列的数据模型(绑一组 score config、装一批待评对象)、一个有意思的5 分钟时间锁(人类尺度的并发控制),以及人工分如何和前两课的 EVAL 分同表汇合——从而成为校准 AI 裁判的标尺。

📋 生活类比
标注队列像医院的待会诊病历筐。每份需要人来看的病历(一条 trace、一次 observation、或整个 session)被放进筐里排队;筐口贴着一张统一的评分表(队列绑定的 score config,决定医生要填哪几项、每项什么刻度)。 医生从筐里取一份来看,系统立刻给这份病历上锁 5 分钟——这样另一位医生同时来取时,不会拿到同一份重复会诊;而要是这位医生中途走开了,锁会自动过期,病历重新可领,不会被永久占住。 看完,医生填表评分、可附一句评语,把病历标成「已会诊」,并记下是谁看的。这些人工评分和前两课 AI 评的分进同一本病案室(scores 表)——于是你能把「人怎么判的」和「AI 怎么判的」摆在一起对照,看 AI 靠不靠谱。

合上闭环:第三种来源,也是「标尺的标尺」

第 28 课说 score 有三种来源:API、EVAL、ANNOTATION。前面几课讲完了 EVAL(裁判与代码),这一课补上 ANNOTATION。三者同表、同 config、同一把尺——但人工分有个独特角色:它常被当作 ground truth,用来回答第 29 课留下的问题「AI 裁判靠不靠谱」。

三来源汇合,人工分还兼任「裁判的标尺」 API(你亲手提交) EVAL(AI 裁判 / 代码) ANNOTATION(人工·本课) scores 表(第 8 课宽表)同名同 config → 可比 人工分 = ground truth在同一批 trace 上和 EVAL 分对照 → 校准 AI 裁判是否可信人机不一致处,正是裁判要改进的地方 人工标注又慢又贵,所以通常只评一小撮关键样本——但这一小撮是衡量 AI 裁判的「金标准」 三来源各取所长:API 灵活、EVAL 可扩展、ANNOTATION 最权威
第三种来源,闭合第 28 课的环:人工分与 API、EVAL 分写进同一张 scores 表(ScoreSourceArray=["API","EVAL","ANNOTATION"],scores.ts:4),享受同一 config 校验。它最大的价值是当 ground truth——在同一批对象上和 AI 裁判分对照,量出裁判的可信度。
示例:标注队列里的一张卡 待标注(trace 输出) trace · chat-support · obs answer「要退款的话,请在 30 天内…」model gpt-4o · latency 1.8s 人来读、来判断 ——机器算不出的「价值/语气/边界」 score-config 决定控件 helpfulness (NUMERIC) 0.8 tone (CATEGORICAL) 好 中 差 标注人 alice · 评语:清楚但啰嗦 提交 → 写一条 score {name:"helpfulness", value:0.8, source:"ANNOTATION", configId}
人来打机器打不了的分(控件由 score-config 决定;值为示例):标注队列把待评的 trace 一张张推到人面前;score-config 决定用什么控件——NUMERIC 滑杆、CATEGORICAL 好/中/差、或 BOOLEAN。提交后写一条 source=ANNOTATION 的 score,和 API/EVAL 的分进同一张表(第 28 课),于是「人审」也能和「机器评」并排比较。
建基准ground truth
很多评估要有「标准答案」才谈得上准不准。人类专家在一小撮代表性样本上打的分,就是这把金标准——AI 裁判、回归测试、模型对比都拿它当参照系。
审裁判审核 AI 判得对不对
第 29 课问过「裁判自己靠不靠谱」。答案就在这:把同一批 trace 既让 AI 裁判评、又让人评,看两者一致不一致。不一致的地方,要么改 prompt、要么换模型——人工标注是校准裁判的反馈回路。
啃边界AI 拿不准的情形
对那些 AI 反复评分摇摆、或明显需要领域知识/价值判断的样本,直接交给人。人工标注专攻机器最不擅长的长尾,把宝贵人力花在刀刃上。

队列的数据模型:一张评分表 + 一筐待评对象

标注队列的数据模型只有两个主角。AnnotationQueue 是「评审任务」本身:有名字、描述,最关键是一组 scoreConfigIds——它绑定第 28 课的 score config,规定这个队列里每位评审员要填哪几项分、每项什么刻度(统一了大家的尺)。AnnotationQueueItem 是「筐里的一份待评对象」:指向一条 trace/observation/session,带一个 PENDING→COMPLETED 的状态。

AnnotationQueue(评审任务) name · description scoreConfigIds[] → 绑定 L28 score config (统一这个队列的尺) AnnotationQueueItem(待评对象) objectType / objectId status: PENDING → COMPLETED lockedAt(5 分钟软锁) annotatorUserId / completedAt 评的对象 TRACE/OBSERVATION/SESSION 1 : N 另有 AnnotationQueueAssignment:把队列结构化指派给特定评审员(谁该评哪个队列)
字段含义为什么重要
AnnotationQueue(评审任务)
name / description队列名与说明一个项目可有多个队列(如「事实性核查」「语气审查」)
scoreConfigIds[]绑定的一组 score config统一评分表:所有评审员按同一组 config 评分,分数才可比(呼应第 28 课)
AnnotationQueueItem(一份待评对象)
objectType / objectId评的是什么:TRACE / OBSERVATION / SESSION三种粒度都能评——整条对话、某一步、或整个会话
statusPENDING / COMPLETED队列的进度就是「还剩多少 PENDING」
lockedAt / lockedByUserId谁在看、何时锁的下一节的 5 分钟锁,防两人重复标注
annotatorUserId / completedAt谁评的、何时完成可审计:每条人工分都知道出自谁手
packages/shared/prisma/schema.prisma:582-633两个模型 + 两个枚举
model AnnotationQueue {
  name           String
  scoreConfigIds String[]   // ← 绑定第28课的 score config,统一这个队列的评分表
  annotationQueueItem  AnnotationQueueItem[]
  annotationQueueAssignment AnnotationQueueAssignment[]  // 队列指派给哪些评审员
}
model AnnotationQueueItem {
  objectId   String;  objectType AnnotationQueueObjectType   // 评的对象
  status     AnnotationQueueStatus @default(PENDING)          // PENDING → COMPLETED
  lockedAt   DateTime?;  lockedByUserId  String?              // ← 软锁(5分钟)
  annotatorUserId String?;  completedAt DateTime?            // 谁评的、何时完成
}
enum AnnotationQueueObjectType { TRACE  OBSERVATION  SESSION }
enum AnnotationQueueStatus     { PENDING  COMPLETED }

还有第三个模型 AnnotationQueueAssignment:把队列指派给特定评审员。于是「谁该评哪个队列」也是结构化的——不是发个链接全凭自觉,而是平台记录在案的分工。

5 分钟软锁:人类尺度的并发控制

多个评审员同时盯着一个队列,怎么避免两人评到同一份、白费功夫?数据库的事务锁太重、也不贴合「人看一会儿」的节奏。Langfuse 用一个轻巧的时间软锁:评审员打开一份 item,就写上 lockedByUserId 和 lockedAt;判断「是否被锁」时,只认最近 5 分钟内的锁。

软锁 = lockedByUserId + lockedAt,只在 5 分钟内有效 评审员 A 打开写 lockedAt=now item 锁定中(lockedAt > now − 5min)其他人看到「已锁」,跳过这份 A 评完 → COMPLETED记 annotatorUserId 若 A 中途离开……5 分钟后锁自动过期,item 重新可领——不会被永久占住 评审员 B 来取锁过期→可领 不是数据库事务锁,而是一个「带过期时间的标记」——贴合人类「看几分钟」的节奏,且永不死锁 关掉浏览器、网断了都不要紧:锁会自己消失
软锁,不是硬锁:isItemLocked 只认 lockedAt 在最近 5 分钟内的锁(annotationQueueItemsRouter.ts:34-36)。它解决「人类并发」——多人同评一个队列不撞车,又因为会自动过期,绝不会因某人关了页面而把 item 永久占死。
annotationQueueItemsRouter.ts:34 · scores/hooks/useScoreMutations.ts:39软锁 + 出分
// 软锁:只有「最近 5 分钟内被锁」才算锁定——超时自动失效
const isItemLocked = (item) =>
  item.lockedByUserId && item.lockedAt &&
  new Date(item.lockedAt) > new Date(Date.now() - 5 * 60 * 1000);   // 5 分钟窗口

// 评完:item 标 COMPLETED 并记下评审员
{ status: AnnotationQueueStatus.COMPLETED, annotatorUserId: ctx.session.user.id }

// 评审员填的每个分:source=ANNOTATION,且带 authorUserId(谁打的)
createAnnotationScore({ source: "ANNOTATION", authorUserId, name, value, dataType, comment });
// → 和第29课 EVAL 分一样,归一后经第12课摄取链路回流同一张 scores 表

评审员填的分,source 标成 ANNOTATION、带上 authorUserId(哪位评审员),还能附 comment 写理由——和第 29 课 EVAL 分一样,最终都归一成 score、经第 12 课摄取链路回流同一张 scores 表。至此,第 28 课的三种来源全部到齐:API、EVAL、ANNOTATION 殊途同归,同表、同尺、可比。Part 5 的主线——「把『发生了什么』变成『做得多好』」——到这里画上闭环。

1

进入被指派的队列

评审员看到指派给自己的队列(AnnotationQueueAssignment)和里面待评的 PENDING item。

2

领一份 → 自动上锁

打开一份 item,系统写 lockedByUserId/lockedAt,5 分钟内别人看到「已锁」、跳过。

3

看对象

按 objectType 展开要评的 TRACE / OBSERVATION / SESSION——直接复用第 25、26 课的详情视图。

4

按 config 评分 + 评语

填队列绑定的每个 score config,可附 comment。分即时写成 source=ANNOTATION、带 authorUserId。

5

提交 → 跳下一份

item 标 COMPLETED + 记 annotatorUserId/completedAt,锁释放,自动推进队列里下一份 PENDING。

🎯 设计取舍
为什么用「5 分钟时间软锁」,而不是数据库事务锁或永久占用标记? 因为这把锁要适配的是人,不是机器。数据库事务锁是毫秒级的、占着连接,根本扛不住「人看几分钟」的时长。而如果用永久标记(「A 正在看」直到 A 主动释放),一旦 A 关了浏览器、断了网、或干脆忘了——这份 item 就被永久占死,再没人能领。时间软锁两头讨好:5 分钟够一个人看完一份、避免并发撞车;又因为会自动过期,任何意外都只会让锁「自愈」,绝不死锁。这是一个把「乐观并发 + 租约(lease)」思想用在人类工作流上的漂亮小设计——锁的粒度,要匹配被锁者的节奏。

为什么人工分一定要和 AI 评的分进同一张表、用同一套 config? 因为只有同尺,才能对照。把人评的「金标准」和 AI 裁判的分摆在同一批 trace 上,一减就知道裁判哪里偏了——这正是第 29 课「怎么信任 AI 裁判」的答案:用人工标注当裁判的标尺。如果两边各用各的存储、各打各的刻度,这种校准根本无从谈起。Langfuse 让三种来源(API/EVAL/ANNOTATION)共享一张 scores 表、一套 score config,本质上就是为了让「人怎么看」和「机器怎么看」永远可以放在一起比较——评估体系的可信度,最终落在这一点上。
🎯 本课要点
  • 人工标注 = 第三种 score 来源(ANNOTATION):补上第 28 课三来源的最后一块。它最贵也最权威,常用作 ground truth——既建基准、又审核「AI 裁判判得对不对」、还啃 AI 拿不准的长尾。
  • 标注队列把无序的评审变有序:AnnotationQueue 绑一组 scoreConfigIds(统一评分表);AnnotationQueueItem 指向一个 TRACE/OBSERVATION/SESSION,带 PENDING→COMPLETED 状态;AnnotationQueueAssignment 把队列指派给评审员。
  • 5 分钟软锁 = 人类并发控制:打开即写 lockedAt,isItemLocked 只认最近 5 分钟内的锁。多人同评不撞车,又因自动过期而永不死锁——锁的粒度匹配人的节奏。
  • 出分同归:评审员按 config 打的分 source=ANNOTATION、带 authorUserId 与可选 comment,item 标 COMPLETED + 记 annotatorUserId;分经第 12 课摄取链路回流同一张 scores 表。
  • 闭环:API/EVAL/ANNOTATION 三来源同表、同尺、可比——人工分因此能当 AI 裁判的标尺。这正是评估可信度的基石,也为 Part 5「把发生变成做得多好」收尾。

🧪 自测 · 想一想为什么这么设计

1. 人工标注是第 28 课三种 score 来源里的 ANNOTATION。除了「人比机器更准」,它在评估体系里还扮演一个独特角色,是什么?
  1. 替代 LLM-as-judge
  2. 让评估跑得更快
  3. 充当 ground truth / 校准 AI 裁判:把人评的「金标准」和 AI 裁判分放在同一批 trace 上对照,量出裁判哪里偏了——这正是第 29 课「怎么信任 AI 裁判」的答案
  4. 降低标注成本
看答案与解析 点击展开
答案:C。人工分又慢又贵,通常只评一小撮关键样本,但这一小撮是衡量 AI 裁判的金标准。因为三来源同表同 config(ScoreSourceArray=[API,EVAL,ANNOTATION]),人评分和 AI 裁判分可在同一批对象上直接对照;不一致处正是裁判要改 prompt/换模型的地方。这把「人工标注当裁判的标尺」,正是评估可信度的基石。
2. 多个评审员同时盯着一个标注队列。Langfuse 用一个「5 分钟时间软锁」防止两人评到同一份。相比数据库事务锁或永久占用标记,软锁的关键好处是?
  1. 能锁住整个队列
  2. 不需要数据库字段
  3. 比事务锁查询更快
  4. 匹配人的节奏且永不死锁:5 分钟够看完一份、避免撞车;又因自动过期,关浏览器/断网都只会让锁自愈,绝不会把 item 永久占死
看答案与解析 点击展开
答案:D。isItemLocked 只认 lockedAt 在最近 5 分钟内的锁。数据库事务锁是毫秒级、占连接,扛不住人看几分钟;永久标记一旦评审员关页面就把 item 占死。时间软锁两头讨好:5 分钟避免并发撞车,又因自动过期而自愈、永不死锁。这是「乐观并发 + 租约」用在人类工作流上——锁的粒度匹配被锁者的节奏。
3. 一个 AnnotationQueue 上绑了一组 scoreConfigIds。这个绑定起什么作用?
  1. 决定队列指派给哪些评审员
  2. 它是这个队列的「统一评分表」:规定所有评审员要按同一组 score config 评分(同名同刻度),分数才可比、才能聚合——直接复用第 28 课的 config
  3. 控制 5 分钟锁的时长
  4. 决定评哪些 trace
看答案与解析 点击展开
答案:B。scoreConfigIds 绑定第 28 课的 score config,规定这个队列里每位评审员要填哪几项分、每项什么刻度(NUMERIC/CATEGORICAL/BOOLEAN)。统一了尺,不同评审员、不同 item 的分才可比可聚合。指派给谁是另一个模型 AnnotationQueueAssignment 的事,评哪些对象是 AnnotationQueueItem 的事——三者各司其职。
💭 发散思考(没有标准答案,动手或动脑想想)
  • Part 5 走到这里,三种 score 来源(API/EVAL/ANNOTATION)全部汇入同一张 scores 表、同一套 config。回头看这条主线:为什么 Langfuse 不为「AI 评的分」和「人评的分」分别建表,而是执意让它们同表同尺?这种「统一数据模型」的设计,在可信度、可维护性、未来扩展上分别带来了什么?你能想到它的代价或局限吗?

The last three lessons let machines score automatically—an LLM as judge, code for deterministic checks. But some judgments require a human: establishing trustworthy ground truth, auditing "is the AI actually right", handling edge cases the AI is unsure about, collecting human preferences for training/alignment. This lesson covers the third score source planted in Lesson 28—human annotation (source=ANNOTATION), and the tool that turns the inherently unordered and collision-prone act of "asking people to review" into an orderly process: the annotation queue. Three focuses: the queue's data model (bound to a set of score configs, holding a batch of objects to review), an interesting 5-minute time lock (human-scale concurrency control), and how human scores converge in the same table as the previous lessons' EVAL scores—becoming the yardstick to calibrate the AI judge.

📋 Analogy
An annotation queue is like a hospital's basket of charts awaiting consultation. Each chart needing human eyes (a trace, an observation, or a whole session) queues in the basket; the basket bears a uniform scoring sheet (the queue's bound score configs, deciding which items each doctor must fill and on what scale). A doctor takes one to review, and the system immediately locks that chart for 5 minutes—so another doctor taking one at the same time won't get the same chart twice; and if this doctor wanders off, the lock auto-expires and the chart becomes available again, never permanently held. After reviewing, the doctor fills the sheet, may add a note, marks the chart "consulted", and records who reviewed it. These human scores go into the same records room (the scores table) as the AI scores from the last lessons—so you can lay "how the human judged" and "how the AI judged" side by side and see whether the AI is reliable.

Closing the loop: the third source, and the "yardstick for yardsticks"

Lesson 28 said a score has three sources: API, EVAL, ANNOTATION. The earlier lessons finished EVAL (judge and code); this one fills in ANNOTATION. All three are same-table, same-config, one ruler—but human scores have a unique role: they're often used as ground truth, answering the question Lesson 29 left open, "is the AI judge reliable".

three sources converge; human scores also serve as "the judge's yardstick" API (you submit) EVAL (AI judge / code) ANNOTATION (human·here) scores table (L8 wide table)same name+config → comparable human scores = ground truthon the same tracescompared with EVAL scores → calibrate if the AI judge is trustworthyhuman-AI disagreements are where the judge must improve human annotation is slow and costly, so usually only a small key sample is reviewed—but that sample is the "gold standard" for the AI judge three sources, each its strength: API flexible, EVAL scalable, ANNOTATION most authoritative
The third source, closing Lesson 28's loop: human scores join API and EVAL scores in the same scores table (ScoreSourceArray=["API","EVAL","ANNOTATION"], scores.ts:4), validated by the same config. Their biggest value is as ground truth—compared against AI-judge scores on the same objects to measure the judge's trustworthiness.
Example: one card in the annotation queue to annotate (trace output) trace · chat-support · obs answer"For a refund, within 30 days please…"model gpt-4o · latency 1.8s a human reads and judges —value/tone/boundaries a machine can't score score-config drives the widget helpfulness (NUMERIC) 0.8 tone (CATEGORICAL) good ok bad annotator alice · comment: clear but verbose submit → writes a score {name:"helpfulness", value:0.8, source:"ANNOTATION", configId}
Humans score what machines can't (the widget is driven by the score-config; values illustrative): the annotation queue pushes traces to a person one card at a time; the score-config picks the widget — a NUMERIC slider, CATEGORICAL good/ok/bad, or BOOLEAN. Submitting writes a source=ANNOTATION score into the same table as API/EVAL scores (Lesson 28), so human review sits side by side with machine eval.
baselineground truth
Many evaluations need a "correct answer" before "how accurate" means anything. Scores from human experts on a small representative sample are that gold standard—AI judges, regression tests, and model comparisons all use it as their frame of reference.
auditcheck if the AI is right
Lesson 29 asked "is the judge itself reliable". The answer is here: have both the AI judge and humans score the same traces and see whether they agree. Where they don't, fix the prompt or change the model—human annotation is the feedback loop that calibrates the judge.
long tailcases the AI is unsure about
For samples where the AI's score keeps wavering, or that clearly need domain knowledge / value judgment, hand them straight to humans. Human annotation specializes in the long tail machines are worst at, spending scarce human effort where it matters most.

The queue's data model: one scoring sheet + a basket of objects

The annotation queue's data model has just two protagonists. AnnotationQueue is the "review task" itself: a name, a description, and crucially a set of scoreConfigIds—it binds Lesson 28's score configs, dictating which scores each reviewer in this queue must fill and on what scale (unifying everyone's ruler). AnnotationQueueItem is "one object to review in the basket": pointing at a trace/observation/session, with a PENDING→COMPLETED status.

AnnotationQueue (review task) name · description scoreConfigIds[] → binds L28 score config (unifies this queue's ruler) AnnotationQueueItem (one to review) objectType / objectId status: PENDING → COMPLETED lockedAt (5-min soft lock) annotatorUserId / completedAt the object TRACE/OBSERVATION/SESSION 1 : N also AnnotationQueueAssignment: structurally assigns queues to specific reviewers (who reviews which queue)
fieldmeaningwhy it matters
AnnotationQueue (review task)
name / descriptionqueue name and notesa project can have several queues (e.g. "fact-check", "tone review")
scoreConfigIds[]the bound set of score configsuniform scoring sheet: all reviewers score by the same configs, so scores are comparable (echoing Lesson 28)
AnnotationQueueItem (one object to review)
objectType / objectIdwhat's reviewed: TRACE / OBSERVATION / SESSIONthree granularities—whole conversation, one step, or whole session
statusPENDING / COMPLETEDqueue progress is "how many PENDING remain"
lockedAt / lockedByUserIdwho's viewing, locked whenthe next section's 5-minute lock, preventing double-annotation
annotatorUserId / completedAtwho reviewed, finished whenauditable: every human score knows whose hand it came from
packages/shared/prisma/schema.prisma:582-633two models + two enums
model AnnotationQueue {
  name           String
  scoreConfigIds String[]   // ← binds Lesson 28's score configs, unifying this queue's sheet
  annotationQueueItem  AnnotationQueueItem[]
  annotationQueueAssignment AnnotationQueueAssignment[]  // which reviewers the queue is assigned to
}
model AnnotationQueueItem {
  objectId   String;  objectType AnnotationQueueObjectType   // the reviewed object
  status     AnnotationQueueStatus @default(PENDING)          // PENDING → COMPLETED
  lockedAt   DateTime?;  lockedByUserId  String?              // ← soft lock (5 minutes)
  annotatorUserId String?;  completedAt DateTime?            // who reviewed, finished when
}
enum AnnotationQueueObjectType { TRACE  OBSERVATION  SESSION }
enum AnnotationQueueStatus     { PENDING  COMPLETED }

There's a third model, AnnotationQueueAssignment: it assigns a queue to specific reviewers. So "who should review which queue" is structured too—not a link shared on the honor system, but a division of labor the platform records.

The 5-minute soft lock: human-scale concurrency control

With several reviewers eyeing one queue, how do you stop two from annotating the same item and wasting effort? A database transaction lock is too heavy and ill-fitting for the "a person reads for a while" rhythm. Langfuse uses a light time-based soft lock: a reviewer opens an item and it stamps lockedByUserId and lockedAt; judging "is it locked" only counts a lock from the last 5 minutes.

soft lock = lockedByUserId + lockedAt, valid only within 5 minutes reviewer A openswrite lockedAt=now item locked (lockedAt > now − 5min)others see "locked", skip this one A done → COMPLETEDrecord annotatorUserId if A leaves midway…lock auto-expires after 5 min, item available again—never permanently held reviewer B takeslock expired→available not a DB transaction lock but a "marker with an expiry"—fits the human "read for a few minutes" rhythm and never deadlocks closing the browser or losing the network doesn't matter: the lock vanishes on its own
A soft lock, not a hard one: isItemLocked only counts a lock whose lockedAt is within the last 5 minutes (annotationQueueItemsRouter.ts:34-36). It solves "human concurrency"—several people on one queue don't collide—and because it auto-expires, it never holds an item permanently just because someone closed their tab.
annotationQueueItemsRouter.ts:34 · scores/hooks/useScoreMutations.ts:39soft lock + emit score
// soft lock: only a "locked within the last 5 minutes" counts as locked—auto-expires
const isItemLocked = (item) =>
  item.lockedByUserId && item.lockedAt &&
  new Date(item.lockedAt) > new Date(Date.now() - 5 * 60 * 1000);   // 5-minute window

// on finish: mark item COMPLETED and record the reviewer
{ status: AnnotationQueueStatus.COMPLETED, annotatorUserId: ctx.session.user.id }

// each score the reviewer fills: source=ANNOTATION, with authorUserId (who scored)
createAnnotationScore({ source: "ANNOTATION", authorUserId, name, value, dataType, comment });
// → like Lesson 29's EVAL scores, normalized and flowed back via L12 ingestion into one scores table

The scores a reviewer fills have source set to ANNOTATION, carry an authorUserId (which reviewer), and may attach a comment with the reasoning—just like Lesson 29's EVAL scores, all normalized into scores and flowed back through Lesson 12's ingestion path into the same scores table. With that, all three sources from Lesson 28 are present: API, EVAL, ANNOTATION all converge—same table, same ruler, comparable. Part 5's through-line—"turn what happened into how well"—comes full circle here.

1

enter your assigned queue

A reviewer sees the queues assigned to them (AnnotationQueueAssignment) and the PENDING items inside.

2

take one → auto-lock

Opening an item stamps lockedByUserId/lockedAt; for 5 minutes others see "locked" and skip it.

3

review the object

Per objectType, expand the TRACE / OBSERVATION / SESSION to review—reusing Lessons 25 & 26's detail views.

4

score by config + comment

Fill each of the queue's bound score configs, optionally with a comment. The scores are written as source=ANNOTATION with authorUserId.

5

submit → next item

The item is marked COMPLETED with annotatorUserId/completedAt, the lock releases, and the next PENDING item in the queue advances automatically.

🎯 Design trade-off
Why a "5-minute time-based soft lock" instead of a DB transaction lock or a permanent-hold marker? Because this lock must fit people, not machines. A DB transaction lock is millisecond-scale and ties up a connection—it can't survive "a person reading for minutes". And a permanent marker ("A is viewing" until A explicitly releases) means that once A closes the browser, loses network, or simply forgets, the item is held forever and no one can take it. The time-based soft lock pleases both ends: 5 minutes is enough for one person to finish one item and avoid collisions; and because it auto-expires, any mishap merely lets the lock "self-heal", never deadlocking. It's a neat application of "optimistic concurrency + a lease" to a human workflow—the granularity of a lock should match the rhythm of who it locks.

Why must human scores go into the same table and use the same config as AI scores? Because only with one ruler can you compare. Lay the human "gold standard" and the AI judge's scores on the same batch of traces, and a subtraction tells you where the judge is biased—this is the answer to Lesson 29's "how to trust the AI judge": use human annotation as the judge's yardstick. If each side used its own storage and its own scale, such calibration would be impossible. Langfuse having the three sources (API/EVAL/ANNOTATION) share one scores table and one set of configs is, at heart, to keep "how humans see it" and "how machines see it" always comparable side by side—the trustworthiness of the whole evaluation system ultimately rests on this.
🎯 Key points
  • Human annotation = the third score source (ANNOTATION): the last piece of Lesson 28's three sources. The most costly and most authoritative, often used as ground truth—setting baselines, auditing "is the AI judge right", and handling the AI's uncertain long tail.
  • The annotation queue makes unordered review orderly: AnnotationQueue binds a set of scoreConfigIds (uniform sheet); AnnotationQueueItem points at a TRACE/OBSERVATION/SESSION with a PENDING→COMPLETED status; AnnotationQueueAssignment assigns queues to reviewers.
  • The 5-minute soft lock = human concurrency control: opening stamps lockedAt, and isItemLocked only counts a lock from the last 5 minutes. Many can review one queue without colliding, and because it auto-expires it never deadlocks—the lock's granularity matches the human rhythm.
  • Scores converge: a reviewer's config-driven scores carry source=ANNOTATION, an authorUserId, and an optional comment; the item is marked COMPLETED with annotatorUserId; the scores flow back through Lesson 12's ingestion into the same scores table.
  • The loop closes: API/EVAL/ANNOTATION are same-table, same-ruler, comparable—so human scores can serve as the AI judge's yardstick. This is the bedrock of evaluation trust, and it caps Part 5's "turn what happened into how well".

🧪 Self-test - think about the design

1. Human annotation is the ANNOTATION of Lesson 28's three score sources. Beyond 'humans are more accurate than machines', it plays a unique role in the evaluation system. What is it?
  1. replaces LLM-as-judge
  2. makes evaluation run faster
  3. serving as ground truth / calibrating the AI judge: laying the human 'gold standard' against AI-judge scores on the same traces measures where the judge is biased — the answer to Lesson 29's 'how to trust the AI judge'
  4. lowers annotation cost
Show answer & explanation click to expand
Answer: C. Human scores are slow and costly, usually only on a small key sample—but that sample is the gold standard for the AI judge. Because the three sources share one table and config (ScoreSourceArray=[API,EVAL,ANNOTATION]), human and AI-judge scores compare directly on the same objects; disagreements are where the judge must fix its prompt/model. Using human annotation as the judge's yardstick is the bedrock of evaluation trust.
2. Several reviewers eye one annotation queue at once. Langfuse uses a '5-minute time-based soft lock' to stop two from annotating the same item. Versus a DB transaction lock or a permanent-hold marker, the soft lock's key benefit is?
  1. can lock the whole queue
  2. needs no database fields
  3. queries faster than a transaction lock
  4. fits the human rhythm and never deadlocks: 5 minutes suffices to finish one item and avoid collisions; and because it auto-expires, closing the browser/losing network just lets the lock self-heal, never holding an item forever
Show answer & explanation click to expand
Answer: D. isItemLocked only counts a lock whose lockedAt is within the last 5 minutes. A DB transaction lock is millisecond-scale and ties up a connection, unfit for minutes of human reading; a permanent marker holds the item forever once a reviewer closes the tab. The time-based soft lock pleases both: 5 minutes avoids collisions, and auto-expiry self-heals, never deadlocking. It's 'optimistic concurrency + a lease' applied to a human workflow — lock granularity matching the locked party's rhythm.
3. An AnnotationQueue binds a set of scoreConfigIds. What does this binding do?
  1. decides which reviewers the queue is assigned to
  2. it's the queue's 'uniform scoring sheet': all reviewers must score by the same set of score configs (same name, same scale) so scores are comparable and aggregatable — directly reusing Lesson 28's configs
  3. controls the 5-minute lock duration
  4. decides which traces to review
Show answer & explanation click to expand
Answer: B. scoreConfigIds bind Lesson 28's score configs, dictating which scores each reviewer in this queue fills and on what scale (NUMERIC/CATEGORICAL/BOOLEAN). With a unified ruler, scores across reviewers and items are comparable and aggregatable. Who it's assigned to is the separate AnnotationQueueAssignment model; which objects to review is AnnotationQueueItem — each with its own job.
💭 Open questions (no single right answer - just think or try)
  • By this point in Part 5, all three score sources (API/EVAL/ANNOTATION) flow into one scores table, one set of configs. Looking back at this through-line: why does Langfuse refuse to build separate tables for 'AI-judged' and 'human-judged' scores, insisting on one table and one ruler? What does this 'unified data model' buy in trust, maintainability, and future extension? Can you think of its costs or limits?