充当 ground truth / 校准 AI 裁判:把人评的「金标准」和 AI 裁判分放在同一批 trace 上对照,量出裁判哪里偏了——这正是第 29 课「怎么信任 AI 裁判」的答案
降低标注成本
看答案与解析 点击展开
答案:C。人工分又慢又贵,通常只评一小撮关键样本,但这一小撮是衡量 AI 裁判的金标准。因为三来源同表同 config(ScoreSourceArray=[API,EVAL,ANNOTATION]),人评分和 AI 裁判分可在同一批对象上直接对照;不一致处正是裁判要改 prompt/换模型的地方。这把「人工标注当裁判的标尺」,正是评估可信度的基石。
The last three lessons let machines score automatically—an LLM as judge, code for deterministic checks. But some judgments require a human: establishing trustworthy ground truth, auditing "is the AI actually right", handling edge cases the AI is unsure about, collecting human preferences for training/alignment. This lesson covers the third score source planted in Lesson 28—human annotation (source=ANNOTATION), and the tool that turns the inherently unordered and collision-prone act of "asking people to review" into an orderly process: the annotation queue.
Three focuses: the queue's data model (bound to a set of score configs, holding a batch of objects to review), an interesting 5-minute time lock (human-scale concurrency control), and how human scores converge in the same table as the previous lessons' EVAL scores—becoming the yardstick to calibrate the AI judge.
📋 Analogy
An annotation queue is like a hospital's basket of charts awaiting consultation. Each chart needing human eyes (a trace, an observation, or a whole session) queues in the basket; the basket bears a uniform scoring sheet (the queue's bound score configs, deciding which items each doctor must fill and on what scale).
A doctor takes one to review, and the system immediately locks that chart for 5 minutes—so another doctor taking one at the same time won't get the same chart twice; and if this doctor wanders off, the lock auto-expires and the chart becomes available again, never permanently held.
After reviewing, the doctor fills the sheet, may add a note, marks the chart "consulted", and records who reviewed it. These human scores go into the same records room (the scores table) as the AI scores from the last lessons—so you can lay "how the human judged" and "how the AI judged" side by side and see whether the AI is reliable.
Closing the loop: the third source, and the "yardstick for yardsticks"
Lesson 28 said a score has three sources: API, EVAL, ANNOTATION. The earlier lessons finished EVAL (judge and code); this one fills in ANNOTATION. All three are same-table, same-config, one ruler—but human scores have a unique role: they're often used as ground truth, answering the question Lesson 29 left open, "is the AI judge reliable".
The third source, closing Lesson 28's loop: human scores join API and EVAL scores in the same scores table (ScoreSourceArray=["API","EVAL","ANNOTATION"], scores.ts:4), validated by the same config. Their biggest value is as ground truth—compared against AI-judge scores on the same objects to measure the judge's trustworthiness.
Humans score what machines can't (the widget is driven by the score-config; values illustrative): the annotation queue pushes traces to a person one card at a time; the score-config picks the widget — a NUMERIC slider, CATEGORICAL good/ok/bad, or BOOLEAN. Submitting writes a source=ANNOTATION score into the same table as API/EVAL scores (Lesson 28), so human review sits side by side with machine eval.
baselineground truth
Many evaluations need a "correct answer" before "how accurate" means anything. Scores from human experts on a small representative sample are that gold standard—AI judges, regression tests, and model comparisons all use it as their frame of reference.
auditcheck if the AI is right
Lesson 29 asked "is the judge itself reliable". The answer is here: have both the AI judge and humans score the same traces and see whether they agree. Where they don't, fix the prompt or change the model—human annotation is the feedback loop that calibrates the judge.
long tailcases the AI is unsure about
For samples where the AI's score keeps wavering, or that clearly need domain knowledge / value judgment, hand them straight to humans. Human annotation specializes in the long tail machines are worst at, spending scarce human effort where it matters most.
The queue's data model: one scoring sheet + a basket of objects
The annotation queue's data model has just two protagonists. AnnotationQueue is the "review task" itself: a name, a description, and crucially a set of scoreConfigIds—it binds Lesson 28's score configs, dictating which scores each reviewer in this queue must fill and on what scale (unifying everyone's ruler). AnnotationQueueItem is "one object to review in the basket": pointing at a trace/observation/session, with a PENDING→COMPLETED status.
field
meaning
why it matters
AnnotationQueue (review task)
name / description
queue name and notes
a project can have several queues (e.g. "fact-check", "tone review")
scoreConfigIds[]
the bound set of score configs
uniform scoring sheet: all reviewers score by the same configs, so scores are comparable (echoing Lesson 28)
AnnotationQueueItem (one object to review)
objectType / objectId
what's reviewed: TRACE / OBSERVATION / SESSION
three granularities—whole conversation, one step, or whole session
status
PENDING / COMPLETED
queue progress is "how many PENDING remain"
lockedAt / lockedByUserId
who's viewing, locked when
the next section's 5-minute lock, preventing double-annotation
annotatorUserId / completedAt
who reviewed, finished when
auditable: every human score knows whose hand it came from
packages/shared/prisma/schema.prisma:582-633two models + two enums
model AnnotationQueue {
name String
scoreConfigIds String[] // ← binds Lesson 28's score configs, unifying this queue's sheet
annotationQueueItem AnnotationQueueItem[]
annotationQueueAssignment AnnotationQueueAssignment[] // which reviewers the queue is assigned to
}
model AnnotationQueueItem {
objectId String; objectType AnnotationQueueObjectType // the reviewed object
status AnnotationQueueStatus @default(PENDING) // PENDING → COMPLETED
lockedAt DateTime?; lockedByUserId String? // ← soft lock (5 minutes)
annotatorUserId String?; completedAt DateTime? // who reviewed, finished when
}
enum AnnotationQueueObjectType { TRACE OBSERVATION SESSION }
enum AnnotationQueueStatus { PENDING COMPLETED }
There's a third model, AnnotationQueueAssignment: it assigns a queue to specific reviewers. So "who should review which queue" is structured too—not a link shared on the honor system, but a division of labor the platform records.
The 5-minute soft lock: human-scale concurrency control
With several reviewers eyeing one queue, how do you stop two from annotating the same item and wasting effort? A database transaction lock is too heavy and ill-fitting for the "a person reads for a while" rhythm. Langfuse uses a light time-based soft lock: a reviewer opens an item and it stamps lockedByUserId and lockedAt; judging "is it locked" only counts a lock from the last 5 minutes.
A soft lock, not a hard one: isItemLocked only counts a lock whose lockedAt is within the last 5 minutes (annotationQueueItemsRouter.ts:34-36). It solves "human concurrency"—several people on one queue don't collide—and because it auto-expires, it never holds an item permanently just because someone closed their tab.
// soft lock: only a "locked within the last 5 minutes" counts as locked—auto-expiresconstisItemLocked = (item) =>
item.lockedByUserId && item.lockedAt &&
new Date(item.lockedAt) > new Date(Date.now() - 5 * 60 * 1000); // 5-minute window// on finish: mark item COMPLETED and record the reviewer
{ status: AnnotationQueueStatus.COMPLETED, annotatorUserId: ctx.session.user.id }
// each score the reviewer fills: source=ANNOTATION, with authorUserId (who scored)
createAnnotationScore({ source: "ANNOTATION", authorUserId, name, value, dataType, comment });
// → like Lesson 29's EVAL scores, normalized and flowed back via L12 ingestion into one scores table
The scores a reviewer fills have source set to ANNOTATION, carry an authorUserId (which reviewer), and may attach a comment with the reasoning—just like Lesson 29's EVAL scores, all normalized into scores and flowed back through Lesson 12's ingestion path into the same scores table. With that, all three sources from Lesson 28 are present: API, EVAL, ANNOTATION all converge—same table, same ruler, comparable. Part 5's through-line—"turn what happened into how well"—comes full circle here.
1
enter your assigned queue
A reviewer sees the queues assigned to them (AnnotationQueueAssignment) and the PENDING items inside.
2
take one → auto-lock
Opening an item stamps lockedByUserId/lockedAt; for 5 minutes others see "locked" and skip it.
3
review the object
Per objectType, expand the TRACE / OBSERVATION / SESSION to review—reusing Lessons 25 & 26's detail views.
4
score by config + comment
Fill each of the queue's bound score configs, optionally with a comment. The scores are written as source=ANNOTATION with authorUserId.
5
submit → next item
The item is marked COMPLETED with annotatorUserId/completedAt, the lock releases, and the next PENDING item in the queue advances automatically.
🎯 Design trade-off
Why a "5-minute time-based soft lock" instead of a DB transaction lock or a permanent-hold marker? Because this lock must fit people, not machines. A DB transaction lock is millisecond-scale and ties up a connection—it can't survive "a person reading for minutes". And a permanent marker ("A is viewing" until A explicitly releases) means that once A closes the browser, loses network, or simply forgets, the item is held forever and no one can take it. The time-based soft lock pleases both ends: 5 minutes is enough for one person to finish one item and avoid collisions; and because it auto-expires, any mishap merely lets the lock "self-heal", never deadlocking. It's a neat application of "optimistic concurrency + a lease" to a human workflow—the granularity of a lock should match the rhythm of who it locks.
Why must human scores go into the same table and use the same config as AI scores? Because only with one ruler can you compare. Lay the human "gold standard" and the AI judge's scores on the same batch of traces, and a subtraction tells you where the judge is biased—this is the answer to Lesson 29's "how to trust the AI judge": use human annotation as the judge's yardstick. If each side used its own storage and its own scale, such calibration would be impossible. Langfuse having the three sources (API/EVAL/ANNOTATION) share one scores table and one set of configs is, at heart, to keep "how humans see it" and "how machines see it" always comparable side by side—the trustworthiness of the whole evaluation system ultimately rests on this.
🎯 Key points
Human annotation = the third score source (ANNOTATION): the last piece of Lesson 28's three sources. The most costly and most authoritative, often used as ground truth—setting baselines, auditing "is the AI judge right", and handling the AI's uncertain long tail.
The annotation queue makes unordered review orderly: AnnotationQueue binds a set of scoreConfigIds (uniform sheet); AnnotationQueueItem points at a TRACE/OBSERVATION/SESSION with a PENDING→COMPLETED status; AnnotationQueueAssignment assigns queues to reviewers.
The 5-minute soft lock = human concurrency control: opening stamps lockedAt, and isItemLocked only counts a lock from the last 5 minutes. Many can review one queue without colliding, and because it auto-expires it never deadlocks—the lock's granularity matches the human rhythm.
Scores converge: a reviewer's config-driven scores carry source=ANNOTATION, an authorUserId, and an optional comment; the item is marked COMPLETED with annotatorUserId; the scores flow back through Lesson 12's ingestion into the same scores table.
The loop closes: API/EVAL/ANNOTATION are same-table, same-ruler, comparable—so human scores can serve as the AI judge's yardstick. This is the bedrock of evaluation trust, and it caps Part 5's "turn what happened into how well".
🧪 Self-test - think about the design
1. Human annotation is the ANNOTATION of Lesson 28's three score sources. Beyond 'humans are more accurate than machines', it plays a unique role in the evaluation system. What is it?
replaces LLM-as-judge
makes evaluation run faster
serving as ground truth / calibrating the AI judge: laying the human 'gold standard' against AI-judge scores on the same traces measures where the judge is biased — the answer to Lesson 29's 'how to trust the AI judge'
lowers annotation cost
Show answer & explanation click to expand
Answer: C. Human scores are slow and costly, usually only on a small key sample—but that sample is the gold standard for the AI judge. Because the three sources share one table and config (ScoreSourceArray=[API,EVAL,ANNOTATION]), human and AI-judge scores compare directly on the same objects; disagreements are where the judge must fix its prompt/model. Using human annotation as the judge's yardstick is the bedrock of evaluation trust.
2. Several reviewers eye one annotation queue at once. Langfuse uses a '5-minute time-based soft lock' to stop two from annotating the same item. Versus a DB transaction lock or a permanent-hold marker, the soft lock's key benefit is?
can lock the whole queue
needs no database fields
queries faster than a transaction lock
fits the human rhythm and never deadlocks: 5 minutes suffices to finish one item and avoid collisions; and because it auto-expires, closing the browser/losing network just lets the lock self-heal, never holding an item forever
Show answer & explanation click to expand
Answer: D. isItemLocked only counts a lock whose lockedAt is within the last 5 minutes. A DB transaction lock is millisecond-scale and ties up a connection, unfit for minutes of human reading; a permanent marker holds the item forever once a reviewer closes the tab. The time-based soft lock pleases both: 5 minutes avoids collisions, and auto-expiry self-heals, never deadlocking. It's 'optimistic concurrency + a lease' applied to a human workflow — lock granularity matching the locked party's rhythm.
3. An AnnotationQueue binds a set of scoreConfigIds. What does this binding do?
decides which reviewers the queue is assigned to
it's the queue's 'uniform scoring sheet': all reviewers must score by the same set of score configs (same name, same scale) so scores are comparable and aggregatable — directly reusing Lesson 28's configs
controls the 5-minute lock duration
decides which traces to review
Show answer & explanation click to expand
Answer: B. scoreConfigIds bind Lesson 28's score configs, dictating which scores each reviewer in this queue fills and on what scale (NUMERIC/CATEGORICAL/BOOLEAN). With a unified ruler, scores across reviewers and items are comparable and aggregatable. Who it's assigned to is the separate AnnotationQueueAssignment model; which objects to review is AnnotationQueueItem — each with its own job.
💭 Open questions (no single right answer - just think or try)
By this point in Part 5, all three score sources (API/EVAL/ANNOTATION) flow into one scores table, one set of configs. Looking back at this through-line: why does Langfuse refuse to build separate tables for 'AI-judged' and 'human-judged' scores, insisting on one table and one ruler? What does this 'unified data model' buy in trust, maintainability, and future extension? Can you think of its costs or limits?