run item = 三元链接的载体:schema.prisma:719 DatasetRuns([datasetId, projectId, name] 唯一),:739 DatasetRunItems 把 datasetRunId+datasetItemId 连到 traceId(+observationId)。一次 run 把整套题各执行一次,每次产出一条真实 trace;run item 记下「哪场×哪题→哪条 trace」。
热力图让回归一眼可见(基于 dataset-run-items 的 scores;值为示例):行是数据集项、列是历次 run,格子按分数着色(红低、黄中、绿高),底部一行是每次 run 的 scores_avg。不用逐条读数,哪次 run 整体更好、哪个项在退步一眼就看出来——这里 item 2 在 run C 从绿掉到红,就是一次需要排查的回归。
一场考试DatasetRuns
一次 run 代表「用某套固定配置(模型/prompt/版本)把整个数据集跑一遍」。它有 name(如「gpt-4o-2025-baseline」)、description、metadata,归属一个 dataset。同一数据集下 run 名唯一,方便你按名字回溯每次实验。
dataset-run-items.ts JOIN scores、按名求平均(agg_scores_avg),得「这场 run 平均分」——交给第36课对比。
packages/shared/src/server/repositories/dataset-run-items.ts按 run 聚合分数
// 每场 run 的平均分:JOIN scores、按 score 名求平均 → agg_scores_avgtype DatasetRunRecord = {
agg_scores_avg: Array<[string, number]>; // 如 [["helpfulness", 0.82], ["toxicity", 0.03]]
scores: ScoreAggregate; …
};
// scoresCte:对数值分按 (name, avg_value) 聚合
WITH scores_aggregated AS ( SELECT …, avg(value) AS scores_avg … GROUP BY name )
// 于是「gpt-4o 场 vs claude 场」可以直接比 agg_scores_avg —— 下一课的实验对比
至此,Part 6 的前两课拼出了完整的实验骨架:数据集(考卷,第34课)→ run(一场考试)→ run item(答题卡存根,把题与答卷钉一起)→ trace(答卷)→ 自动评分(第29–32课)→ 按 run 聚合(这场总评)。下一课只剩最后一步:把多场 run 的总评并排对比,挑出最优配置——这就是「实验」。
🎯 设计取舍
为什么要专门建一个 run item 表,而不是直接给 trace 打个「属于哪场实验」的标签? 因为这是个多对多的关系,而且要支撑高效聚合。一条 trace 概念上属于「某场 run 的某道题」,但「run」「item」「trace」各自都是独立演化的实体(题会改版、run 会重跑、trace 有完整生命周期)。用一张专门的连接表(run item)承载这个三元关系,既让三者解耦、各自独立,又给了你一个干净的聚合入口:要算「这场考了哪些题、平均分多少」,只需扫这张表 JOIN 一下 score,而不用在庞大的 trace 海里捞带特定标签的针。关系独立建模,是它能被高效查询和聚合的前提。
为什么 CH 镜像表要反范式地把 run 字段和题面「快照」进去? 两个理由叠加。其一是聚合速度(第 17、22 课的老主题):把 run name、题面直接内联,算「按 run 分组的平均分」时省掉一次次回表 JOIN——OLAP 就吃这一套。其二更微妙,是记忆:题目会改版(第 34 课),但一场三月的实验,理应永远记得它三月考的是那时的题面。把 input/expectedOutput 在 run 时快照进 run item,等于给「可复现」再上一道保险——即便原题后来面目全非,这场实验的语境也分毫不失。快照不是冗余,而是把「当时的真相」固化下来。
🎯 本课要点
run = 一场考试:DatasetRuns 用某套固定配置把整个数据集跑一遍;同一数据集下 run 名唯一,便于按名回溯每次实验。
答案:B。反范式有两重收益。聚合速度:内联 run 名/题面,省回表 JOIN(OLAP 老主题)。更微妙的是记忆:题会改版,但三月的实验理应永远记得三月那版题面。把 input/expectedOutput 在 run 时快照进 run item,即便原题后来变了,这场实验的语境也不失——第34课「可复现」的再加保险。快照不是冗余,是固化「当时的真相」。
一次 dataset run 产生的 trace 和生产 trace「毫无二致」——同样的观测树、同样能挂 score、同样走摄取链路。这种「实验复用生产基础设施」的设计有什么好处?设想如果实验数据走一套完全独立的存储与查询栈,会带来哪些重复建设和不一致风险?
The last lesson built the "exam paper" (the dataset). This lesson is about how to "sit an exam"—a dataset run takes the whole set and quizzes the app end to end. Two focuses: each question, run once, produces a real trace, and a dataset run item is the record that pins "which exam × which question → which answer sheet (trace)" together; and these run items are mirrored to ClickHouse, snapshotting the question text along with them, so that "aggregate all questions' scores for this exam" is both fast and reproducible.
Once you grasp runs and run items, you connect to Lesson 30's dataset-run-item-upsert eval trigger—run an experiment and the scores get computed automatically.
📋 Analogy
A dataset is the exam paper; a run is one sitting of the exam: with a specific model / prompt config, answer every question on the paper. Each answered question leaves a complete answer sheet (a trace recording the full input/output and every step).
A run item is like an "answer-card stub": it doesn't store the answer itself, only a three-way correspondence—"this is exam #3, question #7, matching answer sheet #920". Once the graders (Lessons 29–32's scoring) score each sheet, aggregating all questions' scores for the same exam yields the exam's overall grade (this run averaged 0.82). So "swap a model, sit another exam—did the grade rise or fall" is obvious—exactly the basis for the next lesson's "experiment comparison".
The three-way link: which exam × which question → which answer sheet
DatasetRuns is simple: a sitting has a name, description, metadata, belongs to a dataset; run names are unique within a dataset ([datasetId, projectId, name]). The real key is DatasetRunItems: it links datasetRunId + datasetItemId to a traceId (plus an optional observationId). In other words, every (run × item) pair maps to one real trace—the run item is the carrier of this three-way relation.
run item = carrier of the three-way link: schema.prisma:719 DatasetRuns (unique [datasetId, projectId, name]), :739 DatasetRunItems links datasetRunId+datasetItemId to traceId(+observationId). A run executes the whole set once, each producing a real trace; the run item records "which exam × which question → which trace".
The heatmap makes regressions obvious (based on dataset-run-items scores; values illustrative): rows are items, columns are runs, cells colored by score (red low, amber mid, green high), with each run's scores_avg at the bottom. Without reading numbers one by one, you see which run is better overall and which item regressed — here item 2 drops green→red in run C, a regression worth investigating.
an examDatasetRuns
A run means "with a fixed config (model/prompt/version), run the whole dataset once". It has a name (e.g. "gpt-4o-2025-baseline"), description, metadata, belongs to a dataset. Run names are unique per dataset, so you can trace each experiment by name.
answer-card stubDatasetRunItems
The connector: datasetRunId (which exam) + datasetItemId (which question) → traceId (which sheet). It stores no input/output—those are in the trace it points to. It only binds "experiment / question / execution record" together, letting you follow a run down to each question's full execution detail.
reuse allthe trace is an ordinary trace
The beauty: a run's traces are indistinguishable from production traces—same observation tree (Lessons 13, 25), same ability to attach scores (Lesson 28), same ingestion path (Lesson 12). Experiments aren't a separate system but "a batch of ordinary traces fed by a dataset", reusing all of the first five parts.
Mirror to ClickHouse: snapshotting the question text too
A run item lives in Postgres (the relational truth), but to compute "what's this exam's average score, up or down from last time" you must aggregate over masses of run items × masses of scores—an OLAP job, belonging to ClickHouse (Lessons 17, 22). So run items are mirrored to CH's dataset_run_items_rmt table. The intriguing part: this table stores not just ids but denormalizes the run fields and snapshots the question text into it.
denormalize = trade space for aggregation speed + memory: the CH table inlines immutable run fields and snapshots item input/expectedOutput (the source comment: "denormalized … but snapshots are relevant"). Even if the question is later versioned up, this run still remembers the question text then—Lesson 34's reproducibility, doubly insured. Engine ReplacingMergeTree (same as Lesson 17). Source: clickhouse/migrations/clustered/0024_dataset_run_items.up.sql.
CREATE TABLE dataset_run_items_rmt (
id String, project_id String, dataset_run_id String,
dataset_item_id String, dataset_id String, trace_id String, observation_id Nullable(String),
-- denormalized: immutable run fields inlined (saves a JOIN)
dataset_run_name String, dataset_run_description Nullable(String), dataset_run_created_at DateTime64(3),
-- snapshot: the question text "as of then" (remembers what was tested even after a version bump)
dataset_item_input Nullable(String) CODEC(ZSTD(3)),
dataset_item_expected_output Nullable(String) CODEC(ZSTD(3)),
event_ts DateTime64(3), is_deleted UInt8
) ENGINE = ReplicatedReplacingMergeTree(event_ts, is_deleted) -- many writes per id, keep latest (L17)ORDER BY (project_id, dataset_id, dataset_run_id, id); -- sort key fits "aggregate by run"
dimension
Postgres (DatasetRunItem)
ClickHouse (dataset_run_items_rmt)
role
relational truth, transactional
aggregation copy, built for OLAP
stores
thin: runId+itemId+traceId relations
fat: also inlines run name/desc + question snapshot
engine
row store, strongly consistent
ReplacingMergeTree(event_ts, is_deleted)
good at
exact single-row reads, FK constraints
"group by run, average" in one scan
This is Lesson 22's "dual-store" pattern replayed in the experiment domain: Postgres as the authority, ClickHouse as the snapshot. The two don't compete but divide labor—one keeps it correct, one keeps it fast.
Run an experiment, scores compute themselves: run-level aggregation
A run item points to a trace, the trace carries scores (Lesson 28). Walk that chain and you can compute each run's average score—exactly what dataset-run-items.ts does: JOIN the scores table, average by score name (agg_scores_avg). And where do the scores come from? Recall Lesson 30: creating a run item emits a dataset-run-item-upsert event, automatically triggering evaluators to score this run's traces. So run an experiment and the scores compute themselves, no manual work.
1
create run + execute per question
Create a run (createOrFetchDatasetRun), run the app once per item with the target config, producing a trace.
2
create run item (pin the version)
Record run×item→traceId, pinning the item's validFrom (L34). Mirror into CH, snapshot the question.
3
auto-trigger evaluation
Persisting the run item emits a dataset-run-item-upsert event → Lesson 30's createEvalJobs queues eval for this trace.
4
scores flow back
Judge / code / human (Lessons 29–32) produce scores, attached back to this trace via the ingestion path.
5
aggregate by run
dataset-run-items.ts JOINs scores, averages by name (agg_scores_avg), yielding "this run's average"—handed to Lesson 36 for comparison.
packages/shared/src/server/repositories/dataset-run-items.tsaggregate scores by run
// each run's average: JOIN scores, average by score name → agg_scores_avgtype DatasetRunRecord = {
agg_scores_avg: Array<[string, number]>; // e.g. [["helpfulness", 0.82], ["toxicity", 0.03]]
scores: ScoreAggregate; …
};
// scoresCte: aggregate numeric scores as (name, avg_value)
WITH scores_aggregated AS ( SELECT …, avg(value) AS scores_avg … GROUP BY name )
// so "gpt-4o sitting vs claude sitting" compares directly on agg_scores_avg — next lesson's comparison
With that, Part 6's first two lessons assemble the full experiment skeleton: dataset (exam paper, L34) → run (a sitting) → run item (answer-card stub, pinning question to sheet) → trace (the sheet) → auto-scoring (Lessons 29–32) → aggregate by run (the exam's grade). The next lesson is the last step: place multiple runs' grades side by side and pick the best config—that's an "experiment".
🎯 Design trade-off
Why build a dedicated run-item table rather than just tag a trace with "which experiment it belongs to"? Because it's a many-to-many relation that must support efficient aggregation. A trace conceptually belongs to "some question of some run", but "run", "item", and "trace" each evolve independently (questions get versioned, runs get re-run, traces have full lifecycles). A dedicated junction table (the run item) carrying this three-way relation both keeps the three decoupled and independent, and gives you a clean aggregation entry: to compute "which questions this exam covered, what the average is", just scan this table and JOIN scores—rather than fishing for tagged needles in a vast trace ocean. Modeling the relation independently is the precondition for efficient query and aggregation.
Why does the CH mirror table denormalize and "snapshot" the run fields and question text? Two compounding reasons. One is aggregation speed (Lessons 17, 22's old theme): inlining run name and question text saves repeated back-to-table JOINs when computing "average grouped by run"—OLAP thrives on this. The other is subtler, memory: questions get versioned (Lesson 34), but a March experiment ought to forever remember the question text it tested then. Snapshotting input/expectedOutput into the run item at run time doubly insures "reproducibility"—even if the original question later changes beyond recognition, this experiment's context is preserved intact. The snapshot isn't redundancy but freezing "the truth as of then".
🎯 Key points
A run = an exam sitting: DatasetRuns runs the whole dataset with a fixed config; run names are unique per dataset, so each experiment is traceable by name.
A run item = carrier of the three-way link: DatasetRunItems pins datasetRunId+datasetItemId→traceId, storing no input/output—the answer is in the trace it points to. This keeps run/item/trace decoupled yet aggregatable.
A run's trace is an ordinary trace: same observation tree, same scores, same ingestion path—experiments reuse all of the first five parts, not a separate system.
Mirror to CH with a denormalized snapshot: dataset_run_items_rmt (ReplacingMergeTree) inlines run fields + snapshots question input/expectedOutput—both speeding "aggregate by run" and making an experiment forever remember what it tested (reproducibility, insured).
Run an experiment, scores auto-compute: creating a run item emits dataset-run-item-upsert → Lesson 30 auto-queues eval → scores flow back → dataset-run-items.ts JOINs scores, averages by name (agg_scores_avg) into "this run's grade", handed to Lesson 36 for comparison.
🧪 Self-test - think about the design
1. Langfuse's DatasetRunItem links datasetRunId + datasetItemId to a traceId. Why a dedicated junction table rather than just tagging a trace with 'which experiment it belongs to'?
to encrypt experiment results
to isolate experiment data from production data
because traces can't take extra fields
because it's a many-to-many relation needing efficient aggregation: run/item/trace evolve independently, a dedicated junction table decouples them and gives a clean aggregation entry (scan it and JOIN scores to compute 'this run's average', not fish tagged needles in a trace ocean)
Show answer & explanation click to expand
Answer: D. A trace conceptually belongs to 'some question of some run', but run, item, and trace evolve independently (questions versioned, runs re-run, traces have full lifecycles). A junction table carrying the three-way relation decouples them and gives an aggregation entry: computing 'which questions this exam covered, what the average' just scans it and JOINs scores. Modeling the relation independently is the precondition for efficient query/aggregation.
2. In ClickHouse's dataset_run_items_rmt mirror table, besides the id/run/item/trace relations it denormalizes the run name and 'snapshots' the item's input/expectedOutput. What unique problem does snapshotting the question text solve?
preventing question deletion
memory: questions get versioned (L34), but the snapshot lets a March experiment forever remember the question text it tested then—even if the original later changes beyond recognition, the experiment's context is preserved, doubly insuring reproducibility
encrypting the question
compressing storage
Show answer & explanation click to expand
Answer: B. Denormalization has two payoffs. Aggregation speed: inlining run name/question saves back-to-table JOINs (the OLAP theme). Subtler is memory: questions get versioned, but a March experiment should forever remember March's question text. Snapshotting input/expectedOutput into the run item at run time preserves the experiment's context even if the original later changes—doubly insuring L34's reproducibility. The snapshot isn't redundancy but freezing 'the truth as of then'.
3. In Langfuse, 'running a dataset run computes scores automatically'. How is this auto-scoring triggered?
creating a run item emits a dataset-run-item-upsert event, triggering Lesson 30's createEvalJobs to queue eval for this run's trace; once scores flow back, dataset-run-items.ts JOINs scores and averages by name (agg_scores_avg)
ClickHouse computes it automatically
the frontend polls and scores every second
manually clicking score when the run finishes
Show answer & explanation click to expand
Answer: A. Recall Lesson 30: createEvalJobs has three trigger sources, one being dataset-run-item-upsert. Creating a run item emits this event → evaluators filter-match this run's trace → queue a JobExecution → judge/code/human produce scores attached back via ingestion. Finally dataset-run-items.ts JOINs scores and averages by name into agg_scores_avg (the run's grade), handed to Lesson 36. The whole chain is automatic.
💭 Open questions (no single right answer - just think or try)
The traces a dataset run produces are 'indistinguishable' from production traces—same observation tree, same scores, same ingestion path. What are the benefits of this 'experiments reuse production infrastructure' design? Imagine if experiment data went through a completely separate storage and query stack—what duplication and inconsistency risks would that bring?