🪢 Langfuse 图解教程Langfuse Visual Guide 第六部分 · 数据集与实验Part 6 · Datasets & Experiments 35 / 55
第六部分 · 数据集与实验Part 6 · Datasets & Experiments

数据集运行与运行项Dataset runs & run items

🧭 读前Before you start
先读 第34课 的数据集与 第30课 的 eval,本课讲数据集运行与运行项。
Read Lesson 34 (datasets) and Lesson 30 (eval); this lesson covers dataset runs and run items.

上一课造好了「考卷」(数据集)。这一课讲怎么「考一场试」——一次 dataset run(数据集运行)就是拿整套题把应用从头到尾考一遍。重点有两个:每道题考一次会产生一条真实 trace,而 dataset run item 正是把「哪场考试 × 哪道题 → 哪份答卷(trace)」三者钉在一起的那条记录;以及,这些 run item 会被镜像到 ClickHouse,连同题面一起快照下来,好让「按这场考试汇总所有题的分数」又快又可复现。 理解了 run 与 run item,你就接上了第 30 课那条 dataset-run-item-upsert 评估触发器——跑一场实验,分数会自动评出来。

📋 生活类比
数据集是试卷,一次 run 就是一场考试:用某个特定的模型 / prompt 配置,把卷子上每道题都答一遍。每道题答完,都留下一份完整的答卷(一条 trace,记着完整的输入输出与每一步)。 run item 就像「答题卡存根」:它不存答案本身,只记一个三元对应——「这是第 3 场考试、第 7 道题、对应第 920 号答卷」。等阅卷老师(第 29–32 课的评分)给每份答卷打了分,把同一场考试所有题的分一汇总,就得到这场考试的总评(这次 run 平均分 0.82)。于是「换个模型再考一场,总评涨了还是跌了」一目了然——这正是下一课「实验对比」的基础。

三元链接:哪场考试 × 哪道题 → 哪份答卷

DatasetRuns 很简单:一场考试有 name、description、metadata,归属某个 dataset;同一数据集下 run 名唯一([datasetId, projectId, name])。真正的关键是 DatasetRunItems:它把 datasetRunId + datasetItemId 连到一个 traceId(外加可选 observationId)。换句话说,每一对(run × item)都对应一条真实 trace——run item 就是这个三元关系的载体。

一次 run = 拿整套题考一遍,每题留一条 trace 数据集item 1item 2item 3考卷(第34课) run「gpt-4o 场」 run items(答题卡存根)run×item1 → trace Arun×item2 → trace Brun×item3 → trace CdatasetRunId+datasetItemId→traceId trace A/B/C每题一条真实 trace run「claude 场」…同套题换配置再考 执行每道题 run item 不存答案本身,只存「run×item→trace」的三元对应——答案在它指向的那条 trace 里
run item = 三元链接的载体:schema.prisma:719 DatasetRuns([datasetId, projectId, name] 唯一),:739 DatasetRunItems 把 datasetRunId+datasetItemId 连到 traceId(+observationId)。一次 run 把整套题各执行一次,每次产出一条真实 trace;run item 记下「哪场×哪题→哪条 trace」。
示例:数据集项 × 运行 的分数热力图 run A (v1) run B (v2) run C (v3) item 1 0.6 0.9 0.9 item 2 0.8 0.8 0.4 item 3 0.3 0.7 0.9 item 4 0.9 1.0 1.0 均值 (scores_avg) 0.65 0.85 0.8 一眼看出:B 整体最稳;item 2 在 run C 退步(绿→红)
热力图让回归一眼可见(基于 dataset-run-items 的 scores;值为示例):行是数据集项、列是历次 run,格子按分数着色(红低、黄中、绿高),底部一行是每次 run 的 scores_avg。不用逐条读数,哪次 run 整体更好、哪个项在退步一眼就看出来——这里 item 2 在 run C 从绿掉到红,就是一次需要排查的回归。
一场考试DatasetRuns
一次 run 代表「用某套固定配置(模型/prompt/版本)把整个数据集跑一遍」。它有 name(如「gpt-4o-2025-baseline」)、description、metadata,归属一个 dataset。同一数据集下 run 名唯一,方便你按名字回溯每次实验。
答题卡存根DatasetRunItems
连接器:datasetRunId(哪场)+ datasetItemId(哪题)→ traceId(哪份答卷)。它本身不存输入输出——那些都在它指向的 trace 里。它只负责把「实验 / 题目 / 执行记录」三者牢牢绑定,让你能从一场 run 顺藤摸到每道题的完整执行细节。
复用一切trace 就是普通 trace
妙在 run 产生的 trace 和生产 trace毫无二致——同样的观测树(第 13、25 课)、同样能挂 score(第 28 课)、同样走摄取链路(第 12 课)。实验不是另一套系统,而是「用数据集喂出来的一批普通 trace」,前 5 部分的能力全部复用。

镜像到 ClickHouse:连题面一起快照下来

run item 在 Postgres 有一份(关系真相),但要算「这场考试平均分多少、和上一场比涨跌如何」,得在海量 run item × 海量 score 上做聚合——这是 OLAP 活,归 ClickHouse(第 17、22 课)。所以 run item 被镜像到 CH 的 dataset_run_items_rmt 表。最值得玩味的是:这张表不只存 id,还反范式地把 run 字段和题面快照了进来。

镜像 + 反范式快照:聚合快、且记住当时题面 Postgres(真相)DatasetRunItemrunId + itemId + traceId只存关系,瘦事务性强一致 ClickHouse: dataset_run_items_rmtid/run/item/trace_id …反范式:run name/desc 内联+ item input/expectedOutput 快照ReplacingMergeTree(event_ts,is_deleted)ORDER BY project/dataset/run/id为聚合而生,胖 按 run 聚合分数JOIN scores → avg这场均分 0.82 镜像 快照题面的妙处:即便日后题被改新版(第34课),这条 run item 仍记着「当时考的到底是什么」——可复现再加一道保险
反范式 = 用空间换聚合速度 + 记忆:CH 表内联了不可变的 run 字段与 item input/expectedOutput 快照(注释原文「denormalized … but snapshots are relevant」)。即使题目日后升新版,这场 run 仍记得当时题面——第 34 课的「可复现」在这里再上一道保险。引擎 ReplacingMergeTree(第 17 课同款)。源码:clickhouse/migrations/clustered/0024_dataset_run_items.up.sql。
packages/shared/clickhouse/migrations/clustered/0024_dataset_run_items.up.sqlCH 镜像表
CREATE TABLE dataset_run_items_rmt (
  id String, project_id String, dataset_run_id String,
  dataset_item_id String, dataset_id String, trace_id String, observation_id Nullable(String),
  -- 反范式:不可变的 run 字段直接内联(省一次 JOIN)
  dataset_run_name String, dataset_run_description Nullable(String), dataset_run_created_at DateTime64(3),
  -- 快照:题面在「当时」的样子(即便日后改版也记得考的是什么)
  dataset_item_input Nullable(String) CODEC(ZSTD(3)),
  dataset_item_expected_output Nullable(String) CODEC(ZSTD(3)),
  event_ts DateTime64(3), is_deleted UInt8
) ENGINE = ReplicatedReplacingMergeTree(event_ts, is_deleted)   -- 同 id 多次写,留最新(第17课)
ORDER BY (project_id, dataset_id, dataset_run_id, id);   -- 排序键贴合「按 run 聚合」
维度Postgres(DatasetRunItem)ClickHouse(dataset_run_items_rmt)
角色关系真相、事务一致聚合副本、为 OLAP 而生
存什么瘦:runId+itemId+traceId 等关系胖:再内联 run 名/描述 + 题面快照
引擎行存、强一致ReplacingMergeTree(event_ts, is_deleted)
擅长精确读单条、外键约束「按 run 分组求平均」扫一遍即得

这正是第 22 课「双存储」格局在实验域的复刻:Postgres 当权威、ClickHouse 当快照。两边不是竞争而是分工——一处保对、一处保快。

跑一场实验,分数自动评出:run 级聚合

run item 指向 trace,trace 上挂着 score(第 28 课)。把这条链走完,就能算出每场 run 的平均分——这正是 dataset-run-items.ts 在做的:JOIN scores 表,按 score 名求平均(agg_scores_avg)。而分数是怎么来的?回想第 30 课:创建 run item 会发出 dataset-run-item-upsert 事件,自动触发评估器给这条 run 的 trace 评分。于是跑一场实验,分数会自动评出来,无需手动。

数据集 N 道题 item run(一场考试) 逐题跑应用 → 每题一条 trace 自动评分 L29–32 评出 score → 挂回这条 trace 按 run 聚合 JOIN scores avg(value) by name agg_scores_avg [[helpfulness, 0.82], [toxicity, 0.03]] → 交给 L36 对比
1

建 run + 逐题执行

新建一场 run(createOrFetchDatasetRun),对每道 item 用目标配置跑一次应用,产出一条 trace。

2

建 run item(钉住版本)

记下 run×item→traceId,并钉住该 item 的 validFrom(第34课)。镜像进 CH,快照题面。

3

自动触发评估

run item 落库发出 dataset-run-item-upsert 事件 → 第30课 createEvalJobs 给这条 trace 排评估工单。

4

评分回流

裁判 / 代码 / 人工(第29–32课)评出 score,经摄取链路挂回这条 trace。

5

按 run 聚合

dataset-run-items.ts JOIN scores、按名求平均(agg_scores_avg),得「这场 run 平均分」——交给第36课对比。

packages/shared/src/server/repositories/dataset-run-items.ts按 run 聚合分数
// 每场 run 的平均分:JOIN scores、按 score 名求平均 → agg_scores_avg
type DatasetRunRecord = {
  agg_scores_avg: Array<[string, number]>;   // 如 [["helpfulness", 0.82], ["toxicity", 0.03]]
  scores: ScoreAggregate; …
};
// scoresCte:对数值分按 (name, avg_value) 聚合
WITH scores_aggregated AS ( SELECT …, avg(value) AS scores_avg … GROUP BY name )
// 于是「gpt-4o 场 vs claude 场」可以直接比 agg_scores_avg —— 下一课的实验对比

至此,Part 6 的前两课拼出了完整的实验骨架:数据集(考卷,第34课)→ run(一场考试)→ run item(答题卡存根,把题与答卷钉一起)→ trace(答卷)→ 自动评分(第29–32课)→ 按 run 聚合(这场总评)。下一课只剩最后一步:把多场 run 的总评并排对比,挑出最优配置——这就是「实验」。

🎯 设计取舍
为什么要专门建一个 run item 表,而不是直接给 trace 打个「属于哪场实验」的标签? 因为这是个多对多的关系,而且要支撑高效聚合。一条 trace 概念上属于「某场 run 的某道题」,但「run」「item」「trace」各自都是独立演化的实体(题会改版、run 会重跑、trace 有完整生命周期)。用一张专门的连接表(run item)承载这个三元关系,既让三者解耦、各自独立,又给了你一个干净的聚合入口:要算「这场考了哪些题、平均分多少」,只需扫这张表 JOIN 一下 score,而不用在庞大的 trace 海里捞带特定标签的针。关系独立建模,是它能被高效查询和聚合的前提。

为什么 CH 镜像表要反范式地把 run 字段和题面「快照」进去? 两个理由叠加。其一是聚合速度(第 17、22 课的老主题):把 run name、题面直接内联,算「按 run 分组的平均分」时省掉一次次回表 JOIN——OLAP 就吃这一套。其二更微妙,是记忆:题目会改版(第 34 课),但一场三月的实验,理应永远记得它三月考的是那时的题面。把 input/expectedOutput 在 run 时快照进 run item,等于给「可复现」再上一道保险——即便原题后来面目全非,这场实验的语境也分毫不失。快照不是冗余,而是把「当时的真相」固化下来。
🎯 本课要点
  • run = 一场考试:DatasetRuns 用某套固定配置把整个数据集跑一遍;同一数据集下 run 名唯一,便于按名回溯每次实验。
  • run item = 三元链接的载体:DatasetRunItems 把 datasetRunId+datasetItemId→traceId 钉在一起,自己不存输入输出——答案在它指向的 trace 里。这让 run/item/trace 三者解耦又可聚合。
  • run 的 trace 就是普通 trace:同样的观测树、同样能挂 score、同样走摄取链路——实验复用前 5 部分的全部能力,不是另起一套系统。
  • 镜像到 CH 并反范式快照:dataset_run_items_rmt(ReplacingMergeTree)内联 run 字段 + 快照题面 input/expectedOutput——既加速「按 run 聚合」,又让实验永远记得当时考的是什么(可复现加保险)。
  • 跑一场实验分数自动评出:建 run item 发 dataset-run-item-upsert 事件 → 第30课自动排评估 → 评分回流 → dataset-run-items.ts JOIN scores 按名求平均(agg_scores_avg)得「这场总评」,交给第36课对比。

🧪 自测 · 想一想为什么这么设计

1. Langfuse 用 DatasetRunItem 把 datasetRunId + datasetItemId 连到一个 traceId。为什么要专门建这个连接表,而不是直接给 trace 打个「属于哪场实验」的标签?
  1. 为了加密实验结果
  2. 为了把实验数据和生产数据隔离
  3. 因为 trace 不能加字段
  4. 因为这是多对多关系且要支撑高效聚合:run/item/trace 各自独立演化,专门的连接表让三者解耦,又给了干净的聚合入口(扫这表 JOIN score 即可算「这场平均分」,不用在 trace 海里捞带标签的针)
看答案与解析 点击展开
答案:D。一条 trace 概念上属于「某场 run 的某道题」,但 run、item、trace 都是独立演化的实体(题会改版、run 会重跑、trace 有完整生命周期)。用连接表承载三元关系,既让三者解耦,又给了聚合入口:算「这场考了哪些题、平均分多少」只需扫这表 JOIN score。关系独立建模是高效查询/聚合的前提。
2. ClickHouse 的 dataset_run_items_rmt 镜像表里,除了 id/run/item/trace 的关系,还反范式地内联了 run 名、并把数据项的 input/expectedOutput「快照」了进去。快照题面解决了什么独特问题?
  1. 防止题目被删除
  2. 记忆:题目会改版(第34课),但快照让一场三月的实验永远记得当时考的是哪版题面——即便原题后来面目全非,实验语境分毫不失,给「可复现」再上一道保险
  3. 加密题面
  4. 压缩存储空间
看答案与解析 点击展开
答案:B。反范式有两重收益。聚合速度:内联 run 名/题面,省回表 JOIN(OLAP 老主题)。更微妙的是记忆:题会改版,但三月的实验理应永远记得三月那版题面。把 input/expectedOutput 在 run 时快照进 run item,即便原题后来变了,这场实验的语境也不失——第34课「可复现」的再加保险。快照不是冗余,是固化「当时的真相」。
3. 在 Langfuse 里「跑一场 dataset run,分数会自动评出来」。这个自动评分是怎么被触发的?
  1. 创建 run item 会发出 dataset-run-item-upsert 事件,触发第30课的 createEvalJobs 给这条 run 的 trace 排评估工单;评分回流后 dataset-run-items.ts JOIN scores 按名求平均(agg_scores_avg)
  2. ClickHouse 自动计算
  3. 前端每秒轮询评分
  4. run 结束时手动点击评分
看答案与解析 点击展开
答案:A。回想第30课:createEvalJobs 有三个触发源,其一就是 dataset-run-item-upsert。建 run item 发此事件 → 评估器按 filter 匹配这条 run 的 trace → 排 JobExecution → 裁判/代码/人工评出 score 经摄取链路挂回 trace。最后 dataset-run-items.ts JOIN scores 按 score 名求平均得 agg_scores_avg(这场总评),交第36课对比。整条链全自动。
💭 发散思考(没有标准答案,动手或动脑想想)
  • 一次 dataset run 产生的 trace 和生产 trace「毫无二致」——同样的观测树、同样能挂 score、同样走摄取链路。这种「实验复用生产基础设施」的设计有什么好处?设想如果实验数据走一套完全独立的存储与查询栈,会带来哪些重复建设和不一致风险?

The last lesson built the "exam paper" (the dataset). This lesson is about how to "sit an exam"—a dataset run takes the whole set and quizzes the app end to end. Two focuses: each question, run once, produces a real trace, and a dataset run item is the record that pins "which exam × which question → which answer sheet (trace)" together; and these run items are mirrored to ClickHouse, snapshotting the question text along with them, so that "aggregate all questions' scores for this exam" is both fast and reproducible. Once you grasp runs and run items, you connect to Lesson 30's dataset-run-item-upsert eval trigger—run an experiment and the scores get computed automatically.

📋 Analogy
A dataset is the exam paper; a run is one sitting of the exam: with a specific model / prompt config, answer every question on the paper. Each answered question leaves a complete answer sheet (a trace recording the full input/output and every step). A run item is like an "answer-card stub": it doesn't store the answer itself, only a three-way correspondence—"this is exam #3, question #7, matching answer sheet #920". Once the graders (Lessons 29–32's scoring) score each sheet, aggregating all questions' scores for the same exam yields the exam's overall grade (this run averaged 0.82). So "swap a model, sit another exam—did the grade rise or fall" is obvious—exactly the basis for the next lesson's "experiment comparison".

The three-way link: which exam × which question → which answer sheet

DatasetRuns is simple: a sitting has a name, description, metadata, belongs to a dataset; run names are unique within a dataset ([datasetId, projectId, name]). The real key is DatasetRunItems: it links datasetRunId + datasetItemId to a traceId (plus an optional observationId). In other words, every (run × item) pair maps to one real trace—the run item is the carrier of this three-way relation.

a run = quiz the whole set once, each question leaving a trace datasetitem 1item 2item 3exam paper (L34) run "gpt-4o sitting" run items (answer-card stubs)run×item1 → trace Arun×item2 → trace Brun×item3 → trace CdatasetRunId+datasetItemId→traceId trace A/B/Cone real trace per question run "claude sitting"…same set, new config execute each question a run item stores no answer—only the "run×item→trace" correspondence; the answer is in the trace it points to
run item = carrier of the three-way link: schema.prisma:719 DatasetRuns (unique [datasetId, projectId, name]), :739 DatasetRunItems links datasetRunId+datasetItemId to traceId(+observationId). A run executes the whole set once, each producing a real trace; the run item records "which exam × which question → which trace".
Example: a dataset-item x run score heatmap run A (v1) run B (v2) run C (v3) item 1 0.6 0.9 0.9 item 2 0.8 0.8 0.4 item 3 0.3 0.7 0.9 item 4 0.9 1.0 1.0 avg (scores_avg) 0.65 0.85 0.8 at a glance: B is most consistent; item 2 regressed in run C (green→red)
The heatmap makes regressions obvious (based on dataset-run-items scores; values illustrative): rows are items, columns are runs, cells colored by score (red low, amber mid, green high), with each run's scores_avg at the bottom. Without reading numbers one by one, you see which run is better overall and which item regressed — here item 2 drops green→red in run C, a regression worth investigating.
an examDatasetRuns
A run means "with a fixed config (model/prompt/version), run the whole dataset once". It has a name (e.g. "gpt-4o-2025-baseline"), description, metadata, belongs to a dataset. Run names are unique per dataset, so you can trace each experiment by name.
answer-card stubDatasetRunItems
The connector: datasetRunId (which exam) + datasetItemId (which question) → traceId (which sheet). It stores no input/output—those are in the trace it points to. It only binds "experiment / question / execution record" together, letting you follow a run down to each question's full execution detail.
reuse allthe trace is an ordinary trace
The beauty: a run's traces are indistinguishable from production traces—same observation tree (Lessons 13, 25), same ability to attach scores (Lesson 28), same ingestion path (Lesson 12). Experiments aren't a separate system but "a batch of ordinary traces fed by a dataset", reusing all of the first five parts.

Mirror to ClickHouse: snapshotting the question text too

A run item lives in Postgres (the relational truth), but to compute "what's this exam's average score, up or down from last time" you must aggregate over masses of run items × masses of scores—an OLAP job, belonging to ClickHouse (Lessons 17, 22). So run items are mirrored to CH's dataset_run_items_rmt table. The intriguing part: this table stores not just ids but denormalizes the run fields and snapshots the question text into it.

mirror + denormalized snapshot: fast aggregation, and remembers the question Postgres (truth)DatasetRunItemrunId + itemId + traceIdstores only relations, thintransactional, consistent CH: dataset_run_items_rmtid/run/item/trace_id …denormalized: run name/desc inlined+ item input/expectedOutput snapshotReplacingMergeTree(event_ts,is_deleted)ORDER BY project/dataset/run/idbuilt for aggregation, fat aggregate by runJOIN scores → avgthis run averaged 0.82 mirror snapshot's payoff: even if the question is later versioned up (L34), this run item still remembers "what was actually tested then"—reproducibility, doubly insured
denormalize = trade space for aggregation speed + memory: the CH table inlines immutable run fields and snapshots item input/expectedOutput (the source comment: "denormalized … but snapshots are relevant"). Even if the question is later versioned up, this run still remembers the question text then—Lesson 34's reproducibility, doubly insured. Engine ReplacingMergeTree (same as Lesson 17). Source: clickhouse/migrations/clustered/0024_dataset_run_items.up.sql.
packages/shared/clickhouse/migrations/clustered/0024_dataset_run_items.up.sqlCH mirror table
CREATE TABLE dataset_run_items_rmt (
  id String, project_id String, dataset_run_id String,
  dataset_item_id String, dataset_id String, trace_id String, observation_id Nullable(String),
  -- denormalized: immutable run fields inlined (saves a JOIN)
  dataset_run_name String, dataset_run_description Nullable(String), dataset_run_created_at DateTime64(3),
  -- snapshot: the question text "as of then" (remembers what was tested even after a version bump)
  dataset_item_input Nullable(String) CODEC(ZSTD(3)),
  dataset_item_expected_output Nullable(String) CODEC(ZSTD(3)),
  event_ts DateTime64(3), is_deleted UInt8
) ENGINE = ReplicatedReplacingMergeTree(event_ts, is_deleted)   -- many writes per id, keep latest (L17)
ORDER BY (project_id, dataset_id, dataset_run_id, id);   -- sort key fits "aggregate by run"
dimensionPostgres (DatasetRunItem)ClickHouse (dataset_run_items_rmt)
rolerelational truth, transactionalaggregation copy, built for OLAP
storesthin: runId+itemId+traceId relationsfat: also inlines run name/desc + question snapshot
enginerow store, strongly consistentReplacingMergeTree(event_ts, is_deleted)
good atexact single-row reads, FK constraints"group by run, average" in one scan

This is Lesson 22's "dual-store" pattern replayed in the experiment domain: Postgres as the authority, ClickHouse as the snapshot. The two don't compete but divide labor—one keeps it correct, one keeps it fast.

Run an experiment, scores compute themselves: run-level aggregation

A run item points to a trace, the trace carries scores (Lesson 28). Walk that chain and you can compute each run's average score—exactly what dataset-run-items.ts does: JOIN the scores table, average by score name (agg_scores_avg). And where do the scores come from? Recall Lesson 30: creating a run item emits a dataset-run-item-upsert event, automatically triggering evaluators to score this run's traces. So run an experiment and the scores compute themselves, no manual work.

dataset N items run (one exam) app per item → one trace each auto-scoring L29–32 → score → back onto trace aggregate per run JOIN scores avg(value) by name agg_scores_avg [[helpfulness, 0.82], [toxicity, 0.03]] → to L36 compare
1

create run + execute per question

Create a run (createOrFetchDatasetRun), run the app once per item with the target config, producing a trace.

2

create run item (pin the version)

Record run×item→traceId, pinning the item's validFrom (L34). Mirror into CH, snapshot the question.

3

auto-trigger evaluation

Persisting the run item emits a dataset-run-item-upsert event → Lesson 30's createEvalJobs queues eval for this trace.

4

scores flow back

Judge / code / human (Lessons 29–32) produce scores, attached back to this trace via the ingestion path.

5

aggregate by run

dataset-run-items.ts JOINs scores, averages by name (agg_scores_avg), yielding "this run's average"—handed to Lesson 36 for comparison.

packages/shared/src/server/repositories/dataset-run-items.tsaggregate scores by run
// each run's average: JOIN scores, average by score name → agg_scores_avg
type DatasetRunRecord = {
  agg_scores_avg: Array<[string, number]>;   // e.g. [["helpfulness", 0.82], ["toxicity", 0.03]]
  scores: ScoreAggregate; …
};
// scoresCte: aggregate numeric scores as (name, avg_value)
WITH scores_aggregated AS ( SELECT …, avg(value) AS scores_avg … GROUP BY name )
// so "gpt-4o sitting vs claude sitting" compares directly on agg_scores_avg — next lesson's comparison

With that, Part 6's first two lessons assemble the full experiment skeleton: dataset (exam paper, L34) → run (a sitting) → run item (answer-card stub, pinning question to sheet) → trace (the sheet) → auto-scoring (Lessons 29–32) → aggregate by run (the exam's grade). The next lesson is the last step: place multiple runs' grades side by side and pick the best config—that's an "experiment".

🎯 Design trade-off
Why build a dedicated run-item table rather than just tag a trace with "which experiment it belongs to"? Because it's a many-to-many relation that must support efficient aggregation. A trace conceptually belongs to "some question of some run", but "run", "item", and "trace" each evolve independently (questions get versioned, runs get re-run, traces have full lifecycles). A dedicated junction table (the run item) carrying this three-way relation both keeps the three decoupled and independent, and gives you a clean aggregation entry: to compute "which questions this exam covered, what the average is", just scan this table and JOIN scores—rather than fishing for tagged needles in a vast trace ocean. Modeling the relation independently is the precondition for efficient query and aggregation.

Why does the CH mirror table denormalize and "snapshot" the run fields and question text? Two compounding reasons. One is aggregation speed (Lessons 17, 22's old theme): inlining run name and question text saves repeated back-to-table JOINs when computing "average grouped by run"—OLAP thrives on this. The other is subtler, memory: questions get versioned (Lesson 34), but a March experiment ought to forever remember the question text it tested then. Snapshotting input/expectedOutput into the run item at run time doubly insures "reproducibility"—even if the original question later changes beyond recognition, this experiment's context is preserved intact. The snapshot isn't redundancy but freezing "the truth as of then".
🎯 Key points
  • A run = an exam sitting: DatasetRuns runs the whole dataset with a fixed config; run names are unique per dataset, so each experiment is traceable by name.
  • A run item = carrier of the three-way link: DatasetRunItems pins datasetRunId+datasetItemId→traceId, storing no input/output—the answer is in the trace it points to. This keeps run/item/trace decoupled yet aggregatable.
  • A run's trace is an ordinary trace: same observation tree, same scores, same ingestion path—experiments reuse all of the first five parts, not a separate system.
  • Mirror to CH with a denormalized snapshot: dataset_run_items_rmt (ReplacingMergeTree) inlines run fields + snapshots question input/expectedOutput—both speeding "aggregate by run" and making an experiment forever remember what it tested (reproducibility, insured).
  • Run an experiment, scores auto-compute: creating a run item emits dataset-run-item-upsert → Lesson 30 auto-queues eval → scores flow back → dataset-run-items.ts JOINs scores, averages by name (agg_scores_avg) into "this run's grade", handed to Lesson 36 for comparison.

🧪 Self-test - think about the design

1. Langfuse's DatasetRunItem links datasetRunId + datasetItemId to a traceId. Why a dedicated junction table rather than just tagging a trace with 'which experiment it belongs to'?
  1. to encrypt experiment results
  2. to isolate experiment data from production data
  3. because traces can't take extra fields
  4. because it's a many-to-many relation needing efficient aggregation: run/item/trace evolve independently, a dedicated junction table decouples them and gives a clean aggregation entry (scan it and JOIN scores to compute 'this run's average', not fish tagged needles in a trace ocean)
Show answer & explanation click to expand
Answer: D. A trace conceptually belongs to 'some question of some run', but run, item, and trace evolve independently (questions versioned, runs re-run, traces have full lifecycles). A junction table carrying the three-way relation decouples them and gives an aggregation entry: computing 'which questions this exam covered, what the average' just scans it and JOINs scores. Modeling the relation independently is the precondition for efficient query/aggregation.
2. In ClickHouse's dataset_run_items_rmt mirror table, besides the id/run/item/trace relations it denormalizes the run name and 'snapshots' the item's input/expectedOutput. What unique problem does snapshotting the question text solve?
  1. preventing question deletion
  2. memory: questions get versioned (L34), but the snapshot lets a March experiment forever remember the question text it tested then—even if the original later changes beyond recognition, the experiment's context is preserved, doubly insuring reproducibility
  3. encrypting the question
  4. compressing storage
Show answer & explanation click to expand
Answer: B. Denormalization has two payoffs. Aggregation speed: inlining run name/question saves back-to-table JOINs (the OLAP theme). Subtler is memory: questions get versioned, but a March experiment should forever remember March's question text. Snapshotting input/expectedOutput into the run item at run time preserves the experiment's context even if the original later changes—doubly insuring L34's reproducibility. The snapshot isn't redundancy but freezing 'the truth as of then'.
3. In Langfuse, 'running a dataset run computes scores automatically'. How is this auto-scoring triggered?
  1. creating a run item emits a dataset-run-item-upsert event, triggering Lesson 30's createEvalJobs to queue eval for this run's trace; once scores flow back, dataset-run-items.ts JOINs scores and averages by name (agg_scores_avg)
  2. ClickHouse computes it automatically
  3. the frontend polls and scores every second
  4. manually clicking score when the run finishes
Show answer & explanation click to expand
Answer: A. Recall Lesson 30: createEvalJobs has three trigger sources, one being dataset-run-item-upsert. Creating a run item emits this event → evaluators filter-match this run's trace → queue a JobExecution → judge/code/human produce scores attached back via ingestion. Finally dataset-run-items.ts JOINs scores and averages by name into agg_scores_avg (the run's grade), handed to Lesson 36. The whole chain is automatic.
💭 Open questions (no single right answer - just think or try)
  • The traces a dataset run produces are 'indistinguishable' from production traces—same observation tree, same scores, same ingestion path. What are the benefits of this 'experiments reuse production infrastructure' design? Imagine if experiment data went through a completely separate storage and query stack—what duplication and inconsistency risks would that bring?