评估需要「考题」。你想知道改了 prompt、换了模型之后应用变好还是变差——就得有一套固定的测试用例反复地考它。这套测试用例,就是数据集(dataset);其中每一道题,就是一个数据项(dataset item):一份输入 + 一份期望输出。 这一课讲数据集和数据项的数据模型,重点是两个让人眼前一亮的设计:数据项可以从真实的生产 trace「提拔」成测试用例(线上遇到的难题,直接变成回归测试题),以及数据项是版本化的——改一次题留一版历史,于是「上次实验用的到底是哪版题」永远查得到。
数据模型很直观:一个 Dataset 装着一组 DatasetItem。Dataset 有名字、描述,还能挂 inputSchema / expectedOutputSchema 给题目定个格式规范。每个 DatasetItem 的核心是三样:input(题干)、expectedOutput(参考答案)、metadata(旁注)。但真正巧妙的是另外两个字段:sourceTraceId 和 sourceObservationId——它们记着「这道题是从哪条真实 trace 来的」。
schema.prisma:685 的 DatasetItem 有 input/expectedOutput/metadata,外加 sourceTraceId/sourceObservationId——后两者让你把一条真实 trace(第 13 课)提拔成测试用例。Dataset(:661)还能用 inputSchema/expectedOutputSchema 约束题目格式。domain/dataset-items.ts;值为示例):每个 item 有 input(喂给应用的输入)、expectedOutput(参考答案)、metadata。一次 run 就是拿这些 input 跑一遍当前应用,把产出和 expectedOutput 比、评分(第 35 课)。于是「改一版好不好」有了固定考卷,可复现、可对比。input 喂给应用的输入、expectedOutput 你期望它给出的参考答案。评估时就拿应用的真实输出和 expectedOutput 对照(或交给第 29–31 课的评判)。metadata 放标签、难度等旁注。sourceTraceId 记下出处。于是你的测试集紧贴真实流量分布,而不是凭空想象的玩具题。inputSchema/expectedOutputSchema)给新题定规矩。一个建好的数据集能干三件大事——它是 Part 6 后两课的地基:
改 prompt / 换模型后,把整套题重跑一遍,对照分数有没有掉——防止「修了一个 bug,带崩三个 case」。
同一套题,用不同配置各跑一次(第 35–36 课的 run),把多次结果并排比,挑出最优组合。
数据项的 input/expectedOutput 也能当作 prompt 里的示例(第 37 课),让模型照着标准答案的样子答。
数据项最精巧的设计是版本化。看主键就懂了:@@id([id, projectId, validFrom])——同一道题(同 id)可以有多行,靠 validFrom 区分版本。每行还有一个 validTo:为空(null)的那行就是当前生效版。改题时,系统在一个事务里做两步——给旧版盖上 validTo(关闭它的有效区间)、再插入一行新版。这正是数据库里经典的「缓慢变化维 / 双时态」手法。(该版本化实现由 LANGFUSE_DATASET_SERVICE_WRITE_TO/READ_FROM_VERSIONED_IMPLEMENTATION 开关控制,executeWithDatasetServiceStrategy 据此在 STATEFUL 与 VERSIONED 间切换。)
validTo IS NULL 的那行。改题在事务里「关旧版(盖 validTo)+ 插新版(新 validFrom)」。一次 dataset run 钉住某个 validFrom,所以它永远看到「当时那版题」——这正是第 30 课 eval 用 datasetItemValidFrom 的原因。源码:dataset-items.ts:418-470。// 在一个事务里:先关旧版,再插新版(缓慢变化维 Type 2) await prisma.$transaction(async (tx) => { // 0. 找到当前生效版(validTo 为空的那行) const current = await tx.datasetItem.findFirst({ where: { id: itemId, projectId, validTo: null }, orderBy: { validFrom: "desc" } }); const newValidFrom = new Date(Math.max(Date.now(), baseTs + 1)); // 1. 给旧版盖上 validTo —— 关闭它的有效区间 if (current) await tx.datasetItem.update({ where: {…current.validFrom}, data: { validTo: newValidFrom } }); // 2. 插入新版 —— validFrom = 刚才那个时刻,validTo 默认 null(成为新的当前版) await tx.datasetItem.create({ data: { ...itemData, validFrom: newValidFrom } }); });
为什么大费周章不直接「原地改」?因为实验必须可复现。你三月跑了一次实验、打了分;六月改了几道题再跑一次。如果题是「原地改」的,那三月那次的分就失去了语境——你不知道它当时考的是哪版题,两次实验根本没法公平比较。版本化把每道题的每一版都钉在时间轴上,于是「三月的 run 用的是 t2 版、六月的 run 用的是 t3 版」清清楚楚,历史结论永远可解释。
| id(逻辑题) | validFrom | validTo | 是否当前 | 内容 |
|---|---|---|---|---|
| item-42 | t1 | t2 | 否(历史) | 题干 v1 |
| item-42 | t2 | t3 | 否(历史) | 题干 v2 |
| item-42 | t3 | null | 是 ✓ | 题干 v3(当前) |
三行同一个 id、不同 validFrom,靠 validTo 是否为空区分「当前 vs 历史」。查「现在这道题长啥样」就 WHERE validTo IS NULL;查「t2 那次实验用的是啥」就找 validFrom ≤ t2 < validTo 的那行——一张表同时回答「现在」和「任意历史时刻」两个问题,这就是双时态的威力。
validFrom/validTo 给每道题的每一版钉上时间区间,让每次 run 都能钉死它当时用的那一版(第 30 课 eval 取数据时正是带着 datasetItemValidFrom 精确回放)。这是「不可变历史 + 可移动当前指针」的经典思路,和第 37 课的 prompt 版本、git 的 commit 是同一种工程信念:真相一旦记录就不抹改,要变就追加新版本。sourceTraceId 让「发现问题 → 收录成回归题 → 下次改进后验证」形成闭环——观测平台不只让你看见问题,还顺手把问题变成防止它复发的「疫苗」。
input(题干)+ expectedOutput(参考答案)+ metadata。它是评估的「考卷」,配合第 28–32 课的 score 用。sourceTraceId/sourceObservationId 让你把一条真实 trace(第 13 课)一键收录成测试用例——回归集紧贴真实流量分布,而非凭空臆造。(id, projectId, validFrom) 让同一道题多版本共存;当前版 = validTo IS NULL。改题在事务里「关旧版(盖 validTo)+ 插新版」——缓慢变化维 / 双时态。validFrom,永远看到「当时那版题」(第 30 课 eval 用 datasetItemValidFrom 精确回放)。历史实验因此永远可解释、可公平对比。Evaluation needs "exam questions". To know whether your app got better or worse after you changed a prompt or swapped a model, you need a fixed set of test cases to quiz it on repeatedly. That set is a dataset; each question in it is a dataset item: one input + one expected output. This lesson covers the data model of datasets and items, focusing on two delightful designs: an item can be "promoted" from a real production trace into a test case (a hard case you hit in production becomes a regression test), and items are versioned—editing a question keeps a version of history, so "which version of the question did last experiment use" is always answerable.
The data model is intuitive: a Dataset holds a set of DatasetItems. A Dataset has a name, a description, and can attach inputSchema / expectedOutputSchema to give questions a format spec. Each DatasetItem's core is three things: input (the prompt), expectedOutput (the reference answer), metadata (side notes). But the truly clever fields are two others: sourceTraceId and sourceObservationId—they record "which real trace this question came from".
schema.prisma:685's DatasetItem has input/expectedOutput/metadata, plus sourceTraceId/sourceObservationId—the latter let you promote a real trace (Lesson 13) into a test case. A Dataset (:661) can also constrain question format via inputSchema/expectedOutputSchema.domain/dataset-items.ts; values illustrative): each item has input (fed to the app), expectedOutput (the reference), and metadata. A run feeds these inputs through the current app and compares the output to expectedOutput, scoring it (Lesson 35). So "is this revision better" gets a fixed exam — reproducible and comparable.input is fed to the app, expectedOutput is the reference answer you expect. Evaluation compares the app's real output against expectedOutput (or hands it to Lessons 29–31's judges). metadata holds tags, difficulty, and other notes.sourceTraceId recording its origin. So your test set hugs the real traffic distribution, not imagined toy questions.inputSchema/expectedOutputSchema) to set rules for new questions.A built dataset can do three big things—it's the foundation for the next two lessons of Part 6:
After changing a prompt / swapping a model, re-run the whole set and check whether scores dropped—preventing "fix one bug, break three cases".
The same set, run once per config (Lessons 35–36's runs), with multiple results placed side by side to pick the best combination.
An item's input/expectedOutput can also serve as examples in a prompt (Lesson 37), letting the model answer in the shape of the reference answers.
An item's most refined design is versioning. The primary key says it all: @@id([id, projectId, validFrom])—the same question (same id) can have multiple rows, distinguished by validFrom. Each row also has a validTo: the row where it's null is the currently active version. On edit, the system does two steps in a transaction—stamp the old version with validTo (close its validity interval), then insert a new version row. This is the database's classic "slowly-changing dimension / bitemporal" technique. (The versioned implementation is gated by LANGFUSE_DATASET_SERVICE_WRITE_TO/READ_FROM_VERSIONED_IMPLEMENTATION; executeWithDatasetServiceStrategy switches between STATEFUL and VERSIONED accordingly.)
validTo IS NULL. An edit, in a transaction, "closes the old version (stamps validTo) + inserts a new version (new validFrom)". A dataset run pins a validFrom, so it always sees "that version of the question"—exactly why Lesson 30's eval uses datasetItemValidFrom. Source: dataset-items.ts:418-470.// in one transaction: close the old version, then insert the new (SCD Type 2) await prisma.$transaction(async (tx) => { // 0. find the current version (the row with validTo null) const current = await tx.datasetItem.findFirst({ where: { id: itemId, projectId, validTo: null }, orderBy: { validFrom: "desc" } }); const newValidFrom = new Date(Math.max(Date.now(), baseTs + 1)); // 1. stamp the old version with validTo — close its validity interval if (current) await tx.datasetItem.update({ where: {…current.validFrom}, data: { validTo: newValidFrom } }); // 2. insert the new version — validFrom = that moment, validTo defaults null (the new current) await tx.datasetItem.create({ data: { ...itemData, validFrom: newValidFrom } }); });
Why go to all this trouble instead of editing in place? Because experiments must be reproducible. You ran an experiment in March and got scores; in June you tweaked a few questions and ran again. If questions were "edited in place", March's scores would lose their context—you wouldn't know which version they tested, and the two experiments couldn't be compared fairly. Versioning pins every version of every question on a timeline, so "March's run used version t2, June's used t3" is crystal clear, and historical conclusions stay forever explainable.
| id (logical question) | validFrom | validTo | current? | content |
|---|---|---|---|---|
| item-42 | t1 | t2 | no (history) | prompt v1 |
| item-42 | t2 | t3 | no (history) | prompt v2 |
| item-42 | t3 | null | yes ✓ | prompt v3 (current) |
Three rows with the same id, different validFrom, distinguished as "current vs history" by whether validTo is null. To ask "what does this question look like now", WHERE validTo IS NULL; to ask "what did the t2 experiment use", find the row where validFrom ≤ t2 < validTo—one table answers both "now" and "any historical moment", the power of bitemporal.
validFrom/validTo to pin a time interval on every version of every question, letting each run pin the version it used (Lesson 30's eval replays exactly via datasetItemValidFrom). It's the classic "immutable history + a movable current pointer", the same belief as Lesson 37's prompt versions and git commits: once truth is recorded it's not erased; to change, append a new version.sourceTraceId closes the loop "find a problem → capture it as a regression question → verify after improving"—an observability platform not only lets you see problems but turns them into "vaccines" against recurrence.
input (prompt) + expectedOutput (reference answer) + metadata. It's the "exam paper" for evaluation, used with Lessons 28–32's scores.sourceTraceId/sourceObservationId let you capture a real trace (Lesson 13) into a test case in one click—the regression set hugs the real traffic distribution, not made-up cases.(id, projectId, validFrom) lets many versions of one question coexist; the current = validTo IS NULL. An edit, in a transaction, "closes the old (stamps validTo) + inserts the new"—slowly-changing dimension / bitemporal.validFrom, always seeing "that version" (Lesson 30's eval replays via datasetItemValidFrom). Historical experiments stay explainable and fairly comparable.