🪢 Langfuse 图解教程Langfuse Visual Guide 第六部分 · 数据集与实验Part 6 · Datasets & Experiments 36 / 55
第六部分 · 数据集与实验Part 6 · Datasets & Experiments

实验与对比Experiments & comparison

🧭 读前Before you start
先读 第35课 的数据集运行,本课讲多次运行如何对比成实验。
Read Lesson 35 on dataset runs; this lesson covers how multiple runs compare into experiments.

前两课造好了「考卷」(数据集)和「考试机制」(run)。这一课讲怎么用它们做决策——这就是实验(experiment)。一次实验,就是拿一个具体的 prompt + 模型配置,在整个数据集上自动跑一遍(产出一次 run),再把多次 run 并排对比,挑出最优的那套。它把「改 prompt、换模型到底好不好」从拍脑袋变成有数据——这是 LLM 工程里最接近「科学方法」的一环。 重点有三:实验是怎么服务端自动跑的(把题目填进 prompt 变量、调 LLM、产 trace);评分怎么自动接上(复用第 30、35 课);以及对比怎么用 baseline + 增量让你一眼看出涨跌。这一课也为 Part 7 的 prompt 管理埋下伏笔——实验里被考的那个 prompt,正是下一课的主角。

📋 生活类比
实验像一场严谨的 A/B 对照试验。同一套题(数据集),A 组用「gpt-4o + prompt v1」、B 组用「claude + prompt v2」,各自认真考一场。 你不用亲自监考:系统自动替你出题(把每道题的输入填进 prompt 的变量空格)、自动阅卷(第 29–32 课的评分)、自动算总评(第 35 课按 run 聚合)。最后把两组成绩单并排摊开,还允许你钦定一组当基准(baseline),其余组就显示「比基准高了多少 / 低了多少」。 于是那个让无数人纠结的问题——「我这个 prompt 改动,到底是变好了还是变坏了?」——不再靠感觉,而是一张并排的、带增量的成绩单说了算。

服务端自动跑:把题填进 prompt 变量,逐题调 LLM

Langfuse 的实验可以完全在服务端跑:你只需选一个数据集、一个带变量的 prompt、一个模型,剩下的交给 worker。createExperimentJobClickhouse(由 ExperimentCreate 队列驱动)会逐道题做三步:把这道题的 input 填进 prompt 的变量空格、用你选的 provider/model 调一次 LLM、把结果写成一条 trace + run item。

逐题:填变量 → 调 LLM → 产 trace + run item prompt(带变量)"回答 {{question}}"第37课的 prompt dataset item.input{question: "退款政策?"} replaceVariablesInPrompt填空 → 完整 messages"回答 退款政策?" 调 LLMprovider/model被考的配置 traceenv=PromptExperiments run item钉住 item 版本 trace 还链接了被考的 prompt(第37课)、metadata 钉住 item 的 validFrom(第34/35课的版本回放) 环境标成 PromptExperiments —— 这正是第30课「langfuse- 前缀防无限循环」要识别的内部环境之一
实验 = 服务端逐题跑 prompt:processLLMCall(由 processItem 委派)调 replaceVariablesInPrompt(prompt, item.input, …) 填变量,用 config.provider/model 调 LLM,写出 trace(environment=PromptExperiments,链接 prompt,metadata 含 itemVersion=item.validFrom)。源码:worker/src/features/experiments/experimentServiceClickhouse.ts:154-225,287。
示例:同一道题,两次 run 并排比 数据集项 · input:退款政策? · expected:30 天内 Run A · prompt v1 「我们支持退款。」 helpfulness 0.6 exact 0 较笼统 Run B · prompt v2 「30 天内可全额退款,逾期不退。」 helpfulness 0.9 exact 1 Δ +0.3 ▲ → 用分数做决策:B 更好,把 production 切到 v2(第 37 课)
把「能评分」升级成「能用分做决策」(基于 dataset-run-items 的 scores;值为示例):同一道题(数据集项)跑两次 run——比如换了 prompt 或 model——把两边 output 与各自 score 并排,一个 Δ 徽章告诉你升降。于是「改一版到底有没有更好」从凭感觉变成有分可依:B 的 helpfulness/exact 都更高,就把 production 切到 v2。
worker/src/features/experiments/experimentServiceClickhouse.tsprocessLLMCall 核心(processItem 委派)
// 1) 把这道题的 input 填进 prompt 的变量空格 → 完整 messages
messages = replaceVariablesInPrompt(
  config.validatedPrompt, datasetItem.input, config.allVariables, config.placeholderNames);

// 2) 组 trace:标实验环境、链接被考 prompt、钉住 item 版本
traceSinkParams = {
  environment: LangfuseInternalTraceEnvironment.PromptExperiments,
  traceId, prompt: config.prompt,
  experimentContext: { …, itemVersion: convert...(datasetItem.validFrom), itemExpectedOutput } };

// 3) 用被考的 provider/model 调 LLM(结果写成这条 trace + run item)
modelParams = { provider: config.provider, model: config.model,
  adapter: config.validatedApiKey.adapter, ...config.model_params };
维度服务端实验(本课主线)远程 / SDK 实验
谁跑应用Langfuse worker(用你选的 prompt/模型)你自己的代码 / 外部系统
怎么触发UI 点「跑」→ ExperimentCreate 队列数据集的 remoteExperimentUrl 回调你的服务
适合纯 prompt/模型对比,零代码复杂自定义链路(多步 agent、检索等)
共同点都产出 run + run item + trace,都走同一套自动评分与对比——下游完全一致

两种模式殊途同归:无论谁来跑应用,最终都落成第 35 课的 run item + trace,享受同一套自动评分(第 30 课)与对比视图。「怎么产生答卷」可以不同,「怎么评判与对比」完全统一——又一次「一个下游,多个上游」的解耦。

对比:选一组当基准,其余看增量

实验跑完,每条 trace 经第 30、35 课自动评分并按 run 聚合,于是每场 run 都有了一张「成绩单」(各 score 名的平均分)。对比的精髓不是看绝对分,而是并排 + 增量:你钦定一场 run 当 baseline(基准),其余各场就显示「比基准高/低多少」。Langfuse 的实验对比页(ExperimentComparisonSelector / ExperimentBaselineControls / ExperimentChartsGrid)正是干这个的。

并排成绩单 + 相对基准的增量 run A(baseline)gpt-4o + prompt v1有用性 0.78毒性通过 0.96延迟 1.2s★ 基准(其余跟它比) run Bclaude + prompt v2有用性 0.85 ▲+0.07毒性通过 0.97 ▲+0.01延迟 2.1s ▼+0.9更准但更慢 run Cgpt-4o + prompt v3有用性 0.74 ▼−0.04毒性通过 0.96 —延迟 1.1s ▲−0.1这版 prompt 反而退步 绝对分难判好坏,增量一目了然:B 用准度换了延迟、C 的 prompt 改动是负优化——决策有据可依
对比靠「基准 + 增量」:钦定一场 run 为 baseline,其余每个 score 显示相对增量(升/降)。源码:web/src/features/experiments/components/ 的 ExperimentBaselineControls、ExperimentComparisonSelector、ExperimentChartsGrid。

选 baseline

ExperimentBaselineControls 让你钦定一场 run 当参照系——通常是「线上现役」那套配置,于是所有候选都和「现状」比,回答「换了会更好吗」。

挑要比的 run

ExperimentComparisonSelector 选若干场 run 并排。同一套题、不同配置,公平可比(因为第 34 课的版本化保证了考的是同一批题)。

看多维增量

ExperimentChartsGrid 把每个 score 维度画成对比图。好坏往往是多维权衡(更准但更慢更贵),增量视图让取舍一目了然。

闭环:一套科学的「改前先验证」方法

把 Part 6 三课串起来,你得到的是一台持续改进的飞轮。它让「优化 LLM 应用」这件本来很玄的事,落地成一个可重复、有数据支撑的工程流程:

持续改进飞轮 把「能评分」升级成「能用分做决策」 ① 攒数据集 L34 提拔生产 case ② 跑实验 本课 · 服务端 run ③ 自动评分 L30/35 聚合成绩单 ④ 对比决策 baseline + 增量 ⑤ 遇新难 case 回流第 1 步
1

攒数据集

把真实生产里答砸的 case 提拔成测试题(第34课),让考卷紧贴真实分布。

2

跑实验

对一个候选 prompt/模型配置,服务端在整套题上跑一场 run(本课),产出一批 trace。

3

自动评分

第30、35课自动给每条 trace 评分、按 run 聚合出这场的成绩单。

4

对比决策

和 baseline(现役配置)并排看增量:真的更好就上线,退步就否决——不靠感觉。

5

回到第1步

上线后又会遇到新的难 case,再提拔成题……飞轮转起来,应用越改越稳。

这就是 Part 6 的全部价值:它把第 5 部分「能给质量评分」升级成「能用分数做决策」。而被实验反复考的那个核心变量——prompt——正是下一部分(Part 7)的主角:它怎么版本化、怎么被高速服务、怎么在 Playground 里交互调试。评估告诉你「现在多好」,实验告诉你「改了会不会更好」,而 prompt 管理则给了你「安全地改」的底气。

🎯 设计取舍
为什么 Langfuse 要在服务端帮你跑实验,而不是只让你在本地跑完把结果传上来? 两种都支持(数据集上的 remoteExperimentUrl 就是给本地/外部跑的),但服务端跑有独特价值:零代码、强一致。产品经理或非工程师也能在 UI 上选个数据集、改个 prompt、点「跑」,不必写一行脚本;而且服务端跑能统一变量替换、版本钉定、评分调度、环境标记——把「实验该怎么做对」固化进平台,而不是指望每个人的本地脚本都写得严谨。代价是平台要承担 LLM 调用的成本与并发,所以它走 ExperimentCreate 队列、复用第 30 课那套调度韧性。

为什么对比要用「baseline + 增量」,而不是直接看绝对分? 因为绝对分几乎没有意义,相对变化才有。「有用性 0.78」是好是坏?没有参照根本说不清。但「比现役配置高了 0.07」就是一个能据以决策的事实。更重要的是,质量往往是多维权衡:B 配置更准但更慢更贵。把每个维度的增量并排摊开,决策者才能做出「这点准度提升值不值这点延迟代价」的清醒取舍——而不是被单一指标牵着走。这也呼应第 33 课监控「只在变化时告警」:有信息量的永远是「变化」,不是「绝对值」。
🎯 本课要点 · 第六部分收官
  • 实验 = prompt × 数据集 × 模型,服务端自动跑:createExperimentJobClickhouse 逐题用 replaceVariablesInPrompt 填变量、调选定 provider/model、产 trace + run item,全程零代码。
  • 实验 trace 带丰富语境:环境标 PromptExperiments、链接被考 prompt(第37课)、metadata 钉住 itemVersion=validFrom(第34/35课的版本回放)——可复现、可追溯。
  • 评分自动接上:复用第 30 课 dataset-run-item-upsert 触发、第 35 课按 run 聚合,跑完即有成绩单,无需手动。
  • 对比靠 baseline + 增量:ExperimentBaselineControls 选基准、ExperimentComparisonSelector 挑 run、ExperimentChartsGrid 画多维增量——绝对分难判,相对变化才能决策。
  • Part 6 闭环:数据集→实验→评分→对比→决策→(新难 case 回流数据集)。把「能评分」升级成「能用分做决策」,下一部分的 prompt 则给你「安全地改」的底气。

🧪 自测 · 想一想为什么这么设计

1. Langfuse 的服务端实验(createExperimentJobClickhouse)对数据集的每道题做什么?
  1. 直接给数据项评分,不调 LLM
  2. 在前端用 JavaScript 跑模型
  3. replaceVariablesInPrompt 把这道题的 input 填进 prompt 的变量空格,用选定 provider/model 调一次 LLM,写成一条 trace(标 PromptExperiments 环境、链接被考 prompt、钉住 item 版本)+ run item
  4. 只把数据项复制到一张新表
看答案与解析 点击展开
答案:C。processLLMCall 三步:① replaceVariablesInPrompt 用 item.input 填 prompt 变量得完整 messages;② 组 trace,环境标 PromptExperiments、链接 config.prompt、metadata 钉 itemVersion=validFrom(第34/35课版本回放);③ 用 config.provider/model 调 LLM。这是「服务端零代码跑实验」——产品经理也能在 UI 点跑,平台统一变量替换/版本钉定/评分调度。
2. 实验对比页用「baseline + 增量」展示结果,而不是直接列绝对分。为什么这种设计更利于决策?
  1. 绝对分几乎没意义(「有用性 0.78」是好是坏说不清),相对变化才有(「比现役高 0.07」可据以决策);且质量常是多维权衡,并排增量让「准度提升值不值延迟代价」这种取舍一目了然
  2. 绝对分会泄露隐私
  3. baseline 能加密结果
  4. 增量计算更省 CPU
看答案与解析 点击展开
答案:A。「有用性 0.78」没有参照根本判不了好坏,但「比现役配置高 0.07」是能决策的事实。更重要的是质量常是多维权衡(B 更准但更慢更贵),把每维增量并排,决策者才能清醒取舍而非被单一指标牵着走。这呼应第33课监控「只在变化时告警」——有信息量的永远是变化,不是绝对值。ExperimentBaselineControls/ComparisonSelector/ChartsGrid 实现这套对比。
3. 把 Part 6 三课串起来,Langfuse 提供的「数据集→实验→评分→对比→决策」回路,本质上把第 5 部分的能力升级成了什么?
  1. 升级成实时监控
  2. 升级成更快的数据库
  3. 升级成自动写 prompt
  4. 从「能给质量评分」升级成「能用分数做决策」:改 prompt/换模型前先在固定测试集上验证,用数据而非感觉判断改动好坏——LLM 工程里最接近科学方法的一环
看答案与解析 点击展开
答案:D。第5部分让你「能看见、能评分」;Part 6 让你「能据此决策」。数据集是固定考卷、实验是一次对照试验、对比是带增量的成绩单——于是「这个改动到底好不好」从拍脑袋变成有数据。回路还自我强化:上线遇新难 case 再提拔成题,飞轮越转应用越稳。被反复考的 prompt 正是下一部分主角。
💭 发散思考(没有标准答案,动手或动脑想想)
  • 实验把「改 prompt 前先验证」变成了科学方法。但它依赖一个前提:你的数据集能代表真实流量、你的评分能反映真实质量。设想这两个前提其中之一不成立(比如数据集偏窄、或 LLM 裁判有系统性偏差),实验对比的结论会怎样误导你?你会怎么防范这种「评估体系本身不可靠」的风险?(提示:回顾第32课人工标注当 ground truth。)

The last two lessons built the "exam paper" (the dataset) and the "exam mechanism" (the run). This lesson is about using them to make decisions—that's an experiment. One experiment takes a specific prompt + model config, runs it automatically over the whole dataset (producing one run), then places multiple runs side by side to pick the best. It turns "is changing this prompt or model actually better" from gut feeling into data—the closest thing to the "scientific method" in LLM engineering. Three focuses: how an experiment runs server-side automatically (fill the question into the prompt's variables, call the LLM, produce a trace); how scoring auto-connects (reusing Lessons 30 and 35); and how comparison uses baseline + deltas so you see gains/losses at a glance. This lesson also sets up Part 7's prompt management—the prompt being tested in an experiment is the next lesson's protagonist.

📋 Analogy
An experiment is like a rigorous A/B controlled trial. The same questions (dataset): group A uses "gpt-4o + prompt v1", group B uses "claude + prompt v2", each sits the exam properly. You needn't proctor yourself: the system automatically sets the questions (fills each question's input into the prompt's variable blanks), automatically grades (Lessons 29–32's scoring), automatically computes the grade (Lesson 35's aggregate-by-run). Finally it lays the report cards side by side, and lets you anoint one group as the baseline, so the rest show "how much above/below baseline". So the question that vexes everyone—"is my prompt change actually better or worse?"—no longer rests on feeling, but on a side-by-side report card with deltas.

Run server-side automatically: fill the question into prompt variables, call the LLM per item

Langfuse experiments can run entirely server-side: you just pick a dataset, a prompt with variables, and a model—the worker does the rest. createExperimentJobClickhouse (driven by the ExperimentCreate queue) does three steps per question: fill the question's input into the prompt's variable blanks, call the LLM with your chosen provider/model, and write the result as a trace + run item.

per question: fill variables → call LLM → produce trace + run item prompt (with variables)"answer {{question}}"Lesson 37's prompt dataset item.input{question: "refund policy?"} replaceVariablesInPromptfill blanks → full messages"answer refund policy?" call LLMprovider/modelthe config under test traceenv=PromptExperiments run itempins item version the trace also links the prompt under test (L37); metadata pins the item's validFrom (L34/35 version replay) environment tagged PromptExperiments — one of the internal envs Lesson 30's "langfuse- prefix loop guard" recognizes
experiment = run a prompt server-side per question: processLLMCall (delegated by processItem) calls replaceVariablesInPrompt(prompt, item.input, …) to fill variables, calls the LLM with config.provider/model, writes a trace (environment=PromptExperiments, linking prompt, metadata with itemVersion=item.validFrom). Source: worker/src/features/experiments/experimentServiceClickhouse.ts:154-225,287.
Example: same item, two runs side by side dataset item · input: refund policy? · expected: within 30 days Run A · prompt v1 "We support refunds." helpfulness 0.6 exact 0 rather generic Run B · prompt v2 "Full refund within 30 days; none after." helpfulness 0.9 exact 1 Δ +0.3 ▲ → decide by score: B is better, point production to v2 (Lesson 37)
Upgrading "able to score" into "able to decide by score" (based on dataset-run-items scores; values illustrative): the same item is run twice — say a different prompt or model — with both outputs and their scores side by side and a Δ badge for the change. So "did this revision actually help" stops being a hunch and becomes evidence: B's helpfulness and exact are both higher, so point production to v2.
worker/src/features/experiments/experimentServiceClickhouse.tsprocessLLMCall core (processItem delegates)
// 1) fill this question's input into the prompt's variable blanks → full messages
messages = replaceVariablesInPrompt(
  config.validatedPrompt, datasetItem.input, config.allVariables, config.placeholderNames);

// 2) build the trace: tag experiment env, link the prompt under test, pin item version
traceSinkParams = {
  environment: LangfuseInternalTraceEnvironment.PromptExperiments,
  traceId, prompt: config.prompt,
  experimentContext: { …, itemVersion: convert...(datasetItem.validFrom), itemExpectedOutput } };

// 3) call the LLM with the provider/model under test (result becomes this trace + run item)
modelParams = { provider: config.provider, model: config.model,
  adapter: config.validatedApiKey.adapter, ...config.model_params };
dimensionserver-side experiment (this lesson)remote / SDK experiment
who runs the appthe Langfuse worker (your chosen prompt/model)your own code / external system
how triggeredclick "run" in the UI → ExperimentCreate queuethe dataset's remoteExperimentUrl calls back your service
suitspure prompt/model comparison, zero-codecomplex custom pipelines (multi-step agents, retrieval, etc.)
in commonboth produce run + run item + trace, both use the same auto-scoring and comparison—downstream is identical

Both modes converge: whoever runs the app, the result lands as Lesson 35's run item + trace, enjoying the same auto-scoring (Lesson 30) and comparison views. "How the answer sheet is produced" can differ; "how it's judged and compared" is fully unified—another "one downstream, many upstreams" decoupling.

Comparison: anoint one as baseline, view the rest as deltas

Once the experiment runs, each trace is auto-scored and aggregated by run via Lessons 30 and 35, so every run has a "report card" (the average of each score name). The essence of comparison isn't absolute scores but side-by-side + deltas: you anoint one run as the baseline, and the rest show "how much above/below baseline". Langfuse's experiment comparison page (ExperimentComparisonSelector / ExperimentBaselineControls / ExperimentChartsGrid) does exactly this.

side-by-side report cards + deltas vs baseline run A (baseline)gpt-4o + prompt v1helpfulness 0.78toxicity-pass 0.96latency 1.2s★ baseline (others compared to it) run Bclaude + prompt v2helpfulness 0.85 ▲+0.07tox-pass 0.97 ▲+0.01latency 2.1s ▼+0.9more accurate but slower run Cgpt-4o + prompt v3helpfulness 0.74 ▼−0.04tox-pass 0.96 —latency 1.1s ▲−0.1this prompt regressed absolute scores are hard to judge; deltas are obvious: B traded latency for accuracy, C's prompt change is a regression—decisions grounded
Comparison via "baseline + deltas": anoint one run as baseline, each score shows its relative delta (up/down). Source: web/src/features/experiments/components/'s ExperimentBaselineControls, ExperimentComparisonSelector, ExperimentChartsGrid.

pick a baseline

ExperimentBaselineControls lets you anoint one run as the reference—usually the "production-active" config, so all candidates compare to "the status quo", answering "would changing be better".

choose runs to compare

ExperimentComparisonSelector picks several runs side by side. Same questions, different configs, fairly comparable (Lesson 34's versioning guarantees they tested the same questions).

view multi-dimensional deltas

ExperimentChartsGrid charts each score dimension as a comparison. Goodness is often a multi-dimensional trade-off (more accurate but slower, costlier); the delta view makes the trade-off obvious.

The loop: a scientific "verify before you change" method

String Part 6's three lessons together and you get a continuous-improvement flywheel. It turns "optimizing an LLM app"—usually mystical—into a repeatable, data-backed engineering process:

continuous-improvement flywheel upgrades "can score" into "can decide with scores" 1 build dataset L34 promote prod cases 2 run experiment this lesson · server run 3 auto-score L30/35 scorecard 4 compare + decide baseline + deltas 5 new hard cases feed back to step 1
1

build a dataset

Promote real botched production cases into test questions (L34), keeping the exam close to the real distribution.

2

run an experiment

For a candidate prompt/model config, the server runs a run over the whole set (this lesson), producing a batch of traces.

3

auto-score

Lessons 30, 35 automatically score each trace and aggregate this run's report card.

4

compare & decide

Side by side with the baseline (active config), view deltas: truly better → ship, regressed → reject—not by feeling.

5

back to step 1

After shipping you hit new hard cases, promote them into questions… the flywheel turns, the app grows steadier.

That's the whole value of Part 6: it upgrades Part 5's "can score quality" into "can make decisions with scores". And the core variable the experiment keeps testing—the prompt—is the protagonist of the next part (Part 7): how it's versioned, served at speed, and tuned interactively in the Playground. Evaluation tells you "how good now", experiments tell you "would changing be better", and prompt management gives you the confidence to change safely.

🎯 Design trade-off
Why does Langfuse run experiments server-side for you rather than just letting you run locally and upload results? Both are supported (the dataset's remoteExperimentUrl is for local/external runs), but server-side has unique value: zero-code, strongly consistent. A product manager or non-engineer can pick a dataset, tweak a prompt, and click "run" in the UI—no script needed; and server-side runs unify variable substitution, version pinning, eval scheduling, and environment tagging—baking "how to do an experiment right" into the platform, rather than hoping every person's local script is rigorous. The cost is the platform bearing LLM-call cost and concurrency, so it goes through the ExperimentCreate queue, reusing Lesson 30's scheduling resilience.

Why compare with "baseline + deltas" rather than absolute scores? Because absolute scores mean almost nothing; relative change is what matters. Is "helpfulness 0.78" good or bad? Without a reference you can't say. But "0.07 above the active config" is a fact you can act on. More importantly, quality is often a multi-dimensional trade-off: config B is more accurate but slower and costlier. Laying each dimension's delta side by side lets a decision-maker make the clear-eyed trade-off "is this much accuracy gain worth this much latency cost"—rather than being led by a single metric. This echoes Lesson 33's monitor "alert only on change": what carries information is always "the change", not "the absolute value".
🎯 Key points · Part 6 wrap-up
  • An experiment = prompt × dataset × model, run server-side: createExperimentJobClickhouse per question uses replaceVariablesInPrompt to fill variables, calls the chosen provider/model, produces a trace + run item—all zero-code.
  • Experiment traces carry rich context: environment tagged PromptExperiments, linking the prompt under test (L37), metadata pinning itemVersion=validFrom (L34/35 version replay)—reproducible and traceable.
  • Scoring auto-connects: reusing Lesson 30's dataset-run-item-upsert trigger and Lesson 35's aggregate-by-run, a report card appears once the run finishes, no manual work.
  • Comparison via baseline + deltas: ExperimentBaselineControls picks the baseline, ExperimentComparisonSelector chooses runs, ExperimentChartsGrid charts multi-dimensional deltas—absolute scores are hard to judge, relative change drives decisions.
  • Part 6 closes the loop: dataset → experiment → score → compare → decide → (new hard cases flow back to the dataset). It upgrades "can score" into "can decide with scores"; the next part's prompt gives you the confidence to "change safely".

🧪 Self-test - think about the design

1. What does Langfuse's server-side experiment (createExperimentJobClickhouse) do for each dataset question?
  1. scores the item directly without calling an LLM
  2. runs the model in the frontend via JavaScript
  3. replaceVariablesInPrompt fills this question's input into the prompt's variable blanks, calls the LLM once with the chosen provider/model, writing a trace (tagged PromptExperiments env, linking the prompt under test, pinning the item version) + run item
  4. just copies the item to a new table
Show answer & explanation click to expand
Answer: C. processLLMCall's three steps: ① replaceVariablesInPrompt fills the prompt's variables with item.input into full messages; ② build the trace, tagging env PromptExperiments, linking config.prompt, metadata pinning itemVersion=validFrom (L34/35 version replay); ③ call the LLM with config.provider/model. This is 'zero-code server-side experiments'—a PM can click run in the UI, with the platform unifying variable substitution/version pinning/eval scheduling.
2. The experiment comparison page shows results via 'baseline + deltas' rather than listing absolute scores. Why does this aid decisions?
  1. absolute scores mean almost nothing ('helpfulness 0.78' good or bad?), relative change does ('0.07 above active' is actionable); and quality is often a multi-dimensional trade-off, with side-by-side deltas making 'is the accuracy gain worth the latency cost' obvious
  2. absolute scores leak privacy
  3. a baseline encrypts results
  4. deltas use less CPU
Show answer & explanation click to expand
Answer: A. 'Helpfulness 0.78' can't be judged without a reference, but '0.07 above the active config' is an actionable fact. More importantly quality is often a multi-dimensional trade-off (B more accurate but slower/costlier); laying each dimension's delta side by side lets decision-makers trade off clearly rather than be led by one metric. This echoes Lesson 33's 'alert only on change'—information lives in change, not absolutes. ExperimentBaselineControls/ComparisonSelector/ChartsGrid implement this.
3. Stringing Part 6's three lessons together, the 'dataset→experiment→score→compare→decide' loop essentially upgrades Part 5's capability into what?
  1. into real-time monitoring
  2. into a faster database
  3. into auto-writing prompts
  4. from 'can score quality' into 'can make decisions with scores': verify on a fixed test set before changing a prompt/model, judging by data not feeling—the closest thing to the scientific method in LLM engineering
Show answer & explanation click to expand
Answer: D. Part 5 lets you 'see and score'; Part 6 lets you 'decide on that basis'. The dataset is a fixed exam, an experiment is a controlled trial, comparison is a report card with deltas—so 'is this change good' goes from gut feeling to data. The loop self-reinforces: post-ship hard cases get promoted into questions, the flywheel steadies the app. The repeatedly-tested prompt is the next part's protagonist.
💭 Open questions (no single right answer - just think or try)
  • Experiments turn 'verify before changing a prompt' into a scientific method. But it rests on premises: your dataset represents real traffic, and your scoring reflects real quality. Imagine one premise fails (e.g. a narrow dataset, or an LLM judge with systematic bias)—how would the comparison's conclusions mislead you? How would you guard against this 'the evaluation system itself is unreliable' risk? (Hint: revisit Lesson 32's human annotation as ground truth.)