The last two lessons built the "exam paper" (the dataset) and the "exam mechanism" (the run). This lesson is about using them to make decisions—that's an experiment. One experiment takes a specific prompt + model config, runs it automatically over the whole dataset (producing one run), then places multiple runs side by side to pick the best. It turns "is changing this prompt or model actually better" from gut feeling into data—the closest thing to the "scientific method" in LLM engineering.
Three focuses: how an experiment runs server-side automatically (fill the question into the prompt's variables, call the LLM, produce a trace); how scoring auto-connects (reusing Lessons 30 and 35); and how comparison uses baseline + deltas so you see gains/losses at a glance. This lesson also sets up Part 7's prompt management—the prompt being tested in an experiment is the next lesson's protagonist.
📋 Analogy
An experiment is like a rigorous A/B controlled trial. The same questions (dataset): group A uses "gpt-4o + prompt v1", group B uses "claude + prompt v2", each sits the exam properly.
You needn't proctor yourself: the system automatically sets the questions (fills each question's input into the prompt's variable blanks), automatically grades (Lessons 29–32's scoring), automatically computes the grade (Lesson 35's aggregate-by-run). Finally it lays the report cards side by side, and lets you anoint one group as the baseline, so the rest show "how much above/below baseline".
So the question that vexes everyone—"is my prompt change actually better or worse?"—no longer rests on feeling, but on a side-by-side report card with deltas.
Run server-side automatically: fill the question into prompt variables, call the LLM per item
Langfuse experiments can run entirely server-side: you just pick a dataset, a prompt with variables, and a model—the worker does the rest. createExperimentJobClickhouse (driven by the ExperimentCreate queue) does three steps per question: fill the question's input into the prompt's variable blanks, call the LLM with your chosen provider/model, and write the result as a trace + run item.
experiment = run a prompt server-side per question: processLLMCall (delegated by processItem) calls replaceVariablesInPrompt(prompt, item.input, …) to fill variables, calls the LLM with config.provider/model, writes a trace (environment=PromptExperiments, linking prompt, metadata with itemVersion=item.validFrom). Source: worker/src/features/experiments/experimentServiceClickhouse.ts:154-225,287.
Upgrading "able to score" into "able to decide by score" (based on dataset-run-items scores; values illustrative): the same item is run twice — say a different prompt or model — with both outputs and their scores side by side and a Δ badge for the change. So "did this revision actually help" stops being a hunch and becomes evidence: B's helpfulness and exact are both higher, so point production to v2.
// 1) fill this question's input into the prompt's variable blanks → full messages
messages = replaceVariablesInPrompt(
config.validatedPrompt, datasetItem.input, config.allVariables, config.placeholderNames);
// 2) build the trace: tag experiment env, link the prompt under test, pin item version
traceSinkParams = {
environment: LangfuseInternalTraceEnvironment.PromptExperiments,
traceId, prompt: config.prompt,
experimentContext: { …, itemVersion: convert...(datasetItem.validFrom), itemExpectedOutput } };
// 3) call the LLM with the provider/model under test (result becomes this trace + run item)
modelParams = { provider: config.provider, model: config.model,
adapter: config.validatedApiKey.adapter, ...config.model_params };
dimension
server-side experiment (this lesson)
remote / SDK experiment
who runs the app
the Langfuse worker (your chosen prompt/model)
your own code / external system
how triggered
click "run" in the UI → ExperimentCreate queue
the dataset's remoteExperimentUrl calls back your service
both produce run + run item + trace, both use the same auto-scoring and comparison—downstream is identical
Both modes converge: whoever runs the app, the result lands as Lesson 35's run item + trace, enjoying the same auto-scoring (Lesson 30) and comparison views. "How the answer sheet is produced" can differ; "how it's judged and compared" is fully unified—another "one downstream, many upstreams" decoupling.
Comparison: anoint one as baseline, view the rest as deltas
Once the experiment runs, each trace is auto-scored and aggregated by run via Lessons 30 and 35, so every run has a "report card" (the average of each score name). The essence of comparison isn't absolute scores but side-by-side + deltas: you anoint one run as the baseline, and the rest show "how much above/below baseline". Langfuse's experiment comparison page (ExperimentComparisonSelector / ExperimentBaselineControls / ExperimentChartsGrid) does exactly this.
Comparison via "baseline + deltas": anoint one run as baseline, each score shows its relative delta (up/down). Source: web/src/features/experiments/components/'s ExperimentBaselineControls, ExperimentComparisonSelector, ExperimentChartsGrid.
pick a baseline
ExperimentBaselineControls lets you anoint one run as the reference—usually the "production-active" config, so all candidates compare to "the status quo", answering "would changing be better".
choose runs to compare
ExperimentComparisonSelector picks several runs side by side. Same questions, different configs, fairly comparable (Lesson 34's versioning guarantees they tested the same questions).
view multi-dimensional deltas
ExperimentChartsGrid charts each score dimension as a comparison. Goodness is often a multi-dimensional trade-off (more accurate but slower, costlier); the delta view makes the trade-off obvious.
The loop: a scientific "verify before you change" method
String Part 6's three lessons together and you get a continuous-improvement flywheel. It turns "optimizing an LLM app"—usually mystical—into a repeatable, data-backed engineering process:
1
build a dataset
Promote real botched production cases into test questions (L34), keeping the exam close to the real distribution.
2
run an experiment
For a candidate prompt/model config, the server runs a run over the whole set (this lesson), producing a batch of traces.
3
auto-score
Lessons 30, 35 automatically score each trace and aggregate this run's report card.
4
compare & decide
Side by side with the baseline (active config), view deltas: truly better → ship, regressed → reject—not by feeling.
5
back to step 1
After shipping you hit new hard cases, promote them into questions… the flywheel turns, the app grows steadier.
That's the whole value of Part 6: it upgrades Part 5's "can score quality" into "can make decisions with scores". And the core variable the experiment keeps testing—the prompt—is the protagonist of the next part (Part 7): how it's versioned, served at speed, and tuned interactively in the Playground. Evaluation tells you "how good now", experiments tell you "would changing be better", and prompt management gives you the confidence to change safely.
🎯 Design trade-off
Why does Langfuse run experiments server-side for you rather than just letting you run locally and upload results? Both are supported (the dataset's remoteExperimentUrl is for local/external runs), but server-side has unique value: zero-code, strongly consistent. A product manager or non-engineer can pick a dataset, tweak a prompt, and click "run" in the UI—no script needed; and server-side runs unify variable substitution, version pinning, eval scheduling, and environment tagging—baking "how to do an experiment right" into the platform, rather than hoping every person's local script is rigorous. The cost is the platform bearing LLM-call cost and concurrency, so it goes through the ExperimentCreate queue, reusing Lesson 30's scheduling resilience.
Why compare with "baseline + deltas" rather than absolute scores? Because absolute scores mean almost nothing; relative change is what matters. Is "helpfulness 0.78" good or bad? Without a reference you can't say. But "0.07 above the active config" is a fact you can act on. More importantly, quality is often a multi-dimensional trade-off: config B is more accurate but slower and costlier. Laying each dimension's delta side by side lets a decision-maker make the clear-eyed trade-off "is this much accuracy gain worth this much latency cost"—rather than being led by a single metric. This echoes Lesson 33's monitor "alert only on change": what carries information is always "the change", not "the absolute value".
🎯 Key points · Part 6 wrap-up
An experiment = prompt × dataset × model, run server-side: createExperimentJobClickhouse per question uses replaceVariablesInPrompt to fill variables, calls the chosen provider/model, produces a trace + run item—all zero-code.
Experiment traces carry rich context: environment tagged PromptExperiments, linking the prompt under test (L37), metadata pinning itemVersion=validFrom (L34/35 version replay)—reproducible and traceable.
Scoring auto-connects: reusing Lesson 30's dataset-run-item-upsert trigger and Lesson 35's aggregate-by-run, a report card appears once the run finishes, no manual work.
Comparison via baseline + deltas: ExperimentBaselineControls picks the baseline, ExperimentComparisonSelector chooses runs, ExperimentChartsGrid charts multi-dimensional deltas—absolute scores are hard to judge, relative change drives decisions.
Part 6 closes the loop: dataset → experiment → score → compare → decide → (new hard cases flow back to the dataset). It upgrades "can score" into "can decide with scores"; the next part's prompt gives you the confidence to "change safely".
🧪 Self-test - think about the design
1. What does Langfuse's server-side experiment (createExperimentJobClickhouse) do for each dataset question?
scores the item directly without calling an LLM
runs the model in the frontend via JavaScript
replaceVariablesInPrompt fills this question's input into the prompt's variable blanks, calls the LLM once with the chosen provider/model, writing a trace (tagged PromptExperiments env, linking the prompt under test, pinning the item version) + run item
just copies the item to a new table
Show answer & explanation click to expand
Answer: C. processLLMCall's three steps: ① replaceVariablesInPrompt fills the prompt's variables with item.input into full messages; ② build the trace, tagging env PromptExperiments, linking config.prompt, metadata pinning itemVersion=validFrom (L34/35 version replay); ③ call the LLM with config.provider/model. This is 'zero-code server-side experiments'—a PM can click run in the UI, with the platform unifying variable substitution/version pinning/eval scheduling.
2. The experiment comparison page shows results via 'baseline + deltas' rather than listing absolute scores. Why does this aid decisions?
absolute scores mean almost nothing ('helpfulness 0.78' good or bad?), relative change does ('0.07 above active' is actionable); and quality is often a multi-dimensional trade-off, with side-by-side deltas making 'is the accuracy gain worth the latency cost' obvious
absolute scores leak privacy
a baseline encrypts results
deltas use less CPU
Show answer & explanation click to expand
Answer: A. 'Helpfulness 0.78' can't be judged without a reference, but '0.07 above the active config' is an actionable fact. More importantly quality is often a multi-dimensional trade-off (B more accurate but slower/costlier); laying each dimension's delta side by side lets decision-makers trade off clearly rather than be led by one metric. This echoes Lesson 33's 'alert only on change'—information lives in change, not absolutes. ExperimentBaselineControls/ComparisonSelector/ChartsGrid implement this.
3. Stringing Part 6's three lessons together, the 'dataset→experiment→score→compare→decide' loop essentially upgrades Part 5's capability into what?
into real-time monitoring
into a faster database
into auto-writing prompts
from 'can score quality' into 'can make decisions with scores': verify on a fixed test set before changing a prompt/model, judging by data not feeling—the closest thing to the scientific method in LLM engineering
Show answer & explanation click to expand
Answer: D. Part 5 lets you 'see and score'; Part 6 lets you 'decide on that basis'. The dataset is a fixed exam, an experiment is a controlled trial, comparison is a report card with deltas—so 'is this change good' goes from gut feeling to data. The loop self-reinforces: post-ship hard cases get promoted into questions, the flywheel steadies the app. The repeatedly-tested prompt is the next part's protagonist.
💭 Open questions (no single right answer - just think or try)
Experiments turn 'verify before changing a prompt' into a scientific method. But it rests on premises: your dataset represents real traffic, and your scoring reflects real quality. Imagine one premise fails (e.g. a narrow dataset, or an LLM judge with systematic bias)—how would the comparison's conclusions mislead you? How would you guard against this 'the evaluation system itself is unreliable' risk? (Hint: revisit Lesson 32's human annotation as ground truth.)