The first four parts let you see what happened on every LLM call. But "what happened" is not the same as "how good was it". This part is about evaluation—how to put a number on a trace. And the atomic unit of all of it is the score.
This lesson nails down the score's data model: what a score looks like, its three data types (numeric / categorical / boolean), what the score config ("the schema of a score") governs, and which three sources a score can come from.
Once you have this model, the next five lessons (LLM-as-a-judge, code eval, human annotation, monitors) are all just "different ways of producing a score".
📋 Analogy
Think of a score as an evaluation sticker slapped on one answer. The sticker says what you're rating (name, e.g. "helpfulness"), the rating (value), and which kind of scale (dataType: a 0–1 number? a "good/ok/bad" tier? a "right/wrong" tick?).
The score config is the printing standard for that kind of sticker: it mandates that "helpfulness" must be a 0–1 number, that "sentiment" must be one of {pos/neutral/neg}—so same-named stickers always use the same ruler, which is what makes them comparable and averageable.
As for who applied the sticker (source): maybe you did (API), maybe an AI judge did (EVAL), maybe a human reviewer did (ANNOTATION)—but as long as the scale matches, all three live in the same comparable space.
Three scales: numeric, categorical, boolean
The soul of a score is its dataType—it decides how the value is read and validated. Langfuse pins down three types with Zod (plus a legacy TEXT):
Three scales, each with a strength: NUMERIC (a number with optional min/max, averageable/sortable), CATEGORICAL (a fixed set of label↔value, both unique, countable/groupable), BOOLEAN (a special category locked to True=1 / False=0, pass-rate computable). Source: packages/shared/src/domain/score-configs.ts:18-80.
Three sources, one table (source enum from domain/scores.ts; values illustrative): a score has three sources — API (reported by code), EVAL (an LLM judge or code eval), ANNOTATION (human) — yet all land in one scores table sharing one shape (name / value or stringValue / dataType / source). So "machine-scored", "human-reviewed" and "API-reported" live in one space, comparable at the same name and scale.
// numeric: optional bounds, no categories
NumericConfigFields = { dataType: "NUMERIC", minValue?, maxValue? };
// categorical: a set of {label, value}; labels unique, values unique (validateCategories)
CategoricalConfigFields = { dataType: "CATEGORICAL", categories: [{label, value}, …] };
// boolean: really a "exactly two" special category, locked to True=1 / False=0
BooleanConfigFields = { dataType: "BOOLEAN",
categories: [{label:"True", value:1}, {label:"False", value:0}] }; // length===2 enforced
Note one elegant move: boolean is not a separate fourth type—it is implemented as a "exactly two categories, locked to True=1/False=0" category. So the core only handles two logics, "numeric" and "categorical", and boolean reuses categorical storage and aggregation—
one fewer special case, one more bit of consistency. This echoes Lesson 8's "wide event" idea: if a general mechanism can express it, don't invent a new one.
One more design that runs throughout: the value is always a number, while stringValue only holds the "label". A NUMERIC value is just that number with an empty stringValue; CATEGORICAL/BOOLEAN put the label in stringValue and the mapped number in value (e.g. "positive"→a number, "True"→1).
Precisely because every score has a numeric value, averaging, trending, and pass-rate work uniformly across all types—the physical basis of "comparable".
dataType
value / stringValue
constraints
aggregations
example
NUMERIC
value=the number; stringValue empty
optional minValue / maxValue
mean, distribution, trend
helpfulness 0.82
CATEGORICAL
stringValue=label; value=mapped number
labels unique, values unique
count, share, group-by
sentiment "positive"
BOOLEAN
stringValue="True"/"False"; value∈{0,1}
locked: True=1 / False=0
pass-rate
is_toxic False (0)
Reminder: scores.ts actually lists 5 dataTypes—the three above are the "comparable scales" a score config supports, plus TEXT (free text, length ≤500, legacy) and CORRECTION (for human-annotation corrections, name fixed to "output"). The latter two aren't aggregatable scales, so this lesson focuses on the first three. Source: scores.ts:46-52 (5 types), score-configs.ts:117-138 (config is a 4-way discriminatedUnion).
score config: a "schema for scoring"
A dataType alone isn't enough. If the same "helpfulness" score is rated 0–1 by one person today and 1–100 by another tomorrow, the data turns to mush. score config fixes exactly this: it is the schema of a score name—
declaring what dataType that name uses, and what bounds / which categories. So no matter where a score comes from, it is validated against the same ruler, which is the precondition for comparison and aggregation.
One config guards one name: a score config declares a name's dataType and constraints (numeric bounds / category set), validating scores from API / EVAL / ANNOTATION into one scores table, one ruler. Source: scores.ts:4 (ScoreSourceArray=["API","EVAL","ANNOTATION"]).
APIyou submit it
Use the SDK's score.create directly—e.g. your own offline eval script, or thumbs-up collected in production. This is one origin of the SCORE_CREATE event on the ingestion path of Lessons 6 & 12.
EVALAI or code judge
Scores computed by LLM-as-a-judge (Lesson 29) or code eval (Lesson 31). They too flow back through the same ingestion path to become scores—evaluation isn't a separate pipeline, it's a producer of scores.
ANNOTATIONhuman annotation
Human reviewers score traces against the config in an annotation queue (Lesson 32). They land in the same scores table, just with source set to ANNOTATION.
What fields a score object actually has
Unpack ScoreSchema and a score splits into three groups of fields—"what & how much", "who & with what", and "attached to what". The most worth remembering is the last: a score must attach to exactly one of trace / session / dataset-run; observationId is not a fourth peer but pins a step within a trace (it requires traceId).
what · how much
name which dimension (e.g. "helpfulness"); value always a number; stringValue the label (categorical/boolean/text only); dataType which scale.
who · with what
source API / EVAL / ANNOTATION; authorUserId the human annotator (nullable); comment a note (nullable); configId which config validates it; queueId which annotation queue it came from.
attached to (one of three)
one of traceId (whole trace), sessionId (whole session), datasetRunId (one dataset run); plus an optional observationId to pin a step within a trace (requires traceId).
This "one-of-three (plus in-trace step) attachment point" matters: it lets a score sit on the whole trace (was this answer good overall), on one observation within it via observationId (was this retrieval step accurate), or on a whole session or a dataset run (the experiment scenarios of Lessons 34–35). (Special case: CORRECTION scores may attach only to trace/observation, never session/datasetRun.) One score model covers every evaluation granularity. Validation rule: packages/shared/src/utils/scores.ts (applyScoreValidation); field defs in domain/scores.ts.
🎯 Design trade-off
Why bother with a dedicated score config instead of letting everyone score freely? Because "being comparable" is the entire value of evaluation. If under the name "helpfulness" team A rates 0–1, team B uses a percentage, and C uses "good/bad",
then those scores are meaningless together—averaging is wrong, trending is wrong, comparing is more wrong still. A score config fixes and enforces "a given name = a given scale", so same-named scores are forever comparable,
and you can confidently ask "did average helpfulness rise this week" or "which model has the higher toxicity pass-rate". This is the same engineering belief as Lesson 8's "provided vs computed" and Lesson 16's "pricing schema": model "what the data should look like" explicitly and validate up front,
and every downstream analysis stands on solid ground. Unifying three sources into one scores table and one config set lets "AI-judged", "human-reviewed", and "API-reported" scores live in one space—which is the shared destination of every remaining lesson in Part 5: producing comparable scores.
🎯 Key points
A score is evaluation's atomic unit: one (name, value, dataType, source) attached to a trace or one observation within it (or a session / dataset run), turning "what happened" into "how well".
Three comparable scales: NUMERIC (a number with optional min/max), CATEGORICAL (a fixed set of label↔value, both unique), BOOLEAN (a special category locked to True=1/False=0).
Boolean reuses categorical: not a separate fourth type but an "exactly two" category—one fewer special case, echoing Lesson 8's "don't invent when a general mechanism works".
score config = a name's schema: declares dataType and constraints (bounds/categories), enforces validation, so same-named scores always share one ruler—comparable and aggregatable.
Three sources: API (you submit), EVAL (AI/code judge), ANNOTATION (human)—all converge, all written to Lesson 8's scores table. The next five lessons are just "producing scores in different ways".
🧪 Self-test - think about the design
1. A Langfuse score config has three comparable scales: NUMERIC, CATEGORICAL, BOOLEAN. But in the source, BOOLEAN is not its own independent validation logic — what is it implemented as?
a special CATEGORICAL with 'exactly two categories, locked to True=1 / False=0' — the core only handles numeric and categorical, and boolean reuses categorical storage and aggregation
a separate boolean table
an alias of NUMERIC
a frontend-only validation
Show answer & explanation click to expand
Answer: A. BooleanConfigFields forces categories length 2, locked to {True:1, False:0} (score-configs.ts). So boolean needs no new storage/aggregation; it rides the categorical path — one fewer special case, more consistency. This echoes Lesson 8's 'if a general mechanism expresses it, don't invent a new one'.
2. For the score 'helpfulness', team A rates 0–1, team B uses a percentage, C uses 'good/bad'. What core problem does a score config solve?
auto-generates scores
comparability: a config is the schema of a name, fixing and enforcing 'this name = this scale', so same-named scores always use one ruler — only then are averaging, trending, and comparison meaningful
reduces database storage
makes scoring faster
Show answer & explanation click to expand
Answer: B. 'Being comparable' is the whole value of evaluation. If same-named scores use different scales, averaging/trending/comparing them is all wrong. A score config explicitly declares dataType and constraints (bounds/categories) and validates up front, keeping same-named scores forever comparable. Same belief as Lesson 8's provided/computed and Lesson 16's pricing schema: model 'what the data should look like' explicitly.
3. A score has three sources: API, EVAL, ANNOTATION (ScoreSourceArray in scores.ts). Which statement about them is most accurate?
each source has its own table and aggregation logic
EVAL scores bypass the ingestion path
all three converge: AI judging (EVAL), human annotation (ANNOTATION), and direct submission (API) are all validated by the same config and written to the same scores table — evaluation isn't a separate pipeline but a producer of scores
only the API source gets stored
Show answer & explanation click to expand
Answer: C. ScoreSourceArray=["API","EVAL","ANNOTATION"] merely labels a row in the same scores table with 'who scored this'. Scores from LLM-as-a-judge, code eval, and human annotation all flow back through the same ingestion path, validated by the same config. This is the shared destination of every remaining Part 5 lesson: producing comparable scores in different ways.
💭 Open questions (no single right answer - just think or try)
Langfuse unifies 'AI-judged, human-reviewed, API-reported' scores into one score type, one table, one config, distinguishing origin only by a source field. In a system you've built or imagined, if you had to unify 'quality signals from many sources' into one comparable metric, how would you design its data model? Would you want a 'schema gatekeeper' like a score config — and why?