上一课,API 把一批事件收下并入队。这一课我们把镜头怼到单个事件上:它长什么样?为什么有 create 和 update 两副面孔?
最关键的是——你的 SDK 先后发来的好几个事件,最后怎么会变成数据库里同一行?这就是 Langfuse 摄取的灵魂:事件流 → 合并成记录。
看懂这一课,第 8 课那张 ReplacingMergeTree(event_ts, is_deleted) 的设计才算真正落地。
generation-create:开始时间、输入的主诉);
检查做完、开完药,医生补一条记录(generation-update:结束时间、诊断结论、用了多少药)。这两条记录都贴着同一个病人号(实体 id)——
护士归档时,不会建两份病历,而是把它们合并进同一份,冲突字段以最新一条为准。而每条记录本身还有自己的流水号(事件 id),
保证同一条记录归档两次也不会重复。「病人号合并、流水号去重」——正是这一课要拆解的两套 id。
每个摄取事件,无论它描述的是 trace、observation 还是 score,都套着同一个信封。信封里有一个公共的「外壳」,再加一个随类型而变的「内容物」body。
Langfuse 用 Zod 把它定义得一清二楚:先有一个 base(外壳),各类型在它上面 extend 出自己的 type 与 body,
最后用 判别联合(z.discriminatedUnion("type", […]))把十几种事件拢成一个 schema。校验时,Zod 看 type 这个字段,
就知道该用哪一套 body 规则去验——这就是「一个字段定身份,一套规则配一种身」。
base 持有 id(事件 id)、timestamp、metadata 和判别字段 type;type 决定 body 套用哪一套 Zod 规则。这正是 z.discriminatedUnion("type", […]) 的工作方式。源码:packages/shared/src/server/ingestion/types.ts:597。IngestionService 合并;值为示例):一次 LLM 调用开始时发一个 observation-create(知道 name/input),结束时发一个 observation-update(补上 output/usage/endTime)。两者 id 相同,按 id 合并成最终一行,后到的非空字段补齐——所以 SDK 不必在内存里攥着整条记录直到结束。const base = z.object({ id: idSchema, // 事件 id(流水号) timestamp: z.iso.datetime({ offset: true }), metadata: jsonSchema.nullish(), }); const generationCreateEvent = base.extend({ type: z.literal(eventTypes.GENERATION_CREATE), body: CreateGenerationBody, // 随 type 而变的内容物 }); // …span-create / span-update / score-create / 共 18 种… const ingestionEvent = z.discriminatedUnion("type", [ traceEvent, scoreEvent, spanCreateEvent, spanUpdateEvent, generationCreateEvent, generationUpdateEvent, /* … */ ]);
判别联合的妙处在于:它不是「把所有字段都设成可选、随便你填」,而是每种 type 配一套严格的 body 规则。
发 generation-create 就必须符合 CreateGenerationBody,发 score-create 就必须符合 ScoreBody——填错了类型、缺了字段,第 12 课那道 Zod 关当场就拦下。
而且这套 schema 还分公共版与内部版两个实例:像 dataset-run-item-create 这种就只允许内部使用,公共 API 发它会被拒。
那些 body 不是各写各的,而是层层继承出来的。最里层是 OptionalObservationBody(traceId、name、startTime、input/output…),
往外 extend 一层加 id 成 CreateEventEvent,再加 endTime 成 CreateSpanBody,再加 model / usage / modelParameters 成 CreateGenerationBody。
这条链精确对应第 3 课的观测三型:EVENT 是时间点(无时长)、SPAN 有起止(多了 endTime)、GENERATION 是带 LLM 字段的 SPAN(多了 model/usage)。
有趣的是,第 3 课提到的那些新观测类型——AGENT / TOOL / CHAIN / RETRIEVER / EVALUATOR / EMBEDDING / GUARDRAIL——它们的 body 全都复用 CreateGenerationBody:
本质上都是「generation 那么丰富的一段观测」,只是语义标签不同。
endTime 有了起止 → GENERATION 再加 model/usage 成为可记 token 与成本的 LLM 调用。AGENT/TOOL/CHAIN 等 7 种新类型直接复用 CreateGenerationBody。源码:types.ts:421-470。这套继承不仅省代码,更让类型语义层层递进:你不可能给一个 EVENT 填 endTime(它的 body 里压根没这字段),也不可能给一个 SPAN 填 usage。 schema 即文档——读完这条链,三型观测「能装什么、不能装什么」一目了然。
现在来回答开篇那个问题。一次 LLM 调用,SDK 通常发两个事件:开始时 generation-create(带 startTime、input),结束时 generation-update(带 endTime、output、usage)。
它们最终却落成 ClickHouse 里同一行 observation。秘密在于两套 id 各司其职:
<eventId>.json——同一事件重发,覆盖同一个文件,天然幂等,不会算两次。ReplacingMergeTree(event_ts) 在写入侧的镜像。把这条「事件流 → 一行」的路径拆成五步,合并的来龙去脉就完全清楚了——前两步在你的应用侧发生,后三步在 Langfuse 服务端发生:
LLM 调用一开始,SDK 发 generation-create,带上 body.id、startTime、input。发完即忘,不在内存里攥着。
调用结束,SDK 再发 generation-update,沿用同一个 body.id,补上 endTime、output、usage。
服务端把 dedupKey = 实体类型-body.id 相同的事件归到一组——这两条就被认成「同一条记录的两次上报」。
逐字段取有值的一方;create 与 update 都给了的字段,以 timestamp 更晚的为准。
合并结果写进 observations 表的同一行(id=body.id);ReplacingMergeTree 在后台再做一次同样的去重兜底(第 8 课)。
事件 type 五花八门,但 worker 写库时只关心一件事:这条该进哪张表。一个叫 getClickhouseEntityType 的小函数把 18 种 type
坍缩成 5 个实体类型(trace / observation / score / sdk_log / dataset_run_item);其中三个核心类型——trace / observation / score——
正好对应第 8 课的三张 ReplacingMergeTree 宽表,另两个(SDK 诊断日志 sdk_log、数据集运行项 dataset_run_item)走各自的处理:
| 实体类型 | 由哪些事件 type 映射而来 | 落到哪张表 |
|---|---|---|
| trace | trace-create | traces(第 8 课) |
| observation | event/span/generation-create·update + agent/tool/chain/retriever/evaluator/embedding/guardrail-create + 旧 observation-* | observations(第 8 课) |
| score | score-create | scores(第 8 课) |
| sdk_log | sdk-log | SDK 诊断日志,单独处理 |
| dataset_run_item | dataset-run-item-create | 数据集运行项,单独处理 |
三个核心类型里,所有「像观测」的 type——无论它叫 span、generation 还是 agent——统统归到 observation,进同一张表,靠 observation 行里的 type 列区分。
这就是第 8 课「宽事件」的威力:一张表容纳所有观测形态,新增一种观测类型不必加表、不必改 schema。
// 按 eventBodyId 把事件分组,使「同一实体」的多个事件能一起存、一起处理 const entityType = getClickhouseEntityType(event.type); // → trace / observation / score const dedupKey = `${entityType}-${event.body.id}`; // 合并键 = 实体类型 + 实体 id // 每个事件以「事件 id」为名落 S3:<eventId>.json(同事件重发→覆盖同文件,幂等) key: safeBlobFilenameStem(event.id, ".json"),
看清这段就懂了全局:实体 id(body.id)决定「合并到哪一行」,事件 id(envelope id)决定「在 S3 里叫什么名」。 前者保证 create/update 汇成一条;后者保证同一个事件重传一百次也只算一次。两套 id,两个目的,配合得严丝合缝。
base: id · timestamp · metadata + 判别字段 type)+ 随 type 而变的 body,由 z.discriminatedUnion("type") 拢成一个 schema。OptionalObservationBody → +id CreateEventEvent → +endTime CreateSpanBody → +model/usage CreateGenerationBody,对应 EVENT⊂SPAN⊂GENERATION。body.id)是合并键——同 id 的 create/update 汇成一行;事件 id(信封 id)是去重键——成为 S3 文件名,保证幂等。ReplacingMergeTree(event_ts) 在写入侧的镜像。getClickhouseEntityType 把 18 种事件 type 坍缩成 5 个实体类型;其中三个核心的 trace / observation / score → 第 8 课的三张宽表(另两个 sdk_log、dataset_run_item 各自处理)。
Last lesson the API accepted a batch of events and enqueued them. Now we zoom into a single event: what does it
look like? Why does it have two faces, create and update? And most crucially — how do the several events your
SDK sends end up as one row in the database? This is the soul of Langfuse ingestion: an event stream merges into a
record. Get this and Lesson 8's ReplacingMergeTree(event_ts, is_deleted) design finally clicks into place.
generation-create: start time, the input complaint). After tests and a prescription, the doctor adds a note
(generation-update: end time, diagnosis, dosage). Both notes carry the same patient number (entity id) — so when
filing, the nurse doesn't create two charts, she merges them into one, latest note winning on conflicts. Each note also has its
own serial number (event id), so filing the same note twice never duplicates it. "Merge by patient number, dedup by serial" —
exactly the two ids this lesson dissects.
Every ingestion event — whether it describes a trace, an observation or a score — rides in the same envelope: a shared "shell"
plus a type-dependent payload body. Langfuse pins this down with Zod: a base (the shell), each type extends it
with its own type and body, and finally a discriminated union (z.discriminatedUnion("type", […]))
gathers the dozen-plus events into one schema. To validate, Zod reads the type field and knows which body rules to apply —
"one field sets the identity, one ruleset per identity".
base holds id (event id), timestamp, metadata and the discriminator type; type decides which Zod ruleset the body follows. That's exactly how z.discriminatedUnion("type", […]) works. Source: packages/shared/src/server/ingestion/types.ts:597.IngestionService merge; values illustrative): an LLM call sends an observation-create at the start (name/input known) and an observation-update at the end (output/usage/endTime). They share an id, so they merge by id into one final row with later non-null fields filled in — the SDK needn't hold the whole record in memory until the end.const base = z.object({ id: idSchema, // event id (serial) timestamp: z.iso.datetime({ offset: true }), metadata: jsonSchema.nullish(), }); const generationCreateEvent = base.extend({ type: z.literal(eventTypes.GENERATION_CREATE), body: CreateGenerationBody, // payload varies by type }); // …span-create / span-update / score-create / 18 in total… const ingestionEvent = z.discriminatedUnion("type", [ traceEvent, scoreEvent, spanCreateEvent, spanUpdateEvent, generationCreateEvent, generationUpdateEvent, /* … */ ]);
The beauty of the discriminated union: it isn't "make every field optional, fill in whatever" — it's a strict body ruleset per
type. Send generation-create and you must match CreateGenerationBody; send score-create and you must
match ScoreBody — wrong type or missing field, and Lesson 12's Zod gate rejects it on the spot. The schema even ships as two
instances, public and internal: things like dataset-run-item-create are internal-only and rejected on the public API.
Those body schemas aren't written separately — they inherit layer by layer. The innermost is
OptionalObservationBody (traceId, name, startTime, input/output…); extend it with id to get
CreateEventEvent, add endTime for CreateSpanBody, then add model / usage / modelParameters for
CreateGenerationBody. This chain maps precisely onto Lesson 3's three observation kinds: EVENT is a point in time (no
duration), SPAN has start & end (adds endTime), GENERATION is a SPAN with LLM fields (adds model/usage). Tellingly,
the newer observation types from Lesson 3 — AGENT / TOOL / CHAIN / RETRIEVER / EVALUATOR / EMBEDDING / GUARDRAIL — all reuse
CreateGenerationBody: at heart they're each "a generation-rich span", differing only in semantic label.
endTime and gains start/end → GENERATION adds model/usage, becoming an LLM call that records tokens and cost. The 7 newer types (AGENT/TOOL/CHAIN…) reuse CreateGenerationBody directly. Source: types.ts:421-470.This inheritance doesn't just save code, it makes the type semantics escalate step by step: you can't put an endTime on an EVENT (its body simply has no such field), nor a usage on a SPAN. The schema is the documentation — read the chain and you see at a glance what each of the three observation kinds can and cannot carry.
Now to answer the opening question. For one LLM call, the SDK usually sends two events: generation-create at the start
(with startTime, input) and generation-update at the end (with endTime, output, usage). Yet they land as the same observation
row in ClickHouse. The secret is two ids, each with its own job:
<eventId>.json — resend the same event, overwrite the same file, naturally idempotent, never counted twice.ReplacingMergeTree(event_ts).Break this "event stream → one row" path into five steps and the whole merge story is clear — the first two happen on your app side, the last three on the Langfuse server:
As the LLM call begins, the SDK sends generation-create with body.id, startTime, input. Fire-and-forget, nothing held in memory.
When the call ends, the SDK sends generation-update, reusing the same body.id, adding endTime, output, usage.
The server groups events with the same dedupKey = entityType-body.id — these two are recognized as "two reports of one record".
Take the set side of each field; where both create and update gave a value, the later timestamp wins.
The merged result writes into one row of observations (id=body.id); ReplacingMergeTree re-dedups in the background as a safety net (Lesson 8).
Event type comes in many flavors, but when the worker writes, it cares about one thing: which table does this go to. A
small function getClickhouseEntityType collapses the 18 types into 5 entity types
(trace / observation / score / sdk_log / dataset_run_item); of these, the three core ones — trace / observation / score — map
exactly onto Lesson 8's three ReplacingMergeTree wide tables, while the other two (SDK diagnostic sdk_log, dataset-run linkage
dataset_run_item) take their own handling:
| entity type | mapped from which event types | which table |
|---|---|---|
| trace | trace-create | traces (Lesson 8) |
| observation | event/span/generation-create·update + agent/tool/chain/retriever/evaluator/embedding/guardrail-create + legacy observation-* | observations (Lesson 8) |
| score | score-create | scores (Lesson 8) |
| sdk_log | sdk-log | SDK diagnostics, handled separately |
| dataset_run_item | dataset-run-item-create | dataset run linkage, handled separately |
Within the three core types, everything "observation-like" — be it span, generation or agent — lands in observation, the same table, distinguished by the
type column on the observation row. That's the power of Lesson 8's "wide events": one table holds every observation shape,
and adding a new observation kind needs no new table, no schema change.
// Group events by eventBodyId so multiple events of "one entity" store & process together const entityType = getClickhouseEntityType(event.type); // → trace / observation / score const dedupKey = `${entityType}-${event.body.id}`; // merge key = entity type + entity id // each event lands in S3 named by its "event id": <eventId>.json (resend → overwrite, idempotent) key: safeBlobFilenameStem(event.id, ".json"),
Read this snippet and the whole picture lands: the entity id (body.id) decides "which row to merge into", the event id (envelope id) decides "what it's named in S3". The former lets create/update converge into one record; the latter ensures resending the same event a hundred times still counts once. Two ids, two purposes, fitting together seamlessly.
base: id · timestamp · metadata + discriminator type) + a type-varying body, gathered by z.discriminatedUnion("type") into one schema.OptionalObservationBody → +id CreateEventEvent → +endTime CreateSpanBody → +model/usage CreateGenerationBody, mapping EVENT⊂SPAN⊂GENERATION.body.id) is the merge key — same-id create/update converge into one row; event id (envelope id) is the dedup key — becomes the S3 filename, guaranteeing idempotency.ReplacingMergeTree(event_ts).getClickhouseEntityType collapses the 18 event types into 5 entity types; the three core ones trace / observation / score → Lesson 8's three wide tables (the other two, sdk_log & dataset_run_item, handled separately).