🪢 Langfuse 图解教程Langfuse Visual Guide 第三部分 · 摄取链路Part 3 · The Ingestion Path 13 / 55
第三部分 · 摄取链路Part 3 · The Ingestion Path

事件类型与合并语义Event types & merge semantics

🧭 读前Before you start
先读 第12课 摄取 API 与 第3课 的 observation 类型,本课讲事件如何合并。
Read Lesson 12 (ingestion API) and Lesson 3 (observation types); this lesson covers how events merge.

上一课,API 把一批事件收下并入队。这一课我们把镜头怼到单个事件上:它长什么样?为什么有 create 和 update 两副面孔? 最关键的是——你的 SDK 先后发来的好几个事件,最后怎么会变成数据库里同一行?这就是 Langfuse 摄取的灵魂:事件流 → 合并成记录。 看懂这一课,第 8 课那张 ReplacingMergeTree(event_ts, is_deleted) 的设计才算真正落地。

🩺 生活类比
把一个观测想成医院里的一份病历。病人挂号那一刻,医生写下第一条就诊记录(generation-create:开始时间、输入的主诉); 检查做完、开完药,医生补一条记录(generation-update:结束时间、诊断结论、用了多少药)。这两条记录都贴着同一个病人号(实体 id)—— 护士归档时,不会建两份病历,而是把它们合并进同一份,冲突字段以最新一条为准。而每条记录本身还有自己的流水号(事件 id), 保证同一条记录归档两次也不会重复。「病人号合并、流水号去重」——正是这一课要拆解的两套 id。

事件的信封:id · timestamp · type · body

每个摄取事件,无论它描述的是 trace、observation 还是 score,都套着同一个信封。信封里有一个公共的「外壳」,再加一个随类型而变的「内容物」body。 Langfuse 用 Zod 把它定义得一清二楚:先有一个 base(外壳),各类型在它上面 extend 出自己的 type 与 body, 最后用 判别联合(z.discriminatedUnion("type", […]))把十几种事件拢成一个 schema。校验时,Zod 看 type 这个字段, 就知道该用哪一套 body 规则去验——这就是「一个字段定身份,一套规则配一种身」。

一个事件的信封:外壳固定,body 随 type 而变 base(公共外壳) id · 事件 id(流水号) timestamp · 事件发生时刻 metadata · 附带信息 type · 判别器 🔑决定下面 body 用哪套规则 按 type 路由 type=generation-create → body: CreateGenerationBodystartTime · model · usage · input/output… type=span-update → body: UpdateSpanBodyid · endTime · output…(补一条) type=score-create → body: ScoreBodyname · value · dataType… …共 18 种 type(trace / 14 个 observation 事件 / score / sdk-log / dataset-run-item)
信封 = 固定外壳 + 可变 body:base 持有 id(事件 id)、timestamp、metadata 和判别字段 type;type 决定 body 套用哪一套 Zod 规则。这正是 z.discriminatedUnion("type", […]) 的工作方式。源码:packages/shared/src/server/ingestion/types.ts:597。
示例:create + update 合并成一行 ① create(开始时) id: obs_7f type: GENERATION name: answer input: {messages} output: — usage: — 先发:知道什么算什么 + ② update(结束时) id: obs_7f ← 同一个 output: "Here is…" usageDetails: {512,88} endTime: …+1.8s 后发:补上结束才知的 = ③ 合并后的一行 id: obs_7f name: answer input: {messages} output: "Here is…" usageDetails: {512,88} endTime: …+1.8s 非空字段补齐
同 id 即合并(语义对齐 IngestionService 合并;值为示例):一次 LLM 调用开始时发一个 observation-create(知道 name/input),结束时发一个 observation-update(补上 output/usage/endTime)。两者 id 相同,按 id 合并成最终一行,后到的非空字段补齐——所以 SDK 不必在内存里攥着整条记录直到结束。
packages/shared/src/server/ingestion/types.tsL597
const base = z.object({
  id: idSchema,                        // 事件 id(流水号)
  timestamp: z.iso.datetime({ offset: true }),
  metadata: jsonSchema.nullish(),
});

const generationCreateEvent = base.extend({
  type: z.literal(eventTypes.GENERATION_CREATE),
  body: CreateGenerationBody,          // 随 type 而变的内容物
});
// …span-create / span-update / score-create / 共 18 种…

const ingestionEvent = z.discriminatedUnion("type", [
  traceEvent, scoreEvent, spanCreateEvent, spanUpdateEvent,
  generationCreateEvent, generationUpdateEvent, /* … */
]);

判别联合的妙处在于:它不是「把所有字段都设成可选、随便你填」,而是每种 type 配一套严格的 body 规则。 发 generation-create 就必须符合 CreateGenerationBody,发 score-create 就必须符合 ScoreBody——填错了类型、缺了字段,第 12 课那道 Zod 关当场就拦下。 而且这套 schema 还分公共版与内部版两个实例:像 dataset-run-item-create 这种就只允许内部使用,公共 API 发它会被拒。

body 的「俄罗斯套娃」:Event ⊂ Span ⊂ Generation

那些 body 不是各写各的,而是层层继承出来的。最里层是 OptionalObservationBody(traceId、name、startTime、input/output…), 往外 extend 一层加 id 成 CreateEventEvent,再加 endTime 成 CreateSpanBody,再加 model / usage / modelParameters 成 CreateGenerationBody。 这条链精确对应第 3 课的观测三型:EVENT 是时间点(无时长)、SPAN 有起止(多了 endTime)、GENERATION 是带 LLM 字段的 SPAN(多了 model/usage)。 有趣的是,第 3 课提到的那些新观测类型——AGENT / TOOL / CHAIN / RETRIEVER / EVALUATOR / EMBEDDING / GUARDRAIL——它们的 body 全都复用 CreateGenerationBody: 本质上都是「generation 那么丰富的一段观测」,只是语义标签不同。

body schema 的继承链(每往外一层,多几个字段) CreateGenerationBody(+ model · usage · modelParameters · completionStartTime) CreateSpanBody(+ endTime) CreateEventEvent(+ id) OptionalObservationBody traceId · name · startTime · input/output · level · parentObservationId… GENERATION SPAN EVENT
越往外,能力越强:EVENT 是无时长的时间点 → SPAN 加 endTime 有了起止 → GENERATION 再加 model/usage 成为可记 token 与成本的 LLM 调用。AGENT/TOOL/CHAIN 等 7 种新类型直接复用 CreateGenerationBody。源码:types.ts:421-470。

这套继承不仅省代码,更让类型语义层层递进:你不可能给一个 EVENT 填 endTime(它的 body 里压根没这字段),也不可能给一个 SPAN 填 usage。 schema 即文档——读完这条链,三型观测「能装什么、不能装什么」一目了然。

灵魂所在:好几个事件,怎么变成一行

现在来回答开篇那个问题。一次 LLM 调用,SDK 通常发两个事件:开始时 generation-create(带 startTime、input),结束时 generation-update(带 endTime、output、usage)。 它们最终却落成 ClickHouse 里同一行 observation。秘密在于两套 id 各司其职:

实体 idbody.id(合并键)
观测/trace 本身的 id。同一个 body.id 的所有事件,注定合并成一行。create 和 update 共享它,所以它们指向同一条记录。
事件 id信封 base.id(去重键)
每个事件自己的流水号。它在 S3 里成为文件名 <eventId>.json——同一事件重发,覆盖同一个文件,天然幂等,不会算两次。
版本timestamp(谁更新)
合并冲突字段时,以时间更晚的事件为准(last-write-wins)。这正是第 8 课 ReplacingMergeTree(event_ts) 在写入侧的镜像。
事件流 → 合并成一条记录(同 body.id,最新者胜) 事件① generation-create eventId=e1 · body.id=obs-42 · t=10:00:00 startTime=10:00 · input="..." model="gpt-4o" 事件② generation-update eventId=e2 · body.id=obs-42 · t=10:00:03 endTime=10:00:03 · output="..." usage={in:120,out:80} 按 body.id 分组合并 合并后的 observation 行 id=obs-42 · type=GENERATION startTime=10:00 · endTime=10:00:03 input + output(各取有值的一方) model=gpt-4o · usage={120,80} 冲突字段:t 更大的②胜
create + update → 一行:两个事件共享 body.id=obs-42,被分到一组合并;各字段取有值的一方,冲突时以 timestamp 更晚者为准。worker 端的合并逻辑(第 15 课)正是第 8 课 ReplacingMergeTree 行为的「写入侧预演」。

把这条「事件流 → 一行」的路径拆成五步,合并的来龙去脉就完全清楚了——前两步在你的应用侧发生,后三步在 Langfuse 服务端发生:

1

开始:发 create

LLM 调用一开始,SDK 发 generation-create,带上 body.id、startTime、input。发完即忘,不在内存里攥着。

2

结束:发 update

调用结束,SDK 再发 generation-update,沿用同一个 body.id,补上 endTime、output、usage。

3

按 body.id 分组

服务端把 dedupKey = 实体类型-body.id 相同的事件归到一组——这两条就被认成「同一条记录的两次上报」。

4

合并字段(最新者胜)

逐字段取有值的一方;create 与 update 都给了的字段,以 timestamp 更晚的为准。

5

落成一行

合并结果写进 observations 表的同一行(id=body.id);ReplacingMergeTree 在后台再做一次同样的去重兜底(第 8 课)。

18 种事件,5 类实体,3 张核心表

事件 type 五花八门,但 worker 写库时只关心一件事:这条该进哪张表。一个叫 getClickhouseEntityType 的小函数把 18 种 type 坍缩成 5 个实体类型(trace / observation / score / sdk_log / dataset_run_item);其中三个核心类型——trace / observation / score—— 正好对应第 8 课的三张 ReplacingMergeTree 宽表,另两个(SDK 诊断日志 sdk_log、数据集运行项 dataset_run_item)走各自的处理:

实体类型由哪些事件 type 映射而来落到哪张表
tracetrace-createtraces(第 8 课)
observationevent/span/generation-create·update + agent/tool/chain/retriever/evaluator/embedding/guardrail-create + 旧 observation-*observations(第 8 课)
scorescore-createscores(第 8 课)
sdk_logsdk-logSDK 诊断日志,单独处理
dataset_run_itemdataset-run-item-create数据集运行项,单独处理

三个核心类型里,所有「像观测」的 type——无论它叫 span、generation 还是 agent——统统归到 observation,进同一张表,靠 observation 行里的 type 列区分。 这就是第 8 课「宽事件」的威力:一张表容纳所有观测形态,新增一种观测类型不必加表、不必改 schema。

packages/shared/src/server/ingestion/processEventBatch.ts分组
// 按 eventBodyId 把事件分组,使「同一实体」的多个事件能一起存、一起处理
const entityType = getClickhouseEntityType(event.type);  // → trace / observation / score
const dedupKey = `${entityType}-${event.body.id}`;       // 合并键 = 实体类型 + 实体 id

// 每个事件以「事件 id」为名落 S3:<eventId>.json(同事件重发→覆盖同文件,幂等)
key: safeBlobFilenameStem(event.id, ".json"),

看清这段就懂了全局:实体 id(body.id)决定「合并到哪一行」,事件 id(envelope id)决定「在 S3 里叫什么名」。 前者保证 create/update 汇成一条;后者保证同一个事件重传一百次也只算一次。两套 id,两个目的,配合得严丝合缝。

🎯 设计取舍
为什么不直接做一个 upsert,非要拆成 create / update 两个事件? 因为SDK 端不该背状态。一次 LLM 调用,开始和结束之间可能隔着几秒、几十秒, 期间应用甚至可能崩、可能换机器。如果要求「等结束后一次性 upsert 全量」,SDK 就得在内存里攥着这条记录直到结束——既占内存又怕丢。 拆成 create + update,SDK 可以开始就发一半、结束再补一半,发完即忘;合并的脏活交给服务端按 body.id 去拼。把状态从客户端推给服务端, 正是分布式系统里「让边缘无状态、让中心兜底」的经典思路——也呼应第 5 课那条「快路 / 慢路」分工。
🎯 本课要点
  • 每个事件 = 信封(base: id · timestamp · metadata + 判别字段 type)+ 随 type 而变的 body,由 z.discriminatedUnion("type") 拢成一个 schema。
  • body 是继承链:OptionalObservationBody → +id CreateEventEvent → +endTime CreateSpanBody → +model/usage CreateGenerationBody,对应 EVENT⊂SPAN⊂GENERATION。
  • 两套 id:实体 id(body.id)是合并键——同 id 的 create/update 汇成一行;事件 id(信封 id)是去重键——成为 S3 文件名,保证幂等。
  • 冲突字段以 timestamp 更晚者为准(last-write-wins),是第 8 课 ReplacingMergeTree(event_ts) 在写入侧的镜像。
  • getClickhouseEntityType 把 18 种事件 type 坍缩成 5 个实体类型;其中三个核心的 trace / observation / score → 第 8 课的三张宽表(另两个 sdk_log、dataset_run_item 各自处理)。

🧪 自测 · 想一想为什么这么设计

1. 一次 LLM 调用,SDK 先发 generation-create、后发 generation-update,最后却落成 ClickHouse 里同一行。靠的是什么?
  1. 两个事件共享同一个实体 id(body.id),服务端按 body.id 分组合并成一行;冲突字段以 timestamp 更晚者为准(last-write-wins)
  2. 靠事件 id 相同来合并
  3. 靠 ClickHouse 随机挑一条保留
  4. 靠 SDK 在本地先合并好再发一条
看答案与解析 点击展开
答案:A。实体 id(body.id)是合并键:同 id 的 create/update 注定汇成一行;事件 id(信封 id)是去重键,成为 S3 文件名 .json 保证幂等。合并冲突时按 timestamp 最新者胜,正是第 8 课 ReplacingMergeTree(event_ts) 在写入侧的镜像。
2. 事件用 z.discriminatedUnion("type", […]) 定义,而不是“一个所有字段都可选的大 schema”。这样做的核心好处是?
  1. 每种 type 配一套严格的 body 规则:发 generation-create 必须符合 CreateGenerationBody,缺字段/填错类型当场被 Zod 拦下,校验既精确又能给出清晰错误
  2. 让事件体积更小
  3. 可以省掉 type 字段
  4. 让所有事件都能随便填字段
看答案与解析 点击展开
答案:A。判别联合用 type 这一个字段决定该套哪套 body 规则,是“一个字段定身份、一套规则配一种身”。比“全字段可选”精确得多——后者无法在入口拦下结构性错误,脏数据会一路流到 worker 才崩。
3. getClickhouseEntityType 把 18 种事件 type 坍缩成 5 个实体类型,其中 span / generation / agent / tool 等都归到哪个,意味着什么?
  1. 全归到 observation,进同一张 observations 表,靠行内 type 列区分;新增一种观测类型不必加表、不必改 schema——这正是第 8 课“宽事件”的威力
  2. 各自进一张独立的表
  3. 归到 score 表
  4. 归到 trace 表
看答案与解析 点击展开
答案:A。5 个实体类型里,三个核心的 trace / observation / score 正好对应第 8 课的三张 ReplacingMergeTree 宽表(另两个 sdk_log、dataset_run_item 各自处理)。所有“像观测”的形态共用 observations 一张表、用 type 列区分,所以 Langfuse 能不断新增观测类型(AGENT/TOOL/CHAIN…)而无需迁移表结构。
💭 发散思考(没有标准答案,动手或动脑想想)
  • create/update 拆成两个事件,把“攒着整条记录”的状态从客户端推给了服务端。你自己的系统里,有没有类似“让边缘无状态、让中心兜底”的地方?这样做的代价和收益各是什么?

Last lesson the API accepted a batch of events and enqueued them. Now we zoom into a single event: what does it look like? Why does it have two faces, create and update? And most crucially — how do the several events your SDK sends end up as one row in the database? This is the soul of Langfuse ingestion: an event stream merges into a record. Get this and Lesson 8's ReplacingMergeTree(event_ts, is_deleted) design finally clicks into place.

🩺 Analogy
Think of one observation as a hospital chart. When the patient checks in, the doctor writes the first visit note (generation-create: start time, the input complaint). After tests and a prescription, the doctor adds a note (generation-update: end time, diagnosis, dosage). Both notes carry the same patient number (entity id) — so when filing, the nurse doesn't create two charts, she merges them into one, latest note winning on conflicts. Each note also has its own serial number (event id), so filing the same note twice never duplicates it. "Merge by patient number, dedup by serial" — exactly the two ids this lesson dissects.

The envelope: id · timestamp · type · body

Every ingestion event — whether it describes a trace, an observation or a score — rides in the same envelope: a shared "shell" plus a type-dependent payload body. Langfuse pins this down with Zod: a base (the shell), each type extends it with its own type and body, and finally a discriminated union (z.discriminatedUnion("type", […])) gathers the dozen-plus events into one schema. To validate, Zod reads the type field and knows which body rules to apply — "one field sets the identity, one ruleset per identity".

An event's envelope: fixed shell, body varies by type base (shared shell) id · event id (serial) timestamp · when it happened metadata · side info type · discriminator 🔑picks the body ruleset below route by type type=generation-create → body: CreateGenerationBodystartTime · model · usage · input/output… type=span-update → body: UpdateSpanBodyid · endTime · output… (a patch) type=score-create → body: ScoreBodyname · value · dataType… …18 types total (trace / 14 observation events / score / sdk-log / dataset-run-item)
Envelope = fixed shell + variable body: base holds id (event id), timestamp, metadata and the discriminator type; type decides which Zod ruleset the body follows. That's exactly how z.discriminatedUnion("type", […]) works. Source: packages/shared/src/server/ingestion/types.ts:597.
Example: create + update merge into one row ① create (at start) id: obs_7f type: GENERATION name: answer input: {messages} output: — usage: — sent first: what is known so far + ② update (at end) id: obs_7f ← same one output: "Here is…" usageDetails: {512,88} endTime: …+1.8s sent later: fills end-time facts = ③ merged row id: obs_7f name: answer input: {messages} output: "Here is…" usageDetails: {512,88} endTime: …+1.8s non-null fields filled in
Same id merges (semantics per IngestionService merge; values illustrative): an LLM call sends an observation-create at the start (name/input known) and an observation-update at the end (output/usage/endTime). They share an id, so they merge by id into one final row with later non-null fields filled in — the SDK needn't hold the whole record in memory until the end.
packages/shared/src/server/ingestion/types.tsL597
const base = z.object({
  id: idSchema,                        // event id (serial)
  timestamp: z.iso.datetime({ offset: true }),
  metadata: jsonSchema.nullish(),
});

const generationCreateEvent = base.extend({
  type: z.literal(eventTypes.GENERATION_CREATE),
  body: CreateGenerationBody,          // payload varies by type
});
// …span-create / span-update / score-create / 18 in total…

const ingestionEvent = z.discriminatedUnion("type", [
  traceEvent, scoreEvent, spanCreateEvent, spanUpdateEvent,
  generationCreateEvent, generationUpdateEvent, /* … */
]);

The beauty of the discriminated union: it isn't "make every field optional, fill in whatever" — it's a strict body ruleset per type. Send generation-create and you must match CreateGenerationBody; send score-create and you must match ScoreBody — wrong type or missing field, and Lesson 12's Zod gate rejects it on the spot. The schema even ships as two instances, public and internal: things like dataset-run-item-create are internal-only and rejected on the public API.

The body "matryoshka": Event ⊂ Span ⊂ Generation

Those body schemas aren't written separately — they inherit layer by layer. The innermost is OptionalObservationBody (traceId, name, startTime, input/output…); extend it with id to get CreateEventEvent, add endTime for CreateSpanBody, then add model / usage / modelParameters for CreateGenerationBody. This chain maps precisely onto Lesson 3's three observation kinds: EVENT is a point in time (no duration), SPAN has start & end (adds endTime), GENERATION is a SPAN with LLM fields (adds model/usage). Tellingly, the newer observation types from Lesson 3 — AGENT / TOOL / CHAIN / RETRIEVER / EVALUATOR / EMBEDDING / GUARDRAIL — all reuse CreateGenerationBody: at heart they're each "a generation-rich span", differing only in semantic label.

The body schema inheritance chain (each outer layer adds fields) CreateGenerationBody (+ model · usage · modelParameters · completionStartTime) CreateSpanBody (+ endTime) CreateEventEvent (+ id) OptionalObservationBody traceId · name · startTime · input/output · level · parentObservationId… GENERATION SPAN EVENT
The further out, the more capable: EVENT is a duration-less point → SPAN adds endTime and gains start/end → GENERATION adds model/usage, becoming an LLM call that records tokens and cost. The 7 newer types (AGENT/TOOL/CHAIN…) reuse CreateGenerationBody directly. Source: types.ts:421-470.

This inheritance doesn't just save code, it makes the type semantics escalate step by step: you can't put an endTime on an EVENT (its body simply has no such field), nor a usage on a SPAN. The schema is the documentation — read the chain and you see at a glance what each of the three observation kinds can and cannot carry.

The heart of it: several events, one row

Now to answer the opening question. For one LLM call, the SDK usually sends two events: generation-create at the start (with startTime, input) and generation-update at the end (with endTime, output, usage). Yet they land as the same observation row in ClickHouse. The secret is two ids, each with its own job:

entity idbody.id (merge key)
The id of the observation/trace itself. All events with the same body.id are destined to merge into one row. create and update share it, so they point at the same record.
event idenvelope base.id (dedup key)
Each event's own serial. In S3 it becomes the filename <eventId>.json — resend the same event, overwrite the same file, naturally idempotent, never counted twice.
versiontimestamp (who wins)
When merging conflicting fields, the later-timestamped event wins (last-write-wins). This is the write-side mirror of Lesson 8's ReplacingMergeTree(event_ts).
Event stream → one record (same body.id, latest wins) event ① generation-create eventId=e1 · body.id=obs-42 · t=10:00:00 startTime=10:00 · input="..." model="gpt-4o" event ② generation-update eventId=e2 · body.id=obs-42 · t=10:00:03 endTime=10:00:03 · output="..." usage={in:120,out:80} group & merge by body.id merged observation row id=obs-42 · type=GENERATION startTime=10:00 · endTime=10:00:03 input + output (each takes the set side) model=gpt-4o · usage={120,80} conflicts: later-t ② wins
create + update → one row: the two events share body.id=obs-42, get grouped and merged; each field takes the side that has a value, conflicts resolved by later timestamp. The worker's merge logic (Lesson 15) is a "write-side rehearsal" of Lesson 8's ReplacingMergeTree behavior.

Break this "event stream → one row" path into five steps and the whole merge story is clear — the first two happen on your app side, the last three on the Langfuse server:

1

Start: send create

As the LLM call begins, the SDK sends generation-create with body.id, startTime, input. Fire-and-forget, nothing held in memory.

2

End: send update

When the call ends, the SDK sends generation-update, reusing the same body.id, adding endTime, output, usage.

3

Group by body.id

The server groups events with the same dedupKey = entityType-body.id — these two are recognized as "two reports of one record".

4

Merge fields (latest wins)

Take the set side of each field; where both create and update gave a value, the later timestamp wins.

5

Land as one row

The merged result writes into one row of observations (id=body.id); ReplacingMergeTree re-dedups in the background as a safety net (Lesson 8).

18 event types, 5 entities, 3 core tables

Event type comes in many flavors, but when the worker writes, it cares about one thing: which table does this go to. A small function getClickhouseEntityType collapses the 18 types into 5 entity types (trace / observation / score / sdk_log / dataset_run_item); of these, the three core ones — trace / observation / score — map exactly onto Lesson 8's three ReplacingMergeTree wide tables, while the other two (SDK diagnostic sdk_log, dataset-run linkage dataset_run_item) take their own handling:

entity typemapped from which event typeswhich table
tracetrace-createtraces (Lesson 8)
observationevent/span/generation-create·update + agent/tool/chain/retriever/evaluator/embedding/guardrail-create + legacy observation-*observations (Lesson 8)
scorescore-createscores (Lesson 8)
sdk_logsdk-logSDK diagnostics, handled separately
dataset_run_itemdataset-run-item-createdataset run linkage, handled separately

Within the three core types, everything "observation-like" — be it span, generation or agent — lands in observation, the same table, distinguished by the type column on the observation row. That's the power of Lesson 8's "wide events": one table holds every observation shape, and adding a new observation kind needs no new table, no schema change.

packages/shared/src/server/ingestion/processEventBatch.tsgrouping
// Group events by eventBodyId so multiple events of "one entity" store & process together
const entityType = getClickhouseEntityType(event.type);  // → trace / observation / score
const dedupKey = `${entityType}-${event.body.id}`;       // merge key = entity type + entity id

// each event lands in S3 named by its "event id": <eventId>.json (resend → overwrite, idempotent)
key: safeBlobFilenameStem(event.id, ".json"),

Read this snippet and the whole picture lands: the entity id (body.id) decides "which row to merge into", the event id (envelope id) decides "what it's named in S3". The former lets create/update converge into one record; the latter ensures resending the same event a hundred times still counts once. Two ids, two purposes, fitting together seamlessly.

🎯 Design trade-off
Why not just do an upsert — why split into two create / update events? Because the SDK shouldn't carry state. An LLM call can take seconds, even tens of seconds, between start and end; meanwhile the app might crash or move machines. Requiring "wait until the end, then upsert the whole thing" forces the SDK to hold that record in memory until completion — costing memory and risking loss. Splitting into create + update lets the SDK send half at the start, the rest at the end, forgetting each as it goes; the dirty work of merging by body.id falls to the server. Pushing state from the client to the center is the classic distributed-systems move — "keep the edge stateless, let the center backstop it" — echoing Lesson 5's fast-lane / slow-lane split.
🎯 Key points
  • Each event = an envelope (base: id · timestamp · metadata + discriminator type) + a type-varying body, gathered by z.discriminatedUnion("type") into one schema.
  • body is an inheritance chain: OptionalObservationBody → +id CreateEventEvent → +endTime CreateSpanBody → +model/usage CreateGenerationBody, mapping EVENT⊂SPAN⊂GENERATION.
  • Two ids: entity id (body.id) is the merge key — same-id create/update converge into one row; event id (envelope id) is the dedup key — becomes the S3 filename, guaranteeing idempotency.
  • Conflicting fields resolve by later timestamp (last-write-wins), the write-side mirror of Lesson 8's ReplacingMergeTree(event_ts).
  • getClickhouseEntityType collapses the 18 event types into 5 entity types; the three core ones trace / observation / score → Lesson 8's three wide tables (the other two, sdk_log & dataset_run_item, handled separately).

🧪 Self-test - think about the design

1. For one LLM call the SDK sends generation-create then generation-update, yet they land as the same row in ClickHouse. How?
  1. Both events share one entity id (body.id); the server groups by body.id and merges into one row, conflicts resolved by later timestamp (last-write-wins)
  2. they merge because their event ids are equal
  3. ClickHouse randomly keeps one of them
  4. the SDK merges locally and sends just one
Show answer & explanation click to expand
Answer: A. The entity id (body.id) is the merge key: same-id create/update converge into one row; the event id (envelope id) is the dedup key, becoming the S3 filename .json for idempotency. Conflicts resolve latest-timestamp-wins — the write-side mirror of L08's ReplacingMergeTree(event_ts).
2. Events are defined with z.discriminatedUnion("type", […]) rather than 'one big schema with all fields optional'. The core benefit?
  1. A strict body ruleset per type: generation-create must match CreateGenerationBody; missing/wrong fields are rejected by Zod on the spot — precise validation with clear errors
  2. makes events smaller on the wire
  3. lets you drop the type field
  4. lets every event fill any field freely
Show answer & explanation click to expand
Answer: A. The discriminated union uses the single type field to pick the body ruleset — 'one field sets identity, one ruleset per identity'. Far more precise than 'all-optional', which can't catch structural errors at the entry and lets dirty data crash the worker downstream.
3. getClickhouseEntityType collapses 18 event types into 5 entity types; where do span/generation/agent/tool land among them, and what does that imply?
  1. all into observation, the same observations table, distinguished by the row's type column; adding a new observation kind needs no new table or schema change — the power of L08's 'wide events'
  2. each into its own separate table
  3. into the score table
  4. into the trace table
Show answer & explanation click to expand
Answer: A. Among the 5 entity types, the three core ones trace / observation / score map onto L08's three ReplacingMergeTree wide tables (the other two, sdk_log & dataset_run_item, are handled separately). Every 'observation-like' shape shares the one observations table, distinguished by the type column, so Langfuse can keep adding observation kinds (AGENT/TOOL/CHAIN…) without schema migrations.
💭 Open questions (no single right answer - just think or try)
  • Splitting create/update into two events pushes the 'hold the whole record' state from client to server. In your own systems, is there a similar 'keep the edge stateless, let the center backstop' spot? What are its costs and benefits?