第 14 课 worker 取走任务、照 fileKey 把事件从 S3 读了回来。这一课是整条写入链路的心脏:那一堆事件,外加数据库里这条记录的当前状态,
究竟怎么被揉成一条最终记录?答案藏在 IngestionService 里——mergeAndWrite 按实体类型分流,
mergeRecords 用一招「左折叠 + 后者覆盖」把多份输入合一,再盖上 event_ts=now 的版本戳。读懂它,第 8 课 ReplacingMergeTree 的「写入侧另一半」就补齐了。
IngestionService 干的事:底稿来自 ClickHouse,修订来自 S3,叠出来的合并稿写回库。
mergeAndWrite 本身只是个调度员。它拿到第 13 课算出的实体类型(trace / observation / score / dataset_run_item),
用一个 switch 把活儿分给对应的 process{Type}EventList。每条线路的骨架一样——读旧、转换、合并、写回——只是字段细节随类型而异。
mergeAndWrite 按实体类型把任务交给四条专用处理线,各自写回第 8 课的对应表。observation 那条最复杂(要算 token/成本、补包装 trace)。源码:worker/src/services/IngestionService/index.ts:149-195。IngestionService;值为示例):update 只带变更字段,合并器把它叠到当前行上得到结果行,再以更新的 event_ts 追加写入。ClickHouse 的 ReplacingMergeTree 查询时按 event_ts 留最新——于是「改」被实现成了「追加 + 查时取最新」。每条处理线,合并的输入其实有两个来源,缺一不可:
worker 用 listFiles 把这个实体在 S3 里的所有事件都读回来(create、update……一个不漏),再按 timestamp 排好序。这是「修订流」。
同时去 observations 表 getClickhouseRecord 把已存在的那一行读出来当底稿。它排在合并数组的最前面,作为 immutable 字段的基线。
为什么两个来源都要?看似冗余,其实各有不可替代的作用。S3 那份是事件的真账本(第 14 课),重读全部事件能保证合并出的记录完整无遗漏, 即便这次任务是个迟到的补传;ClickHouse 那份则提供 immutable 字段的权威基线(created_at、start_time 这些「第一次写定就锁死」的值), 还兜住那些可能已经从 S3 滚动清理掉的早期事件。两者叠在一起,既不丢新、又不改旧——这就是「读旧底稿 + 叠新修订」的精髓。
核心算法只有几行:把「ClickHouse 底稿 + 按时间排序的事件记录」拼成一个数组,从左往右一个个 overwriteObject 叠上去,最后盖一个 event_ts = now。
关键全在 overwriteObject 的三条「不覆盖」规则里。
undefined、空对象。所以 update 只带 endTime/output/usage,create 的 startTime/input/model 安然无恙。源码:IngestionService/index.ts:984-1005、utils.ts:56-81。// 左折叠:底稿在前,事件按时间在后,逐个叠加 let result = { id: records[0].id, project_id: records[0].project_id }; for (const record of records) result = overwriteObject(result, record, immutableEntityKeys); result.event_ts = new Date().getTime(); // 版本戳 → 交给第 8 课 ReplacingMergeTree // overwriteObject:用 lodash mergeWith,命中任一条就保留旧值 objValue if (nonOverwritableKeys.includes(key) // ① immutable || srcValue === undefined // ② 没带这个字段 || (isObject(srcValue) && isEmpty(srcValue))) // ③ 空 usage/cost return objValue; return srcValue; // 否则后者覆盖前者
哪些字段被锁成 immutable?看 observations 这张表:[id, project_id, trace_id, start_time, created_at, environment]。
它们是记录的身份与锚点——一旦第一条事件写定,后续 update 再怎么改都动不了;其余 payload 字段(output、usage、end_time、level、metadata……)才走 last-write-wins。
合并过程中还藏着一个体贴的向后兼容细节:如果一个 observation 合并完发现没有 trace_id(很老的 SDK,< 2.0 时代埋点不带 trace),
IngestionService 会顺手造一条「包装 trace」——用 observation 自己的 id 当 trace id,补写进 traces 表,再把它挂上去。这样无论数据多老,
每个 observation 都不会变成「孤儿」,第 5 课那棵 trace 树永远完整。一个小小的补偿逻辑,体现的是这套系统对历史数据与多版本 SDK 的尊重与照顾。
| 字段类别 | 例子 | 合并规则 |
|---|---|---|
| 身份 / 锚点(immutable) | id · project_id · trace_id · start_time · created_at · environment | first-wins:第一次写定,永不被覆盖 |
| 载荷(mutable) | end_time · output · usage · level · metadata · model… | last-write-wins:后到事件覆盖,但 undefined / 空对象不抹旧值 |
把最复杂的 processObservationEventList 拆开看,从一堆事件到写回库,正好七步:
先查 Redis:这个 fileKey 最近几分钟处理过吗?处理过就直接跳过(配合第 13 课「事件 id 当文件名」,重复投递不重复算)。
toTimeSortedEventList 把事件按 timestamp 排好;同一时刻 create 排在 update 前。
getClickhouseRecord 取出该 observation 已存在的行(带类型与 start_time 过滤),当合并基线。
把事件映射成记录,mergeRecords 左折叠出 mergedObservationRecord。
从最新往回找第一条非空的 input / output(事件里没有就退回 ClickHouse 的值),并归一化 tool 数据。
getGenerationUsage 做模型匹配、token 计数、成本计算(第 16 课详解),并进最终记录。
clickhouseWriter.addToQueue(Observations, finalRecord)——不直接写库,交给第 17 课的批写器攒批落盘。
第 7 步要特别留意:IngestionService 自己从不直接写 ClickHouse,它只把合并好的记录塞进 ClickhouseWriter 的队列。
为什么?因为逐条写 ClickHouse 是性能灾难(第 8 课说过它爱大批量)。把「合并」和「落盘」再切开,正是第 6 课「每层只干一件事」的又一次体现——合并归 IngestionService,攒批归 ClickhouseWriter。这种「先在内存里把记录算到位、再交给专门的批写器统一落盘」的两段式,让高频写入既保持正确,又不至于把列式存储压垮;第 17 课会看到批写器如何按表攒批、定时刷新。
event_ts=now 这个版本戳把两层咬合起来——预合并产出的新行永远有更大的 event_ts,于是它一定能在存储层的去重里胜出。这是典型的纵深防御。
mergeAndWrite 是调度员:按实体类型 switch 到 process{Trace,Observation,Score,DatasetRunItem}EventList,骨架都是「读旧→转换→合并→写回」。mergeRecords 把它们左折叠(overwriteObject 后者覆盖前者),并打 event_ts=now 版本戳。undefined、空对象——所以 update 只带变化字段、不抹掉 create 的值。addToQueue,不直接写库(落盘归第 17 课)。写入侧预合并求「即时正确」,存储侧 ReplacingMergeTree 求「最终唯一」,靠 event_ts 咬合——纵深防御。
Lesson 14's worker took the job and read the events back from S3 by fileKey. This lesson is the heart of the whole write path: how
does that pile of events, plus the record's current state in the database, get kneaded into one final record? The answer
lives in IngestionService — mergeAndWrite dispatches by entity type, mergeRecords folds many inputs into one with a
single "left-fold + later-overwrites" move, then stamps a event_ts=now version. Get it, and Lesson 8's ReplacingMergeTree gains its
"other (write-side) half".
IngestionService does:
the base from ClickHouse, the revisions from S3, the merged draft written back.
mergeAndWrite itself is just a dispatcher. It takes Lesson 13's computed entity type (trace / observation / score /
dataset_run_item) and a switch hands the work to the matching process{Type}EventList. Every line has the same skeleton —
read old, transform, merge, write back — only the field details differ by type.
mergeAndWrite hands the job by entity type to four dedicated lines, each writing back to Lesson 8's matching table. The observation line is the richest (token/cost, wrapper-trace backfill). Source: worker/src/services/IngestionService/index.ts:149-195.IngestionService; values illustrative): the update carries only changed fields; the merger overlays them onto the current row to get the result row, then appends it with an updated event_ts. ClickHouse's ReplacingMergeTree keeps the latest by event_ts at read time — so “update” is implemented as “append + keep-latest-at-query”.On each line, the merge actually has two sources, both indispensable:
The worker uses listFiles to read back all events of this entity in S3 (create, update… none missed), then sorts them by timestamp. This is the "revision stream".
Meanwhile it getClickhouseRecords the existing row from observations as the base draft. It goes first in the merge array, the baseline for immutable fields.
Why need both? Seemingly redundant, but each is irreplaceable. The S3 one is the events' real ledger (Lesson 14); re-reading all events guarantees the merged record is complete, even if this job is a late backfill. The ClickHouse one provides the authoritative baseline for immutable fields (created_at, start_time — those "locked once first written" values) and backstops early events that may have aged out of S3. Stacked together, they lose nothing new while changing nothing old — the essence of "read the old base + apply new revisions".
The core algorithm is just a few lines: assemble "ClickHouse base + time-sorted event records" into an array, fold them left-to-right with
overwriteObject, then stamp event_ts = now. The crux is entirely in overwriteObject's three "don't
overwrite" rules.
undefined values, empty objects. So update carrying only endTime/output/usage leaves create's startTime/input/model untouched. Source: IngestionService/index.ts:984-1005, utils.ts:56-81.// Left-fold: base first, events (time-sorted) after, stacked one by one let result = { id: records[0].id, project_id: records[0].project_id }; for (const record of records) result = overwriteObject(result, record, immutableEntityKeys); result.event_ts = new Date().getTime(); // version stamp → Lesson 8's ReplacingMergeTree // overwriteObject: lodash mergeWith; hit any rule → keep old objValue if (nonOverwritableKeys.includes(key) // ① immutable || srcValue === undefined // ② field not provided || (isObject(srcValue) && isEmpty(srcValue))) // ③ empty usage/cost return objValue; return srcValue; // else later overwrites earlier
Which fields are locked immutable? For the observations table: [id, project_id, trace_id, start_time, created_at, environment]. These are
the record's identity and anchors — set once by the first event, untouchable by any later update; everything else (output, usage, end_time,
level, metadata…) follows last-write-wins.
| field class | examples | merge rule |
|---|---|---|
| identity / anchor (immutable) | id · project_id · trace_id · start_time · created_at · environment | first-wins: set once, never overwritten |
| payload (mutable) | end_time · output · usage · level · metadata · model… | last-write-wins: later event overwrites, but undefined / empty object won't wipe old |
The merge also hides a thoughtful backward-compat detail: if a merged observation turns out to have no trace_id (a very
old SDK from the pre-2.0 era that instrumented without a trace), IngestionService quietly creates a "wrapper trace" — using
the observation's own id as the trace id, writing it into the traces table and attaching the observation to it. So no matter how old the data, no
observation becomes an orphan, and Lesson 5's trace tree stays whole. A tiny compensating bit of logic that respects historical
data and multi-version SDKs.
Unpack the most complex processObservationEventList and the path from a pile of events to a DB write is exactly seven steps:
Check Redis first: was this fileKey processed in the last few minutes? If so, skip outright (with Lesson 13's "event id as filename", redelivery never double-counts).
toTimeSortedEventList orders events by timestamp; at the same instant, create comes before update.
getClickhouseRecord fetches the observation's existing row (filtered by type and start_time) as the merge baseline.
Map events to records, mergeRecords left-folds into mergedObservationRecord.
Find the first non-null input/output scanning from newest back (fallback to ClickHouse's), and normalize tool data.
getGenerationUsage does model matching, token counting, cost calculation (detailed in Lesson 16), folded into the final record.
clickhouseWriter.addToQueue(Observations, finalRecord) — not a direct DB write, handed to Lesson 17's batch writer.
Note step 7: IngestionService never writes ClickHouse directly — it only pushes the merged record into
ClickhouseWriter's queue. Why? Because row-by-row ClickHouse writes are a performance disaster (Lesson 8 noted it loves big
batches). Splitting "merge" from "persist" is again Lesson 6's "each layer does one thing" — merging to IngestionService, batching to ClickhouseWriter.
This two-stage "compute the record fully in memory, then hand it to a dedicated batch writer" keeps high-frequency writes correct without crushing the
columnar store; Lesson 17 shows how the writer batches per table and flushes on a timer.
event_ts=now meshes the two — a pre-merged new row always has
a larger event_ts, so it always wins the storage layer's dedup. Classic defense in depth.
mergeAndWrite is a dispatcher: switch by entity type to process{Trace,Observation,Score,DatasetRunItem}EventList, all sharing "read old → transform → merge → write back".mergeRecords left-folds them (overwriteObject, later overwrites earlier) and stamps the event_ts=now version.undefined, empty objects — so update carries only changed fields without wiping create's values.addToQueues, not a direct write (persist is Lesson 17). Write-side pre-merge for "immediate correctness", storage-side ReplacingMergeTree for "eventual uniqueness", meshed by event_ts — defense in depth.