🪢 Langfuse 图解教程Langfuse Visual Guide 第三部分 · 摄取链路Part 3 · The Ingestion Path 15 / 55
第三部分 · 摄取链路Part 3 · The Ingestion Path

IngestionService:合并的心脏IngestionService: the merge heart

🧭 读前Before you start
先读 第13课 的合并语义与 第14课 的队列,本课进入合并的心脏 IngestionService。
Read Lesson 13 (merge semantics) and Lesson 14 (the queue); this lesson enters the merge heart, IngestionService.

第 14 课 worker 取走任务、照 fileKey 把事件从 S3 读了回来。这一课是整条写入链路的心脏:那一堆事件,外加数据库里这条记录的当前状态, 究竟怎么被揉成一条最终记录?答案藏在 IngestionService 里——mergeAndWrite 按实体类型分流, mergeRecords 用一招「左折叠 + 后者覆盖」把多份输入合一,再盖上 event_ts=now 的版本戳。读懂它,第 8 课 ReplacingMergeTree 的「写入侧另一半」就补齐了。

📝 生活类比
把合并想成协作编辑一篇维基词条。数据库里当前发布的版本是底稿;这次收到的每个事件,就像一条按时间排好的修订, 依次盖在底稿上。后改的覆盖先改的(同一字段,新值赢);但某条修订没碰的字段,保持原样(你只改了「结论」,「正文」不会被清空)。 还有几样东西被锁死,谁都不能改:词条的创建时间、唯一 id。所有修订叠完,得到一份合并稿,标上「本次合并时刻」当版本号—— 这正是 IngestionService 干的事:底稿来自 ClickHouse,修订来自 S3,叠出来的合并稿写回库。

mergeAndWrite:先按实体类型分流

mergeAndWrite 本身只是个调度员。它拿到第 13 课算出的实体类型(trace / observation / score / dataset_run_item), 用一个 switch 把活儿分给对应的 process{Type}EventList。每条线路的骨架一样——读旧、转换、合并、写回——只是字段细节随类型而异。

mergeAndWrite:一个 switch,四条处理线 mergeAndWriteswitch (entityType) processTraceEventList→ traces 表 processObservationEventList→ observations 表(最复杂) processScoreEventList→ scores 表 processDatasetRunItemEventList仅 create,不合并
分流而非合流:mergeAndWrite 按实体类型把任务交给四条专用处理线,各自写回第 8 课的对应表。observation 那条最复杂(要算 token/成本、补包装 trace)。源码:worker/src/services/IngestionService/index.ts:149-195。
示例:合并 = 旧行 + 补丁 → 新行 当前存储行 id: obs_7f output: null total_cost: null event_ts: 12:00:00.1 incoming update(补丁) id: obs_7f output: "Here is…" total_cost: 0.0041 只带变更的字段 结果行(写回 CH) id: obs_7f output: "Here is…" total_cost: 0.0041 event_ts: 12:00:01.9 event_ts 更新 → 留这条
合并是「叠补丁」,不是「改原行」(语义对齐 IngestionService;值为示例):update 只带变更字段,合并器把它叠到当前行上得到结果行,再以更新的 event_ts 追加写入。ClickHouse 的 ReplacingMergeTree 查询时按 event_ts 留最新——于是「改」被实现成了「追加 + 查时取最新」。

每条处理线,合并的输入其实有两个来源,缺一不可:

📜 S3:本实体的全部事件

worker 用 listFiles 把这个实体在 S3 里的所有事件都读回来(create、update……一个不漏),再按 timestamp 排好序。这是「修订流」。

🗄️ ClickHouse:当前这条记录

同时去 observations 表 getClickhouseRecord 把已存在的那一行读出来当底稿。它排在合并数组的最前面,作为 immutable 字段的基线。

为什么两个来源都要?看似冗余,其实各有不可替代的作用。S3 那份是事件的真账本(第 14 课),重读全部事件能保证合并出的记录完整无遗漏, 即便这次任务是个迟到的补传;ClickHouse 那份则提供 immutable 字段的权威基线(created_at、start_time 这些「第一次写定就锁死」的值), 还兜住那些可能已经从 S3 滚动清理掉的早期事件。两者叠在一起,既不丢新、又不改旧——这就是「读旧底稿 + 叠新修订」的精髓。

mergeRecords:左折叠,后者覆盖

核心算法只有几行:把「ClickHouse 底稿 + 按时间排序的事件记录」拼成一个数组,从左往右一个个 overwriteObject 叠上去,最后盖一个 event_ts = now。 关键全在 overwriteObject 的三条「不覆盖」规则里。

左折叠:底稿 → 叠事件 → 最终记录 CH 底稿已存在的行(可能为空) 事件① createstartTime·inputmodel 事件② updateendTime·outputusage 最终记录各字段最新有值者event_ts = now overwriteObject ×N overwriteObject 的三条「不覆盖」规则(后者的值被忽略,保留前者): ① immutable 键id·start_time·created_at… ② 值为 undefined没带的字段不抹旧值 ③ 空对象 {}空 usage/cost 不覆盖
叠的是「有值的新字段」:后到的事件覆盖先到的,但三种情况例外——immutable 键、值为 undefined、空对象。所以 update 只带 endTime/output/usage,create 的 startTime/input/model 安然无恙。源码:IngestionService/index.ts:984-1005、utils.ts:56-81。
worker/src/services/IngestionService/index.ts & utils.ts合并核心
// 左折叠:底稿在前,事件按时间在后,逐个叠加
let result = { id: records[0].id, project_id: records[0].project_id };
for (const record of records)
  result = overwriteObject(result, record, immutableEntityKeys);
result.event_ts = new Date().getTime();   // 版本戳 → 交给第 8 课 ReplacingMergeTree

// overwriteObject:用 lodash mergeWith,命中任一条就保留旧值 objValue
if (nonOverwritableKeys.includes(key)        // ① immutable
    || srcValue === undefined             // ② 没带这个字段
    || (isObject(srcValue) && isEmpty(srcValue)))  // ③ 空 usage/cost
  return objValue;
return srcValue;                            // 否则后者覆盖前者

哪些字段被锁成 immutable?看 observations 这张表:[id, project_id, trace_id, start_time, created_at, environment]。 它们是记录的身份与锚点——一旦第一条事件写定,后续 update 再怎么改都动不了;其余 payload 字段(output、usage、end_time、level、metadata……)才走 last-write-wins。

合并过程中还藏着一个体贴的向后兼容细节:如果一个 observation 合并完发现没有 trace_id(很老的 SDK,< 2.0 时代埋点不带 trace), IngestionService 会顺手造一条「包装 trace」——用 observation 自己的 id 当 trace id,补写进 traces 表,再把它挂上去。这样无论数据多老, 每个 observation 都不会变成「孤儿」,第 5 课那棵 trace 树永远完整。一个小小的补偿逻辑,体现的是这套系统对历史数据与多版本 SDK 的尊重与照顾。

字段类别例子合并规则
身份 / 锚点(immutable)id · project_id · trace_id · start_time · created_at · environmentfirst-wins:第一次写定,永不被覆盖
载荷(mutable)end_time · output · usage · level · metadata · model…last-write-wins:后到事件覆盖,但 undefined / 空对象不抹旧值

observation 那条线的完整一生

把最复杂的 processObservationEventList 拆开看,从一堆事件到写回库,正好七步:

① 幂等跳过 查 Redis 去重 ② 按时间排序 create 先于 update ③ 读 CH 底稿 已存在行当基线 ④ 合并 mergeRecords 左折叠 ⑤ 补 input/output 回找最新非空 ⑥ 算 token+成本 getGenerationUsage ⑦ 入写库队列 addToQueue ClickhouseWriter 批写落盘(L17) IngestionService 只合并、从不直接写库——第 7 步只把最终记录塞进批写器队列
1

幂等跳过

先查 Redis:这个 fileKey 最近几分钟处理过吗?处理过就直接跳过(配合第 13 课「事件 id 当文件名」,重复投递不重复算)。

2

按时间排序

toTimeSortedEventList 把事件按 timestamp 排好;同一时刻 create 排在 update 前。

3

读 ClickHouse 底稿

getClickhouseRecord 取出该 observation 已存在的行(带类型与 start_time 过滤),当合并基线。

4

事件 → 记录 → 合并

把事件映射成记录,mergeRecords 左折叠出 mergedObservationRecord。

5

补 input/output

从最新往回找第一条非空的 input / output(事件里没有就退回 ClickHouse 的值),并归一化 tool 数据。

6

算 token 与成本

getGenerationUsage 做模型匹配、token 计数、成本计算(第 16 课详解),并进最终记录。

7

入写库队列

clickhouseWriter.addToQueue(Observations, finalRecord)——不直接写库,交给第 17 课的批写器攒批落盘。

第 7 步要特别留意:IngestionService 自己从不直接写 ClickHouse,它只把合并好的记录塞进 ClickhouseWriter 的队列。 为什么?因为逐条写 ClickHouse 是性能灾难(第 8 课说过它爱大批量)。把「合并」和「落盘」再切开,正是第 6 课「每层只干一件事」的又一次体现——合并归 IngestionService,攒批归 ClickhouseWriter。这种「先在内存里把记录算到位、再交给专门的批写器统一落盘」的两段式,让高频写入既保持正确,又不至于把列式存储压垮;第 17 课会看到批写器如何按表攒批、定时刷新。

🎯 设计取舍
既然第 8 课的 ReplacingMergeTree 自己就会按 event_ts 去重,为什么写入侧还要再合并一遍? 因为两者解决的是不同时刻的问题。 写入侧的预合并保证:每次写出的那一行,当下就是正确的完整状态——查询立刻就能读到对的数据,不必等后台合并。而 ReplacingMergeTree 的后台去重是最终发生的, 用来收拾「同一条记录被并发处理、产生多行」这类竞态的尾巴。两层各管一段:写入侧求「即时正确」,存储侧求「最终唯一」。 再加上 event_ts=now 这个版本戳把两层咬合起来——预合并产出的新行永远有更大的 event_ts,于是它一定能在存储层的去重里胜出。这是典型的纵深防御。
🎯 本课要点
  • mergeAndWrite 是调度员:按实体类型 switch 到 process{Trace,Observation,Score,DatasetRunItem}EventList,骨架都是「读旧→转换→合并→写回」。
  • 合并有两个输入:S3 里本实体的全部事件(listFiles,按时间排序)+ ClickHouse 里当前那一行(当 immutable 基线)。
  • mergeRecords 把它们左折叠(overwriteObject 后者覆盖前者),并打 event_ts=now 版本戳。
  • 三条不覆盖规则:immutable 键(id/start_time/created_at…first-wins)、undefined、空对象——所以 update 只带变化字段、不抹掉 create 的值。
  • 合并完只 addToQueue,不直接写库(落盘归第 17 课)。写入侧预合并求「即时正确」,存储侧 ReplacingMergeTree 求「最终唯一」,靠 event_ts 咬合——纵深防御。

🧪 自测 · 想一想为什么这么设计

1. 一个 observation 先收到 create(带 startTime、input、model),后收到 update(带 endTime、output、usage)。mergeRecords 合并后,startTime 和 model 还在吗?为什么?
  1. 都被清空:后到的 update 整体覆盖了 create
  2. 都还在:update 没带这些字段(值为 undefined),overwriteObject 的“不覆盖”规则会保留旧值;startTime 还是 immutable 键,本就锁死
  3. 只有 model 还在,startTime 被清空
  4. 随机保留其一
看答案与解析 点击展开
答案:B。overwriteObject 用 lodash mergeWith,命中三条之一就保留旧值:① immutable 键(id/project_id/trace_id/start_time/created_at/environment)② 值为 undefined ③ 空对象。update 只带变化字段,没带的 input/model 是 undefined 故不被抹;start_time 更是 immutable。这正是 create/update 能正确合并的根本。
2. 合并时,mergeRecords 的输入数组里第一个元素总是 ClickHouse 已存在的那条记录。把它放第一位有什么用?
  1. 为了先把它删掉
  2. 纯粹是数组顺序习惯,没有语义
  3. 为了让 ClickHouse 记录覆盖所有新事件
  4. 它当 immutable 字段的基线:左折叠从它开始,后续事件只能覆盖 mutable 字段;created_at、start_time 等锁死字段以它为准,保证身份/锚点稳定
看答案与解析 点击展开
答案:D。折叠从最左(CH 底稿)往右叠事件,后者覆盖前者。但 immutable 键不被覆盖,于是 created_at/start_time 等永远保留 CH 底稿(即第一次写定)的值。这让“读旧底稿 + 叠新修订”既纳新又不改旧,是合并正确性的关键。
3. 既然第 8 课的 ReplacingMergeTree 会按 event_ts 自动去重,为什么 IngestionService 在写入侧还要预合并一遍?
  1. 为了把数据写两遍更安全
  2. 纯粹是历史遗留代码
  3. 因为 ReplacingMergeTree 其实不工作
  4. 两层解决不同时刻的问题:写入侧预合并保证“写出的那一行当下就完整正确”,查询立即可读;ReplacingMergeTree 的后台去重是最终发生的,收拾并发竞态的尾巴。event_ts=now 让新行总能胜出,两层咬合成纵深防御
看答案与解析 点击展开
答案:D。写入侧求“即时正确”——不预合并的话,查询会读到半成品(只有 create 没有 update);存储侧求“最终唯一”——收拾“同一记录被并发处理产生多行”的竞态。event_ts=now 这个版本戳把两层咬合:预合并产出的新行 event_ts 更大,必在 ReplacingMergeTree 去重里胜出。这是典型的纵深防御。
💭 发散思考(没有标准答案,动手或动脑想想)
  • IngestionService 把“合并”和“落盘”切成两层,又把“即时正确”和“最终唯一”分给写入侧与存储侧。这种“同一目标、两层各管一段、用一个版本戳咬合”的纵深防御,你在自己接触过的系统里见过类似设计吗?它的好处和复杂度代价你怎么看?

Lesson 14's worker took the job and read the events back from S3 by fileKey. This lesson is the heart of the whole write path: how does that pile of events, plus the record's current state in the database, get kneaded into one final record? The answer lives in IngestionService — mergeAndWrite dispatches by entity type, mergeRecords folds many inputs into one with a single "left-fold + later-overwrites" move, then stamps a event_ts=now version. Get it, and Lesson 8's ReplacingMergeTree gains its "other (write-side) half".

📝 Analogy
Think of merging as collaboratively editing a wiki article. The database's currently published version is the base; each event received this time is like a chronologically ordered revision, applied onto the base in turn. Later edits override earlier ones (same field, newer value wins); but a field a revision didn't touch stays as is (you only changed the "conclusion", the "body" isn't wiped). A few things are locked, untouchable by anyone: the article's creation time, its unique id. Stack all revisions and you get a merged draft, tagged with "this merge's moment" as the version — exactly what IngestionService does: the base from ClickHouse, the revisions from S3, the merged draft written back.

mergeAndWrite: split by entity type first

mergeAndWrite itself is just a dispatcher. It takes Lesson 13's computed entity type (trace / observation / score / dataset_run_item) and a switch hands the work to the matching process{Type}EventList. Every line has the same skeleton — read old, transform, merge, write back — only the field details differ by type.

mergeAndWrite: one switch, four lines mergeAndWriteswitch (entityType) processTraceEventList→ traces table processObservationEventList→ observations (richest) processScoreEventList→ scores table processDatasetRunItemEventListcreate only, no merge
Split, not converge: mergeAndWrite hands the job by entity type to four dedicated lines, each writing back to Lesson 8's matching table. The observation line is the richest (token/cost, wrapper-trace backfill). Source: worker/src/services/IngestionService/index.ts:149-195.
Example: merge = stored row + patch → new row stored row id: obs_7f output: null total_cost: null event_ts: 12:00:00.1 incoming update (patch) id: obs_7f output: "Here is…" total_cost: 0.0041 only the changed fields result row (written to CH) id: obs_7f output: "Here is…" total_cost: 0.0041 event_ts: 12:00:01.9 event_ts updated → keep this one
Merge is “apply a patch”, not “edit the row in place” (semantics per IngestionService; values illustrative): the update carries only changed fields; the merger overlays them onto the current row to get the result row, then appends it with an updated event_ts. ClickHouse's ReplacingMergeTree keeps the latest by event_ts at read time — so “update” is implemented as “append + keep-latest-at-query”.

On each line, the merge actually has two sources, both indispensable:

📜 S3: all events of this entity

The worker uses listFiles to read back all events of this entity in S3 (create, update… none missed), then sorts them by timestamp. This is the "revision stream".

🗄️ ClickHouse: the current record

Meanwhile it getClickhouseRecords the existing row from observations as the base draft. It goes first in the merge array, the baseline for immutable fields.

Why need both? Seemingly redundant, but each is irreplaceable. The S3 one is the events' real ledger (Lesson 14); re-reading all events guarantees the merged record is complete, even if this job is a late backfill. The ClickHouse one provides the authoritative baseline for immutable fields (created_at, start_time — those "locked once first written" values) and backstops early events that may have aged out of S3. Stacked together, they lose nothing new while changing nothing old — the essence of "read the old base + apply new revisions".

mergeRecords: left-fold, later overwrites

The core algorithm is just a few lines: assemble "ClickHouse base + time-sorted event records" into an array, fold them left-to-right with overwriteObject, then stamp event_ts = now. The crux is entirely in overwriteObject's three "don't overwrite" rules.

Left-fold: base → stack events → final record CH baseexisting row(may be empty) event ① createstartTime·inputmodel event ② updateendTime·outputusage final recordeach field: latest set valueevent_ts = now overwriteObject ×N overwriteObject's three "don't overwrite" rules (later value ignored, earlier kept): ① immutable keyid·start_time·created_at… ② value is undefinedabsent field won't wipe old ③ empty object {}empty usage/cost no-op
It stacks "new fields that have a value": a later event overwrites an earlier one, with three exceptions — immutable keys, undefined values, empty objects. So update carrying only endTime/output/usage leaves create's startTime/input/model untouched. Source: IngestionService/index.ts:984-1005, utils.ts:56-81.
worker/src/services/IngestionService/index.ts & utils.tsmerge core
// Left-fold: base first, events (time-sorted) after, stacked one by one
let result = { id: records[0].id, project_id: records[0].project_id };
for (const record of records)
  result = overwriteObject(result, record, immutableEntityKeys);
result.event_ts = new Date().getTime();   // version stamp → Lesson 8's ReplacingMergeTree

// overwriteObject: lodash mergeWith; hit any rule → keep old objValue
if (nonOverwritableKeys.includes(key)        // ① immutable
    || srcValue === undefined             // ② field not provided
    || (isObject(srcValue) && isEmpty(srcValue)))  // ③ empty usage/cost
  return objValue;
return srcValue;                            // else later overwrites earlier

Which fields are locked immutable? For the observations table: [id, project_id, trace_id, start_time, created_at, environment]. These are the record's identity and anchors — set once by the first event, untouchable by any later update; everything else (output, usage, end_time, level, metadata…) follows last-write-wins.

field classexamplesmerge rule
identity / anchor (immutable)id · project_id · trace_id · start_time · created_at · environmentfirst-wins: set once, never overwritten
payload (mutable)end_time · output · usage · level · metadata · model…last-write-wins: later event overwrites, but undefined / empty object won't wipe old

The merge also hides a thoughtful backward-compat detail: if a merged observation turns out to have no trace_id (a very old SDK from the pre-2.0 era that instrumented without a trace), IngestionService quietly creates a "wrapper trace" — using the observation's own id as the trace id, writing it into the traces table and attaching the observation to it. So no matter how old the data, no observation becomes an orphan, and Lesson 5's trace tree stays whole. A tiny compensating bit of logic that respects historical data and multi-version SDKs.

The full life of the observation line

Unpack the most complex processObservationEventList and the path from a pile of events to a DB write is exactly seven steps:

1 idempotency skip dedup via Redis 2 sort by time create before update 3 read CH base existing row as baseline 4 merge mergeRecords left-fold 5 backfill in/out latest non-empty 6 token+cost getGenerationUsage 7 enqueue write addToQueue ClickhouseWriter batched persist (L17) IngestionService only merges, never writes directly — step 7 just drops the final record into the writer's queue
1

idempotency skip

Check Redis first: was this fileKey processed in the last few minutes? If so, skip outright (with Lesson 13's "event id as filename", redelivery never double-counts).

2

sort by time

toTimeSortedEventList orders events by timestamp; at the same instant, create comes before update.

3

read ClickHouse base

getClickhouseRecord fetches the observation's existing row (filtered by type and start_time) as the merge baseline.

4

events → records → merge

Map events to records, mergeRecords left-folds into mergedObservationRecord.

5

fill input/output

Find the first non-null input/output scanning from newest back (fallback to ClickHouse's), and normalize tool data.

6

compute tokens & cost

getGenerationUsage does model matching, token counting, cost calculation (detailed in Lesson 16), folded into the final record.

7

enqueue the write

clickhouseWriter.addToQueue(Observations, finalRecord) — not a direct DB write, handed to Lesson 17's batch writer.

Note step 7: IngestionService never writes ClickHouse directly — it only pushes the merged record into ClickhouseWriter's queue. Why? Because row-by-row ClickHouse writes are a performance disaster (Lesson 8 noted it loves big batches). Splitting "merge" from "persist" is again Lesson 6's "each layer does one thing" — merging to IngestionService, batching to ClickhouseWriter. This two-stage "compute the record fully in memory, then hand it to a dedicated batch writer" keeps high-frequency writes correct without crushing the columnar store; Lesson 17 shows how the writer batches per table and flushes on a timer.

🎯 Design trade-off
If Lesson 8's ReplacingMergeTree already dedups by event_ts, why merge again on the write side? Because they solve problems at different moments. The write-side pre-merge guarantees: the very row written out is the correct, complete state right now — queries read correct data immediately, no waiting for background merges. ReplacingMergeTree's background dedup happens eventually, cleaning up the tail of races like "one record processed concurrently, producing multiple rows". Each layer owns a stretch: the write side seeks "immediate correctness", the storage side "eventual uniqueness". And event_ts=now meshes the two — a pre-merged new row always has a larger event_ts, so it always wins the storage layer's dedup. Classic defense in depth.
🎯 Key points
  • mergeAndWrite is a dispatcher: switch by entity type to process{Trace,Observation,Score,DatasetRunItem}EventList, all sharing "read old → transform → merge → write back".
  • Merge has two inputs: all events of this entity from S3 (listFiles, time-sorted) + the current row from ClickHouse (the immutable baseline).
  • mergeRecords left-folds them (overwriteObject, later overwrites earlier) and stamps the event_ts=now version.
  • Three don't-overwrite rules: immutable keys (id/start_time/created_at… first-wins), undefined, empty objects — so update carries only changed fields without wiping create's values.
  • After merging it only addToQueues, not a direct write (persist is Lesson 17). Write-side pre-merge for "immediate correctness", storage-side ReplacingMergeTree for "eventual uniqueness", meshed by event_ts — defense in depth.

🧪 Self-test - think about the design

1. An observation gets create (startTime, input, model) then update (endTime, output, usage). After mergeRecords, do startTime and model survive? Why?
  1. both wiped: the later update wholly overwrites create
  2. both survive: update doesn't carry them (undefined), so overwriteObject's 'don't overwrite' rule keeps the old values; startTime is also an immutable key, locked anyway
  3. only model survives, startTime is wiped
  4. one of them is randomly kept
Show answer & explanation click to expand
Answer: B. overwriteObject uses lodash mergeWith and keeps the old value on any of three: ① immutable keys (id/project_id/trace_id/start_time/created_at/environment) ② undefined value ③ empty object. update carries only changed fields, so absent input/model are undefined and not wiped; start_time is immutable anyway. This is exactly why create/update merge correctly.
2. In mergeRecords, the first element of the input array is always the existing ClickHouse record. What's the point of putting it first?
  1. to delete it first
  2. purely an array-order habit, no semantics
  3. so the ClickHouse record overwrites all new events
  4. it's the baseline for immutable fields: the left-fold starts from it, later events can only overwrite mutable fields; locked fields like created_at, start_time take its value, keeping identity/anchors stable
Show answer & explanation click to expand
Answer: D. The fold goes left (CH base) to right (events), later overwriting earlier. But immutable keys aren't overwritten, so created_at/start_time keep the CH base's (first-written) value. This lets 'read old base + apply new revisions' absorb new while preserving old — key to merge correctness.
3. Since Lesson 8's ReplacingMergeTree auto-dedups by event_ts, why does IngestionService still pre-merge on the write side?
  1. to write the data twice for safety
  2. purely legacy code
  3. because ReplacingMergeTree doesn't actually work
  4. the two layers solve problems at different moments: write-side pre-merge ensures 'the written row is complete and correct now', readable immediately; ReplacingMergeTree's background dedup is eventual, cleaning up concurrency races. event_ts=now lets the new row always win, meshing the layers into defense in depth
Show answer & explanation click to expand
Answer: D. The write side seeks 'immediate correctness' — without pre-merge, queries would read a half-baked row (create but no update); the storage side seeks 'eventual uniqueness', cleaning races that produce multiple rows for one record. The event_ts=now stamp meshes them: a pre-merged new row has a larger event_ts and always wins ReplacingMergeTree's dedup. Classic defense in depth.
💭 Open questions (no single right answer - just think or try)
  • IngestionService splits 'merge' from 'persist', and assigns 'immediate correctness' to the write side and 'eventual uniqueness' to the storage side. Have you seen this 'one goal, two layers each owning a stretch, meshed by a version stamp' defense-in-depth elsewhere? How do you weigh its benefits against the added complexity?