🪢 Langfuse 图解教程Langfuse Visual Guide 第三部分 · 摄取链路Part 3 · The Ingestion Path 17 / 55
第三部分 · 摄取链路Part 3 · The Ingestion Path

ClickhouseWriter:批量落盘ClickhouseWriter: batched persistence

🧭 读前Before you start
先读 第14课 的队列与 第8课 的 CH 表,本课讲 ClickhouseWriter 如何批量落盘。
Read Lesson 14 (the queue) and Lesson 8 (the CH tables); this lesson covers how ClickhouseWriter batches to disk.

第 15 课合并完,最后一步是 clickhouseWriter.addToQueue(...)——记录被塞进一个队列,而不是马上写库。这一课就讲那个队列背后的 ClickhouseWriter:一个单例,把潮水般涌来的单条记录攒成少数几个大批次,再一次性 INSERT 进 ClickHouse。 为什么非要攒?因为第 8 课说过,ClickHouse 极度偏爱大批量写入、最怕逐条插入。这一层,就是把「高频小写」翻译成「低频大写」的缓冲带。

🚌 生活类比
把 ClickhouseWriter 想成摆渡班车。乘客(记录)一个个到站,司机不会来一个就开一趟——那样油费(写入开销)高得离谱。 他等到车坐满了(达到 batchSize),或者发车时刻到了(writeInterval 定时器响),就一趟把整车人送过去。 两个条件谁先到就按谁发:高峰期车一会儿就满、靠「坐满」发车;低谷期人少,就靠「到点」发车,免得最后几个乘客干等太久。 而且每条线路(每张表)都有自己的一辆车:traces 一辆、observations 一辆、scores 一辆,各坐各的、各发各的,互不耽误。

缓冲带:每张表一个队列,两个发车条件

ClickhouseWriter 内部为每张表各开一个内存队列(traces / observations / scores / …)。 第 15 课的 addToQueue 把合并好的记录推进对应队列;记录在队列里排队等发车。发车(flush)由两个条件触发,谁先到算谁:

ClickhouseWriter:攒批 → 批量 INSERT IngestionService第 15 课addToQueue一条条推进来 queue[traces] ▢▢▢▢攒到 batchSize… queue[observations] ▢▢▢各表独立 queue[scores] ▢▢各发各的 flush 触发① 队列 ≥ batchSize② writeInterval 到点谁先到算谁 ClickHouse一次 INSERT整批写入
攒批缓冲带:每张表一个内存队列;addToQueue 喂入,达 batchSize 立刻 flush 该表,或定时器每 writeInterval 触发 flushAll 刷所有表。flush 把整批一次 INSERT(JSONEachRow)。源码:worker/src/services/ClickhouseWriter/index.ts:34-135, 550-600。
示例:批写器的攒批与 flush 时刻 时间 → 缓冲量 batchSize 阈值 ① 按量 flush ① 按量 flush ② 按时 flush writeInterval 到点 ③ flushAll(true)关机排空·不丢 谁先到算谁:攒满 batchSize 或到 writeInterval 就批量 INSERT 一次;优雅停机一次写尽
攒批落盘:按量 + 按时 双触发(常量名对齐 ClickhouseWriter;值为示例):事件先进每表内存缓冲,batchSize 攒满就按量 flush,否则每 writeInterval 按时 flush,谁先到算谁;关机时 flushAll(true) 一次写尽不丢数据。用「批量 INSERT」换 ClickHouse 的高写入吞吐。
worker/src/services/ClickhouseWriter/index.ts两个触发
// 触发①:入队时,本表攒够一批就立刻刷
addToQueue(tableName, data) {
  this.queue[tableName].push({ createdAt: Date.now(), data });
  if (this.queue[tableName].length >= this.batchSize)
    this.flush(tableName);          // 坐满发车
}
// 触发②:定时器,每 writeInterval 毫秒刷所有表
setInterval(() => this.flushAll(), this.writeInterval);   // 到点发车

// flush:从队列取一批,批量 INSERT(带退避重试 maxAttempts)
const items = entityQueue.splice(0, fullQueue ? all : this.batchSize);
await backOff(() => writeToClickhouse({ table, records }), { numOfAttempts });

为什么 ClickHouse 这么怕「逐条写」?回到第 8 课的存储机制:MergeTree 每收一次 INSERT 就在磁盘上落一个新的数据分片(part), 之后还要靠后台不断把小分片合并成大分片。如果你每条记录都单独 INSERT,瞬间会产生海量小分片——磁盘元数据爆炸、后台合并永远追不上、查询也被拖慢 (甚至触发 ClickHouse 的「too many parts」保护而拒写)。把一万条记录攒成一次 INSERT,就只生成一个大分片,写入、合并、查询全都轻松。 所以这一层不是可有可无的优化,而是让 ClickHouse 这种列式存储能扛住高频摄取的必要前提——没有这道缓冲带,再好的表设计也会被海量小写拖垮。

两个发车条件,外加一次优雅排空

「坐满」和「到点」这两个条件,分别照顾了两种负载,缺一不可:

谁先到算谁:吞吐与延迟的平衡 高峰期 ▶ 瞬间坐满 batchSize ① 立刻 flush(按量) 延迟极低、批批都满,吞吐拉满 低谷期 ▶ 半天才两条,凑不满 ② 到点 flush(按时) 每 writeInterval 一刷,不让数据干等 关机时 flushAll(true) 排空 清定时器 + 把所有队列一次写尽,不丢数据
两条件互补 + 优雅关机:高峰靠 batchSize 保吞吐,低谷靠 writeInterval 保延迟上限;shutdown 先 clearInterval 再 flushAll(true) 把内存里剩的全部写尽。flushAll 用 Promise.all 并行刷各表。
触发条件解决什么
① 按量(batchSize)某表队列长度 ≥ batchSize高吞吐时批批写满,把 ClickHouse 的写入效率拉满
② 按时(writeInterval)定时器每 writeInterval 毫秒触发 flushAll低吞吐时给延迟封顶,数据不会无限期滞留内存
③ 关机排空shutdown() 调 flushAll(true)进程退出前把缓冲全部落盘,避免丢数据

这两个参数 batchSize 与 writeInterval 其实是一对吞吐 / 延迟旋钮:调大 batchSize、拉长 interval,批更大、写库更省,但单条记录从「算完」到「能被查到」的延迟变长; 反之延迟低但写得更碎。Langfuse 把它们做成环境变量,让你按自己的流量与时效要求自由去拧,无需改代码。

把一条合并好的记录在写入器里的旅程拆开,正好五步:

1

入队

addToQueue(table, record) 把记录连同 createdAt 时间戳推进该表队列尾部。

2

等发车

记录在内存队列里排队。期间系统还会记录它等了多久(wait_time 指标),用来观测积压。

3

触发 flush

本表攒到 batchSize(按量)或定时器到点(按时),splice 取出最多一批。

4

批量写入

writeToClickhouse 把整批以 JSONEachRow 一次 INSERT;带 backOff 退避重试。

5

成功 / 重试

成功则这批出队完成;可重试错误退避再试,超长批拆小重排队——尽量不连累整批。

为什么是单例,为什么敢用内存缓冲

ClickhouseWriter 是单例(私有构造 + getInstance):整个 worker 进程只有一个写入器、一组队列。这很关键—— 只有汇到一处,才能把来自无数任务的零散记录真正攒成大批;如果每个任务各写各的,批就永远攒不大。写入时还带退避重试(maxAttempts), 遇到可重试错误就指数退避再试,遇到「字符串过长」这类错误则把批拆小重试——既追求大批量,又不被个别异常记录拖垮整批。这种「整批共进退、坏的单独拎出来」的策略,和第 12 课摄取 API 的「部分成功」一脉相承:系统在每一层都尽量做到容错而不牵连,让一两条问题数据不至于阻塞整条管道。

S3 · 事件账本 绝不丢 · 持久层 队列任务 写库成功前不 ack ClickhouseWriter 内存缓冲(易失·攒大批) ⚡ 崩溃 → 这批丢失 ClickHouse 最终行 数据一条不少 崩了就重投:从 S3 重读 → 重合并 → 重写 持久性由 S3 + 队列兜底,这一层才敢用易失内存缓冲去换写入吞吐

🧩 单例 + 按表队列

一个进程一个 writer,把所有任务的记录汇聚成大批;按表分队列,让 traces/observations/scores 各自满批、并行刷,互不阻塞。

🛡️ 批量写 + 退避重试

一次 INSERT 写整批(JSONEachRow),失败按 maxAttempts 退避重试;超长批自动拆分。把「写得快」和「写得稳」一起拿下。

🎯 设计取舍
记录攒在内存队列里,万一 worker 突然崩了,这批不就丢了吗? 会丢——但这不要紧,因为真正的「账本」不在这里。回想第 14 课: 每个事件都原原本本躺在 S3,而队列任务在写库成功前不会被确认。于是 worker 半路崩溃,最多是这批合并结果没落盘, 对应的队列任务会重新投递、从 S3 重新读事件、重新合并重新写——数据一条不少。正因为持久性已经由上游的 S3 + 队列兜底, 这一层才敢放心地用易失的内存缓冲去换写入吞吐。这是一个典型的分层设计智慧:把持久和高效分给不同层,各自做到极致—— S3 负责「绝不丢」,ClickhouseWriter 负责「写得快」,谁也不必为对方的目标妥协。
🎯 本课要点
  • ClickhouseWriter 是单例,把高频单条记录攒成大批再一次 INSERT——因为第 8 课 ClickHouse 偏爱大批量、最怕逐条。
  • 每张表一个内存队列(traces/observations/scores…),addToQueue 喂入,flushAll 用 Promise.all 并行刷各表。
  • 两个 flush 触发:① 队列 ≥ batchSize(按量,保吞吐)② 每 writeInterval 定时器(按时,封顶延迟);谁先到算谁。
  • shutdown 先清定时器、再 flushAll(true) 排空所有队列,进程退出不丢数据;写入带退避重试、超长批自动拆分。
  • 敢用易失内存缓冲换吞吐,是因为持久性由上游 S3 + 队列兜底(第 14 课):崩了就重投重算,一条不丢——分层各司其职。

🧪 自测 · 想一想为什么这么设计

1. ClickhouseWriter 为什么要把单条记录攒成大批再写,而不是 IngestionService 算完一条就直接 INSERT 一条?
  1. 因为 ClickHouse 的 MergeTree 每次 INSERT 都落一个磁盘分片(part),逐条写会产生海量小分片、后台合并追不上、查询变慢甚至触发 too-many-parts 拒写;攒成一批只生成一个大分片
  2. 因为 IngestionService 不能访问数据库
  3. 因为单条写会丢数据
  4. 因为攒批能加密数据
看答案与解析 点击展开
答案:A。第 8 课讲过 ClickHouse 偏爱大批量。MergeTree 每个 INSERT 生成一个数据分片,小分片越多、磁盘元数据和后台合并压力越大。把高频小写翻译成低频大写,是列式存储扛住高频摄取的必要前提,而非可有可无的优化。
2. ClickhouseWriter 的两个 flush 触发条件是什么?为什么需要两个而不是一个?
  1. ① 白天 ② 夜间
  2. 只有一个:队列满才写
  3. ① 内存满 ② 磁盘满
  4. ① 某表队列 ≥ batchSize(按量)② 定时器每 writeInterval 触发 flushAll(按时);谁先到算谁——高峰靠“坐满”保吞吐,低谷靠“到点”保延迟上限,缺一就会要么延迟失控、要么批永远凑不满
看答案与解析 点击展开
答案:D。只靠 batchSize:低谷期凑不满一批,记录会无限期滞留内存、延迟失控。只靠定时器:高峰期一个 interval 内涌入远超一批,仍可能积压。两者“谁先到算谁”互补:高吞吐批批写满拉高效率,低吞吐到点即发封顶延迟。关机时再 flushAll(true) 排空,不丢数据。
3. 记录攒在内存队列里,worker 崩溃这批就没了。Langfuse 为什么敢用这种“易失”的内存缓冲?
  1. 因为丢一点数据无所谓
  2. 因为 worker 永远不会崩
  3. 因为持久性由上游兜底:每个事件原原本本在 S3,队列任务写库成功前不会被确认;崩了任务会重投、从 S3 重读重合并重写,一条不丢——所以这层可放心用内存换吞吐
  4. 因为内存比磁盘更可靠
看答案与解析 点击展开
答案:C。这是典型的分层智慧:S3(第 14 课)负责“绝不丢”,ClickhouseWriter 负责“写得快”,谁也不为对方妥协。因为队列 + S3 已经保证了持久性与可重放,写入器才敢用易失内存缓冲去换吞吐——崩溃只是触发一次重投重算。
💭 发散思考(没有标准答案,动手或动脑想想)
  • ClickhouseWriter 把“持久”交给上游 S3、自己专注“高效”,于是敢用易失内存缓冲换吞吐。这种“把不同的质量目标分给不同层、各自做到极致”的思路,你在自己设计系统时会怎么运用?如果某一层既要快又要绝不丢,会带来哪些额外复杂度?

Lesson 15's merge ended with clickhouseWriter.addToQueue(...) — the record goes into a queue, not straight to the DB. This lesson covers what's behind that queue, the ClickhouseWriter: a singleton that gathers the flood of single records into a handful of big batches, then INSERTs them into ClickHouse in one shot. Why batch at all? Because, as Lesson 8 said, ClickHouse strongly prefers large batched writes and dreads row-by-row inserts. This layer is the buffer that translates "high-frequency small writes" into "low-frequency large writes".

🚌 Analogy
Think of ClickhouseWriter as a shuttle bus. Passengers (records) arrive one by one, but the driver doesn't depart for each one — the fuel cost (write overhead) would be absurd. He waits until the bus is full (hits batchSize), or the departure time arrives (the writeInterval timer fires), then takes the whole busload across in one trip. Whichever condition comes first wins: at rush hour the bus fills fast and leaves on "full"; in a lull, few passengers, so it leaves on "time" so the last few don't wait forever. And each route (each table) has its own bus: one for traces, one for observations, one for scores, each loading and departing independently, never holding up the others.

The buffer: one queue per table, two departure conditions

Inside, ClickhouseWriter keeps an in-memory queue per table (traces / observations / scores / …). Lesson 15's addToQueue pushes the merged record into the matching queue; records wait in line to depart. Departure (flush) is triggered by two conditions, whichever comes first:

ClickhouseWriter: batch up → bulk INSERT IngestionServiceLesson 15addToQueuepushed one by one queue[traces] ▢▢▢▢fills to batchSize… queue[observations] ▢▢▢per-table independent queue[scores] ▢▢each departs on its own flush trigger① queue ≥ batchSize② writeInterval elapsedwhichever first ClickHouseone INSERTwhole batch
A batching buffer: one in-memory queue per table; addToQueue feeds it, hitting batchSize flushes that table immediately, or the timer fires flushAll every writeInterval. flush sends the whole batch in one INSERT (JSONEachRow). Source: worker/src/services/ClickhouseWriter/index.ts:34-135, 550-600.
Example: the batch writer accumulating then flushing time → buffered batchSize threshold ① flush by size ① flush by size ② flush by time writeInterval elapsed ③ flushAll(true)drain on shutdown · no loss whichever comes first: when batchSize fills or writeInterval elapses, batch-INSERT once; graceful shutdown flushes all
Batched persistence: flush by size and by time (constant names per ClickhouseWriter; values illustrative): events enter a per-table memory buffer; a full batchSize triggers a flush by size, otherwise every writeInterval triggers a flush by time, whichever comes first; at shutdown flushAll(true) drains everything losslessly. Batched INSERTs buy ClickHouse's high write throughput.
worker/src/services/ClickhouseWriter/index.tstwo triggers
// Trigger ①: on enqueue, flush this table once it has a full batch
addToQueue(tableName, data) {
  this.queue[tableName].push({ createdAt: Date.now(), data });
  if (this.queue[tableName].length >= this.batchSize)
    this.flush(tableName);          // depart when full
}
// Trigger ②: a timer, flush all tables every writeInterval ms
setInterval(() => this.flushAll(), this.writeInterval);   // depart on time

// flush: take a batch from the queue, bulk INSERT (with backoff retry maxAttempts)
const items = entityQueue.splice(0, fullQueue ? all : this.batchSize);
await backOff(() => writeToClickhouse({ table, records }), { numOfAttempts });

Why does ClickHouse so dread "row-by-row writes"? Back to Lesson 8's storage mechanics: a MergeTree drops a new data part on disk for every INSERT, then relies on background merges to combine small parts into large ones. INSERT each record separately and you instantly spawn a flood of tiny parts — exploding disk metadata, background merges that never catch up, slowed queries (and even ClickHouse's "too many parts" guard rejecting writes). Batch ten thousand records into one INSERT and you create just one big part — easy to write, merge, and query. So this layer isn't an optional optimization; it's a precondition for a columnar store like ClickHouse to withstand high-frequency ingestion — without this buffer, even the best table design gets crushed by a deluge of small writes.

Two departure conditions, plus a graceful drain

"Full" and "on time" each serve a different load, and both are needed:

Whichever first: balancing throughput and latency rush hour ▶ fills batchSize instantly ① flush now (by size) low latency, full batches, max throughput lull ▶ only two in ages, never fills ② flush on time flush every writeInterval, no waiting forever on shutdown flushAll(true) drain clear timer + write out every queue, no loss
Complementary conditions + graceful shutdown: rush hour leans on batchSize for throughput, lulls on writeInterval to cap latency; shutdown first clearInterval then flushAll(true) writes out everything left in memory. flushAll fans out across tables with Promise.all.
triggerconditionwhat it solves
① by size (batchSize)a table's queue length ≥ batchSizeat high throughput, write full batches, maxing ClickHouse write efficiency
② by time (writeInterval)the timer fires flushAll every writeInterval msat low throughput, caps latency so data doesn't linger in memory indefinitely
③ shutdown drainshutdown() calls flushAll(true)persist all buffers before the process exits, avoiding data loss

The two parameters batchSize and writeInterval are really a pair of throughput / latency knobs: a bigger batchSize and longer interval mean larger batches and cheaper writes, but a longer delay from "computed" to "queryable" for each record; the reverse means lower latency but more fragmented writes. Langfuse makes them environment variables so you can tune them to your traffic and freshness needs without touching code.

Unpack a merged record's journey through the writer, and it's exactly five steps:

1

enqueue

addToQueue(table, record) pushes the record, with a createdAt timestamp, onto that table's queue tail.

2

wait to depart

The record waits in the in-memory queue. Meanwhile the system records how long it waited (a wait_time metric) to observe backlog.

3

trigger flush

This table hits batchSize (by size) or the timer fires (by time); splice takes up to a batch.

4

bulk write

writeToClickhouse INSERTs the whole batch as JSONEachRow in one go; with backOff retry.

5

success / retry

On success the batch is done; retryable errors back off and retry, oversized batches split and requeue — sparing the rest of the batch.

Why a singleton, and why dare to buffer in memory

ClickhouseWriter is a singleton (private constructor + getInstance): one writer, one set of queues per worker process. That's crucial — only by converging to one place can scattered records from countless jobs actually batch up large; if each job wrote on its own, batches would never grow. Writes carry backoff retry (maxAttempts): retryable errors back off and retry, "string too long" errors split the batch and retry — chasing big batches without letting one bad record drag down the whole batch. This "the batch rises and falls together, the bad ones get pulled out alone" strategy echoes Lesson 12's ingestion-API "partial success": at every layer the system tries to be fault-tolerant without collateral damage, so a problem record or two won't block the whole pipeline.

S3 · event ledger never loses · durable queue job not acked until write ok ClickhouseWriter memory buffer (volatile · batches) ⚡ crash → batch lost ClickHouse final row nothing lost crash ⇒ redeliver: re-read S3 → re-merge → re-write durability handled by S3 + queue, so this layer dares to trade a volatile buffer for write throughput

🧩 singleton + per-table queues

One writer per process gathers all jobs' records into big batches; per-table queues let traces/observations/scores each fill and flush in parallel, never blocking each other.

🛡️ bulk write + backoff retry

One INSERT per whole batch (JSONEachRow); failures retry with maxAttempts backoff; oversized batches auto-split. "Fast" and "stable" together.

🎯 Design trade-off
Records sit in an in-memory queue — if the worker suddenly crashes, isn't the batch lost? It is — but that doesn't matter, because the real "ledger" isn't here. Recall Lesson 14: every event rests verbatim in S3, and the queue job isn't acked until the DB write succeeds. So a mid-flight worker crash, at worst, means this batch of merged results didn't land; the corresponding queue job is redelivered, re-reads events from S3, re-merges and re-writes — not a single record lost. Precisely because durability is already backstopped by upstream S3 + the queue, this layer can safely trade volatile in-memory buffering for write throughput. A classic layered-design insight: assign durability and efficiency to different layers, each taken to the extreme — S3 owns "never lose", ClickhouseWriter owns "write fast", neither compromising for the other's goal.
🎯 Key points
  • ClickhouseWriter is a singleton that gathers high-frequency single records into big batches for one INSERT — because Lesson 8's ClickHouse prefers large batches and dreads row-by-row.
  • One in-memory queue per table (traces/observations/scores…); addToQueue feeds it, flushAll fans out across tables with Promise.all.
  • Two flush triggers: ① queue ≥ batchSize (by size, for throughput) ② every writeInterval timer (by time, capping latency); whichever first.
  • shutdown clears the timer then flushAll(true) drains all queues so nothing is lost on exit; writes carry backoff retry, oversized batches auto-split.
  • It dares to trade volatile memory buffering for throughput because durability is backstopped by upstream S3 + the queue (Lesson 14): crash → redeliver and recompute, nothing lost — layers each doing their job.

🧪 Self-test - think about the design

1. Why does ClickhouseWriter batch single records before writing, instead of IngestionService INSERTing each record as soon as it's computed?
  1. because ClickHouse's MergeTree drops a disk part per INSERT; row-by-row spawns a flood of tiny parts, background merges can't keep up, queries slow, even hitting the too-many-parts guard; one batch makes just one big part
  2. because IngestionService can't access the database
  3. because single writes lose data
  4. because batching encrypts the data
Show answer & explanation click to expand
Answer: A. Lesson 8 noted ClickHouse prefers large batches. Each INSERT creates a MergeTree part; more tiny parts means heavier disk metadata and background merges. Translating high-frequency small writes into low-frequency large ones is a precondition for a columnar store to withstand high-frequency ingestion, not an optional optimization.
2. What are ClickhouseWriter's two flush triggers, and why two rather than one?
  1. ① daytime ② nighttime
  2. only one: write when the queue is full
  3. ① memory full ② disk full
  4. ① a table's queue ≥ batchSize (by size) ② a timer fires flushAll every writeInterval (by time); whichever first — rush hour uses 'full' for throughput, lulls use 'on time' to cap latency; with only one, either latency runs away or batches never fill
Show answer & explanation click to expand
Answer: D. batchSize alone: in a lull a batch never fills, so records linger in memory indefinitely with runaway latency. Timer alone: at peak far more than a batch floods in per interval, risking backlog. The two complement 'whichever first': high throughput writes full batches for efficiency, low throughput flushes on time to cap latency. On shutdown, flushAll(true) drains everything, losing nothing.
3. Records sit in an in-memory queue, so a worker crash loses that batch. Why does Langfuse dare to use such 'volatile' in-memory buffering?
  1. because losing some data is fine
  2. because the worker never crashes
  3. because durability is backstopped upstream: every event sits verbatim in S3, and the queue job isn't acked until the DB write succeeds; on crash the job is redelivered, re-reads from S3, re-merges and re-writes, losing nothing — so this layer can trade memory for throughput
  4. because memory is more reliable than disk
Show answer & explanation click to expand
Answer: C. Classic layered wisdom: S3 (Lesson 14) owns 'never lose', ClickhouseWriter owns 'write fast', neither compromising for the other. Because the queue + S3 already guarantee durability and replayability, the writer can trade volatile memory buffering for throughput — a crash merely triggers a redeliver-and-recompute.
💭 Open questions (no single right answer - just think or try)
  • ClickhouseWriter delegates durability to upstream S3 and focuses on efficiency, so it dares to trade volatile memory buffering for throughput. How would you apply this 'assign different quality goals to different layers, each taken to the extreme' approach in your own designs? If one layer must be both fast and never-lose, what extra complexity does that bring?