第 15 课合并完,最后一步是 clickhouseWriter.addToQueue(...)——记录被塞进一个队列,而不是马上写库。这一课就讲那个队列背后的
ClickhouseWriter:一个单例,把潮水般涌来的单条记录攒成少数几个大批次,再一次性 INSERT 进 ClickHouse。
为什么非要攒?因为第 8 课说过,ClickHouse 极度偏爱大批量写入、最怕逐条插入。这一层,就是把「高频小写」翻译成「低频大写」的缓冲带。
ClickhouseWriter 想成摆渡班车。乘客(记录)一个个到站,司机不会来一个就开一趟——那样油费(写入开销)高得离谱。
他等到车坐满了(达到 batchSize),或者发车时刻到了(writeInterval 定时器响),就一趟把整车人送过去。
两个条件谁先到就按谁发:高峰期车一会儿就满、靠「坐满」发车;低谷期人少,就靠「到点」发车,免得最后几个乘客干等太久。
而且每条线路(每张表)都有自己的一辆车:traces 一辆、observations 一辆、scores 一辆,各坐各的、各发各的,互不耽误。
ClickhouseWriter 内部为每张表各开一个内存队列(traces / observations / scores / …)。
第 15 课的 addToQueue 把合并好的记录推进对应队列;记录在队列里排队等发车。发车(flush)由两个条件触发,谁先到算谁:
addToQueue 喂入,达 batchSize 立刻 flush 该表,或定时器每 writeInterval 触发 flushAll 刷所有表。flush 把整批一次 INSERT(JSONEachRow)。源码:worker/src/services/ClickhouseWriter/index.ts:34-135, 550-600。ClickhouseWriter;值为示例):事件先进每表内存缓冲,batchSize 攒满就按量 flush,否则每 writeInterval 按时 flush,谁先到算谁;关机时 flushAll(true) 一次写尽不丢数据。用「批量 INSERT」换 ClickHouse 的高写入吞吐。// 触发①:入队时,本表攒够一批就立刻刷 addToQueue(tableName, data) { this.queue[tableName].push({ createdAt: Date.now(), data }); if (this.queue[tableName].length >= this.batchSize) this.flush(tableName); // 坐满发车 } // 触发②:定时器,每 writeInterval 毫秒刷所有表 setInterval(() => this.flushAll(), this.writeInterval); // 到点发车 // flush:从队列取一批,批量 INSERT(带退避重试 maxAttempts) const items = entityQueue.splice(0, fullQueue ? all : this.batchSize); await backOff(() => writeToClickhouse({ table, records }), { numOfAttempts });
为什么 ClickHouse 这么怕「逐条写」?回到第 8 课的存储机制:MergeTree 每收一次 INSERT 就在磁盘上落一个新的数据分片(part),
之后还要靠后台不断把小分片合并成大分片。如果你每条记录都单独 INSERT,瞬间会产生海量小分片——磁盘元数据爆炸、后台合并永远追不上、查询也被拖慢
(甚至触发 ClickHouse 的「too many parts」保护而拒写)。把一万条记录攒成一次 INSERT,就只生成一个大分片,写入、合并、查询全都轻松。
所以这一层不是可有可无的优化,而是让 ClickHouse 这种列式存储能扛住高频摄取的必要前提——没有这道缓冲带,再好的表设计也会被海量小写拖垮。
「坐满」和「到点」这两个条件,分别照顾了两种负载,缺一不可:
batchSize 保吞吐,低谷靠 writeInterval 保延迟上限;shutdown 先 clearInterval 再 flushAll(true) 把内存里剩的全部写尽。flushAll 用 Promise.all 并行刷各表。| 触发 | 条件 | 解决什么 |
|---|---|---|
| ① 按量(batchSize) | 某表队列长度 ≥ batchSize | 高吞吐时批批写满,把 ClickHouse 的写入效率拉满 |
| ② 按时(writeInterval) | 定时器每 writeInterval 毫秒触发 flushAll | 低吞吐时给延迟封顶,数据不会无限期滞留内存 |
| ③ 关机排空 | shutdown() 调 flushAll(true) | 进程退出前把缓冲全部落盘,避免丢数据 |
这两个参数 batchSize 与 writeInterval 其实是一对吞吐 / 延迟旋钮:调大 batchSize、拉长 interval,批更大、写库更省,但单条记录从「算完」到「能被查到」的延迟变长;
反之延迟低但写得更碎。Langfuse 把它们做成环境变量,让你按自己的流量与时效要求自由去拧,无需改代码。
把一条合并好的记录在写入器里的旅程拆开,正好五步:
addToQueue(table, record) 把记录连同 createdAt 时间戳推进该表队列尾部。
记录在内存队列里排队。期间系统还会记录它等了多久(wait_time 指标),用来观测积压。
本表攒到 batchSize(按量)或定时器到点(按时),splice 取出最多一批。
writeToClickhouse 把整批以 JSONEachRow 一次 INSERT;带 backOff 退避重试。
成功则这批出队完成;可重试错误退避再试,超长批拆小重排队——尽量不连累整批。
ClickhouseWriter 是单例(私有构造 + getInstance):整个 worker 进程只有一个写入器、一组队列。这很关键——
只有汇到一处,才能把来自无数任务的零散记录真正攒成大批;如果每个任务各写各的,批就永远攒不大。写入时还带退避重试(maxAttempts),
遇到可重试错误就指数退避再试,遇到「字符串过长」这类错误则把批拆小重试——既追求大批量,又不被个别异常记录拖垮整批。这种「整批共进退、坏的单独拎出来」的策略,和第 12 课摄取 API 的「部分成功」一脉相承:系统在每一层都尽量做到容错而不牵连,让一两条问题数据不至于阻塞整条管道。
一个进程一个 writer,把所有任务的记录汇聚成大批;按表分队列,让 traces/observations/scores 各自满批、并行刷,互不阻塞。
一次 INSERT 写整批(JSONEachRow),失败按 maxAttempts 退避重试;超长批自动拆分。把「写得快」和「写得稳」一起拿下。
ClickhouseWriter 是单例,把高频单条记录攒成大批再一次 INSERT——因为第 8 课 ClickHouse 偏爱大批量、最怕逐条。addToQueue 喂入,flushAll 用 Promise.all 并行刷各表。batchSize(按量,保吞吐)② 每 writeInterval 定时器(按时,封顶延迟);谁先到算谁。shutdown 先清定时器、再 flushAll(true) 排空所有队列,进程退出不丢数据;写入带退避重试、超长批自动拆分。
Lesson 15's merge ended with clickhouseWriter.addToQueue(...) — the record goes into a queue, not straight to the DB. This lesson covers
what's behind that queue, the ClickhouseWriter: a singleton that gathers the flood of single records into a
handful of big batches, then INSERTs them into ClickHouse in one shot. Why batch at all? Because, as Lesson 8 said, ClickHouse
strongly prefers large batched writes and dreads row-by-row inserts. This layer is the buffer that translates
"high-frequency small writes" into "low-frequency large writes".
ClickhouseWriter as a shuttle bus. Passengers (records) arrive one by one, but the driver doesn't depart
for each one — the fuel cost (write overhead) would be absurd. He waits until the bus is full (hits batchSize), or
the departure time arrives (the writeInterval timer fires), then takes the whole busload across in one trip. Whichever condition comes
first wins: at rush hour the bus fills fast and leaves on "full"; in a lull, few passengers, so it leaves on "time" so the last few don't wait forever.
And each route (each table) has its own bus: one for traces, one for observations, one for scores, each loading and departing
independently, never holding up the others.
Inside, ClickhouseWriter keeps an in-memory queue per table (traces / observations / scores / …). Lesson 15's
addToQueue pushes the merged record into the matching queue; records wait in line to depart. Departure (flush) is triggered by
two conditions, whichever comes first:
addToQueue feeds it, hitting batchSize flushes that table immediately, or the timer fires flushAll every writeInterval. flush sends the whole batch in one INSERT (JSONEachRow). Source: worker/src/services/ClickhouseWriter/index.ts:34-135, 550-600.ClickhouseWriter; values illustrative): events enter a per-table memory buffer; a full batchSize triggers a flush by size, otherwise every writeInterval triggers a flush by time, whichever comes first; at shutdown flushAll(true) drains everything losslessly. Batched INSERTs buy ClickHouse's high write throughput.// Trigger ①: on enqueue, flush this table once it has a full batch addToQueue(tableName, data) { this.queue[tableName].push({ createdAt: Date.now(), data }); if (this.queue[tableName].length >= this.batchSize) this.flush(tableName); // depart when full } // Trigger ②: a timer, flush all tables every writeInterval ms setInterval(() => this.flushAll(), this.writeInterval); // depart on time // flush: take a batch from the queue, bulk INSERT (with backoff retry maxAttempts) const items = entityQueue.splice(0, fullQueue ? all : this.batchSize); await backOff(() => writeToClickhouse({ table, records }), { numOfAttempts });
Why does ClickHouse so dread "row-by-row writes"? Back to Lesson 8's storage mechanics: a MergeTree drops a new data part on disk for
every INSERT, then relies on background merges to combine small parts into large ones. INSERT each record separately and you
instantly spawn a flood of tiny parts — exploding disk metadata, background merges that never catch up, slowed queries (and even ClickHouse's
"too many parts" guard rejecting writes). Batch ten thousand records into one INSERT and you create just one big part — easy to write,
merge, and query. So this layer isn't an optional optimization; it's a precondition for a columnar store like ClickHouse to withstand
high-frequency ingestion — without this buffer, even the best table design gets crushed by a deluge of small writes.
"Full" and "on time" each serve a different load, and both are needed:
batchSize for throughput, lulls on writeInterval to cap latency; shutdown first clearInterval then flushAll(true) writes out everything left in memory. flushAll fans out across tables with Promise.all.| trigger | condition | what it solves |
|---|---|---|
| ① by size (batchSize) | a table's queue length ≥ batchSize | at high throughput, write full batches, maxing ClickHouse write efficiency |
| ② by time (writeInterval) | the timer fires flushAll every writeInterval ms | at low throughput, caps latency so data doesn't linger in memory indefinitely |
| ③ shutdown drain | shutdown() calls flushAll(true) | persist all buffers before the process exits, avoiding data loss |
The two parameters batchSize and writeInterval are really a pair of throughput / latency knobs: a bigger
batchSize and longer interval mean larger batches and cheaper writes, but a longer delay from "computed" to "queryable" for each record; the reverse means
lower latency but more fragmented writes. Langfuse makes them environment variables so you can tune them to your traffic and freshness
needs without touching code.
Unpack a merged record's journey through the writer, and it's exactly five steps:
addToQueue(table, record) pushes the record, with a createdAt timestamp, onto that table's queue tail.
The record waits in the in-memory queue. Meanwhile the system records how long it waited (a wait_time metric) to observe backlog.
This table hits batchSize (by size) or the timer fires (by time); splice takes up to a batch.
writeToClickhouse INSERTs the whole batch as JSONEachRow in one go; with backOff retry.
On success the batch is done; retryable errors back off and retry, oversized batches split and requeue — sparing the rest of the batch.
ClickhouseWriter is a singleton (private constructor + getInstance): one writer, one set of queues per worker
process. That's crucial — only by converging to one place can scattered records from countless jobs actually batch up large; if each job
wrote on its own, batches would never grow. Writes carry backoff retry (maxAttempts): retryable errors back off and retry,
"string too long" errors split the batch and retry — chasing big batches without letting one bad record drag down the whole batch. This
"the batch rises and falls together, the bad ones get pulled out alone" strategy echoes Lesson 12's ingestion-API "partial success": at
every layer the system tries to be fault-tolerant without collateral damage, so a problem record or two won't block the whole pipeline.
One writer per process gathers all jobs' records into big batches; per-table queues let traces/observations/scores each fill and flush in parallel, never blocking each other.
One INSERT per whole batch (JSONEachRow); failures retry with maxAttempts backoff; oversized batches auto-split. "Fast" and "stable" together.
ClickhouseWriter is a singleton that gathers high-frequency single records into big batches for one INSERT — because Lesson 8's ClickHouse prefers large batches and dreads row-by-row.addToQueue feeds it, flushAll fans out across tables with Promise.all.batchSize (by size, for throughput) ② every writeInterval timer (by time, capping latency); whichever first.shutdown clears the timer then flushAll(true) drains all queues so nothing is lost on exit; writes carry backoff retry, oversized batches auto-split.