🪢 Langfuse 图解教程Langfuse Visual Guide 第三部分 · 摄取链路Part 3 · The Ingestion Path 14 / 55
第三部分 · 摄取链路Part 3 · The Ingestion Path

摄取队列The ingestion queue

🧭 读前Before you start
先读 第12课 摄取 API 与 第7课 里 Redis 的角色,本课讲摄取队列如何削峰。
Read Lesson 12 (ingestion API) and Redis's role in Lesson 7; this lesson covers how the ingestion queue absorbs spikes.

第 12 课 API「入队」后就闪人了,第 13 课讲清了入的是什么「事件」。这一课的主角,就是那条队列本身——Langfuse 用 Redis + BullMQ 搭起来的传送带。它是第 5 课「快路 / 慢路」之间的交接点:web 把活儿往上一放就走人,worker 在另一头不紧不慢地取走慢慢做。 这一课要回答三个问题:队列里到底放了什么(剧透:不是事件本体,而是一张「取件票」)、为什么要分主队列 / 次队列、以及高并发下怎么分片扛住洪峰。

🛄 生活类比
把队列想成机场的行李传送带。值机柜台(API)不会把你的行李箱直接堆在柜台上——它把箱子送进后仓(S3),只在传送带上放一张行李条(指针:箱子在后仓的编号)。 分拣员(worker)从传送带上取下行李条,照着编号去后仓取箱子,再处理。这样设计的妙处:传送带(Redis 内存)只跑轻飘飘的纸条,又快又省; 真正笨重的箱子(事件本体,可能很大)安安静静躺在后仓。万一分拣员忙不过来,纸条就在带上排队等着,箱子一件都不会丢——因为后仓才是货物的「真账本」。

队列里放的是「取件票」,不是事件本体

这是整条摄取链路最容易误解的一点:Redis 队列里并不存事件的完整内容。第 12 课 processEventBatch 在入队前,先把每个事件按 <eventId>.json 的名字写进 S3(第 13 课的事件 id 在这里当文件名);队列里只放一个轻量任务,载荷里关键的就一个 fileKey—— 指向 S3 里那个文件。worker 取到任务,再照着 fileKey 去 S3 把事件读回来。

交接:S3 存本体,队列只递指针 webprocessEventBatch(第 12 课) S3 事件日志<eventId>.json(本体,可能很大) Redis 队列(BullMQ)任务载荷:fileKey · eventBodyId · type worker取任务 → 读 S3→ IngestionService ① 写本体 ② 放指针 ③ 取指针 ④ 按 fileKey 读回 队列轻 → Redis 内存省、入队快;本体重 → 落 S3 当「真账本」,任务丢了也能从 S3 重建
指针与本体分离:事件本体写入 S3(<eventId>.json),队列里只放含 fileKey 的轻量任务。worker 取任务后照 fileKey 回 S3 读本体。源码:processEventBatch.ts:340-398(queue.add(IngestionJob, {payload:{data:{type, eventBodyId, fileKey,…}, authCheck}}}))。
示例:摄取队列的水位与消费滞后 web 入队(快) pending events ≈ 18k 水位线/watermark worker 消费 primary 队列 大多数项目共用 lag 偏高消费滞后(s) secondary 队列 高吞吐项目隔离到此 大客户的洪峰不拖垮别人 primary/secondary 物理隔离
队列削峰,水位预警(队列契约见 packages/shared/src/server/queues.ts;值为示例):web 把事件飞快塞进 Redis 就返回,worker 在另一头慢慢取。队列深度就是积压,水位线越过就该加 worker;消费滞后是「现在查得到多旧的数据」。高吞吐项目走 secondary 队列,和 primary 物理隔离,互不拖累。
packages/shared/src/server/ingestion/processEventBatch.ts入队
const shardingKey = `${projectId}-${eventData.eventBodyId}`;   // 分片键(见下)
const queue = IngestionQueue.getInstance({ shardingKey });

await queue.add(QueueJobs.IngestionJob, {
  id: randomUUID(), timestamp: new Date(),
  payload: {
    data: { type, eventBodyId, fileKey: eventData.key, /* 指向 S3 */ },
    authCheck,                          // 把租户身份也带上(第 10 课)
  },
}, { delay });

📨 队列只放指针

载荷是 { type, eventBodyId, fileKey, authCheck } 这种几十字节的小任务。Redis 内存金贵,任务越小,入队越快、积压时也撑得住。

📦 S3 存本体

事件原文(input/output 可能很大)躺在 S3。它还是第 15 课「合并」要回读的历史事件源,更是任务万一丢失时的可重建依据。

入队时还藏着一个细节:queue.add(…, { delay }) 给任务挂了一小段延迟(默认约 5 秒)。为什么要故意「慢一拍」?因为同一实体的 create 和 update 可能先后脚到达,若 worker 抢在 update 落 S3 之前就处理 create,容易产生重复或乱序写入。延迟让一批相关事件先「落定」,worker 再统一取走,天然减少重复处理。 更妙的是 getDelay 在每天 UTC 的 23:45–00:15 这段「日界线」会把延迟拉长——因为第 8 课的排序键里有 toDate(start_time),跨午夜的事件若处理太急, 可能被分到相邻两天的分区里造成重复。这个不起眼的延迟,正是为第 8 课那套按天组织的存储「保驾护航」,是工程上「用一点延迟换一致性」的典型权衡。

分片:把一条队列摊成 N 条,扛住洪峰

单条 Redis 队列总有上限。当部署开启 Redis 集群(REDIS_CLUSTER_ENABLED=true)时,摄取队列会分片成多条: ingestion-queue、ingestion-queue-1、ingestion-queue-2……数量由 LANGFUSE_INGESTION_QUEUE_SHARD_COUNT 决定。 分到哪条,取决于分片键 projectId-eventBodyId 的哈希:用 SHA-256 算出一个数,对分片数取模,得到 shard 下标。

一致性哈希分片:同一实体永远落同一 shard 分片键projectId-eventBodyId SHA-256 % NgetShardIndex() ingestion-queue(shard 0) ingestion-queue-1 ingestion-queue-2 …直到 SHARD_COUNT-1 同一 projectId-eventBodyId → 同一哈希 → 同一 shard:create 和 update 永远在一条队列里、有序处理
分片但不打乱实体:哈希的是 projectId-eventBodyId,所以同一实体的所有事件(create/update)必然落到同一条 shard,既把负载摊到多个 Redis 节点,又保证一条记录的多次上报被同一个 worker 有序处理。源码:redis/ingestionQueue.ts:38-52、redis/sharding.ts:9。

注意分片键选 projectId-eventBodyId 而不是随机数,是有讲究的:随机分会把同一条记录的 create 和 update 打散到不同 shard, 合并时就得跨队列协调,既复杂又可能乱序。用实体 id 当分片键,「同一实体同一 shard」天然成立——这和第 8 课用 project_id 领头排序键、 第 13 课用 body.id 当合并键,是同一种「让相关数据物理上聚在一起」的思路。

主队列 vs 次队列:别让吵闹的租户堵住所有人

所有项目共用一条主队列,会有个隐患:某个超高吞吐的项目瞬间灌进百万事件,把队列塞满,其他项目的事件只能干等。Langfuse 的解法是次队列隔离: worker 在处理每个任务前先判断——这个 project 该不该改道去次队列?判断有两条:

项目 A 项目 B 项目 C 高吞吐 改道判断 ① env 白名单(静态) ② S3 SlowDown(动态) 命中 → 次队列 否则 → 主队列 主队列 常规分片 · 给 A、B 主 worker 次队列 独立分片 · 给 C 次 worker 物理隔离:C 改道后,主队列立刻腾手处理 A、B——吵闹租户堵不住别人
判断触发条件含义
env 白名单project 在 LANGFUSE_SECONDARY_INGESTION_QUEUE_ENABLED_PROJECT_IDS 里已知的高吞吐大户,静态隔离到次队列
S3 限流标志hasS3SlowdownFlag(projectId) 为真该项目刚把 S3 写出 SlowDown,动态临时改道,给主队列减压
都不满足—留在主队列正常处理

命中任一条,worker 就把这条任务原样 add 进 SecondaryIngestionQueue 然后 return,主队列立刻腾出手处理别人的。 次队列有自己独立的分片数(LANGFUSE_INGESTION_SECONDARY_QUEUE_SHARD_COUNT)和独立的 worker,洪峰与日常彻底物理隔离,互不拖累。OTel 摄取也有同样的主 / 次一对(第 18 课)。

1

worker 取下一个任务

从某条 shard 队列拿到 { fileKey, eventBodyId, type, authCheck }。

2

判断是否改道次队列

命中 env 白名单或 S3 限流标志 → 重新入次队列并结束;否则继续。

3

照 fileKey 读 S3

取回事件本体(以及该实体此前的历史事件,供第 15 课合并)。

4

交给 IngestionService

mergeAndWrite 合并成一条记录(第 15 课),再交 ClickhouseWriter 攒批写库(第 17 课)。

5

失败则重试

任务配 attempts: 6 + 指数退避(5s 起);成功即 removeOnComplete,失败保留便于排查。

🎯 设计取舍
为什么非要队列 + S3 这一套,不能 API 直接写库? 因为这套组合同时买到了解耦和持久两样东西。解耦:web 和 worker 各按自己的节奏跑, worker 挂了、ClickHouse 抖了,事件就在队列里排队、在 S3 里躺着,一条不丢,等下游恢复再慢慢消化(第 5 课最终一致)。持久:S3 是事件的「真账本」, 即便某个队列任务彻底失败,数据也能从 S3 重放重建。代价是多了一跳(写 S3 + 入队 + 读 S3)和最终一致的延迟,但换来的是一个削峰、容错、可重放的摄取管道—— 对一个每天吞百亿事件的系统,这笔买卖太划算了。
🎯 本课要点
  • 队列用 Redis + BullMQ,是 web(快路)与 worker(慢路)的交接点;任务载荷只放指针(fileKey 等),事件本体在 S3。
  • S3 是真账本:本体落 <eventId>.json,既供第 15 课合并回读,也让任务丢失时能重建——队列轻、内存省、可重放。
  • 分片:集群下队列拆成 ingestion-queue-N,按 projectId-eventBodyId 的 SHA-256 取模定 shard——同一实体永远同一 shard,有序又均衡。
  • 主 / 次队列:按 env 白名单(静态)或 S3 限流标志(动态)把高吞吐项目改道次队列,物理隔离,避免一个吵闹租户堵住所有人。
  • 韧性:任务 attempts: 6(次级队列为 5)+ 指数退避;成功即删、失败保留。队列 + S3 = 削峰 + 容错 + 可重放。

🧪 自测 · 想一想为什么这么设计

1. 摄取队列里到底放了什么?为什么不直接把事件本体塞进 Redis?
  1. 放一份加密后的事件副本
  2. 只放含 fileKey 的轻量任务(指针);事件本体先写进 S3 的 .json,worker 再照 fileKey 回 S3 读回——队列轻则 Redis 内存省、入队快,本体留在 S3 当“真账本”,任务丢了也能重建
  3. 放事件的完整内容,包括可能很大的 input/output
  4. 什么都不放,worker 自己去数据库查
看答案与解析 点击展开
答案:B。S3 存本体、队列递指针是这条链路最易误解的点。Redis 内存金贵,任务越小越快越扛积压;S3 既是合并要回读的历史事件源(第 15 课),也是任务失败时的可重放依据。这就买到了“轻队列 + 持久存储”两样好处。
2. 分片键为什么选 projectId-eventBodyId,而不是随机分配?
  1. 因为 projectId-eventBodyId 最短
  2. 对实体 id 哈希取模能保证“同一实体永远落同一 shard”:一条记录的 create/update 始终在同一队列、被同一 worker 有序处理,合并无需跨队列协调;随机分会把它们打散、合并时复杂又可能乱序
  3. 为了让每个 shard 负载完全相等
  4. 因为随机分配根本无法实现
看答案与解析 点击展开
答案:B。用一致性哈希(SHA-256 取模)把负载摊到多个 Redis 节点,同时用实体 id 当键,让“同一实体同一 shard”天然成立——这与第 8 课用 project_id 领头排序键、第 13 课用 body.id 当合并键,是同一种“让相关数据物理聚在一起”的思路。
3. 主队列 vs 次队列的隔离机制解决了什么问题?worker 何时把任务改道次队列?
  1. 次队列只处理失败的任务
  2. 让次队列处理得更慢以省钱
  3. 防止某个超高吞吐项目塞满主队列、饿死其他项目;命中 env 白名单(静态隔离已知大户)或 S3 限流标志(动态临时改道)就把该任务原样入次队列并 return,主队列立刻服务别人
  4. 主队列只处理 trace,次队列只处理 score
看答案与解析 点击展开
答案:C。次队列有独立的分片数和独立的 worker,把洪峰与日常物理隔离。两条判断:env 白名单(LANGFUSE_SECONDARY_INGESTION_QUEUE_ENABLED_PROJECT_IDS)做静态隔离,S3 SlowDown 标志做动态减压。这是典型的“吵闹邻居”隔离,保障多租户公平。
💭 发散思考(没有标准答案,动手或动脑想想)
  • “队列 + S3 当真账本”换来了削峰、容错、可重放,代价是多一跳和最终一致的延迟。如果让你给一个高并发写入系统做选型,你会在“直接写库”和“队列缓冲 + 对象存储”之间怎么权衡?哪些信号会让你倒向后者?

Lesson 12 had the API "enqueue" and vanish; Lesson 13 explained what "event" it enqueues. The star of this lesson is the queue itself — a conveyor belt built on Redis + BullMQ. It's the handoff point between Lesson 5's fast lane and slow lane: web drops the work and leaves, while the worker pulls it off the other end and processes at its own pace. Three questions to answer: what's actually in the queue (spoiler: not the event itself, but a "claim ticket"), why split into a primary / secondary queue, and how sharding survives spikes under high concurrency.

🛄 Analogy
Think of the queue as an airport baggage belt. The check-in desk (API) doesn't pile your suitcase on the counter — it sends the case to the back room (S3) and puts only a baggage tag (a pointer: the case's id in the back room) on the belt. A handler (worker) takes the tag off the belt, fetches the case by its id, then processes it. The beauty: the belt (Redis memory) only carries featherweight tags, fast and cheap; the bulky cases (event bodies, possibly large) rest quietly in the back. If handlers fall behind, tags simply queue on the belt and not a single case is lost — because the back room is the goods' "real ledger".

The queue holds a "claim ticket", not the event body

Here's the most-misunderstood point of the whole ingestion path: the Redis queue does not store the event's full content. Before enqueuing, Lesson 12's processEventBatch first writes each event to S3 named <eventId>.json (Lesson 13's event id becomes the filename); the queue holds only a lightweight job whose key field is a fileKey — a pointer to that S3 file. The worker takes the job, then reads the event back from S3 by fileKey.

Handoff: S3 stores the body, the queue passes a pointer webprocessEventBatch(Lesson 12) S3 event log<eventId>.json (body, may be large) Redis queue (BullMQ)job: fileKey · eventBodyId · type workertake job → read S3→ IngestionService ① write body ② put pointer ③ take pointer ④ read back by fileKey light queue → Redis memory saved, fast enqueue; heavy body → S3 as "real ledger", rebuildable if a job is lost
Pointer and body, separated: the event body is written to S3 (<eventId>.json); the queue holds only a lightweight job carrying fileKey. The worker reads the body back from S3 by fileKey. Source: processEventBatch.ts:340-398.
Example: ingestion queue depth and consumer lag web enqueue (fast) pending events ≈ 18k watermark worker consume primary queue shared by most projects lag highconsumer lag (s) secondary queue high-throughput projects isolated here a big tenant's spike won't drag others down primary/secondary physical isolation
The queue absorbs spikes, the watermark warns (queue contract in packages/shared/src/server/queues.ts; values illustrative): web pushes events into Redis and returns fast; workers drain the other end. Queue depth is the backlog; crossing the watermark means add workers; consumer lag is “how stale the queryable data is”. High-throughput projects use a secondary queue physically isolated from primary, so one tenant's flood can't starve others.
packages/shared/src/server/ingestion/processEventBatch.tsenqueue
const shardingKey = `${projectId}-${eventData.eventBodyId}`;   // shard key (below)
const queue = IngestionQueue.getInstance({ shardingKey });

await queue.add(QueueJobs.IngestionJob, {
  id: randomUUID(), timestamp: new Date(),
  payload: {
    data: { type, eventBodyId, fileKey: eventData.key, /* points to S3 */ },
    authCheck,                          // carry the tenant identity too (Lesson 10)
  },
}, { delay });

📨 the queue holds a pointer

The payload is a tiny { type, eventBodyId, fileKey, authCheck } job of a few dozen bytes. Redis memory is precious; the smaller the job, the faster the enqueue and the better it holds up under backlog.

📦 S3 holds the body

The raw event (input/output can be large) sits in S3. It's also the historical event source Lesson 15's merge reads back, and the basis for rebuilding should a job ever be lost.

One more detail hides in the enqueue: queue.add(…, { delay }) attaches a small delay (about 5s by default). Why be deliberately "a beat slow"? Because the create and update for one entity may arrive back to back; if a worker processes the create before the update lands in S3, duplicates or out-of-order writes can result. The delay lets a batch of related events "settle" first, so the worker takes them together, naturally reducing duplicate processing. Better still, getDelay lengthens the delay around the UTC 23:45–00:15 "date boundary" — because Lesson 8's ordering key contains toDate(start_time), events straddling midnight, if processed too eagerly, could split across two date partitions and duplicate. This unassuming delay is exactly what safeguards Lesson 8's day-organized storage — a classic "trade a little latency for consistency".

Sharding: split one queue into N, survive the spike

A single Redis queue always has a ceiling. When a deployment enables Redis cluster (REDIS_CLUSTER_ENABLED=true), the ingestion queue is sharded into several: ingestion-queue, ingestion-queue-1, ingestion-queue-2… the count set by LANGFUSE_INGESTION_QUEUE_SHARD_COUNT. Which shard depends on the hash of the shard key projectId-eventBodyId: SHA-256 produces a number, taken modulo the shard count to yield a shard index.

Consistent-hash sharding: one entity always lands on one shard shard keyprojectId-eventBodyId SHA-256 % NgetShardIndex() ingestion-queue (shard 0) ingestion-queue-1 ingestion-queue-2 …up to SHARD_COUNT-1 same projectId-eventBodyId → same hash → same shard: create & update always in one queue, processed in order
Shard without scattering an entity: the hash is over projectId-eventBodyId, so all events of one entity (create/update) necessarily land on the same shard — spreading load across Redis nodes while keeping a record's multiple reports ordered on one worker. Source: redis/ingestionQueue.ts:38-52, redis/sharding.ts:9.

Choosing projectId-eventBodyId as the shard key — rather than a random number — is deliberate: random sharding would scatter one record's create and update across different shards, forcing cross-queue coordination at merge time, complex and prone to disorder. Using the entity id as the shard key makes "same entity, same shard" hold naturally — the same idea as Lesson 8 leading the ordering key with project_id and Lesson 13 using body.id as the merge key: keep related data physically together.

Primary vs secondary: don't let a noisy tenant block everyone

All projects sharing one primary queue has a hazard: one ultra-high-throughput project floods in a million events, packs the queue, and everyone else's events just wait. Langfuse's answer is secondary-queue isolation: before processing each job, the worker decides — should this project be redirected to the secondary queue? Two checks:

project A project B project C high-throughput redirect check (1) env allowlist (static) (2) S3 SlowDown (dynamic) match → secondary else → primary primary queue normal shards · for A, B primary worker secondary queue own shards · for C secondary worker physical isolation: once C is redirected, the primary instantly serves A, B — a noisy tenant can't block others
checktriggermeaning
env allowlistproject is in LANGFUSE_SECONDARY_INGESTION_QUEUE_ENABLED_PROJECT_IDSa known high-throughput heavy hitter, statically isolated to the secondary
S3 slowdown flaghasS3SlowdownFlag(projectId) is truethis project just got an S3 SlowDown, dynamically redirected to relieve the primary
neither—stays on the primary, processed normally

On either hit, the worker adds the job as-is to SecondaryIngestionQueue and returns, freeing the primary to serve others immediately. The secondary has its own shard count (LANGFUSE_INGESTION_SECONDARY_QUEUE_SHARD_COUNT) and its own workers, so spike and steady-state are physically isolated, neither dragging the other. OTel ingestion has the same primary/secondary pair (Lesson 18).

1

worker takes the next job

Pulls { fileKey, eventBodyId, type, authCheck } off one shard queue.

2

decide secondary redirect

Hit the env allowlist or the S3 slowdown flag → re-enqueue to the secondary and finish; else continue.

3

read S3 by fileKey

Fetch the event body (plus this entity's prior events, for Lesson 15's merge).

4

hand to IngestionService

mergeAndWrite merges into one record (Lesson 15), then ClickhouseWriter batches the write (Lesson 17).

5

retry on failure

Jobs carry attempts: 6 + exponential backoff (from 5s); removeOnComplete on success, kept on failure for debugging.

🎯 Design trade-off
Why insist on queue + S3 — why not have the API write the DB directly? Because the combo buys both decoupling and durability. Decoupling: web and worker each run at their own pace; if the worker dies or ClickHouse hiccups, events queue in Redis and rest in S3, not one lost, digested once downstream recovers (Lesson 5's eventual consistency). Durability: S3 is the events' "real ledger" — even if a queue job fails outright, the data can be replayed and rebuilt from S3. The cost is an extra hop (write S3 + enqueue + read S3) and eventual-consistency latency, but in return you get a spike-absorbing, fault-tolerant, replayable ingestion pipeline — for a system swallowing tens of billions of events a day, a bargain.
🎯 Key points
  • The queue is Redis + BullMQ, the handoff between web (fast lane) and worker (slow lane); the job carries only a pointer (fileKey etc.), the event body lives in S3.
  • S3 is the real ledger: bodies land in <eventId>.json, read back for Lesson 15's merge and able to rebuild a lost job — light queue, saved memory, replayable.
  • Sharding: under cluster the queue splits into ingestion-queue-N, shard chosen by SHA-256 modulo of projectId-eventBodyId — same entity always same shard, ordered and balanced.
  • Primary / secondary: redirect high-throughput projects to the secondary by env allowlist (static) or S3 slowdown flag (dynamic), physically isolating them so one noisy tenant can't block everyone.
  • Resilience: jobs use attempts: 6 (secondary queue: 5) + exponential backoff; removed on success, kept on failure. Queue + S3 = spike absorption + fault tolerance + replayability.

🧪 Self-test - think about the design

1. What's actually in the ingestion queue, and why not stuff the event body straight into Redis?
  1. an encrypted copy of the event
  2. only a lightweight job carrying fileKey (a pointer); the body is first written to S3 as .json, the worker reads it back by fileKey — a light queue saves Redis memory and speeds enqueue, while the body stays in S3 as the 'real ledger', rebuildable if a job is lost
  3. the full event content, including possibly huge input/output
  4. nothing; the worker queries the DB itself
Show answer & explanation click to expand
Answer: B. S3 holds the body, the queue passes a pointer — the most-misunderstood point of the path. Redis memory is precious, so smaller jobs are faster and survive backlog; S3 is both the historical event source the merge reads back (L15) and the replay basis on job failure. You get 'light queue + durable storage' at once.
2. Why is the shard key projectId-eventBodyId rather than a random assignment?
  1. because projectId-eventBodyId is the shortest string
  2. hashing the entity id guarantees 'same entity always on the same shard': a record's create/update stay in one queue, processed in order by one worker, no cross-queue coordination at merge; random scatter would split them, making merge complex and prone to disorder
  3. to make every shard's load exactly equal
  4. because random assignment is impossible to implement
Show answer & explanation click to expand
Answer: B. Consistent hashing (SHA-256 modulo) spreads load across Redis nodes, while using the entity id as key makes 'same entity, same shard' hold naturally — the same 'keep related data physically together' idea as L08 leading the ordering key with project_id and L13 using body.id as the merge key.
3. What problem does primary-vs-secondary isolation solve, and when does the worker redirect a job to the secondary?
  1. the secondary only handles failed jobs
  2. to make the secondary slower to save money
  3. it prevents one ultra-high-throughput project from packing the primary and starving others; on hitting the env allowlist (statically isolating known heavy hitters) or the S3 slowdown flag (dynamic temporary redirect), the worker re-enqueues the job to the secondary and returns, freeing the primary to serve others
  4. primary handles only traces, secondary only scores
Show answer & explanation click to expand
Answer: C. The secondary has its own shard count and workers, physically isolating spike from steady-state. Two checks: the env allowlist (LANGFUSE_SECONDARY_INGESTION_QUEUE_ENABLED_PROJECT_IDS) for static isolation, the S3 SlowDown flag for dynamic relief. A classic 'noisy neighbor' isolation safeguarding multi-tenant fairness.
💭 Open questions (no single right answer - just think or try)
  • 'Queue + S3 as real ledger' buys spike absorption, fault tolerance and replayability, at the cost of an extra hop and eventual-consistency latency. Designing a high-concurrency write system, how would you weigh 'write the DB directly' against 'queue buffer + object storage'? What signals would tip you toward the latter?