Lesson 12 had the API "enqueue" and vanish; Lesson 13 explained what "event" it enqueues. The star of this lesson is the queue
itself — a conveyor belt built on Redis + BullMQ. It's the handoff point between Lesson 5's fast lane and slow lane:
web drops the work and leaves, while the worker pulls it off the other end and processes at its own pace. Three questions to answer: what's actually
in the queue (spoiler: not the event itself, but a "claim ticket"), why split into a primary / secondary queue, and how
sharding survives spikes under high concurrency.
🛄 Analogy
Think of the queue as an airport baggage belt. The check-in desk (API) doesn't pile your suitcase on the counter — it sends the case
to the back room (S3) and puts only a baggage tag (a pointer: the case's id in the back room) on the belt. A handler
(worker) takes the tag off the belt, fetches the case by its id, then processes it. The beauty: the belt (Redis memory) only carries
featherweight tags, fast and cheap; the bulky cases (event bodies, possibly large) rest quietly in the back. If handlers fall behind,
tags simply queue on the belt and not a single case is lost — because the back room is the goods' "real ledger".
The queue holds a "claim ticket", not the event body
Here's the most-misunderstood point of the whole ingestion path: the Redis queue does not store the event's full content. Before
enqueuing, Lesson 12's processEventBatch first writes each event to S3 named <eventId>.json (Lesson 13's
event id becomes the filename); the queue holds only a lightweight job whose key field is a fileKey — a pointer to that S3
file. The worker takes the job, then reads the event back from S3 by fileKey.
Pointer and body, separated: the event body is written to S3 (<eventId>.json); the queue holds only a lightweight job carrying fileKey. The worker reads the body back from S3 by fileKey. Source: processEventBatch.ts:340-398.
The queue absorbs spikes, the watermark warns (queue contract in packages/shared/src/server/queues.ts; values illustrative): web pushes events into Redis and returns fast; workers drain the other end. Queue depth is the backlog; crossing the watermark means add workers; consumer lag is “how stale the queryable data is”. High-throughput projects use a secondary queue physically isolated from primary, so one tenant's flood can't starve others.
The payload is a tiny { type, eventBodyId, fileKey, authCheck } job of a few dozen bytes. Redis memory is precious; the smaller the job, the faster the enqueue and the better it holds up under backlog.
📦 S3 holds the body
The raw event (input/output can be large) sits in S3. It's also the historical event sourceLesson 15's merge reads back, and the basis for rebuilding should a job ever be lost.
One more detail hides in the enqueue: queue.add(…, { delay }) attaches a small delay (about 5s by default). Why be
deliberately "a beat slow"? Because the create and update for one entity may arrive back to back; if a worker processes the create
before the update lands in S3, duplicates or out-of-order writes can result. The delay lets a batch of related events "settle" first, so the worker
takes them together, naturally reducing duplicate processing. Better still, getDelay lengthens the delay around the UTC
23:45–00:15 "date boundary" — because Lesson 8's ordering key contains toDate(start_time), events straddling midnight, if
processed too eagerly, could split across two date partitions and duplicate. This unassuming delay is exactly what safeguards Lesson 8's
day-organized storage — a classic "trade a little latency for consistency".
Sharding: split one queue into N, survive the spike
A single Redis queue always has a ceiling. When a deployment enables Redis cluster (REDIS_CLUSTER_ENABLED=true), the ingestion queue is
sharded into several: ingestion-queue, ingestion-queue-1, ingestion-queue-2… the count set by
LANGFUSE_INGESTION_QUEUE_SHARD_COUNT. Which shard depends on the hash of the shard keyprojectId-eventBodyId:
SHA-256 produces a number, taken modulo the shard count to yield a shard index.
Shard without scattering an entity: the hash is over projectId-eventBodyId, so all events of one entity (create/update) necessarily land on the same shard — spreading load across Redis nodes while keeping a record's multiple reports ordered on one worker. Source: redis/ingestionQueue.ts:38-52, redis/sharding.ts:9.
Choosing projectId-eventBodyId as the shard key — rather than a random number — is deliberate: random sharding would scatter
one record's create and update across different shards, forcing cross-queue coordination at merge time, complex and prone to disorder. Using the entity
id as the shard key makes "same entity, same shard" hold naturally — the same idea as Lesson 8 leading the ordering key with
project_id and Lesson 13 using body.id as the merge key: keep related data physically together.
Primary vs secondary: don't let a noisy tenant block everyone
All projects sharing one primary queue has a hazard: one ultra-high-throughput project floods in a million events, packs the queue,
and everyone else's events just wait. Langfuse's answer is secondary-queue isolation: before processing each job, the worker decides —
should this project be redirected to the secondary queue? Two checks:
check
trigger
meaning
env allowlist
project is in LANGFUSE_SECONDARY_INGESTION_QUEUE_ENABLED_PROJECT_IDS
a known high-throughput heavy hitter, statically isolated to the secondary
S3 slowdown flag
hasS3SlowdownFlag(projectId) is true
this project just got an S3 SlowDown, dynamically redirected to relieve the primary
neither
—
stays on the primary, processed normally
On either hit, the worker adds the job as-is to SecondaryIngestionQueue and returns, freeing the primary to serve
others immediately. The secondary has its own shard count (LANGFUSE_INGESTION_SECONDARY_QUEUE_SHARD_COUNT) and its own workers, so spike and
steady-state are physically isolated, neither dragging the other. OTel ingestion has the same primary/secondary pair (Lesson 18).
1
worker takes the next job
Pulls { fileKey, eventBodyId, type, authCheck } off one shard queue.
2
decide secondary redirect
Hit the env allowlist or the S3 slowdown flag → re-enqueue to the secondary and finish; else continue.
3
read S3 by fileKey
Fetch the event body (plus this entity's prior events, for Lesson 15's merge).
4
hand to IngestionService
mergeAndWrite merges into one record (Lesson 15), then ClickhouseWriter batches the write (Lesson 17).
5
retry on failure
Jobs carry attempts: 6 + exponential backoff (from 5s); removeOnComplete on success, kept on failure for debugging.
🎯 Design trade-off
Why insist on queue + S3 — why not have the API write the DB directly? Because the combo buys both decoupling and
durability. Decoupling: web and worker each run at their own pace; if the worker dies or ClickHouse hiccups, events queue in Redis and
rest in S3, not one lost, digested once downstream recovers (Lesson 5's eventual consistency). Durability: S3 is the events' "real
ledger" — even if a queue job fails outright, the data can be replayed and rebuilt from S3. The cost is an extra hop (write S3 + enqueue + read S3)
and eventual-consistency latency, but in return you get a spike-absorbing, fault-tolerant, replayable ingestion pipeline — for a
system swallowing tens of billions of events a day, a bargain.
🎯 Key points
The queue is Redis + BullMQ, the handoff between web (fast lane) and worker (slow lane); the job carries only a pointer (fileKey etc.), the event body lives in S3.
S3 is the real ledger: bodies land in <eventId>.json, read back for Lesson 15's merge and able to rebuild a lost job — light queue, saved memory, replayable.
Sharding: under cluster the queue splits into ingestion-queue-N, shard chosen by SHA-256 modulo of projectId-eventBodyId — same entity always same shard, ordered and balanced.
Primary / secondary: redirect high-throughput projects to the secondary by env allowlist (static) or S3 slowdown flag (dynamic), physically isolating them so one noisy tenant can't block everyone.
Resilience: jobs use attempts: 6 (secondary queue: 5) + exponential backoff; removed on success, kept on failure. Queue + S3 = spike absorption + fault tolerance + replayability.
🧪 Self-test - think about the design
1. What's actually in the ingestion queue, and why not stuff the event body straight into Redis?
an encrypted copy of the event
only a lightweight job carrying fileKey (a pointer); the body is first written to S3 as .json, the worker reads it back by fileKey — a light queue saves Redis memory and speeds enqueue, while the body stays in S3 as the 'real ledger', rebuildable if a job is lost
the full event content, including possibly huge input/output
nothing; the worker queries the DB itself
Show answer & explanation click to expand
Answer: B. S3 holds the body, the queue passes a pointer — the most-misunderstood point of the path. Redis memory is precious, so smaller jobs are faster and survive backlog; S3 is both the historical event source the merge reads back (L15) and the replay basis on job failure. You get 'light queue + durable storage' at once.
2. Why is the shard key projectId-eventBodyId rather than a random assignment?
because projectId-eventBodyId is the shortest string
hashing the entity id guarantees 'same entity always on the same shard': a record's create/update stay in one queue, processed in order by one worker, no cross-queue coordination at merge; random scatter would split them, making merge complex and prone to disorder
to make every shard's load exactly equal
because random assignment is impossible to implement
Show answer & explanation click to expand
Answer: B. Consistent hashing (SHA-256 modulo) spreads load across Redis nodes, while using the entity id as key makes 'same entity, same shard' hold naturally — the same 'keep related data physically together' idea as L08 leading the ordering key with project_id and L13 using body.id as the merge key.
3. What problem does primary-vs-secondary isolation solve, and when does the worker redirect a job to the secondary?
the secondary only handles failed jobs
to make the secondary slower to save money
it prevents one ultra-high-throughput project from packing the primary and starving others; on hitting the env allowlist (statically isolating known heavy hitters) or the S3 slowdown flag (dynamic temporary redirect), the worker re-enqueues the job to the secondary and returns, freeing the primary to serve others
primary handles only traces, secondary only scores
Show answer & explanation click to expand
Answer: C. The secondary has its own shard count and workers, physically isolating spike from steady-state. Two checks: the env allowlist (LANGFUSE_SECONDARY_INGESTION_QUEUE_ENABLED_PROJECT_IDS) for static isolation, the S3 SlowDown flag for dynamic relief. A classic 'noisy neighbor' isolation safeguarding multi-tenant fairness.
💭 Open questions (no single right answer - just think or try)
'Queue + S3 as real ledger' buys spike absorption, fault tolerance and replayability, at the cost of an extra hop and eventual-consistency latency. Designing a high-concurrency write system, how would you weigh 'write the DB directly' against 'queue buffer + object storage'? What signals would tip you toward the latter?