Part 1 is nearly done. You now know what the data looks like (trace/observation/score), where the code
lives (monorepo & narrow waist), and the design philosophy behind it (wide events). This lesson
strings them into one line: follow one concrete trace from the moment your app calls
langfuse.trace(...) all the way to it appearing in the UI list, being opened, even being auto-scored. It's a
trailer — each leg is tagged with the later lesson that covers it, so by the end you have the whole map's
route plan. Put differently, the first four lessons took the parts out one by one; this one puts them back and
fires the engine once, so you see the data actually flowing — with that dynamic panorama, digging into each part
lesson by lesson won't lose the forest for the trees.
🔌 Analogy
Think of a trace's life as shipping a parcel: at home you place the order and seal the box (create
trace/observation in the SDK); a courier picks it up, scans it into a bag (the public API accepts the events and
quickly enqueues); the parcel queues in a sorting center and is processed in the back (the worker
pulls from the queue, merges, computes "shipping cost" = cost); then it's shelved into the warehouse (written to
ClickHouse); finally you open your phone to track it (see the trace in the UI). The key: placing the order
is not the same as being shelved — pickup, sorting and shelving sit in between, so there's a brief processing
delay. That's the essence of Langfuse's "async" ingestion.
A trace's full journey: the top half is the write path (SDK→API→Redis→worker→ClickHouse, with S3 alongside), each leg tagged with its lesson (L12–19); the bottom half is the read path (ClickHouse→repository→tRPC→UI, L20–27). The two paths "meet" at the ClickHouse wide table: writes put data in, reads take it out.
The write path is one async handshake (per api/public/ingestion and packages/shared/src/server/queues.ts): SDK sends a batch → web validates and enqueues to Redis → returns 207 Multi-Status immediately (with per-event successes/errors, not waiting for persistence); only across the async boundary does the worker dequeue, merge and write ClickHouse+S3. So between "written" and "queryable" there is an eventual-consistency window — exactly why a just-sent trace is sometimes not visible yet.
Top half · write: SDK to ClickHouse
First the parcel's "shipped to shelved" journey. It's deliberately async — the API only accepts and enqueues
fast, leaving the heavy lifting to the worker:
web/src/pages/api/public/ingestion.ts authenticates, Zod-validates a batch, then quickly drops it into the Redis queue and returns — without waiting for processing.
IngestionService pulls events from the queue, fetches this record's prior events from S3 and merges (multiple updates to one observation fold into one), deciding the final shape.
Batch processed records and write them to ClickHouse traces/observations/scores in one go (ReplacingMergeTree handles "later write of same id wins").
In this path, S3 plays "the raw event manuscript": each incoming event lands in S3 first, and the worker re-reads S3
when merging. That allows replay on failure and lets "multiple updates to one observation" merge correctly — details
unpacked in L15–17. For now, remember the spine: accept → enqueue → merge → cost → store.
It's worth calling out why step 3, "merge", can't be skipped. The reason: an SDK often reports one observation
not all at once. For an LLM call, the SDK might first send an "observation-create" saying "I started, model is gpt-4o",
then after the call returns send an "observation-update" adding "output is X, used 1240 tokens". Those two events arrive
separately, and the worker must merge them into one final record. That's why it re-reads S3 for "this id's
prior events" — stitching the fragments that arrived at different times into one complete row. Grasp this and you see how Lesson 3's
"observations can arrive out of order via parentObservationId" is actually realized in ingestion: events may come in multiple
pieces, out of order, and the worker converges them into a consistent final state.
Out-of-order events → one final state: the same observation's create/update arrive in multiple pieces, possibly out of order; the worker fetches that id's history from S3, groups and merges by id (conflicts resolved by the later timestamp), converging into one final row written to ClickHouse. Details in Lessons 13·15.
Bottom half · read: from UI back to ClickHouse
Once stored, seeing it in your browser takes a different, more real-time path — you click, it queries:
The handler calls functions in packages/shared/src/server/repositories, translating your filters into ClickHouse SQL (with time windows, selecting only the needed columns).
The list reads only compact narrow fields (fast); only when you open a trace does it fetch the full input/output and build the observation tree.
Note that write and read converge on this one ClickHouse table: writes put the wide event in, reads pull it out
every which way. Lesson 2's "lists read narrow fields, detail reads big fields" is realized on exactly this read path.
Why separate "accept" from "process"
The cleverest step in this path is the API "accept and return". It splits the whole thing into a "fast lane" and a
"slow lane":
One clean cut: between app and API is the fast lane (returns in ms, almost never slow); after the API is the slow lane (enqueue → worker batches in the background). The queue is a reservoir: spikes queue up, and you add workers when overwhelmed.
Consider: a high-traffic LLM app may produce thousands of observations per second. If the API had to synchronously
merge, compute cost and write ClickHouse before returning, two bad things happen: one, every app call waits those
extra hundreds of ms, needlessly slowed by your monitoring tool; two, if ClickHouse hiccups or a spike maxes it out, the API
times out and even bounces errors back into your business. Cutting "accept" from "process" reduces the API to a
featherweight "validate + enqueue" endpoint — almost impossible to slow, almost impossible to drag your app down via
a downstream issue.
The queue is a reservoir: spikes queue up, the worker consumes in batches at its own pace, and you
add workers (scale out) when it can't keep up. Langfuse even splits the ingestion queue into primary/secondary
lanes (L14), isolating high-throughput big projects from ordinary ones so one project's spike can't drown everyone. The cost
is the bit of lag mentioned next — but keep Δ to seconds and users barely notice. This is why nearly every high-throughput telemetry
system (not just Langfuse) looks like this: lightweight accept + async process + batched store. Grasp this spine and
you can predict what each Part 3 lesson is solving.
The cost of async: eventual consistency
Stitch the halves together and one fact must be clear in your mind: from SDK send to UI visibility there's queue processing
in between, a few seconds of lag. This is eventual consistency.
What does "eventual consistency" mean in practice? It means that right after a call runs in your app, if you immediately switch to
the Langfuse UI and refresh, you might not see it yet — wait a second or two, refresh, and it appears. For almost all
observability use this is perfectly fine: you come after the fact to triage, view dashboards, evaluate — a few seconds
don't matter. But it flags two things: first, don't treat Langfuse as a strongly-consistent business database — it's a
telemetry system, not a "write-then-read-immediately" store; second, if you write automated tests to verify "is instrumentation
correct", remember to allow ingestion time to process, or you'll falsely conclude "the data never arrived". Understand
this boundary and the lag won't surprise you — you'll appreciate the "never drag the business" trade behind it.
Why tolerate that lag and go async? Because ingestion must absorb very high write throughput while
your app can't afford to wait. If the API synchronously merged, computed cost and wrote before returning, every LLM
call would be slowed by Langfuse and could fail on downstream hiccups. Async decouples "accept" from "process": the API always
returns in milliseconds and almost never drags your app; the worker can digest spikes in batches,
retryably, scaling by queue. The cost is that bit of processing lag — and Lesson 2's "preserve near-real-time debugging"
principle demands keeping Δ to seconds so observability stays "almost live". Trade a little lag for an ingestion layer that
never drags the business and survives spikes. This "lightweight accept + async process" trade recurs throughout Part 3 — it's
not one module's trick but the guiding idea across the whole ingestion path.
🎯 Key points · Part 1 wrap-up
A trace's life: SDK → public API (enqueue) → Redis → worker (merge·cost) → ClickHouse (write), then UI → tRPC → repository → ClickHouse (read).
Writes are async (seconds of lag, eventual consistency), reads are real-time; the two paths converge on the ClickHouse wide table.
S3 stores the raw event manuscript for merge and replay; the Redis queue decouples accept from process.
Every leg maps to a later lesson: writing is Part 3 (L12–19), reading is Part 4 (L20–27).
Part 1 ends here: you now hold the master map — data model + repo map + design philosophy + end-to-end path — and we dive in leg by leg next.
🧪 Self-test - think about the design
1. Why is Langfuse ingestion async (the API enqueues and returns, the worker processes in the background) rather than the API synchronously writing ClickHouse before returning?
So the API always returns in ms and barely drags your app, and the queue buffers: spikes queue up, the worker consumes in batches retryably, and you scale out when overwhelmed
To deliberately add latency to save money
Because the API cannot access the database
Because ClickHouse doesn't support synchronous writes
Show answer & explanation click to expand
Answer: A. If the API synchronously merged/costed/wrote, every LLM call is slowed and a downstream hiccup times out the API or bounces errors into your business. Async decouples accept from process: the API is featherweight and rarely slow; the worker digests spikes in batches, retryably, scalably. The cost is seconds of lag (eventual consistency).
2. Why does the worker re-read 'this id's prior events' from S3 and merge during processing?
Because ClickHouse loses data and must restore from S3
Because merging must happen in the frontend
Because S3 is faster than the queue
Because one observation is often reported in multiple, possibly out-of-order events (create then update), and the worker must merge these fragments into one consistent final record
Show answer & explanation click to expand
Answer: D. SDK reporting often isn't one-shot: a create ('I started') then an update ('output/usage') after the call returns. They arrive separately; the worker re-reads S3 to stitch the fragments into one row. This supports 'events arrive out of order, worker converges to final state' — the runtime realization of L03's parentObservationId design.
3. What practical caveat does 'eventual consistency' imply when using Langfuse?
Data will never appear
Just-instrumented data may take a second or two to appear in the UI; don't treat it as a strongly-consistent business DB, and when testing instrumentation, allow ingestion time to process
Traces are randomly lost
You must manually refresh the DB connection
Show answer & explanation click to expand
Answer: B. Async ingestion has seconds of processing lag — fine for after-the-fact triage/dashboards/eval, but it means 'written isn't instantly readable'. So don't treat it as a strongly-consistent store; when verifying instrumentation in tests, wait for ingestion or you'll falsely conclude 'no data arrived'.
💭 Open questions (no single right answer - just think or try)
Recap Part 1: in your own words, tell a colleague who's never used Langfuse the 'life of a trace' from SDK to UI. Which leg do you think is most overlooked yet most crucial?