Last lesson, data left your app as "events". This lesson is about where it lands. Many assume one database suffices,
but Langfuse uses four stores — Postgres, ClickHouse, Redis, S3 — each for its own job. Not
showing off: different data has wildly different access patterns, and the wrong store will crush the hot path.
Understand the division of labor and you understand the foundation of Langfuse's performance and scalability.
🔌 Analogy
Picture the four stores as a company's four "ways to keep things": Postgres is the filing cabinet —
important records (who's an employee, project config), needing to be accurate, editable, quick to look up but not
huge; ClickHouse is the giant warehouse — mountains of goods (hundreds of millions of events a day),
needing bulk stock-taking by category; Redis is the conveyor belt + staging table —
goods queue on the belt (queues), and frequently-used small items stay at hand (cache); S3 is off-site
storage — big, rarely-flipped originals (event manuscripts, uploaded images, large exports). Each excels at its own; cram
them into one and you lose.
The four stores' division of labor: Postgres=config/metadata (txn, consistent, small); ClickHouse=wide events (columnar, huge, fast aggregation); Redis=queue+cache (in-memory, fast, temporary); S3=event manuscript/media/exports (object store, big, cheap). web leans read, worker leans write, but both pick the store that fits the data.
Not "primary/replica", but division of labor (per prisma/schema.prisma, CH migrations, server/queues.ts, the media service): the x-axis is transactional↔analytical, the y-axis structured rows↔bulk objects. Postgres governs "how it's configured", ClickHouse "what happened", Redis absorbs spikes, S3 backstops the originals — each datum lands in the quadrant that fits its nature.
Why not just one
The core reason in one line: transactional (OLTP) and analytical (OLAP) workloads are opposite optimization targets,
and no single store nails both extremes.
Config/metadata (projects, users, API keys, prompt definitions) is classic OLTP: small, frequent precise single-row read/write, strong consistency (create a project, use it instantly), with foreign keys. Postgres's home turf.
Telemetry (trace/observation/score) is classic OLAP: enormous (hundreds of millions/day), write-mostly, queries are mostly time-window + dimensional aggregation ("cost by model over 7 days"). Columnar ClickHouse's home turf.
time-window scans + aggregation over a few columns of huge rows, columnar + compressed. Fits massive, append-write, dimensional telemetry. Bad at: frequent single-row edits + strong consistency.
Cram telemetry into Postgres and huge rows + frequent aggregation overwhelm it; cram config into ClickHouse and its weak
transactions, eventual consistency fail "instantly usable". So Langfuse isn't being greedy — it's letting each kind of
data live in the store that fits it best, the storage-layer landing point of Lesson 2's "design around columnar access" and
"denormalize carefully".
Two opposite access patterns: OLTP uses an index to hit exactly one row (read config, change settings); OLAP must scan a few columns of a big slice of rows and aggregate (cost by time/model). Row stores excel at the former, columnar at the latter — which is why config goes to Postgres and telemetry to ClickHouse.
What each store holds
Store
Holds
Client in the repo
Postgres
org/project/user, API keys, prompts, eval config, pricing… (control plane)
raw event manuscript (for merge/replay), multimodal media, bulk exports
.../server/s3/index.ts
One detail: S3 isn't just "big-file storage". In the ingestion path, each incoming event lands in S3 first, and the
worker re-reads S3 when merging (L05, L15) — here S3 plays the "event log / source of truth" role. The wide events in
ClickHouse are essentially a "materialized view" of the merged S3 manuscripts: if ClickHouse data ever needs rebuilding,
it can in principle be replayed from the raw S3 events. This "raw log + derived table" split is Lesson 2's "immutable /
append-oriented events" in action.
control planePostgres
who can use it, configured how: org/project/user, API keys, prompts, eval config, pricing. Few but critical changes, needs strong consistency.
data planeClickHouse
the actual telemetry flood: traces/observations/scores wide-event tables. Huge, append, dimensional aggregation.
transportRedis
the queue that makes ingestion async, plus a cache for hot config. In-memory, temporary, very fast.
object storeS3 / blob
the event source of truth, uploaded media, bulk exports. Big, rarely-flipped things live here — cheap and durable.
These four layers collaborate at runtime, not in isolation. Recall Lesson 5's path: on ingestion, events first enter the
Redis queue and originals land in S3; after merging the worker writes ClickHouse; and to
merge it must know "which prompt version this observation links to" and "what model pricing applies", so it reads config back from
Postgres. One ingestion, four stores each pulling their weight. Reads are similar: the UI list queries
ClickHouse, but "what's this project called, who has access" comes from Postgres, and hot config goes
through the Redis cache. So understanding Langfuse is essentially understanding how data flows among these four
stores — which is why Part 2 spends a whole lesson on the storage division of labor: it's the shared foundation
of every later path lesson.
🎯 Design trade-off
Four stores = four operational burdens, so why split this much?Lesson 2 has a principle: "treat cost and
operational simplicity as constraints" — each extra database means more backups, monitoring, upgrades, DR. Langfuse still keeps four
because each solves a hard problem a single store can't: without ClickHouse, huge aggregation queries are unusably
slow; without the Redis queue, ingestion can't go async to absorb spikes; without S3, big fields and media bloat the database and you
can't replay. In other words, these four aren't "add because we can" — each earned its existence. That's also the
test for "should we add another store": does the problem it solves justify its long-term operational cost?
One project: each side stores half
Grounding the abstraction in real tables: the same project lives in Postgres as an editable row (config, relations, strong consistency) and in ClickHouse as a wide-event table (append, columnar, later-write-wins). The two halves are stitched by the same project_id — the left side defines it, the right side partitions and sorts by it.
prisma/schema.prisma · clickhouse/migrations/unclustered/0001_traces.up.sqlcontrol plane vs data plane
// ── Control plane: in Postgres a project is an "editable row" (schema.prisma) ──modelProject {
id String @id @default(cuid())
orgId String @map("org_id")
name String
retentionDays Int? @map("retention_days") // config: strongly consistent, editable anytime
apiKeys ApiKey[] // relation: OLTP fetches by key
}
-- ── Data plane: in ClickHouse the same project's trace is a "wide-event row" (0001_traces.up.sql) ──CREATE TABLEtraces (
`id` String,
`timestamp` DateTime64(3),
`project_id` String, -- the same project_id
...
) ENGINE = ReplacingMergeTree(event_ts, is_deleted)
PARTITION BY toYYYYMM(timestamp) -- prune by monthORDER BY (project_id, toDate(timestamp), id); -- clustered within a project
🎯 Key points
Four stores, each its own job: Postgres (config/metadata·OLTP), ClickHouse (wide events·OLAP), Redis (queue+cache), S3 (event manuscript+media+exports).
Core reason: OLTP and OLAP are opposite targets — config needs precise single-row + strong consistency, telemetry needs massive scan + aggregation, so split stores.
S3 is the "source of truth": events land in S3 first; ClickHouse wide events are the merged derived table, replayable.
Control plane (Postgres) vs data plane (ClickHouse) is the macro version of Lesson 3's "domain shape vs physical storage".
Four stores aren't tech-stacking — each earned its existence, which is also the test for "should we add another store".
🧪 Self-test - think about the design
1. Why does Langfuse put config/metadata in Postgres and trace/observation/score in ClickHouse, rather than one database for all?
Because ClickHouse can't store strings
Purely to use more tech
Because Postgres has no indexes
Because they're opposite workloads: config is OLTP (small, precise single-row, strongly consistent), telemetry is OLAP (massive, time-window + dimensional aggregation); no single store nails both extremes
Show answer & explanation click to expand
Answer: D. Config needs precise single-row read/write and strong consistency (instantly usable) — row-store Postgres's turf; telemetry is massive, write-mostly, dimensionally aggregated — columnar ClickHouse's turf. Forcing one store loses both ways. This is 'choose storage by access pattern'.
2. What key role does S3 play in the ingestion path?
The events' 'source of truth / raw log': events land in S3 first, the worker re-reads S3 to merge; ClickHouse wide events are the merged derived table, in principle replayable from S3
Just stores uploaded images, unrelated to ingestion
Stores user passwords
Replaces Redis as the queue
Show answer & explanation click to expand
Answer: A. S3 isn't just a big-file store. Events land in S3 as an immutable raw log, from which the worker merges ClickHouse wide events (derived/materialized). This is L02's 'immutable/append-oriented events', and makes data replayable.
3. Langfuse uses four stores, yet L02 says 'treat operational simplicity as a constraint'. How to reconcile? What's the test for 'should we add another store'?
The frontend framework decides
You may only ever use one database
Each store must earn its existence — does the hard problem it solves justify its long-term ops cost (backup/monitoring/upgrade/DR)
More stores are always better
Show answer & explanation click to expand
Answer: C. The four aren't 'add because we can': ClickHouse solves huge aggregation, Redis solves async spike absorption, S3 solves big fields + replay — each solving what one store can't, thus earning its place. That's exactly the test for adding another store.
💭 Open questions (no single right answer - just think or try)
Imagine designing a new logging/monitoring system on a tight budget. Would you start with one Postgres, or bring in ClickHouse from day one? At what scale/query pattern does a columnar store truly 'earn its existence'?