🪢 Langfuse 图解教程Langfuse Visual Guide 第一部分 · 宏观全景Part 1 · The Big Picture 02 / 55
第一部分 · 宏观全景Part 1 · The Big Picture

可观测性 2.0 与宽事件Observability 2.0 & wide events

上一课说 Langfuse 的「设计灵魂」是宽事件 / 可观测性 2.0。这一课把这句口号讲透——它不是营销词, 而是写在 .agents/ARCHITECTURE_PRINCIPLES.md 里、约束着 Langfuse 每一处存储与查询设计的第一性原则。 理解了它,后面「为什么用 ClickHouse」「为什么事件不可变」「为什么列表只读窄字段」这些问题,答案都会自己浮现。 可以说,这一课是整份指南的地基课:它不讲某个具体模块,而讲一种看问题的方式,后面五十多课都建在它之上。

🔌 生活类比
传统「可观测性 1.0」像医院把你的体检拆成三个互不相通的科室:一个只记数字(血压、心率——指标 metrics), 一个只写文字(医生手记——日志 logs),一个只画就诊路线(你今天先挂号再抽血——链路 traces)。 想搞清「为什么这次不舒服」,你得自己跨三个科室对账。「可观测性 2.0」反过来:每看一次病,就写一份 完整病历——数字、描述、过程全在一张纸上。这张「完整病历」,就是宽事件(wide event)。 好处显而易见:换个医生、换个时间回看,都能从这一张纸上还原全部情况,而不必再去三个科室翻档案对时间线。
可观测性 1.0 · 三件套分离 指标 metrics 预聚合数字 · 丢细节 日志 logs 零散文本 · 难聚合 链路 traces 有结构 · 缺业务属性 🧩 事后还要人肉拼接 (reassemble later) vs 可观测性 2.0 · 一条宽事件 model · tokens · cost · latency · user · tag · env … 一行里同时有「数字 + 结构 + 业务字段」 按用户切片 按模型聚合 按时间下钻 ✅ 同一张宽表,换个查询就是新视角
1.0 vs 2.0:传统三件套把信息拆散,回答跨维度问题前先得人肉拼接;宽事件把一步操作的数字、结构、业务字段放进同一行,「下钻」和「聚合」只是对同一张宽表的不同查询。这正是 ARCHITECTURE_PRINCIPLES.md 说的「偏好宽而属性丰富的事件,而非需要事后重建的零散指标/日志/链路」。
同一步操作:拆成三条 vs 装进一行 ① 可观测性 1.0:三系统、三条记录 metric · gen.latency = 820msPrometheus · 只有数字,没有上下文 log · INFO answered q_88Loki · 只有文本,难聚合 trace span · id=sp_3Jaeger · 只有调用链,没数字/业务 事后按 traceId 人肉拼接,跨维度提问很难 vs ② 可观测性 2.0:一行宽 observation observation(同一行装下全部) input {messages:[…]} · output 「Here is how…」 model gpt-4o · latency 820ms usageDetails {input: 512, output: 88} costDetails {total: 0.0041} userId u_42 · metadata {env: prod} · tags [v3] → 下钻 / 聚合 / 过滤 都只是对这一张宽表的不同列查询,无需 JOIN
拆散 vs 一行(字段对齐 domain/observations.ts;值为示例):左边 1.0 把一步操作的数字、文本、调用链分给三套系统,回答「这个用户这小时花了多少」要先按 traceId 拼三处;右边 2.0 把它们焊进同一行宽 observation,同样的问题只是这张宽表的一次列查询。

可观测性 1.0 的困境

「指标 + 日志 + 链路」这套经典组合(业内常称可观测性 1.0)每一件单看都没错,问题出在三者分离:

于是真要排查一个跨维度的问题,你得在三套系统之间来回对账。Langfuse 的架构原则第一条就是冲着这个来的—— 把 observation 当作主分析单元,trace 只是关联句柄(原文:"Model observations as the primary analytical unit. A trace is a correlation handle that links related observations, not the only useful entry point.")。

举个具体的对比。线上有人投诉「晚高峰回答变慢」,你想知道「是不是某个模型在某些用户上特别慢」。 在 1.0 里:指标只有「全局平均延迟」这一条曲线,看不出是哪个模型、哪批用户;日志里能翻到单条记录,但几万行文本没法快速按模型分组算分位数; 链路能看单次调用的耗时,却没记 user 和 model 这些业务字段。三套都「沾边」,但没有一套能直接回答,你只能导出数据自己写脚本拼。 在 2.0 里:每条 observation 都同时带着 model、userId、latency,于是这个问题退化成一句普通查询—— 「按 model + userId 分组、看 latency 的 p95」。同一个问题,1.0 要跨三系统拼半天,2.0 改个查询就答了。

宽事件:把一切放进一行

「可观测性 2.0」的解法很直接:不要事后拼接,写的时候就别拆开。把一步操作的所有高基数上下文 (哪个模型、多少 token、多少钱、多久、哪个用户、哪些标签、什么环境……)原样塞进同一条记录——这条记录就是 Langfuse 里的 observation。它既是「链路节点」(有 parentObservationId 能拼出调用树), 又自带可聚合的数字(usageDetails / costDetails / latency),还携带任意业务字段 (metadata / tags / userId)。三件套想分开做的事,一条宽事件全包了。

你可能会问:那为什么不把这些字段拆成多张规整的表、用外键关联,岂不是更「干净」?答案是——查询时的 JOIN 是热路径上的隐藏成本。 可观测数据量极大(一个活跃项目一天可能上亿条 observation),如果每次「按模型看成本」都要把 observation 表和一张「模型表」「用量表」JOIN 起来, 扫描与连接的代价会随数据量爆炸。Langfuse 的原则是谨慎反范式化(denormalize carefully):把常用来过滤、聚合的字段直接内联到 observation 行上,让「按模型 / 按用户 / 按时间」这些高频查询变成直接的列谓词,而不是 JOIN。这就是「宽」的另一层含义—— 不只是字段多,更是有意把该放在一起的放在一起,用一点存储冗余换查询时不必连表。

😕 1.0:先拆再拼

指标、日志、链路各存一处;问一个跨维度问题,要在三套系统里查三次再人肉对账。能问什么,上线前就被字段设计锁死了。

😀 2.0:一行装下

一条 observation 同时有数字、结构、业务字段。切片、分组、过滤都是对同一张宽表的查询,事后想问什么都行。代价:行更宽、存得更多。

一条 observation = 三类字段塞进同一行 observations 宽表的一行(ObservationSchema) ① 关联字段(拼调用树)id · traceId · projectIdparentObservationIdtype · startTime/endTime让它既是「链路节点」又能独立查询 ② 数字字段(可聚合)latency · timeToFirstTokenusageDetails(token 用量)costDetails · totalCost直接 sum/avg/p95,不必从日志里 parse ③ 业务字段(可过滤分组)model · internalModelIdmetadata · environmentpromptName/Version「按模型 / 按环境」退化成列谓词 三类字段同处一行 → 「按模型看 p95 延迟」「按用户看花费」都是对这张宽表的普通列查询,无需 JOIN
宽 observation 的解剖:packages/shared/src/domain/observations.ts:55 ObservationSchema 把关联(id/traceId/parentObservationId)、数字(latency/usageDetails/costDetails)、业务(model/metadata/environment/promptName)三类字段塞进同一行。这就是「宽」的字面含义——也是「下钻=查这张表、聚合=也查这张表」的根。
packages/shared/src/domain/observations.ts:55一条 observation 的真实结构
export const ObservationSchema = z.object({
  // ① 关联字段:拼出调用树,也让本行能独立查询
  id: z.string(),  traceId: z.string().nullable(),  projectId: z.string(),
  parentObservationId: z.string().nullable(),  type: ObservationTypeDomain,
  // ② 数字字段:可直接 sum/avg/p95,无需从日志 parse
  latency: z.number().nullable(),  timeToFirstToken: z.number().nullable(),
  usageDetails: z.record(z.string(), z.number()),  costDetails: z.record(z.string(), z.number()),
  totalCost: z.number().nullable(),  totalUsage: z.number(),
  // ③ 业务字段:让「按模型 / 按环境 / 按 prompt」变成列谓词
  model: z.string().nullable(),  environment: z.string(),  metadata: MetadataDomain,
  promptName: z.string().nullable(),  promptVersion: z.number().nullable(),
  input: jsonSchema.nullable(),  output: jsonSchema.nullable(),  // 大字段,详情页才读(第24课)
});

这套原则如何贯穿 Langfuse

ARCHITECTURE_PRINCIPLES.md 里的每一条,都能在后面的课里找到对应的工程落点。下表是「原则 → 它换来什么 → 哪一课展开」:

架构原则(原文要义)它换来什么对应
observation 为主分析单元,trace 是关联句柄任何一步都能独立查询,不必从 trace 进入第 3 课
偏好宽而属性丰富的事件,而非零散三件套下钻与聚合统一成「查同一张宽表」第 8 课
保留高基数上下文,应对 unknown unknowns事后能问上线时没想到的问题本课
偏好不可变 / 追加式事件记录避免读时去重的隐藏开销;用 ReplacingMergeTree第 8 课
谨慎反范式化,消除热路径 JOIN常用过滤变成直接的列谓词第 8 · 22 课
围绕列式访问设计(窄字段、时间窗口、排序键、剪枝)大表上也能毫秒级扫描聚合第 8 · 23 课
列表/仪表盘用紧凑表示,详情页才读大字段列表快、详情全,互不拖累第 24 课
API 契约要「规模感知」:强制时间窗口、token 分页防止一不小心扫全表历史第 27 课
把成本与运维简单性当作架构约束不轻易加库/队列/物化视图第 53 课
🎯 设计取舍
宽事件不是免费的:行更宽、写入数据更多、还得有聪明的写入与压缩。Langfuse 的取舍是——用一点存储与写入复杂度,换「事后任意提问」的自由。 它靠三招把代价压下来:列式存储(只读你查的列)、ZSTD 压缩(input/output 大字段压到很小)、 延迟读取(列表只读窄字段,大字段只在详情页拉)。这三招正是后面第 8、24 课的主题。

为什么值得:unknown unknowns

宽事件最大的价值,是回答你当初没想到要问的问题。预先聚合的指标只能回答已知问题(你建了哪张仪表盘就只能看哪个维度); 而把高基数上下文整行留下,新问题来了,换个查询就能答,不必改埋点、不必等下次上线。

这一点对一个平台尤其关键。Langfuse 要给成千上万个项目用,它不可能预先知道每个团队会关心什么维度—— 有人想看「按 prompt 版本对比成本」,有人想看「按 session 看用户多轮对话的总延迟」,有人想按自定义 metadata 标签切分。 如果走预聚合的老路,平台就得为每种需求预先建一张指标——这既做不完、也僵化。宽事件让 Langfuse 把「定义新视图」这件事, 从「改埋点 + 重新上线」降级成「写一条新查询」:仪表盘、自定义指标、过滤器(第 23、40 课)本质上都是在同一张宽表上换查询。 这就是为什么「保留高基数上下文」不是锦上添花,而是整个平台可扩展性的地基。

预聚合指标:只能问「已知问题」 ✅ 平均延迟是多少?(建过这张图) ✅ 每分钟请求数?(建过这张图) ❌ 用户 X 在模型 Y 上为何变慢?没预聚合这个维度 → 答不了 宽事件:还能问「新问题」 ✅ 平均延迟?聚合 latency 列 ✅ 用户 X + 模型 Y 的慢?加两个过滤 ✅ 任何事后才想到的切法高基数整行还在 → 换个查询即可
已知 vs 未知:预聚合把「能问什么」在上线时就固定了;宽事件把高基数上下文整行留存,于是面对「unknown unknowns」——那些你当初根本没想到要问的问题——只需换一个查询。这就是 Langfuse 宁可让行更宽、也要保留上下文的根本理由。
🎯 本课要点
  • 可观测性 1.0 = 指标 + 日志 + 链路三件套分离,跨维度问题要人肉拼接,能问什么被字段设计锁死。
  • 可观测性 2.0 / 宽事件 = 把一步操作的数字、结构、业务字段放进同一条 observation;下钻与聚合统一成「查同一张宽表」。
  • 这是写在 .agents/ARCHITECTURE_PRINCIPLES.md 的第一性原则,逐条对应后面的存储与查询设计。
  • 代价是行更宽、存得更多,靠列存 + ZSTD 压缩 + 延迟读取三招平衡(第 8 · 24 课)。
  • 核心收益:能回答 unknown unknowns——事后才想到、没预先聚合的问题。

十条原则,四个主题

九到十条原则乍看零散,其实围着四个主题转——它们层层递进,正好对应这份指南后面四大块内容。把它们分组记,比逐条背更有用:

数据模型observation 为主 · 高基数
先把「记什么」定对:以 observation 为主分析单元、保留高基数上下文。记错了,后面再快的存储也救不回来。
存储宽 · 不可变 · 反范式 · 列式
再把「怎么存」定对:宽事件、追加不可变(ReplacingMergeTree)、谨慎反范式化、围绕列式访问设计排序键与分区。
查询紧凑列表 · 详情才读大字段
然后把「怎么读」定对:列表/仪表盘用紧凑的查询优化表示,大块 input/output 只在单条详情页才拉取。
契约与成本规模感知 API · 运维从简
最后把「对外/对运维」定对:API 强制时间窗口与 token 分页,避免扫全表;把额外的库、队列、物化视图的运维负担当成真实成本来权衡。

这四组也正是本指南的主线:数据模型是第一部分(本部分),存储是第二部分,写入与查询两条链路是第三、四部分, 契约与成本则散落在公共 API(第 27 课)与运维(第 53 课)。换句话说,你现在读的这份「设计哲学」,就是后面五十多课的提纲。

还有一条容易被忽略但很关键的原则:保留近实时的调试体验(原文:"Preserve real-time or near-real-time debugging workflows. Batch processing can help, but it should not make fresh production behavior invisible.")。这解释了为什么 Langfuse 的摄取虽是异步的,却要把处理延迟压到秒级——可观测性如果「看不到刚刚发生的事」,就失去了一半意义。这条会在第 14–17 课的摄取链路里反复出现。

🧪 自测 · 想一想为什么这么设计

1. “可观测性 1.0(指标+日志+链路)”和“2.0(宽事件)”最本质的区别是什么?
  1. 2.0 要求每个 trace 只能有一条 observation
  2. 1.0 更快,因为数据更宽
  3. 2.0 不再存储任何数字指标
  4. 1.0 把信息拆进三套分离的系统、回答跨维度问题前要人肉拼接;2.0 把一步的数字/结构/业务字段放进同一行,下钻与聚合都成了对同一张宽表的查询
看答案与解析 点击展开
答案:D。关键不在“用不用指标”,而在“分不分离”。1.0 的三件套各存一处,跨维度问题要在三套系统间对账;2.0 把高基数上下文整行留存,新问题只是换个查询。这正是 ARCHITECTURE_PRINCIPLES.md 第一条与第二条。
2. 为什么说宽事件能回答“unknown unknowns”,而预聚合指标不能?
  1. 因为宽事件用了更贵的数据库
  2. 因为宽事件会自动猜测你想问的问题
  3. 预聚合在上线前就把维度固定了,只能回答已建好的图;宽事件保留高基数整行,事后想到的新切法换个查询即可,无需改埋点或重新上线
  4. 因为预聚合指标不支持时间维度
看答案与解析 点击展开
答案:C。“unknown unknowns”是你当初没想到要问的问题。预聚合把“能问什么”锁死在上线时;宽事件把上下文整行留下,于是定义新视图从“改埋点+重新上线”降级成“写一条新查询”——这是平台可扩展性的地基。
3. Langfuse 为什么把 model、usageDetails、costDetails 等内联到 observation 行上,而不是拆成多张表用外键关联?
  1. 因为外键在任何数据库里都非法
  2. 为了消除热路径上的 JOIN:可观测数据量极大,常用过滤/聚合若每次都要连表代价会爆炸;内联后“按模型/用户/时间”变成直接的列谓词
  3. 因为 ClickHouse 不支持多张表
  4. 纯粹为了占用更多磁盘
看答案与解析 点击展开
答案:B。这就是“谨慎反范式化”原则:用一点存储冗余换查询时不必连表。在上亿行规模下,把常用来过滤聚合的字段内联到行上,让高频查询变成直接列谓词,是把宽事件做快的关键一招(第 8 课展开)。
💭 发散思考(没有标准答案,动手或动脑想想)
  • 想象你维护一个公司内部的 LLM 网关,老板突然问“上周哪个部门、用哪个模型、花了最多钱”。如果你只有传统的“平均成本”指标,会卡在哪?换成宽事件后这个问题怎么变简单?

Last lesson called Langfuse's "design soul" wide events / Observability 2.0. This lesson makes the slogan concrete — it is not marketing; it's a first principle written into .agents/ARCHITECTURE_PRINCIPLES.md that constrains every storage and query decision in Langfuse. Grasp it and the later questions — "why ClickHouse", "why immutable events", "why lists read only narrow fields" — answer themselves. This lesson is the guide's foundation lesson: not about one module but about a way of seeing that the fifty-odd later lessons build on.

🔌 Analogy
Classic "Observability 1.0" is like a hospital splitting your check-up across three disconnected departments: one records only numbers (blood pressure, heart rate — metrics), one writes only text (the doctor's notes — logs), one draws only the route (register, then blood draw — traces). To understand "why I feel unwell" you must reconcile across all three yourself. "Observability 2.0" flips it: each visit writes one complete medical record — numbers, description and process on a single sheet. That single sheet is the wide event. The benefit is obvious: a different doctor, or you revisiting later, can reconstruct the whole picture from that one sheet — no re-digging three departments to line up the timeline.
Observability 1.0 · three split pillars metrics pre-aggregated · lose detail logs scattered text · hard to aggregate traces structured · no business attrs 🧩 reassemble by hand later vs Observability 2.0 · one wide event model · tokens · cost · latency · user · tag · env … numbers + structure + business fields, one row by user by model by time ✅ same wide table, new query = new view
1.0 vs 2.0: the classic three pillars fragment information, forcing manual reassembly before any cross-dimension question; a wide event puts the numbers, structure and business fields of one step on a single row, so "drill down" and "aggregate" are just different queries over one wide table. This is exactly what ARCHITECTURE_PRINCIPLES.md means by "prefer wide, richly attributed events over fragmented metrics, logs, and trace records that require later reconstruction".
One step: split into three vs put on one row ① Observability 1.0: three systems, three records metric · gen.latency = 820msPrometheus · numbers only, no context log · INFO answered q_88Loki · text only, hard to aggregate trace span · id=sp_3Jaeger · call tree only, no numbers stitch by traceId after the fact — cross-cutting Qs are hard vs ② Observability 2.0: one wide observation observation (all on one row) input {messages:[…]} · output 「Here is how…」 model gpt-4o · latency 820ms usageDetails {input: 512, output: 88} costDetails {total: 0.0041} userId u_42 · metadata {env: prod} · tags [v3] → drill down / aggregate / filter all just column queries over this one wide table, no JOIN
Split vs one row (fields per domain/observations.ts; values illustrative): on the left, 1.0 scatters one step's numbers, text and call tree across three systems — answering "how much did this user spend this hour" means stitching three places by traceId; on the right, 2.0 welds them onto one wide observation row, and the same question is one column query.

The trouble with Observability 1.0

The classic "metrics + logs + traces" combo (often called Observability 1.0) is fine piece by piece; the problem is that the three are separate:

So to triage a cross-dimension problem you end up reconciling across three systems. Langfuse's very first architecture principle targets exactly this — treat the observation as the primary analytical unit, with the trace as a correlation handle (verbatim: "Model observations as the primary analytical unit. A trace is a correlation handle that links related observations, not the only useful entry point.").

A concrete contrast. Someone complains that "answers get slow at peak hours" and you want to know "is a particular model slow for particular users". In 1.0: metrics give you one "global average latency" curve that can't tell which model or which users; logs have the individual records but tens of thousands of text lines won't quickly group-by-model and compute percentiles; traces show per-call duration but never recorded the user and model business fields. All three are "related", yet none can answer directly — you end up exporting data and scripting the join yourself. In 2.0: every observation already carries model, userId and latency, so the question collapses to one ordinary query — "group by model + userId, look at p95 of latency". Same question: 1.0 stitches across three systems for ages; 2.0 just changes the query.

Wide events: put everything on one row

The "Observability 2.0" fix is direct: don't reassemble later — don't split it when you write. Cram all the high-cardinality context of one step (which model, how many tokens, how much money, how long, which user, which tags, what environment…) into one record — that record is a Langfuse observation. It's both a "trace node" (with parentObservationId to rebuild the call tree), a bag of aggregatable numbers (usageDetails / costDetails / latency), and a carrier of arbitrary business fields (metadata / tags / userId). Everything the three pillars do separately, one wide event does together.

You might ask: why not split these fields into several normalized tables with foreign keys — isn't that "cleaner"? Because JOINs at query time are a hidden hot-path cost. Observability data is enormous (an active project can write hundreds of millions of observations a day); if every "cost by model" had to JOIN the observation table with a "models" table and a "usage" table, the scan-and-join cost explodes with volume. Langfuse's principle is to denormalize carefully: inline the fields commonly used to filter and aggregate directly onto the observation row, so "by model / by user / by time" become plain column predicates instead of JOINs. That's the other meaning of "wide" — not just many fields, but deliberately keeping together what belongs together, trading a little storage redundancy for not joining at query time.

😕 1.0: split then stitch

Metrics, logs and traces live apart; a cross-dimension question means three queries plus manual reconciliation. What you can ask is locked by field design before you ship.

😀 2.0: one row holds it

One observation carries numbers, structure and business fields at once. Slice, group and filter are all queries over one wide table; ask anything later. Cost: wider rows, more storage.

One observation = three kinds of fields on one row one row of the observations wide table (ObservationSchema) ① link fields (build call tree)id · traceId · projectIdparentObservationIdtype · startTime/endTimemakes it a "call-tree node"AND independently queryable ② numeric fields (aggregatable)latency · timeToFirstTokenusageDetails (token usage)costDetails · totalCostsum/avg/p95 directly,no parsing from logs ③ business fields (filter/group)model · internalModelIdmetadata · environmentpromptName/Version"by model / by env"collapses to a column predicate Three kinds of fields on one row → "p95 latency by model", "cost by user" are ordinary column queries over this wide table, no JOIN
Anatomy of a wide observation: packages/shared/src/domain/observations.ts:55 ObservationSchema packs link (id/traceId/parentObservationId), numeric (latency/usageDetails/costDetails), and business (model/metadata/environment/promptName) fields onto one row. That's the literal meaning of "wide" — and the root of "drill-down = query this table, aggregate = also query this table".
packages/shared/src/domain/observations.ts:55the real shape of one observation
export const ObservationSchema = z.object({
  // ① link fields: build the call tree, and let this row be queried alone
  id: z.string(),  traceId: z.string().nullable(),  projectId: z.string(),
  parentObservationId: z.string().nullable(),  type: ObservationTypeDomain,
  // ② numeric fields: sum/avg/p95 directly, no parsing from logs
  latency: z.number().nullable(),  timeToFirstToken: z.number().nullable(),
  usageDetails: z.record(z.string(), z.number()),  costDetails: z.record(z.string(), z.number()),
  totalCost: z.number().nullable(),  totalUsage: z.number(),
  // ③ business fields: make "by model / by env / by prompt" a column predicate
  model: z.string().nullable(),  environment: z.string(),  metadata: MetadataDomain,
  promptName: z.string().nullable(),  promptVersion: z.number().nullable(),
  input: jsonSchema.nullable(),  output: jsonSchema.nullable(),  // big fields, read on the detail page (Lesson 24)
});

How the principles thread through Langfuse

Every line in ARCHITECTURE_PRINCIPLES.md has an engineering landing spot in a later lesson. Below: "principle → what it buys → where".

Architecture principle (gist)What it buysLesson
observation = primary unit; trace = correlation handleany step is independently queryableL03
prefer wide, richly-attributed events over fragmented pillarsdrill-down & aggregation unify into one wide tableL08
preserve high-cardinality context for unknown unknownsask questions you didn't foreseethis
favor immutable / append-oriented recordsno read-time dedup cost; ReplacingMergeTreeL08
denormalize carefully to kill hot-path JOINscommon filters become column predicatesL08·22
design around columnar access (narrow fields, time bounds, ordering, pruning)ms scans/aggregations even on big tablesL08·23
compact list/dashboard views; read big payloads only on detailfast lists, full detail, no mutual dragL24
scale-aware API contracts: time windows, token paginationavoid accidental full-history scansL27
treat cost & operational simplicity as constraintsdon't casually add DBs/queues/MVsL53
🎯 Design trade-off
Wide events aren't free: wider rows, more data written, and you need smart writes + compression. Langfuse's trade is a little storage and write complexity for the freedom to ask anything later. It tames the cost three ways: columnar storage (read only the columns you query), ZSTD compression (big input/output fields shrink hard), deferred reads (lists read narrow fields; big fields only on the detail page). Those three are the subject of lessons 8 and 24.

Why it's worth it: unknown unknowns

The biggest payoff of wide events is answering questions you didn't think to ask up front. Pre-aggregated metrics answer only known questions (you see only the dimensions of dashboards you built); keep the high-cardinality context on the row and a new question is just a new query — no re-instrumenting, no waiting for the next deploy.

This matters especially for a platform. Langfuse serves thousands of projects and cannot know in advance which dimensions each team cares about — one wants "cost by prompt version", another "total latency of a user's multi-turn session", another to slice by custom metadata tags. The pre-aggregation route would force the platform to predefine a metric per need — never-ending and rigid. Wide events let Langfuse demote "define a new view" from "change instrumentation + redeploy" to "write a new query": dashboards, custom metrics and filters (lessons 23, 40) are all just different queries over the same wide table. That's why "preserve high-cardinality context" isn't a nicety — it's the foundation of the platform's extensibility.

Pre-aggregated: only "known questions" ✅ average latency? (you built this chart) ✅ requests/min? (you built this chart) ❌ why is user X slow on model Y?dimension not pre-aggregated → can't Wide events: also "new questions" ✅ avg latency? aggregate the latency col ✅ user X + model Y slow? add two filters ✅ any slice thought of laterrow still has the context → just re-query
Known vs unknown: pre-aggregation fixes "what you can ask" at ship time; wide events keep the high-cardinality context on the row, so facing unknown unknowns — the questions you never thought to ask — you just write a new query. That's the root reason Langfuse keeps rows wide.
🎯 Key points
  • Observability 1.0 = metrics + logs + traces, three split pillars; cross-dimension questions need manual reassembly, and what you can ask is locked by field design.
  • Observability 2.0 / wide events = put numbers, structure and business fields of one step on one observation; drill-down and aggregation unify into "query one wide table".
  • This is a first principle in .agents/ARCHITECTURE_PRINCIPLES.md, mapping line-by-line to later storage/query design.
  • The cost is wider rows and more storage, balanced by columnar storage + ZSTD compression + deferred reads (L08·24).
  • Core payoff: answering unknown unknowns — questions thought of after the fact, never pre-aggregated.

Ten principles, four themes

The nine-to-ten principles look scattered but revolve around four themes — layered, and matching the four big blocks of this guide. Grouping them beats memorizing them one by one:

data modelobservation-first · high cardinality
Get "what to record" right first: observation as the primary unit, keep high-cardinality context. Get this wrong and no amount of fast storage saves you.
storagewide · immutable · denormalized · columnar
Then "how to store": wide events, append-only immutability (ReplacingMergeTree), careful denormalization, ordering keys & partitions designed for columnar access.
querycompact lists · big fields on detail only
Then "how to read": lists/dashboards on compact query-optimized representations; big input/output fetched only on a single detail page.
contract & costscale-aware API · lean ops
Finally "outward/ops": APIs force time windows and token pagination to avoid full scans; treat extra DBs, queues and materialized views as real operational cost.

These four groups are also the spine of this guide: data model is Part 1 (here), storage is Part 2, the write and read paths are Parts 3–4, and contract & cost show up in the public API (L27) and operations (L53). In other words, the "design philosophy" you're reading now is the outline of the fifty-odd lessons ahead.

One easily-missed but crucial principle: preserve near-real-time debugging (verbatim: "Preserve real-time or near-real-time debugging workflows. Batch processing can help, but it should not make fresh production behavior invisible."). This is why Langfuse's ingestion, though async, keeps processing lag to seconds — observability that "can't see what just happened" loses half its value. This recurs across the ingestion path in lessons 14–17.

🧪 Self-test - think about the design

1. What is the most essential difference between Observability 1.0 (metrics+logs+traces) and 2.0 (wide events)?
  1. 2.0 requires exactly one observation per trace
  2. 1.0 is faster because the data is wider
  3. 2.0 stops storing any numeric metrics
  4. 1.0 splits info across three separate systems and needs manual reassembly for cross-dimension questions; 2.0 puts numbers/structure/business fields of one step on one row, so drill-down and aggregation are both queries over one wide table
Show answer & explanation click to expand
Answer: D. The point isn't whether you use metrics, but whether things are separated. 1.0's three pillars live apart, forcing reconciliation across systems; 2.0 keeps high-cardinality context on the row, so a new question is just a new query. That's principles one and two in ARCHITECTURE_PRINCIPLES.md.
2. Why can wide events answer 'unknown unknowns' while pre-aggregated metrics cannot?
  1. Because wide events use a more expensive database
  2. Because wide events automatically guess your questions
  3. Pre-aggregation fixes dimensions before shipping, answering only charts you built; wide events keep the full high-cardinality row, so any slice thought of later is just a new query — no re-instrumenting or redeploy
  4. Because pre-aggregated metrics don't support a time dimension
Show answer & explanation click to expand
Answer: C. 'Unknown unknowns' are questions you didn't foresee. Pre-aggregation locks 'what you can ask' at ship time; wide events keep the context on the row, demoting 'define a new view' from 're-instrument + redeploy' to 'write a new query' — the foundation of platform extensibility.
3. Why does Langfuse inline model, usageDetails, costDetails onto the observation row instead of splitting into normalized tables with foreign keys?
  1. Because foreign keys are illegal in any database
  2. To kill hot-path JOINs: observability data is huge, so joining for common filters/aggregations would explode in cost; inlining turns 'by model/user/time' into plain column predicates
  3. Because ClickHouse doesn't support multiple tables
  4. Purely to use more disk
Show answer & explanation click to expand
Answer: B. This is the 'denormalize carefully' principle: trade a little storage redundancy for not joining at query time. At hundreds-of-millions-of-rows scale, inlining the fields used to filter/aggregate so hot queries become column predicates is key to making wide events fast (expanded in L08).
💭 Open questions (no single right answer - just think or try)
  • Imagine you run an internal LLM gateway and your boss asks 'which department, on which model, spent the most last week'. If you only had a traditional 'average cost' metric, where would you get stuck? How does a wide event make this easy?