🪢 Langfuse 图解教程Langfuse Visual Guide 第十一部分 · 设计专题与终章Part 11 · Design Themes & Capstone 54 / 55
第十一部分 · 设计专题与终章Part 11 · Design Themes & Capstone

设计专题综合Design themes synthesis

🧭 读前Before you start
先回顾 第2课 的宽事件与 第7课 的双存储两大基石,本课把六大设计专题综合起来。
Revisit Lesson 2 (wide events) and Lesson 7 (dual storage); this lesson synthesizes the six design themes.

走到这里,你已经把 Langfuse 从一条 trace 的出生看到了它的落幕、从前端点按看到了后台队列。这一课往后退一步,做一件事:把 53 课里反复出现的设计决策,收拢成六个名字。你会发现,那些看似各管一摊的实现——ClickHouse 为什么用列存、为什么要用队列、为什么删除要跨三个库、为什么处处带着 projectId——背后其实是同一套世界观在反复落地。Langfuse 官方的 ARCHITECTURE_PRINCIPLES.md 把它一句话点明:「为宽的、结构化的事件数据上的高吞吐、探索式可观测而优化」。 六个主题——宽事件、不可变、异步、双存储、多租户、成本——不是六个孤立的技巧,而是这一句话的六个切面。看懂它们,你就不只是「读过 Langfuse 的代码」,而是真正理解了「一个高规模可观测平台该怎么想」。

📋 生活类比
前面 53 课像是逐间参观一座大厦的每个房间——这一课,是带你登上对面的天台,看清这栋楼的整体设计语言。你会发现设计师其实只信奉几条原则,然后在每个房间里反复运用:承重墙都朝同一个方向、管线都走同一套规范、连灯开关的位置都遵循同一个习惯。 一座好建筑的精妙,不在于某个房间多花哨,而在于整栋楼贯穿着一致的取舍——每一处局部选择,回头看都能归到那几条原则上。Langfuse 也是如此:宽事件决定了「数据长什么样」,不可变决定了「数据怎么写」,异步决定了「活儿在哪儿干」,双存储决定了「数据存哪儿」,多租户决定了「数据归谁」,成本决定了「什么该做什么不该做」。六根梁,撑起同一座楼。

主题一·二:宽事件 + 不可变(数据怎么长、怎么写)

宽事件(wide events)是整套架构的地基,也是「可观测性 2.0」的核心主张。传统做法把监控数据拆成三摊——指标(metrics)、日志(logs)、链路(traces),事后再费劲拼回去。Langfuse 反其道:以 observation 为主要分析单位,把一次操作的全部上下文(输入、输出、模型、用量、耗时、自定义属性……)塞进一条又宽又富的事件里;trace 只是把相关 observation 串起来的「关联句柄」,不是唯一入口。这样保住了高基数上下文——你能任意切片、分组、过滤,去调查那些事前没预料到的问题(unknown unknowns),而不必预先为每个未来的问题建好指标(第5/6课的领域模型、第41课的查询引擎都是这个主张的落地)。

不可变(immutability)是宽事件的天然搭档。高吞吐遥测下,Langfuse 偏好追加而非更新:事件写进去就基本不动。为什么?因为「更新」在海量数据上代价惊人——它逼着查询时做读时去重,凭空多出隐藏的查询成本。第8课你已见识过 ClickHouse 怎么用 ReplacingMergeTree(event_ts, is_deleted) + 读时 FINAL 去重来吞下「同一实体多次写入」:不真的去改老行,而是追加新行、查询时按 event_ts 保留最新版本。把「改」变成「追加 + 查时去重」,是高规模写入能扛住的关键。

一条又宽又富的事件,胜过三摊事后拼图 传统:拆成三摊 metrics(聚合数值,丢了细节) logs(散乱文本) traces(又一套) 事后费劲拼回去 Langfuse:一条宽事件 observation输入 · 输出 · 模型 · 用量 · 耗时+ 任意自定义属性(高基数)trace = 把相关 observation 串起来的句柄可任意切片/分组/过滤调查 unknown unknowns,不必预建指标 不可变:追加写进去就不动,只追加新行 为何不更新?更新逼着读时去重= 海量数据的隐藏查询成本 第8课:ClickHouse 用 ReplacingMergeTree(event_ts,is_deleted) + 读时 FINAL,把「同实体多次写入」变成「追加新行、查时保留最新」 把「改」变成「追加 + 合并」——这是高规模写入扛得住的关键,也是不可变与宽事件的共谋
宽事件 + 不可变:ARCHITECTURE_PRINCIPLES.md 原则 1-4——「observation 为主要分析单位」「宽而富的事件优于碎片化的 metrics/logs/traces」「保住高基数上下文调查 unknown unknowns」「偏好不可变/追加,更新会造成读时去重的隐藏成本」。落地见第 5/6 课领域模型、第 8 课 ReplacingMergeTree、第 41 课查询引擎。
示例:六大设计主题 → 各自落到的真实机制 同一套信念,落到全书各处的具体代码 主题 theme 落到的真实机制 课次 宽事件 observations 作宽行、保高基数列 L02·L08 不可变 + 去重 ReplacingMergeTree FINAL · 事件 id 幂等 L08·L13 异步解耦 API 落 S3+Redis 即返 207 · worker 消费 L12·L15 双存储 Postgres 配置/事务 + ClickHouse 分析 L09·L08 多租户 每行带 projectId · RBAC 角色×scope L06·L49 成本意识 列式扫描 · 按 token 定价 · 缓存命中 L16·L42 这六条不是孤立的设计,而是反复在每条链路上重演——这正是「读懂一处,处处相通」
整本指南其实只在反复讲六件事(机制均取自对应课的真实源码;课号为导航):①宽事件——把 observation 做成宽行、保住高基数,于是能任意切片(L02/L08);②不可变+去重——只追加事件、靠 ReplacingMergeTree FINAL 和事件 id 幂等收敛(L08/L13);③异步解耦——摄取只落 S3+Redis 就返 207,重活交 worker(L12/L15);④双存储——Postgres 管配置/事务、ClickHouse 管分析(L09/L08);⑤多租户——每行带 projectId、RBAC 角色×scope 判权(L06/L49);⑥成本意识——列式扫描、按 token 定价、缓存命中(L16/L42)。读懂一处,处处相通。
主题一宽事件(observability 2.0)
一次操作 = 一条带全部上下文的宽事件。observation 是分析单位、trace 是关联句柄。保住高基数 → 能回答事前没想到的问题,而不必为每个未来问题预建指标。这是「探索式可观测」的根。
主题二不可变(追加而非更新)
高吞吐下偏好追加。更新会逼出读时去重的隐藏成本,所以用 ReplacingMergeTree(event_ts, is_deleted)「追加新行 + 查时按 event_ts 保留最新」替代「原地改」。不可变让写入路径简单、可预测、能水平扩展。

主题三·四:异步 + 双存储(活在哪儿干、数据存哪儿)

异步(async)是 Langfuse 扛住高吞吐的骨架。请求路径上只做最轻的事——校验、落 S3、塞进 Redis 队列就立刻返回;真正的重活全推给 BullMQ 队列 + worker 在后台慢慢消化(第12-19课的摄取链路)。这套「快速接收、异步处理」的分工,在后面反复变奏:第30课 eval、第43课计量、第46课分析集成、第52课后台迁移——你见过的两级 fan-out(调度→逐项目)、幂等、水位线、分布式锁,全是「异步」这个主题为应对「失败会重试、规模会很大」演化出的纪律。把活儿从请求里搬到队列里,是高规模系统「削峰、解耦、可重试」的通用答案。

双存储(实为三存储)决定数据的归宿,让读写各得其所。Postgres 扛 OLTP——配置、用户、权限这类强一致、低频改的结构化元数据;ClickHouse 扛 OLAP——海量、可分析的事件,靠列存 + 时间分区 + 排序键 + 数据裁剪把「扫几十亿行做聚合」变得可行;对象存储 S3 放体积大的原始负载(完整输入输出、媒体)。配套的取舍是谨慎反范式化(去掉热路径上的 join,把常用过滤变成直接列谓词)、列表/仪表盘只读紧凑表示、详情才取大字段(第22/24课)。不同形状的数据放进最合适的引擎——这是「为列式访问设计」的直接结果。

快接收→异步处理;数据按形状各归各家 请求路径(薄)校验·落S3·塞队列→立刻返回 BullMQ + worker重活后台慢慢消化 异步主题的反复变奏两级 fan-out · 幂等 · 水位线 · 分布式锁(第30/43/46/52课) Postgres(OLTP)配置/用户/权限·强一致低频改结构化元数据「源真相」 ClickHouse(OLAP)海量事件·列存+分区+排序键扫几十亿行做聚合可行谨慎反范式去 join 对象存储 S3大块负载·完整输入输出/媒体列表只取紧凑表示详情才取大字段(第22/24课)
异步 + 双(三)存储:ARCHITECTURE_PRINCIPLES.md 原则 5-7、10——反范式去热路径 join、为列式访问设计(窄选列/时间界/排序键/裁剪)、列表用紧凑表示大字段延后、保住近实时调试。异步骨架见第 12-19 课摄取,反复变奏见第 30/43/46/52 课;三存储见第 2/22/24/52 课。
主题三异步(队列 + worker)
请求路径只接收、不干重活;重活进 BullMQ 由 worker 后台处理。削峰、解耦、可重试。其衍生纪律——fan-out、幂等、水位、锁——在 eval/计量/集成/迁移里反复出现,是同一主题的不同变奏。
主题四双(三)存储(各归各家)
Postgres 扛 OLTP 元数据、ClickHouse 扛 OLAP 海量事件(列存)、S3 放大块负载。配套谨慎反范式、列表紧凑/详情取大字段。让不同形状的数据落进最合适的引擎,是性能与成本的根。
packages/shared/src/server/redis/ingestionQueue.ts异步·分片队列
// 异步主题落到代码:摄取队列按「分片」fan-out,由 worker 后台消费
export class IngestionQueue {
  public static getShardNames() {
    return Array.from(
      { length: env.LANGFUSE_INGESTION_QUEUE_SHARD_COUNT },
      (_, i) => `${QueueName.IngestionQueue}${i > 0 ? `-${i}` : ""}`,
    );   // → IngestionQueue, IngestionQueue-1, IngestionQueue-2 …
  }
}

主题五·六:多租户 + 成本(数据归谁、什么值得做)

多租户(multi-tenancy)是贯穿全栈的一条隐线:几乎每张表、每个查询、每条队列消息都带着 projectId。从第48课的登录(session 注入组织/项目成员关系)、到第49课的 RBAC scope 与 API key scope、再到第50课按 plan 给的 entitlement,构成层层收窄的隔离;第46课的分析集成、第52课的删除都按项目 fan-out。多租户不是某一处的功能,而是一条「任何数据都明确归属、任何访问都明确授权」的纪律——它决定了一个平台能不能安全地让成千上万个互不信任的租户共用同一套基础设施。

成本(cost)是把前五个主题「管住」的那条约束。架构原则里写得很直白:把成本与运维简单性当作架构约束——「额外的数据库、队列、物化视图、迁移,都必须挣回它们的长期运维负担」。这解释了很多克制的选择:API 契约要求时间窗口、暴露字段选择、用 token 分页、不给「能扫全部历史」的危险默认(第27课);模型与用量成本被显式计量(第42/43课)。不是「能加就加」,而是「每一个新增的活动部件,都要先证明自己值这份维护成本」——这份克制,恰恰是系统能长期简单、可维护的根。

目标:在宽的结构化事件上做高吞吐、探索式可观测(且是平台) 前提:探索式 前提:高吞吐+扩展 前提:平台+长期 ① 宽事件:保住高基数,调查 unknown unknowns ② 不可变(查时去重) ③ 异步(削峰) ④ 双存储(按形状) ⑤ 多租户(处处 projectId) ⑥ 成本(部件须挣回运维) 拆开任一个,其余都显别扭——架构是一组彼此印证、共同服务于同一目标的取舍
主题一句话在哪些课见过
① 宽事件一条又宽又富的事件,保住高基数调查 unknown unknownsL05/L06 领域模型 · L41 查询引擎
② 不可变追加而非更新;改 = 追加新行 + 查时去重(保留最新)L08 ReplacingMergeTree · L13
③ 异步快接收、队列里慢处理;fan-out/幂等/水位/锁L12–19 摄取 · L30/L43/L46/L52
④ 双(三)存储PG(OLTP)+CH(OLAP)+S3(负载),各归各家L02/L22/L24 · L52 删除
⑤ 多租户处处带 projectId;鉴权→RBAC→entitlement 层层隔离L48/L49/L50 · L46/L52 按项目
⑥ 成本每个活动部件须挣回运维负担;契约要 scale-awareL27 公共API · L42/L43 成本
🎯 设计取舍
这六个主题,为什么说是「一套世界观的六个切面」,而不是六个互不相干的技巧? 因为它们全都从同一个目标推导而来:在宽的、结构化的事件数据上,做高吞吐、探索式的可观测。把这句话拆开看——「探索式」要求你能回答事前没想到的问题,于是必须保住高基数(宽事件),而不能预先把数据揉成几个固定指标;「高吞吐」要求写入便宜、查询能横向扩展,于是写要不可变(避免读时去重)、活要异步(削峰解耦)、存要按访问形状分到列式引擎(双存储);而「平台」二字意味着它要服务很多租户、要长期能维护,于是必须多租户隔离、必须把成本当硬约束。你看,每一个主题都不是凭空的偏好,而是从「目标 + 规模」这两个前提一步步逼出来的必然。这正是「架构」和「一堆技术选型」的区别:架构是一组彼此印证、共同服务于同一目标的取舍,拆开任何一个,其余的都会显得别扭。

这套思维方式,怎么迁移到你自己的系统? 别照搬 Langfuse 的具体选型(你未必需要 ClickHouse、未必要 fan-out 队列),而要学它推导的方式:先把你系统的「目标 + 规模前提」想清楚——你要回答的是已知问题还是探索式问题?写多还是读多?一个租户还是很多租户?能容忍多大延迟?把这几个前提钉死,很多「该不该上消息队列」「该不该分库」「要不要反范式」的争论会自动有答案。而架构原则文档里那句最朴素的「每个额外的活动部件都要挣回它的长期运维负担」,则是一把万能的减法尺子——在加任何东西之前,先问它配不配。能用更少的部件达成目标,几乎永远是更好的架构。
🎯 本课要点
  • 六个主题 = 一套世界观的六个切面:都从「为宽的结构化事件数据做高吞吐、探索式可观测」推导而来(ARCHITECTURE_PRINCIPLES.md)。
  • ① 宽事件:observation 为分析单位、trace 为关联句柄;宽而富 + 高基数 → 调查 unknown unknowns,不必预建指标。
  • ② 不可变 + ③ 异步:写不可变(追加+合并,避读时去重)、活异步(队列削峰解耦,衍生 fan-out/幂等/水位/锁)——高吞吐扛得住的两根支柱。
  • ④ 双(三)存储 + ⑤ 多租户:PG/CH/S3 按形状各归各家(反范式、列式、紧凑列表);处处 projectId、鉴权→RBAC→entitlement 层层隔离。
  • ⑥ 成本是约束:每个额外的库/队列/视图/迁移都要挣回运维负担;契约 scale-aware(时间窗/字段选择/token 分页)。迁移这套思维的关键不是照搬选型,而是先钉死「目标+规模」前提、再用「部件配不配」做减法。

🧪 自测 · 想一想为什么这么设计

1. 这一课把 Langfuse 反复出现的设计归纳为六个主题(宽事件、不可变、异步、双存储、多租户、成本)。这一课的核心论点是?
  1. 这六个主题应该照搬到任何系统里
  2. 这六个主题互相冲突,需要权衡取舍
  3. Langfuse 的设计是六个独立团队各自决定的
  4. 这六个不是互不相干的技巧,而是同一套世界观的六个切面——全都从「为宽的、结构化的事件数据做高吞吐、探索式可观测」这个目标推导而来;它们彼此印证、共同服务一个目标,这正是「架构」区别于「一堆技术选型」之处
看答案与解析 点击展开
答案:D。这一课的关键不在于列出六个名词,而在于揭示它们的「同源性」。把目标「为宽的结构化事件数据做高吞吐、探索式可观测」拆开:探索式→要保住高基数(宽事件)、不能预揉成固定指标;高吞吐→写要便宜(不可变避读时去重)、活要削峰(异步)、存要按访问形状分(双存储);平台→要服务多租户(隔离)、要长期可维护(成本约束)。每个主题都是从「目标+规模」逼出来的必然,而非孤立偏好。这就是架构与「技术选型清单」的根本区别:架构是一组彼此印证、共同指向同一目标的取舍,拆掉任何一个,其余都会显得别扭。
2. 「宽事件」与「不可变」这两个主题是怎么互相配合的?
  1. 不可变要求事件越窄越好,与宽事件矛盾
  2. 宽事件主张把一次操作的全部上下文塞进一条又宽又富、写完基本不动的事件里;不可变则用「追加而非更新」承接这种高吞吐写入——改用「追加新行+查时去重保留最新」(ClickHouse ReplacingMergeTree(event_ts,is_deleted)+FINAL)替代原地更新,避免更新逼出的读时去重隐藏成本
  3. 宽事件和不可变其实是同一件事的两个名字
  4. 它们没有关系,只是凑在一节里讲
看答案与解析 点击展开
答案:B。宽事件决定「数据长什么样」——以 observation 为单位、把全部上下文(输入输出模型用量耗时+高基数自定义属性)装进一条事件,trace 只是关联句柄。这种又宽又富、海量产生的事件,写入方式必须便宜且可扩展,于是不可变接力:写完就不动、只追加。为什么不更新?因为在几十亿行规模上,更新会逼着查询做读时去重,凭空增加隐藏成本。ClickHouse 的 ReplacingMergeTree(event_ts, is_deleted) 正是这个主题的落地——同一实体多次写入不原地改,而是追加新行、查询时用 FINAL 按 event_ts 保留最新版本(读时去重)。所以两者是共谋:宽事件定义了数据形状,不可变让这种形状的海量写入扛得住。
3. 架构原则文档里有一句反复被引用的话:「额外的数据库、队列、物化视图、迁移,都必须挣回它们的长期运维负担」。这句话体现的「成本」主题,对设计决策意味着什么?
  1. 意思是要尽量用最便宜的云服务
  2. 意思是性能比一切都重要
  3. 把成本与运维简单性当作硬约束,而非「能加就加」:每个新增的活动部件都要先证明自己值这份长期维护成本。它解释了很多克制的选择(API契约要时间窗/字段选择/token分页、不给能扫全历史的危险默认)——能用更少部件达成目标几乎永远是更好的架构
  4. 意思是不应该有任何数据库或队列
看答案与解析 点击展开
答案:C。这句话把「成本」从「钱」上升为「架构约束」。它针对的不是云账单,而是每个活动部件带来的长期运维负担——多一个数据库要备份/监控/升级、多一个队列要保活/排障、多一个物化视图要维护一致性、多一个迁移要小心演进。所以默认姿态是减法:加任何东西之前先问「它配不配」。这解释了 Langfuse 一系列克制选择:公共 API 强制时间窗口、暴露字段选择、用 token 分页、拒绝「默认扫全部历史」这种危险便利。其底层智慧是:能用更少的活动部件达成同样目标,几乎永远是更可维护、更不容易出事的架构。复杂度是要还的债,成本主题就是那把催你少借债的尺子。
💭 发散思考(没有标准答案,动手或动脑想想)
  • 这一课点出:迁移 Langfuse 的设计智慧,关键不是照搬具体选型(ClickHouse、fan-out 队列),而是学它的推导方式——先钉死「目标+规模」前提(探索式还是已知问题?写多还是读多?一个还是多个租户?容忍多大延迟?),再用「每个活动部件都要挣回运维负担」做减法。请挑一个你熟悉或正在做的系统,试着走一遍这个推导:它的目标+规模前提是什么?据此,六个主题里哪些适用、哪些不必要?你现有架构里有没有「没挣回运维负担」的部件?

By now you've followed a trace from birth to curtain call, from front-end clicks to background queues. This lesson takes a step back to do one thing: gather the design decisions that kept recurring across 53 lessons into six names. You'll find that the seemingly-separate implementations — why ClickHouse uses columnar storage, why queues, why deletion spans three stores, why projectId is everywhere — are actually one worldview landing over and over. Langfuse's official ARCHITECTURE_PRINCIPLES.md states it in a sentence: optimize for high-scale, exploratory observability on wide, structured event data. The six themes — wide events, immutability, async, dual storage, multi-tenancy, cost — aren't six isolated tricks but six facets of that one sentence. Grasp them and you've not just "read Langfuse's code" but truly understood "how a high-scale observability platform should think."

📋 Analogy
The previous 53 lessons were like touring each room of a great building one by one — this lesson takes you up to the rooftop across the street to see the building's overall design language. You'll find the architect actually believes in just a few principles, then applies them repeatedly in every room: load-bearing walls all face the same way, conduits all follow one standard, even the light-switch positions follow one habit. A good building's elegance isn't in any one room being fancy, but in consistent trade-offs running through the whole building — every local choice, in hindsight, traces back to those few principles. Langfuse is the same: wide events decide "what the data looks like," immutability decides "how it's written," async decides "where the work happens," dual storage decides "where it's stored," multi-tenancy decides "who it belongs to," cost decides "what should and shouldn't be done." Six beams, holding up one building.

Themes 1 & 2: wide events + immutability (how data looks and is written)

Wide events are the foundation of the whole architecture and the core claim of "observability 2.0." The traditional approach splits monitoring data into three piles — metrics, logs, traces — then laboriously stitches them back later. Langfuse does the reverse: with the observation as the primary analytical unit, it packs an operation's full context (input, output, model, usage, latency, custom attributes…) into one wide, rich event; a trace is just the "correlation handle" linking related observations, not the only entry point. This preserves high-cardinality context — you can slice, group, filter at will to investigate problems you didn't anticipate (unknown unknowns), without pre-building a metric for every future question (Lessons 5/6 domain models and Lesson 41's query engine are this claim landing).

Immutability is wide events' natural partner. Under high-throughput telemetry, Langfuse favors append over update: an event written basically doesn't move. Why? Because "updates" are shockingly costly at massive scale — they force read-time deduplication, conjuring hidden query costs. In Lesson 8 you saw how ClickHouse uses ReplacingMergeTree(event_ts, is_deleted) + read-time FINAL deduplication to swallow "the same entity written multiple times": it doesn't really modify old rows but appends new ones and keeps the latest version by event_ts at query time. Turning "modify" into "append + dedup at query" is the key to write-side surviving at high scale.

One wide rich event beats three piles stitched later tradition: three piles metrics (aggregated, detail lost) logs (scattered text) traces (yet another set) laboriously stitched back later Langfuse: one wide event = observationinput · output · model · usage · latency+ any custom attributes (high cardinality)trace = handle linking related observationsslice/group/filter at willinvestigate unknown unknowns, no pre-built metrics immutable: appendwritten stays put, only append new rows why not update?updates force read-time dedup= hidden query cost at scale L8: ReplacingMergeTree(event_ts, is_deleted) + read-time FINAL → "entity written N times" = "append rows, keep latest at query" Turning "modify" into "append + merge" — the key to high-scale write survival, and the conspiracy between immutability and wide events
Wide events + immutability: ARCHITECTURE_PRINCIPLES.md principles 1-4 — "observation as the primary analytical unit," "wide richly-attributed events over fragmented metrics/logs/traces," "preserve high-cardinality context to investigate unknown unknowns," "favor immutable/append; updates create read-time-dedup hidden cost." Landed in Lessons 5/6 domain models, Lesson 8 ReplacingMergeTree, Lesson 41 query engine.
Example: six design themes → the real mechanism each lands in one set of beliefs, realized as concrete code throughout the guide theme the real mechanism lessons wide events observations as wide, high-cardinality rows L02·L08 immutable + dedup ReplacingMergeTree FINAL · event-id idempotency L08·L13 async decouple API lands S3+Redis → 207 · worker consumes L12·L15 dual storage Postgres config/txn + ClickHouse analytics L09·L08 multi-tenancy projectId on every row · RBAC role×scope L06·L49 cost-aware columnar scans · token pricing · cache hits L16·L42 these six aren't isolated choices but recur on every path — understand one and the rest connect
The whole guide really repeats just six things (every mechanism is from that lesson's real source; lesson numbers are for navigation): ① wide events — make each observation a wide row and keep high cardinality, so you can slice freely (L02/L08); ② immutable + dedup — append-only events converged via ReplacingMergeTree FINAL and event-id idempotency (L08/L13); ③ async decoupling — ingestion just lands S3+Redis and returns 207, the heavy work goes to a worker (L12/L15); ④ dual storage — Postgres for config/txn, ClickHouse for analytics (L09/L08); ⑤ multi-tenancy — projectId on every row, RBAC role×scope for authz (L06/L49); ⑥ cost-awareness — columnar scans, token pricing, cache hits (L16/L42). Understand one and the rest connect.
theme 1wide events (observability 2.0)
One operation = one wide event with full context. The observation is the analytical unit, the trace a correlation handle. Preserving high cardinality → answer questions you didn't anticipate, without pre-building a metric per future question. This is the root of "exploratory observability."
theme 2immutability (append not update)
High-throughput favors append. Updates force read-time-dedup hidden cost, so ReplacingMergeTree(event_ts, is_deleted)'s "append new rows + keep the latest by event_ts at query" replaces "modify in place." Immutability keeps the write path simple, predictable, and horizontally scalable.

Themes 3 & 4: async + dual storage (where work happens, where data lives)

Async is the skeleton that lets Langfuse survive high throughput. The request path does only the lightest things — validate, land in S3, push to a Redis queue and return immediately; the real heavy lifting goes to BullMQ queues + workers to digest slowly in the background (Lessons 12-19's ingestion path). This "receive fast, process async" division recurs in variations later: Lesson 30 eval, Lesson 43 metering, Lesson 46 analytics integrations, Lesson 52 background migrations — the two-level fan-out (schedule→per-object), idempotency, watermarks, distributed locks you've seen are all disciplines the "async" theme evolved to handle "failures retry, scale gets big." Moving work from the request into the queue is the universal answer to high-scale "peak-shaving, decoupling, retryability."

Dual storage (really tri-store) decides data's home, letting reads and writes each fit. Postgres handles OLTP — strongly-consistent, rarely-changed structured metadata like config, users, permissions; ClickHouse handles OLAP — massive analyzable events, using columnar storage + time partitions + ordering keys + data pruning to make "scan billions of rows to aggregate" feasible; object storage S3 holds bulky raw payloads (full inputs/outputs, media). The accompanying trade-offs are careful denormalization (kill hot-path joins, turn common filters into direct column predicates) and lists/dashboards read only compact representations, detail views fetch big fields (Lessons 22/24). Putting differently-shaped data into the most-fitting engine — a direct result of "designing for columnar access."

Receive fast→process async; data goes home by shape request path (thin)validate·land S3·queue→return now BullMQ + workerheavy work digested in background recurring variations of the async themetwo-level fan-out · idempotency · watermark · distributed lock (L30/43/46/52) Postgres (OLTP)config/users/perms·consistent rarely-changedstructured metadata"source of truth" ClickHouse (OLAP)massive events·columnar+partition+order keyscan billions to aggregate, feasiblecareful denorm to kill joins object storage S3bulky payloads·full I/O / medialists fetch only compact representationsdetail fetches big fields (L22/24)
Async + dual (tri) storage: ARCHITECTURE_PRINCIPLES.md principles 5-7, 10 — denormalize to kill hot-path joins, design for columnar access (narrow field selection/time bounds/ordering keys/pruning), lists use compact representations deferring big fields, preserve near-real-time debugging. The async skeleton is Lessons 12-19 ingestion, variations in Lessons 30/43/46/52; tri-store in Lessons 2/22/24/52.
theme 3async (queue + worker)
The request path only receives, no heavy work; heavy work enters BullMQ for the worker to process in the background. Peak-shaving, decoupling, retryability. Its derived disciplines — fan-out, idempotency, watermark, lock — recur in eval/metering/integration/migration, variations of one theme.
theme 4dual (tri) storage (each goes home)
Postgres handles OLTP metadata, ClickHouse OLAP massive events (columnar), S3 bulky payloads. With careful denormalization, compact lists / big-field detail. Letting differently-shaped data land in the most-fitting engine is the root of performance and cost.
packages/shared/src/server/redis/ingestionQueue.tsasync · sharded queue
// The async theme in code: the ingestion queue fans out across "shards", consumed by the worker
export class IngestionQueue {
  public static getShardNames() {
    return Array.from(
      { length: env.LANGFUSE_INGESTION_QUEUE_SHARD_COUNT },
      (_, i) => `${QueueName.IngestionQueue}${i > 0 ? `-${i}` : ""}`,
    );   // → IngestionQueue, IngestionQueue-1, IngestionQueue-2 …
  }
}

Themes 5 & 6: multi-tenancy + cost (who data belongs to, what's worth doing)

Multi-tenancy is a hidden thread running full-stack: nearly every table, every query, every queue message carries a projectId. From Lesson 48's login (session injecting org/project memberships), to Lesson 49's RBAC scopes and API-key scopes, to Lesson 50's plan-based entitlements, they form layer-upon-layer narrowing isolation; Lesson 46's analytics integrations and Lesson 52's deletion both fan out per project. Multi-tenancy isn't a feature in one place but a discipline of "every datum has a clear owner, every access has clear authorization" — it decides whether a platform can safely let thousands of mutually-untrusting tenants share one infrastructure.

Cost is the constraint that "reins in" the first five themes. The architecture principles say it bluntly: treat cost and operational simplicity as architectural constraints — "extra databases, queues, materialized views, and migrations must earn their long-term operational burden." This explains many restrained choices: API contracts require time windows, expose field selection, use token pagination, give no dangerous "can scan all history" defaults (Lesson 27); model and usage cost are explicitly metered (Lessons 42/43). Not "add it because you can," but "every new moving part must first prove it's worth its maintenance cost" — this restraint is precisely the root of a system staying simple and maintainable long-term.

goal: high-throughput, exploratory observability over wide structured events (a platform) premise: exploratory premise: throughput+scale premise: platform+long-term 1 wide events: keep high cardinality, investigate unknown unknowns 2 immutable (dedup) 3 async (absorb) 4 dual store (by shape) 5 multi-tenancy (projectId) 6 cost (parts earn maintenance) remove any one and the rest feel off — architecture is a set of mutually-reinforcing trade-offs for one goal
ThemeIn a sentenceSeen in lessons
① wide eventsone wide rich event, preserving high cardinality to investigate unknown unknownsL05/L06 domain models · L41 query engine
② immutabilityappend not update; modify = append new rows + dedup at query (keep latest)L08 ReplacingMergeTree · L13
③ asyncreceive fast, process slowly in queues; fan-out/idempotency/watermark/lockL12–19 ingestion · L30/L43/L46/L52
④ dual (tri) storagePG(OLTP)+CH(OLAP)+S3(payloads), each goes homeL02/L22/L24 · L52 deletion
⑤ multi-tenancyprojectId everywhere; auth→RBAC→entitlement layered isolationL48/L49/L50 · L46/L52 per-project
⑥ costevery moving part must earn its burden; contracts must be scale-awareL27 public API · L42/L43 cost
🎯 Design trade-off
Why are these six themes "six facets of one worldview" rather than six unrelated tricks? Because they all derive from one goal: high-throughput, exploratory observability on wide, structured event data. Unpack that sentence — "exploratory" requires answering questions you didn't anticipate, so you must preserve high cardinality (wide events), not pre-mash data into a few fixed metrics; "high-throughput" requires cheap writes and horizontally-scalable queries, so writes are immutable (avoid read-time dedup), work is async (peak-shave, decouple), storage is split by access shape into columnar engines (dual storage); and the word "platform" means it serves many tenants and must be maintainable long-term, so it must isolate by multi-tenancy and treat cost as a hard constraint. See — each theme isn't an arbitrary preference but a necessity forced step by step from the two premises "goal + scale." This is exactly the difference between "architecture" and "a pile of tech choices": architecture is a set of trade-offs that mutually corroborate and jointly serve one goal; pull out any one and the rest look awkward.

How does this way of thinking transfer to your own system? Don't copy Langfuse's specific choices (you may not need ClickHouse, may not need fan-out queues), but learn its way of deriving: first get clear on your system's "goal + scale premises" — are you answering known questions or exploratory ones? Write-heavy or read-heavy? One tenant or many? How much latency can you tolerate? Nail these premises and many debates over "should we add a message queue," "should we shard," "should we denormalize" answer themselves. And the principles doc's plainest line — "every extra moving part must earn its long-term operational burden" — is a universal subtraction ruler: before adding anything, ask whether it deserves to be there. Achieving the goal with fewer parts is almost always the better architecture.
🎯 Key points
  • Six themes = six facets of one worldview: all derived from "high-throughput, exploratory observability on wide structured event data" (ARCHITECTURE_PRINCIPLES.md).
  • ① wide events: observation as the analytical unit, trace as the correlation handle; wide + rich + high cardinality → investigate unknown unknowns, no pre-built metrics.
  • ② immutability + ③ async: immutable writes (append+merge, avoid read-time dedup), async work (queues peak-shave/decouple, deriving fan-out/idempotency/watermark/lock) — the two pillars of surviving high throughput.
  • ④ dual (tri) storage + ⑤ multi-tenancy: PG/CH/S3 each go home by shape (denorm, columnar, compact lists); projectId everywhere, auth→RBAC→entitlement layered isolation.
  • ⑥ cost is a constraint: every extra DB/queue/view/migration must earn its burden; contracts scale-aware (time windows/field selection/token pagination). The key to transferring this thinking isn't copying choices but nailing "goal+scale" premises first, then subtracting via "does this part deserve to exist."

🧪 Self-test - think about the design

1. This lesson distills Langfuse's recurring design into six themes (wide events, immutability, async, dual storage, multi-tenancy, cost). What's the lesson's central argument?
  1. these six themes should be copied verbatim into any system
  2. the six themes conflict and require trade-offs against each other
  3. Langfuse's design was decided independently by six separate teams
  4. the six aren't unrelated tricks but six facets of one worldview — all derived from the goal 'high-throughput, exploratory observability on wide, structured event data'; they mutually corroborate and jointly serve one goal, which is exactly what distinguishes 'architecture' from 'a pile of tech choices'
Show answer & explanation click to expand
Answer: D. This lesson's key isn't listing six nouns but revealing their common origin. Unpack the goal 'high-throughput, exploratory observability on wide structured event data': exploratory→preserve high cardinality (wide events), don't pre-mash into fixed metrics; high-throughput→cheap writes (immutability avoids read-time dedup), peak-shaving work (async), storage split by access shape (dual storage); platform→serve many tenants (isolation), maintainable long-term (cost constraint). Each theme is a necessity forced from 'goal+scale', not an isolated preference. That's the fundamental difference between architecture and a 'tech-choice checklist': architecture is a set of mutually-corroborating trade-offs pointing at one goal; remove any and the rest look awkward.
2. How do the 'wide events' and 'immutability' themes reinforce each other?
  1. immutability requires the narrowest events, conflicting with wide events
  2. wide events advocate packing an operation's full context into one wide, rich event that basically doesn't move after writing; immutability handles this high-throughput write via 'append not update' — using 'append new rows + dedup at query keeping latest' (ClickHouse ReplacingMergeTree(event_ts,is_deleted)+FINAL) instead of in-place update, avoiding the read-time-dedup hidden cost updates would force
  3. wide events and immutability are really two names for the same thing
  4. they're unrelated, just discussed in one section
Show answer & explanation click to expand
Answer: B. Wide events decide 'what the data looks like'—per observation, packing all context (input/output/model/usage/latency + high-cardinality custom attributes) into one event, with the trace as just a correlation handle. Such wide, rich, massively-produced events need a write path that's cheap and scalable, so immutability takes over: written stays put, only append. Why not update? Because at billions of rows, updates force read-time dedup, conjuring hidden cost. ClickHouse's ReplacingMergeTree(event_ts, is_deleted) is this theme landing—the same entity written multiple times isn't modified in place but appended as new rows, with FINAL keeping the latest version by event_ts at query (read-time deduplication). So the two conspire: wide events define the data shape, immutability makes that shape's massive writes survivable.
3. The architecture principles doc has an oft-cited line: 'extra databases, queues, materialized views, and migrations must earn their long-term operational burden.' What does this 'cost' theme mean for design decisions?
  1. it means using the cheapest cloud services possible
  2. it means performance matters more than anything
  3. treating cost and operational simplicity as hard constraints, not 'add it because you can': every new moving part must first prove it's worth its long-term maintenance cost. It explains many restrained choices (API contracts require time windows/field selection/token pagination, no dangerous scan-all-history defaults) — achieving the goal with fewer parts is almost always the better architecture
  4. it means there should be no databases or queues at all
Show answer & explanation click to expand
Answer: C. This line elevates 'cost' from 'money' to an 'architectural constraint'. It targets not the cloud bill but the long-term operational burden of each moving part—an extra database to back up/monitor/upgrade, an extra queue to keep alive/debug, an extra materialized view to keep consistent, an extra migration to evolve carefully. So the default stance is subtraction: before adding anything, ask 'does it deserve to be here'. This explains Langfuse's restrained choices: the public API forces time windows, exposes field selection, uses token pagination, refuses the dangerous convenience of 'scan all history by default'. The underlying wisdom: achieving the same goal with fewer moving parts is almost always a more maintainable, less failure-prone architecture. Complexity is debt to be repaid; the cost theme is the ruler nudging you to borrow less.
💭 Open questions (no single right answer - just think or try)
  • This lesson notes: transferring Langfuse's design wisdom isn't about copying specific choices (ClickHouse, fan-out queues) but learning its way of deriving — first nail the 'goal+scale' premises (exploratory or known questions? write-heavy or read-heavy? one tenant or many? how much latency tolerable?), then subtract via 'every moving part must earn its burden'. Pick a system you know or are building, and walk this derivation: what are its goal+scale premises? Given those, which of the six themes apply and which are unnecessary? Does your current architecture have any parts that 'haven't earned their operational burden'?