🪢 Langfuse 图解教程Langfuse Visual Guide 第一部分 · 宏观全景Part 1 · The Big Picture 01 / 55
第一部分 · 宏观全景Part 1 · The Big Picture

Langfuse 是什么What is Langfuse

LLM 应用最让人头疼的地方是非确定性:同样的输入,今天和明天可能给出不一样的答案; 一次多步的 agent 调用里,到底是哪一步出了错、用了多少 token、花了 多少钱、慢在哪一环,光看几行日志几乎无从判断。 Langfuse 是一个开源的 LLM 工程平台 (langfuse/langfuse,MIT 许可)。一句话:它把你 LLM 应用里那些看不见的每一步, 变成可观测、可评估、可复现的数据——围绕开发 → 监控 → 评估 → 调试这一个闭环转。 说白了,它要回答的是工程师每天都在问、却很难答的三个问题:这次到底发生了什么?好不好?怎么变得更好?

🔌 生活类比
把一个 LLM 应用想成一架飞机:平时能飞,可一旦出状况,你两眼一抹黑。Langfuse 给它装上两样东西—— 黑匣子(飞行记录仪)和实验记录本。黑匣子把每一次模型调用、每一步检索、用到的 prompt、 消耗的 token、花掉的钱、各环节延迟,统统录下来;实验记录本则让你把“换个 prompt / 换个模型 / 改个参数”的每一次尝试 并排摆开对比。出了事能回放,要改进有依据——这就是 Langfuse 想给 LLM 应用的东西。说到底,黑匣子负责「看见」,实验记录本负责「比较」,两者合起来才让「凭感觉调」变成「拿数据改」。
Langfuse 站在你的应用「旁边」,而不是「里面」 你的 LLM 应用 + SDK / OpenTelemetry 事件 Langfuse 平台 🔭 可观测 trace · observation ✅ 评估 score · LLM 评审 📝 Prompt 管理 版本 · 线上拉取 🧪 数据集/实验 dataset · 对比 工程师 看·查·改 持续改进:换 prompt / 换模型 / 修链路
一个闭环:你的应用经 SDK 把运行事件送进 Langfuse;平台沉淀成 可观测 / 评估 / Prompt / 数据集 四类能力;工程师据此发现问题、改进应用,再把改进送回线上。Langfuse 始终在应用旁边观测,不侵入业务逻辑。

它到底是什么

一句话定义:一个把 LLM 应用的运行过程「数据化」的平台。它不替你训练模型,也不是又一个调用大模型的框架; 它做的事发生在你的应用之外——把每一次运行原原本本地记录、度量,并支撑你对它评估与改进。 官方把能力归为四块(在仓库里就是 web/src/features/ 下一个个目录):

能力解决什么问题仓库里的实现
🔭 可观测 Tracing看清每一次调用:调用树、输入输出、token、成本、延迟web/src/features/traces · tracing-tables
✅ 评估 Evaluation给输出评分:LLM-as-judge、人工标注、自定义代码评估web/src/features/evals · scores · annotation-queues
📝 Prompt 管理把 prompt 版本化,线上动态拉取、回滚、对比web/src/features/prompts
🧪 数据集与实验攒测试集、跑实验、并排对比不同版本web/src/features/datasets · experiments

要特别说清楚它不是什么:它不是一个像 LangChain 那样帮你「写」LLM 应用的框架——你用什么框架写应用都行, Langfuse 只在运行时把发生的事记下来;它也不是一个模型服务或网关,不替你「调」模型(虽然 Playground 能帮你试)。 它的位置是旁路观测者:你的应用照常跑,只是顺手把每一步「报告」给 Langfuse。

🗺️ 宏观理解
Langfuse 既是 SaaS 云服务(cloud.langfuse.com),也能完全自托管(self-host)——因为它开源、且只依赖 Postgres、ClickHouse、Redis、S3 这几样常见基础设施。本教程讲的就是自托管那一套代码:你在 langfuse/langfuse 仓库里看到的,正是云上跑的同一套。理解了这套代码,你既能把它用好,也能改它、给它提 PR。

三大数据支柱:trace / observation / score

Langfuse 里所有「可观测」数据,归根结底是三种东西。它们的领域模型定义在 packages/shared/src/domain/ 下的 traces.ts / observations.ts / scores.ts,分析存储则落在 ClickHouse 同名的三张表里 (packages/shared/clickhouse/migrations/unclustered/0001_traces、0002_observations、 0003_scores):

trace 是「外壳」,observation 是「主角」,score 是「评判」 trace · 一次用户问答 / session observation · span:handle-request span:retrieve-docs input: 用户问题 output: 命中的 3 段文档 generation:llm-call · gpt-4o model · 1,240 tok · $0.012 · 1.8s input: prompt + 文档 output: 回答 score:helpfulness 0.9(LLM 评审) ↑ score 可挂在 trace 或某个 observation 上
包含关系:一个 trace 把一次完整交互的所有步骤装在一起;每一步是一个 observation(可嵌套);score 则是对 trace 或某个 observation 的评分。注意架构取向——observation 才是主分析单元,trace 更像把它们串起来的「关联句柄」。
示例:一条 trace 实际长什么样 trace · id=chat-support_a1b2 · userId=u_42 totalCost≈$0.0042 · latency 1.93s SPAN retrieve-docs latency 120ms · input {query} · output {3 docs} startTime +0ms GENERATION answer model gpt-4o · modelParameters {temperature: 0.2} usageDetails {input: 512, output: 88} · costDetails {total: 0.0041} input {messages:[…]} · output 「Here is how…」 latency 1.8s EVENT cache-write latency 4ms · metadata {key: q:hash} startTime +1.92s scorename=helpfulness · value=0.9 · dataType=NUMERIC source=EVAL
示例 trace(字段对齐 domain/observations.ts 与 scores.ts;值为示例):一个 trace 串起三个 observation——SPAN 检索 → GENERATION 生成(带真实字段 model/usageDetails/costDetails/latency)→ EVENT 缓存写;score 以 NUMERIC 形式挂在 trace 上。这就是上图三支柱的具体一例。

🧵 trace · 关联句柄

代表一次完整的请求/会话。它本身信息不多(名字、用户、会话、标签、时间),主要作用是把同一次交互里的所有 observation 关联起来,方便你顺藤摸瓜。定义见 domain/traces.ts。

⭐ observation · 主角

一步具体操作。最核心是三型:GENERATION(一次 LLM 调用,带 model/token/cost)、 SPAN(一段有耗时的逻辑,如检索)、EVENT(一个瞬时事件);新版领域模型在此之上还细分出 AGENT / TOOL / CHAIN / RETRIEVER 等更具体的类型。它带着 input/output、时间、用量、成本等丰富属性,是查询与聚合的主体。

那第三支柱 score 呢?它是评分,可挂在 trace 或某个 observation 上。取值主要有三类—— NUMERIC 数值(如 0.9)、CATEGORICAL 分类(如「正确 / 错误」)、BOOLEAN 布尔(领域模型里还有 CORRECTION、TEXT); 来源有三种——EVAL(自动评审)、ANNOTATION(人工标注)、API(SDK 直接提交),见 packages/shared/src/domain/scores.ts。 正是 score 把「可观测」升级成「可评估」:有了可比较的分数,你才能量化「这次改动到底有没有让应用变好」,而不是凭感觉。score 的配置(数值范围、分类项) 存在 Postgres 的 ScoreConfig 里,分数本身则和 trace/observation 一样进 ClickHouse。

🔬 源码细节
observation 的类型在领域层是一个枚举:最初的 LegacyPrismaObservationType 只有 SPAN / EVENT / GENERATION 三种(该旧枚举如今已同步扩展到 10 种), 而当前的领域模型 packages/shared/src/domain/observations.ts 的 ObservationType 已扩展到 10 种 (新增 AGENT / TOOL / CHAIN / RETRIEVER / EMBEDDING / EVALUATOR / GUARDRAIL)。ClickHouse 的 observations 表把 model、 provided_usage_details、cost_details、prompt_id 等都内联成一行的列与 Map—— 这正是「宽事件」的样子(下一节展开)。

把三大支柱落到真实类型上——它们都是 Zod schema,定义在 packages/shared/src/domain/:

packages/shared/src/domain/{traces,observations,scores}.ts三大支柱
// ① trace = 一次完整调用的「根」(traces.ts)
export const TraceDomain = z.object({
  id: z.string(),  name: z.string().nullable(),  timestamp: z.date(),
  input: jsonSchema.nullable(),   output: jsonSchema.nullable(),
  sessionId: z.string().nullable(),  userId: z.string().nullable(),
  projectId: z.string(),                 // 每条都带 projectId:硬隔离键
});

// ② observation = trace 里的每一步,10 种类型(observations.ts)
export const ObservationType = {
  SPAN, EVENT, GENERATION, AGENT, TOOL,
  CHAIN, RETRIEVER, EVALUATOR, EMBEDDING, GUARDRAIL,
} as const;

// ③ score = 挂在 trace/observation 上的评分(scores.ts)
//   NUMERIC(value) · CATEGORICAL/BOOLEAN(stringValue) · CORRECTION

看得见,才改得动:一个排错场景

抽象的定义不如一个具体场景。假设你做了个客服问答机器人,用户反馈「它偶尔会答非所问」。 没有 Langfuse 时,你只有应用日志里几行 prompt=... response=...,既不知道是哪一步错了, 也复现不出来。接上 Langfuse 后,排查会变成这样:

① 在 trace 列表里按「用户点了踩」或「低分」过滤,几秒就翻出那几条出问题的 trace; ② 点开一条,observation 树一目了然——原来是 retrieve-docs 这一步召回了不相关的文档, 而不是模型本身的问题;③ 看这条 generation 的 input,确认被污染的上下文正是那几段错误文档; ④ 你把这条 trace 加进一个数据集,改完检索逻辑后跑实验对比新旧版本,用评分确认真的修好了。 整个过程,靠的就是「每一步都被原样记下来」。

⚠️ 没有可观测时的常见坑
只盯着「模型输出不好」很容易错怪模型。真实世界里大量「LLM 答得差」其实是检索召回错了、 prompt 拼错了、上游某步超时返回了空。这些只有把整条链路摊开才看得见—— 这就是为什么 Langfuse 把重点放在 observation 这一「步」,而不只是最终答案。

它由什么组成

把镜头拉到系统层面,Langfuse 是一个 pnpm + Turbo 的 monorepo,运行时主要分两个容器,外加四类存储 (见根目录 docker-compose.yml 与 AGENTS.md 的项目结构说明):

两个容器(web · worker)+ 四类存储 SDK / OTel 你的应用 web · Next.js UI · tRPC · REST Redis 队列 BullMQ worker 队列消费 · 后台处理 Postgres 配置 / 元数据 ClickHouse 宽事件 · 列存 S3 / blob 原始事件 · 媒体 Redis 缓存 worker 写入 web 读写配置 / 读分析
数据怎么流:SDK 把事件发给 web 的公共 API,web 快速入队 Redis;worker 消费队列、做合并与成本计算,把结果写进 ClickHouse(宽事件)与 S3(原始事件/媒体);UI 再由 web 从 Postgres(配置)和 ClickHouse(分析)读出来渲染。摄取走异步,所以 web 端永远轻快。
packages/shared共享内核
领域模型、Prisma/ClickHouse 访问、队列契约、仓储层。被 web 与 worker 共用;它不反向依赖任何一方。
webNext.js 应用
UI(Pages Router)+ 给 UI 用的 tRPC + 给 SDK 用的公共 REST API。依赖 @langfuse/shared 与 @langfuse/ee。
worker后台处理
BullMQ 队列消费者:摄取、评估、实验、导出、删除、集成……几十个队列。只依赖 @langfuse/shared。
ee企业版
企业功能与 license 校验,被 web 引用;其余皆 MIT。依赖方向严格单向:web/worker/ee → shared。

为什么把核心抽进 packages/shared、让 web 与 worker 都只单向依赖它?因为 web 与 worker 是两种完全不同的负载: web 面向用户、追求低延迟、要能随时重启;worker 啃重活、要可重试、按队列伸缩。把共享的领域模型、存储访问、队列契约 收进一个「谁都不反向依赖」的窄腰,两边就能各自独立演进、独立部署、独立扩容,同时又共用同一套契约、不会各写一套导致漂移。 这条单向依赖纪律(web/worker/ee → shared,而 shared 谁都不依赖),是整个 monorepo 能长期维护的关键。

为什么这么设计:宽事件 / 可观测性 2.0

Langfuse 的存储与查询路径,都被一条设计原则贯穿——把 observation 当作主分析单元,用「宽事件」记录一切 (明确写在 .agents/ARCHITECTURE_PRINCIPLES.md 里)。所谓宽事件,就是不把信息拆成零散的 指标(metrics)、日志(logs)、链路(traces)再事后拼接,而是把一步操作的所有高基数上下文 (模型、用量、成本、prompt、用户、标签、环境……)原样塞进同一行。

对比一下传统的「可观测性三件套」:指标(metrics)是预先聚合好的数字(如 QPS、平均延迟),省空间但丢了细节, 事后想按「某个用户 + 某个模型 + 某段时间」下钻就抓瞎;日志(logs)是零散的文本行,细节有了却难以聚合与关联; 链路(traces)能看调用结构,却往往不带业务属性。宽事件的思路是把这三者合一:每条 observation 既是一条 带结构的「链路节点」,又自带可聚合的数值(token/cost/latency),还携带任意业务字段(user/tag/metadata)。于是「下钻」和「聚合」 不再是两套系统,而是对同一张宽表的不同查询。这也是 Langfuse 选择 ClickHouse 的根本原因——列存天生擅长 「在一张宽表上做时间窗口扫描 + 任意维度聚合」。

🎯 设计取舍
为什么宁可让每行很「宽」,也不拆成规整的多张表? 因为可观测的本质是事后回答你当时没想到的问题 (unknown unknowns)。如果上线前就把字段拆进固定指标,那你只能查预设好的维度;而把高基数上下文整行留下, 你就能在事后任意切片、分组、过滤。代价是单行更大、存储更多、写入要更聪明——Langfuse 用列存 + ZSTD 压缩 + 仅在详情页才读大字段来平衡(后续课展开)。用一点存储,换无限的提问自由。
开发 Develop
接 SDK·埋点
→
监控 Monitor
看 trace·成本·延迟
→
评估 Evaluate
评分·标注·实验
→
调试 Debug
定位坏 case·改
↺
再开发
闭环

这套教程怎么读

本指南沿着数据走:先打地基,再顺着「写入 → 读取」两条主链路,然后逐个拆功能子系统,最后讲平台与运维。 每一课都配真实源码对应(文件:行号)、手绘图与设计取舍框,并以一段自测题收尾。 本指南有一条硬承诺:每一处技术论断都对应 langfuse/langfuse 仓库里真实存在的源码,绝不臆造; 引用以「文件:行号」标注,方便你打开仓库逐一核对。看不懂的地方,顺着引用去读源码,往往比读十段解释都管用。

1

宏观与基础(第 1–11 课)

数据模型、双存储、ClickHouse 宽事件表、多租户、部署拓扑。

2

摄取链路 · 写(第 12–19 课)

从 SDK 到 ClickHouse:API、队列、合并、token/成本、OTel、媒体。

3

查询链路 · 读(第 20–27 课)

tRPC、仓储层、过滤/搜索、列表、trace 详情、公共 API。

4

功能子系统(第 28–47 课)

评估/评分、数据集/实验、Prompt/Playground、仪表盘/成本、自动化/集成。

5

平台·运维·设计专题(第 48–55 课)

鉴权/RBAC、EE、运维、删除、构建测试,最后是设计专题与终章。

🎯 本课要点
  • Langfuse 是开源 LLM 工程平台(MIT):围绕开发→监控→评估→调试闭环,把 LLM 应用的每一步变成数据。
  • 三大支柱:trace(关联句柄)⊃ observation(主分析单元,分 GENERATION/SPAN/EVENT)+ score(评分)。
  • 两容器四存储:web + worker;Postgres(配置) + ClickHouse(宽事件) + Redis(队列/缓存) + S3(原始/媒体)。
  • 摄取是异步的:SDK→web 入队→worker 处理→写存储,所以 web 端轻快、能抗尖峰。
  • 设计灵魂是「宽事件 / 可观测性 2.0」:用高基数整行换取事后任意提问的自由。

🧪 自测 · 想一想为什么这么设计

1. Langfuse 把 observation(而不是 trace)当作“主分析单元”。这个定位最主要带来什么不同?
  1. 它取消了 trace 概念,只保留扁平日志
  2. 它把所有数据压缩成一个 trace 大对象,查询时整体读出
  3. 它要求每个 trace 只能有一个 observation
  4. 每个 LLM 调用/检索步骤本身就是一条带丰富属性的宽事件,可独立切片聚合;trace 只是把它们关联起来的句柄
看答案与解析 点击展开
答案:D。这是“可观测性 2.0 / 宽事件”的核心:把高基数上下文留在每条 observation 上,用户就能在不预设问题的前提下任意切片、分组、过滤。trace 仍在,但它是关联句柄,不是唯一入口(见 .agents/ARCHITECTURE_PRINCIPLES.md)。
2. Langfuse 为什么要同时用 Postgres 和 ClickHouse 两套数据库,而不是只用一套?
  1. 因为 Postgres 不能存字符串
  2. 纯粹为了用更多技术显得复杂
  3. ClickHouse 用来做用户登录,Postgres 用来跑 ML
  4. 事务性配置/元数据(项目、用户、prompt、配置)适合 Postgres;海量、可追加的宽事件(trace/observation/score)适合列存的 ClickHouse 做时间窗口扫描与聚合
看答案与解析 点击展开
答案:D。两类负载的访问模式根本不同:元数据是小数据量、强一致、频繁更新——交给 Postgres;遥测是大数据量、写多于改、按时间和项目聚合——交给 ClickHouse(宽事件、列存、ReplacingMergeTree)。用对存储,热路径才不会被拖垮。
3. Langfuse 的摄取为什么走“SDK → 公共 API → 队列 → worker → ClickHouse”这条异步链路,而不是 API 直接写 ClickHouse?
  1. 队列是用来加密数据的
  2. 为了让写入更慢以节省成本
  3. 因为 ClickHouse 只接受来自 worker 的连接
  4. 解耦与削峰:API 只需快速收下事件并入队,重活(合并、token/成本计算、批量写)交给 worker 异步处理,既抗流量尖峰又能批量高效写入
看答案与解析 点击展开
答案:D。把“接收”和“处理”分开,是高吞吐遥测系统的经典做法。API 端轻量、可水平扩展、对 SDK 低延迟;worker 端可重试、可批量、可分片(主/次队列隔离高吞吐项目)。代价是“最终一致”——数据有秒级处理延迟,这正是 §设计取舍要权衡的。
💭 发散思考(没有标准答案,动手或动脑想想)
  • 如果让你给一个自研的 LLM 应用加“可观测性”,你会先记录哪三类信息到每条 observation 上?为什么这三类最能帮你事后 debug?

The hardest thing about LLM apps is non-determinism: the same input can give different answers today and tomorrow; and inside one multi-step agent run, figuring out which step went wrong, how many tokens it burned, how much it cost, or where the latency went is nearly impossible from a few log lines. Langfuse is an open-source LLM engineering platform (langfuse/langfuse, MIT-licensed). In one line: it turns every invisible step inside your LLM app into observable, evaluable, reproducible data — around one loop: develop → monitor → evaluate → debug. Put plainly, it answers the three questions engineers ask daily but struggle to answer: what actually happened this time? was it any good? how do I make it better?

🔌 Analogy
Think of an LLM app as an airplane: it flies fine until something goes wrong — and then you're blind. Langfuse bolts on two things: a black box (flight recorder) and a lab notebook. The black box records every model call, every retrieval step, the prompt used, the tokens spent, the money burned and the latency of each leg; the lab notebook lets you line up every “try another prompt / model / parameter” experiment side by side. When things break you can replay; when you want to improve you have evidence — that's what Langfuse gives an LLM app. In short, the black box is for "seeing" and the lab notebook is for "comparing"; together they turn "tuning by vibes" into "improving with data".
Langfuse sits BESIDE your app, not INSIDE it Your LLM app + SDK / OpenTelemetry events Langfuse platform 🔭 Observe trace · observation ✅ Evaluate score · LLM judge 📝 Prompts version · fetch live 🧪 Datasets dataset · compare Engineer see·query·fix improve: swap prompt / model / fix the chain
One loop: your app ships run-time events through the SDK into Langfuse; the platform distills them into four capabilities — observe / evaluate / prompts / datasets; engineers find problems, improve the app, and push improvements back to production. Langfuse always observes from beside the app, never intruding on business logic.

What it actually is

A one-line definition: a platform that turns an LLM app's run-time behavior into data. It does not train models for you, and it is not yet another framework for calling an LLM; its work happens outside your app — faithfully recording and measuring every run, and supporting you to evaluate and improve it. The official capabilities map to one folder each under web/src/features/:

CapabilityProblem it solvesWhere it lives
🔭 TracingSee every call: the call tree, I/O, tokens, cost, latencyweb/src/features/traces · tracing-tables
✅ EvaluationScore outputs: LLM-as-judge, human annotation, custom codeweb/src/features/evals · scores · annotation-queues
📝 Prompt mgmtVersion prompts; fetch live, roll back, compareweb/src/features/prompts
🧪 Datasets & experimentsBuild test sets, run experiments, compare versionsweb/src/features/datasets · experiments

It helps to say what it is not: it is not a framework like LangChain that helps you write an LLM app — use whatever framework you like; Langfuse only records what happens at run time. It is also not a model server or gateway that calls models for you (though the Playground lets you experiment). Its place is the side-channel observer: your app runs as usual and simply "reports" each step to Langfuse.

🗺️ Big picture
Langfuse is both a SaaS cloud (cloud.langfuse.com) and fully self-hostable — because it's open source and depends only on commodity infra: Postgres, ClickHouse, Redis, S3. This guide explains the self-host codebase: what you see in langfuse/langfuse is the same stack that runs in the cloud. Understand this code and you can both use it well and change it / send a PR.

The three pillars: trace / observation / score

Every piece of "observable" data in Langfuse boils down to three things. Their domain models live under packages/shared/src/domain/ (traces.ts / observations.ts / scores.ts); their analytical storage is three same-named ClickHouse tables (packages/shared/clickhouse/migrations/unclustered/0001_traces, 0002_observations, 0003_scores):

trace is the shell, observation is the star, score is the verdict trace · one user interaction / session observation · span: handle-request span: retrieve-docs input: user question output: 3 matched chunks generation: llm-call · gpt-4o model · 1,240 tok · $0.012 · 1.8s input: prompt + docs output: answer score: helpfulness 0.9 (LLM judge) ↑ a score attaches to a trace or an observation
Containment: a trace wraps all steps of one interaction; each step is an observation (which can nest); a score rates the trace or one observation. Note the architectural stance — the observation is the primary analytical unit; the trace is more of a "correlation handle" stringing them together.
Example: what a trace actually looks like trace · id=chat-support_a1b2 · userId=u_42 totalCost≈$0.0042 · latency 1.93s SPAN retrieve-docs latency 120ms · input {query} · output {3 docs} startTime +0ms GENERATION answer model gpt-4o · modelParameters {temperature: 0.2} usageDetails {input: 512, output: 88} · costDetails {total: 0.0041} input {messages:[…]} · output 「Here is how…」 latency 1.8s EVENT cache-write latency 4ms · metadata {key: q:hash} startTime +1.92s scorename=helpfulness · value=0.9 · dataType=NUMERIC source=EVAL
Example trace (fields per domain/observations.ts and scores.ts; values illustrative): one trace strings three observations — SPAN retrieve → GENERATION answer (with real fields model/usageDetails/costDetails/latency) → EVENT cache-write; a score attaches to the trace as NUMERIC. A concrete instance of the three pillars above.

🧵 trace · correlation handle

Represents one complete request/session. It carries little itself (name, user, session, tags, time); its main job is to correlate all the observations of one interaction so you can follow the thread. See domain/traces.ts.

⭐ observation · the star

One concrete step. The three core types are GENERATION (an LLM call, with model/token/cost), SPAN (a timed unit of logic, e.g. retrieval) and EVENT (an instantaneous event); the current domain model further specializes these into AGENT / TOOL / CHAIN / RETRIEVER and more. It carries rich attributes — input/output, timing, usage, cost — and is the subject of querying and aggregation.

And the third pillar, score? It is a rating attached to a trace or an observation. Its value is mainly one of three types — NUMERIC (e.g. 0.9), CATEGORICAL (e.g. "correct / wrong"), BOOLEAN (the domain model also has CORRECTION and TEXT); from three sources — EVAL (auto-judge), ANNOTATION (human), API (submitted via SDK), see packages/shared/src/domain/scores.ts. Score is what upgrades "observable" into "evaluable": with comparable numbers you can quantify "did this change actually make the app better" instead of guessing. A score's config (numeric range, categories) lives in Postgres as ScoreConfig, while the scores themselves go into ClickHouse like traces/observations.

🔬 Source detail
The observation type is an enum in the domain layer: the original LegacyPrismaObservationType had only SPAN / EVENT / GENERATION (that legacy enum has since been extended to 10 too), while the current domain model packages/shared/src/domain/observations.ts defines an ObservationType with 10 values (adding AGENT / TOOL / CHAIN / RETRIEVER / EMBEDDING / EVALUATOR / GUARDRAIL). The ClickHouse observations table inlines model, provided_usage_details, cost_details, prompt_id and more as columns and Maps on a single row — exactly what a "wide event" looks like (next section).

Grounding the three pillars in real types — all are Zod schemas in packages/shared/src/domain/:

packages/shared/src/domain/{traces,observations,scores}.tsthree pillars
// (1) trace = the "root" of one full call (traces.ts)
export const TraceDomain = z.object({
  id: z.string(),  name: z.string().nullable(),  timestamp: z.date(),
  input: jsonSchema.nullable(),   output: jsonSchema.nullable(),
  sessionId: z.string().nullable(),  userId: z.string().nullable(),
  projectId: z.string(),                 // every row carries projectId: the isolation key
});

// (2) observation = each step inside a trace, 10 types (observations.ts)
export const ObservationType = {
  SPAN, EVENT, GENERATION, AGENT, TOOL,
  CHAIN, RETRIEVER, EVALUATOR, EMBEDDING, GUARDRAIL,
} as const;

// (3) score = a grade attached to a trace/observation (scores.ts)
//   NUMERIC(value) · CATEGORICAL/BOOLEAN(stringValue) · CORRECTION

You can only fix what you can see: a debugging scene

An abstract definition is weaker than a concrete scene. Say you built a customer-support bot and users report "it sometimes answers off-topic". Without Langfuse you only have a few app-log lines like prompt=... response=... — you can't tell which step failed, and you can't reproduce it. With Langfuse, triage looks like this:

① In the trace list, filter by "user thumbs-down" or "low score" and surface the offending traces in seconds; ② open one, and the observation tree makes it obvious — it was the retrieve-docs step that fetched irrelevant documents, not the model itself; ③ inspect that generation's input and confirm the polluted context is exactly those wrong docs; ④ add this trace to a dataset, fix the retrieval logic, run an experiment comparing old vs new, and use a score to confirm it's truly fixed. The whole flow rests on "every step was recorded as-is".

⚠️ The trap without observability
Staring only at "the model output is bad" makes it easy to blame the model wrongly. In the real world a huge share of "the LLM answered poorly" is actually bad retrieval, a mis-assembled prompt, or an upstream step that timed out and returned empty. You only see these by laying out the whole chain — which is why Langfuse centers on the observation (the "step"), not just the final answer.

What it's made of

Zoom out to the system level: Langfuse is a pnpm + Turbo monorepo, running as two main containers plus four kinds of storage (see the root docker-compose.yml and the project structure in AGENTS.md):

Two containers (web · worker) + four stores SDK / OTel your app web · Next.js UI · tRPC · REST Redis queue BullMQ worker queue consumer · jobs Postgres config / metadata ClickHouse wide events · columnar S3 / blob raw events · media Redis cache worker writes web reads/writes config / reads analytics
How data flows: the SDK posts events to web's public API, which quickly enqueues into Redis; the worker consumes the queue, merges and computes cost, and writes ClickHouse (wide events) and S3 (raw events/media); the UI is then rendered by web reading from Postgres (config) and ClickHouse (analytics). Ingestion is async, so the web tier stays light.
packages/sharedshared core
Domain models, Prisma/ClickHouse access, queue contracts, repositories. Used by both web and worker; it depends on neither.
webNext.js app
UI (Pages Router) + tRPC for the UI + a public REST API for SDKs. Depends on @langfuse/shared and @langfuse/ee.
workerbackground jobs
BullMQ queue consumers: ingestion, evaluation, experiments, exports, deletion, integrations… dozens of queues. Depends only on @langfuse/shared.
eeenterprise
Enterprise features + license check, used by web; everything else is MIT. Dependency direction is strictly one-way: web/worker/ee → shared.

Why pull the core into packages/shared and have both web and worker depend on it one-way only? Because web and worker are two very different workloads: web faces users, wants low latency and must restart anytime; the worker chews through heavy jobs, must be retryable and scales by queue. Putting the shared domain models, storage access and queue contracts into a narrow waist that nothing depends on backwards lets the two sides evolve, deploy and scale independently while sharing one contract that can't drift. That one-way dependency discipline (web/worker/ee → shared, and shared depends on none) is key to keeping the whole monorepo maintainable.

Why this design: wide events / Observability 2.0

Langfuse's storage and query paths are all threaded by one design principle — treat the observation as the primary analytical unit and record everything as "wide events" (stated explicitly in .agents/ARCHITECTURE_PRINCIPLES.md). A wide event means you do not shatter information into separate metrics, logs and traces to be reassembled later; instead you cram all the high-cardinality context of one step (model, usage, cost, prompt, user, tags, environment…) onto a single row.

Contrast the classic "three pillars of observability": metrics are pre-aggregated numbers (QPS, average latency) — cheap but they lose detail, so drilling into "this user + this model + this time window" later is hopeless; logs are scattered text lines — detail is there but they're hard to aggregate and correlate; traces show call structure but usually carry no business attributes. The wide-event idea fuses all three: each observation is at once a structured "trace node", a bag of aggregatable numbers (token/cost/latency), and a carrier of arbitrary business fields (user/tag/metadata). So "drill down" and "aggregate" stop being two systems and become different queries over one wide table. That's also the root reason Langfuse picked ClickHouse — columnar storage is built for "time-bounded scans + arbitrary-dimension aggregation over a wide table".

🎯 Design trade-off
Why keep each row "wide" instead of splitting it into tidy normalized tables? Because the essence of observability is answering questions you didn't think of at the time (unknown unknowns). If you split fields into fixed metrics before shipping, you can only query the dimensions you predefined; keep the high-cardinality context on the row and you can slice, group and filter arbitrarily after the fact. The cost is bigger rows, more storage and smarter writes — Langfuse balances it with columnar storage + ZSTD compression + reading big fields only on the detail page (later lessons). Trade a little storage for unlimited freedom to ask questions.
Develop
wire SDK · instrument
→
Monitor
traces · cost · latency
→
Evaluate
score · annotate · experiment
→
Debug
find bad cases · fix
↺
Develop again
loop

How to read this guide

This guide follows the data: lay the foundations first, then walk the two main paths "write → read", then take apart the feature subsystems one by one, and finally cover platform & operations. Every lesson ships real source citations (file:line), hand-drawn diagrams and a design trade-off box, and closes with a short self-test. This guide makes one hard promise: every technical claim corresponds to source that really exists in langfuse/langfuse, never invented; citations carry "file:line" so you can open the repo and check. Where something is unclear, following the citation into the source often beats ten paragraphs of explanation.

1

Big picture & foundations (L01–11)

Data model, dual store, ClickHouse wide-event tables, multi-tenancy, deployment topology.

2

Ingestion · write (L12–19)

SDK to ClickHouse: API, queue, merge, token/cost, OTel, media.

3

Query · read (L20–27)

tRPC, repositories, filtering/search, lists, trace detail, public API.

4

Feature subsystems (L28–47)

Eval/scoring, datasets/experiments, prompts/playground, dashboards/cost, automation/integrations.

5

Platform · ops · design themes (L48–55)

Auth/RBAC, EE, operations, deletion, build & test, then design themes and the capstone.

🎯 Key points
  • Langfuse is an open-source LLM engineering platform (MIT): a develop → monitor → evaluate → debug loop that turns every step of an LLM app into data.
  • Three pillars: trace (correlation handle) ⊃ observation (primary unit; GENERATION/SPAN/EVENT) + score (rating).
  • Two containers, four stores: web + worker; Postgres (config) + ClickHouse (wide events) + Redis (queue/cache) + S3 (raw/media).
  • Ingestion is async: SDK → web enqueues → worker processes → writes storage, so web stays light and absorbs spikes.
  • The soul of the design is "wide events / Observability 2.0": trade high-cardinality rows for the freedom to ask any question later.

🧪 Self-test - think about the design

1. Langfuse treats the observation (not the trace) as the primary analytical unit. What is the main difference this brings?
  1. It removes the trace concept and keeps only flat logs
  2. It compresses everything into one big trace object read whole at query time
  3. It requires each trace to have exactly one observation
  4. Each LLM call/retrieval step is itself a richly-attributed wide event you can slice & aggregate; the trace is just a handle that correlates them
Show answer & explanation click to expand
Answer: D. This is the heart of 'Observability 2.0 / wide events': keep high-cardinality context on each observation so users can slice, group and filter without predefining questions. The trace still exists, but as a correlation handle, not the only entry point (see .agents/ARCHITECTURE_PRINCIPLES.md).
2. Why does Langfuse use both Postgres and ClickHouse instead of just one database?
  1. Because Postgres cannot store strings
  2. Purely to look more complex by using more tech
  3. ClickHouse handles logins while Postgres runs ML
  4. Transactional config/metadata (projects, users, prompts, settings) fits Postgres; high-volume append-oriented wide events (trace/observation/score) fit columnar ClickHouse for time-bounded scans & aggregation
Show answer & explanation click to expand
Answer: D. The two workloads have fundamentally different access patterns: metadata is small, strongly-consistent and frequently updated — Postgres; telemetry is huge, write-mostly and aggregated by time/project — ClickHouse (wide events, columnar, ReplacingMergeTree). Matching storage to pattern keeps the hot path fast.
3. Why does ingestion go 'SDK → public API → queue → worker → ClickHouse' asynchronously instead of the API writing ClickHouse directly?
  1. The queue is there to encrypt the data
  2. To make writes slower and save cost
  3. Because ClickHouse only accepts connections from the worker
  4. Decoupling & smoothing: the API just accepts and enqueues fast, while the heavy work (merge, token/cost compute, batched writes) runs async in the worker — absorbing spikes and writing in efficient batches
Show answer & explanation click to expand
Answer: D. Separating 'accept' from 'process' is the classic shape for high-throughput telemetry. The API stays lightweight, horizontally scalable and low-latency to the SDK; the worker can retry, batch and shard (primary/secondary queues isolate high-throughput projects). The cost is eventual consistency — a few seconds of processing lag, exactly the trade-off to weigh.
💭 Open questions (no single right answer - just think or try)
  • If you were adding 'observability' to your own LLM app, which three kinds of data would you record on each observation first, and why would those three help you debug later?