LLM 应用最让人头疼的地方是非确定性:同样的输入,今天和明天可能给出不一样的答案;
一次多步的 agent 调用里,到底是哪一步出了错、用了多少 token、花了
多少钱、慢在哪一环,光看几行日志几乎无从判断。
Langfuse 是一个开源的 LLM 工程平台
(langfuse/langfuse,MIT 许可)。一句话:它把你 LLM 应用里那些看不见的每一步,
变成可观测、可评估、可复现的数据——围绕开发 → 监控 → 评估 → 调试这一个闭环转。
说白了,它要回答的是工程师每天都在问、却很难答的三个问题:这次到底发生了什么?好不好?怎么变得更好?
一句话定义:一个把 LLM 应用的运行过程「数据化」的平台。它不替你训练模型,也不是又一个调用大模型的框架;
它做的事发生在你的应用之外——把每一次运行原原本本地记录、度量,并支撑你对它评估与改进。
官方把能力归为四块(在仓库里就是 web/src/features/ 下一个个目录):
| 能力 | 解决什么问题 | 仓库里的实现 |
|---|---|---|
| 🔭 可观测 Tracing | 看清每一次调用:调用树、输入输出、token、成本、延迟 | web/src/features/traces · tracing-tables |
| ✅ 评估 Evaluation | 给输出评分:LLM-as-judge、人工标注、自定义代码评估 | web/src/features/evals · scores · annotation-queues |
| 📝 Prompt 管理 | 把 prompt 版本化,线上动态拉取、回滚、对比 | web/src/features/prompts |
| 🧪 数据集与实验 | 攒测试集、跑实验、并排对比不同版本 | web/src/features/datasets · experiments |
要特别说清楚它不是什么:它不是一个像 LangChain 那样帮你「写」LLM 应用的框架——你用什么框架写应用都行, Langfuse 只在运行时把发生的事记下来;它也不是一个模型服务或网关,不替你「调」模型(虽然 Playground 能帮你试)。 它的位置是旁路观测者:你的应用照常跑,只是顺手把每一步「报告」给 Langfuse。
langfuse/langfuse
仓库里看到的,正是云上跑的同一套。理解了这套代码,你既能把它用好,也能改它、给它提 PR。
Langfuse 里所有「可观测」数据,归根结底是三种东西。它们的领域模型定义在
packages/shared/src/domain/ 下的 traces.ts / observations.ts /
scores.ts,分析存储则落在 ClickHouse 同名的三张表里
(packages/shared/clickhouse/migrations/unclustered/0001_traces、0002_observations、
0003_scores):
domain/observations.ts 与 scores.ts;值为示例):一个 trace 串起三个 observation——SPAN 检索 → GENERATION 生成(带真实字段 model/usageDetails/costDetails/latency)→ EVENT 缓存写;score 以 NUMERIC 形式挂在 trace 上。这就是上图三支柱的具体一例。代表一次完整的请求/会话。它本身信息不多(名字、用户、会话、标签、时间),主要作用是把同一次交互里的所有
observation 关联起来,方便你顺藤摸瓜。定义见 domain/traces.ts。
一步具体操作。最核心是三型:GENERATION(一次 LLM 调用,带 model/token/cost)、
SPAN(一段有耗时的逻辑,如检索)、EVENT(一个瞬时事件);新版领域模型在此之上还细分出
AGENT / TOOL / CHAIN / RETRIEVER 等更具体的类型。它带着 input/output、时间、用量、成本等丰富属性,是查询与聚合的主体。
那第三支柱 score 呢?它是评分,可挂在 trace 或某个 observation 上。取值主要有三类——
NUMERIC 数值(如 0.9)、CATEGORICAL 分类(如「正确 / 错误」)、BOOLEAN 布尔(领域模型里还有 CORRECTION、TEXT);
来源有三种——EVAL(自动评审)、ANNOTATION(人工标注)、API(SDK 直接提交),见 packages/shared/src/domain/scores.ts。
正是 score 把「可观测」升级成「可评估」:有了可比较的分数,你才能量化「这次改动到底有没有让应用变好」,而不是凭感觉。score 的配置(数值范围、分类项)
存在 Postgres 的 ScoreConfig 里,分数本身则和 trace/observation 一样进 ClickHouse。
LegacyPrismaObservationType 只有
SPAN / EVENT / GENERATION 三种(该旧枚举如今已同步扩展到 10 种),
而当前的领域模型 packages/shared/src/domain/observations.ts 的 ObservationType 已扩展到 10 种
(新增 AGENT / TOOL / CHAIN / RETRIEVER / EMBEDDING / EVALUATOR / GUARDRAIL)。ClickHouse 的 observations 表把 model、
provided_usage_details、cost_details、prompt_id 等都内联成一行的列与 Map——
这正是「宽事件」的样子(下一节展开)。
把三大支柱落到真实类型上——它们都是 Zod schema,定义在 packages/shared/src/domain/:
// ① trace = 一次完整调用的「根」(traces.ts) export const TraceDomain = z.object({ id: z.string(), name: z.string().nullable(), timestamp: z.date(), input: jsonSchema.nullable(), output: jsonSchema.nullable(), sessionId: z.string().nullable(), userId: z.string().nullable(), projectId: z.string(), // 每条都带 projectId:硬隔离键 }); // ② observation = trace 里的每一步,10 种类型(observations.ts) export const ObservationType = { SPAN, EVENT, GENERATION, AGENT, TOOL, CHAIN, RETRIEVER, EVALUATOR, EMBEDDING, GUARDRAIL, } as const; // ③ score = 挂在 trace/observation 上的评分(scores.ts) // NUMERIC(value) · CATEGORICAL/BOOLEAN(stringValue) · CORRECTION
抽象的定义不如一个具体场景。假设你做了个客服问答机器人,用户反馈「它偶尔会答非所问」。
没有 Langfuse 时,你只有应用日志里几行 prompt=... response=...,既不知道是哪一步错了,
也复现不出来。接上 Langfuse 后,排查会变成这样:
① 在 trace 列表里按「用户点了踩」或「低分」过滤,几秒就翻出那几条出问题的 trace;
② 点开一条,observation 树一目了然——原来是 retrieve-docs 这一步召回了不相关的文档,
而不是模型本身的问题;③ 看这条 generation 的 input,确认被污染的上下文正是那几段错误文档;
④ 你把这条 trace 加进一个数据集,改完检索逻辑后跑实验对比新旧版本,用评分确认真的修好了。
整个过程,靠的就是「每一步都被原样记下来」。
把镜头拉到系统层面,Langfuse 是一个 pnpm + Turbo 的 monorepo,运行时主要分两个容器,外加四类存储
(见根目录 docker-compose.yml 与 AGENTS.md 的项目结构说明):
@langfuse/shared 与 @langfuse/ee。@langfuse/shared。web/worker/ee → shared。为什么把核心抽进 packages/shared、让 web 与 worker 都只单向依赖它?因为 web 与 worker 是两种完全不同的负载:
web 面向用户、追求低延迟、要能随时重启;worker 啃重活、要可重试、按队列伸缩。把共享的领域模型、存储访问、队列契约
收进一个「谁都不反向依赖」的窄腰,两边就能各自独立演进、独立部署、独立扩容,同时又共用同一套契约、不会各写一套导致漂移。
这条单向依赖纪律(web/worker/ee → shared,而 shared 谁都不依赖),是整个 monorepo 能长期维护的关键。
Langfuse 的存储与查询路径,都被一条设计原则贯穿——把 observation 当作主分析单元,用「宽事件」记录一切
(明确写在 .agents/ARCHITECTURE_PRINCIPLES.md 里)。所谓宽事件,就是不把信息拆成零散的
指标(metrics)、日志(logs)、链路(traces)再事后拼接,而是把一步操作的所有高基数上下文
(模型、用量、成本、prompt、用户、标签、环境……)原样塞进同一行。
对比一下传统的「可观测性三件套」:指标(metrics)是预先聚合好的数字(如 QPS、平均延迟),省空间但丢了细节, 事后想按「某个用户 + 某个模型 + 某段时间」下钻就抓瞎;日志(logs)是零散的文本行,细节有了却难以聚合与关联; 链路(traces)能看调用结构,却往往不带业务属性。宽事件的思路是把这三者合一:每条 observation 既是一条 带结构的「链路节点」,又自带可聚合的数值(token/cost/latency),还携带任意业务字段(user/tag/metadata)。于是「下钻」和「聚合」 不再是两套系统,而是对同一张宽表的不同查询。这也是 Langfuse 选择 ClickHouse 的根本原因——列存天生擅长 「在一张宽表上做时间窗口扫描 + 任意维度聚合」。
本指南沿着数据走:先打地基,再顺着「写入 → 读取」两条主链路,然后逐个拆功能子系统,最后讲平台与运维。
每一课都配真实源码对应(文件:行号)、手绘图与设计取舍框,并以一段自测题收尾。
本指南有一条硬承诺:每一处技术论断都对应 langfuse/langfuse 仓库里真实存在的源码,绝不臆造;
引用以「文件:行号」标注,方便你打开仓库逐一核对。看不懂的地方,顺着引用去读源码,往往比读十段解释都管用。
数据模型、双存储、ClickHouse 宽事件表、多租户、部署拓扑。
从 SDK 到 ClickHouse:API、队列、合并、token/成本、OTel、媒体。
tRPC、仓储层、过滤/搜索、列表、trace 详情、公共 API。
评估/评分、数据集/实验、Prompt/Playground、仪表盘/成本、自动化/集成。
鉴权/RBAC、EE、运维、删除、构建测试,最后是设计专题与终章。
observation(而不是 trace)当作“主分析单元”。这个定位最主要带来什么不同?.agents/ARCHITECTURE_PRINCIPLES.md)。
The hardest thing about LLM apps is non-determinism: the same input can give different
answers today and tomorrow; and inside one multi-step agent run, figuring out which step
went wrong, how many tokens it burned, how much it cost, or where the
latency went is nearly impossible from a few log lines.
Langfuse is an open-source LLM engineering
platform (langfuse/langfuse, MIT-licensed). In one line: it turns every
invisible step inside your LLM app into observable, evaluable, reproducible
data — around one loop: develop → monitor → evaluate → debug.
Put plainly, it answers the three questions engineers ask daily but struggle to answer: what actually happened this
time? was it any good? how do I make it better?
A one-line definition: a platform that turns an LLM app's run-time behavior into data.
It does not train models for you, and it is not yet another framework for
calling an LLM; its work happens outside your app — faithfully recording and
measuring every run, and supporting you to evaluate and improve it. The official
capabilities map to one folder each under web/src/features/:
| Capability | Problem it solves | Where it lives |
|---|---|---|
| 🔭 Tracing | See every call: the call tree, I/O, tokens, cost, latency | web/src/features/traces · tracing-tables |
| ✅ Evaluation | Score outputs: LLM-as-judge, human annotation, custom code | web/src/features/evals · scores · annotation-queues |
| 📝 Prompt mgmt | Version prompts; fetch live, roll back, compare | web/src/features/prompts |
| 🧪 Datasets & experiments | Build test sets, run experiments, compare versions | web/src/features/datasets · experiments |
It helps to say what it is not: it is not a framework like LangChain that helps you write an LLM app — use whatever framework you like; Langfuse only records what happens at run time. It is also not a model server or gateway that calls models for you (though the Playground lets you experiment). Its place is the side-channel observer: your app runs as usual and simply "reports" each step to Langfuse.
langfuse/langfuse is the same stack that runs in the
cloud. Understand this code and you can both use it well and change it / send a PR.
Every piece of "observable" data in Langfuse boils down to three things. Their domain
models live under packages/shared/src/domain/ (traces.ts /
observations.ts / scores.ts); their analytical storage is three
same-named ClickHouse tables (packages/shared/clickhouse/migrations/unclustered/0001_traces,
0002_observations, 0003_scores):
domain/observations.ts and scores.ts; values illustrative): one trace strings three observations — SPAN retrieve → GENERATION answer (with real fields model/usageDetails/costDetails/latency) → EVENT cache-write; a score attaches to the trace as NUMERIC. A concrete instance of the three pillars above.Represents one complete request/session. It carries little itself (name, user,
session, tags, time); its main job is to correlate all the observations of one
interaction so you can follow the thread. See domain/traces.ts.
One concrete step. The three core types are GENERATION (an LLM call, with
model/token/cost), SPAN (a timed unit of logic, e.g. retrieval) and EVENT (an instantaneous
event); the current domain model further specializes these into AGENT / TOOL / CHAIN / RETRIEVER and more. It
carries rich attributes — input/output, timing, usage, cost — and is the subject of querying and
aggregation.
And the third pillar, score? It is a rating attached to a trace or an observation. Its value
is mainly one of three types — NUMERIC (e.g. 0.9), CATEGORICAL (e.g. "correct / wrong"),
BOOLEAN (the domain model also has CORRECTION and TEXT); from three sources —
EVAL (auto-judge), ANNOTATION (human), API (submitted via SDK), see
packages/shared/src/domain/scores.ts. Score is what upgrades "observable" into "evaluable": with comparable numbers
you can quantify "did this change actually make the app better" instead of guessing. A score's config (numeric range,
categories) lives in Postgres as ScoreConfig, while the scores themselves go into ClickHouse like
traces/observations.
LegacyPrismaObservationType had only
SPAN / EVENT / GENERATION (that legacy enum has since been extended to 10 too), while the current domain model
packages/shared/src/domain/observations.ts defines an ObservationType with 10 values
(adding AGENT / TOOL / CHAIN / RETRIEVER / EMBEDDING / EVALUATOR / GUARDRAIL). The ClickHouse
observations table inlines model, provided_usage_details,
cost_details, prompt_id and more as columns and Maps on a single row — exactly what a
"wide event" looks like (next section).
Grounding the three pillars in real types — all are Zod schemas in packages/shared/src/domain/:
// (1) trace = the "root" of one full call (traces.ts) export const TraceDomain = z.object({ id: z.string(), name: z.string().nullable(), timestamp: z.date(), input: jsonSchema.nullable(), output: jsonSchema.nullable(), sessionId: z.string().nullable(), userId: z.string().nullable(), projectId: z.string(), // every row carries projectId: the isolation key }); // (2) observation = each step inside a trace, 10 types (observations.ts) export const ObservationType = { SPAN, EVENT, GENERATION, AGENT, TOOL, CHAIN, RETRIEVER, EVALUATOR, EMBEDDING, GUARDRAIL, } as const; // (3) score = a grade attached to a trace/observation (scores.ts) // NUMERIC(value) · CATEGORICAL/BOOLEAN(stringValue) · CORRECTION
An abstract definition is weaker than a concrete scene. Say you built a customer-support bot and
users report "it sometimes answers off-topic". Without Langfuse you only have a few app-log lines like
prompt=... response=... — you can't tell which step failed, and you can't reproduce it.
With Langfuse, triage looks like this:
① In the trace list, filter by "user thumbs-down" or "low score" and surface the offending traces in
seconds; ② open one, and the observation tree makes it obvious — it was the retrieve-docs step
that fetched irrelevant documents, not the model itself; ③ inspect that generation's input
and confirm the polluted context is exactly those wrong docs; ④ add this trace to a dataset, fix the
retrieval logic, run an experiment comparing old vs new, and use a score to confirm it's
truly fixed. The whole flow rests on "every step was recorded as-is".
Zoom out to the system level: Langfuse is a pnpm + Turbo monorepo, running as two main
containers plus four kinds of storage (see the root docker-compose.yml and the project structure in
AGENTS.md):
@langfuse/shared and @langfuse/ee.@langfuse/shared.web/worker/ee → shared.Why pull the core into packages/shared and have both web and worker depend on it one-way only?
Because web and worker are two very different workloads: web faces users, wants low latency and must restart
anytime; the worker chews through heavy jobs, must be retryable and scales by queue. Putting the shared domain models,
storage access and queue contracts into a narrow waist that nothing depends on backwards lets the two
sides evolve, deploy and scale independently while sharing one contract that can't drift. That one-way dependency
discipline (web/worker/ee → shared, and shared depends on none) is key to keeping the whole monorepo
maintainable.
Langfuse's storage and query paths are all threaded by one design principle — treat the observation
as the primary analytical unit and record everything as "wide events" (stated explicitly in
.agents/ARCHITECTURE_PRINCIPLES.md). A wide event means you do not shatter
information into separate metrics, logs and traces to be reassembled later; instead you cram all the
high-cardinality context of one step (model, usage, cost, prompt, user, tags, environment…)
onto a single row.
Contrast the classic "three pillars of observability": metrics are pre-aggregated numbers (QPS, average latency) — cheap but they lose detail, so drilling into "this user + this model + this time window" later is hopeless; logs are scattered text lines — detail is there but they're hard to aggregate and correlate; traces show call structure but usually carry no business attributes. The wide-event idea fuses all three: each observation is at once a structured "trace node", a bag of aggregatable numbers (token/cost/latency), and a carrier of arbitrary business fields (user/tag/metadata). So "drill down" and "aggregate" stop being two systems and become different queries over one wide table. That's also the root reason Langfuse picked ClickHouse — columnar storage is built for "time-bounded scans + arbitrary-dimension aggregation over a wide table".
This guide follows the data: lay the foundations first, then walk the two main paths
"write → read", then take apart the feature subsystems one by one, and finally cover platform & operations.
Every lesson ships real source citations (file:line), hand-drawn diagrams and a design trade-off
box, and closes with a short self-test. This guide makes one hard promise: every technical claim corresponds to
source that really exists in langfuse/langfuse, never invented; citations carry
"file:line" so you can open the repo and check. Where something is unclear, following the citation into the source often beats
ten paragraphs of explanation.
Data model, dual store, ClickHouse wide-event tables, multi-tenancy, deployment topology.
SDK to ClickHouse: API, queue, merge, token/cost, OTel, media.
tRPC, repositories, filtering/search, lists, trace detail, public API.
Eval/scoring, datasets/experiments, prompts/playground, dashboards/cost, automation/integrations.
Auth/RBAC, EE, operations, deletion, build & test, then design themes and the capstone.
observation (not the trace) as the primary analytical unit. What is the main difference this brings?.agents/ARCHITECTURE_PRINCIPLES.md).