🪢 Langfuse 图解教程Langfuse Visual Guide 第二部分 · 前置基础Part 2 · Foundations 11 / 55
第二部分 · 前置基础Part 2 · Foundations

部署拓扑与依赖Deployment topology

🧭 读前Before you start
先读 第7课 的四存储分工,本课讲它们如何在容器拓扑里部署与互相依赖。
Read Lesson 7 on the four stores; this lesson covers how they deploy and depend on each other in the container topology.

第二部分讲了数据模型与存储「是什么」。这一课收尾,讲「跑起来要什么」——自托管一套 Langfuse,你实际要启动哪些容器、它们怎么连。 答案很干净:2 个应用容器 + 4 个基础设施,全写在根目录的 docker-compose.yml 里。把前面几课抽象的架构图,落到这张具体的「装箱清单」上, 你对 Langfuse 的「形」就完整了。

🔌 生活类比
把一套 Langfuse 部署想成开一家餐厅需要的几个「房间」:前厅(web)接待客人、上菜单、端菜;后厨(worker)在后面默默备菜、洗碗、处理大单; 冷库(ClickHouse)存海量食材(遥测);文件柜(Postgres)放菜单和员工档案(配置);传菜口(Redis)排订单(队列); 大仓库(S3/MinIO)堆又大又不常用的存货(原件、媒体)。 少了哪个房间,餐厅都开不利索——但凑齐这 6 个,店就能营业了。这一课就带你认全这六个「房间」,看清它们各自是谁、为什么少不了、又怎么连成一家能营业的店。
docker-compose 拓扑:2 应用 + 4 基础设施 langfuse-weblangfuse/langfuse:3 · UI+API langfuse-workerlangfuse-worker:3 · 后台 postgres:17配置/元数据 clickhouse-server宽事件 redis:7队列 + 缓存 minio(S3 兼容)事件底稿·媒体 每个基础设施都有自己的 named volume 做持久化(pg / clickhouse data+logs / redis / minio) web 偏读 PG+CH;worker 偏写 CH/S3、消费 Redis 队列(连线为示意)
自托管装箱清单:两个应用容器 langfuse-web(langfuse/langfuse:3)与 langfuse-worker(langfuse/langfuse-worker:3),加四个基础设施 postgres:17 · clickhouse-server · redis:7 · minio(S3 兼容)。每个基础设施配一个 named volume 持久化。这正是第 5、7 课架构图的「实体版」。
部署拓扑:三层栈,按层扩缩 ① 边缘 · 负载均衡 / 入口(TLS、路由) ② 应用层(无状态 · 可水平扩展) web ×NNext.js · UI+API+tRPC …副本 worker ×M消费 BullMQ 队列 …副本 ③ 后端服务(有状态) Postgres ClickHouse Redis S3
三层、按层扩缩(依据 docker-compose/helm 与 server/queues.ts):无状态的 web 和 worker 各自多副本、按负载独立扩缩(web 顶用户流量,worker 顶队列积压);有状态的 Postgres/ClickHouse/Redis/S3 在底层被两者共享。这正是第 5、7 课「读写分离、四存储分工」落到容器上的样子。

最小自托管栈:2 应用 + 4 基础设施

服务镜像角色
langfuse-weblangfuse/langfuse:3面向用户:UI + tRPC + 公共 REST API(摄取入口)
langfuse-workerlangfuse/langfuse-worker:3后台:消费队列、合并写入、评估、导出…
postgrespostgres:17控制面:配置/元数据(第 9 课)
clickhouseclickhouse/clickhouse-server数据面:宽事件表(第 8 课)
redisredis:7BullMQ 队列 + 缓存
miniochainguard/minioS3 兼容对象存储(本地替身):事件底稿·媒体·导出

注意 minio:它是一个兼容 S3 协议的开源对象存储,在本地/自托管时充当 AWS S3 的替身。这意味着同一套代码, 在云上用真 S3、在自托管用 MinIO,应用层完全无感——这正是「面向接口(S3 协议)而非实现」带来的可移植性。四个基础设施容器各自挂 named volume 做持久化 (postgres / clickhouse data+logs / redis / minio),而 web 与 worker 无状态、不挂卷;所以你 docker compose down 再 up,数据还在。

应用层web · worker
无状态的业务容器,可随时重启、按需扩容。它们启动后连上下面的存储干活。
数据层postgres · clickhouse
两类数据库:控制面(配置)与数据面(遥测)。有状态,靠 named volume 持久化,是整套系统的根基。
支撑层redis · minio
队列/缓存与对象存储。让摄取异步、让大字段/媒体有处可放。同样有状态、需持久化。

这三层有个朴素的启动次序:先把数据层与支撑层拉起、等它们健康,再启动应用层——所以 compose 里应用容器会 depends_on 这些基础设施并等待其 healthcheck。 理解这个分层,你 debug 部署问题时就有了方向:连不上库先看数据层、摄取不动先看 Redis、媒体存不进先看 MinIO。

web 与 worker:分工与协作

两个应用容器是 Langfuse 的「左右手」。它们形态相似(都连同一套存储),但职责相反:

web 面向用户 · 要低延迟 无状态 · 可随时重启/扩容 收请求、入队、读数据渲染 扩容看:并发用户/QPS worker 啃重活 · 可慢 可重试 · 按队列伸缩 消费队列、合并写库、评估 扩容看:队列积压 Redis 队列 通过队列解耦:两者可各自独立部署、独立扩容、互不阻塞
左右手分工:web 面向用户、追求低延迟、可随时重启;worker 啃重活、可重试、按队列积压伸缩。二者通过 Redis 队列解耦(第 5 课),所以你能分别给它们扩容——用户多了加 web,队列堵了加 worker。这正是第 4 课「窄腰」让两个容器独立演进的部署体现。

web 怎么扩

无状态、面向请求 → 看并发用户 / QPS,水平多开几份、前面挂负载均衡即可。

worker 怎么扩

面向队列 → 看队列积压,积压上来就多开 worker 实例消费(还能按队列分片,第 14 课)。

配置全靠环境变量

这六个容器怎么知道彼此地址、密钥、各种开关?答案是环境变量。仓库里有一组 .env.*.example 模板(如 .env.dev.example、.env.prod.example), 列出了所有需要配置的变量(数据库连接、S3 端点、加密密钥…)。更重要的是,这些 env 不是随便读的——它们经过 Zod 校验 (packages/shared/src/env.ts、worker/src/env.ts 等),启动时就检查类型与必填项。配错了、漏填了,进程会直接启动失败并报清楚哪个变量有问题, 而不是带病运行到半路才崩。这条「启动即校验配置」的纪律,是把「配置错误」从难查的运行时 bug 变成一眼可见的启动错误(第 51 课展开)。

把这些拼起来,第一次自托管的体验其实很简单:克隆仓库 → 复制一份 .env.dev.example 填好关键变量 → docker compose up, 六个容器就按依赖次序拉起、连通,几分钟后你就能打开浏览器看到 Langfuse 界面。仓库里还备了多套 compose 文件应对不同场景—— docker-compose.dev.yml(本地开发,常只拉基础设施、应用跑在本机便于热重载)、docker-compose.yml(完整一套)、 以及针对特定云的变体(如 Azure)。这种「一份编排管全栈、按场景给变体」的安排,正是把前面说的「四依赖的复杂度」收进了几个 yml 文件里—— 你不需要手动一个个装数据库、配网络,compose 替你把这些都串好了。理解了这张拓扑,你不仅会「跑起来」,遇到问题也知道该去哪个容器看日志。

同一镜像 + 不同 .env = 不同环境(配置外置) langfuse/langfuse:3 同一个镜像 .env.dev .env.staging .env.prod 本地 dev:localhost · MinIO 替身 预发 staging 生产 prod:托管 PG/CH · 真 S3 env 经 Zod 启动即校验:配错/漏填 → 进程直接启动失败并指明哪个变量
.env.dev.example(节选,按「4 基础设施 + 认证」分组)配置即接线
# ① Postgres —— 配置/用户/权限等元数据(第7课 OLTP)
DATABASE_URL="postgresql://postgres:postgres@localhost:5432/postgres"
# ② ClickHouse —— 海量 trace/observation/score 事件(第8课列存)
CLICKHOUSE_URL="http://localhost:8123"
CLICKHOUSE_USER="clickhouse"   CLICKHOUSE_PASSWORD="clickhouse"
# ③ Redis —— 摄取队列 + 缓存(第12课快接收的落点)
REDIS_HOST=localhost   REDIS_PORT=6379
# ④ S3 / MinIO —— 大块负载与媒体(第18课);MinIO 是本地 S3 替身
LANGFUSE_S3_MEDIA_UPLOAD_BUCKET=langfuse
LANGFUSE_S3_..._ENDPOINT="http://localhost:9090"   ..._FORCE_PATH_STYLE=true
# 认证与密钥 —— 启动时由 Zod 校验,缺一即启动失败(第51课)
NEXTAUTH_URL="http://localhost:3000"   NEXTAUTH_SECRET="secret"
SALT="salt"   # API key 哈希用盐(第49课 createShaHash)
🎯 设计取舍
四个基础设施依赖,对自托管者是不是负担? 坦白说,是——相比「一个二进制 + 一个 SQLite」,Langfuse 的自托管门槛确实更高。但这正是第 7 课那条取舍的部署侧后果: 为了能扛生产级规模(海量遥测、异步摄取、对象存储),这四个组件每一个都不可省。Langfuse 的缓解办法是用 docker-compose 一键拉起全部六个容器、用 MinIO 替身免去对云 S3 的依赖, 让「在自己机器上跑起完整一套」变得简单。复杂度无法消除,但可以被打包好、一键交付。这也是为什么官方强调「self-host in minutes」——不是没有依赖,而是把依赖编排好了。对自托管者来说,真正要权衡的不是「依赖多不多」,而是「这套规模化能力你用不用得上」。
🎯 本课要点 · 第二部分收官
  • 最小自托管栈 = 2 应用 + 4 基础设施:langfuse-web + langfuse-worker,加 postgres · clickhouse · redis · minio(S3),全在 docker-compose.yml。
  • web vs worker:同连一套存储但职责相反——web 面向用户低延迟、按 QPS 扩;worker 啃重活、按队列积压扩;靠 Redis 队列解耦。
  • MinIO 是 S3 协议替身:同一套代码云上用真 S3、本地用 MinIO,应用无感。
  • 配置全靠环境变量,并经 Zod 启动即校验(env.ts)——配错就启动失败、报清楚,而非带病运行。
  • 四依赖是规模化的必要代价(第 7 课),用 docker-compose 一键编排来缓解——「依赖不少,但都打包好了」。

🧪 自测 · 想一想为什么这么设计

1. 自托管一套 Langfuse,最小栈包含哪些容器?
  1. 只需要 ClickHouse
  2. web + worker + 一个 SQLite 文件
  3. 只要一个 langfuse 容器
  4. 2 个应用(langfuse-web、langfuse-worker)+ 4 个基础设施(postgres、clickhouse、redis、minio/S3),全在 docker-compose.yml
看答案与解析 点击展开
答案:D。docker-compose.yml 列出两个应用容器(web/worker)加四个基础设施(postgres:17、clickhouse-server、redis:7、minio)。这正是第 5、7 课架构图的实体版。
2. web 和 worker 都连同一套存储,为什么还要拆成两个容器、还能分别扩容?
  1. 因为一个容器装不下代码
  2. 职责相反:web 面向用户、要低延迟、按 QPS 扩;worker 啃重活、可重试、按队列积压扩。靠 Redis 队列解耦,所以能各自独立扩容、互不阻塞
  3. 因为它们用不同的数据库
  4. 纯粹为了多开机器
看答案与解析 点击展开
答案:B。用户压力(QPS)和后台压力(队列积压)是两条独立的扩容信号。拆成两个容器、用队列解耦,就能“用户多了加 web、队列堵了加 worker”,互不牵连。这是第 4 课窄腰让两容器独立演进的部署体现。
3. Langfuse 用 Zod 在“启动时”校验环境变量。这条纪律的好处是什么?
  1. 让启动更慢以便检查
  2. 自动修正错误的配置
  3. 把环境变量加密
  4. 配错/漏填会让进程直接启动失败并指出哪个变量有问题——把“配置错误”从难查的运行时 bug 变成一眼可见的启动错误,而不是带病运行到半路才崩
看答案与解析 点击展开
答案:D。fail-fast:env.ts 用 Zod 在启动时校验类型与必填项,错了立刻报清楚。这比“带着错误配置跑半天、在某个边角功能才崩”可调试得多(第 51 课展开)。
💭 发散思考(没有标准答案,动手或动脑想想)
  • Langfuse 的自托管需要 4 个基础设施,门槛比“一个二进制”高。如果你做一个开源工具,会怎么在“能扛规模”和“易自托管”之间取舍?docker-compose 一键编排算不算一个好答案?

Part 2 covered what the data model and storage are. This lesson closes it with what it takes to run — to self-host Langfuse, which containers you actually start and how they connect. The answer is clean: 2 app containers + 4 infrastructure engines, all in the root docker-compose.yml. Land the abstract architecture of earlier lessons onto this concrete "packing list" and your mental model of Langfuse's "shape" is complete.

🔌 Analogy
Picture a Langfuse deployment as the rooms a restaurant needs: the front of house (web) greets guests, shows the menu, serves dishes; the kitchen (worker) quietly preps, washes up, handles big orders; the cold storage (ClickHouse) holds mountains of ingredients (telemetry); the filing cabinet (Postgres) keeps the menu and staff records (config); the order pass (Redis) queues tickets; the warehouse (S3/MinIO) stacks big, rarely-used stock (manuscripts, media). Miss a room and the restaurant limps — but assemble these 6 and you can open for business.
docker-compose topology: 2 apps + 4 infra langfuse-weblangfuse/langfuse:3 · UI+API langfuse-workerlangfuse-worker:3 · jobs postgres:17config/metadata clickhouse-serverwide events redis:7queue + cache minio (S3-compatible)event log · media each infra has its own named volume for persistence (pg / clickhouse data+logs / redis / minio) web leans read PG+CH; worker leans write CH/S3, consumes the Redis queue (lines are illustrative)
The self-host packing list: two app containers langfuse-web (langfuse/langfuse:3) and langfuse-worker (langfuse/langfuse-worker:3), plus four infra postgres:17 · clickhouse-server · redis:7 · minio (S3-compatible). Each infra gets a named volume for persistence. This is the "physical version" of the architecture in Lessons 5 and 7.
Deployment topology: a three-tier stack, scaled per tier ① edge · load balancer / ingress (TLS, routing) ② app tier (stateless · horizontally scalable) web ×NNext.js · UI+API+tRPC …replicas worker ×Mconsume BullMQ queues …replicas ③ backing services (stateful) Postgres ClickHouse Redis S3
Three tiers, scaled per tier (per docker-compose/helm and server/queues.ts): stateless web and worker each run multiple replicas and scale independently (web for user traffic, worker for queue backlog); stateful Postgres/ClickHouse/Redis/S3 sit underneath, shared by both. This is Lessons 5 and 7's "read/write split, four-store division of labor" landed on containers.

The minimal self-host stack: 2 apps + 4 infra

ServiceImageRole
langfuse-weblangfuse/langfuse:3user-facing: UI + tRPC + public REST API (ingestion entry)
langfuse-workerlangfuse/langfuse-worker:3background: consume queues, merge-write, eval, export…
postgrespostgres:17control plane: config/metadata (L09)
clickhouseclickhouse/clickhouse-serverdata plane: wide-event tables (L08)
redisredis:7BullMQ queues + cache
miniochainguard/minioS3-compatible object store (local stand-in): event log · media · exports

Note minio: an open-source object store that speaks the S3 protocol, standing in for AWS S3 locally / when self-hosting. So the same code uses real S3 in the cloud and MinIO when self-hosted, entirely transparent to the app layer — the portability of "program to an interface (the S3 protocol), not an implementation". The four infra containers each mount a named volume for persistence (postgres / clickhouse data+logs / redis / minio), while web and worker are stateless — no volumes; so docker compose down then up keeps your data.

app tierweb · worker
stateless business containers, restartable and scalable on demand. They start, connect to the storage below, and work.
data tierpostgres · clickhouse
two databases: control plane (config) and data plane (telemetry). Stateful, persisted via named volumes — the bedrock.
support tierredis · minio
queue/cache and object store. Make ingestion async and give big fields/media a home. Also stateful, also persisted.

These tiers have a plain startup order: bring up the data and support tiers and wait for them to be healthy, then start the app tier — so in compose the app containers depends_on the infra and wait for healthchecks. Grasp this layering and you have direction when debugging deployments: can't reach the DB → check the data tier; ingestion stalled → check Redis; media won't save → check MinIO.

web and worker: division and collaboration

The two app containers are Langfuse's "two hands". Similar in form (both connect the same storage) but opposite in duty:

web user-facing · low latency stateless · restart/scale anytime accept requests, enqueue, read & render scale by: concurrent users / QPS worker heavy jobs · may be slow retryable · scales by queue consume queue, merge-write, eval scale by: queue backlog Redis queue decoupled by the queue: deploy and scale each independently, without blocking the other
Two hands: web is user-facing, low-latency, restartable; worker chews heavy jobs, retryable, scales by queue backlog. The two are decoupled by the Redis queue (L05), so you scale them separately — more users, add web; backed-up queue, add worker. This is the deployment-side expression of Lesson 4's "narrow waist" letting the two containers evolve independently.

scaling web

stateless, request-facing → watch concurrent users / QPS, run a few replicas behind a load balancer.

scaling worker

queue-facing → watch queue backlog, spin up more worker instances to consume (and shard by queue, L14).

Configuration is all environment variables

How do the six containers know each other's addresses, secrets and toggles? Environment variables. The repo has a set of .env.*.example templates (e.g. .env.dev.example, .env.prod.example) listing every variable to set (DB connections, S3 endpoint, encryption keys…). More importantly, these envs aren't read loosely — they pass through Zod validation (packages/shared/src/env.ts, worker/src/env.ts, etc.), checking types and required fields at startup. Misconfigure or miss one and the process fails to start and tells you exactly which variable is wrong, rather than running sick and crashing midway. This "validate config at startup" discipline turns "config errors" from hard-to-find runtime bugs into obvious startup errors (expanded in L51).

Put it together and the first self-host experience is actually simple: clone the repo → copy an .env.dev.example and fill the key variables → docker compose up, and the six containers come up in dependency order, connect, and within minutes you open a browser to the Langfuse UI. The repo also ships several compose files for different scenarios — docker-compose.dev.yml (local dev, often just the infra while the apps run on your machine for hot reload), docker-compose.yml (the full stack), and cloud-specific variants (e.g. Azure). This "one orchestration for the whole stack, variants per scenario" is exactly how the earlier "four-dependency complexity" gets folded into a few yml files — you needn't hand-install each database or wire the network; compose strings it all together. Grasp this topology and you'll not only "get it running" but also know which container's logs to check when something breaks.

same image + different .env = different environment (externalized config) langfuse/langfuse:3 one image .env.dev .env.staging .env.prod local dev: localhost · MinIO stand-in staging production: managed PG/CH · real S3 env validated by Zod at boot: a misconfig/missing var fails startup and names the variable
.env.dev.example (excerpt, grouped by "4 infra + auth")config = wiring
# ① Postgres — metadata: config/users/permissions (L07 OLTP)
DATABASE_URL="postgresql://postgres:postgres@localhost:5432/postgres"
# ② ClickHouse — massive trace/observation/score events (L08 columnar)
CLICKHOUSE_URL="http://localhost:8123"
CLICKHOUSE_USER="clickhouse"   CLICKHOUSE_PASSWORD="clickhouse"
# ③ Redis — ingestion queue + cache (where L12's "receive fast" lands)
REDIS_HOST=localhost   REDIS_PORT=6379
# ④ S3 / MinIO — bulky payloads & media (L18); MinIO is a local S3 stand-in
LANGFUSE_S3_MEDIA_UPLOAD_BUCKET=langfuse
LANGFUSE_S3_..._ENDPOINT="http://localhost:9090"   ..._FORCE_PATH_STYLE=true
# auth & secrets — Zod-validated at startup, miss one = fail to start (L51)
NEXTAUTH_URL="http://localhost:3000"   NEXTAUTH_SECRET="secret"
SALT="salt"   // salt for API-key hashing (L49 createShaHash)
🎯 Design trade-off
Are four infra dependencies a burden for self-hosters? Honestly, yes — versus "one binary + one SQLite", Langfuse's self-host bar is genuinely higher. But that's the deployment-side consequence of Lesson 7's trade-off: to handle production scale (massive telemetry, async ingestion, object storage), none of the four is droppable. Langfuse's mitigation is bringing up all six containers with one docker-compose command and using a MinIO stand-in to remove the dependency on cloud S3, making "run the whole thing on your own machine" easy. Complexity can't be erased, but it can be packaged and delivered in one command. That's why the docs say "self-host in minutes" — not no dependencies, but dependencies orchestrated for you.
🎯 Key points · Part 2 wrap-up
  • Minimal self-host stack = 2 apps + 4 infra: langfuse-web + langfuse-worker, plus postgres · clickhouse · redis · minio(S3), all in docker-compose.yml.
  • web vs worker: same storage, opposite duties — web user-facing/low-latency, scales by QPS; worker heavy/retryable, scales by queue backlog; decoupled by the Redis queue.
  • MinIO is an S3-protocol stand-in: same code uses real S3 in the cloud, MinIO locally — transparent to the app.
  • Config is all env vars, Zod-validated at startup (env.ts) — misconfigure and it fails to start with a clear message, no sick running.
  • Four deps are the necessary cost of scale (L07), mitigated by one-command docker-compose orchestration — "plenty of deps, but packaged".

🧪 Self-test - think about the design

1. What containers make up the minimal stack to self-host Langfuse?
  1. Only ClickHouse
  2. web + worker + one SQLite file
  3. Just one langfuse container
  4. 2 apps (langfuse-web, langfuse-worker) + 4 infra (postgres, clickhouse, redis, minio/S3), all in docker-compose.yml
Show answer & explanation click to expand
Answer: D. docker-compose.yml lists two app containers (web/worker) plus four infra (postgres:17, clickhouse-server, redis:7, minio). It's the physical version of the architecture in L05/L07.
2. web and worker connect the same storage — why split into two containers, and why scale them separately?
  1. Because one container can't hold the code
  2. Opposite duties: web is user-facing, low-latency, scales by QPS; worker chews heavy jobs, retryable, scales by queue backlog. Decoupled by the Redis queue, so each scales independently without blocking the other
  3. Because they use different databases
  4. Just to run more machines
Show answer & explanation click to expand
Answer: B. User pressure (QPS) and background pressure (queue backlog) are independent scaling signals. Splitting into two containers decoupled by a queue lets you 'add web for more users, add worker for a backed-up queue', without entanglement. It's the deployment-side expression of L04's narrow waist.
3. Langfuse validates env vars with Zod 'at startup'. What's the benefit of this discipline?
  1. To make startup slower for checking
  2. To auto-fix wrong config
  3. To encrypt env vars
  4. A misconfig/missing var makes the process fail to start and names the bad variable — turning 'config errors' from hard-to-find runtime bugs into obvious startup errors, instead of running sick and crashing midway
Show answer & explanation click to expand
Answer: D. Fail-fast: env.ts uses Zod to validate types and required fields at startup, reporting clearly on error. Far more debuggable than 'running with bad config and crashing in some corner feature later' (expanded in L51).
💭 Open questions (no single right answer - just think or try)
  • Langfuse self-host needs 4 infra engines, a higher bar than 'one binary'. If you built an open-source tool, how would you trade off 'handles scale' vs 'easy to self-host'? Is one-command docker-compose orchestration a good answer?