🪢 Langfuse 图解教程Langfuse Visual Guide 第二部分 · 前置基础Part 2 · Foundations 09 / 55
第二部分 · 前置基础Part 2 · Foundations

元数据 schema(Postgres/Prisma)The metadata schema (Postgres)

🧭 读前Before you start
先读 第7课 知道 Postgres 管配置元数据,本课细看它的 Prisma schema。
Read Lesson 7 — Postgres holds config metadata; this lesson details its Prisma schema.

前两课的主角是 ClickHouse(海量遥测)。这一课换到另一半:Postgres。如果说 ClickHouse 存的是「发生了什么」(trace/observation/score), 那 Postgres 存的就是「这套平台是怎么配置的」——谁是你的组织和项目、谁有权限、有哪些 API key、定义了哪些 prompt、配了哪些评估规则、各模型怎么定价。 这一面叫控制面(control plane),数据量不大却事关全局。它的形状全写在 packages/shared/prisma/schema.prisma 里。

🔌 生活类比
把两套存储想成一家公司的两类系统:Postgres 是行政/人事系统——记录谁是员工、分在哪个部门、有什么权限、签了哪些规章。 数据不多,但要求准确、可改、立刻生效(新员工入职,权限马上得能用)。ClickHouse 则是业务流水账——海量交易记录, 平时只往里追加,事后拿来按月按品类统计。 你不会把规章制度记到流水账里,也不会把每笔交易塞进人事档案——两类数据,两套系统,各得其所。这一课就专门看「人事系统」这一半:它存了什么、为什么这么存。
控制面(Postgres)vs 数据面(ClickHouse) Postgres · 控制面 少而关键 · 强一致 · 频繁精确读写 org / project / user API key · 权限 prompt 定义 eval / 数据集 配置 模型定价 集成 · 审计日志 「这套平台怎么配置」 ClickHouse · 数据面 海量 · 追加 · 按维度聚合 traces(宽事件) observations(宽事件) scores(宽事件)
两面分工:Postgres 存「平台怎么配置」——组织/项目/用户、API key、prompt、eval/数据集配置、定价、集成、审计;ClickHouse 存「实际发生了什么」——三张宽事件表。这正是第 3 课「领域形状 vs 物理存储」的宏观版:一个管控制、一个管数据。
元数据 schema:实体与关系(节选) Organizationid · name · cloudConfig计费/成员的顶层 1 : N Projectid · orgId · name数据边界 · 一切都挂在它下 ApiKeyid · projectId · publicKeyhashedSecretKey(只存哈希) Promptid · projectId · nameversion · labels(版本化) Datasetid · projectId · name+ items / runs Userid · email · password经 *Membership 多对多入组/项目 N : N(角色)
一切挂在 Project 下(模型名取自 packages/shared/prisma/schema.prisma;列为示例):Organization 1─N Project,Project 1─N ApiKey/Prompt/Dataset;User 经 OrganizationMembership/ProjectMembership 与组织/项目多对多并带角色。因为「一切挂在 project 下」,鉴权只需回答「你属不属于这个 project」(第 10、21 课)。

Postgres 里装的是「控制面」

打开 schema.prisma,你会看到几十个模型。它们可以按用途归成几组(都用 @@map 映射成 snake_case 表名、用 cuid 做主键):

分组主要模型管什么
身份与租户Organization · Project · User · *Membership · Role · ApiKey谁是组织/项目、谁有什么权限、用哪把 key
PromptPrompt · PromptDependency · PromptProtectedLabelsprompt 的版本化定义与依赖
数据集 / 评估Dataset · DatasetItem · DatasetRuns · DatasetRunItems · ScoreConfig · EvalTemplate · JobConfiguration · JobExecution测试集、评分配置、评估任务的定义与编排
模型与计费Model · Price · PricingTier模型定价表(成本计算的依据,第 16 课)
集成与运维*Integration · BatchExport · BatchAction · Media · AuditLog · BackgroundMigration对外集成、导出、媒体、审计、后台迁移

看出规律了吗?这些全是「配置」和「定义」,而不是「海量发生的事」。它们数量级在千到百万之间,要被频繁精确地读写单条 (创建一个项目、改一个 prompt、查一把 key 是否有效),还要强一致——这正是关系型数据库 Postgres 的拿手好戏。

✅ 配置放 Postgres

要外键关系(project → 它的 prompt/key/配置)、要事务(一次改多张表要么全成要么全败)、要「改完立刻可读」。这些都是关系型数据库的强项。

❌ 配置不放 ClickHouse

ClickHouse 弱事务、最终一致,且不擅长频繁改单条。把「创建项目立刻可用」交给它,会出现刚建好却查不到的尴尬。配置类数据放它纯属错配。

第一层身份与租户
Organization / Project / User / Membership / Role / ApiKey——决定「谁、在哪个项目、有什么权限」。是其它一切的前提。
第二层功能配置
Prompt、Dataset、ScoreConfig、EvalTemplate/JobConfiguration、Model/Price——各功能的「定义与编排」,都挂在 project 下。
第三层集成与运维
各种 *Integration、BatchExport/Action、Media、AuditLog、BackgroundMigration——对外连接与平台自身的运维记录。

Project 是枢纽

在这几十个模型里,Project 是当之无愧的中心。打开它的定义你会发现,它字段不多(id、orgId、name、deletedAt 软删、retentionDays…), 但挂着几十条关系——几乎所有其它东西都从属于某个 project:

Organization计费 · 成员 Project数据边界 · 枢纽 1 : N API key 成员/权限 prompt 数据集/eval 模型/定价 仪表盘 自动化/监控 集成 媒体/导出
Project 是控制面的中心:Organization 之下,几乎一切(API key、成员、prompt、数据集/eval、模型/定价、仪表盘、自动化/监控、集成、媒体/导出)都1:N 地挂在某个 project 上。所以 project 是天然的「数据边界」——一把 API key 属于某个 project,你查的所有数据也都被限定在它所属的 project 里(多租户,下一课展开)。

「project 是枢纽」还带来两个很实在的后果。其一是级联:删掉一个 project,它名下的 key、prompt、数据集、配置会跟着被清理(schema 里很多关系标了 onDelete: Cascade), 这让「删一个项目」变成一个干净的操作,而不是到处留下孤儿数据。其二是边界即权限:因为一切都挂在 project 下,鉴权只要回答「你属不属于这个 project」就够了—— 这也是下一课多租户、以及第 21 课 tRPC 中间件 protectedProjectProcedure 的工作基础。记住这张「以 project 为中心」的图,后面很多看似分散的功能,其实都挂在它上面。

为什么 Postgres 里还有 trace 表?

翻 schema.prisma 时你可能会困惑:怎么还有 LegacyPrismaTrace、LegacyPrismaObservation、LegacyPrismaScore? 不是说遥测都在 ClickHouse 吗?答案在名字里——Legacy(遗留)。这是 Langfuse 早期把 trace/observation/score 存在 Postgres 的历史残留; 后来为了应对规模,分析数据整体迁到了 ClickHouse(这正是第 7、8 课的存在理由)。这些 legacy 模型留着主要是兼容与迁移用,不再是热路径。 看到带 Legacy 前缀的,就知道「这是老路,别照着学」。

API key 与权限(预览)

身份这组里,ApiKey 有个值得现在就记住的安全细节:

packages/shared/prisma/schema.prismamodel ApiKey
publicKey            // 公钥,明文
hashedSecretKey      // 密钥的哈希(不存明文!)
fastHashedSecretKey  // 快速校验用的哈希
displaySecretKey     // 只用于 UI 展示的脱敏片段
scope                // ApiKeyScope,默认 PROJECT

注意:密钥本身从不以明文存库,只存哈希——这样即使数据库泄露,攻击者也拿不到可用的 key。权限则用 Role 枚举表达: OWNER / ADMIN / MEMBER / VIEWER / NONE,配合组织级与项目级的 membership 决定「谁能在哪个项目做什么」。这套鉴权与权限会在第 49、53 课展开, 这里你只需建立印象:控制面不仅存「配置」,也存「谁能碰这些配置」。

API key:明文只给你一次,库里只存哈希 你 / SDK 持有 sk-lf-xxxxx(明文) 仅创建时显示一次 Postgres · model ApiKey publicKey — 明文(公钥) hashedSecretKey — 哈希 fastHashedSecretKey — 哈希(快校验) displaySecretKey — 脱敏片段 ✗ 密钥明文:从不入库 校验一次请求 ① 把请求里的 key 哈希 ② 比对 fastHashedSecretKey ✓ 放行 / ✗ 401 明文从不参与比较 即使数据库泄露,攻击者也拿不到可用的 key —— 只有哈希、无法逆推
🎯 设计取舍
把「控制面」和「数据面」分到两个数据库,最大的好处是什么? 是让两者能各自按自己的规律演进与伸缩。控制面数据量小、改动要强一致, 适合一台(或主从)Postgres 稳稳扛住;数据面是洪流,要水平扩展的 ClickHouse 集群。如果硬塞进一个库,要么委屈了配置的强一致,要么拖垮了遥测的吞吐。 代价是:一次操作有时要跨两个库(比如摄取时要回 Postgres 查模型定价、查 prompt 关联)——但这种「读配置、写数据」的跨库协作是可控且高频缓存的, 远比「让一个库同时做好 OLTP 和 OLAP」现实。按职责分库,让每个库只做自己最擅长的事。
🎯 本课要点
  • Postgres = 控制面:存「平台怎么配置」——org/project/user、API key、prompt、eval/数据集配置、模型定价、集成、审计。定义在 schema.prisma。
  • Project 是枢纽:字段不多但挂着几十条关系,几乎一切都 1:N 属于某个 project——它是天然的数据边界(多租户基础)。
  • LegacyPrismaTrace/Observation/Score 是历史残留:分析数据早已迁到 ClickHouse,这些只为兼容/迁移而留。
  • API key 存哈希、不存明文;权限用 Role(OWNER/ADMIN/MEMBER/VIEWER/NONE)+ 组织/项目 membership(第 49、53 课)。
  • 控制面 vs 数据面,是第 3 课「领域形状 vs 物理存储」的宏观版:按职责分库,各做擅长的事。

🧪 自测 · 想一想为什么这么设计

1. Langfuse 为什么把配置/元数据(org/project/user、API key、prompt、定价…)放 Postgres,而不是和遥测一起放 ClickHouse?
  1. 因为 Postgres 查询更慢,适合不常用的数据
  2. 纯粹历史原因,没有道理
  3. 配置要外键关系、事务、强一致(创建即可用),是关系型 Postgres 的强项;ClickHouse 弱事务、最终一致、不擅长频繁改单条,放配置是错配
  4. 因为 ClickHouse 容量太小
看答案与解析 点击展开
答案:C。配置是 OLTP:量小、要精确单条读写、要强一致、有外键关系。这正是 Postgres 的主场;ClickHouse 适合海量追加+聚合(OLAP)。按职责分库,各做擅长的事——这是第 7 课的延续。
2. 为什么说 Project 是控制面的“枢纽”,以及它为什么是多租户的“数据边界”?
  1. 因为 Project 存了所有 trace
  2. 因为它没有任何关系
  3. 因为 Project 表行数最多
  4. Project 字段不多但挂着几十条 1:N 关系,几乎一切(key/成员/prompt/数据集/仪表盘/集成…)都属于某个 project;一把 API key 属于某 project,查询也被限定在该 project 内
看答案与解析 点击展开
答案:D。几乎所有控制面对象都 1:N 挂在 project 下,所以它是中心;又因为 key、数据都按 project 划分,project 成了天然的隔离边界(也对应 ClickHouse 排序键以 project_id 打头)。多租户下一课展开。
3. schema.prisma 里为什么还有 LegacyPrismaTrace/Observation/Score?API key 又是怎么存的?
  1. Legacy 用于备份 ClickHouse;API key 不存储
  2. Legacy 是主力表;API key 明文存储
  3. 两者都已从代码删除
  4. Legacy 是早期把遥测存在 Postgres 的历史残留,分析数据已迁到 ClickHouse,仅为兼容/迁移保留;API key 的密钥只存哈希、不存明文
看答案与解析 点击展开
答案:D。带 Legacy 前缀=老路:遥测早已迁 ClickHouse(第 7、8 课的存在理由),这些只为兼容/迁移留着。API key 只存 hashedSecretKey 等哈希,即使库泄露也拿不到可用 key——安全细节,第 49/53 课展开。
💭 发散思考(没有标准答案,动手或动脑想想)
  • Langfuse 把“配置”和“数据”分到两个库。如果你自己的系统也有“少而关键的配置”+“海量的事件”,你会怎么划分?跨库读配置的代价你打算怎么压(提示:缓存)?

The last two lessons starred ClickHouse (massive telemetry). This one switches to the other half: Postgres. If ClickHouse stores "what happened" (trace/observation/score), Postgres stores "how the platform is configured" — who your orgs and projects are, who has access, which API keys exist, which prompts are defined, which eval rules are set, how each model is priced. This side is the control plane — small in volume yet global in importance. Its shape is all in packages/shared/prisma/schema.prisma.

🔌 Analogy
Think of the two stores as a company's two systems: Postgres is the HR/admin system — who's an employee, in which department, with what permissions, under which policies. Not much data, but it must be accurate, editable, instantly effective (a new hire's permissions must work right away). ClickHouse is the transaction ledger — mountains of records, append-only, used later for monthly/category stats. You wouldn't write policies into the ledger, nor stuff every transaction into HR files — two kinds of data, two systems, each in its place.
control plane (Postgres) vs data plane (ClickHouse) Postgres · control plane small but critical · consistent · precise read/write org / project / user API key · permissions prompt definitions eval / dataset config model pricing integrations · audit "how the platform is configured" ClickHouse · data plane huge · append · dimensional aggregation traces (wide events) observations (wide events) scores (wide events)
Two halves: Postgres stores "how the platform is configured" — org/project/user, API keys, prompts, eval/dataset config, pricing, integrations, audit; ClickHouse stores "what actually happened" — three wide-event tables. This is the macro version of Lesson 3's "domain shape vs physical storage": one governs control, one governs data.
The metadata schema: entities & relations (excerpt) Organizationid · name · cloudConfigtop of billing/membership 1 : N Projectid · orgId · namedata boundary · everything hangs here ApiKeyid · projectId · publicKeyhashedSecretKey (hash only) Promptid · projectId · nameversion · labels (versioned) Datasetid · projectId · name+ items / runs Userid · email · passwordjoins org/project via *Membership N : N (role)
Everything hangs under Project (model names from packages/shared/prisma/schema.prisma; columns illustrative): Organization 1─N Project, Project 1─N ApiKey/Prompt/Dataset; User is many-to-many to org/project via OrganizationMembership/ProjectMembership with a role. Because "everything hangs under project", auth only needs to answer "do you belong to this project" (Lessons 10, 21).

Postgres holds the "control plane"

Open schema.prisma and you'll see dozens of models. They group by purpose (all map to snake_case tables via @@map, with cuid primary keys):

GroupMain modelsGoverns
Identity & tenancyOrganization · Project · User · *Membership · Role · ApiKeywho the org/project is, who has what permission, which key
PromptPrompt · PromptDependency · PromptProtectedLabelsversioned prompt definitions and dependencies
Datasets / evalDataset · DatasetItem · DatasetRuns · DatasetRunItems · ScoreConfig · EvalTemplate · JobConfiguration · JobExecutionthe definition & orchestration of test sets, score configs, eval jobs
Models & billingModel · Price · PricingTiermodel pricing tables (basis for cost computation, L16)
Integrations & ops*Integration · BatchExport · BatchAction · Media · AuditLog · BackgroundMigrationoutbound integrations, exports, media, audit, background migrations

See the pattern? These are all "config" and "definition", not "mountains of events". They number in the thousands to millions, are read/written precisely and often one at a time (create a project, edit a prompt, check a key's validity), and need strong consistency — exactly the relational database Postgres's forte.

✅ Config in Postgres

You want foreign keys (project → its prompts/keys/config), transactions (a multi-table change all-or-nothing), and "readable right after writing". All relational-DB strengths.

❌ Config not in ClickHouse

ClickHouse has weak transactions, eventual consistency, and dislikes frequent single-row edits. Put "create-project-instantly-usable" there and you'd get the awkward "just created but can't find it". Config there is a mismatch.

tier 1identity & tenancy
Organization / Project / User / Membership / Role / ApiKey — decide "who, in which project, with what permission". The prerequisite for everything else.
tier 2feature config
Prompt, Dataset, ScoreConfig, EvalTemplate/JobConfiguration, Model/Price — each feature's "definition & orchestration", all hung under a project.
tier 3integrations & ops
the *Integration models, BatchExport/Action, Media, AuditLog, BackgroundMigration — outbound connections and the platform's own ops records.

Project is the hub

Among these dozens of models, Project is the undeniable center. Open its definition and you'll find few fields (id, orgId, name, deletedAt soft-delete, retentionDays…) but dozens of relations — almost everything else belongs to some project:

Organizationbilling · members Projectdata boundary · hub 1 : N API key members/roles prompts datasets/eval models/pricing dashboards automation integrations media/export
Project is the center of the control plane: under Organization, almost everything (API keys, members, prompts, datasets/eval, models/pricing, dashboards, automation, integrations, media/export) hangs off some project 1:N. So project is the natural "data boundary" — an API key belongs to a project, and all the data you query is scoped to its project (multi-tenancy, next lesson).

"Project is the hub" has two very real consequences. One is cascade: delete a project and its keys, prompts, datasets and config are cleaned up with it (many relations are marked onDelete: Cascade), making "delete a project" a clean operation rather than leaving orphan data everywhere. The other is boundary equals permission: because everything hangs under a project, authorization only needs to answer "do you belong to this project" — the basis for the next lesson's multi-tenancy and for L21's tRPC middleware protectedProjectProcedure. Remember this "project-centric" picture and many seemingly-scattered features later turn out to hang off it.

Why are there trace tables in Postgres?

Browsing schema.prisma you may be puzzled: why are there LegacyPrismaTrace, LegacyPrismaObservation, LegacyPrismaScore? Isn't telemetry all in ClickHouse? The answer is in the name — Legacy. This is a historical remnant of Langfuse's early days storing trace/observation/score in Postgres; later, to handle scale, the analytical data moved to ClickHouse (the very reason for Lessons 7 and 8). These legacy models remain mainly for compatibility and migration — no longer the hot path. See a Legacy prefix and you know "that's the old way, don't copy it".

API keys & permissions (preview)

In the identity group, ApiKey has a security detail worth remembering now:

packages/shared/prisma/schema.prismamodel ApiKey
publicKey            // public key, plaintext
hashedSecretKey      // hash of the secret (no plaintext!)
fastHashedSecretKey  // a hash for fast verification
displaySecretKey     // a masked fragment, UI display only
scope                // ApiKeyScope, default PROJECT

Note: the secret is never stored in plaintext, only hashed — so even a DB leak yields no usable key. Permissions use the Role enum: OWNER / ADMIN / MEMBER / VIEWER / NONE, combined with org-level and project-level memberships to decide "who can do what in which project". Auth and permissions get their own lessons (L49, L53); for now just register the impression: the control plane stores not only "config" but "who may touch that config".

API key: plaintext shown once, only hashes stored you / SDK hold sk-lf-xxxxx (plaintext) shown once at creation Postgres · model ApiKey publicKey — plaintext (public) hashedSecretKey — hash fastHashedSecretKey — hash (fast check) displaySecretKey — masked fragment ✗ plaintext secret: never stored verify a request 1 hash the key in the request 2 compare to fastHashedSecretKey ✓ allow / ✗ 401 plaintext never compared even a DB leak yields no usable key — only hashes, not reversible
🎯 Design trade-off
What's the biggest payoff of splitting "control plane" and "data plane" into two databases? Letting each evolve and scale by its own rules. The control plane is small and needs strong consistency — fine for a single (or primary/replica) Postgres; the data plane is a flood needing a horizontally-scalable ClickHouse cluster. Force them into one store and you either shortchange config consistency or crush telemetry throughput. The cost: some operations span two stores (e.g. ingestion reads model pricing and prompt links back from Postgres) — but this "read config, write data" cross-store collaboration is controllable and heavily cached, far more realistic than "making one store great at both OLTP and OLAP". Split by responsibility; let each store do only what it's best at.
🎯 Key points
  • Postgres = control plane: stores "how the platform is configured" — org/project/user, API keys, prompts, eval/dataset config, model pricing, integrations, audit. Defined in schema.prisma.
  • Project is the hub: few fields but dozens of relations; almost everything belongs 1:N to a project — the natural data boundary (multi-tenancy basis).
  • LegacyPrismaTrace/Observation/Score are historical remnants: analytical data long moved to ClickHouse; these remain only for compatibility/migration.
  • API keys are stored hashed, never plaintext; permissions use Role (OWNER/ADMIN/MEMBER/VIEWER/NONE) + org/project memberships (L49, L53).
  • Control plane vs data plane is the macro version of Lesson 3's "domain shape vs physical storage": split by responsibility, each doing what it's best at.

🧪 Self-test - think about the design

1. Why does Langfuse keep config/metadata (org/project/user, API keys, prompts, pricing…) in Postgres rather than in ClickHouse with telemetry?
  1. Because Postgres is slower, fit for rarely-used data
  2. Pure history, no reason
  3. Config needs foreign keys, transactions and strong consistency (instantly usable) — relational Postgres's strength; ClickHouse has weak transactions, eventual consistency and dislikes frequent single-row edits, so config there is a mismatch
  4. Because ClickHouse is too small
Show answer & explanation click to expand
Answer: C. Config is OLTP: small, precise single-row, strongly consistent, with foreign keys — Postgres's turf; ClickHouse fits massive append + aggregation (OLAP). Split by responsibility, each at its best — a continuation of L07.
2. Why is Project the 'hub' of the control plane, and why is it the multi-tenant 'data boundary'?
  1. Because Project stores all traces
  2. Because it has no relations
  3. Because the Project table has the most rows
  4. Project has few fields but dozens of 1:N relations; almost everything (keys/members/prompts/datasets/dashboards/integrations…) belongs to a project; an API key belongs to a project and queries are scoped to it
Show answer & explanation click to expand
Answer: D. Almost all control-plane objects hang 1:N off a project, making it the center; and since keys and data are scoped by project, project is the natural isolation boundary (matching ClickHouse's project_id-first ordering key). Multi-tenancy next lesson.
3. Why are there still LegacyPrismaTrace/Observation/Score in schema.prisma? And how are API keys stored?
  1. Legacy back up ClickHouse; API keys aren't stored
  2. Legacy are the primary tables; API keys are stored in plaintext
  3. Both have been removed from the code
  4. Legacy models are historical remnants of storing telemetry in Postgres; analytical data moved to ClickHouse, kept only for compat/migration; API key secrets are stored hashed, never plaintext
Show answer & explanation click to expand
Answer: D. Legacy prefix = old way: telemetry long moved to ClickHouse (the reason for L07/L08); these remain for compat/migration. API keys store only hashedSecretKey etc., so a DB leak yields no usable key — a security detail expanded in L49/L53.
💭 Open questions (no single right answer - just think or try)
  • Langfuse splits 'config' and 'data' into two databases. If your system also had 'small critical config' + 'massive events', how would you divide them? How would you tame the cross-store config-read cost (hint: caching)?