🪢 Langfuse 图解教程Langfuse Visual Guide 第三部分 · 摄取链路Part 3 · The Ingestion Path 12 / 55
第三部分 · 摄取链路Part 3 · The Ingestion Path

摄取 APIThe ingestion API

🧭 读前Before you start
先读 第6课 的埋点入口,本课讲数据进平台的第一站——摄取 API。
Read Lesson 6 on instrumentation entry points; this lesson covers the first stop into the platform — the ingestion API.

第二部分把「数据长什么样、存在哪」讲完了。从这一课起,我们正式走摄取链路——也就是第 5 课那条「写入主干」的放大版。 第一站是公共摄取 API:你的应用发来一批事件,服务端这一头是怎么收下的。剧透一下它的性格——极快、极轻: 它只做「鉴权 + 校验 + 入队」三件事,然后立刻返回,绝不在这里合并、算成本、写库(那些是 worker 的活)。这就是第 5 课说的「快路」。

🔌 生活类比
把摄取 API 想成快递的收件口:快递员(你的 SDK)抱来一批包裹(events),收件员(API)做三件事就放行—— 核对寄件人身份(鉴权:你这把 key 是哪个项目的?)、扫码登记面单(校验:每个包裹的格式对不对?)、丢上传送带(入队)。 然后当场给你一张回执(HTTP 207),就让你走了——不会当着你的面去分拣、称重、送到目的地。那些慢活,传送带后面的人(worker)慢慢做。 收件口的唯一目标,就是收得飞快、永不堵门。
一个摄取请求的一生(毫秒级) SDK POST一批 events ① 鉴权key→project ② Zod 校验格式对不对 ③ processEventBatch入队(+落 S3) 207立即返回 合并、算成本、写 ClickHouse 都不在这里——那是 worker 的活(第 15–17 课) API 的全部职责:收下 → 校验 → 入队 → 回执
极简的收件流程:鉴权(API key 解析出 project,第 9·10 课)→ Zod 校验事件格式 → processEventBatch 把事件入队(同时把原件落 S3)→ 立即 207。整条链路上「重活」一件都不在这里做,所以 API 永远毫秒级返回。
示例:一次摄取请求与 207 响应 POST /api/public/ingestion Authorization: Basic <pk:sk> { "batch": [ { "type":"trace-create", "body":{"id":"chat_a1b2","name":"qa"} }, { "type":"observation-create", "body":{"id":"obs_7f","type":"GENERATION", "traceId":"chat_a1b2"} } ] } ↑ 请求(攒批一次发) 207 Multi-Status { "successes": [ {"id":"chat_a1b2","status":201}, {"id":"obs_7f","status":201} ], "errors": [] } 每事件各自成败,整批一次 207 ↑ 响应(早返回,不等落库)
一来一回(状态码与 batch 形状对齐 web/src/pages/api/public/ingestion.ts;值为示例):客户端把多个事件攒成一个 batch POST 上来,服务端校验入队后回 207 Multi-Status——successes/errors 逐事件给结果。207 而非 200,是因为「一批里可能有的成有的败」。

一个摄取请求的一生

入口在 web/src/pages/api/public/ingestion.ts 的 handler。它收下请求体后,真正干活的是一个叫 processEventBatch 的共享函数:

web/src/pages/api/public/ingestion.tshandler
import { processEventBatch } from "@langfuse/shared/src/server";

export default async function handler(req, res) {
  // …鉴权、解析 body…
  const result = await processEventBatch(batchForProcessing, authCheck);
  // 立即把结果返回(成功/部分失败),不等 worker 处理
}

注意一个设计细节:processEventBatch 是共享的——除了 /ingestion,连「单独提交 score」的接口也复用它。 为什么?因为「收一批事件、校验、入队」这套逻辑到处都用得上,抽成一个函数就不必每个端点各写一遍、各错一遍(又是第 6 课「复杂度收敛到一处」的体现)。 它支持的事件类型涵盖了 TRACE_CREATE / OBSERVATION_CREATE / SPAN_CREATE / GENERATION_CREATE / SCORE_CREATE 等 (还包括 SDK_LOG、DATASET_RUN_ITEM_CREATE)——正是第 6 课说的那套事件信封。

为什么是「一批」,不是「一条」

SDK 不是每产生一个事件就发一次 HTTP,而是攒成一批再发。这个「批」的设计省了大事:

😖 一条一发

每个 observation 都来一次 HTTP 往返,高频应用下网络开销巨大、连接数爆炸,对你的应用是实打实的拖累。

😀 攒批一发

几十上百个事件打成一个请求,一次往返搞定。网络开销摊薄,API 也能一次性校验、一次性入队,吞吐高得多。

批处理还带来一个语义上的好处:同一批里可能既有某 observation 的 CREATE 又有它的 UPDATE,服务端可以整批一起处理, 减少来回。当然,批也不能无限大——请求体过大时 API 会拒绝(返回「请求体过大」),这也是一道保护:别让单个请求把服务端撑爆。

1

SDK 攒批 → POST /ingestion

应用侧把一段时间内的事件打成一批,一次发出。

2

handler 鉴权

从 API key 解析 project、校验写入权限(失败整批拒)。

3

Zod 校验每个事件

结构不对的事件被拒;其余继续(可部分成功)。

4

processEventBatch 入队 + 落 S3

合法事件进 Redis 队列、原件存 S3(第 14 课)。

5

立即返回结果

告诉调用方哪些收下、哪些被拒——全程毫秒级,不等 worker。

一批里好坏混杂?逐条处理,207 告诉你哪条成哪条败 一批 N 个事件 event ✓ 合法 event ✗ 结构错 event ✓ 合法 event ✓ 合法 逐条校验合法 → 进队列处理非法 → 逐条记 rejectedErrors坏事件不连累好事件 successes:已收下进 Redis 队列、原件落 S3 errors:被拒(带原因)调用方可只重发这几条 最终 HTTP 207 Multi-Status,返回 {successes, errors}——这就是「批量但可部分成功」的具体含义
批量 = 可部分成功:ingestion.ts:153 filterBatchForEventsOnly 逐条筛、非法事件记 rejectedErrors;:158 processEventBatch 处理合法的;:162 最终 res.status(207).json(result) 返回 {successes, errors}。一个混合批次里,坏事件不连累好事件,调用方可只重发被拒的几条。
web/src/pages/api/public/ingestion.ts:132-162批量解析 + 逐条筛 + 207
// 请求体就是「一个 batch 数组」,先整体 Zod 校验外壳
const batchType = z.object({ batch: z.array(z.unknown()) });
const parsedSchema = batchType.safeParse(req.body);

// 逐条筛:非法事件被单独拒,好事件继续——混合批次仍能处理
const { batchForProcessing, rejectedErrors } = filterBatchForEventsOnly(parsedSchema.data.batch);
const result = await processEventBatch(batchForProcessing, authCheck);  // 入队 + 落 S3
if (rejectedErrors.length > 0) result.errors = [...result.errors, ...rejectedErrors];

return res.status(207).json(result);  // 207 Multi-Status:{ successes, errors } 各是哪些

守门的两道关:鉴权 + 校验

API 在「入队」之前,必须先过两道关。它们看似例行公事,却是整个系统安全与健壮的第一道防线:

① 鉴权关 API key → 哪个 project? 有没有写入权限? ② 校验关(Zod) 每个事件结构对不对? 坏事件早拒,别进系统 入队 两关都过才放行 鉴权决定「你是谁、能不能写」,校验决定「你发的东西干不干净」
两道关:鉴权从 API key 解析出 project(第 9 课 key 属于某 project)并确认有写入权限——这一步也把租户钉死(第 10 课 project_id 贯穿);Zod 校验逐个事件检查结构,坏数据在门口就被拒,不会污染下游。两关都过,才允许入队。

第二道关「Zod 校验」尤其重要:摄取系统每天吞下海量来自各种 SDK、各种版本的数据,难免有格式不对、字段缺失的「脏事件」。 在入口就用 Zod schema 把它们挡下、给出清晰错误,比让它们流进队列、到 worker 处理时才崩要好得多——越早拒绝坏数据,整个系统越省心。 这一批里如果有几个坏事件,API 还能返回「部分成功」,告诉你哪几条没收下,而不是整批一刀切。

这一关把守什么失败会怎样
鉴权API key 有效吗?属于哪个 project?能写吗?401/403,整批拒绝
Zod 校验每个事件的结构/类型是否合法坏事件被拒,可返回部分成功
体积保护请求体是否过大返回「请求体过大」,防撑爆

「部分成功」不是说说而已,它写进了响应契约:ingestion.ts 的 handler 最后一行是 res.status(207).json(result), 而 result 形如 { successes, errors }——两个数组,分别列出收下的事件与被拒的事件(每条带错误原因)。207 Multi-Status 这个 HTTP 状态码 天生就是为「一个请求里多个子操作、结果各异」设计的。于是即便你这批 100 个事件里混进 3 个脏的,服务端也会收下 97 个、退回 3 个并告诉你为什么, 而不是因为 3 颗老鼠屎把整锅粥倒掉。这对 SDK 极其友好:客户端可以只重传失败的那几条,无需整批重来。

🎯 设计取舍
为什么 API 只做「鉴权+校验+入队」,把一切重活都推给 worker? 因为摄取 API 是直面你应用的那一层,它的首要 KPI 是 「永远快、永远不拖累调用方」。任何放在这里的额外工作——合并、查定价、写库——都会变成你每次埋点调用的延迟,还会因为下游抖动而失败。 所以 Langfuse 把这条线切得极干净:API 端只保留「无论如何都必须当场做」的最小集合(鉴权防止越权写入、校验防止脏数据入库、入队完成交接), 剩下能异步的一律异步。代价是「最终一致」(第 5 课),但收益是一个几乎不可能慢、不可能拖垮你业务的摄取入口。入口越薄,越扛得住洪峰。
🎯 本课要点
  • 摄取 API 的全部职责:鉴权 + Zod 校验 + 入队,然后立即返回——合并/算成本/写库都不在这里(是 worker 的活)。
  • 入口 ingestion.ts 的 handler 调用共享的 processEventBatch(连单独提交 score 的接口也复用它)。
  • 攒批上报:几十上百个事件一次 HTTP,摊薄网络开销、一次校验一次入队;批过大会被拒(体积保护)。
  • 两道关:鉴权(key→project、权限,把租户钉死)+ Zod 校验(坏数据门口就拒,可部分成功)。
  • 取舍:入口只留「必须当场做」的最小集合,能异步的全异步——入口越薄越扛洪峰,代价是最终一致。

🧪 自测 · 想一想为什么这么设计

1. 摄取 API 收到一批事件后只做三件事就立即返回。是哪三件?为什么不在这里合并/算成本/写库?
  1. 鉴权 + 合并 + 写库;因为这样最快
  2. 鉴权 + Zod 校验 + 入队;因为 API 直面你的应用,首要目标是“永远快、不拖累调用方”,重活放这里会增加每次埋点的延迟、还会因下游抖动而失败
  3. 压缩 + 加密 + 入队
  4. 只做入队,不鉴权也不校验
看答案与解析 点击展开
答案:B。入口只保留“必须当场做”的最小集合(鉴权防越权、校验拦脏数据、入队完成交接),能异步的全交给 worker。入口越薄越扛洪峰、越不可能拖垮你的业务,代价是最终一致(第 5 课)。
2. ingestion.ts 的 handler 把真正的活交给一个叫 processEventBatch 的共享函数,连单独提交 score 的接口也复用它。这么做的好处是?
  1. 为了让 score 接口更慢
  2. “收一批、校验、入队”到处都用得上,抽成一处就不必每个端点各写一遍、各错一遍(复杂度收敛到一处,第 6 课)
  3. 纯粹为了多写代码
  4. 因为 score 不能走队列
看答案与解析 点击展开
答案:B。多个端点共享同一套摄取逻辑,维护在一处、测试在一处,避免分散实现导致的不一致。这与第 6 课“把复杂度收敛到一个能严格测试的地方”一脉相承。
3. 为什么说“在入口用 Zod 校验、越早拒绝坏数据越好”,而且 API 还能返回“部分成功”?
  1. 摄取每天吞下各种 SDK/版本的数据,难免有脏事件;入口拦下并给清晰错误,比让它流进队列、到 worker 才崩好得多;一批里坏几条只拒那几条、其余照收
  2. 因为坏数据更快
  3. 因为部分成功能省钱
  4. 因为 Zod 能加密数据
看答案与解析 点击展开
答案:A。fail-fast + 容错:在门口用 schema 拦下脏数据,避免污染下游、避免 worker 崩;同时按事件粒度返回部分成功,不因个别坏事件牵连整批。这让海量异构来源的摄取既健壮又友好。
💭 发散思考(没有标准答案,动手或动脑想想)
  • 把“接收”和“处理”切开、入口只做最小集合——这条思路你能用到自己写的 API 上吗?哪些工作“必须当场做”,哪些可以丢给后台异步?切错了会怎样?

Part 2 finished "what the data looks like and where it lives". From here we walk the ingestion path — the close-up of Lesson 5's "write trunk". First stop: the public ingestion API — how the server receives a batch of events your app sends. Spoiler on its character: extremely fast and light. It does just three things — auth + validate + enqueue — then returns immediately, never merging, computing cost, or writing the DB here (those are the worker's job). This is Lesson 5's "fast lane".

🔌 Analogy
Picture the ingestion API as a courier's drop-off counter: the courier (your SDK) brings a batch of parcels (events), and the clerk (the API) does three things before waving you through — check the sender's identity (auth: which project is this key?), scan and log the labels (validate: is each parcel well-formed?), drop them on the belt (enqueue). Then it hands you a receipt on the spot (HTTP 207) and lets you go — it won't sort, weigh and deliver in front of you. The slow work is done by people down the belt (the worker). The counter's only goal: accept fast, never block the door.
the life of one ingestion request (milliseconds) SDK POSTa batch of events 1 authkey→project 2 Zod validatewell-formed? 3 processEventBatchenqueue (+ S3) 207return now merge, cost, ClickHouse writes are not here — that's the worker (L15–17) the API's entire duty: accept → validate → enqueue → receipt
A minimal accept flow: auth (API key resolves a project, L09·10) → Zod-validate event shapes → processEventBatch enqueues (and lands originals in S3) → immediate 207. No "heavy work" happens here, so the API always returns in milliseconds.
Example: one ingestion request and the 207 response POST /api/public/ingestion Authorization: Basic <pk:sk> { "batch": [ { "type":"trace-create", "body":{"id":"chat_a1b2","name":"qa"} }, { "type":"observation-create", "body":{"id":"obs_7f","type":"GENERATION", "traceId":"chat_a1b2"} } ] } ↑ request (batched, sent once) 207 Multi-Status { "successes": [ {"id":"chat_a1b2","status":201}, {"id":"obs_7f","status":201} ], "errors": [] } per-event success/failure, one 207 for the batch ↑ response (returns early, before persistence)
One round trip (status and batch shape per web/src/pages/api/public/ingestion.ts; values illustrative): the client batches several events into one POST; the server validates, enqueues and returns 207 Multi-Status — successes/errors give a per-event result. It's 207, not 200, because a batch can be partly OK and partly failed.

The life of one ingestion request

The entry is the handler in web/src/pages/api/public/ingestion.ts. After taking the body, the real work is a shared function called processEventBatch:

web/src/pages/api/public/ingestion.tshandler
import { processEventBatch } from "@langfuse/shared/src/server";

export default async function handler(req, res) {
  // …auth, parse body…
  const result = await processEventBatch(batchForProcessing, authCheck);
  // return the result immediately (success / partial failure), not waiting for the worker
}

One design detail: processEventBatch is shared — besides /ingestion, even the standalone "submit a score" endpoint reuses it. Why? Because "accept a batch, validate, enqueue" is needed everywhere; factoring it into one function avoids re-implementing (and re-bugging) it per endpoint (again Lesson 6's "converge complexity to one place"). The event types it supports cover TRACE_CREATE / OBSERVATION_CREATE / SPAN_CREATE / GENERATION_CREATE / SCORE_CREATE and more (including SDK_LOG, DATASET_RUN_ITEM_CREATE) — exactly Lesson 6's event envelope.

Why a "batch", not "one"

The SDK doesn't fire one HTTP per event — it collects a batch and sends it. This "batch" design saves a lot:

😖 one-by-one

Every observation a full HTTP round trip; at high frequency the network overhead is huge and connections explode — a real drag on your app.

😀 batched

Dozens or hundreds of events in one request, one round trip. Network overhead amortizes, and the API validates and enqueues in one go — far higher throughput.

Batching also gives a semantic benefit: a batch may hold both a CREATE and an UPDATE for one observation, and the server can process them together, cutting round trips. Of course a batch can't be infinite — when the body is too large the API rejects it ("payload too large"), another guard: don't let one request blow up the server.

1

SDK batches → POST /ingestion

the app packs a window of events into one batch and sends it.

2

handler authenticates

resolve the project from the API key and check write permission (fail → reject the whole batch).

3

Zod-validate each event

ill-formed events are rejected; the rest continue (partial success possible).

4

processEventBatch enqueues + lands S3

valid events enter the Redis queue; originals are stored in S3 (L14).

5

return the result immediately

tell the caller which were accepted, which rejected — all in milliseconds, not waiting for the worker.

Good and bad mixed in a batch? Per-event, 207 tells you which won/failed a batch of N events event ✓ valid event ✗ malformed event ✓ valid event ✓ valid validate per eventvalid → enter the queueinvalid → record rejectedErrorsbad don't drag down good successes: acceptedinto Redis queue, originals to S3 errors: rejected (with reasons)caller can resend just these Finally HTTP 207 Multi-Status, returning {successes, errors} — the concrete meaning of "batched but partially successful"
Batch = partial success: ingestion.ts:153 filterBatchForEventsOnly filters per-event, recording invalid ones as rejectedErrors; :158 processEventBatch handles the valid ones; :162 finally res.status(207).json(result) returns {successes, errors}. In a mixed batch, bad events don't drag down good ones, and the caller can resend just the rejected few.
web/src/pages/api/public/ingestion.ts:132-162batch parse + per-event filter + 207
// the request body is "a batch array"; first Zod-validate the outer shell
const batchType = z.object({ batch: z.array(z.unknown()) });
const parsedSchema = batchType.safeParse(req.body);

// filter per-event: invalid ones rejected individually, good ones continue — mixed batch still processes
const { batchForProcessing, rejectedErrors } = filterBatchForEventsOnly(parsedSchema.data.batch);
const result = await processEventBatch(batchForProcessing, authCheck);  // enqueue + land S3
if (rejectedErrors.length > 0) result.errors = [...result.errors, ...rejectedErrors];

return res.status(207).json(result);  // 207 Multi-Status: which are { successes, errors }

Two gates: auth + validation

Before "enqueue", the API must pass two gates. They look routine but are the system's first line of security and robustness:

1 auth gate API key → which project? may it write? 2 validate gate (Zod) is each event well-formed? reject bad events early enqueue only if both pass auth decides "who you are and may you write", validation decides "is what you sent clean"
Two gates: auth resolves the project from the API key (L09: a key belongs to a project) and confirms write permission — this also pins the tenant (L10's project_id threading); Zod validation checks each event's shape, so bad data is rejected at the door and never pollutes downstream. Pass both, then enqueue.

The second gate, Zod validation, especially matters: the ingestion system swallows a flood of data from many SDKs and versions daily, inevitably with malformed or missing-field "dirty events". Rejecting them at the entry with a Zod schema and a clear error beats letting them flow into the queue and crash the worker — the earlier you reject bad data, the calmer the whole system. If a few events in a batch are bad, the API can still return "partial success", telling you which weren't accepted, rather than failing the whole batch.

This gateGuardsOn failure
authIs the API key valid? Which project? May it write?401/403, whole batch rejected
Zod validationIs each event's shape/type legalbad events rejected; partial success possible
size guardIs the body too large"payload too large", prevents blow-up

"Partial success" isn't just talk — it's baked into the response contract: the last line of ingestion.ts's handler is res.status(207).json(result), where result looks like { successes, errors } — two arrays listing the accepted events and the rejected ones (each with its error reason). The 207 Multi-Status HTTP code is purpose-built for "one request, many sub-operations, mixed outcomes". So even if 3 of your 100 events are dirty, the server accepts 97, returns the 3 with reasons, instead of throwing out the whole pot for a few bad apples. Hugely SDK-friendly: the client can retransmit just the failed ones, no full-batch retry.

🎯 Design trade-off
Why does the API only "auth + validate + enqueue", pushing all heavy work to the worker? Because the ingestion API is the layer facing your app directly, and its top KPI is "always fast, never drag the caller". Any extra work here — merging, looking up pricing, writing the DB — becomes latency on every instrumentation call and can fail on downstream hiccups. So Langfuse cuts this line very cleanly: the API keeps only the minimum that must happen on the spot (auth to prevent unauthorized writes, validation to keep dirty data out, enqueue to hand off), and everything else that can be async is async. The cost is "eventual consistency" (L05), but the gain is an ingestion entry that's almost impossible to slow or to drag your business down. The thinner the entry, the better it survives spikes.
🎯 Key points
  • The ingestion API's entire job: auth + Zod validation + enqueue, then return immediately — merge/cost/DB-write are not here (the worker's job).
  • The handler in ingestion.ts calls the shared processEventBatch (even the standalone score endpoint reuses it).
  • Batched reporting: dozens/hundreds of events per HTTP, amortizing network cost, validated and enqueued in one go; oversized batches are rejected (size guard).
  • Two gates: auth (key→project, permission, pins the tenant) + Zod validation (reject bad data at the door, partial success possible).
  • Tradeoff: the entry keeps only "must happen now", everything async-able is async — the thinner the entry the better it survives spikes, at the cost of eventual consistency.

🧪 Self-test - think about the design

1. The ingestion API does only three things on a batch, then returns immediately. Which three, and why not merge/cost/write-DB here?
  1. auth + merge + write-DB; because it's fastest
  2. auth + Zod validation + enqueue; because the API faces your app directly, its top goal is 'always fast, never drag the caller'; heavy work here would add latency to every call and fail on downstream hiccups
  3. compress + encrypt + enqueue
  4. only enqueue, no auth or validation
Show answer & explanation click to expand
Answer: B. The entry keeps only the must-do-now minimum (auth prevents unauthorized writes, validation blocks dirty data, enqueue hands off), deferring everything async to the worker. The thinner the entry, the better it survives spikes and the less it drags your business — at the cost of eventual consistency (L05).
2. The ingestion.ts handler delegates the real work to a shared processEventBatch, reused even by the standalone score endpoint. The benefit?
  1. to make the score endpoint slower
  2. 'accept a batch, validate, enqueue' is needed everywhere; factoring it into one place avoids re-implementing (and re-bugging) it per endpoint (converge complexity, L06)
  3. just to write more code
  4. because scores can't use the queue
Show answer & explanation click to expand
Answer: B. Multiple endpoints share one ingestion routine, maintained and tested in one place, avoiding inconsistency from scattered implementations — in line with L06's 'converge complexity to one strictly testable place'.
3. Why 'validate with Zod at the entry, reject bad data as early as possible', and why can the API return 'partial success'?
  1. Ingestion swallows data from many SDKs/versions daily, inevitably with dirty events; rejecting at the entry with a clear error beats letting them crash the worker; a few bad events in a batch get rejected while the rest are accepted
  2. because bad data is faster
  3. because partial success saves money
  4. because Zod encrypts data
Show answer & explanation click to expand
Answer: A. Fail-fast + fault tolerance: a schema blocks dirty data at the door, sparing downstream and the worker; per-event partial success avoids letting a few bad events doom the whole batch. This keeps ingestion from massive heterogeneous sources both robust and friendly.
💭 Open questions (no single right answer - just think or try)
  • Splitting 'accept' from 'process' so the entry does the minimum — can you apply this to your own APIs? Which work 'must happen now', and which can be deferred to async background? What if you split it wrong?