第 43 课讲的是「平台向你收费」,这一课反过来:把你的数据,源源不断地送回到你自己的地盘。Langfuse 攒下的 trace、observation、score,你可能想拉进自家的 PostHog / Mixpanel 做产品分析,或者落到 S3 / GCS / Azure 的对象存储里长期归档、喂给数据仓库。这一课讲这套「你的数据,你的去处」的分析集成怎么实现——三个长得几乎一样的集成(PostHog、Mixpanel、Blob Storage),共享同一套漂亮的工程骨架:两级 fan-out 队列(定时调度 → 逐项目处理)、lastSyncAt 水位线的增量同步(只送新数据、不重不漏)、以及 blob 导出的流式处理(再大的数据量也不会撑爆内存)。 看完你会明白:一个成熟平台「往外导数据」这件事,远不止「写个定时任务 SELECT 一把发出去」那么简单。
lastSyncAt 水位线)。② 不能一个人扛着所有部门的抄写活——派一个调度员清点「哪些部门要抄送」,再给每个部门单独派一个抄写员并行干(两级 fan-out)。③ 给档案库抄那种超厚的账本时,不能先把整本搬到桌上再抄(桌子会塌=内存爆)——而是翻一页抄一页、抄完即走(流式处理)。这三条朴素的智慧,正是这一课的全部精髓。
分析集成有三个:PostHog、Mixpanel(产品分析),Blob Storage(S3 / S3 兼容 / Azure 对象存储)。它们的数据模型长得几乎一样,都以 projectId 为主键(一个项目一份配置)、都把访问凭据加密保管(又是第 39 课的加密)、都有一个 lastSyncAt 记「上次同步到哪」、一个 enabled 开关、一个 exportSource 选「导哪些数据」(TRACES_OBSERVATIONS / +EVENTS / 仅 EVENTS)。Blob Storage 配置更丰富:bucket、prefix、文件格式(CSV/JSON/JSONL)、是否压缩、导出频率等。
schema.prisma:1183 PosthogIntegration、:1196 MixpanelIntegration、:1209 BlobStorageIntegration——均以 projectId 为主键,存加密凭据(encryptedPosthogApiKey / encryptedMixpanelProjectToken / secretAccessKey)、lastSyncAt、enabled、exportSource(AnalyticsIntegrationExportSource:TRACES_OBSERVATIONS / TRACES_OBSERVATIONS_EVENTS / EVENTS)。PosthogIntegration 与 BlobStorageIntegration;值为示例):traces/observations 向上以 事件的形式 capture({event, distinct_id, properties}) 进 PostHog(发往 posthogHostName、用 phc_ 项目 key),做产品分析、留存漏斗;向下以文件的形式周期导出到 BlobStorage(type=S3/S3_COMPATIBLE/AZURE_BLOB_STORAGE,fileType∈CSV/JSON/JSONL,落到 bucket/prefix),喂数仓/离线分析。一个增量游标保证每次只导新增行、不重不漏。Langfuse 不替代这些工具,而是把数据准确地送过去。| 集成 | 去处 | 凭据(加密) | 特有配置 |
|---|---|---|---|
| PostHog | 产品分析平台 | encryptedPosthogApiKey + host | — |
| Mixpanel | 产品分析平台 | encryptedMixpanelProjectToken | mixpanelRegion(数据驻留区) |
| Blob Storage | S3 / S3兼容 / Azure | secretAccessKey(可选→也可用 IAM 角色) | bucket/prefix/fileType/compressed/exportFrequency/exportMode |
导数据不能用「一个大任务串行处理所有项目」——某个项目慢或失败会拖累全部。Langfuse 用经典的两级 fan-out:每个集成有两条队列。① 调度队列(cron 定时触发):handlePostHogIntegrationSchedule 查出所有 enabled: true 的集成,addBulk 给每个项目各塞一个处理任务。② 处理队列:每个任务独立处理一个项目的导出,包在 instrumentAsync 里各开一条新 trace(OTel CONSUMER span,方便单独观测)。三个集成各有自己独立的两条队列——PostHog 堵了不影响 Mixpanel,互不拖累。
postHogIntegrationProcessor(调度,postHogIntegrationQueue.ts:11)→ handlePostHogIntegrationSchedule 查 enabled:true 后 addBulk(:35)扇出;postHogIntegrationProcessingProcessor(:23)每任务 instrumentAsync(startNewTrace) 处理一个项目。去重 jobId=projectId-lastSyncAt(:50)、removeOnFail(:51)。Mixpanel/BlobStorage 同构。// 调度队列:查出所有启用的集成,给每个项目扇出一个处理任务 const projects = await prisma.posthogIntegration.findMany({ select: { lastSyncAt: true, projectId: true }, where: { enabled: true }, // 只调度开了开关的 }); await processingQueue.addBulk(projects.map((integration) => ({ name: QueueJobs.PostHogIntegrationProcessingJob, data: { payload: { projectId: integration.projectId } }, opts: { // 去重:同一项目 + 同一 lastSyncAt 窗口,不会被排两次 jobId: `${integration.projectId}-${integration.lastSyncAt?.toISOString() ?? ""}`, removeOnFail: true, // 失败立即清理,不挡下个周期 }, })));
findMany({enabled:true}) 查出所有该同步的项目,addBulk 一次性把「每项目一个任务」塞进处理队列。本身不干重活,只负责「派活」。instrumentAsync(startNewTrace) 各开一条 trace,可被多 worker 并行消费、各自独立观测。jobId=projectId-lastSyncAt 保证「同项目同窗口」不重复入队(幂等);removeOnFail 让失败任务立即出队、不挡下个 cron 周期重排。三集成各有独立队列,互不连累。每个项目的处理任务,核心是增量:只导上次之后的新数据。窗口下界 minTimestamp = lastSyncAt || project.createdAt(首次跑就从项目创建时间起);上界 maxTimestamp = min(下一个UTC日界, 当前-30分钟)——留 30 分钟缓冲避免抓到还在陆续到达的数据,按日切块避免一个卡住的集成每次都重扫越来越大的窗口(还顺带对齐 ClickHouse 的分区裁剪)。导完后 update({lastSyncAt: maxTimestamp}) 把书签往前挪。对 Blob Storage 这种可能很大的导出,还用 Node stream pipeline:一行行查、一行行转格式(CSV/JSON/JSONL)、可选 gzip 压缩、直接流进 S3/GCS——绝不把整个结果集装进内存。
minTimestamp = 上次的 lastSyncAt(首次 = 项目 createdAt);maxTimestamp = min(下一个 UTC 日界, now−30min)。空窗口直接跳过。
decrypt 还原 API key / token / secret,从 ClickHouse 查 [min, max) 区间的 trace/observation/score(按 exportSource 选类型)。
PostHog/Mixpanel:转成事件 posthog.capture() 逐条上报;Blob:流式 pipeline 一行行转 CSV/JSON/JSONL(±gzip) 经 StorageServiceFactory 写 S3/S3兼容/Azure。
成功后 update({ lastSyncAt: maxTimestamp })。下个周期从这里接着导——天然增量、不重不漏。
handlePostHogIntegrationProjectJob.ts:322 minTimestamp = lastSyncAt || project.createdAt;:324 上界先取 now−30min,再 :339 与「下一个 UTC 日界」取 min(按日切块、对齐 CH 分区);:394 成功后 posthogIntegration.update({lastSyncAt: maxTimestamp})。Blob 导出用 StorageServiceFactory + stream pipeline/Transform(handleBlobStorageIntegrationProjectJob.ts:1,9)。removeOnFail 掉,其余照常。② 并行——N 个项目的处理任务可以被多个 worker 并行消费,而不是排成一长串串行干。③ 可观测——每个项目的处理 instrumentAsync(startNewTrace) 各开一条 trace,出问题时能精确定位是哪个项目、哪一步。这套「一个调度员清点、给每个对象单独派活」的 fan-out,在第 30 课 eval、第 43 课计量里反复出现——是 Langfuse 处理「对一批对象各做一遍」的标准答案。lastSyncAt 落后很多,如果一次性扫「从那时到现在」的超大窗口,每小时重试都在重扫巨量数据、永远追不上;按 UTC 日界切成一天一段,既限制单次工作量,又正好对齐 ClickHouse 按天分区的裁剪,查得更快。增量同步的精髓不是「同步到最新」,而是「每次稳稳推进一小步、且这一步可控」。
projectId 为主键、加密凭据(复用第39课)、lastSyncAt 水位、enabled、exportSource。addBulk 逐项目扇出,jobId=projectId-lastSyncAt 去重、removeOnFail)+ 处理队列(每项目 instrumentAsync 各开新 trace)。三集成各有独立两队列,互不拖累。min=lastSyncAt||createdAt,max=min(次日界, now−30min);30 分钟缓冲避在途数据、按日切块限单次工作量并对齐 CH 分区;成功后挪 lastSyncAt。只导新数据、不重不漏。stream pipeline/Transform 一行行转 CSV/JSON/JSONL(±gzip),经 StorageServiceFactory 写 S3/GCS/Azure——再大也不撑爆内存。fan-out/增量/流式,是「往外导数据」的三件套智慧。Lesson 43 was "the platform charging you"; this lesson is the reverse: sending your data, continuously, back to your own turf. The traces, observations, and scores Langfuse accumulates — you may want to pull them into your own PostHog / Mixpanel for product analytics, or land them in S3 / GCS / Azure object storage for long-term archival and feeding a data warehouse. This lesson covers how that "your data, your destinations" analytics integration works — three nearly-identical integrations (PostHog, Mixpanel, Blob Storage) sharing one elegant engineering skeleton: a two-level fan-out queue (scheduled dispatch → per-project processing), incremental sync via a lastSyncAt watermark (only new data, no-dup-no-miss), and streaming for blob export (no memory blowup no matter the volume). By the end you'll see: a mature platform "exporting data out" is far more than "write a cron job, SELECT once, send it off."
lastSyncAt watermark). ② Don't have one person shoulder the copying for all departments — assign a dispatcher to tally "which departments need a copy," then assign a separate scribe per department to work in parallel (two-level fan-out). ③ When copying a super-thick ledger to the vault, don't haul the whole book onto the desk first (the desk collapses = memory blows up) — instead flip a page, copy a page, move on (streaming). These three plain pieces of wisdom are the whole essence of this lesson.
There are three analytics integrations: PostHog, Mixpanel (product analytics), and Blob Storage (S3 / S3-compatible / Azure object storage). Their data models look nearly identical — all keyed by projectId (one config per project), all keeping access credentials encrypted (Lesson 39's encryption again), all with a lastSyncAt recording "where the last sync reached," an enabled switch, and an exportSource choosing "which data to export" (TRACES_OBSERVATIONS / +EVENTS / EVENTS only). Blob Storage's config is richer: bucket, prefix, file format (CSV/JSON/JSONL), whether to compress, export frequency, etc.
schema.prisma:1183 PosthogIntegration, :1196 MixpanelIntegration, :1209 BlobStorageIntegration — all keyed by projectId, storing encrypted credentials (encryptedPosthogApiKey / encryptedMixpanelProjectToken / secretAccessKey), lastSyncAt, enabled, exportSource (AnalyticsIntegrationExportSource: TRACES_OBSERVATIONS / TRACES_OBSERVATIONS_EVENTS / EVENTS).PosthogIntegration and BlobStorageIntegration; values illustrative): traces/observations go up as events via capture({event, distinct_id, properties}) into PostHog (to posthogHostName with a phc_ project key) for product analytics and funnels; and down as files exported periodically to BlobStorage (type=S3/S3_COMPATIBLE/AZURE_BLOB_STORAGE, fileType∈CSV/JSON/JSONL, into a bucket/prefix) for the warehouse / offline analysis. An incremental cursor makes each run export only new rows — no gaps, no dupes. Langfuse doesn't replace these tools; it delivers the data to them accurately.| Integration | Destination | Credentials (encrypted) | Specific config |
|---|---|---|---|
| PostHog | product analytics platform | encryptedPosthogApiKey + host | — |
| Mixpanel | product analytics platform | encryptedMixpanelProjectToken | mixpanelRegion (data residency) |
| Blob Storage | S3 / S3-compatible / Azure | secretAccessKey (optional → can use IAM role) | bucket/prefix/fileType/compressed/exportFrequency/exportMode |
Exporting data can't use "one big task serially processing all projects" — one slow or failing project would drag down all the rest. Langfuse uses the classic two-level fan-out: each integration has two queues. ① Scheduling queue (cron-triggered): handlePostHogIntegrationSchedule finds all enabled: true integrations and addBulks one processing task per project. ② Processing queue: each task independently handles one project's export, wrapped in instrumentAsync that starts a fresh trace per project (an OTel CONSUMER span, for isolated observability). The three integrations each have their own independent pair of queues — a PostHog jam doesn't affect Mixpanel, no mutual drag.
postHogIntegrationProcessor (scheduling, postHogIntegrationQueue.ts:11) → handlePostHogIntegrationSchedule finds enabled:true then addBulk (:35) fans out; postHogIntegrationProcessingProcessor (:23) per-task instrumentAsync(startNewTrace) handles one project. Dedup jobId=projectId-lastSyncAt (:50), removeOnFail (:51). Mixpanel/BlobStorage are isomorphic.// scheduling queue: find all enabled integrations, fan out one task per project const projects = await prisma.posthogIntegration.findMany({ select: { lastSyncAt: true, projectId: true }, where: { enabled: true }, // only schedule the switched-on ones }); await processingQueue.addBulk(projects.map((integration) => ({ name: QueueJobs.PostHogIntegrationProcessingJob, data: { payload: { projectId: integration.projectId } }, opts: { // dedup: same project + same lastSyncAt window won't be scheduled twice jobId: `${integration.projectId}-${integration.lastSyncAt?.toISOString() ?? ""}`, removeOnFail: true, // clean failures immediately, don't block next cycle }, })));
findMany({enabled:true}) finds all projects due for sync, addBulk stuffs "one task per project" into the processing queue at once. It does no heavy lifting itself — only "assigns work."instrumentAsync(startNewTrace) opens its own trace, consumable by multiple workers in parallel, each independently observable.jobId=projectId-lastSyncAt ensures "same project, same window" isn't enqueued twice (idempotent); removeOnFail dequeues failed tasks immediately, not blocking the next cron cycle's re-queue. Three integrations, independent queues, no mutual drag.Each project's processing task is at its core incremental: export only the new data since last time. The window's lower bound minTimestamp = lastSyncAt || project.createdAt (first run starts from project creation); the upper bound maxTimestamp = min(next UTC day boundary, now − 30 minutes) — leaving a 30-minute buffer to avoid grabbing still-arriving data, and chunking by day so a stuck integration doesn't re-scan an ever-growing window each run (also aligning with ClickHouse partition pruning). After exporting, update({lastSyncAt: maxTimestamp}) advances the bookmark. For potentially-huge Blob Storage exports, it uses a Node stream pipeline: query row by row, transform format row by row (CSV/JSON/JSONL), optionally gzip, and stream straight into S3/GCS — never loading the whole result set into memory.
minTimestamp = previous lastSyncAt (first time = project createdAt); maxTimestamp = min(next UTC day boundary, now−30min). An empty window is skipped.
decrypt the API key / token / secret, query the [min, max) slice of trace/observation/score from ClickHouse (type chosen by exportSource).
PostHog/Mixpanel: transform into events, posthog.capture() one by one; Blob: streaming pipeline transforms row by row into CSV/JSON/JSONL (±gzip), writes to S3/S3-compatible/Azure via StorageServiceFactory.
On success update({ lastSyncAt: maxTimestamp }). The next cycle continues from here — naturally incremental, no-dup-no-miss.
handlePostHogIntegrationProjectJob.ts:322 minTimestamp = lastSyncAt || project.createdAt; :324 upper bound first takes now−30min, then :339 min'd with the "next UTC day boundary" (day-chunking, aligns CH partitions); :394 on success posthogIntegration.update({lastSyncAt: maxTimestamp}). Blob export uses StorageServiceFactory + stream pipeline/Transform (handleBlobStorageIntegrationProjectJob.ts:1,9).removeOnFails itself, the rest carry on. ② parallelism — N projects' processing tasks can be consumed by multiple workers in parallel, rather than queued into one long serial chain. ③ observability — each project's processing instrumentAsync(startNewTrace) opens its own trace, so when something breaks you can pinpoint which project, which step. This "one dispatcher tallies, assigns separate work per object" fan-out recurs in Lesson 30's eval and Lesson 43's metering — it's Langfuse's standard answer for "do one pass over a batch of objects."lastSyncAt lags far behind; scanning "from then to now" as one giant window means every hourly retry re-scans massive data, never catching up. Slicing by UTC day boundary bounds per-run work and aligns exactly with ClickHouse's day partition pruning, querying faster. The essence of incremental sync isn't "sync to the latest," but "advance one small, controlled step steadily each time."
projectId, encrypted credentials (reuse Lesson 39), lastSyncAt watermark, enabled, exportSource.addBulk fans out per project, jobId=projectId-lastSyncAt dedup, removeOnFail) + processing queue (each project instrumentAsync opens a fresh trace). Each integration has its own queue pair, no mutual drag.min=lastSyncAt||createdAt, max=min(next-day-boundary, now−30min); the 30-min buffer avoids in-flight data, day-chunking bounds per-run work and aligns CH partitions; advance lastSyncAt on success. Only new data, no-dup-no-miss.stream pipeline/Transform transforms row by row into CSV/JSON/JSONL (±gzip), writes to S3/GCS/Azure via StorageServiceFactory — no memory blowup at any size. Fan-out / incremental / streaming are the trio of wisdom for "exporting data out."