🪢 Langfuse 图解教程Langfuse Visual Guide 第十部分 · 平台与运维Part 10 · Platform & Operations 52 / 55
第十部分 · 平台与运维Part 10 · Platform & Operations

数据生命周期与删除Data lifecycle & deletion

🧭 读前Before you start
先读 第7课 的四存储与 第19课 的媒体,本课讲数据保留期与跨存储删除。
Read Lesson 7 (the four stores) and Lesson 19 (media); this lesson covers retention and cross-store deletion.

数据有一生:被摄取写入(出生)、被查询分析(活着)、最终被删除(落幕)。前面的课讲足了「写」和「读」,这一课讲「删」——以及一个容易被忽略却很关键的事实:在 Langfuse 里,一条 trace 不是一行数据,而是散在三个存储里的三份:元数据在 Postgres、海量事件在 ClickHouse、大块输入输出/媒体在 对象存储(S3)。所以「删一条 trace / 删一个项目 / 按保留期清旧数据」都不是删一行那么简单,而是要跨三个存储协调一致地删干净——漏删任何一处,都是数据残留(合规与隐私的隐患)。 这一课讲三种删除(按保留期自动清理、按需删 trace、级联删项目)怎么跨存储删、为什么删除有讲究的顺序;最后看后台迁移——怎么在不停机的前提下,给亿万行的表「在线做手术」。

📋 生活类比
把一条 trace 想成一份跨三个档案室存放的卷宗:登记处(Postgres)有它的索引卡(谁、何时、属于哪个项目),主库房(ClickHouse)有它海量的明细记录,大件仓库(S3)放着它体积过大的附件(完整输入输出、上传的图片)。 现在要「销毁」这份卷宗——只撕掉登记处那张索引卡,可远远不够:主库房的明细、大件仓库的附件还在,这就是数据残留。真正的销毁必须三个档案室一处不漏地清。而且销毁有讲究的次序:要先清主库房和大件仓库、最后才撕登记处的卡——因为那张索引卡是「这份卷宗还没销毁完」的唯一凭据,万一中途断电,靠它才能知道「还得回来接着销毁」;要是先撕了卡,剩下的明细和附件就成了没人认领的孤儿,再也对不上是谁的了。

一条 trace 散在三处:删除要跨存储

回顾全书的存储分工:Postgres 存配置与元数据(项目、用户、API key、各种设置);ClickHouse 存海量、可分析的事件(trace/observation/score,第3-4部分的主角);对象存储 S3 存体积大的原始负载(完整的输入输出、上传的媒体)。这套分工让读写各得其所,但也意味着:删除必须同时清三处。以「删整个项目」为例,projectDeleteProcessor 的顺序很讲究——先并行清 ClickHouse 与 S3(traces/observations/scores/events + blob 引用),清完才删 Postgres。这个「PG 最后删」不是随意的:Postgres 里的项目记录是「这个项目还没删完」的锚点,留到最后,万一中途崩了还能靠它重试;若先删 PG,剩下的 CH/S3 数据就成了无主孤儿,再难清理。

一条 trace = 三个存储里的三份,删除要清干净 Postgres元数据:项目/用户/key/设置「源真相」+ 重试锚点 ClickHouse海量事件 trace/obs/score/event可分析的明细 对象存储 S3大块负载:完整输入输出/媒体 projectDelete 顺序① 并行清 CH + S3Promise.all(traces,obs,scores,events, blob 引用)② 最后才删 Postgres 为何 PG 留到最后?它是「还没删完」的锚点崩了能靠它重试 若先删 PG?CH/S3 数据成无主孤儿再难对上、难清理
跨存储删除:projectDelete.ts:22 projectDeleteProcessor——Promise.all 先清 ClickHouse(deleteTracesByProjectId/deleteObservationsByProjectId/deleteScoresByProjectId/v4 的 deleteEventsByProjectId)与 S3(removeIngestionEventsFromS3AndDeleteClickhouseRefsForProject),再删 dataset run items,最后删 Postgres。注释明确「Delete project data from ClickHouse first」。
示例:一条 trace 的生命周期与删除 ①创建 ②保留窗口 ③到期/请求删除 ④跨存储删除 data-retention / data-access-days ④a 软标记(先快后稳) pendingDeletion.create({isDeleted:false}) → 延迟队列 (LANGFUSE_TRACE_DELETE_DELAY_MS) ④b 硬删除 · worker 并行 Promise.all Postgrestrace 行 / pending_deletions ClickHousetraces·observations·scores S3 media 只删孤儿(NOT EXISTS 守卫)被别的 trace 引用的共享 media 保留 全部成功后置 isDeleted=true;分批(LANGFUSE_DELETE_BATCH_SIZE)、可重试
删除不是「DROP 一行」,而是一条跨三处存储的有序流程(取自 traceDeletionProcessor 与 processClickhouseTraceDelete;值为示例):数据生而被保留一段窗口(data-retention / data-access-days)。到期或主动删除时,先在 Postgres 的 pending_deletions 里软标记并入延迟队列(先快速响应),再由 worker Promise.all 并行删 Postgres 与 ClickHouse 的 traces/observations/scores;ClickHouse 这步还顺带删 S3 上的 孤儿 media——用 NOT EXISTS 守卫,被别的 trace 引用的共享 media 不误删。全部成功才把 isDeleted 置真,整个过程分批、可重试。
元数据Postgres(最后删)
项目、用户、API key、各种配置——结构化、强一致、量不大。它是整个系统的「源真相」,删除流程里留到最后:作为「这条删除还没完成」的锚点,保证崩溃后能安全重试。
事件ClickHouse(先删)
trace/observation/score 这些海量、可分析的明细(第 3–4 部分讲过它为何用列存)。删除量可能巨大,用按 projectId / 按时间的批量删除,和 S3 并行清理。
负载对象存储 S3(先删)
完整的输入输出、上传的媒体等大块二进制。删 CH 行的同时,也要把它在 S3 的对象和引用清掉——否则就是「指针没了、数据还占着空间」的泄漏。

三种删除:按期清、按需删、级联删

Langfuse 有三类删除,场景不同但都要跨存储:① 数据保留(data retention)——自动、周期性、按年龄清旧数据。它是 EE 功能(第 50 课的 entitlement),每个项目可设 retentionDays;调度走和第 46 课一模一样的两级 fan-out:dataRetentionProcessor 找出设了保留期的项目、给每个派一个处理任务,dataRetentionProcessingProcessor 算出 cutoffDate = 今天 − retentionDays,删掉比它更老的 CH 数据与 S3 媒体。② trace 删除——按需删指定 trace(第 47 课批量操作的 trace-delete 就走它,traceDeleteProcessor)。③ 项目删除——级联删整个项目的所有数据(上一节的 projectDeleteProcessor)。

三类删除,场景不同,但都要跨存储 ① 数据保留(EE) 自动 · 周期触发 按龄 cutoff = now − retentionDays 两级 fan-out(同 L46) 清 CH + S3 老数据 ② trace 删除 按需触发(L47 批量) 指定 traceIds traceDeleteProcessor 清 CH + S3 ③ 项目删除 级联触发 整个项目所有数据 projectDeleteProcessor 清 CH+S3 → PG 最后 共同点:都跨 ClickHouse + S3(项目删除还含 Postgres,且 PG 留到最后)
1

数据保留:调度 fan-out(EE)

dataRetentionProcessor 周期触发,查出设了 retentionDays 的项目,给每个项目派一个处理任务——和第 46 课分析集成同一套两级 fan-out骨架。

2

数据保留:逐项目按龄清理

每任务算 cutoffDate = now − retentionDays,deleteTracesOlderThanDays 等并行清 CH 的 traces/observations/scores/events,并清 S3 媒体——只删过期的,不动新数据。

3

trace 删除:按需

用户或第 47 课批量操作要删指定 trace,traceDeleteProcessor 按 traceIds 跨存储删(同样 CH + S3 一起清)。

4

项目删除:级联

删整个项目时 projectDeleteProcessor 先并行清 CH + S3、再删 dataset run items、最后删 PG(上一节的「PG 留到最后」)。

worker/src/ee/dataRetention/handleDataRetentionProcessingJob.ts按龄跨存储清理
// 数据保留是 EE 功能:每个项目可设 retentionDays(第50课 entitlement)
const { retentionDays } = await prisma.project.findUnique({ where: { id: projectId },
                                                       select: { retentionDays: true } });
if (!retentionDays) return;                          // 没设保留期 → 不清理

const cutoffDate = new Date(now − retentionDays * 86400_000);   // 今天往前推 N 天

// 跨存储并行清理「比 cutoff 更老」的数据:先 S3 媒体 + ClickHouse
await Promise.all([
  removeIngestionEventsFromS3AndDeleteClickhouseRefsForProject(projectId, cutoffDate),
  deleteTracesOlderThanDays(projectId, cutoffDate),
  deleteObservationsOlderThanDays(projectId, cutoffDate),
  deleteScoresOlderThanDays(projectId, cutoffDate),
  deleteEventsOlderThanDays(projectId, cutoffDate),
]);   // 只删过期的,新数据原封不动

后台迁移:给亿万行的表「在线做手术」

数据生命周期还有一面:schema 演进。普通的数据库迁移(加个列、改个约束)能在部署时几秒内跑完。但 Langfuse 的 ClickHouse 里动辄几十亿行,要给它们回填一个新字段、或重组数据布局,可能要跑几小时——你不可能为此停机几小时。BackgroundMigration 就是为这种「大手术」设计的:把迁移做成一个能后台慢慢跑、能断点续传、全程不停机的长任务。

它的模型(schema.prisma:284)藏着三个关键字段:state(Json)——存「跑到哪了」的进度,崩溃重启后从这里接着跑(和第 46 课 lastSyncAt 水位线同一种「可恢复」智慧);lockedAt/workerId——分布式锁,保证同一迁移只有一个 worker 在跑(多 worker 部署下不重复、不打架);finishedAt/failedReason——记结果。BackgroundMigrationManager 用一个带心跳(heartbeat)的循环驱动它:持锁、跑一批、更新 state、续约心跳,直到完成。整个过程应用照常服务,用户无感。

不停机给亿万行的表做手术:慢跑 + 续传 + 单跑 BackgroundMigration 模型 state (Json)进度 → 崩溃从此续跑 lockedAt / workerId分布式锁 → 只一个 worker 跑 finishedAt / failedReason结果留痕 Manager 心跳循环① 抢锁(lockedAt)② 跑一批③ 更新 state④ 续心跳 heartBeat循环直到 finishedAt 不停机应用照常服务用户无感几小时的大手术在后台慢慢完成 每批回写 state,可中断可续
在线 schema 演进:schema.prisma:284 BackgroundMigration(name 唯一、script、args、state Json 可恢复、finishedAt/failedAt/failedReason、workerId/lockedAt 锁);backgroundMigrationManager.ts:6 BackgroundMigrationManager.run(:17 heartBeat 心跳、:42 while 循环、持锁跑批、:127 migration.run(args)),被 instrumentAsync 包裹。接口 IBackgroundMigration(validate/run/abort)。
挑战普通迁移后台迁移 BackgroundMigration
给亿万行回填字段部署时跑、可能要停机几小时后台慢慢跑、不停机,应用照常服务
中途崩了要么回滚、要么从头再来state 记进度,从断点续跑(同第46课水位)
多 worker 部署—lockedAt/workerId 锁,同一迁移只一个在跑
跑到哪了 / 失败原因看部署日志finishedAt/failedReason 落库可查
🎯 设计取舍
删除为什么要讲究「先删 ClickHouse/S3,最后才删 Postgres」这个顺序?反过来不行吗? 这是一个「保留重试锚点」的经典考量,和第 47 课「幂等」「快照边界」是同一类清醒。一次跨三存储的删除不是原子的——它由多步组成,任何一步都可能因崩溃、超时、网络抖动而中断。关键问题是:中断后,系统还知不知道「这件事没做完、得回来接着做」? Postgres 里的项目记录,恰好就是这个「待办凭据」:只要它还在,删除任务重试时就知道「这个项目还没清干净」,继续把 CH/S3 的残留扫掉。可一旦你先删了 PG,这个凭据就没了——此时若 CH/S3 还有残留,系统再也无从知道它们的存在,它们就成了永久占着空间、却没人认领的孤儿数据。所以「源真相 / 锚点最后删」是分布式删除的通用纪律:先清外围、后清锚点,让「未完成」始终可被发现、可被重试。

明明可以写个一次性脚本跑迁移,为什么要专门做一套带 state、锁、心跳的 BackgroundMigration 框架? 因为「给生产环境亿万行的表做变更」这件事,同时撞上三个硬约束:① 不能停机——业务要持续服务,不可能锁表几小时;② 会中断——跑几小时的任务,期间一定会遇到部署、重启、崩溃,必须能从断点续跑而非从头再来(这就需要把进度持久化进 state,和第 46 课增量同步的 lastSyncAt 是同一招);③ 多实例——生产有多个 worker,必须保证同一迁移只被一个 worker 执行,否则重复跑会把数据改乱(这就需要 lockedAt 分布式锁 + 心跳续约)。一次性脚本三条全占不住。把这些共性沉淀成一个框架后,每个具体迁移只需实现 validate/run/abort,复杂的「续传、加锁、留痕」由框架统一保证——把「安全地在线演进数据」这件难事,做成一个可复用、可信赖的底座。
🎯 本课要点
  • 一条 trace 散在三处:元数据在 Postgres、海量事件在 ClickHouse、大块负载在对象存储 S3。所以删除必须跨三个存储一处不漏地清,漏删即数据残留(合规/隐私隐患)。
  • 删除有顺序:projectDelete 先并行清 CH + S3、最后才删 Postgres。PG 记录是「这条删除没完成」的重试锚点,留到最后保证崩溃可重试;先删 PG 会让 CH/S3 数据成无主孤儿。
  • 三种删除:① 数据保留(EE,按 retentionDays 自动清旧,走第46课同款两级 fan-out,cutoffDate 只删过期);② trace 删除(按需,第47课批量操作走它);③ 项目删除(级联清整个项目)。
  • 后台迁移在线演进 schema:给亿万行的表回填/重组,几小时的大手术不能停机。BackgroundMigration:state 记进度可断点续传(同第46课水位)、lockedAt/workerId 锁保证单 worker 执行、心跳续约、结果落库。
  • 一脉相承:删除顺序=保留重试锚点(同第47课幂等)、后台迁移 state=可恢复(同第46课增量)、单 worker 锁(同第43课计量互斥)。分布式系统里「安全地做有副作用的大事」,反复用同一套智慧。

🧪 自测 · 想一想为什么这么设计

1. Langfuse 删除一个项目时,projectDelete 的顺序是「先并行清 ClickHouse 和 S3,最后才删 Postgres」。为什么 Postgres 要留到最后删?
  1. ClickHouse 必须在 Postgres 之前初始化
  2. PG里的项目记录是「这次删除还没完成」的重试锚点:跨存储删除不是原子的,中途可能崩。只要PG记录还在,重试就知道「还得继续清CH/S3残留」;若先删PG,剩下的CH/S3数据就成了没人认领、再也对不上的孤儿,永久占空间
  3. 这只是代码书写的习惯,没有实际意义
  4. Postgres 删除最慢,放最后省时间
看答案与解析 点击展开
答案:B。一次删除要跨 Postgres + ClickHouse + S3 三个存储,这不是一个原子事务——它是多步操作,任何一步都可能因崩溃/超时/网络抖动中断。核心问题是:中断之后,系统还知不知道「这事没做完、得回来接着做」?PG 里的项目记录恰好是这个「待办凭据」:留着它,删除任务重试时就知道该项目还没清干净,继续扫 CH/S3 残留;而一旦先删了 PG,这个凭据没了,若 CH/S3 还有残留,系统再也无从知道它们的存在,它们就成了永久孤儿。所以「锚点/源真相最后删」是分布式删除的通用纪律——先清外围、后清锚点,让「未完成」始终可发现、可重试。这与第47课「快照边界/幂等」一脉相承。
2. 在 Langfuse 里,删除一条 trace 不是「删一行数据库记录」那么简单。最根本的原因是?
  1. 因为 trace 数据是加密的,删除前要先解密
  2. 因为 trace 太大,一次删不完
  3. 一条trace的数据散在三个存储里:元数据在Postgres、海量事件明细在ClickHouse、大块输入输出/媒体在对象存储S3。要真正删干净必须跨三处协调一致地删,漏删任何一处都是数据残留——既占空间,更是合规与隐私隐患(如GDPR的被遗忘权)
  4. 因为删除需要管理员审批
看答案与解析 点击展开
答案:C。这是 Langfuse 存储架构的直接后果。为了让读写各得其所,一条 trace 被拆成三份存:Postgres 放结构化元数据、ClickHouse 放海量可分析的事件明细(列存,第3-4部分讲过)、S3 放体积过大的原始负载(完整输入输出、媒体)。这套分工对性能很好,但对删除提出了要求:必须三处同时清。只删 PG 那行「索引」,CH 的明细和 S3 的附件还在,就是数据残留——不仅白占存储,在隐私合规上更要命:用户行使「删除我的数据」的权利时,残留意味着没真正删除。所以 Langfuse 的每种删除(保留期清理、trace 删除、项目删除)都是跨存储操作。
3. Langfuse 的 BackgroundMigration(后台迁移)模型有 state、lockedAt/workerId、心跳等机制。它要解决的核心问题是?
  1. 压缩迁移后的数据
  2. 自动生成迁移脚本
  3. 给生产环境亿万行的表做变更(回填新字段/重组布局),同时撞上不能停机、会被部署/崩溃中断、有多个worker三个约束:state记进度可断点续传(同第46课水位)、lockedAt/workerId锁保证同一迁移只一个worker跑、心跳续约——让几小时的大手术在后台不停机完成
  4. 让数据库迁移跑得更快
看答案与解析 点击展开
答案:C。普通迁移(加列、改约束)部署时几秒跑完即可,不需要这套重型机制。但当你要给 ClickHouse 里几十亿行回填一个新字段、或重组数据布局,任务可能跑几小时,这就同时撞上三个硬约束:① 不能停机——业务得持续服务,不可能锁表几小时;② 会中断——几小时里必然遇到部署、重启、崩溃,必须能从断点续跑而非从头来(靠把进度存进 state,和第46课增量同步的 lastSyncAt 同一招);③ 多 worker——生产多实例,必须保证同一迁移只被一个 worker 执行,否则重复跑改乱数据(靠 lockedAt 分布式锁 + 心跳续约)。一次性脚本三条都占不住。BackgroundMigration 把这些共性沉淀成框架,每个迁移只实现 validate/run/abort,复杂的续传/加锁/留痕由框架统一保证。
💭 发散思考(没有标准答案,动手或动脑想想)
  • 这一课反复出现一个主题:在分布式系统里「安全地做有副作用的大事」要靠几招——删除留重试锚点、迁移用 state 可恢复、用锁保证单 worker 执行。请结合你的经验谈谈:你做过的「跨多个系统/存储的批量删除或数据迁移」遇到过哪些坑(中途失败、孤儿数据、重复执行)?你是怎么保证「不重不漏、可恢复」的?如果数据还涉及合规删除(如 GDPR 被遗忘权),又会带来哪些额外要求?

Data has a lifetime: ingested and written (born), queried and analyzed (alive), finally deleted (curtain call). Earlier lessons covered "write" and "read" thoroughly; this one covers "delete" — and an easily-overlooked but crucial fact: in Langfuse, one trace isn't one row but three copies scattered across three stores: metadata in Postgres, massive events in ClickHouse, bulky inputs/outputs/media in object storage (S3). So "delete a trace / delete a project / clean old data by retention period" is never as simple as deleting one row — it requires coordinated, consistent deletion across all three stores; missing any one leaves data residue (a compliance and privacy hazard). This lesson covers three deletions (auto-clean by retention period, on-demand trace delete, cascading project delete), how they delete across stores, why deletion has a deliberate order; and finally background migrations — how to "operate online" on a table with billions of rows without downtime.

📋 Analogy
Think of a trace as a dossier stored across three archive rooms: the registry (Postgres) has its index card (who, when, which project), the main vault (ClickHouse) has its massive detailed records, the oversized-item warehouse (S3) holds its bulky attachments (full inputs/outputs, uploaded images). Now to "destroy" this dossier — just tearing up the registry's index card is far from enough: the vault's details and the warehouse's attachments are still there, which is data residue. True destruction must clean all three archive rooms, missing none. And destruction has a deliberate order: clean the main vault and warehouse first, tear up the registry card last — because that index card is the only proof that "this dossier isn't fully destroyed yet"; if the power cuts mid-way, it's how you know to "come back and finish destroying." Tear up the card first and the remaining details and attachments become unclaimed orphans, no longer matchable to anyone.

One trace scattered across three: deletion spans stores

Recall the book-wide storage division: Postgres stores config and metadata (projects, users, API keys, various settings); ClickHouse stores massive analyzable events (trace/observation/score, the stars of Parts 3-4); object storage S3 stores bulky raw payloads (full inputs/outputs, uploaded media). This division serves reads and writes well, but also means: deletion must clean all three at once. Take "delete a whole project": projectDeleteProcessor's order is deliberate — first clean ClickHouse and S3 in parallel (traces/observations/scores/events + blob refs), only then delete Postgres. This "delete PG last" isn't arbitrary: the project record in Postgres is the anchor for "this project isn't fully deleted yet"; kept to the end, it allows a retry if a crash happens mid-way; delete PG first and the remaining CH/S3 data becomes ownerless orphans, hard to clean up.

One trace = three copies across three stores; clean them all Postgresmetadata: project/user/key/settings"source of truth" + retry anchor ClickHousemassive events trace/obs/score/eventanalyzable detail object storage S3bulky payloads: full I/O / media projectDelete order① clean CH + S3 in parallelPromise.all(traces,obs,scores,events, blob refs)② delete Postgres last Why PG last?it's the "not done yet" anchora crash can retry from it If PG deleted first?CH/S3 data become orphanshard to match, hard to clean
Cross-store deletion: projectDelete.ts:22 projectDeleteProcessor — Promise.all first cleans ClickHouse (deleteTracesByProjectId/deleteObservationsByProjectId/deleteScoresByProjectId/v4's deleteEventsByProjectId) and S3 (removeIngestionEventsFromS3AndDeleteClickhouseRefsForProject), then deletes dataset run items, finally deletes Postgres. The comment is explicit: "Delete project data from ClickHouse first."
Example: a trace's lifecycle and deletion ①created ②retention window ③expire / delete req ④cross-store delete data-retention / data-access-days ④a soft mark (fast, safe) pendingDeletion.create({isDeleted:false}) → delayed queue (LANGFUSE_TRACE_DELETE_DELAY_MS) ④b hard delete · worker Promise.all Postgrestrace rows / pending_deletions ClickHousetraces·observations·scores S3 media only orphans (NOT EXISTS guard)shared media referenced by others survives set isDeleted=true after success; batched (LANGFUSE_DELETE_BATCH_SIZE), retryable
Deletion isn't "DROP a row" but an ordered flow across three stores (from traceDeletionProcessor and processClickhouseTraceDelete; values illustrative): data is born and retained for a window (data-retention / data-access-days). On expiry or an explicit delete, it's first soft-marked in Postgres' pending_deletions and put on a delayed queue (respond fast), then a worker deletes Postgres and ClickHouse traces/observations/scores in parallel via Promise.all; the ClickHouse step also removes orphaned media from S3 — guarded by NOT EXISTS so shared media referenced by other traces is never wrongly deleted. Only after all succeed is isDeleted set true; the whole process is batched and retryable.
metadataPostgres (deleted last)
Projects, users, API keys, various config — structured, strongly consistent, small in volume. It's the whole system's "source of truth," kept last in deletion: serving as the anchor for "this deletion isn't done yet," ensuring a safe retry after a crash.
eventsClickHouse (deleted first)
trace/observation/score — massive analyzable detail (Parts 3-4 explained why it uses columnar storage). The delete volume can be huge, using batched deletion by projectId / by time, cleaned in parallel with S3.
payloadobject storage S3 (deleted first)
Full inputs/outputs, uploaded media, and other bulky binaries. Deleting CH rows must also clear their S3 objects and references — otherwise it's a "pointer gone, data still taking space" leak.

Three deletions: clean by period, delete on demand, cascade-delete

Langfuse has three kinds of deletion, different scenarios but all cross-store: ① data retention — auto, periodic, age-based cleaning of old data. It's an EE feature (Lesson 50's entitlement); each project can set retentionDays; scheduling uses exactly the same two-level fan-out as Lesson 46: dataRetentionProcessor finds projects with a retention period set and dispatches a processing job per project, dataRetentionProcessingProcessor computes cutoffDate = today − retentionDays and deletes CH data and S3 media older than it. ② trace deletion — on-demand deletion of specific traces (Lesson 47's batch-action trace-delete uses it, traceDeleteProcessor). ③ project deletion — cascading deletion of all of a project's data (the previous section's projectDeleteProcessor).

three kinds of deletion, different scenarios, all cross-store 1 data retention (EE) auto · periodic age: cutoff = now − retentionDays two-level fan-out (like L46) clean old CH + S3 2 trace deletion on-demand (L47 batch) specific traceIds traceDeleteProcessor clean CH + S3 3 project deletion cascade all of a project's data projectDeleteProcessor clean CH+S3 → PG last in common: all cross ClickHouse + S3 (project deletion also Postgres, kept last)
1

Data retention: scheduling fan-out (EE)

dataRetentionProcessor fires periodically, finds projects with retentionDays set, dispatches a processing job per project — the same two-level fan-out skeleton as Lesson 46's analytics integrations.

2

Data retention: per-project age cleaning

Each job computes cutoffDate = now − retentionDays, deleteTracesOlderThanDays etc. clean CH's traces/observations/scores/events in parallel and clean S3 media — deleting only the expired, untouching new data.

3

Trace deletion: on-demand

A user or Lesson 47's batch action wants to delete specific traces; traceDeleteProcessor deletes by traceIds across stores (likewise cleaning CH + S3 together).

4

Project deletion: cascade

To delete a whole project, projectDeleteProcessor first cleans CH + S3 in parallel, then dataset run items, finally deletes PG (the previous section's "PG kept last").

worker/src/ee/dataRetention/handleDataRetentionProcessingJob.tsage-based cross-store cleaning
// data retention is an EE feature: each project can set retentionDays (Lesson 50 entitlement)
const { retentionDays } = await prisma.project.findUnique({ where: { id: projectId },
                                                       select: { retentionDays: true } });
if (!retentionDays) return;                          // no retention period → don't clean

const cutoffDate = new Date(now − retentionDays * 86400_000);   // roll back N days from today

// clean data "older than cutoff" across stores in parallel: S3 media + ClickHouse first
await Promise.all([
  removeIngestionEventsFromS3AndDeleteClickhouseRefsForProject(projectId, cutoffDate),
  deleteTracesOlderThanDays(projectId, cutoffDate),
  deleteObservationsOlderThanDays(projectId, cutoffDate),
  deleteScoresOlderThanDays(projectId, cutoffDate),
  deleteEventsOlderThanDays(projectId, cutoffDate),
]);   // delete only the expired, new data untouched

Background migrations: "operating online" on billion-row tables

The data lifecycle has another facet: schema evolution. An ordinary DB migration (add a column, change a constraint) runs in seconds at deploy time. But Langfuse's ClickHouse often holds billions of rows, and to backfill a new field or reorganize data layout across them may run for hours — you can't take hours of downtime for it. BackgroundMigration is designed for such "big surgery": making the migration a long task that runs slowly in the background, resumes from a checkpoint, and never takes downtime.

Its model (schema.prisma:284) hides three key fields: state (Json) — stores "how far it got" as progress, resuming from here after a crash-restart (the same "resumable" wisdom as Lesson 46's lastSyncAt watermark); lockedAt/workerId — a distributed lock ensuring only one worker runs a given migration (no duplication or conflict under multi-worker deployments); finishedAt/failedReason — records the result. BackgroundMigrationManager drives it with a heartbeat loop: hold the lock, run a batch, update state, renew the heartbeat, until done. Throughout, the app keeps serving and users are none the wiser.

Operate on billion-row tables without downtime: slow run + resume + single-run BackgroundMigration model state (Json)progress → resume on crash lockedAt / workerIddistributed lock → one worker runs finishedAt / failedReasonresult kept Manager heartbeat loop① grab lock (lockedAt)② run a batch③ update state④ renew heartBeatloop until finishedAt no downtimeapp keeps servingusers none the wiserhours-long big surgerycompletes slowly in the background write state back each batch, interruptible & resumable
Online schema evolution: schema.prisma:284 BackgroundMigration (name unique, script, args, state Json resumable, finishedAt/failedAt/failedReason, workerId/lockedAt lock); backgroundMigrationManager.ts:6 BackgroundMigrationManager.run (:17 heartBeat, :42 while loop, hold-lock run-batch, :127 migration.run(args)), wrapped in instrumentAsync. Interface IBackgroundMigration (validate/run/abort).
ChallengeOrdinary migrationBackgroundMigration
Backfill a field across billions of rowsrun at deploy, may need hours of downtimerun slowly in background, no downtime, app keeps serving
Crashed mid-wayeither roll back or start overstate records progress, resume from checkpoint (like Lesson 46's watermark)
Multi-worker deployment—lockedAt/workerId lock, only one runs a given migration
How far / failure reasoncheck deploy logsfinishedAt/failedReason persisted and queryable
🎯 Design trade-off
Why does deletion insist on the order "delete ClickHouse/S3 first, Postgres last"? Won't the reverse do? This is a classic "keep the retry anchor" consideration, the same clarity as Lesson 47's "idempotency" and "snapshot boundary." A deletion across three stores isn't atomic — it's multiple steps, any of which may be interrupted by a crash, timeout, or network blip. The key question: after an interruption, does the system still know "this isn't done, come back and finish"? The project record in Postgres is exactly that "to-do proof": as long as it exists, a deletion retry knows "this project isn't fully cleaned" and continues sweeping the CH/S3 residue. But delete PG first and that proof is gone — if CH/S3 still have residue, the system can no longer know they exist, and they become orphan data permanently taking space with no claimant. So "delete the source-of-truth / anchor last" is the universal discipline of distributed deletion: clean the periphery first, the anchor last, so "unfinished" remains discoverable and retryable.

You could write a one-off script to run a migration — why build a whole BackgroundMigration framework with state, locks, and heartbeats? Because "changing a billion-row production table" simultaneously hits three hard constraints: ① no downtime — the business must keep serving, you can't lock the table for hours; ② will be interrupted — a multi-hour task will surely meet a deploy, restart, or crash, so it must resume from a checkpoint rather than start over (which needs persisting progress into state, the same trick as Lesson 46's incremental lastSyncAt); ③ multi-instance — production has many workers, so a given migration must run on only one worker, else duplicate runs corrupt data (which needs the lockedAt distributed lock + heartbeat renewal). A one-off script meets none of the three. Distilling these commonalities into a framework lets each concrete migration just implement validate/run/abort, while the complex "resume, lock, audit" is uniformly guaranteed by the framework — turning the hard thing of "safely evolving data online" into a reusable, trustworthy foundation.
🎯 Key points
  • One trace scattered across three: metadata in Postgres, massive events in ClickHouse, bulky payloads in object storage S3. So deletion must clean all three, missing none; a miss is data residue (compliance/privacy hazard).
  • Deletion has an order: projectDelete cleans CH + S3 in parallel first, deletes Postgres last. The PG record is the retry anchor for "this deletion isn't done," kept last to ensure crash-retryability; deleting PG first orphans CH/S3 data.
  • Three deletions: ① data retention (EE, auto-cleans old data by retentionDays, same two-level fan-out as Lesson 46, cutoffDate deletes only the expired); ② trace deletion (on-demand, Lesson 47's batch action uses it); ③ project deletion (cascade-cleans a whole project).
  • Background migrations evolve schema online: backfill/reorganize billion-row tables, hours of big surgery without downtime. BackgroundMigration: state records progress for checkpoint resume (like Lesson 46's watermark), lockedAt/workerId lock ensures single-worker execution, heartbeat renewal, result persisted.
  • One lineage: deletion order = keep the retry anchor (like Lesson 47's idempotency), background-migration state = resumable (like Lesson 46's incremental), single-worker lock (like Lesson 43's metering mutex). "Safely doing big side-effectful things" in a distributed system reuses the same wisdom over and over.

🧪 Self-test - think about the design

1. When Langfuse deletes a project, projectDelete's order is 'clean ClickHouse and S3 in parallel first, delete Postgres last'. Why is Postgres kept for last?
  1. ClickHouse must be initialized before Postgres
  2. the project record in PG is the retry anchor for 'this deletion isn't done': cross-store deletion isn't atomic and may crash mid-way; as long as the PG record exists, a retry knows 'keep cleaning the CH/S3 residue'; delete PG first and the remaining CH/S3 data becomes unclaimed, unmatchable orphans permanently taking space
  3. it's just a coding habit with no real meaning
  4. Postgres deletion is slowest, last saves time
Show answer & explanation click to expand
Answer: B. A deletion spans three stores—Postgres + ClickHouse + S3—and this isn't an atomic transaction; it's multi-step, any step interruptible by crash/timeout/network blip. The key question: after interruption, does the system still know 'this isn't done, come back and finish'? The PG project record is exactly that 'to-do proof': keep it and a retry knows the project isn't fully cleaned, continuing to sweep CH/S3 residue; delete PG first and that proof is gone—if CH/S3 still have residue, the system can no longer know it exists, and it becomes a permanent orphan. So 'delete the anchor/source-of-truth last' is the universal discipline of distributed deletion—clean the periphery first, the anchor last, so 'unfinished' stays discoverable and retryable. Of one lineage with Lesson 47's 'snapshot boundary/idempotency'.
2. In Langfuse, deleting a trace isn't as simple as 'delete one database row'. The most fundamental reason is?
  1. because trace data is encrypted and must be decrypted before deletion
  2. because a trace is too big to delete at once
  3. a trace's data is scattered across three stores: metadata in Postgres, massive event detail in ClickHouse, bulky inputs/outputs/media in object storage S3. To truly delete it you must delete consistently across all three; missing any one is data residue—wasting space and, worse, a compliance and privacy hazard (e.g. GDPR's right to be forgotten)
  4. because deletion needs admin approval
Show answer & explanation click to expand
Answer: C. This is a direct consequence of Langfuse's storage architecture. To serve reads and writes well, a trace is split into three: Postgres for structured metadata, ClickHouse for massive analyzable event detail (columnar, Parts 3-4), S3 for oversized raw payloads (full I/O, media). This division is great for performance but demands of deletion: clean all three at once. Delete only the PG 'index' row and the CH detail and S3 attachments remain—data residue—not just wasted storage but, worse, a privacy-compliance problem: when a user exercises 'delete my data', residue means it wasn't truly deleted. So each Langfuse deletion (retention cleaning, trace delete, project delete) is a cross-store operation.
3. Langfuse's BackgroundMigration model has state, lockedAt/workerId, heartbeat mechanisms. What core problem does it solve?
  1. compresses data after migration
  2. auto-generates migration scripts
  3. changing a billion-row production table (backfill a field/reorganize layout) hits three constraints at once—no downtime, interruptible by deploy/crash, multiple workers: state records progress for checkpoint resume (like Lesson 46's watermark), lockedAt/workerId lock ensures only one worker runs a given migration, heartbeat renewal—letting hours-long big surgery complete in the background without downtime
  4. makes database migrations run faster
Show answer & explanation click to expand
Answer: C. An ordinary migration (add column, change constraint) runs in seconds at deploy and needs none of this heavy machinery. But backfilling a new field across billions of ClickHouse rows or reorganizing layout may run for hours, hitting three hard constraints at once: ① no downtime—the business must keep serving, can't lock the table for hours; ② will be interrupted—over hours you'll surely meet a deploy, restart, crash, so it must resume from a checkpoint not start over (by persisting progress into state, the same trick as Lesson 46's incremental lastSyncAt); ③ multi-worker—production has many instances, so a given migration must run on only one worker, else duplicate runs corrupt data (via the lockedAt distributed lock + heartbeat renewal). A one-off script meets none of the three. BackgroundMigration distills these commonalities into a framework; each migration just implements validate/run/abort, while resume/lock/audit is guaranteed by the framework.
💭 Open questions (no single right answer - just think or try)
  • This lesson recurs on a theme: in distributed systems, 'safely doing big side-effectful things' relies on a few moves — deletion keeps a retry anchor, migrations use state for resumability, locks ensure single-worker execution. Drawing on your experience: what pitfalls have you hit in 'bulk deletion or data migration across multiple systems/stores' (mid-way failure, orphan data, duplicate execution)? How did you ensure 'no-dup-no-miss, resumable'? And if the data also involves compliant deletion (e.g. GDPR right to be forgotten), what additional requirements arise?