这一课反复出现一个主题:在分布式系统里「安全地做有副作用的大事」要靠几招——删除留重试锚点、迁移用 state 可恢复、用锁保证单 worker 执行。请结合你的经验谈谈:你做过的「跨多个系统/存储的批量删除或数据迁移」遇到过哪些坑(中途失败、孤儿数据、重复执行)?你是怎么保证「不重不漏、可恢复」的?如果数据还涉及合规删除(如 GDPR 被遗忘权),又会带来哪些额外要求?
Data has a lifetime: ingested and written (born), queried and analyzed (alive), finally deleted (curtain call). Earlier lessons covered "write" and "read" thoroughly; this one covers "delete" — and an easily-overlooked but crucial fact: in Langfuse, one trace isn't one row but three copies scattered across three stores: metadata in Postgres, massive events in ClickHouse, bulky inputs/outputs/media in object storage (S3). So "delete a trace / delete a project / clean old data by retention period" is never as simple as deleting one row — it requires coordinated, consistent deletion across all three stores; missing any one leaves data residue (a compliance and privacy hazard).
This lesson covers three deletions (auto-clean by retention period, on-demand trace delete, cascading project delete), how they delete across stores, why deletion has a deliberate order; and finally background migrations — how to "operate online" on a table with billions of rows without downtime.
📋 Analogy
Think of a trace as a dossier stored across three archive rooms: the registry (Postgres) has its index card (who, when, which project), the main vault (ClickHouse) has its massive detailed records, the oversized-item warehouse (S3) holds its bulky attachments (full inputs/outputs, uploaded images).
Now to "destroy" this dossier — just tearing up the registry's index card is far from enough: the vault's details and the warehouse's attachments are still there, which is data residue. True destruction must clean all three archive rooms, missing none. And destruction has a deliberate order: clean the main vault and warehouse first, tear up the registry card last — because that index card is the only proof that "this dossier isn't fully destroyed yet"; if the power cuts mid-way, it's how you know to "come back and finish destroying." Tear up the card first and the remaining details and attachments become unclaimed orphans, no longer matchable to anyone.
One trace scattered across three: deletion spans stores
Recall the book-wide storage division: Postgres stores config and metadata (projects, users, API keys, various settings); ClickHouse stores massive analyzable events (trace/observation/score, the stars of Parts 3-4); object storage S3 stores bulky raw payloads (full inputs/outputs, uploaded media). This division serves reads and writes well, but also means: deletion must clean all three at once. Take "delete a whole project": projectDeleteProcessor's order is deliberate — first clean ClickHouse and S3 in parallel (traces/observations/scores/events + blob refs), only then delete Postgres. This "delete PG last" isn't arbitrary: the project record in Postgres is the anchor for "this project isn't fully deleted yet"; kept to the end, it allows a retry if a crash happens mid-way; delete PG first and the remaining CH/S3 data becomes ownerless orphans, hard to clean up.
Cross-store deletion: projectDelete.ts:22 projectDeleteProcessor — Promise.all first cleans ClickHouse (deleteTracesByProjectId/deleteObservationsByProjectId/deleteScoresByProjectId/v4's deleteEventsByProjectId) and S3 (removeIngestionEventsFromS3AndDeleteClickhouseRefsForProject), then deletes dataset run items, finally deletes Postgres. The comment is explicit: "Delete project data from ClickHouse first."
Deletion isn't "DROP a row" but an ordered flow across three stores (from traceDeletionProcessor and processClickhouseTraceDelete; values illustrative): data is born and retained for a window (data-retention / data-access-days). On expiry or an explicit delete, it's first soft-marked in Postgres' pending_deletions and put on a delayed queue (respond fast), then a worker deletes Postgres and ClickHouse traces/observations/scoresin parallel via Promise.all; the ClickHouse step also removes orphaned media from S3 — guarded by NOT EXISTS so shared media referenced by other traces is never wrongly deleted. Only after all succeed is isDeleted set true; the whole process is batched and retryable.
metadataPostgres (deleted last)
Projects, users, API keys, various config — structured, strongly consistent, small in volume. It's the whole system's "source of truth," kept last in deletion: serving as the anchor for "this deletion isn't done yet," ensuring a safe retry after a crash.
eventsClickHouse (deleted first)
trace/observation/score — massive analyzable detail (Parts 3-4 explained why it uses columnar storage). The delete volume can be huge, using batched deletion by projectId / by time, cleaned in parallel with S3.
payloadobject storage S3 (deleted first)
Full inputs/outputs, uploaded media, and other bulky binaries. Deleting CH rows must also clear their S3 objects and references — otherwise it's a "pointer gone, data still taking space" leak.
Three deletions: clean by period, delete on demand, cascade-delete
Langfuse has three kinds of deletion, different scenarios but all cross-store: ① data retention — auto, periodic, age-based cleaning of old data. It's an EE feature (Lesson 50's entitlement); each project can set retentionDays; scheduling uses exactly the same two-level fan-out as Lesson 46: dataRetentionProcessor finds projects with a retention period set and dispatches a processing job per project, dataRetentionProcessingProcessor computes cutoffDate = today − retentionDays and deletes CH data and S3 media older than it. ② trace deletion — on-demand deletion of specific traces (Lesson 47's batch-action trace-delete uses it, traceDeleteProcessor). ③ project deletion — cascading deletion of all of a project's data (the previous section's projectDeleteProcessor).
1
Data retention: scheduling fan-out (EE)
dataRetentionProcessor fires periodically, finds projects with retentionDays set, dispatches a processing job per project — the same two-level fan-out skeleton as Lesson 46's analytics integrations.
2
Data retention: per-project age cleaning
Each job computes cutoffDate = now − retentionDays, deleteTracesOlderThanDays etc. clean CH's traces/observations/scores/events in parallel and clean S3 media — deleting only the expired, untouching new data.
3
Trace deletion: on-demand
A user or Lesson 47's batch action wants to delete specific traces; traceDeleteProcessor deletes by traceIds across stores (likewise cleaning CH + S3 together).
4
Project deletion: cascade
To delete a whole project, projectDeleteProcessor first cleans CH + S3 in parallel, then dataset run items, finally deletes PG (the previous section's "PG kept last").
// data retention is an EE feature: each project can set retentionDays (Lesson 50 entitlement)const { retentionDays } = await prisma.project.findUnique({ where: { id: projectId },
select: { retentionDays: true } });
if (!retentionDays) return; // no retention period → don't cleanconst cutoffDate = new Date(now − retentionDays * 86400_000); // roll back N days from today// clean data "older than cutoff" across stores in parallel: S3 media + ClickHouse firstawait Promise.all([
removeIngestionEventsFromS3AndDeleteClickhouseRefsForProject(projectId, cutoffDate),
deleteTracesOlderThanDays(projectId, cutoffDate),
deleteObservationsOlderThanDays(projectId, cutoffDate),
deleteScoresOlderThanDays(projectId, cutoffDate),
deleteEventsOlderThanDays(projectId, cutoffDate),
]); // delete only the expired, new data untouched
Background migrations: "operating online" on billion-row tables
The data lifecycle has another facet: schema evolution. An ordinary DB migration (add a column, change a constraint) runs in seconds at deploy time. But Langfuse's ClickHouse often holds billions of rows, and to backfill a new field or reorganize data layout across them may run for hours — you can't take hours of downtime for it. BackgroundMigration is designed for such "big surgery": making the migration a long task that runs slowly in the background, resumes from a checkpoint, and never takes downtime.
Its model (schema.prisma:284) hides three key fields: state (Json) — stores "how far it got" as progress, resuming from here after a crash-restart (the same "resumable" wisdom as Lesson 46's lastSyncAt watermark); lockedAt/workerId — a distributed lock ensuring only one worker runs a given migration (no duplication or conflict under multi-worker deployments); finishedAt/failedReason — records the result. BackgroundMigrationManager drives it with a heartbeat loop: hold the lock, run a batch, update state, renew the heartbeat, until done. Throughout, the app keeps serving and users are none the wiser.
run slowly in background, no downtime, app keeps serving
Crashed mid-way
either roll back or start over
state records progress, resume from checkpoint (like Lesson 46's watermark)
Multi-worker deployment
—
lockedAt/workerId lock, only one runs a given migration
How far / failure reason
check deploy logs
finishedAt/failedReasonpersisted and queryable
🎯 Design trade-off
Why does deletion insist on the order "delete ClickHouse/S3 first, Postgres last"? Won't the reverse do? This is a classic "keep the retry anchor" consideration, the same clarity as Lesson 47's "idempotency" and "snapshot boundary." A deletion across three stores isn't atomic — it's multiple steps, any of which may be interrupted by a crash, timeout, or network blip. The key question: after an interruption, does the system still know "this isn't done, come back and finish"? The project record in Postgres is exactly that "to-do proof": as long as it exists, a deletion retry knows "this project isn't fully cleaned" and continues sweeping the CH/S3 residue. But delete PG first and that proof is gone — if CH/S3 still have residue, the system can no longer know they exist, and they become orphan data permanently taking space with no claimant. So "delete the source-of-truth / anchor last" is the universal discipline of distributed deletion: clean the periphery first, the anchor last, so "unfinished" remains discoverable and retryable.
You could write a one-off script to run a migration — why build a whole BackgroundMigration framework with state, locks, and heartbeats? Because "changing a billion-row production table" simultaneously hits three hard constraints: ① no downtime — the business must keep serving, you can't lock the table for hours; ② will be interrupted — a multi-hour task will surely meet a deploy, restart, or crash, so it must resume from a checkpoint rather than start over (which needs persisting progress into state, the same trick as Lesson 46's incremental lastSyncAt); ③ multi-instance — production has many workers, so a given migration must run on only one worker, else duplicate runs corrupt data (which needs the lockedAt distributed lock + heartbeat renewal). A one-off script meets none of the three. Distilling these commonalities into a framework lets each concrete migration just implement validate/run/abort, while the complex "resume, lock, audit" is uniformly guaranteed by the framework — turning the hard thing of "safely evolving data online" into a reusable, trustworthy foundation.
🎯 Key points
One trace scattered across three: metadata in Postgres, massive events in ClickHouse, bulky payloads in object storage S3. So deletion must clean all three, missing none; a miss is data residue (compliance/privacy hazard).
Deletion has an order: projectDelete cleans CH + S3 in parallel first, deletes Postgres last. The PG record is the retry anchor for "this deletion isn't done," kept last to ensure crash-retryability; deleting PG first orphans CH/S3 data.
Three deletions: ① data retention (EE, auto-cleans old data by retentionDays, same two-level fan-out as Lesson 46, cutoffDate deletes only the expired); ② trace deletion (on-demand, Lesson 47's batch action uses it); ③ project deletion (cascade-cleans a whole project).
Background migrations evolve schema online: backfill/reorganize billion-row tables, hours of big surgery without downtime. BackgroundMigration: state records progress for checkpoint resume (like Lesson 46's watermark), lockedAt/workerId lock ensures single-worker execution, heartbeat renewal, result persisted.
One lineage: deletion order = keep the retry anchor (like Lesson 47's idempotency), background-migration state = resumable (like Lesson 46's incremental), single-worker lock (like Lesson 43's metering mutex). "Safely doing big side-effectful things" in a distributed system reuses the same wisdom over and over.
🧪 Self-test - think about the design
1. When Langfuse deletes a project, projectDelete's order is 'clean ClickHouse and S3 in parallel first, delete Postgres last'. Why is Postgres kept for last?
ClickHouse must be initialized before Postgres
the project record in PG is the retry anchor for 'this deletion isn't done': cross-store deletion isn't atomic and may crash mid-way; as long as the PG record exists, a retry knows 'keep cleaning the CH/S3 residue'; delete PG first and the remaining CH/S3 data becomes unclaimed, unmatchable orphans permanently taking space
it's just a coding habit with no real meaning
Postgres deletion is slowest, last saves time
Show answer & explanation click to expand
Answer: B. A deletion spans three stores—Postgres + ClickHouse + S3—and this isn't an atomic transaction; it's multi-step, any step interruptible by crash/timeout/network blip. The key question: after interruption, does the system still know 'this isn't done, come back and finish'? The PG project record is exactly that 'to-do proof': keep it and a retry knows the project isn't fully cleaned, continuing to sweep CH/S3 residue; delete PG first and that proof is gone—if CH/S3 still have residue, the system can no longer know it exists, and it becomes a permanent orphan. So 'delete the anchor/source-of-truth last' is the universal discipline of distributed deletion—clean the periphery first, the anchor last, so 'unfinished' stays discoverable and retryable. Of one lineage with Lesson 47's 'snapshot boundary/idempotency'.
2. In Langfuse, deleting a trace isn't as simple as 'delete one database row'. The most fundamental reason is?
because trace data is encrypted and must be decrypted before deletion
because a trace is too big to delete at once
a trace's data is scattered across three stores: metadata in Postgres, massive event detail in ClickHouse, bulky inputs/outputs/media in object storage S3. To truly delete it you must delete consistently across all three; missing any one is data residue—wasting space and, worse, a compliance and privacy hazard (e.g. GDPR's right to be forgotten)
because deletion needs admin approval
Show answer & explanation click to expand
Answer: C. This is a direct consequence of Langfuse's storage architecture. To serve reads and writes well, a trace is split into three: Postgres for structured metadata, ClickHouse for massive analyzable event detail (columnar, Parts 3-4), S3 for oversized raw payloads (full I/O, media). This division is great for performance but demands of deletion: clean all three at once. Delete only the PG 'index' row and the CH detail and S3 attachments remain—data residue—not just wasted storage but, worse, a privacy-compliance problem: when a user exercises 'delete my data', residue means it wasn't truly deleted. So each Langfuse deletion (retention cleaning, trace delete, project delete) is a cross-store operation.
3. Langfuse's BackgroundMigration model has state, lockedAt/workerId, heartbeat mechanisms. What core problem does it solve?
compresses data after migration
auto-generates migration scripts
changing a billion-row production table (backfill a field/reorganize layout) hits three constraints at once—no downtime, interruptible by deploy/crash, multiple workers: state records progress for checkpoint resume (like Lesson 46's watermark), lockedAt/workerId lock ensures only one worker runs a given migration, heartbeat renewal—letting hours-long big surgery complete in the background without downtime
makes database migrations run faster
Show answer & explanation click to expand
Answer: C. An ordinary migration (add column, change constraint) runs in seconds at deploy and needs none of this heavy machinery. But backfilling a new field across billions of ClickHouse rows or reorganizing layout may run for hours, hitting three hard constraints at once: ① no downtime—the business must keep serving, can't lock the table for hours; ② will be interrupted—over hours you'll surely meet a deploy, restart, crash, so it must resume from a checkpoint not start over (by persisting progress into state, the same trick as Lesson 46's incremental lastSyncAt); ③ multi-worker—production has many instances, so a given migration must run on only one worker, else duplicate runs corrupt data (via the lockedAt distributed lock + heartbeat renewal). A one-off script meets none of the three. BackgroundMigration distills these commonalities into a framework; each migration just implements validate/run/abort, while resume/lock/audit is guaranteed by the framework.
💭 Open questions (no single right answer - just think or try)
This lesson recurs on a theme: in distributed systems, 'safely doing big side-effectful things' relies on a few moves — deletion keeps a retry anchor, migrations use state for resumability, locks ensure single-worker execution. Drawing on your experience: what pitfalls have you hit in 'bulk deletion or data migration across multiple systems/stores' (mid-way failure, orphan data, duplicate execution)? How did you ensure 'no-dup-no-miss, resumable'? And if the data also involves compliant deletion (e.g. GDPR right to be forgotten), what additional requirements arise?