🪢 Langfuse 图解教程Langfuse Visual Guide 第九部分 · 自动化与集成Part 9 · Automation & Integrations 47 / 55
第九部分 · 自动化与集成Part 9 · Automation & Integrations

批量导出与批量操作Batch exports & actions

🧭 读前Before you start
先读 第46课 的集成与 第30课 的状态机,本课讲批量导出与批量操作(幂等)。
Read Lesson 46 (integrations) and Lesson 30 (the state machine); this lesson covers batch exports and actions (idempotency).

你在 trace 列表里筛出「过去 7 天、出错的、来自某用户的」5 万条 trace,想一次性导成 CSV,或者一键全删。这种「对一票筛出来的数据,整体干一件事」的需求,就是这一课的主角:批量导出(Batch Export)和批量操作(Batch Action)。它俩看着不同——一个是只读地产出文件、一个是危险地批量改数据——骨子里却共享同一套工程姿态:都基于一个可能命中海量行的查询/过滤条件,都因为耗时太长不能卡在请求里而扔进队列异步跑,都用流式 + 分块避免一次性把几万行塞进内存。 作为「自动化与集成」这一部分的收官课,它把前面攒下的流式导出(第46课)、任务状态机(第30/35课)、签名 URL 与对象存储(第46课)又一次组装起来,还会讲清楚一个批量改数据时绕不开的硬约束:幂等。

📋 生活类比
想象一个巨型图书馆,你下了一道指令:「把所有 2020 年前、关于天文的、已破损的书找出来」。接下来要么导出(把这些书的清单誊抄成一份目录交给你——只读,产出一份文件),要么操作(把这些书全部销毁或搬到别的馆——动真格地改变馆藏)。 这么大的活,图书管理员不会让你在前台干等着(耗时太长 → 交给后台慢慢办,办完通知你 = 队列异步);也不会把几万本书一次搬到桌上(桌子塌 = 内存爆 → 一推车一推车地搬 = 流式分块)。而「销毁/搬运」这种改变馆藏的操作还有个讲究:万一中途断电了要重来,已经销毁的书可不能被「再销毁一次」搞出乱子——所以每一步都得设计成「重复做也不出错」(幂等)。导出那份目录则没这顾虑(誊抄两遍只是浪费纸)。这一课讲的,就是图书馆怎么把这两类「批量大活」办得又稳又安全。

一个查询,两条路:只读导出 vs 改数据操作

两者都从同一个起点出发:一个过滤/查询条件(和第 23 课列表筛选同源),它可能命中成千上万行。然后分叉:批量导出是只读的——把命中的行流式查出、转成 CSV/JSON、写进对象存储、给你一个限时下载链接。批量操作是会改数据的——把命中的行的 id 流式取出、每 1000 个一块地执行删除 / 加进标注队列 / 加进数据集 / 重新评估。共同点是:都耗时(所以走队列异步、有状态机可追踪)、都数据量大(所以流式 + 分块、绝不全装内存)。

同一个过滤条件,命中海量行 → 分两条路处理 过滤/查询条件第23课同款筛选可能命中数万行 批量导出(只读)流式查 → 转 CSV/JSON→ 写对象存储→ 限时签名URL → 邮件 批量操作(改数据)流式取 id → 每 1000 一块删除 / 加标注队列 / 加数据集/ 重新评估(eval-create)⚠ 必须幂等(重试会重跑已处理块) 共同骨架耗时 → 队列异步有状态机可追踪量大 → 流式 + 分块绝不全装内存
同源分叉:批量导出 worker/src/features/batchExport/handleBatchExportJob.ts(只读产文件);批量操作 worker/src/features/batchAction/handleBatchActionJob.ts(CHUNK_SIZE=1000,switch 分发到 trace-delete / *-add-to-annotation-queue / dataset-delete / eval-create 等处理器)。两者都基于第 23 课的过滤条件、都走 BullMQ 队列。
示例:一次批量导出(CSV)+ 状态机 QUEUED PROCESSING COMPLETED FAILED 状态存 batch_exports 表 📄 export_traces_2026-06-27.csv流式分块写出 id,timestamp,name,userId,totalCost,latency t_9f3a,2026-06-27T10:00Z,chat,u_8f3a,0.0123,1.84 t_2b71,2026-06-27T10:01Z,chat,u_2b71,0.0098,1.32 t_55c9,2026-06-27T10:02Z,summarize,u_55c9,0.0211,2.07 ⟲ 幂等:同一导出请求重跑得同一文件,不会重复扣费或写两份
大导出是「有状态、可续跑」的后台作业,不是一次性内存操作(状态枚举取自 features/batchExport/types.ts;值为示例):一次导出在库里走 QUEUED → PROCESSING → COMPLETED 状态机(出错转 FAILED),产物是一份流式分块写出的 CSV(图中表头 + 数据行)。为什么要状态机而不是「点一下立刻下载」?因为导出可能是百万行——必须能后台跑、能断点续、能重试,还要幂等(同一请求重跑得同一结果,不重复计费)。这正是第 30 课「创建与执行分离」在导出场景的复用。
维度批量导出 Batch Export批量操作 Batch Action
本质只读:产出一份文件改数据:删除/修改海量行
产物对象存储里的 CSV/JSON + 限时签名 URL(邮件发你)无文件;副作用落在数据上
大数据策略流式 pipeline(逐行转格式写出)流式取 id + 每 1000 一块处理
关键约束签名 URL 限时过期(不长期公开)必须幂等(重试会重跑已处理的块)
状态机QUEUED→PROCESSING→COMPLETED/FAILED/CANCELLED同样有 BatchActionStatus 追踪

批量导出:流式 → 文件 → 限时链接 → 邮件

导出任务是一台状态机(BatchExportStatus:QUEUED→PROCESSING→COMPLETED/FAILED/CANCELLED),和第 30 课 eval 的 JobExecution、第 35 课数据集运行同根同源。一进来先校验状态(已取消就跳过、非 QUEUED 也跳过——幂等防重复执行),置为 PROCESSING。核心是流式管道:getDatabaseReadStreamPaginated 分页读出命中的行,pipeline 一行行经 streamTransformations[format] 转成 CSV/JSON,uploadFileBuffered 分片传进对象存储(和第 46 课同一个 StorageService)。传完生成一个限时签名 URL(getSignedUrl(expiresInSeconds)),把状态置 COMPLETED、记下 url 和 expiresAt,最后发邮件把下载链接送到用户邮箱。

导出状态机:QUEUED → PROCESSING → COMPLETED(幂等防重复执行) ① 流式查 分页读命中行 getDatabaseRead… ② 转格式 逐行 CSV/JSON 内存恒定 ③ 传存储 uploadFileBuffered → S3 / GCS ④ 限时链接 getSignedUrl 记 expiresAt ⑤ 发邮件 下载链接 → 用户邮箱 流式管道全程内存恒定:几万行也不撑爆;限时签名 URL 不长期公开数据
1

校验状态 + 置 PROCESSING

已 CANCELLED → 跳过;非 QUEUED → 跳过(防同一任务被重复执行)。否则 update({status: PROCESSING}) 占住它。

2

流式查 + 转格式

getDatabaseReadStreamPaginated 分页读命中行;pipeline 逐行经 streamTransformations[format] 转 CSV/JSON——内存恒定,几万行也不撑爆。

3

传对象存储 + 签名 URL

uploadFileBuffered 分片传进 S3/GCS;getSignedUrl(expiresInSeconds) 生成限时下载链接——不长期公开数据。

4

置 COMPLETED + 发邮件

update({status: COMPLETED, url, finishedAt, expiresAt});sendBatchExportSuccessEmail 把 downloadLink 发到用户邮箱。

worker/src/features/batchExport/handleBatchExportJob.ts流式导出 + 限时链接
// 1) 状态机校验:幂等防重复执行
if (jobDetails.status === BatchExportStatus.CANCELLED) return;       // 已取消,跳过
if (jobDetails.status !== BatchExportStatus.QUEUED)   return;       // 已被处理,跳过
await prisma.batchExport.update({ ..., data: { status: BatchExportStatus.PROCESSING } });

// 2) 流式管道:分页读 → 逐行转 CSV/JSON(内存恒定)
const fileStream = pipeline(
  await getDatabaseReadStreamPaginated({ ...streamParams }),
  streamTransformations[jobDetails.format](),                  // CSV / JSON / JSONL
);

// 3) 传对象存储 → 生成限时签名 URL
await storageService.uploadFileBuffered({ fileName, data: fileStream, ... });
const signedUrl = await storageService.getSignedUrl(fileName, expiresInSeconds);

// 4) 置完成 + 记过期时间 + 发邮件
await prisma.batchExport.update({ ..., data: {
  status: BatchExportStatus.COMPLETED, url: signedUrl,
  finishedAt: new Date(), expiresAt: new Date(Date.now() + expiresInSeconds*1000) } });
await sendBatchExportSuccessEmail({ receiverEmail: user.email, downloadLink: signedUrl, ... });

批量操作:分块 1000,且必须幂等

批量操作要改数据,分块流程是:用过滤条件 + 一个 cutoffCreatedAt 截止时间(快照边界——任务启动后才新建的行不算进来,目标集合稳定)流式取出命中行的 id,攒够 1000 个一块,交给 processActionChunk 按 actionId 分发:trace-delete 删 trace、*-add-to-annotation-queue 加进第 32 课的标注队列、dataset-delete 删数据集、eval-create 给历史数据补跑第 29 课的评估……源码在这里留了一句醒目的红线注释:「所有操作必须幂等。任务失败会重试,重试时已处理过的块可能被再处理一遍。」这不是建议,是硬约束。

流式取 id → 每 1000 一块 → 分发处理(且必须幂等) 流式取命中 id过滤 + cutoffCreatedAt快照边界:固定目标集不卷入任务后新建的行 块 1·1000 块 2·1000 块 3 … processActionChunkswitch(actionId)逐块分发CHUNK_SIZE=1000:保护数据库不被一次性巨量冲击 trace-delete(删 trace) *-add-to-annotation-queue(第32课) dataset-delete / add-to-dataset eval-create(补跑第29课评估) ⚠ 红线:所有操作必须幂等任务失败会整体重试 → 已处理过的块可能被再跑一遍所以「删一个已删的」「加一个已加的」都不能报错或副作用翻倍
分块 + 幂等:handleBatchActionJob.ts:46 CHUNK_SIZE=1000;:61 processActionChunk switch 分发(trace-delete / *-add-to-annotation-queue / score-delete / dataset-delete / eval-create);:57-60 的注释明确「所有操作必须幂等……重试时已处理块可能被再处理」。cutoffCreatedAt 作快照边界固定目标集合。
取数流式 + 快照边界
用过滤条件 + cutoffCreatedAt 流式取命中行的 id。cutoffCreatedAt 是关键:把目标集合钉死在任务启动那一刻,任务跑的几分钟里新写入的行不会被卷进来,避免「边删边新增、永远删不完」。
分块CHUNK_SIZE = 1000
不一次性对几万行下手,而是攒 1000 个 id 一块、逐块处理。既保护数据库不被一条巨型语句冲击,也让失败时的重试粒度停在「块」而非「整个任务从零再来」。
铁律幂等(idempotent)
队列任务失败会整体重试,重试时前面已成功的块会被再跑一遍。所以每种操作都必须「重复执行不出错、副作用不翻倍」:删一个已删的 = 静默成功,加一个已在队列里的 = 不重复加。这是批量改数据绕不过的硬约束。
🎯 设计取舍
为什么批量操作「必须幂等」是一条铁律,而不是「尽量做到」? 因为它和「队列会重试」这条现实正面相撞。一个删 5 万行的任务,分成 50 块跑,假设跑到第 30 块时 worker 崩了——BullMQ 会把整个任务重新派发(它不知道你内部跑到了第几块),于是前 29 块会被再执行一遍。如果「删除」不是幂等的(比如删一个不存在的 id 就抛异常),重试会直接失败、任务永远卡死;如果「加入队列」不是幂等的,同一批 trace 会被加两次。幂等不是锦上添花,而是「在一个会重试的系统里,正确性的前提」。源码作者把这句话写成醒目注释顶在函数上,就是在反复提醒每一个未来改这里的人:你写的每个 action,都活在「可能被重复调用」的世界里。这与第 43 课计量的「恰好一次」、第 30 课 eval 的去重,是同一种对「分布式世界不可靠」的清醒。

为什么导出给的是「限时签名 URL」,而不是一个永久公开的下载地址? 因为导出的文件里往往是敏感数据(用户的 trace、输入输出、可能含 PII)。如果生成一个永久公开链接,它一旦被转发、被日志记录、被缓存,就等于把这批数据永久泄露到了一个谁拿到链接谁就能下载的地方。签名 URL 的设计是「临时授权」:链接里带着加密签名和过期时间,expiresAt 一到就自动失效。再配合 BatchExport 记录里的 expiresAt,平台对「这份导出还能不能下」始终有据可查。对敏感数据的访问,默认应该是有时限的、可收回的,而不是一次给出、永久敞开。
🎯 本课要点 · 第九部分收官
  • 一个查询,两条路:批量导出(只读,产文件)与批量操作(改数据,删/加队列/加数据集/重评)都基于第 23 课的过滤条件,都因耗时长而走队列异步、有状态机追踪,都用流式 + 分块避免内存爆。
  • 批量导出 = 流式 → 文件 → 限时链接 → 邮件:BatchExportStatus 状态机(QUEUED→PROCESSING→COMPLETED);getDatabaseReadStreamPaginated+pipeline 转 CSV/JSON;uploadFileBuffered 传对象存储;getSignedUrl 给限时下载链接;记 expiresAt 后发邮件。
  • 批量操作 = 分块 1000 + 幂等:cutoffCreatedAt 快照边界固定目标集;CHUNK_SIZE=1000 保护 DB、细化重试粒度;processActionChunk 按 actionId 分发(trace-delete / 加标注队列 / dataset-delete / eval-create)。
  • 幂等是铁律:队列失败会整体重试、已处理块会被重跑,所以每个 action 必须「重复执行不出错、副作用不翻倍」。这是分布式重试系统里正确性的前提,与第 43 课「恰好一次」、第 30 课去重同源。
  • 限时签名 URL:导出含敏感数据,给带过期时间的临时授权链接而非永久公开地址——对敏感数据的访问默认应有时限、可收回。
  • 第九部分收官:自动化(L44 webhook/SSRF)→ Slack(L45)→ 分析集成(L46 fan-out/增量/流式)→ 批量导出与操作(L47)。一条主线:把平台的数据安全、可靠、可控地与外部世界连接。

🧪 自测 · 想一想为什么这么设计

1. 批量导出和批量操作看着很不同(一个产文件、一个改数据),但这一课强调它们共享同一套工程骨架。这套共同骨架是什么?
  1. 都只能在自托管版使用
  2. 都需要用户付费解锁
  3. 都把数据加密后存储
  4. 都基于一个可能命中海量行的过滤条件、都因耗时长而走队列异步(有状态机可追踪)、都用流式+分块避免一次性把几万行装进内存——区别只在『只读产文件』vs『改数据』
看答案与解析 点击展开
答案:D。识别「不同功能的共同骨架」是这一课的核心。批量导出(只读)和批量操作(改数据)表面差异大,但都面对同样两个工程现实:① 数据量可能极大(命中几万行)→ 必须流式 + 分块,不能全装内存;② 处理耗时长 → 不能卡在 HTTP 请求里,必须扔进队列异步跑,并用状态机(BatchExportStatus / BatchActionStatus)让用户能追踪进度。看穿这层共性,你就知道:凡是『对一大票数据整体做一件事』的需求,都该往这个骨架上套。
2. 批量操作的源码在 processActionChunk 上方顶着一句醒目注释:「所有操作必须幂等。任务失败会重试,重试时已处理过的块可能被再处理一遍。」为什么幂等在这里是「硬约束」而非「最好做到」?
  1. 因为数据库要求所有写操作幂等
  2. 因为它和「队列会重试」正面相撞:删5万行分50块,跑到第30块崩了,BullMQ 会重发整个任务(它不知道内部跑到第几块),前29块会被再执行一遍——若删除/加队列不幂等,要么重试直接失败任务卡死,要么同批数据被加两次
  3. 因为幂等能让任务跑得更快
  4. 因为幂等能节省存储空间
看答案与解析 点击展开
答案:B。队列的「失败重试」是可靠性的来源,但它和「批量改数据」相遇时产生一个尖锐后果:重试是把整个任务重发,而任务内部已经跑完的块,队列并不知道、也会跟着重跑。于是每个 action 都活在「可能被重复调用」的世界里。如果不幂等:删一个已删的 id 抛异常 → 重试永远失败、任务卡死;把一批 trace 重复加进标注队列 → 数据被污染。所以幂等是「在会重试的系统里保证正确」的前提,不是可选项。这与第43课计量「恰好一次」、第30课 eval 去重,都是对『分布式不可靠』的同一种清醒。
3. 批量导出完成后,给用户的是一个带过期时间的「签名 URL」,而不是一个永久公开的下载地址。这个设计主要在防什么?
  1. 防敏感数据被永久泄露:导出文件常含用户trace/输入输出/PII,永久公开链接一旦被转发/记日志/缓存,就等于把数据永久敞开给任何拿到链接的人;签名URL是带加密签名和过期时间的临时授权,expiresAt一到自动失效
  2. 防止下载速度太慢
  3. 防止文件占用太多存储
  4. 防止用户重复下载
看答案与解析 点击展开
答案:A。导出的本质是把一批(往往敏感的)数据打包成一个可下载文件。如果给的是永久公开 URL,这个链接就成了数据的「长期后门」:它可能被贴进聊天、写进日志、被代理缓存,之后任何拿到它的人都能下载——相当于把这批数据永久泄露。签名 URL 的设计是「临时、可收回的授权」:URL 里嵌入加密签名和 expiresAt,过期即失效,BatchExport 记录也存着 expiresAt 备查。背后的安全原则是:对敏感数据的访问,默认应该是有时限、可收回的,而不是一次给出、永久敞开。
💭 发散思考(没有标准答案,动手或动脑想想)
  • 这一课是第九部分(自动化与集成)的收官:从 webhook/SSRF(L44)、Slack(L45)、分析集成(L46)到批量导出与操作(L47),主线是「把平台数据安全、可靠、可控地与外部世界连接」。回看这四课,你能总结出几条反复出现的『与外部世界打交道』的工程原则吗(如:不可信输入要纵深防御、敏感凭据/数据要加密与限时、耗时大活要队列+流式、会重试就必须幂等)?这些原则在你自己做过的『对接外部系统』里,哪些被遵守了、哪些被忽略并酿成过问题?

In the trace list you filter down to "the past 7 days, errored, from a certain user" — 50,000 traces — and want to export them all to CSV in one go, or delete them all with one click. This "do one thing to a whole batch of filtered data" need is this lesson's lead: Batch Export and Batch Action. They look different — one read-only produces a file, the other dangerously mutates data in bulk — but at heart they share one engineering posture: both built on a query/filter that may match a huge number of rows, both thrown onto a queue to run async because they're too slow to block a request, both using streaming + chunking to avoid stuffing tens of thousands of rows into memory at once. As the closing lesson of "Automation & Integrations," it once again assembles the streaming export (Lesson 46), job state machine (Lessons 30/35), signed URLs and object storage (Lesson 46) accumulated earlier, and spells out a hard constraint unavoidable when mutating data in bulk: idempotency.

📋 Analogy
Picture a giant library where you issue an order: "find all books before 2020, about astronomy, that are damaged." Then either export (copy a catalog of those books for you — read-only, produces a file) or act (destroy them all or move them to another branch — really changing the collection). For such a big job, the librarian won't make you wait at the front desk (too slow → hand it to the back office, notify you when done = async queue); nor will they haul tens of thousands of books onto the desk at once (desk collapses = memory blows up → cart by cart = streaming chunks). And "destroy/move" — operations that change the collection — have a catch: if the power cuts mid-way and you must start over, an already-destroyed book mustn't cause chaos by being "destroyed again" — so every step must be designed to be "safe to repeat" (idempotent). Exporting a catalog has no such worry (copying twice just wastes paper). This lesson is about how the library does these two kinds of "bulk big jobs" steadily and safely.

One query, two paths: read-only export vs data-mutating action

Both start from the same point: a filter/query condition (same origin as Lesson 23's list filter) that may match thousands of rows. Then they fork: batch export is read-only — stream the matched rows out, transform to CSV/JSON, write to object storage, give you a time-limited download link. Batch action mutates data — stream out the matched rows' ids, 1000 per chunk, executing delete / add-to-annotation-queue / add-to-dataset / re-evaluate. The commonality: both are slow (so async via queue, with a trackable state machine) and large (so streaming + chunking, never all-in-memory).

One filter, matching massive rows → handled in two paths filter/query conditionsame as Lesson 23 filtermay match tens of thousands batch export (read-only)stream query → CSV/JSON→ write object storage→ time-limited signed URL → email batch action (mutate)stream ids → 1000 per chunkdelete / add-to-annotation-queue / dataset/ re-evaluate (eval-create)⚠ must be idempotent (retry reruns processed chunks) shared skeletonslow → async queuetrackable state machinelarge → streaming + chunkingnever all-in-memory
Same-origin fork: batch export worker/src/features/batchExport/handleBatchExportJob.ts (read-only, produces a file); batch action worker/src/features/batchAction/handleBatchActionJob.ts (CHUNK_SIZE=1000, switch dispatches to trace-delete / *-add-to-annotation-queue / dataset-delete / eval-create processors). Both built on Lesson 23's filter, both via BullMQ queues.
Example: a batch export (CSV) + state machine QUEUED PROCESSING COMPLETED FAILED status in batch_exports 📄 export_traces_2026-06-27.csvstreamed in chunks id,timestamp,name,userId,totalCost,latency t_9f3a,2026-06-27T10:00Z,chat,u_8f3a,0.0123,1.84 t_2b71,2026-06-27T10:01Z,chat,u_2b71,0.0098,1.32 t_55c9,2026-06-27T10:02Z,summarize,u_55c9,0.0211,2.07 ⟲ idempotent: re-running the same export request yields the same file, no double work
A big export is a stateful, resumable background job, not a one-shot in-memory call (status enum from features/batchExport/types.ts; values illustrative): an export moves through a QUEUED → PROCESSING → COMPLETED state machine in the DB (to FAILED on error), producing a streamed, chunked CSV (header + rows here). Why a state machine instead of "click and download now"? Because an export can be millions of rows — it must run in the background, resume, retry, and be idempotent (re-running the same request gives the same result, no double billing). This reuses Lesson 30's "creation separate from execution" for the export case.
DimensionBatch ExportBatch Action
Essenceread-only: produces a filemutates data: delete/modify massive rows
OutputCSV/JSON in object storage + time-limited signed URL (emailed to you)no file; side effects land on the data
Big-data strategystreaming pipeline (transform format row by row)stream ids + process 1000 per chunk
Key constraintsigned URL expires (not public forever)must be idempotent (retry reruns processed chunks)
State machineQUEUED→PROCESSING→COMPLETED/FAILED/CANCELLEDalso tracked via BatchActionStatus

Batch export: stream → file → time-limited link → email

An export job is a state machine (BatchExportStatus: QUEUED→PROCESSING→COMPLETED/FAILED/CANCELLED), of the same lineage as Lesson 30's eval JobExecution and Lesson 35's dataset runs. On entry it first validates status (skip if CANCELLED, skip if not QUEUED — idempotent guard against re-execution), sets PROCESSING. The core is a streaming pipeline: getDatabaseReadStreamPaginated reads the matched rows by page, pipeline transforms them row by row through streamTransformations[format] into CSV/JSON, uploadFileBuffered multipart-uploads into object storage (the same StorageService as Lesson 46). After upload it generates a time-limited signed URL (getSignedUrl(expiresInSeconds)), sets status COMPLETED, records url and expiresAt, and finally emails the download link to the user.

export state machine: QUEUED → PROCESSING → COMPLETED (idempotent) 1 stream read page-read rows getDatabaseRead… 2 transform row → CSV/JSON constant memory 3 upload uploadFileBuffered → S3 / GCS 4 signed URL getSignedUrl time-limited 5 email download link → user inbox constant memory throughout: tens of thousands of rows won't blow up; the signed URL never exposes data long-term
1

Validate status + set PROCESSING

CANCELLED → skip; not QUEUED → skip (guard against re-executing the same job). Otherwise update({status: PROCESSING}) claims it.

2

Stream query + transform format

getDatabaseReadStreamPaginated reads matched rows by page; pipeline transforms row by row through streamTransformations[format] into CSV/JSON — constant memory, no blowup at tens of thousands of rows.

3

Upload to object storage + signed URL

uploadFileBuffered multipart-uploads to S3/GCS; getSignedUrl(expiresInSeconds) generates a time-limited download link — data isn't public forever.

4

Set COMPLETED + email

update({status: COMPLETED, url, finishedAt, expiresAt}); sendBatchExportSuccessEmail sends the downloadLink to the user's inbox.

worker/src/features/batchExport/handleBatchExportJob.tsstreaming export + time-limited link
// 1) state-machine guard: idempotent against re-execution
if (jobDetails.status === BatchExportStatus.CANCELLED) return;       // cancelled, skip
if (jobDetails.status !== BatchExportStatus.QUEUED)   return;       // already handled, skip
await prisma.batchExport.update({ ..., data: { status: BatchExportStatus.PROCESSING } });

// 2) streaming pipeline: paginated read → transform to CSV/JSON row by row (constant memory)
const fileStream = pipeline(
  await getDatabaseReadStreamPaginated({ ...streamParams }),
  streamTransformations[jobDetails.format](),                  // CSV / JSON / JSONL
);

// 3) upload to object storage → generate a time-limited signed URL
await storageService.uploadFileBuffered({ fileName, data: fileStream, ... });
const signedUrl = await storageService.getSignedUrl(fileName, expiresInSeconds);

// 4) set completed + record expiry + send email
await prisma.batchExport.update({ ..., data: {
  status: BatchExportStatus.COMPLETED, url: signedUrl,
  finishedAt: new Date(), expiresAt: new Date(Date.now() + expiresInSeconds*1000) } });
await sendBatchExportSuccessEmail({ receiverEmail: user.email, downloadLink: signedUrl, ... });

Batch action: chunks of 1000, and must be idempotent

A batch action mutates data; the chunking flow is: using the filter + a cutoffCreatedAt cutoff time (a snapshot boundary — rows created after the job starts aren't included, so the target set is stable), stream out the matched rows' ids, accumulate 1000 per chunk, and hand each to processActionChunk, which dispatches by actionId: trace-delete deletes traces, *-add-to-annotation-queue adds to Lesson 32's annotation queue, dataset-delete deletes datasets, eval-create backfills Lesson 29's evaluation on historic data… The source leaves a conspicuous red-line comment here: "All operations must be idempotent. The job is retried on failure, and on retry chunks already processed may be processed again." This isn't a suggestion — it's a hard constraint.

Stream ids → 1000 per chunk → dispatch (and must be idempotent) stream matched idsfilter + cutoffCreatedAtsnapshot boundary: fixed setexcludes rows born after job chunk 1·1000 chunk 2·1000 chunk 3 … processActionChunkswitch(actionId)dispatch per chunkCHUNK_SIZE=1000: protectsthe DB from one giant hit trace-delete (delete traces) *-add-to-annotation-queue (Lesson 32) dataset-delete / add-to-dataset eval-create (backfill Lesson 29 eval) ⚠ red line: all operations must be idempotentjob failure retries the whole thing → already-processed chunks may run againso "delete an already-deleted" / "add an already-added" must not error or double side effects
Chunking + idempotency: handleBatchActionJob.ts:46 CHUNK_SIZE=1000; :61 processActionChunk switch dispatches (trace-delete / *-add-to-annotation-queue / score-delete / dataset-delete / eval-create); the comment at :57-60 states clearly "all operations must be idempotent… on retry processed chunks may be re-processed." cutoffCreatedAt serves as the snapshot boundary fixing the target set.
fetchstream + snapshot boundary
Stream the matched rows' ids using the filter + cutoffCreatedAt. cutoffCreatedAt is key: it pins the target set to the instant the job started, so rows written during the job's few-minute run aren't swept in — avoiding "delete while new ones keep arriving, never finishing."
chunkCHUNK_SIZE = 1000
Rather than hitting tens of thousands of rows at once, accumulate 1000 ids per chunk and process chunk by chunk. This both protects the database from one giant statement and keeps the retry granularity at "chunk" rather than "the whole job from scratch."
iron ruleidempotency
A failed queue job retries the whole thing, and on retry already-succeeded chunks run again. So every operation must "not error on repeat, not double side effects": deleting an already-deleted = silent success, adding an already-queued one = no duplicate add. This is the unavoidable hard constraint for bulk data mutation.
🎯 Design trade-off
Why is "batch actions must be idempotent" an iron rule, not "try your best"? Because it collides head-on with the reality that "queues retry." A job deleting 50,000 rows runs in 50 chunks; say the worker crashes at chunk 30 — BullMQ re-dispatches the whole job (it has no idea which chunk you reached internally), so the first 29 chunks run again. If "delete" isn't idempotent (e.g. deleting a nonexistent id throws), the retry fails outright and the job is stuck forever; if "add to queue" isn't idempotent, the same batch of traces gets added twice. Idempotency isn't a nicety but "the precondition for correctness in a system that retries." The author pinned this sentence as a conspicuous comment atop the function precisely to remind every future editor: every action you write lives in a world where "it may be called more than once." This is the same clarity about "the distributed world is unreliable" as Lesson 43's metering "exactly once" and Lesson 30's eval dedup.

Why does export give a "time-limited signed URL" rather than a permanently-public download address? Because the export file often holds sensitive data (users' traces, inputs/outputs, possibly PII). Generating a permanent public link means that once forwarded, logged, or cached, it permanently leaks this data to a place where anyone with the link can download it. A signed URL is "temporary authorization": the link carries a cryptographic signature and an expiry, auto-invalidating once expiresAt passes. Combined with the expiresAt on the BatchExport record, the platform always has an auditable answer to "can this export still be downloaded." Access to sensitive data should default to time-bound and revocable, not given-once and open-forever.
🎯 Key points · Part 9 wrap-up
  • One query, two paths: batch export (read-only, produces a file) and batch action (mutates data — delete/add-to-queue/add-to-dataset/re-eval) both build on Lesson 23's filter, both go async via queue with a state machine for being slow, both use streaming + chunking to avoid memory blowup.
  • Batch export = stream → file → time-limited link → email: BatchExportStatus state machine (QUEUED→PROCESSING→COMPLETED); getDatabaseReadStreamPaginated+pipeline transform to CSV/JSON; uploadFileBuffered to object storage; getSignedUrl gives a time-limited download link; record expiresAt then email.
  • Batch action = chunks of 1000 + idempotency: cutoffCreatedAt snapshot boundary fixes the target set; CHUNK_SIZE=1000 protects the DB and refines retry granularity; processActionChunk dispatches by actionId (trace-delete / add-to-annotation-queue / dataset-delete / eval-create).
  • Idempotency is the iron rule: queue failure retries the whole job, processed chunks rerun, so every action must "not error on repeat, not double side effects." It's the precondition for correctness in a distributed retry system, of the same origin as Lesson 43's "exactly once" and Lesson 30's dedup.
  • Time-limited signed URL: exports hold sensitive data, so give a temporary-authorization link with an expiry rather than a permanently-public address — access to sensitive data should default to time-bound and revocable.
  • Part 9 finale: Automation (L44 webhook/SSRF) → Slack (L45) → analytics integrations (L46 fan-out/incremental/streaming) → batch export & actions (L47). One through-line: connecting the platform's data to the outside world securely, reliably, and controllably.

🧪 Self-test - think about the design

1. Batch export and batch action look quite different (one produces a file, one mutates data), yet this lesson stresses they share one engineering skeleton. What is that shared skeleton?
  1. both only work in self-host
  2. both require paid unlock
  3. both encrypt data before storing
  4. both build on a filter that may match massive rows, both go async via queue for being slow (with a trackable state machine), both use streaming + chunking to avoid loading tens of thousands of rows into memory at once—the only difference is 'read-only produces a file' vs 'mutates data'
Show answer & explanation click to expand
Answer: D. Recognizing 'the shared skeleton of different features' is this lesson's core. Batch export (read-only) and batch action (mutating) look very different but face the same two engineering realities: ① data volume may be enormous (matching tens of thousands of rows) → must stream + chunk, not all-in-memory; ② processing is slow → can't block an HTTP request, must run async on a queue, with a state machine (BatchExportStatus / BatchActionStatus) so users can track progress. See through this commonality and you know: any 'do one thing to a whole batch of data' need should map onto this skeleton.
2. The batch action source pins a conspicuous comment atop processActionChunk: 'All operations must be idempotent. The job retries on failure, and on retry chunks already processed may be processed again.' Why is idempotency a 'hard constraint' here, not 'nice to have'?
  1. because the database requires all writes to be idempotent
  2. because it collides head-on with 'queues retry': deleting 50k rows in 50 chunks, crash at chunk 30, BullMQ re-dispatches the whole job (it doesn't know which chunk you reached), the first 29 rerun—if delete/add-to-queue isn't idempotent, either the retry fails outright and the job sticks, or the same batch gets added twice
  3. because idempotency makes the job run faster
  4. because idempotency saves storage space
Show answer & explanation click to expand
Answer: B. A queue's 'retry on failure' is the source of reliability, but meeting 'bulk data mutation' yields a sharp consequence: a retry re-dispatches the whole job, and the chunks already done inside it—the queue doesn't know, and reruns them too. So every action lives in a world where 'it may be called again.' Without idempotency: deleting an already-deleted id throws → the retry fails forever, job stuck; re-adding a batch of traces to the annotation queue → data polluted. So idempotency is the precondition for 'correctness in a retrying system', not optional. Same clarity about 'distributed unreliability' as Lesson 43's 'exactly once' and Lesson 30's eval dedup.
3. After a batch export completes, the user gets a time-limited 'signed URL' rather than a permanently-public download address. What does this design mainly guard against?
  1. guards against permanent leakage of sensitive data: export files often contain users' traces/inputs-outputs/PII; a permanent public link, once forwarded/logged/cached, opens the data forever to anyone with the link; a signed URL is temporary authorization with a cryptographic signature and expiry, auto-invalidating once expiresAt passes
  2. prevents slow download speed
  3. prevents files from using too much storage
  4. prevents users downloading repeatedly
Show answer & explanation click to expand
Answer: A. An export essentially packages a batch of (often sensitive) data into a downloadable file. A permanent public URL becomes a 'long-term backdoor' to that data: it may be pasted into chats, written to logs, cached by proxies, and anyone who later gets it can download—effectively a permanent leak. A signed URL is 'temporary, revocable authorization': the URL embeds a cryptographic signature and expiresAt, invalidating on expiry, and the BatchExport record stores expiresAt for audit. The security principle: access to sensitive data should default to time-bound and revocable, not given-once and open-forever.
💭 Open questions (no single right answer - just think or try)
  • This lesson closes Part 9 (Automation & Integrations): from webhook/SSRF (L44), Slack (L45), analytics integrations (L46) to batch export & actions (L47), the through-line is 'connecting platform data to the outside world securely, reliably, controllably.' Reviewing these four lessons, can you distill a few recurring engineering principles for 'dealing with the outside world' (e.g.: untrusted input needs defense-in-depth, sensitive credentials/data need encryption and time-limits, slow big jobs need queue+streaming, anything that retries must be idempotent)? In your own 'integrate with external systems' work, which were honored, and which were ignored and caused problems?