In the trace list you filter down to "the past 7 days, errored, from a certain user" — 50,000 traces — and want to export them all to CSV in one go, or delete them all with one click. This "do one thing to a whole batch of filtered data" need is this lesson's lead: Batch Export and Batch Action. They look different — one read-only produces a file, the other dangerously mutates data in bulk — but at heart they share one engineering posture: both built on a query/filter that may match a huge number of rows, both thrown onto a queue to run async because they're too slow to block a request, both using streaming + chunking to avoid stuffing tens of thousands of rows into memory at once.
As the closing lesson of "Automation & Integrations," it once again assembles the streaming export (Lesson 46), job state machine (Lessons 30/35), signed URLs and object storage (Lesson 46) accumulated earlier, and spells out a hard constraint unavoidable when mutating data in bulk: idempotency.
📋 Analogy
Picture a giant library where you issue an order: "find all books before 2020, about astronomy, that are damaged." Then either export (copy a catalog of those books for you — read-only, produces a file) or act (destroy them all or move them to another branch — really changing the collection).
For such a big job, the librarian won't make you wait at the front desk (too slow → hand it to the back office, notify you when done = async queue); nor will they haul tens of thousands of books onto the desk at once (desk collapses = memory blows up → cart by cart = streaming chunks). And "destroy/move" — operations that change the collection — have a catch: if the power cuts mid-way and you must start over, an already-destroyed book mustn't cause chaos by being "destroyed again" — so every step must be designed to be "safe to repeat" (idempotent). Exporting a catalog has no such worry (copying twice just wastes paper). This lesson is about how the library does these two kinds of "bulk big jobs" steadily and safely.
One query, two paths: read-only export vs data-mutating action
Both start from the same point: a filter/query condition (same origin as Lesson 23's list filter) that may match thousands of rows. Then they fork: batch export is read-only — stream the matched rows out, transform to CSV/JSON, write to object storage, give you a time-limited download link. Batch actionmutates data — stream out the matched rows' ids, 1000 per chunk, executing delete / add-to-annotation-queue / add-to-dataset / re-evaluate. The commonality: both are slow (so async via queue, with a trackable state machine) and large (so streaming + chunking, never all-in-memory).
Same-origin fork: batch export worker/src/features/batchExport/handleBatchExportJob.ts (read-only, produces a file); batch action worker/src/features/batchAction/handleBatchActionJob.ts (CHUNK_SIZE=1000, switch dispatches to trace-delete / *-add-to-annotation-queue / dataset-delete / eval-create processors). Both built on Lesson 23's filter, both via BullMQ queues.
A big export is a stateful, resumable background job, not a one-shot in-memory call (status enum from features/batchExport/types.ts; values illustrative): an export moves through a QUEUED → PROCESSING → COMPLETED state machine in the DB (to FAILED on error), producing a streamed, chunked CSV (header + rows here). Why a state machine instead of "click and download now"? Because an export can be millions of rows — it must run in the background, resume, retry, and be idempotent (re-running the same request gives the same result, no double billing). This reuses Lesson 30's "creation separate from execution" for the export case.
Dimension
Batch Export
Batch Action
Essence
read-only: produces a file
mutates data: delete/modify massive rows
Output
CSV/JSON in object storage + time-limited signed URL (emailed to you)
no file; side effects land on the data
Big-data strategy
streaming pipeline (transform format row by row)
stream ids + process 1000 per chunk
Key constraint
signed URL expires (not public forever)
must be idempotent (retry reruns processed chunks)
State machine
QUEUED→PROCESSING→COMPLETED/FAILED/CANCELLED
also tracked via BatchActionStatus
Batch export: stream → file → time-limited link → email
An export job is a state machine (BatchExportStatus: QUEUED→PROCESSING→COMPLETED/FAILED/CANCELLED), of the same lineage as Lesson 30's eval JobExecution and Lesson 35's dataset runs. On entry it first validates status (skip if CANCELLED, skip if not QUEUED — idempotent guard against re-execution), sets PROCESSING. The core is a streaming pipeline: getDatabaseReadStreamPaginated reads the matched rows by page, pipeline transforms them row by row through streamTransformations[format] into CSV/JSON, uploadFileBuffered multipart-uploads into object storage (the same StorageService as Lesson 46). After upload it generates a time-limited signed URL (getSignedUrl(expiresInSeconds)), sets status COMPLETED, records url and expiresAt, and finally emails the download link to the user.
1
Validate status + set PROCESSING
CANCELLED → skip; not QUEUED → skip (guard against re-executing the same job). Otherwise update({status: PROCESSING}) claims it.
2
Stream query + transform format
getDatabaseReadStreamPaginated reads matched rows by page; pipeline transforms row by row through streamTransformations[format] into CSV/JSON — constant memory, no blowup at tens of thousands of rows.
3
Upload to object storage + signed URL
uploadFileBuffered multipart-uploads to S3/GCS; getSignedUrl(expiresInSeconds) generates a time-limited download link — data isn't public forever.
4
Set COMPLETED + email
update({status: COMPLETED, url, finishedAt, expiresAt}); sendBatchExportSuccessEmail sends the downloadLink to the user's inbox.
worker/src/features/batchExport/handleBatchExportJob.tsstreaming export + time-limited link
Batch action: chunks of 1000, and must be idempotent
A batch action mutates data; the chunking flow is: using the filter + a cutoffCreatedAt cutoff time (a snapshot boundary — rows created after the job starts aren't included, so the target set is stable), stream out the matched rows' ids, accumulate 1000 per chunk, and hand each to processActionChunk, which dispatches by actionId: trace-delete deletes traces, *-add-to-annotation-queue adds to Lesson 32's annotation queue, dataset-delete deletes datasets, eval-create backfills Lesson 29's evaluation on historic data… The source leaves a conspicuous red-line comment here: "All operations must be idempotent. The job is retried on failure, and on retry chunks already processed may be processed again." This isn't a suggestion — it's a hard constraint.
Chunking + idempotency: handleBatchActionJob.ts:46CHUNK_SIZE=1000; :61 processActionChunk switch dispatches (trace-delete / *-add-to-annotation-queue / score-delete / dataset-delete / eval-create); the comment at :57-60 states clearly "all operations must be idempotent… on retry processed chunks may be re-processed." cutoffCreatedAt serves as the snapshot boundary fixing the target set.
fetchstream + snapshot boundary
Stream the matched rows' ids using the filter + cutoffCreatedAt. cutoffCreatedAt is key: it pins the target set to the instant the job started, so rows written during the job's few-minute run aren't swept in — avoiding "delete while new ones keep arriving, never finishing."
chunkCHUNK_SIZE = 1000
Rather than hitting tens of thousands of rows at once, accumulate 1000 ids per chunk and process chunk by chunk. This both protects the database from one giant statement and keeps the retry granularity at "chunk" rather than "the whole job from scratch."
iron ruleidempotency
A failed queue job retries the whole thing, and on retry already-succeeded chunks run again. So every operation must "not error on repeat, not double side effects": deleting an already-deleted = silent success, adding an already-queued one = no duplicate add. This is the unavoidable hard constraint for bulk data mutation.
🎯 Design trade-off
Why is "batch actions must be idempotent" an iron rule, not "try your best"? Because it collides head-on with the reality that "queues retry." A job deleting 50,000 rows runs in 50 chunks; say the worker crashes at chunk 30 — BullMQ re-dispatches the whole job (it has no idea which chunk you reached internally), so the first 29 chunks run again. If "delete" isn't idempotent (e.g. deleting a nonexistent id throws), the retry fails outright and the job is stuck forever; if "add to queue" isn't idempotent, the same batch of traces gets added twice. Idempotency isn't a nicety but "the precondition for correctness in a system that retries." The author pinned this sentence as a conspicuous comment atop the function precisely to remind every future editor: every action you write lives in a world where "it may be called more than once." This is the same clarity about "the distributed world is unreliable" as Lesson 43's metering "exactly once" and Lesson 30's eval dedup.
Why does export give a "time-limited signed URL" rather than a permanently-public download address? Because the export file often holds sensitive data (users' traces, inputs/outputs, possibly PII). Generating a permanent public link means that once forwarded, logged, or cached, it permanently leaks this data to a place where anyone with the link can download it. A signed URL is "temporary authorization": the link carries a cryptographic signature and an expiry, auto-invalidating once expiresAt passes. Combined with the expiresAt on the BatchExport record, the platform always has an auditable answer to "can this export still be downloaded." Access to sensitive data should default to time-bound and revocable, not given-once and open-forever.
🎯 Key points · Part 9 wrap-up
One query, two paths: batch export (read-only, produces a file) and batch action (mutates data — delete/add-to-queue/add-to-dataset/re-eval) both build on Lesson 23's filter, both go async via queue with a state machine for being slow, both use streaming + chunking to avoid memory blowup.
Batch export = stream → file → time-limited link → email: BatchExportStatus state machine (QUEUED→PROCESSING→COMPLETED); getDatabaseReadStreamPaginated+pipeline transform to CSV/JSON; uploadFileBuffered to object storage; getSignedUrl gives a time-limited download link; record expiresAt then email.
Batch action = chunks of 1000 + idempotency: cutoffCreatedAt snapshot boundary fixes the target set; CHUNK_SIZE=1000 protects the DB and refines retry granularity; processActionChunk dispatches by actionId (trace-delete / add-to-annotation-queue / dataset-delete / eval-create).
Idempotency is the iron rule: queue failure retries the whole job, processed chunks rerun, so every action must "not error on repeat, not double side effects." It's the precondition for correctness in a distributed retry system, of the same origin as Lesson 43's "exactly once" and Lesson 30's dedup.
Time-limited signed URL: exports hold sensitive data, so give a temporary-authorization link with an expiry rather than a permanently-public address — access to sensitive data should default to time-bound and revocable.
Part 9 finale: Automation (L44 webhook/SSRF) → Slack (L45) → analytics integrations (L46 fan-out/incremental/streaming) → batch export & actions (L47). One through-line: connecting the platform's data to the outside world securely, reliably, and controllably.
🧪 Self-test - think about the design
1. Batch export and batch action look quite different (one produces a file, one mutates data), yet this lesson stresses they share one engineering skeleton. What is that shared skeleton?
both only work in self-host
both require paid unlock
both encrypt data before storing
both build on a filter that may match massive rows, both go async via queue for being slow (with a trackable state machine), both use streaming + chunking to avoid loading tens of thousands of rows into memory at once—the only difference is 'read-only produces a file' vs 'mutates data'
Show answer & explanation click to expand
Answer: D. Recognizing 'the shared skeleton of different features' is this lesson's core. Batch export (read-only) and batch action (mutating) look very different but face the same two engineering realities: ① data volume may be enormous (matching tens of thousands of rows) → must stream + chunk, not all-in-memory; ② processing is slow → can't block an HTTP request, must run async on a queue, with a state machine (BatchExportStatus / BatchActionStatus) so users can track progress. See through this commonality and you know: any 'do one thing to a whole batch of data' need should map onto this skeleton.
2. The batch action source pins a conspicuous comment atop processActionChunk: 'All operations must be idempotent. The job retries on failure, and on retry chunks already processed may be processed again.' Why is idempotency a 'hard constraint' here, not 'nice to have'?
because the database requires all writes to be idempotent
because it collides head-on with 'queues retry': deleting 50k rows in 50 chunks, crash at chunk 30, BullMQ re-dispatches the whole job (it doesn't know which chunk you reached), the first 29 rerun—if delete/add-to-queue isn't idempotent, either the retry fails outright and the job sticks, or the same batch gets added twice
because idempotency makes the job run faster
because idempotency saves storage space
Show answer & explanation click to expand
Answer: B. A queue's 'retry on failure' is the source of reliability, but meeting 'bulk data mutation' yields a sharp consequence: a retry re-dispatches the whole job, and the chunks already done inside it—the queue doesn't know, and reruns them too. So every action lives in a world where 'it may be called again.' Without idempotency: deleting an already-deleted id throws → the retry fails forever, job stuck; re-adding a batch of traces to the annotation queue → data polluted. So idempotency is the precondition for 'correctness in a retrying system', not optional. Same clarity about 'distributed unreliability' as Lesson 43's 'exactly once' and Lesson 30's eval dedup.
3. After a batch export completes, the user gets a time-limited 'signed URL' rather than a permanently-public download address. What does this design mainly guard against?
guards against permanent leakage of sensitive data: export files often contain users' traces/inputs-outputs/PII; a permanent public link, once forwarded/logged/cached, opens the data forever to anyone with the link; a signed URL is temporary authorization with a cryptographic signature and expiry, auto-invalidating once expiresAt passes
prevents slow download speed
prevents files from using too much storage
prevents users downloading repeatedly
Show answer & explanation click to expand
Answer: A. An export essentially packages a batch of (often sensitive) data into a downloadable file. A permanent public URL becomes a 'long-term backdoor' to that data: it may be pasted into chats, written to logs, cached by proxies, and anyone who later gets it can download—effectively a permanent leak. A signed URL is 'temporary, revocable authorization': the URL embeds a cryptographic signature and expiresAt, invalidating on expiry, and the BatchExport record stores expiresAt for audit. The security principle: access to sensitive data should default to time-bound and revocable, not given-once and open-forever.
💭 Open questions (no single right answer - just think or try)
This lesson closes Part 9 (Automation & Integrations): from webhook/SSRF (L44), Slack (L45), analytics integrations (L46) to batch export & actions (L47), the through-line is 'connecting platform data to the outside world securely, reliably, controllably.' Reviewing these four lessons, can you distill a few recurring engineering principles for 'dealing with the outside world' (e.g.: untrusted input needs defense-in-depth, sensitive credentials/data need encryption and time-limits, slow big jobs need queue+streaming, anything that retries must be idempotent)? In your own 'integrate with external systems' work, which were honored, and which were ignored and caused problems?