🪢 Langfuse 图解教程Langfuse Visual Guide 第三部分 · 摄取链路Part 3 · The Ingestion Path 19 / 55
第三部分 · 摄取链路Part 3 · The Ingestion Path

媒体与 blob 存储Media & blob storage

🧭 读前Before you start
先读 第12课 摄取与 第7课 里 S3 的角色,本课讲媒体与大块 blob 怎么存。
Read Lesson 12 (ingestion) and S3's role in Lesson 7; this lesson covers how media and large blobs are stored.

多模态时代,一次 LLM 调用的 input/output 里可能塞着一张图、一段音频、一个 PDF——动辄几 MB 的二进制。如果把这些原样写进第 8 课的 ClickHouse 宽表, input/output 列会瞬间膨胀,查询变慢、存储浪费。Langfuse 的解法干净利落:大块二进制根本不进 ClickHouse, 而是单独存进 S3 媒体桶,事件里只留一个小小的引用串。这一课,就讲这套「把笨重的 blob 挡在热路径之外」的媒体存储设计。

📚 生活类比
把它想成图书馆。目录卡片(ClickHouse 里的事件)上不会抄整本书,只写一个索书号(引用串 @@@langfuseMedia:…@@@); 真正厚重的书(媒体二进制)摆在书库的架子上(S3 媒体桶)。两个人上传同一本书,图书馆不会摆两本——靠书的「指纹」(sha256 哈希)认出是同一本,只存一份。 你需要看书时,凭索书号去把书取出来。卡片轻、检索快;书库便宜、容量大——各得其所。

blob 与引用分离:事件里只留索书号

核心思想一句话:把 blob 从事件里拆出去。媒体二进制存进 S3 媒体桶;事件的 input/output 里,原本那张图的位置被替换成一个引用串 @@@langfuseMedia:…@@@。于是 ClickHouse 存的是几十字节的索书号,而不是几 MB 的图片。

blob 与引用分离:重的进 S3,轻的进 ClickHouse 原始消息(多模态) "text": "这张图里是什么?" "image": <3.4 MB base64>🖼️ 笨重的二进制绝不能进 ClickHouse ▶拆分 S3 媒体桶存图片本体(一份)按 sha256 命名/去重 ClickHouse 事件"text": "这张图里是什么?""image": @@@langfuseMedia:…@@@ blob_storage_file_log台账:哪个 blob属于哪个 entity
重的进 S3、轻的进 ClickHouse:多模态消息里的图片被替换成 @@@langfuseMedia:…@@@ 引用串,本体存入 S3 媒体桶(按 sha256 去重),blob_storage_file_log 记录每个 blob 归属哪个 entity。源码:packages/shared/src/utils/mediaReferences.ts:6(MEDIA_REFERENCE_PATTERN)。
示例:媒体存指针,本体落 S3 observation.input(小) { "role": "user", "content": [ {"type":"text","text":"这张图…"}, @@@langfuseMedia:…id=med_9@@@ ] } 只存「指针」,行很小、可聚合 解引用 S3 对象(大) key: {projectId}/{mediaId} proj_ab/med_9.png content-type: image/png size: 1.8 MB 🖼 本体(可能很大)安静躺这 既不撑大宽表,又能按需取
指针进表,本体落 S3(token 形如 MEDIA_REFERENCE_PATTERN;值为示例):图片/音频不直接塞进 observation,而是替换成一个 @@@langfuseMedia:…@@@ 引用 token,真正的字节按 {{projectId}}/{{mediaId}} 存到 S3。于是宽表的行保持小而可聚合,UI 要看时再按指针去 S3 取——「队列轻、本体重」在媒体场景的延续。
packages/shared/src/utils/mediaReferences.ts & clickhouse/migrations/.../0011_*.sql引用 + 台账
// 事件里媒体被替换成这种小引用串,ClickHouse 只存它(几十字节)
export const MEDIA_REFERENCE_PATTERN = /@@@langfuseMedia:.+?@@@/g;
// 例:@@@langfuseMedia:type=image/png|id=abc123|source=base64@@@

// blob_storage_file_log:所有 blob 的台账(又是第 8 课的 ReplacingMergeTree 宽表)
CREATE TABLE blob_storage_file_log (
  project_id String, entity_type String, entity_id String, event_id String,
  bucket_name String, bucket_path String,        -- 这个 blob 在哪
  event_ts DateTime64(3), is_deleted UInt8,
) ENGINE = ReplacingMergeTree(event_ts, is_deleted)
  ORDER BY (project_id, entity_type, entity_id, event_id);

为什么非得这么分?因为遥测数据和媒体 blob 的「体质」根本不同,硬放在一起会两头受罪——一个被撑大、一个被低效访问。对比一下就一目了然:

维度遥测数据(trace/observation)媒体 blob(图/音/文件)
大小小,几 KB 的结构化 JSON/文本大,动辄几 MB 的二进制
访问方式高频查询、过滤、聚合几乎只按 id 整取,不查内部
该放哪ClickHouse 宽表(第 8 课)——为分析而生S3 对象存储——为大文件而生、便宜
事件里留什么—只留 @@@langfuseMedia:…@@@ 引用串

上传:直传 S3、按指纹去重

媒体怎么进到 S3?关键是SDK 直接传给 S3,本体不经过 Langfuse 的应用服务器。流程是:SDK 先把媒体的指纹(sha256 哈希)、 类型、大小报给 POST /api/public/media;服务端据此决定——这媒体之前传过吗?

presigned 直传 + 内容寻址去重 SDK报 sha256+类型+大小 media 端点查 prisma.media 去重 命中:已传过同指纹→ uploadUrl=null,跳过上传 未命中:返回 presigned URL一个有时效的 S3 直传地址 S3 媒体桶SDK 凭 URL 直传本体 关键:媒体本体走 SDK→S3 直连,绝不经过 Langfuse 应用服务器 内容寻址:媒体按 (project, sha256) 唯一;同一文件传一百次也只占一份存储大小受 LANGFUSE_S3_MEDIA_MAX_CONTENT_LENGTH 限制
presigned 直传 + 去重:SDK 报指纹 → 服务端查 prisma.media(按 projectId+sha256 唯一);已传过则回 uploadUrl: null 跳过,否则回 presigned URL 让 SDK 直传 S3。本体不过应用层。源码:web/src/features/media/server/mediaService.ts:66-122。
1

SDK 报指纹

计算媒体的 sha256、contentType、contentLength,POST /api/public/media(先不传本体)。

2

服务端去重判断

按 (projectId, sha256Hash) 查 prisma.media;已成功上传过同类型 → 返回 uploadUrl: null,秒过。

3

否则发 presigned URL

新媒体则算出 mediaId、bucketPath,返回一个有时效的 S3 直传地址。

4

SDK 直传 S3

SDK 凭 URL 把本体直接 PUT 进 S3 媒体桶,不经 Langfuse 应用服务器——省带宽、不堵 API。

5

关联 + 记台账

媒体与 trace/observation 关联;blob_storage_file_log 记下这个 blob 归属哪个 entity,供生命周期管理与导出。

内容寻址去重在真实场景里收益惊人。设想你做了个「图片问答」应用,每条对话都带同一张企业 logo 当水印,或者一个 few-shot 提示里固定塞着同几张示例图—— 一天下来可能有上万条 trace 引用同一张图。靠 sha256 指纹,这张图在 S3 里只存一份,后续每次「上传」都会在第 2 步被识破、uploadUrl: null 秒过, 既省存储、又省带宽、还省了 SDK 反复上传的时间。这就是把「同一性」交给内容哈希、而非文件名或时间戳来判断的妙处——相同的字节,永远只有一份,无论它被多少条 trace、多少个项目内的对话引用。

取用时反过来:GET /api/public/media/[mediaId] 返回一个有时效的下载地址,UI 或你的程序凭它去 S3 取回本体。 这个「有时效」也顺带管住了访问控制:下载链接由服务端在鉴权之后才签发、且很快过期,媒体既不会被公开裸奔在互联网上、也不会绕过项目边界(第 10 课)被别的租户读到。 整个生命周期里,几 MB 的二进制只在 SDK 和 S3 之间流动,Langfuse 的应用层和 ClickHouse 全程只碰那个轻飘飘的引用串——这正是把笨重负载挡在热路径之外的关键。

UI / 你的程序 只拿到引用串 Langfuse 应用层 鉴权 → 签 presigned GET S3 媒体桶 blob 本体(几 MB) ① ② ③ ① 请求 mediaId → ② 应用层鉴权后签有时效 URL → ③ 凭 URL 直接取 S3 本体 应用层与 ClickHouse 全程只碰引用串;URL 很快过期、不跨项目边界(L10)

😖 假如塞进 ClickHouse

input 列里躺着几 MB base64,宽表体积暴涨;连「查最近 100 条 trace」这种压根不碰图的查询,也要扫过这些大字段,全表跟着变慢——一颗老鼠屎坏一锅汤。

😀 分离到 S3

ClickHouse 只存几十字节引用,宽表始终精瘦、扫描飞快;图片躺在便宜的对象存储里,要用才按 id 取。各自待在最适合的地方,互不拖累,这也是第 7 课「双存储分工」在媒体场景的延续。

🎯 设计取舍
为什么大费周章地把媒体单拎出来,而不是图省事直接塞进事件? 因为blob 和遥测数据的「体质」完全不同,硬塞在一起会两败俱伤。 遥测数据(trace/observation)小、结构化、要被高频查询和聚合,所以放进 ClickHouse 宽表(第 8 课);而媒体大、是不透明二进制、几乎只按 id 整取, 天生属于对象存储。如果把几 MB 的图片塞进 ClickHouse 的 input 列,会同时毁掉两件事:宽表被撑爆、扫描变慢(连不带图的查询都被拖累),又用昂贵的分析库去干对象存储的活。 Langfuse 的做法是各按本性安置:结构化的进 ClickHouse、二进制的进 S3,中间用一个 @@@langfuseMedia@@@ 引用串牵线;再用 sha256 内容寻址天然去重、 用 presigned URL 让本体绕开应用层直传。这套组合拳背后是一条朴素的工程智慧:让每种数据待在最适合它的存储里——这也为第三部分「摄取链路」画上句点。
🎯 本课要点 · 第三部分收官
  • 多模态媒体(图/音/文件)是大块二进制,绝不能塞进第 8 课的 ClickHouse 宽表——会撑爆 input/output 列、拖慢所有查询。
  • blob 与引用分离:本体存进 S3 媒体桶,事件里只留一个小引用串 @@@langfuseMedia:…@@@(MEDIA_REFERENCE_PATTERN)。
  • presigned 直传:SDK 凭服务端发的有时效 URL 把媒体直接 PUT 进 S3,本体不经 Langfuse 应用层,省带宽、不堵 API。
  • 内容寻址去重:媒体按 (projectId, sha256) 唯一;已传过则回 uploadUrl: null 跳过——同一文件只占一份存储。
  • blob_storage_file_log(第 8 课同款 ReplacingMergeTree 宽表)记录每个 blob 归属哪个 entity,供生命周期管理与导出。核心智慧:让每种数据待在最适合它的存储里。

🧪 自测 · 想一想为什么这么设计

1. 一次多模态调用的 input 里有一张 3MB 的图。Langfuse 是怎么处理它的,ClickHouse 里最终存了什么?
  1. 把 3MB base64 直接存进 ClickHouse 的 input 列
  2. 把图片压缩后存进 Postgres
  3. 把图片丢弃,只留文字
  4. 图片本体存进 S3 媒体桶,事件里那张图被替换成一个小引用串 @@@langfuseMedia:…@@@;ClickHouse 只存这个几十字节的引用,绝不存 3MB 二进制
看答案与解析 点击展开
答案:D。大块二进制绝不能进 ClickHouse 宽表,否则 input/output 列暴涨、拖慢所有查询。Langfuse 把 blob 与引用分离:本体进 S3 媒体桶,事件里只留 MEDIA_REFERENCE_PATTERN 匹配的小引用串,ClickHouse 始终精瘦。
2. 媒体上传时,SDK 先把 sha256 报给服务端、而不是直接把文件 POST 过去。这个“先报指纹”的设计带来哪两个好处?
  1. ① 加密文件 ② 压缩文件
  2. ① 去重:按 (projectId, sha256) 查 prisma.media,已传过就回 uploadUrl=null 跳过,同一文件只存一份;② 直传:未命中则返回 presigned URL,SDK 把本体直接 PUT 进 S3,绕开应用服务器省带宽
  3. ① 给文件改名 ② 给文件分类
  4. ① 让上传更慢 ② 让 API 更忙
看答案与解析 点击展开
答案:B。先报指纹让服务端能在传输前就判断“是否已有同一文件”,实现内容寻址去重;命中则秒过,未命中才发 presigned URL 让 SDK 直传 S3。本体全程不经 Langfuse 应用层,既省带宽又不堵 API——这是把大文件挡在热路径之外的关键一招。
3. 为什么 Langfuse 要把遥测数据放 ClickHouse、媒体 blob 放 S3,而不是图省事都放一处?背后的核心原则是什么?
  1. 因为两者“体质”不同:遥测小、结构化、要高频查询聚合(适合 ClickHouse 宽表);媒体大、是不透明二进制、几乎只按 id 整取(适合对象存储)。让每种数据待在最适合它的存储里,互不拖累
  2. 因为 S3 比 ClickHouse 更快查询
  3. 纯粹是历史原因
  4. 因为 ClickHouse 不能存文本
看答案与解析 点击展开
答案:A。把几 MB 图片塞进 ClickHouse 会同时毁两件事:宽表撑爆、连不碰图的查询也被拖慢,又用昂贵分析库干对象存储的活。各按本性安置——结构化进 ClickHouse、二进制进 S3、引用串牵线、sha256 去重——这条“让每种数据待在最适合的存储里”正是第 7 课双存储分工的延续,也为第三部分收尾。
💭 发散思考(没有标准答案,动手或动脑想想)
  • “让每种数据待在最适合它的存储里,用一个小引用把它们牵起来”——这条原则在你做过的系统里有没有类似场景(比如大文件、日志、缩略图)?你会怎么决定“什么放主库、什么放对象存储/外部存储”,那条边界怎么画?内容寻址去重又能帮你省下什么?

In the multimodal era, an LLM call's input/output may carry an image, an audio clip, a PDF — binaries of several MB. Write those verbatim into Lesson 8's ClickHouse wide tables and the input/output columns balloon instantly, queries slow, storage is wasted. Langfuse's answer is clean: large binaries never enter ClickHouse at all; they're stored separately in an S3 media bucket, and the event keeps only a tiny reference string. This lesson covers that media-storage design — keeping bulky blobs off the hot path.

📚 Analogy
Think of a library. The catalog card (the ClickHouse event) never transcribes the whole book, just a call number (the reference string @@@langfuseMedia:…@@@); the heavy books themselves (the media binaries) sit on the stacks (the S3 media bucket). Two people upload the same book and the library doesn't shelve two copies — it recognizes them as one by the book's "fingerprint" (the sha256 hash), storing just one. When you want to read, you fetch the book by its call number. Cards are light and searchable; stacks are cheap and roomy — each in its right place.

Blob and reference, split: the event keeps only a call number

The core idea in one line: pull the blob out of the event. Media binaries go to the S3 media bucket; in the event's input/output, where that image used to sit, a reference string @@@langfuseMedia:…@@@ takes its place. So ClickHouse stores a few-dozen-byte call number, not a multi-MB image.

Blob/reference split: heavy to S3, light to ClickHouse raw message (multimodal) "text": "what's in this image?" "image": <3.4 MB base64>🖼️ bulky binarymust NOT enter ClickHouse ▶split S3 media bucketstores the image (once)named/deduped by sha256 ClickHouse event"text": "what's in this image?""image": @@@langfuseMedia:…@@@ blob_storage_file_logledger: which blobbelongs to which entity
Heavy to S3, light to ClickHouse: the image in a multimodal message is replaced by a @@@langfuseMedia:…@@@ reference; the body goes to the S3 media bucket (deduped by sha256), and blob_storage_file_log records which entity each blob belongs to. Source: packages/shared/src/utils/mediaReferences.ts:6 (MEDIA_REFERENCE_PATTERN).
Example: store a media pointer, body to S3 observation.input (small) { "role": "user", "content": [ {"type":"text","text":"this image…"}, @@@langfuseMedia:…id=med_9@@@ ] } stores only a pointer; the row stays small and aggregatable dereference S3 object (big) key: {projectId}/{mediaId} proj_ab/med_9.png content-type: image/png size: 1.8 MB 🖼 the blob (possibly large) sits quietly here neither bloats the wide table nor blocks on-demand fetch
Pointer in the table, body in S3 (token shape per MEDIA_REFERENCE_PATTERN; values illustrative): images/audio aren't embedded in the observation; they're replaced by a @@@langfuseMedia:…@@@ reference token, while the real bytes go to S3 at {{projectId}}/{{mediaId}}. The wide-table row stays small and aggregatable, and the UI dereferences to S3 on demand — the “light queue, heavy body” idea applied to media.
packages/shared/src/utils/mediaReferences.ts & clickhouse/migrations/.../0011_*.sqlreference + ledger
// media in events is replaced by this small reference string; ClickHouse stores only it (~bytes)
export const MEDIA_REFERENCE_PATTERN = /@@@langfuseMedia:.+?@@@/g;
// e.g. @@@langfuseMedia:type=image/png|id=abc123|source=base64@@@

// blob_storage_file_log: a ledger of all blobs (again Lesson 8's ReplacingMergeTree wide table)
CREATE TABLE blob_storage_file_log (
  project_id String, entity_type String, entity_id String, event_id String,
  bucket_name String, bucket_path String,        -- where this blob lives
  event_ts DateTime64(3), is_deleted UInt8,
) ENGINE = ReplacingMergeTree(event_ts, is_deleted)
  ORDER BY (project_id, entity_type, entity_id, event_id);

Why split this way? Because telemetry data and media blobs have fundamentally different "constitutions", and forcing them together hurts both. A quick comparison makes it obvious:

dimensiontelemetry (trace/observation)media blob (image/audio/file)
sizesmall, a few KB of structured JSON/textlarge, often several MB of binary
accessfrequent query, filter, aggregatealmost only fetched whole by id, never queried inside
where it belongsClickHouse wide tables (Lesson 8) — built for analyticsS3 object storage — built for big files, cheap
what stays in the event—only the @@@langfuseMedia:…@@@ reference

Upload: direct to S3, deduped by fingerprint

How does media get into S3? The key: the SDK uploads directly to S3, the body never passing through Langfuse's app servers. The flow: the SDK first reports the media's fingerprint (sha256 hash), type and size to POST /api/public/media; the server decides — has this media been uploaded before?

Presigned direct upload + content-addressed dedup SDKreports sha256+type+size media endpointchecks prisma.media hit: same fingerprint exists→ uploadUrl=null, skip upload miss: return presigned URLa time-limited direct-to-S3 address S3 media bucketSDK uploads body via URL Key: media body goes SDK→S3 directly, never through Langfuse app servers content-addressed: media unique by (project, sha256); upload the same file 100x, store one copysize capped by LANGFUSE_S3_MEDIA_MAX_CONTENT_LENGTH
Presigned direct upload + dedup: the SDK reports the fingerprint → the server checks prisma.media (unique by projectId+sha256); already uploaded → returns uploadUrl: null to skip, else a presigned URL for the SDK to upload directly to S3. The body never touches the app layer. Source: web/src/features/media/server/mediaService.ts:66-122.
1

SDK reports the fingerprint

Computes the media's sha256, contentType, contentLength, POST /api/public/media (no body yet).

2

server dedup check

Looks up prisma.media by (projectId, sha256Hash); already uploaded with same type → returns uploadUrl: null, instant pass.

3

else issue a presigned URL

For new media, compute mediaId, bucketPath, and return a time-limited direct-to-S3 address.

4

SDK uploads directly to S3

The SDK PUTs the body straight into the S3 media bucket via the URL, bypassing Langfuse app servers — saves bandwidth, doesn't clog the API.

5

link + log

The media is linked to the trace/observation; blob_storage_file_log records which entity this blob belongs to, for lifecycle and export.

Content-addressed dedup pays off dramatically in practice. Imagine an "image Q&A" app where every conversation carries the same company logo as a watermark, or a few-shot prompt fixed with the same example images — over a day that could be tens of thousands of traces referencing one image. By sha256 fingerprint, that image is stored once in S3; every subsequent "upload" is caught at step 2 with an instant uploadUrl: null — saving storage, bandwidth, and the SDK's repeated upload time. That's the beauty of deciding "sameness" by content hash rather than filename or timestamp — identical bytes, only ever one copy, however many traces or conversations within a project reference it.

Retrieval is the reverse: GET /api/public/media/[mediaId] returns a time-limited download address, and the UI or your code fetches the body from S3 with it. That "time-limited" also handles access control: the download link is signed by the server only after auth and expires quickly, so media is neither left publicly exposed nor readable across the project boundary (Lesson 10) by another tenant. Across the whole lifecycle, the multi-MB binary only flows between the SDK and S3; Langfuse's app layer and ClickHouse only ever touch the featherweight reference string — exactly what keeps bulky payloads off the hot path.

UI / your code has only the reference Langfuse app layer auth → sign presigned GET S3 media bucket blob body (several MB) 1 2 3 1 request mediaId → 2 app signs a time-limited URL after auth → 3 fetch the body straight from S3 app layer and ClickHouse only touch the reference; the URL expires fast and never crosses the project boundary (L10)

😖 if stuffed into ClickHouse

Several MB of base64 sit in the input column, the wide table bloats; even a query like "fetch the latest 100 traces" that never touches the image must scan past these huge fields, slowing the whole table — one bad apple spoils the barrel.

😀 split out to S3

ClickHouse stores a few-dozen-byte reference, the wide table stays lean and scans fast; images rest in cheap object storage, fetched by id only when needed. Each in its best-fit place, neither dragging the other.

🎯 Design trade-off
Why go to all this trouble to pull media out, rather than just stuffing it into the event? Because blobs and telemetry have utterly different constitutions, and cramming them together hurts both. Telemetry (trace/observation) is small, structured, queried and aggregated frequently, so it lives in ClickHouse wide tables (Lesson 8); media is large, opaque binary, almost only fetched whole by id, inherently belonging in object storage. Stuff a multi-MB image into a ClickHouse input column and you ruin two things at once: the wide table bloats and scans slow (dragging down even image-free queries), and you waste a pricey analytical store doing object storage's job. Langfuse instead places each by its nature: structured into ClickHouse, binary into S3, linked by a @@@langfuseMedia@@@ reference; plus sha256 content addressing for natural dedup and presigned URLs to let bodies bypass the app layer. Behind this combo is a plain engineering wisdom: let each kind of data live in the storage that fits it best — fittingly closing out Part 3, "the ingestion path".
🎯 Key points · Part 3 wrap-up
  • Multimodal media (image/audio/file) are large binaries that must never enter Lesson 8's ClickHouse wide tables — they'd bloat input/output columns and slow every query.
  • Blob/reference split: the body goes to the S3 media bucket, the event keeps only a small reference @@@langfuseMedia:…@@@ (MEDIA_REFERENCE_PATTERN).
  • Presigned direct upload: the SDK PUTs media straight to S3 via a server-issued time-limited URL, the body bypassing Langfuse's app layer — saving bandwidth, not clogging the API.
  • Content-addressed dedup: media unique by (projectId, sha256); already uploaded → uploadUrl: null skip — the same file stored only once.
  • blob_storage_file_log (Lesson 8's ReplacingMergeTree wide table) records which entity each blob belongs to, for lifecycle and export. The core wisdom: let each kind of data live in the storage that fits it best.

🧪 Self-test - think about the design

1. A multimodal call's input has a 3MB image. How does Langfuse handle it, and what ends up in ClickHouse?
  1. stores the 3MB base64 straight into ClickHouse's input column
  2. compresses the image into Postgres
  3. discards the image, keeping only text
  4. the image body goes into the S3 media bucket, and in the event the image is replaced by a small reference @@@langfuseMedia:…@@@; ClickHouse stores only that few-dozen-byte reference, never the 3MB binary
Show answer & explanation click to expand
Answer: D. Large binaries must never enter ClickHouse wide tables, or input/output columns balloon and slow every query. Langfuse splits blob from reference: the body to the S3 media bucket, the event keeping only a small reference matched by MEDIA_REFERENCE_PATTERN, keeping ClickHouse lean.
2. On media upload, the SDK first reports the sha256 to the server instead of POSTing the file directly. What two benefits does this 'fingerprint first' design bring?
  1. ① encrypt the file ② compress the file
  2. ① dedup: look up prisma.media by (projectId, sha256), if already uploaded return uploadUrl=null to skip, storing the same file once; ② direct upload: on miss return a presigned URL so the SDK PUTs the body straight to S3, bypassing app servers and saving bandwidth
  3. ① rename the file ② categorize it
  4. ① slow the upload ② busy the API
Show answer & explanation click to expand
Answer: B. Reporting the fingerprint first lets the server decide 'do we already have this file' before any transfer, achieving content-addressed dedup; on hit it passes instantly, on miss it issues a presigned URL for the SDK to upload directly to S3. The body never passes through Langfuse's app layer, saving bandwidth and not clogging the API — key to keeping big files off the hot path.
3. Why does Langfuse put telemetry in ClickHouse and media blobs in S3, rather than everything in one place? What's the core principle?
  1. because their 'constitutions' differ: telemetry is small, structured, frequently queried/aggregated (fits ClickHouse wide tables); media is large, opaque binary, almost only fetched whole by id (fits object storage). Let each kind of data live in the storage that fits it best, neither dragging the other
  2. because S3 queries faster than ClickHouse
  3. purely historical reasons
  4. because ClickHouse can't store text
Show answer & explanation click to expand
Answer: A. Stuffing a multi-MB image into ClickHouse ruins two things: the wide table bloats and even image-free queries slow, and you waste a pricey analytical store on object storage's job. Placing each by its nature — structured to ClickHouse, binary to S3, linked by a reference, deduped by sha256 — extends Lesson 7's dual-store division of labor and closes out Part 3.
💭 Open questions (no single right answer - just think or try)
  • 'Let each kind of data live in the storage that fits it best, linked by a small reference' — is there a similar scenario in systems you've built (large files, logs, thumbnails)? How would you decide 'what goes in the main DB vs object/external storage', and where to draw that line? What could content-addressed dedup save you?