🪢 Langfuse 图解教程Langfuse Visual Guide 第四部分 · 查询链路Part 4 · The Read Path 22 / 55
第四部分 · 查询链路Part 4 · The Read Path

仓储层:从 ClickHouse 读The repository layer

🧭 读前Before you start
先读 第8课 的 CH 表与 第21课 的 tRPC,本课讲仓储层如何从 ClickHouse 读。
Read Lesson 8 (the CH tables) and Lesson 21 (tRPC); this lesson covers how the repository layer reads from ClickHouse.

上一课结束在「resolver 调 @langfuse/shared 的仓储读库」。这一课就推开仓储层那扇门,看它怎么从 ClickHouse 把数据捞出来。核心是一条纪律: 所有分析查询都汇到同一个执行器——queryClickhouse()。它给每条查询统一套上:OTel 追踪、查询标签、失败退避重试、 资源错误友好包装。再加上一个让跨表 JOIN 变得便宜的关键技巧——回看时间窗。看懂这一层,就看懂了 Langfuse 所有读取的「总闸门」。

🏬 生活类比
把 ClickHouse 想成一个巨大的仓库。如果让每个工人(每段查询代码)各自闯进仓库乱翻,既没人记录谁拿了什么,出了乱子也无从追查。 Langfuse 的做法是开唯一一个领料窗口(queryClickhouse):所有人都从这儿递单子。窗口管理员会登记是谁来领的(查询标签)、给你贴张追踪票(OTel span)、 货架一时占用就稍等再试(退避重试);要是你一张单子想搬走整个仓库(内存超限),他不会让仓库塌掉,而是礼貌地退回一句「把范围缩小点再来」(资源错误)。 一个窗口,把观测、追责、容错全管住了。

唯一的领料窗口:queryClickhouse

每段读取代码都不直接连 ClickHouse,而是把查询和一组标签交给 queryClickhouse()。它在真正发查询的外面,统一裹了好几层横切关注:

所有读取汇到一个执行器 traces 服务 dashboard 引擎 public API …几十处调用 queryClickhouse() ① OTel span(每条查询可追踪) ② log_comment 标签(谁/哪个面/路由) ③ backOff 退避重试(网络抖动) ④ 资源错误友好包装 ClickHouseJSONEachRow 流式变体Stream / WithProgress(SSE)
一个窗口,四样统一:queryClickhouse() 给每条查询裹上 OTel span、log_comment 标签、退避重试、资源错误包装,再发往 ClickHouse(大读走 queryClickhouseStream/WithProgress)。源码:packages/shared/src/server/repositories/clickhouse.ts:639。
示例:仓储层发出的一条 CH 查询与结果 SELECTid, name, type, start_time, total_cost, usage_details FROMobservations WHEREproject_id = {projectId}AND start_time >= {from} ORDER BYstart_time DESCLIMIT50FINAL FINAL=取合并后最新 idnametypestart_timetotal_costusage obs_7fanswerGENERATION12:00:01.90.0041512/88 obs_3aretrieveSPAN12:00:00.1—— obs_9cplanGENERATION11:59:58.00.0009120/40 仓储层把这些行组装成 UI 要的对象
读路径的核心:参数化的列查询(依据 packages/shared/src/server/repositories 与 0002_observations;值为示例):仓储层永远带 WHERE project_id = ?(租户隔离)、ORDER BY start_time、LIMIT,并加 FINAL 取 ReplacingMergeTree 合并后的最新行。返回的若干行再被组装成列表/详情 UI 要的形状。
packages/shared/src/server/repositories/clickhouse.tsqueryClickhouse :639
export async function queryClickhouse<T>(opts) {
  assertNoLegacyEventsRead(opts.query);          // 守卫:别读旧 events 表
  return await instrumentAsync({ name: "clickhouse-query" }, async (span) => {
    setSpanQueryAttributes(span, opts.query);        // ① 把 SQL 挂到 OTel span
    return await backOff(                          // ③ 网络错可重试,指数退避
      () => sendClickhouseQuery({ ...opts, span }),     // ② 内部设 log_comment=JSON(tags)
      { numOfAttempts, retry: isRetryableError },
    ).catch((e) => { throw ClickHouseResourceError.wrapIfResourceError(e, tags); });
  });                                                // ④ 内存/超时 → 友好错误(第 21 课接住)
}

这四样里,查询标签尤其值得一提。sendClickhouseQuery 会把标签塞进 ClickHouse 的 log_comment 设置(即查询日志的注释字段): 于是每一条打到 ClickHouse 的 SQL,都自带「是哪个 project、哪个面(UI/公共 API/worker)、哪个路由发起的」。当某条慢查询拖垮集群时,运维能直接在 ClickHouse 的查询日志里 按标签定位元凶——这是把可观测性焊进数据访问层的一招,散落各处的裸查询永远做不到。

同一个领料窗口还备了几种取货方式,应对不同读取场景——但它们共享同一套追踪/标签/重试机制:

一个窗口,三种取货方式 —— 共享 OTel span · log_comment 标签 · 退避重试 queryClickhouse 一次取回全部(结果读成数组) queryClickhouseStream 边读边吐(流式 · 内存有界) queryClickhouseWithProgress 带进度的流 列表 / 详情 / 聚合(装得下内存) 批量导出(结果可能巨大) SSE 流式仪表盘(边查边推进度)
queryClickhouse一次取回全部
最常用:跑查询、把结果一次性读成数组返回。绝大多数列表、详情、聚合都走它。
queryClickhouseStream边读边吐
大读不必把整个结果攒在内存里,而是流式一行行吐出——批量导出(第 12 课提过的导出)这类「结果可能巨大」的场景靠它,内存有界。
queryClickhouseWithProgress带进度的流
给 SSE 流式仪表盘用:一边查一边把进度推给前端,让大仪表盘查询「有反馈、不假死」。

回看时间窗:让跨表 JOIN 不再扫全表

还记得第 8 课吗——trace、observation、score 是三张独立的宽表,彼此没有外键,而是靠时间关联。问题来了:要把一条 trace 的观测和评分拼起来, 难道要在动辄几个月、上亿行的 observations 表里全表搜一遍?那会慢到无法接受。仓储层的答案是回看时间窗:利用「一条 trace 的观测,几乎都集中在它发生后的一小段时间内」这个事实, 只在一个有界的时间范围里找。

以 trace 时间戳为锚,只看有界的回看窗 时间 → trace.timestamp(锚点) 观测窗:start_time ≥ 锚 − 2 天 评分窗:1 小时 窗外:不扫(几个月的旧分区被跳过) 窗外:不扫 OBSERVATIONS_TO_TRACE_INTERVAL=2 天 · SCORE_TO_TRACE_OBSERVATIONS_INTERVAL=1 小时
有界回看,跳过旧分区:以 trace 的 timestamp 为锚,观测只在 start_time ≥ 锚 − 2 天 内找、评分在 1 小时内找。这些常量被注入 WHERE 子句,配合第 8 课「按月分区」让 JOIN 只碰几个分区。源码:repositories/constants.ts:4,9。

😖 不设窗:全表 JOIN

为一条 trace 找观测,要在整张 observations 表(几个月、上亿行)里搜。每次列表查询都这样,集群很快被拖垮——这正是把分析库当事务库用的典型翻车。

😀 设回看窗:有界扫描

把 start_time ≥ trace.timestamp − 2 天 注入 WHERE,配合按月分区,JOIN 只读那几个相关分区。扫描量从「几个月」骤降到「两天」,查询飞快。

当然,窗也有代价:极少数跨度异常长的 observation(晚于 2 天才结束)可能落在窗外、被漏掉。这是一个有意识的取舍—— 用「放过极长尾的一点点正确性」换「JOIN 不再扫全表」的巨大成本节省。Langfuse 据源码记录的分布(repositories/README.md:约 96% 的观测在 trace 之后 2 分钟内开始,但有很长的长尾)把窗设成 2 天,留足极大的安全裕量,几乎不会真漏。 正确性与成本之间,工程师按真实数据分布画了一条务实的线。

这一层加了什么解决什么
OTel span每条查询可追踪、可看耗时——观测性焊进数据层
log_comment 标签每条 SQL 自带 project/面/路由,慢查询能在 CH 查询日志里按标签追责
退避重试网络抖动等可重试错误自动重试,不打扰调用方
资源错误包装内存/超时翻成友好的 ClickHouseResourceError,交第 21 课给出「缩小范围」建议
回看时间窗跨表 JOIN 只扫有界分区,把全表扫描挡在门外
legacy 守卫 + 流式变体禁读旧 events 表;大读用 Stream、SSE 仪表盘用 WithProgress
🎯 设计取舍
为什么把所有读取硬挤到一个执行器里,而不是哪儿要查哪儿写 SQL? 因为横切关注最忌散落。可观测性、追责、重试、资源保护——这些和「业务查什么」无关、 却每条查询都需要的能力,如果让一百处裸查询各自实现,必然有的写、有的漏、写法还不一致。收敛成一个执行器,它们就一次实现、处处生效: 加一项能力(比如新的查询标签维度),全平台的查询一起获得。回看时间窗则是另一种智慧:它承认第 8 课「三张独立宽表、靠时间关联」的设计代价(JOIN 没有外键可走), 并用「数据的真实时间局部性」把这个代价摊薄到几乎为零。一个总闸门管住所有共性,一条务实的时间线管住所有 JOIN——这就是仓储层的全部精髓。
🎯 本课要点
  • 所有 ClickHouse 读取都汇到一个执行器 queryClickhouse(),沿途统一套上 OTel span、查询标签、退避重试、资源错误包装。
  • 查询标签经 log_comment 进 ClickHouse 查询日志,每条 SQL 自带 project/面/路由——慢查询可按标签追责。
  • 资源错误(内存/超时)被包装成 ClickHouseResourceError,交第 21 课的错误中间件翻成「缩小时间范围」的可操作建议。
  • 回看时间窗:trace/observation/score 是三张独立宽表、靠时间关联;JOIN 时以 trace 时间戳为锚、只在 2 天(观测)/1 小时(评分)内找,避免全表扫描。
  • 取舍:横切能力收敛到一处「一次实现处处生效」;时间窗用「真实时间局部性」摊薄第 8 课分表的 JOIN 成本,以极小长尾正确性换巨大成本节省。

🧪 自测 · 想一想为什么这么设计

1. Langfuse 让所有 ClickHouse 读取都经过一个 queryClickhouse() 执行器,而不是各处直接写查询。这样做最核心的收益是?
  1. 因为 ClickHouse 只接受一个连接
  2. 让查询语法更简单
  3. 为了隐藏 SQL 不让人看
  4. 横切关注(OTel 追踪、查询标签、退避重试、资源错误包装)一次实现、处处生效;加一项能力全平台查询一起获得,不会有的写有的漏、写法不一
看答案与解析 点击展开
答案:D。可观测性、追责、重试、资源保护都和「业务查什么」无关却每条查询都需要。收敛到一个执行器,它们一次实现、处处复用;散落各处的裸查询必然写法不一、漏东漏西。这与第 6 课「把复杂度收敛到一处」一脉相承。
2. queryClickhouse 通过 log_comment 给每条 SQL 打上 project/面/路由标签。这个标签的实际用处是?
  1. 标签让查询更快
  2. 标签替代了 WHERE 条件
  3. 标签随查询进入 ClickHouse 的查询日志;当某条慢查询拖垮集群时,运维能直接在日志里按标签定位是哪个 project/面/路由的元凶——把可观测性焊进数据访问层
  4. 标签用来加密查询
看答案与解析 点击展开
答案:C。sendClickhouseQuery 把标签塞进 clickhouse_settings.log_comment,于是每条打到 CH 的 SQL 都自带「谁/哪个面/哪个路由发起」。出现慢查询或资源耗尽时,可在 CH 自己的查询日志里按标签追责——这是散落各处的裸查询永远做不到的。
3. 把一条 trace 的观测拼起来时,Langfuse 不全表扫 observations,而是只在 trace 时间戳前后一个有界窗口(如 2 天)内找。为什么能、且为什么要这样?
  1. 纯粹是随便定的限制
  2. 因为 2 天前的数据会被自动删除
  3. 因为 ClickHouse 不支持大于 2 天的查询
  4. 能:trace/observation/score 是三张独立宽表、靠时间关联,且一条 trace 的观测几乎都集中在它发生后很短时间内;要:否则跨表 JOIN 要扫几个月上亿行,把分析库当事务库用必然拖垮集群
看答案与解析 点击展开
答案:D。第 8 课让三表独立、靠时间关联,JOIN 没有外键可走。回看时间窗(OBSERVATIONS_TO_TRACE_INTERVAL=2 天、SCORE=1 小时)注入 WHERE,配合按月分区只扫几个相关分区。代价是极长尾观测可能漏(源码 README 记录约 96% 在 2 分钟内开始,长尾很长,2 天留足裕量)——用极小正确性换巨大成本节省。
💭 发散思考(没有标准答案,动手或动脑想想)
  • 「把所有数据访问收敛到一个执行器,统一加上追踪、标签、重试、资源保护」——这条原则你能用到自己的系统吗?如果你的代码里到处是裸 SQL/裸 HTTP 调用,把它们收敛到一个执行器,你最想先统一加上哪三样能力?为什么?

The last lesson ended at "the resolver calls a @langfuse/shared repository to read". This lesson opens that repository layer door and shows how it pulls data out of ClickHouse. The core is a discipline: every analytical query funnels through one executor — queryClickhouse(). It uniformly wraps each query with OTel tracing, query tags, backoff retry, and friendly resource-error wrapping. Plus a key trick that makes cross-table JOINs cheap — the look-back window. Get this layer and you get the "master valve" for all of Langfuse's reads.

🏬 Analogy
Think of ClickHouse as a vast warehouse. If every worker (each piece of query code) barges into the warehouse to rummage on their own, nobody records who took what, and when things go wrong there's no trail. Langfuse instead opens one single dispatch window (queryClickhouse): everyone hands in their slip here. The clerk logs who's collecting (query tags), stamps a tracking ticket (OTel span), and waits and retries if a shelf is briefly busy (backoff). If your slip tries to haul off the whole warehouse (out of memory), the clerk won't let the warehouse collapse — they politely hand back "narrow it down and come again" (resource error). One window handles observability, accountability, and fault tolerance all at once.

The one dispatch window: queryClickhouse

No reading code connects to ClickHouse directly; it hands the query and a set of tags to queryClickhouse(). Around the actual query, it uniformly wraps several cross-cutting concerns:

All reads funnel into one executor traces service dashboard engine public API …dozens of callers queryClickhouse() ① OTel span (every query traceable) ② log_comment tags (who/surface/route) ③ backOff retry (network blips) ④ friendly resource-error wrap ClickHouseJSONEachRow streaming variantsStream / WithProgress(SSE)
One window, four things unified: queryClickhouse() wraps each query with an OTel span, log_comment tags, backoff retry, and resource-error wrapping, then sends it to ClickHouse (big reads via queryClickhouseStream/WithProgress). Source: packages/shared/src/server/repositories/clickhouse.ts:639.
Example: a CH query the repository runs, and its rows SELECTid, name, type, start_time, total_cost, usage_details FROMobservations WHEREproject_id = {projectId}AND start_time >= {from} ORDER BYstart_time DESCLIMIT50FINAL FINAL = the merged latest idnametypestart_timetotal_costusage obs_7fanswerGENERATION12:00:01.90.0041512/88 obs_3aretrieveSPAN12:00:00.1—— obs_9cplanGENERATION11:59:58.00.0009120/40 the repository assembles these rows into the objects the UI needs
The heart of the read path: a parameterized column query (per packages/shared/src/server/repositories and 0002_observations; values illustrative): the repository always carries WHERE project_id = ? (tenant isolation), ORDER BY start_time, LIMIT, and adds FINAL to read the merged-latest rows from ReplacingMergeTree. The returned rows are then assembled into the shape the list/detail UI needs.
packages/shared/src/server/repositories/clickhouse.tsqueryClickhouse :639
export async function queryClickhouse<T>(opts) {
  assertNoLegacyEventsRead(opts.query);          // guard: don't read legacy events tables
  return await instrumentAsync({ name: "clickhouse-query" }, async (span) => {
    setSpanQueryAttributes(span, opts.query);        // ① attach SQL to the OTel span
    return await backOff(                          // ③ retry retryable (network) errors
      () => sendClickhouseQuery({ ...opts, span }),     // ② sets log_comment=JSON(tags) inside
      { numOfAttempts, retry: isRetryableError },
    ).catch((e) => { throw ClickHouseResourceError.wrapIfResourceError(e, tags); });
  });                                                // ④ memory/timeout → friendly error (caught by L21)
}

Of these four, the query tags deserve a special mention. sendClickhouseQuery stuffs the tags into ClickHouse's log_comment setting (the query log's comment field): so every SQL hitting ClickHouse carries "which project, which surface (UI/public API/worker), which route initiated it". When one slow query drags down the cluster, operators can locate the culprit by tag directly in ClickHouse's own query log — a way to weld observability into the data-access layer that scattered raw queries can never achieve.

The same dispatch window also offers several pickup modes for different read scenarios — but they share the same tracing/tags/retry machinery:

one window, three pickup modes — sharing OTel span · log_comment tags · backoff retry queryClickhouse fetch all at once (into an array) queryClickhouseStream stream as it reads (bounded memory) queryClickhouseWithProgress a progress-pushing stream lists / detail / aggregations (fits memory) batch exports (result may be huge) SSE dashboards (push progress live)
queryClickhousefetch all at once
The common case: run the query, read the whole result into an array, return. Most lists, details, aggregations go through it.
queryClickhouseStreamstream as it reads
Big reads needn't buffer the whole result in memory; they stream row by row — batch exports (the exports mentioned in L12) and other "result might be huge" cases use it, with bounded memory.
queryClickhouseWithProgressstreaming with progress
For SSE streaming dashboards: pushes progress to the frontend as it queries, so big dashboard queries "give feedback, don't freeze".

Look-back windows: cross-table JOINs without full scans

Remember Lesson 8 — trace, observation, score are three independent wide tables, with no foreign keys, correlated by time. The problem: to stitch a trace's observations and scores together, must we full-scan an observations table of months and billions of rows? Unacceptably slow. The repository layer's answer is the look-back window: exploiting the fact that "a trace's observations almost all cluster in a short window after it happened", it searches only within a bounded time range.

Anchored at the trace timestamp, only a bounded look-back window time → trace.timestamp (anchor) obs window: start_time ≥ anchor − 2 days score window: 1 hour outside: not scanned(months of old partitions skipped) outside: not scanned OBSERVATIONS_TO_TRACE_INTERVAL=2 days · SCORE_TO_TRACE_OBSERVATIONS_INTERVAL=1 hour
Bounded look-back, skip old partitions: anchored at the trace's timestamp, observations are found within start_time ≥ anchor − 2 days, scores within 1 hour. These constants are injected into WHERE clauses, with Lesson 8's monthly partitioning letting JOINs touch only a few partitions. Source: repositories/constants.ts:4,9.

😖 no window: full-table JOIN

To find a trace's observations, search the whole observations table (months, billions of rows). Every list query doing this would soon drag the cluster down — the classic crash of using an analytical store like a transactional one.

😀 look-back window: bounded scan

Inject start_time ≥ trace.timestamp − 2 days into WHERE, and with monthly partitioning the JOIN reads only those few relevant partitions. The scan shrinks from "months" to "two days" — queries fly.

The window has a cost, of course: a rare unusually long observation (ending more than 2 days late) could fall outside the window and be missed. This is a deliberate trade-off — trading "letting go of a tiny long tail of correctness" for "JOINs no longer scanning the whole table". Per a documented distribution (~96% of observations start within 2 minutes of the trace per repositories/README.md, with a very long tail), Langfuse sets the window to 2 days, leaving a huge safety margin, so it almost never truly misses. Between correctness and cost, engineers drew a pragmatic line by the real data distribution.

what this layer addswhat it solves
OTel spanevery query traceable, its latency visible — observability welded into the data layer
log_comment tagsevery SQL carries project/surface/route, so slow queries can be traced by tag in CH's query log
backoff retryretryable errors (network blips) auto-retry without bothering the caller
resource-error wrappingmemory/timeout become a friendly ClickHouseResourceError, handed to L21 for "narrow it" advice
look-back windowcross-table JOINs scan only bounded partitions, keeping full scans out
legacy guard + streaming variantsforbids reading legacy events; big reads use Stream, SSE dashboards use WithProgress
🎯 Design trade-off
Why cram all reads into one executor, instead of writing SQL wherever you need it? Because cross-cutting concerns most dread being scattered. Observability, accountability, retry, resource protection — capabilities unrelated to "what business queries" yet needed by every query — if a hundred raw queries each implement them, some will write them, some omit them, and inconsistently. Converge into one executor and they're implemented once, in effect everywhere: add a capability (say a new query-tag dimension) and every query platform-wide gains it together. The look-back window is another kind of wisdom: it accepts the cost of Lesson 8's "three independent wide tables, correlated by time" design (JOINs have no foreign key to follow), and uses "the data's real temporal locality" to amortize that cost to near zero. One master valve for all commonalities, one pragmatic timeline for all JOINs — that's the whole essence of the repository layer.
🎯 Key points
  • All ClickHouse reads funnel into one executor queryClickhouse(), uniformly wrapped with OTel span, query tags, backoff retry, resource-error wrapping.
  • Query tags go via log_comment into ClickHouse's query log, so every SQL carries project/surface/route — slow queries traceable by tag.
  • Resource errors (memory/timeout) are wrapped into ClickHouseResourceError, handed to L21's error middleware for "narrow the time range" advice.
  • Look-back windows: trace/observation/score are three independent wide tables correlated by time; JOINs anchor at the trace timestamp and search only within 2 days (obs) / 1 hour (scores), avoiding full scans.
  • Tradeoff: cross-cutting capabilities converge to one place "implemented once, everywhere"; the time window uses "real temporal locality" to amortize Lesson 8's split-table JOIN cost, trading a tiny long-tail of correctness for huge cost savings.

🧪 Self-test - think about the design

1. Langfuse routes all ClickHouse reads through one queryClickhouse() executor rather than writing queries everywhere. The core benefit?
  1. because ClickHouse accepts only one connection
  2. to make query syntax simpler
  3. to hide the SQL from view
  4. cross-cutting concerns (OTel tracing, query tags, backoff retry, resource-error wrapping) are implemented once and apply everywhere; add a capability and every query gains it, with no some-write-some-omit inconsistency
Show answer & explanation click to expand
Answer: D. Observability, accountability, retry, resource protection are unrelated to 'what business queries' yet needed by every query. Converged into one executor they're implemented once and reused; scattered raw queries inevitably drift and omit. In line with L06's 'converge complexity to one place'.
2. queryClickhouse tags every SQL via log_comment with project/surface/route. What's the practical use?
  1. the tags speed up the query
  2. the tags replace the WHERE clause
  3. the tags enter ClickHouse's query log with the query; when a slow query drags the cluster down, operators can locate the culprit project/surface/route by tag right in the log — observability welded into the data-access layer
  4. the tags encrypt the query
Show answer & explanation click to expand
Answer: C. sendClickhouseQuery stuffs tags into clickhouse_settings.log_comment, so every SQL hitting CH carries 'who/which surface/which route initiated it'. On a slow query or resource exhaustion, you can trace the culprit by tag in CH's own query log — impossible for scattered raw queries.
3. To stitch a trace's observations, Langfuse doesn't full-scan observations but searches only a bounded window (e.g. 2 days) around the trace timestamp. Why can it, and why must it?
  1. an arbitrary limit
  2. because data older than 2 days is auto-deleted
  3. because ClickHouse can't query beyond 2 days
  4. can: trace/observation/score are three independent wide tables correlated by time, and a trace's observations almost all cluster shortly after it; must: otherwise the cross-table JOIN scans months and billions of rows, crashing the cluster by using an analytical store like a transactional one
Show answer & explanation click to expand
Answer: D. L08 made the three tables independent and time-correlated, so JOINs have no FK to follow. Look-back windows (OBSERVATIONS_TO_TRACE_INTERVAL=2 days, SCORE=1 hour) injected into WHERE, with monthly partitioning, scan only a few relevant partitions. The cost is missing extreme-long-tail observations (~96% start within 2 min per the repo README, with a long tail; 2 days leaves margin) — a tiny correctness trade for huge cost savings.
💭 Open questions (no single right answer - just think or try)
  • 'Converge all data access into one executor, uniformly adding tracing, tags, retry, resource protection' — can you apply this to your systems? If your code is full of raw SQL/raw HTTP calls, converging them into one executor, which three capabilities would you add first, and why?