🪢 Langfuse 图解教程Langfuse Visual Guide 第八部分 · 仪表盘·指标·成本Part 8 · Dashboards, Metrics & Cost 41 / 55
第八部分 · 仪表盘·指标·成本Part 8 · Dashboards, Metrics & Cost

查询引擎The query engine

🧭 读前Before you start
先读 第8课 的 CH 表与 第22课 的仓储层,本课讲通用查询引擎。
Read Lesson 8 (the CH tables) and Lesson 22 (the repository layer); this lesson covers the general query engine.

上一课的 widget 声明了 view + dimensions + metrics + filters。但 ClickHouse 只懂 SQL,根本不认识这些声明。中间那座桥,就是查询引擎(query engine)——它把这套声明编译成 SQL,跑出数据。这一课拆开它,看清两个核心部件:一个语义层(data model,把「成本」「按模型」这样的友好逻辑名,映射到真实的 ClickHouse 列与表达式),和一个编译器(QueryBuilder.build(),把声明拼装成参数化的 SQL)。 理解了它,你就明白第 40 课仪表盘、第 33 课监控、第 35 课实验聚合为何能共用一份查询形状——因为它们背后都是这同一台编译器;也会再次见到第 23 课「参数化防注入」的纪律,在聚合查询里如何贯彻。

📋 生活类比
查询引擎像餐厅里的「翻译官 + 一本菜单」。你(widget)用人话点菜:「按模型分组、统计总成本,只看 production 的」。翻译官(QueryBuilder)翻开一本菜单(data model)——菜单上每道「逻辑菜名」都标注着后厨用的真实食材和做法(「总成本」=后厨表里的 sum(total_cost) 列、「按模型」=按 provided_model_name 分组)。 翻译官照着菜单,把你的人话精确翻译成后厨(ClickHouse)能执行的工单(SQL)。妙处有二:其一,你永远不用知道后厨的食材叫什么、放在哪个冷柜——菜单这层把你和后厨的物理细节隔开了,后厨改了食材摆放(换列名、改表结构),只要菜单跟着更新,你的点法一字不用改。其二,凡是你填的「具体值」(如 production),翻译官都另用一张小纸条夹着递进去(参数化),绝不直接写进工单正文——这样哪怕你写个奇怪的店名,也绝串改不了工单本身(防 SQL 注入,第 23 课)。

语义层:把逻辑名翻译成真实 SQL

引擎的第一个部件是 data model(数据模型 / 语义层)。它为每个 view(traces / observations / scores-numeric / scores-categorical)声明一份「字典」:这个视图有哪些 dimensions(维度,可分组的字段)、哪些 measures(度量,可聚合的量),以及它们各自对应哪一段真实的 ClickHouse SQL。比如维度 name 对应 traces.name、时间维度对应一段 formatDateTime(...)。

data model:逻辑名 ↔ 真实 SQL 的字典 你说的(逻辑名)view: tracesdimension: namemetric: count友好、稳定、与物理解耦 data model(字典)每个逻辑名 →{ sql, alias }还声明可分组维度、可聚合度量支持 v1/v2 版本 后厨懂的(真实 SQL)FROM traces …traces.namecount(*)物理列名、可演化 语义层把「你说的」和「后厨懂的」隔开:物理列改了名,只动字典,所有 widget/监控/API 的查询照常
语义层 = 一本逻辑名↔SQL 的字典:dataModel.ts 为每个 view 声明 dimensions 与 measures,每项形如 { sql: "traces.name", alias: "name" }(维度),时间维度甚至是 formatDateTime(traces.timestamp, …) 这样的表达式。源码:packages/shared/src/features/query/dataModel.ts:10-70、types.ts:9(viewDeclaration)。
示例:查询规格 → 编译出的 ClickHouse SQL query spec (JSON) · 仪表盘提交 { view: "observations", metrics: [ { measure:"count", aggregation:"count" }, { measure:"totalCost", aggregation:"sum" } ], dimensions: [{ field:"providedModelName" }], timeDimension: { granularity:"day" }, filters: [{ column:"environment", operator:"=", value:"production" }], fromTimestamp:"2026-06-01…", toTimestamp:"2026-06-30…" } queryBuilder compiled ClickHouse SQL SELECT provided_model_name, count() AS count, sum(total_cost) AS sum_totalCost, toStartOfDay(start_time) AS day FROM observations FINAL WHERE project_id = {projectId} AND environment = 'production' AND start_time >= {from} AND start_time < {to} GROUP BY provided_model_name, day ORDER BY day 参数化占位符({projectId}/{from}/{to})杜绝 SQL 注入
一份声明式规格,引擎替你写 SQL(规格形状取自 features/query/types.ts,列名取自 observations 迁移;值为示例):你只描述要什么——view 选表、metrics 是 {measure, aggregation}、dimensions 是 {field}、timeDimension.granularity 定时间桶;queryBuilder 把它编译成真正的 ClickHouse SQL:度量变 count()/sum(total_cost)、维度进 GROUP BY、时间桶变 toStartOfDay(start_time)。所有用户值都走参数化占位符,从根上挡住 SQL 注入。换个 view 就查另一张表——同一套规格语言覆盖全部图表。
packages/shared/src/features/query/dataModel.ts一个 view 的声明
// 每个 view 声明:有哪些维度/度量,各自对应哪段真实 SQL
{
  name: "traces",
  dimensions: {
    name:        { sql: "traces.name",      alias: "name" },        // 逻辑名 → 物理列
    environment: { sql: "traces.environment", alias: "environment" },
    timestampMonth: { sql: "formatDateTime(traces.timestamp, '%Y-%m')", … },  // 可是表达式
  },
  measures: { count: { … }, totalCost: { sql: "sum(total_cost)", … }, … },   // 可聚合的量
}
// widget 只引用 "name"/"totalCost" 这些逻辑名,永不直接碰物理列
view(数据源)聚合的对象典型维度 / 度量
traces整条 trace维度:name、environment、userId;度量:count、总成本
observations单步观测(含 generation)维度:model、type;度量:count、token 数、成本、延迟
scores-numeric数值分维度:name、source;度量:avg(value)、count
scores-categorical分类分维度:name、stringValue;度量:count、各类占比

四个 view 各对应一类核心数据(呼应第 8、13、28 课的领域模型)。一个 widget 选定一个 view,再从它声明好的维度/度量里挑要分组、要算的——能问什么,由 view 的字典划定边界,既好用又防越界。

编译器:把声明拼成参数化 SQL

第二个部件是 QueryBuilder.build()——真正的编译器。它拿到一个声明式的 QueryType,对照 data model,分几步把它拼成完整 SQL:维度翻译成 SELECT … GROUP BY 的列(mapDimensions)、度量翻译成 count()/sum()/avg() 等聚合(mapMetrics + translateAggregation)、时间按粒度分桶(applyBucketingDimension,做时序图)、过滤翻译成 WHERE,必要时连表(buildJoins)。

QueryBuilder.build():声明 → 参数化 SQL QueryType(声明)viewdimensionsmetricsfilterstimeDimension QueryBuilder.build()mapDimensions → GROUP BYmapMetrics → count/sum/avgapplyBucketing → 时间分桶mapFilters → 参数化 WHEREbuildJoins → 必要时连表对照 data model 翻译每一项 SQL(带 {占位符})SELECT … GROUP BY …WHERE env = {p1} parameters{ p1: "production" } queryExecutor把 SQL+参数发给 ClickHouse 关键:用户填的具体值(production)永远走 parameters,不拼进 SQL 正文——参数化防注入(第23课) SQL 与值分离:模板可缓存/可审计,值随调用绑定,恶意输入串改不了结构
编译器对照字典逐项翻译:queryBuilder.ts 的 build() 串起 mapDimensions(:270) / mapMetrics+translateAggregation(:290,137) / applyBucketingDimension(:182) / mapFilters+buildWhereClause(:490,858)。过滤用 FilterList 的 apply() 产出 SQL+参数——用户值绑成参数,绝不拼进正文(第 23 课同款防注入)。
packages/shared/src/features/query/server/queryBuilder.ts · queryExecutor.ts声明 → SQL → 执行
// 1) 声明式查询(来自 widget / monitor / 实验聚合,都是这个形状)
const query = { view: "observations", dimensions: [{ field: "model" }],
  metrics: [{ measure: "totalCost", aggregation: "sum" }],
  filters: [{ column: "environment", operator: "=", value: "production" }] };

// 2) 编译:对照 data model,拼成参数化 SQL(GROUP BY/聚合/WHERE/分桶/JOIN)
const { sql, parameters } = new QueryBuilder(chartConfig, version).build(query, projectId);
// sql:  SELECT model, sum(total_cost) … WHERE environment = {p_env} GROUP BY model
// parameters: { p_env: "production" }   ← 值走参数,不拼进 SQL

// 3) 执行:queryExecutor 把 sql+parameters 发给 ClickHouse
1

取 view 字典

getViewDeclaration(view, version) 按版本拿到该 view 的维度/度量字典——后续每一步都对照它翻译。

2

维度 → GROUP BY

mapDimensions 把每个逻辑维度译成 SELECT 列和 GROUP BY;时间维度交给 applyBucketingDimension 按粒度分桶。

3

度量 → 聚合

mapMetrics + translateAggregation 把度量译成 count()/sum()/avg() 等聚合函数。

4

过滤 → 参数化 WHERE

mapFilters + buildWhereClause 用 FilterList.apply 产出 WHERE,用户值绑成 parameters。

5

连表 + 出 SQL

需要跨表时 buildJoins 补 JOIN;最终拼出 { sql, parameters },交 queryExecutor 跑 ClickHouse。

为什么值得造这台引擎

把声明编译成 SQL,本可以让每个 widget 自己拼字符串。但 Langfuse 偏要造一台统一引擎,因为它一次解决了四个问题:

① 上层声明(声明式 · 与物理无关 · 注入安全) view + dimensions + metrics + filters ② 查询引擎 = 指标语言的编译器 语义层:逻辑名 ↔ SQL(解耦) 参数化:值走 parameters(防注入) 视图版本化:v1/v2(不破历史) 一引擎多消费(口径一致) ③ 参数化 SQL → ClickHouse 执行 一引擎多消费 仪表盘 L40 · 监控 L33 · 实验聚合 → 同一指标永远同一个数
设计解决什么呼应
语义层(逻辑名↔SQL)物理列改名/换表,只改字典,所有查询不动;用户也无需懂物理 schema解耦「想要什么」与「怎么存」
参数化(值走 parameters)用户值绝不拼进 SQL 正文,根除 SQL 注入第 23 课 FilterState 同款
视图版本化(v1/v2)物理 schema 演进时,老查询仍按老视图编译,不破坏历史第 34/37 课「不可变 + 演进」
一引擎多消费仪表盘/监控/实验聚合共用,指标口径全局一致第 40 课「一份查询形状三处复用」

把这四点连起来看,查询引擎其实是一个小型「指标语言 + 编译器」:上层用一种声明式、与物理无关、注入安全的小语言描述「我要什么指标」,引擎负责把它正确、一致、安全地落成 ClickHouse 能跑的 SQL。这和第 23 课把 FilterState 编译成过滤 SQL 是同一种思路,只是从「过滤」升级到了完整的「聚合查询」。声明在上、编译在下,物理细节被牢牢封在引擎里——这正是 Langfuse 能让仪表盘、监控、API 对同一指标永远算出同一个数的底层原因。

🎯 设计取舍
为什么要造一个语义层,而不让 widget 直接写 SQL? 直接写 SQL 看似省事,实则埋三颗雷。其一是耦合:每个 widget 都硬编码物理列名,哪天 ClickHouse 改了表结构、换了列名,成百上千张 widget 一起崩,得逐个改。语义层把「逻辑名→物理 SQL」收进一本字典,物理一变只改字典,上层岿然不动。其二是安全:让每个 widget 自己拼 SQL,就等于把防注入的责任摊给每个写 widget 的人——总有人会忘、会拼错,一个口子就是一次数据泄露。统一引擎把参数化焊死在唯一出口,注入安全靠结构兜底,不靠自觉。其三是一致:散落各处的 SQL 迟早对同一指标算出不同数。把指标定义收进语义层,全公司一种算法。一台引擎,同时买下解耦、安全、一致三件大事——这就是为什么值得为它多写这几千行。

为什么 view 要带 v1/v2 版本? 因为物理 schema 会演进,但历史查询不该因此失效。ClickHouse 表会重构、列会拆分合并、为性能引入新的物化视图——这些都是底层的正常演化。如果查询直接绑死当前物理结构,那每次底层一动,所有保存好的 widget、monitor、API 查询都可能算错或报错。给 view 引入版本,等于让「语义」也有了不可变的快照:一个 v1 的查询永远按 v1 的字典编译,哪怕底层早已迁到 v2 的新表。这和第 34 课数据项版本、第 37 课 prompt 版本是同一种信念在查询层的回响——底层可以大胆演进,因为上层被一层稳定的版本化语义保护着。
🎯 本课要点
  • 查询引擎 = 声明式查询 → SQL 的编译器:widget 的 view+dimensions+metrics+filters 声明,由它编译成 ClickHouse SQL 并执行,是第 40 课仪表盘背后的发动机。
  • 语义层(data model):为每个 view 声明 dimensions/measures,每项把逻辑名映射到真实 SQL(如 name→traces.name、时间→formatDateTime(...))——上层永不直接碰物理列。
  • 编译器(QueryBuilder.build):对照字典把维度译成 GROUP BY、度量译成聚合、时间分桶、过滤译成 WHERE、必要时连表,产出 SQL + 参数。
  • 参数化防注入:用户填的值绑成 parameters、绝不拼进 SQL 正文——第 23 课 FilterState 的纪律在聚合查询里同样贯彻,注入安全靠结构兜底。
  • 一引擎四收益:语义层解耦物理、参数化保安全、视图 v1/v2 版本化护历史、一引擎多消费保口径一致——仪表盘/监控/实验对同一指标永远同一个数。

🧪 自测 · 想一想为什么这么设计

1. Langfuse 的查询引擎有一个「语义层」(data model)。它的核心作用是什么?
  1. 缓存查询结果
  2. 压缩 ClickHouse 数据
  3. 把友好的逻辑名(如维度 name、度量 totalCost)映射到真实的 ClickHouse 列/表达式(traces.name、sum(total_cost)),让上层永不直接碰物理列——物理 schema 改名换表只需改字典
  4. 把 SQL 翻译成自然语言
看答案与解析 点击展开
答案:C。dataModel.ts 为每个 view 声明 dimensions/measures,每项形如 { sql: "traces.name", alias: "name" },把逻辑名映射到真实 SQL(甚至是 formatDateTime 这样的表达式)。widget 只引用逻辑名,永不碰物理列。于是物理列改名/换表只需改字典,成百上千张 widget 不动——这是「想要什么」与「怎么存」的解耦。
2. QueryBuilder.build() 把过滤条件编译进 SQL 时,用户填的具体值(如 environment="production")是怎么处理的?
  1. 存进 Redis 再引用
  2. 绑成 parameters、绝不拼进 SQL 正文(SQL 里只放 {占位符},值随调用绑定)——和第23课 FilterState 同款,从结构上根除 SQL 注入
  3. 直接字符串拼接进 SQL
  4. 先做 HTML 转义再拼接
看答案与解析 点击展开
答案:B。mapFilters/buildWhereClause 用 FilterList 的 apply() 产出 SQL+参数:SQL 模板里是 {占位符},用户值放进 parameters 字典,由 ClickHouse 驱动安全绑定。SQL 与值分离,模板可缓存/审计,恶意输入串改不了结构。这正是第23课把 FilterState 编译成过滤 SQL 的同一种防注入纪律,在聚合查询里同样贯彻——注入安全靠结构兜底,不靠每个开发者自觉。
3. Langfuse 的 view 带有 v1/v2 版本。为什么查询引擎要给「视图」也引入版本?
  1. v2 比 v1 查询更快
  2. 为了支持多个 ClickHouse 实例
  3. 版本号用于计费
  4. 因为物理 schema 会演进(表重构、列拆分、加物化视图),但历史查询不该因此失效;一个 v1 查询永远按 v1 字典编译,底层迁到 v2 也不破坏——和第34/37课「不可变+演进」同一信念
看答案与解析 点击展开
答案:D。ClickHouse 表会重构、列会拆分合并、为性能加物化视图——底层正常演化。若查询绑死当前物理结构,每次底层一动,所有保存的 widget/monitor/API 查询都可能算错或报错。给 view 引入 v1/v2,等于给「语义」也加了不可变快照:v1 查询永远按 v1 编译。这和第34课数据项版本、第37课 prompt 版本同一信念——底层敢大胆演进,因为上层被稳定的版本化语义保护。
💭 发散思考(没有标准答案,动手或动脑想想)
  • 查询引擎本质是一个「语义层 + 编译器」——这模式在业界很常见(Cube、LookML、Malloy 等)。它用「声明式 + 物理无关 + 注入安全」换取了什么?又付出了什么代价(引擎得支持所有需要的查询形状,逃逸到原生 SQL 受限)?如果你的团队要做内部分析平台,你会选「人人写 SQL」还是「统一语义层」,为什么?两者能并存吗?

The last lesson's widget declared view + dimensions + metrics + filters. But ClickHouse only understands SQL—it knows nothing of these declarations. The bridge between them is the query engine—it compiles that declaration into SQL and runs it. This lesson takes it apart, revealing two core parts: a semantic layer (the data model, mapping friendly logical names like "cost" and "by model" to real ClickHouse columns and expressions), and a compiler (QueryBuilder.build(), assembling the declaration into parameterized SQL). Understanding it explains why Lesson 40's dashboards, Lesson 33's monitors, and Lesson 35's experiment aggregation can share one query shape—because they're all backed by this same compiler; you'll also see again how Lesson 23's "parameterized = injection-safe" discipline carries into aggregation queries.

📋 Analogy
The query engine is like a restaurant's "interpreter + a menu". You (the widget) order in plain words: "group by model, total the cost, only production". The interpreter (QueryBuilder) opens a menu (the data model)—where each "logical dish name" is annotated with the real ingredients and methods the kitchen uses ("total cost" = the kitchen table's sum(total_cost) column, "by model" = group by provided_model_name). Following the menu, the interpreter precisely translates your plain words into a work ticket the kitchen (ClickHouse) can execute (SQL). Two beauties: one, you never need to know what the kitchen's ingredients are called or which fridge they're in—this menu layer isolates you from the kitchen's physical details; if the kitchen rearranges ingredients (renames columns, restructures tables), as long as the menu updates, your ordering style doesn't change one word. Two, any "concrete value" you fill in (like production) the interpreter passes in on a separate slip (parameterized), never written into the ticket body—so even a weird store name can't alter the ticket itself (SQL injection prevention, Lesson 23).

The semantic layer: translate logical names into real SQL

The engine's first part is the data model (semantic layer). For each view (traces / observations / scores-numeric / scores-categorical) it declares a "dictionary": which dimensions (groupable fields) and measures (aggregatable quantities) this view has, and which real ClickHouse SQL each maps to. For example, the dimension name maps to traces.name, and the time dimension to a formatDateTime(...) expression.

data model: a dictionary of logical name ↔ real SQL what you say (logical)view: tracesdimension: namemetric: countfriendly, stable, physical-decoupled data model (dictionary)each logical name →{ sql, alias }declares groupable dims, aggregatable measuressupports v1/v2 versions what the kitchen knows (real SQL)FROM traces …traces.namecount(*)physical column names, evolvable the semantic layer isolates "what you say" from "what the kitchen knows": rename a physical column, only the dictionary changes, all widget/monitor/API queries go on
semantic layer = a dictionary of logical name↔SQL: dataModel.ts declares dimensions and measures per view, each shaped like { sql: "traces.name", alias: "name" } (a dimension); a time dimension can even be an expression like formatDateTime(traces.timestamp, …). Source: packages/shared/src/features/query/dataModel.ts:10-70, types.ts:9 (viewDeclaration).
Example: a query spec → compiled ClickHouse SQL query spec (JSON) · dashboard submits { view: "observations", metrics: [ { measure:"count", aggregation:"count" }, { measure:"totalCost", aggregation:"sum" } ], dimensions: [{ field:"providedModelName" }], timeDimension: { granularity:"day" }, filters: [{ column:"environment", operator:"=", value:"production" }], fromTimestamp:"2026-06-01…", toTimestamp:"2026-06-30…" } queryBuilder compiled ClickHouse SQL SELECT provided_model_name, count() AS count, sum(total_cost) AS sum_totalCost, toStartOfDay(start_time) AS day FROM observations FINAL WHERE project_id = {projectId} AND environment = 'production' AND start_time >= {from} AND start_time < {to} GROUP BY provided_model_name, day ORDER BY day parameterized placeholders ({projectId}/{from}/{to}) prevent SQL injection
One declarative spec, the engine writes the SQL (spec shape from features/query/types.ts, column names from the observations migration; values illustrative): you only describe what you want — view picks the table, metrics are {measure, aggregation}, dimensions are {field}, timeDimension.granularity sets the time bucket; queryBuilder compiles it into real ClickHouse SQL: measures become count()/sum(total_cost), dimensions go into GROUP BY, the bucket becomes toStartOfDay(start_time). Every user value flows through parameterized placeholders, blocking SQL injection at the root. Swap the view and you query a different table — one spec language covers every chart.
packages/shared/src/features/query/dataModel.tsone view's declaration
// each view declares: which dimensions/measures, each mapping to some real SQL
{
  name: "traces",
  dimensions: {
    name:        { sql: "traces.name",      alias: "name" },        // logical name → physical column
    environment: { sql: "traces.environment", alias: "environment" },
    timestampMonth: { sql: "formatDateTime(traces.timestamp, '%Y-%m')", … },  // can be an expression
  },
  measures: { count: { … }, totalCost: { sql: "sum(total_cost)", … }, … },   // aggregatable quantities
}
// a widget only references logical names like "name"/"totalCost", never touches physical columns
view (data source)what it aggregatestypical dimensions / measures
traceswhole tracesdims: name, environment, userId; measures: count, total cost
observationssingle observations (incl. generations)dims: model, type; measures: count, tokens, cost, latency
scores-numericnumeric scoresdims: name, source; measures: avg(value), count
scores-categoricalcategorical scoresdims: name, stringValue; measures: count, share per category

The four views each correspond to a core data type (echoing Lessons 8, 13, 28's domain model). A widget picks a view, then chooses dimensions/measures to group and compute from those it declares—what you can ask is bounded by the view's dictionary, both usable and guarded against over-reach.

The compiler: assemble the declaration into parameterized SQL

The second part is QueryBuilder.build()—the real compiler. It takes a declarative QueryType, consults the data model, and assembles full SQL in steps: dimensions become SELECT … GROUP BY columns (mapDimensions); measures become count()/sum()/avg() aggregations (mapMetrics + translateAggregation); time is bucketed by granularity (applyBucketingDimension, for time-series charts); filters become WHERE; and tables are joined when needed (buildJoins).

QueryBuilder.build(): declaration → parameterized SQL QueryType (declaration)viewdimensionsmetricsfilterstimeDimension QueryBuilder.build()mapDimensions → GROUP BYmapMetrics → count/sum/avgapplyBucketing → time bucketsmapFilters → parameterized WHEREbuildJoins → join when neededtranslate each item via the data model SQL (with {placeholders})SELECT … GROUP BY …WHERE env = {p1} parameters{ p1: "production" } queryExecutorsends SQL+params to ClickHouse key: user-filled concrete values (production) always go via parameters, never into the SQL body—parameterized = injection-safe (Lesson 23) SQL separate from values: the template is cacheable/auditable, values bound per call, malicious input can't alter the structure
the compiler translates item by item via the dictionary: queryBuilder.ts's build() chains mapDimensions(:270) / mapMetrics+translateAggregation(:290,137) / applyBucketingDimension(:182) / mapFilters+buildWhereClause(:490,858). Filters use FilterList's apply() to produce SQL+parameters—user values are bound as parameters, never spliced into the body (same injection prevention as Lesson 23).
packages/shared/src/features/query/server/queryBuilder.ts · queryExecutor.tsdeclaration → SQL → execute
// 1) a declarative query (from widget / monitor / experiment aggregation, all this shape)
const query = { view: "observations", dimensions: [{ field: "model" }],
  metrics: [{ measure: "totalCost", aggregation: "sum" }],
  filters: [{ column: "environment", operator: "=", value: "production" }] };

// 2) compile: consult the data model, assemble parameterized SQL (GROUP BY/aggregate/WHERE/bucket/JOIN)
const { sql, parameters } = new QueryBuilder(chartConfig, version).build(query, projectId);
// sql:  SELECT model, sum(total_cost) … WHERE environment = {p_env} GROUP BY model
// parameters: { p_env: "production" }   ← values go via parameters, not spliced into SQL

// 3) execute: queryExecutor sends sql+parameters to ClickHouse
1

fetch the view dictionary

getViewDeclaration(view, version) gets that view's dimension/measure dictionary by version—every later step translates against it.

2

dimensions → GROUP BY

mapDimensions translates each logical dimension into SELECT columns and GROUP BY; the time dimension goes to applyBucketingDimension for granularity bucketing.

3

measures → aggregations

mapMetrics + translateAggregation turn measures into count()/sum()/avg() aggregation functions.

4

filters → parameterized WHERE

mapFilters + buildWhereClause use FilterList.apply to produce WHERE, binding user values as parameters.

5

join + emit SQL

When crossing tables, buildJoins adds JOINs; finally it assembles { sql, parameters }, handed to queryExecutor to run on ClickHouse.

Why it's worth building this engine

Compiling a declaration to SQL could let each widget assemble strings itself. But Langfuse insists on one unified engine, because it solves four problems at once:

1 declaration (declarative · physical-agnostic · safe) view + dimensions + metrics + filters 2 query engine = a metric-language compiler semantic layer: name ↔ SQL parameterized (no injection) view versioning: v1/v2 one engine, many consumers 3 parameterized SQL → ClickHouse one engine, many consumers dashboards L40 · monitors L33 · experiment aggregation → same metric, same number
designwhat it solvesechoes
semantic layer (logical name↔SQL)rename/restructure physical columns—only the dictionary changes, all queries untouched; users needn't know the physical schemadecouple "what you want" from "how it's stored"
parameterization (values via parameters)user values never spliced into the SQL body, rooting out SQL injectionsame as Lesson 23's FilterState
view versioning (v1/v2)as the physical schema evolves, old queries still compile against the old view, not breaking historyLessons 34/37 "immutable + evolving"
one engine, many consumersdashboards/monitors/experiment-aggregation share it, keeping metric definitions globally consistentLesson 40 "one query shape, three reuses"

Connect these four and the query engine is really a small "metrics language + compiler": the upper layers describe "which metric I want" in a declarative, physical-agnostic, injection-safe little language, and the engine renders it correctly, consistently, safely into SQL ClickHouse can run. It's the same idea as Lesson 23 compiling FilterState into filter SQL, just upgraded from "filtering" to full "aggregation queries". Declaration on top, compilation below, physical details sealed firmly inside the engine—the underlying reason Langfuse can make dashboards, monitors, and the API always compute the same number for the same metric.

🎯 Design trade-off
Why build a semantic layer instead of letting widgets write SQL directly? Direct SQL seems easier but plants three landmines. One is coupling: every widget hardcodes physical column names, so the day ClickHouse restructures a table or renames a column, hundreds of widgets break together and must be fixed one by one. The semantic layer collects "logical name → physical SQL" into one dictionary, so a physical change touches only the dictionary, the upper layer unmoved. Two is safety: letting each widget assemble SQL spreads injection-prevention responsibility to everyone who writes a widget—someone will forget or get it wrong, and one hole is one data leak. A unified engine welds parameterization into its single exit, so injection safety is backstopped by structure, not diligence. Three is consistency: SQL scattered everywhere eventually computes different numbers for the same metric. Collecting metric definitions into the semantic layer gives one algorithm company-wide. One engine buys decoupling, safety, and consistency all at once—that's why it's worth the extra few thousand lines.

Why do views carry v1/v2 versions? Because the physical schema evolves, but historical queries shouldn't break for it. ClickHouse tables get refactored, columns split and merge, new materialized views appear for performance—all normal evolution of the underlying. If a query binds tightly to the current physical structure, then every underlying move risks miscomputing or erroring every saved widget, monitor, and API query. Introducing versions to a view gives "semantics" an immutable snapshot too: a v1 query forever compiles against the v1 dictionary, even if the underlying has long migrated to v2's new tables. This is the same belief echoing at the query layer as Lesson 34's dataset-item versions and Lesson 37's prompt versions—the underlying can evolve boldly because the upper layer is protected by a stable layer of versioned semantics.
🎯 Key points
  • The query engine = a declarative-query → SQL compiler: a widget's view+dimensions+metrics+filters declaration is compiled to ClickHouse SQL and executed by it—the engine behind Lesson 40's dashboards.
  • Semantic layer (data model): declares dimensions/measures per view, each mapping a logical name to real SQL (e.g. name→traces.name, time→formatDateTime(...))—the upper layer never touches physical columns.
  • Compiler (QueryBuilder.build): consults the dictionary to translate dimensions into GROUP BY, measures into aggregations, time into buckets, filters into WHERE, joining tables when needed, producing SQL + parameters.
  • Parameterized = injection-safe: user-filled values are bound as parameters, never spliced into the SQL body—Lesson 23's FilterState discipline carried into aggregation queries, injection safety backstopped by structure.
  • One engine, four payoffs: the semantic layer decouples physical, parameterization ensures safety, view v1/v2 versioning protects history, one engine for many consumers keeps definitions consistent—dashboards/monitors/experiments always the same number for the same metric.

🧪 Self-test - think about the design

1. Langfuse's query engine has a 'semantic layer' (data model). What is its core role?
  1. caching query results
  2. compressing ClickHouse data
  3. mapping friendly logical names (e.g. dimension name, measure totalCost) to real ClickHouse columns/expressions (traces.name, sum(total_cost)), so the upper layer never touches physical columns—a physical rename/restructure only changes the dictionary
  4. translating SQL into natural language
Show answer & explanation click to expand
Answer: C. dataModel.ts declares dimensions/measures per view, each shaped like { sql: "traces.name", alias: "name" }, mapping logical names to real SQL (even expressions like formatDateTime). A widget only references logical names, never physical columns. So a physical rename/restructure only touches the dictionary, leaving hundreds of widgets unmoved—decoupling 'what you want' from 'how it's stored'.
2. When QueryBuilder.build() compiles filter conditions into SQL, how are user-filled concrete values (e.g. environment="production") handled?
  1. stored in Redis then referenced
  2. bound as parameters, never spliced into the SQL body (the SQL holds only {placeholders}, values bound per call)—same as Lesson 23's FilterState, rooting out SQL injection structurally
  3. directly string-concatenated into the SQL
  4. HTML-escaped then concatenated
Show answer & explanation click to expand
Answer: B. mapFilters/buildWhereClause use FilterList's apply() to produce SQL+parameters: the SQL template has {placeholders}, user values go into a parameters dict, safely bound by the ClickHouse driver. SQL separate from values, the template cacheable/auditable, malicious input can't alter the structure. This is the same injection-prevention discipline as Lesson 23 compiling FilterState into filter SQL, carried into aggregation—injection safety backstopped by structure, not each developer's diligence.
3. Langfuse's views carry v1/v2 versions. Why does the query engine introduce versions to 'views' too?
  1. v2 queries are faster than v1
  2. to support multiple ClickHouse instances
  3. version numbers are used for billing
  4. because the physical schema evolves (table refactors, column splits, materialized views), yet historical queries shouldn't break for it; a v1 query forever compiles against the v1 dictionary even after the underlying migrates to v2—same belief as Lessons 34/37's 'immutable + evolving'
Show answer & explanation click to expand
Answer: D. ClickHouse tables get refactored, columns split/merge, materialized views added for performance—normal underlying evolution. If a query binds to the current physical structure, every underlying move risks miscomputing/erroring every saved widget/monitor/API query. Versioning a view gives semantics an immutable snapshot: a v1 query forever compiles against v1. Same belief as Lesson 34's dataset-item versions and Lesson 37's prompt versions—the underlying evolves boldly because the upper layer is protected by stable versioned semantics.
💭 Open questions (no single right answer - just think or try)
  • The query engine is essentially a 'semantic layer + compiler'—a common industry pattern (Cube, LookML, Malloy). What does 'declarative + physical-agnostic + injection-safe' buy, and at what cost (the engine must support every needed query shape, escaping to raw SQL is limited)? If your team built an internal analytics platform, would you choose 'everyone writes SQL' or 'a unified semantic layer', and why? Can the two coexist?