The last lesson's widget declared view + dimensions + metrics + filters. But ClickHouse only understands SQL—it knows nothing of these declarations. The bridge between them is the query engine—it compiles that declaration into SQL and runs it. This lesson takes it apart, revealing two core parts: a semantic layer (the data model, mapping friendly logical names like "cost" and "by model" to real ClickHouse columns and expressions), and a compiler (QueryBuilder.build(), assembling the declaration into parameterized SQL).
Understanding it explains why Lesson 40's dashboards, Lesson 33's monitors, and Lesson 35's experiment aggregation can share one query shape—because they're all backed by this same compiler; you'll also see again how Lesson 23's "parameterized = injection-safe" discipline carries into aggregation queries.
📋 Analogy
The query engine is like a restaurant's "interpreter + a menu". You (the widget) order in plain words: "group by model, total the cost, only production". The interpreter (QueryBuilder) opens a menu (the data model)—where each "logical dish name" is annotated with the real ingredients and methods the kitchen uses ("total cost" = the kitchen table's sum(total_cost) column, "by model" = group by provided_model_name).
Following the menu, the interpreter precisely translates your plain words into a work ticket the kitchen (ClickHouse) can execute (SQL). Two beauties: one, you never need to know what the kitchen's ingredients are called or which fridge they're in—this menu layer isolates you from the kitchen's physical details; if the kitchen rearranges ingredients (renames columns, restructures tables), as long as the menu updates, your ordering style doesn't change one word. Two, any "concrete value" you fill in (like production) the interpreter passes in on a separate slip (parameterized), never written into the ticket body—so even a weird store name can't alter the ticket itself (SQL injection prevention, Lesson 23).
The semantic layer: translate logical names into real SQL
The engine's first part is the data model (semantic layer). For each view (traces / observations / scores-numeric / scores-categorical) it declares a "dictionary": which dimensions (groupable fields) and measures (aggregatable quantities) this view has, and which real ClickHouse SQL each maps to. For example, the dimension name maps to traces.name, and the time dimension to a formatDateTime(...) expression.
semantic layer = a dictionary of logical name↔SQL: dataModel.ts declares dimensions and measures per view, each shaped like { sql: "traces.name", alias: "name" } (a dimension); a time dimension can even be an expression like formatDateTime(traces.timestamp, …). Source: packages/shared/src/features/query/dataModel.ts:10-70, types.ts:9 (viewDeclaration).
One declarative spec, the engine writes the SQL (spec shape from features/query/types.ts, column names from the observations migration; values illustrative): you only describe what you want — view picks the table, metrics are {measure, aggregation}, dimensions are {field}, timeDimension.granularity sets the time bucket; queryBuilder compiles it into real ClickHouse SQL: measures become count()/sum(total_cost), dimensions go into GROUP BY, the bucket becomes toStartOfDay(start_time). Every user value flows through parameterized placeholders, blocking SQL injection at the root. Swap the view and you query a different table — one spec language covers every chart.
dims: name, stringValue; measures: count, share per category
The four views each correspond to a core data type (echoing Lessons 8, 13, 28's domain model). A widget picks a view, then chooses dimensions/measures to group and compute from those it declares—what you can ask is bounded by the view's dictionary, both usable and guarded against over-reach.
The compiler: assemble the declaration into parameterized SQL
The second part is QueryBuilder.build()—the real compiler. It takes a declarative QueryType, consults the data model, and assembles full SQL in steps: dimensions become SELECT … GROUP BY columns (mapDimensions); measures become count()/sum()/avg() aggregations (mapMetrics + translateAggregation); time is bucketed by granularity (applyBucketingDimension, for time-series charts); filters become WHERE; and tables are joined when needed (buildJoins).
the compiler translates item by item via the dictionary: queryBuilder.ts's build() chains mapDimensions(:270) / mapMetrics+translateAggregation(:290,137) / applyBucketingDimension(:182) / mapFilters+buildWhereClause(:490,858). Filters use FilterList's apply() to produce SQL+parameters—user values are bound as parameters, never spliced into the body (same injection prevention as Lesson 23).
// 1) a declarative query (from widget / monitor / experiment aggregation, all this shape)const query = { view: "observations", dimensions: [{ field: "model" }],
metrics: [{ measure: "totalCost", aggregation: "sum" }],
filters: [{ column: "environment", operator: "=", value: "production" }] };
// 2) compile: consult the data model, assemble parameterized SQL (GROUP BY/aggregate/WHERE/bucket/JOIN)const { sql, parameters } = new QueryBuilder(chartConfig, version).build(query, projectId);
// sql: SELECT model, sum(total_cost) … WHERE environment = {p_env} GROUP BY model// parameters: { p_env: "production" } ← values go via parameters, not spliced into SQL// 3) execute: queryExecutor sends sql+parameters to ClickHouse
1
fetch the view dictionary
getViewDeclaration(view, version) gets that view's dimension/measure dictionary by version—every later step translates against it.
2
dimensions → GROUP BY
mapDimensions translates each logical dimension into SELECT columns and GROUP BY; the time dimension goes to applyBucketingDimension for granularity bucketing.
3
measures → aggregations
mapMetrics + translateAggregation turn measures into count()/sum()/avg() aggregation functions.
4
filters → parameterized WHERE
mapFilters + buildWhereClause use FilterList.apply to produce WHERE, binding user values as parameters.
5
join + emit SQL
When crossing tables, buildJoins adds JOINs; finally it assembles { sql, parameters }, handed to queryExecutor to run on ClickHouse.
Why it's worth building this engine
Compiling a declaration to SQL could let each widget assemble strings itself. But Langfuse insists on one unified engine, because it solves four problems at once:
design
what it solves
echoes
semantic layer (logical name↔SQL)
rename/restructure physical columns—only the dictionary changes, all queries untouched; users needn't know the physical schema
decouple "what you want" from "how it's stored"
parameterization (values via parameters)
user values never spliced into the SQL body, rooting out SQL injection
Connect these four and the query engine is really a small "metrics language + compiler": the upper layers describe "which metric I want" in a declarative, physical-agnostic, injection-safe little language, and the engine renders it correctly, consistently, safely into SQL ClickHouse can run. It's the same idea as Lesson 23 compiling FilterState into filter SQL, just upgraded from "filtering" to full "aggregation queries". Declaration on top, compilation below, physical details sealed firmly inside the engine—the underlying reason Langfuse can make dashboards, monitors, and the API always compute the same number for the same metric.
🎯 Design trade-off
Why build a semantic layer instead of letting widgets write SQL directly? Direct SQL seems easier but plants three landmines. One is coupling: every widget hardcodes physical column names, so the day ClickHouse restructures a table or renames a column, hundreds of widgets break together and must be fixed one by one. The semantic layer collects "logical name → physical SQL" into one dictionary, so a physical change touches only the dictionary, the upper layer unmoved. Two is safety: letting each widget assemble SQL spreads injection-prevention responsibility to everyone who writes a widget—someone will forget or get it wrong, and one hole is one data leak. A unified engine welds parameterization into its single exit, so injection safety is backstopped by structure, not diligence. Three is consistency: SQL scattered everywhere eventually computes different numbers for the same metric. Collecting metric definitions into the semantic layer gives one algorithm company-wide. One engine buys decoupling, safety, and consistency all at once—that's why it's worth the extra few thousand lines.
Why do views carry v1/v2 versions? Because the physical schema evolves, but historical queries shouldn't break for it. ClickHouse tables get refactored, columns split and merge, new materialized views appear for performance—all normal evolution of the underlying. If a query binds tightly to the current physical structure, then every underlying move risks miscomputing or erroring every saved widget, monitor, and API query. Introducing versions to a view gives "semantics" an immutable snapshot too: a v1 query forever compiles against the v1 dictionary, even if the underlying has long migrated to v2's new tables. This is the same belief echoing at the query layer as Lesson 34's dataset-item versions and Lesson 37's prompt versions—the underlying can evolve boldly because the upper layer is protected by a stable layer of versioned semantics.
🎯 Key points
The query engine = a declarative-query → SQL compiler: a widget's view+dimensions+metrics+filters declaration is compiled to ClickHouse SQL and executed by it—the engine behind Lesson 40's dashboards.
Semantic layer (data model): declares dimensions/measures per view, each mapping a logical name to real SQL (e.g. name→traces.name, time→formatDateTime(...))—the upper layer never touches physical columns.
Compiler (QueryBuilder.build): consults the dictionary to translate dimensions into GROUP BY, measures into aggregations, time into buckets, filters into WHERE, joining tables when needed, producing SQL + parameters.
Parameterized = injection-safe: user-filled values are bound as parameters, never spliced into the SQL body—Lesson 23's FilterState discipline carried into aggregation queries, injection safety backstopped by structure.
One engine, four payoffs: the semantic layer decouples physical, parameterization ensures safety, view v1/v2 versioning protects history, one engine for many consumers keeps definitions consistent—dashboards/monitors/experiments always the same number for the same metric.
🧪 Self-test - think about the design
1. Langfuse's query engine has a 'semantic layer' (data model). What is its core role?
caching query results
compressing ClickHouse data
mapping friendly logical names (e.g. dimension name, measure totalCost) to real ClickHouse columns/expressions (traces.name, sum(total_cost)), so the upper layer never touches physical columns—a physical rename/restructure only changes the dictionary
translating SQL into natural language
Show answer & explanation click to expand
Answer: C. dataModel.ts declares dimensions/measures per view, each shaped like { sql: "traces.name", alias: "name" }, mapping logical names to real SQL (even expressions like formatDateTime). A widget only references logical names, never physical columns. So a physical rename/restructure only touches the dictionary, leaving hundreds of widgets unmoved—decoupling 'what you want' from 'how it's stored'.
2. When QueryBuilder.build() compiles filter conditions into SQL, how are user-filled concrete values (e.g. environment="production") handled?
stored in Redis then referenced
bound as parameters, never spliced into the SQL body (the SQL holds only {placeholders}, values bound per call)—same as Lesson 23's FilterState, rooting out SQL injection structurally
directly string-concatenated into the SQL
HTML-escaped then concatenated
Show answer & explanation click to expand
Answer: B. mapFilters/buildWhereClause use FilterList's apply() to produce SQL+parameters: the SQL template has {placeholders}, user values go into a parameters dict, safely bound by the ClickHouse driver. SQL separate from values, the template cacheable/auditable, malicious input can't alter the structure. This is the same injection-prevention discipline as Lesson 23 compiling FilterState into filter SQL, carried into aggregation—injection safety backstopped by structure, not each developer's diligence.
3. Langfuse's views carry v1/v2 versions. Why does the query engine introduce versions to 'views' too?
v2 queries are faster than v1
to support multiple ClickHouse instances
version numbers are used for billing
because the physical schema evolves (table refactors, column splits, materialized views), yet historical queries shouldn't break for it; a v1 query forever compiles against the v1 dictionary even after the underlying migrates to v2—same belief as Lessons 34/37's 'immutable + evolving'
Show answer & explanation click to expand
Answer: D. ClickHouse tables get refactored, columns split/merge, materialized views added for performance—normal underlying evolution. If a query binds to the current physical structure, every underlying move risks miscomputing/erroring every saved widget/monitor/API query. Versioning a view gives semantics an immutable snapshot: a v1 query forever compiles against v1. Same belief as Lesson 34's dataset-item versions and Lesson 37's prompt versions—the underlying evolves boldly because the upper layer is protected by stable versioned semantics.
💭 Open questions (no single right answer - just think or try)
The query engine is essentially a 'semantic layer + compiler'—a common industry pattern (Cube, LookML, Malloy). What does 'declarative + physical-agnostic + injection-safe' buy, and at what cost (the engine must support every needed query shape, escaping to raw SQL is limited)? If your team built an internal analytics platform, would you choose 'everyone writes SQL' or 'a unified semantic layer', and why? Can the two coexist?