🪢 Langfuse 图解教程Langfuse Visual Guide 第五部分 · 评估与评分Part 5 · Evaluation & Scoring 33 / 55
第五部分 · 评估与评分Part 5 · Evaluation & Scoring

监控器与告警Monitors & alerting

🧭 读前Before you start
先读 第28课 的 score 与 第30课 的 eval,本课讲监控器与告警怎么联动。
Read Lesson 28 (scores) and Lesson 30 (eval); this lesson covers how monitors and alerting tie together.

Part 5 让你能给质量评分了——但分数躺在库里,你不可能一直盯着看。这一课讲 Part 5 的收尾,也是让评估真正起作用的最后一环:监控器(monitor)。它是「主动」的那一半——按固定节奏盯着某个指标或分数,一旦越过你设的线,就主动发出告警。一句话:把质量监测从「你去拉」(看仪表盘)变成「它来推」(异常即告警)。 我们会看到一个优雅的复用:monitor 本质上是一个「会自己盯着自己」的仪表盘组件;一台把指标值映射成 OK/WARNING/ALERT 的严重度状态机;以及一条「只在状态变化时才告警、并把投递解耦出去」的链路。

📋 生活类比
仪表盘(第 40 课)像一份体检报告:你主动去翻,才知道指标如何。监控器则像一只可穿戴心率警报器:它自己盯着,平时不吭声,一旦心率冲出安全区间就主动报警——你不必时时盯屏。 你给它设三样东西:正常的边界(阈值,如「错误率别超 5%」)、多久测一次(cadence,如每 5 分钟)、看最近多久的数据(window,如过去 1 小时)。它每到点就量一次、和边界比一比,得出一个严重度(正常 / 警告 / 告警)。 贴心的是:它只在状态发生变化时响一声(从「正常」变「告警」才推送),而不是每 5 分钟都吵你一遍;要是异常一直不好,你还能设「每隔一阵再提醒一次」(renotify)。报警铃本身也不直接接到某一个设备——而是发到一个统一的通知中枢,再由它转给短信、Slack 或值班系统。

一台会自己盯着自己的仪表盘组件

monitor 最妙的设计,是它直接复用了仪表盘的查询形状。源码注释写得明明白白:Monitor 的查询字段「mirrors DashboardWidget」——同样的 view(看 observation 还是 score)、filters(过滤哪些数据)、metric(算什么指标)。区别只在于:仪表盘组件等你打开页面才算一次;monitor 则由一个调度器按 cadenceMs 反复地算、并把结果拿去比阈值。

监控流水线:定时算指标 → 比阈值 → 变化才告警 ① 调度器nextRunAt 到期claim 后入队 MonitorJob ② 处理器算指标view/filters/metric(= 仪表盘组件的查询)在最近 windowMs 窗口 ③ computeSeverityvalue <op> 阈值→ OK/WARNING/ALERT ④ 状态机严重度变了吗?变了才 emit 告警 ⑤ WebhookQueue → 自动化解耦投递:Slack / webhook(第 44 课) 仅当 emit=true 仪表盘组件「等你看」;monitor 多了①调度 + ③④阈值与状态机——于是它「自己看,异常才喊你」 同一套查询机制,pull(仪表盘)与 push(监控)两种姿态
monitor = 自带调度与阈值的仪表盘组件。schema.prisma:1752 的 Monitor 把 view/filters/metric(注释原文 mirrors DashboardWidget)和 windowMs/cadenceMs/thresholdOperator/alertThreshold 装在一起。worker 端 monitorQueue.ts 的 MonitorProcessor.process 跑②③④,越线才发⑤。
示例:聚合指标越过阈值就告警 阈值 5% 指标:每小时错误率 当前 7.2% · 已越线 0%10%+ 🔔 告警触发→ webhook(第44课)/ Slack(第45课)/ GitHub dispatch 监控器怎么工作在 trace/score 流上按窗口聚合越过你设的阈值 → 触发 Action
盯聚合,越线才喊人(阈值与窗口来自监控配置;值为示例):监控器不盯单条 score,而是在滚动窗口上算某个聚合指标(错误率、平均成本、平均分……),越过你设的阈值才触发一条告警,去 webhook/Slack/GitHub dispatch。把「事后翻日志」变成「越线主动找你」——这正是第 5 部分「能评分」升级成「能盯着分做决策」的一环。
看什么view / filters / metric
和仪表盘组件同款的查询三件套:view 决定盯 OBSERVATIONS 还是 SCORES_NUMERIC / SCORES_CATEGORICAL(能直接监控 Part 5 算出的分数),filters 圈定范围,metric 定义算什么(如平均分、错误率、P95 延迟)。一个 monitor 就是「一条仪表盘曲线 + 一条警戒线」。
怎么测window / cadence
windowMs 是看最近多久的数据(滑动窗口,如过去 1 小时),cadenceMs 是多久测一次(如每 5 分钟)。两者解耦:你可以每 5 分钟、回看 1 小时——窗口给平滑、节奏给及时。
何时响threshold / severity
thresholdOperator(GT/GTE/LT/LTE/EQ/NEQ)+ alertThreshold(必填)+ warningThreshold(选填两档)决定指标值落进哪个严重度。再由状态机决定这次要不要真的发告警(下两节细讲)。

严重度状态机:从指标值到 OK / WARNING / ALERT

算出指标值后,computeSeverity 把它映射成一个严重度。逻辑很干净:先看值在不在(处理「没数据」),再按 operator 依次比告警线、警告线——撞上告警线就 ALERT,撞上警告线就 WARNING,都没撞就 OK。

MonitorSeverity 六态(computeSeverity + 状态机) 指标值 value?先判空 NO_DATA按 noData 模式:补零/沿用/标无数据 ALERT WARNING OK PAUSED UNKNOWN 用户暂停初始/未知 空 撞告警线 撞警告线 都没撞 三个「生命周期态」NO_DATA/PAUSED/UNKNOWN + 三个「比阈值得到的态」ALERT/WARNING/OK
六态严重度:MonitorSeverity = { PAUSED, UNKNOWN, NO_DATA, OK, WARNING, ALERT }(schema.prisma:1736)。computeSeverity 先按 noData 模式处理空值,再按 operator 比 alert / warning 阈值。先比 ALERT 后比 WARNING——就高不就低,确保最严重的状态优先。
配置项取值作用
thresholdOperatorGT / GTE / LT / LTE / EQ / NEQ怎么比:错误率用 GT「超过」,成功率用 LT「低于」
alertThreshold / warningThreshold两条线(警告可选)两档预警:先黄牌后红牌,给你缓冲
noData.modeSUBSTITUTE_ZERO / LAST_SEVERITY / SHOW_NO_DATA / NOTIFY_NO_DATA没数据时怎么办:补 0 再判 / 沿用上次 / 标「无数据」/ 还为无数据告警
packages/shared/src/features/monitors/processor/computeSeverity.ts值 → 严重度
// 空值:按 monitor 的 noData 模式决定
if (value === null) {
  if (mode === "SUBSTITUTE_ZERO") value = 0;          // 当作 0 继续比
  else if (mode === "LAST_SEVERITY") return prevSeverity;  // 沿用上次
  else return "NO_DATA";                              // 标记无数据
}
// 非空:先比告警线、再比警告线——就高不就低
if (matches(value, operator, alertThreshold))   return "ALERT";
if (warningThreshold !== null &&
    matches(value, operator, warningThreshold))  return "WARNING";
return "OK";
// matches: GT→value>th, LT→value<th, …(六种 operator)

把抽象配置落到真实场景,三个典型 monitor 长这样——注意前两个直接盯着 Part 5 算出的分数:

有用性均分掉了

view=SCORES_NUMERIC,metric=「有用性」平均分,operator=LT,warning 0.75 / alert 0.70。窗口 1 小时、每 5 分钟测。质量回归的第一道哨兵。

毒性通过率不达标

view=SCORES_NUMERIC(布尔分的通过率),metric=「无毒」占比,operator=LT,alert 0.95。安全红线一破即告警。

错误率飙升

view=OBSERVATIONS,metric=错误占比,operator=GT,alert 5%。不依赖 score,直接盯第 13 课的 observation 状态——上游一抖就知道。

只在变化时告警,且把投递解耦出去

算出严重度,不等于就要发告警。如果每 5 分钟算一次、每次都因为「还在 ALERT」而推一条,你的 Slack 会被刷爆。applyStateMachine 解决这个:它对比上一次和这一次的严重度,原则上只在状态发生变化时才发(OK→ALERT 发、ALERT→ALERT 不发);若异常持续,再按 renotify 策略隔一阵补一次。而且,被暂停(PAUSED)的 monitor 直接跳过——不覆盖用户意图。

变化才 emit · 投递走统一队列 上次 OK → 这次 ALERT变化 → 发! ALERT → 仍 ALERT默认不发(防刷屏) 持续异常 + renotify 到点补一次提醒 emit 决策applyStateMachinePAUSED 直接跳过 WebhookQueue(统一通知中枢)monitor 只管发布,不管怎么投 Slack webhook 「变化才发」防刷屏 + 「统一队列再分发」让 monitor 不必认识每种投递渠道(第 44 课自动化)
两层克制:applyStateMachine(processor/applyStateMachine.ts)按「严重度是否变化 + renotify」决定 emit,PAUSED 不覆盖用户意图;真要发时,monitorQueue.ts 只把告警发布到 WebhookQueue,由自动化系统(Trigger→Action)决定投到 Slack 还是 webhook。
processor/applyStateMachine.ts · worker/src/queues/monitorQueue.ts是否发 + 怎么发
// 是否发:暂停跳过;否则按「严重度变化 + renotify」决定 emit
if (prev.severity === "PAUSED") return { emit: false, … };   // 不覆盖用户意图
const severityChanged = prev.severity !== next.severity;
const emit = shouldEmit({ prev, next, prevAlertedAt, renotify, … });  // 变化或到点才 true

// 怎么发:monitor 不直接发 Slack,只发布到 WebhookQueue(解耦投递)
const processor = new MonitorProcessor(prisma, async (input) => {
  await webhookQueue.add(QueueName.WebhookQueue, { payload: input, … });
});
await processor.process(job.data.payload, new Date());   // 算指标→定severity→(必要时)发布

这条链路把 Part 5 推到了终点:从第 28 课「一个 score 长什么样」,到 29–32 课「四种方式生产 score」,再到这一课「盯着这些分数/指标,越线就主动喊人」。评估终于闭环成行动——你不再只是能看到质量,而是质量一掉,系统就替你看到、并立刻通知你。而「发布到 WebhookQueue、由自动化转投递」这一步,正好接上 Part 7 的自动化与集成(第 44 课)——监控只是众多「会触发自动化的事件源」之一。

🎯 设计取舍
为什么 monitor 要复用仪表盘的查询形状,而不是自成一套? 因为「监控的指标」和「仪表盘看的指标」本来就是同一个东西——平均分、错误率、P95 延迟。让 Monitor 的 view/filters/metric 直接 mirror DashboardWidget,意味着:你在仪表盘上画得出的任何曲线,都能一键变成一个会自己盯着的告警;底层那套 ClickHouse 查询机制只写一遍、两处复用。pull 与 push 共享同一个指标定义,这是少有的「既省代码又顺直觉」的设计。

为什么只在「状态变化」时告警,而不是每次越线都发? 因为告警疲劳是监控系统的头号杀手。如果错误率持续高,每 5 分钟一条「还在告警」,工程师很快就会把整个频道静音——于是真正的新问题也被淹没。「变化才发 + renotify 周期补发」让每条告警都携带信息量(状态真的变了),既不漏报、又不刷屏。severity 做成状态机而非每次重算的瞬时值,正是为了能问出「和上次比变了吗」这个关键问题。

为什么告警不直接发 Slack,而要先发布到 WebhookQueue? 又是那个贯穿全书的信念——解耦。monitor 的职责是「判断要不要告警」,不该关心「告警怎么送到人手里」(Slack?webhook?邮件?值班系统?)。把告警发布到统一队列,再由自动化系统(第 44 课的 Trigger→Action)决定投递渠道,于是:投递方式可独立演进、可重试、可对接任意新渠道,而 monitor 一行不改。判断与投递分离,和第 30 课「创建与执行分离」、第 12 课「一个入口多个生产者」是同一种架构品味。
🎯 本课要点 · 第五部分收官
  • monitor = 主动的评估:把质量监测从「你去拉仪表盘」变成「它来推告警」。按 cadenceMs 定时在最近 windowMs 窗口算指标,越线即告警。
  • 复用仪表盘查询形状:view/filters/metric 注释原文「mirrors DashboardWidget」——能画成曲线的指标就能变成告警;view 含 SCORES_NUMERIC/CATEGORICAL,可直接监控 Part 5 算出的分数。
  • 严重度状态机:computeSeverity 按 operator(GT/GTE/LT/LTE/EQ/NEQ)先比告警线、再比警告线,得 OK/WARNING/ALERT;空值按 noData 模式处理(补零/沿用/标无数据);另有 PAUSED/UNKNOWN 生命周期态。
  • 只在变化时告警:applyStateMachine 比上次与这次的严重度,原则上变化才发,持续异常按 renotify 补发,PAUSED 不覆盖——根治告警疲劳。
  • 解耦投递:monitor 只把告警发布到 WebhookQueue,由自动化(第 44 课 Trigger→Action)转投 Slack / webhook。判断与投递分离,呼应第 30 课「创建/执行分离」。
  • Part 5 闭环:从「score 的模型」(28)→「四种生产 score 的方式」(29–32)→「盯着 score/指标主动告警」(33)。评估从「能看见质量」升级为「质量一掉就被通知」——真正成为行动。

🧪 自测 · 想一想为什么这么设计

1. Langfuse 的 monitor(监控器)和 dashboard(仪表盘)有什么本质关系?
  1. monitor 复用仪表盘组件的查询形状(view/filters/metric,源码注释 mirrors DashboardWidget),区别在于它由调度器定时算、并拿去比阈值——pull(你看)变 push(它喊)
  2. dashboard 是 monitor 的告警历史
  3. monitor 是 dashboard 的只读快照
  4. 两者用完全不同的查询引擎
看答案与解析 点击展开
答案:A。Monitor 的 view/filters/metric 直接 mirror DashboardWidget——能在仪表盘画出的曲线,都能一键变成会自己盯着的告警,底层 ClickHouse 查询机制只写一遍。区别只在 monitor 多了①调度(按 cadenceMs 定时)+③④阈值与状态机。pull 与 push 共享同一个指标定义,既省代码又顺直觉。
2. 一个 monitor 持续处于 ALERT 状态。它每 5 分钟算一次,但并不会每 5 分钟都发一条告警。这种克制由什么实现,解决什么问题?
  1. applyStateMachine:原则上只在严重度「变化」时 emit(OK→ALERT 发、ALERT→ALERT 不发),持续异常按 renotify 周期补发——根治告警疲劳
  2. 每次都发,靠 Slack 自己去重
  3. 随机丢弃一部分告警
  4. 只在工作时间发
看答案与解析 点击展开
答案:A。告警疲劳是监控系统头号杀手:持续 ALERT 每 5 分钟一条,工程师会把频道静音,真问题也被淹没。applyStateMachine 对比 prev/next 严重度,severityChanged 才考虑 emit,并按 renotify 决定是否补发;PAUSED 直接跳过不覆盖用户意图。把 severity 做成状态机而非瞬时值,正是为了能问「和上次比变了吗」。
3. monitor 判定要告警后,并不直接调 Slack API,而是把告警发布到 WebhookQueue,再由自动化系统转投递。为什么这样设计?
  1. 判断与投递分离(解耦):monitor 只管「要不要告警」,不必认识每种投递渠道;投递方式可独立演进/重试/接新渠道,monitor 一行不改——呼应第30课创建/执行分离
  2. Slack 不支持直接调用
  3. 为了把告警存档
  4. WebhookQueue 比直接调 Slack 快
看答案与解析 点击展开
答案:A。monitor 的职责是判断是否越线告警,不该耦合「告警怎么送到人手里」(Slack/webhook/邮件/值班)。发布到统一 WebhookQueue,由自动化(第44课 Trigger→Action)决定渠道,于是投递可独立演进、可重试、可接任意新渠道,monitor 不变。这与第30课创建/执行分离、第12课一个入口多生产者,是同一种「解耦」品味。
💭 发散思考(没有标准答案,动手或动脑想想)
  • Part 5 到此完整:score 模型(28)→ 四种生产 score 的方式(29–32)→ 主动监控告警(33)。回顾整个第五部分,Langfuse 让「评估」从一个静态的「能评分、能查」系统,变成一个动态的「能自动评、能主动告警、还能用人工分校准 AI」的闭环。如果让你为自己的 LLM 应用设计一套评估体系,你会怎么组合这五种能力(人工/LLM裁判/代码/监控)?哪些质量信号你会持续监控并告警,为什么?

Part 5 let you score quality—but scores sit in a database, and you can't watch them forever. This lesson caps Part 5 and adds the last link that makes evaluation actually act: the monitor. It's the "active" half—watching a metric or score on a fixed cadence, and when it crosses a line you set, raising an alert on its own. In one phrase: turn quality monitoring from "you pull" (look at a dashboard) into "it pushes" (alert on anomaly). We'll see an elegant reuse: a monitor is essentially a dashboard widget that watches itself; a severity state machine mapping a metric value into OK/WARNING/ALERT; and a path that "only alerts on a state change and decouples delivery away".

📋 Analogy
A dashboard (Lesson 40) is like a checkup report: you flip it open to learn how things are. A monitor is like a wearable heart-rate alarm: it watches on its own, stays quiet, and the moment your rate leaves the safe zone it sounds an alert—you needn't stare at a screen. You set it three things: the normal boundary (threshold, e.g. "error rate under 5%"), how often to measure (cadence, e.g. every 5 minutes), and how far back to look (window, e.g. the past hour). At each tick it measures once, compares against the boundary, and yields a severity (OK / warning / alert). The thoughtful bit: it only beeps when the state changes (it pushes when "OK" becomes "alert"), not every 5 minutes; and if the anomaly persists, you can set it to "remind again every so often" (renotify). The alarm bell itself isn't wired to one device either—it goes to a unified notification hub, which forwards to SMS, Slack, or an on-call system.

A dashboard widget that watches itself

The neatest part of a monitor is that it directly reuses the dashboard's query shape. The source comment says it plainly: a Monitor's query fields "mirror DashboardWidget"—the same view (observations or scores), filters (which data), metric (what to compute). The only difference: a dashboard widget computes once when you open the page; a monitor is computed repeatedly by a scheduler per cadenceMs and the result is taken to compare against a threshold.

monitor pipeline: scheduled metric → threshold → alert only on change ① schedulernextRunAt dueclaim → enqueue MonitorJob ② processor computesview/filters/metric(= a dashboard widget's query)over the recent windowMs ③ computeSeverityvalue <op> threshold→ OK/WARNING/ALERT ④ state machinedid severity change?emit only if so ⑤ WebhookQueue → automationdecoupled delivery: Slack / webhook (Lesson 44) only if emit=true a dashboard widget "waits for you to look"; a monitor adds ① scheduling + ③④ thresholds and state machine—so it "looks itself, calls you only on anomaly" one query mechanism, two postures: pull (dashboard) and push (monitor)
monitor = a dashboard widget with built-in scheduling and thresholds. schema.prisma:1752's Monitor packs view/filters/metric (the comment literally says mirrors DashboardWidget) together with windowMs/cadenceMs/thresholdOperator/alertThreshold. On the worker, monitorQueue.ts's MonitorProcessor.process runs ②③④ and only emits ⑤ when a line is crossed.
Example: an aggregate crosses the threshold, alert fires threshold 5% metric: hourly error rate now 7.2% · over 0%10%+ 🔔 alert fires→ webhook (L44) / Slack (L45) / GitHub dispatch how the monitor worksaggregates the trace/score stream over a windowcrossing your threshold → triggers an Action
Watch the aggregate, page only on crossing (threshold and window from the monitor config; values illustrative): a monitor doesn't watch single scores; it aggregates a metric (error rate, avg cost, avg score…) over a rolling window and fires an alert only when it crosses your threshold, to webhook/Slack/GitHub dispatch. It turns "grep the logs later" into "it pages you on crossing" — part of how Part 5 upgrades "able to score" into "able to act on scores".
whatview / filters / metric
The same query trio as a dashboard widget: view decides whether to watch OBSERVATIONS or SCORES_NUMERIC / SCORES_CATEGORICAL (so it can directly monitor the scores Part 5 produced), filters scope the data, metric defines what to compute (avg score, error rate, P95 latency). A monitor is "one dashboard curve + one alarm line".
howwindow / cadence
windowMs is how far back to look (a sliding window, e.g. the past hour); cadenceMs is how often to measure (e.g. every 5 minutes). The two are decoupled: you can measure every 5 minutes looking back 1 hour—the window gives smoothing, the cadence gives timeliness.
whenthreshold / severity
thresholdOperator (GT/GTE/LT/LTE/EQ/NEQ) + alertThreshold (required) + warningThreshold (optional two-tier) decide which severity the value falls into. The state machine then decides whether to actually send an alert this time (next two sections).

The severity state machine: from a value to OK / WARNING / ALERT

Once the metric value is computed, computeSeverity maps it to a severity. The logic is clean: first check whether the value exists (handle "no data"), then compare by operator against the alert line, then the warning line—hit the alert line → ALERT, hit the warning line → WARNING, neither → OK.

MonitorSeverity, six states (computeSeverity + state machine) metric value?check null first NO_DATAper noData mode: zero/carry-over/flag ALERT WARNING OK PAUSED UNKNOWN user-pausedinitial null meets alert meets warning neither three "lifecycle" states NO_DATA/PAUSED/UNKNOWN + three "threshold-derived" states ALERT/WARNING/OK
Six-state severity: MonitorSeverity = { PAUSED, UNKNOWN, NO_DATA, OK, WARNING, ALERT } (schema.prisma:1736). computeSeverity first handles null per the noData mode, then compares the alert / warning thresholds by operator. Alert is checked before warning—highest wins, ensuring the most severe state takes priority.
configvaluesrole
thresholdOperatorGT / GTE / LT / LTE / EQ / NEQhow to compare: error rate uses GT "exceeds", success rate uses LT "falls below"
alertThreshold / warningThresholdtwo lines (warning optional)two-tier warning: yellow card then red card, giving you a buffer
noData.modeSUBSTITUTE_ZERO / LAST_SEVERITY / SHOW_NO_DATA / NOTIFY_NO_DATAwhat to do with no data: treat as 0 / carry over last / flag "no data" / even alert for no data
packages/shared/src/features/monitors/processor/computeSeverity.tsvalue → severity
// null value: decide per the monitor's noData mode
if (value === null) {
  if (mode === "SUBSTITUTE_ZERO") value = 0;          // treat as 0 and continue
  else if (mode === "LAST_SEVERITY") return prevSeverity;  // carry over last
  else return "NO_DATA";                              // flag no data
}
// present value: check alert line first, then warning line — highest wins
if (matches(value, operator, alertThreshold))   return "ALERT";
if (warningThreshold !== null &&
    matches(value, operator, warningThreshold))  return "WARNING";
return "OK";
// matches: GT→value>th, LT→value<th, … (six operators)

Grounding the abstract config in real scenarios, three typical monitors look like this—note the first two watch the scores Part 5 produced:

helpfulness average dropped

view=SCORES_NUMERIC, metric=avg "helpfulness", operator=LT, warning 0.75 / alert 0.70. Window 1 hour, measured every 5 minutes. The first sentinel for quality regression.

toxicity pass-rate below target

view=SCORES_NUMERIC (a boolean score's pass-rate), metric=share "non-toxic", operator=LT, alert 0.95. A safety red line alerts the moment it's crossed.

error rate spiking

view=OBSERVATIONS, metric=share of errors, operator=GT, alert 5%. No score needed—watches Lesson 13's observation status directly, so an upstream wobble shows immediately.

Alert only on change, and decouple delivery away

Computing a severity does not mean an alert must fire. If you compute every 5 minutes and push each time because it's "still ALERT", your Slack drowns. applyStateMachine fixes this: it compares the previous and current severity and, as a rule, only emits on a state change (OK→ALERT emits, ALERT→ALERT doesn't); if the anomaly persists, it tops up per the renotify policy. And a PAUSED monitor is skipped outright—not overwriting user intent.

emit only on change · delivery via one queue prev OK → now ALERTchange → emit! ALERT → still ALERTdefault no emit (anti-spam) persists + renotify duetop up a reminder emit decisionapplyStateMachinePAUSED skipped WebhookQueue (one notification hub)monitor only publishes, not how to deliver Slack webhook "emit only on change" prevents spam + "one queue then fan out" frees the monitor from knowing each delivery channel (Lesson 44 automation)
Two layers of restraint: applyStateMachine (processor/applyStateMachine.ts) decides emit by "did severity change + renotify", and PAUSED doesn't overwrite user intent; when it does fire, monitorQueue.ts only publishes the alert to the WebhookQueue, leaving the automation system (Trigger→Action) to decide Slack vs webhook.
processor/applyStateMachine.ts · worker/src/queues/monitorQueue.tswhether to send + how
// whether: skip if paused; else decide emit by "severity change + renotify"
if (prev.severity === "PAUSED") return { emit: false, … };   // don't overwrite user intent
const severityChanged = prev.severity !== next.severity;
const emit = shouldEmit({ prev, next, prevAlertedAt, renotify, … });  // true only on change or due

// how: a monitor doesn't send Slack directly, only publishes to WebhookQueue (decoupled)
const processor = new MonitorProcessor(prisma, async (input) => {
  await webhookQueue.add(QueueName.WebhookQueue, { payload: input, … });
});
await processor.process(job.data.payload, new Date());   // compute→severity→(if needed) publish

This path pushes Part 5 to its finish: from Lesson 28's "what a score looks like", through Lessons 29–32's "four ways to produce a score", to this lesson's "watch those scores/metrics and call someone the moment a line is crossed". Evaluation finally closes the loop into action—you no longer merely can see quality; the moment quality drops, the system sees it for you and notifies you at once. And the "publish to WebhookQueue, let automation deliver" step dovetails into Part 7's automation and integrations (Lesson 44)—monitoring is just one of many "event sources that can trigger automation".

🎯 Design trade-off
Why reuse the dashboard's query shape instead of inventing one? Because "a monitored metric" and "a dashboarded metric" are the same thing—avg score, error rate, P95 latency. Having Monitor's view/filters/metric directly mirror DashboardWidget means: any curve you can draw on a dashboard can become a self-watching alert in one click; the underlying ClickHouse query machinery is written once, reused twice. Pull and push share one metric definition—a rare "saves code and matches intuition" design.

Why alert only on a "state change" rather than every time a line is crossed? Because alert fatigue is a monitoring system's number-one killer. If error rate stays high and "still alerting" pings every 5 minutes, engineers soon mute the whole channel—and real new problems get buried. "Emit on change + renotify top-ups" makes every alert carry information (the state really changed), neither missing nor spamming. Making severity a state machine rather than a per-tick instantaneous value is precisely so it can ask the key question: "did it change from last time".

Why not send Slack directly, but publish to the WebhookQueue first? Again the belief that runs through this whole guide—decoupling. A monitor's job is "decide whether to alert"; it shouldn't care "how the alert reaches a human" (Slack? webhook? email? on-call?). Publishing alerts to one queue, with the automation system (Lesson 44's Trigger→Action) deciding the channel, means delivery can evolve independently, be retried, and connect to any new channel—while the monitor changes not a line. Decision separate from delivery, the same architectural taste as Lesson 30's "creation separate from execution" and Lesson 12's "one entry, many producers".
🎯 Key points · Part 5 wrap-up
  • A monitor = active evaluation: turn quality monitoring from "you pull a dashboard" into "it pushes an alert". It computes a metric over the recent windowMs per cadenceMs, alerting when a line is crossed.
  • Reuses the dashboard query shape: view/filters/metric (the comment literally says "mirrors DashboardWidget")—any chartable metric can become an alert; view includes SCORES_NUMERIC/CATEGORICAL, so it can directly monitor the scores Part 5 produced.
  • Severity state machine: computeSeverity compares the alert line then the warning line by operator (GT/GTE/LT/LTE/EQ/NEQ), yielding OK/WARNING/ALERT; null is handled per noData mode (zero/carry-over/flag); plus PAUSED/UNKNOWN lifecycle states.
  • Alert only on change: applyStateMachine compares previous and current severity, as a rule emitting only on change, topping up persistent anomalies per renotify, never overwriting PAUSED—curing alert fatigue.
  • Decoupled delivery: a monitor only publishes alerts to the WebhookQueue, with automation (Lesson 44's Trigger→Action) forwarding to Slack / webhook. Decision separate from delivery, echoing Lesson 30's "creation/execution split".
  • Part 5 closes the loop: from "the score model" (28) → "four ways to produce a score" (29–32) → "watch scores/metrics and alert actively" (33). Evaluation graduates from "can see quality" to "gets notified the moment quality drops"—truly becoming action.

🧪 Self-test - think about the design

1. What is the essential relationship between a Langfuse monitor and a dashboard?
  1. a monitor reuses the dashboard widget's query shape (view/filters/metric, the source comment says mirrors DashboardWidget), differing in that a scheduler computes it on a cadence and compares against thresholds — pull (you look) becomes push (it calls you)
  2. a dashboard is a monitor's alert history
  3. a monitor is a read-only snapshot of a dashboard
  4. they use entirely different query engines
Show answer & explanation click to expand
Answer: A. Monitor's view/filters/metric directly mirror DashboardWidget—any curve you can draw on a dashboard becomes a self-watching alert in one click, with the ClickHouse query machinery written once. The difference is only that a monitor adds ① scheduling (per cadenceMs) + ③④ thresholds and a state machine. Pull and push share one metric definition: saves code and matches intuition.
2. A monitor stays in ALERT. It computes every 5 minutes but does not send an alert every 5 minutes. What implements this restraint, and what problem does it solve?
  1. applyStateMachine: as a rule emit only on a severity 'change' (OK→ALERT sends, ALERT→ALERT doesn't), topping up persistent anomalies per the renotify period — curing alert fatigue
  2. it sends every time, relying on Slack to dedupe
  3. it randomly drops some alerts
  4. it only sends during work hours
Show answer & explanation click to expand
Answer: A. Alert fatigue is a monitoring system's top killer: a persistent ALERT pinging every 5 minutes makes engineers mute the channel and bury real issues. applyStateMachine compares prev/next severity, considering emit only when severityChanged, and tops up per renotify; PAUSED is skipped to not overwrite user intent. Making severity a state machine rather than an instantaneous value is exactly to ask 'did it change from last time'.
3. After deciding to alert, a monitor doesn't call the Slack API directly but publishes the alert to the WebhookQueue, with the automation system forwarding it. Why this design?
  1. decision separate from delivery (decoupling): the monitor only decides 'whether to alert' and needn't know each delivery channel; delivery can evolve/retry/add channels independently with the monitor unchanged — echoing Lesson 30's create/execute split
  2. Slack can't be called directly
  3. to archive the alerts
  4. the WebhookQueue is faster than calling Slack directly
Show answer & explanation click to expand
Answer: A. A monitor's job is deciding whether a line is crossed, not coupling to 'how the alert reaches a human' (Slack/webhook/email/on-call). Publishing to one WebhookQueue, with automation (Lesson 44's Trigger→Action) choosing the channel, lets delivery evolve/retry/add channels independently while the monitor stays put. Same decoupling taste as Lesson 30's create/execute split and Lesson 12's one-entry-many-producers.
💭 Open questions (no single right answer - just think or try)
  • Part 5 is now complete: the score model (28) → four ways to produce scores (29–32) → active monitoring and alerting (33). Reviewing the whole part, Langfuse turns 'evaluation' from a static 'can score, can query' system into a dynamic closed loop that 'auto-evaluates, alerts actively, and calibrates AI with human scores'. If you designed an evaluation system for your own LLM app, how would you combine these five capabilities (human/LLM-judge/code/monitoring)? Which quality signals would you continuously monitor and alert on, and why?