Read Lesson 28 (scores) and Lesson 30 (eval); this lesson covers how monitors and alerting tie together.
Part 5 让你能给质量评分了——但分数躺在库里,你不可能一直盯着看。这一课讲 Part 5 的收尾,也是让评估真正起作用的最后一环:监控器(monitor)。它是「主动」的那一半——按固定节奏盯着某个指标或分数,一旦越过你设的线,就主动发出告警。一句话:把质量监测从「你去拉」(看仪表盘)变成「它来推」(异常即告警)。
我们会看到一个优雅的复用:monitor 本质上是一个「会自己盯着自己」的仪表盘组件;一台把指标值映射成 OK/WARNING/ALERT 的严重度状态机;以及一条「只在状态变化时才告警、并把投递解耦出去」的链路。
Part 5 let you score quality—but scores sit in a database, and you can't watch them forever. This lesson caps Part 5 and adds the last link that makes evaluation actually act: the monitor. It's the "active" half—watching a metric or score on a fixed cadence, and when it crosses a line you set, raising an alert on its own. In one phrase: turn quality monitoring from "you pull" (look at a dashboard) into "it pushes" (alert on anomaly).
We'll see an elegant reuse: a monitor is essentially a dashboard widget that watches itself; a severity state machine mapping a metric value into OK/WARNING/ALERT; and a path that "only alerts on a state change and decouples delivery away".
📋 Analogy
A dashboard (Lesson 40) is like a checkup report: you flip it open to learn how things are. A monitor is like a wearable heart-rate alarm: it watches on its own, stays quiet, and the moment your rate leaves the safe zone it sounds an alert—you needn't stare at a screen.
You set it three things: the normal boundary (threshold, e.g. "error rate under 5%"), how often to measure (cadence, e.g. every 5 minutes), and how far back to look (window, e.g. the past hour). At each tick it measures once, compares against the boundary, and yields a severity (OK / warning / alert).
The thoughtful bit: it only beeps when the state changes (it pushes when "OK" becomes "alert"), not every 5 minutes; and if the anomaly persists, you can set it to "remind again every so often" (renotify). The alarm bell itself isn't wired to one device either—it goes to a unified notification hub, which forwards to SMS, Slack, or an on-call system.
A dashboard widget that watches itself
The neatest part of a monitor is that it directly reuses the dashboard's query shape. The source comment says it plainly: a Monitor's query fields "mirror DashboardWidget"—the same view (observations or scores), filters (which data), metric (what to compute). The only difference: a dashboard widget computes once when you open the page; a monitor is computed repeatedly by a scheduler per cadenceMs and the result is taken to compare against a threshold.
monitor = a dashboard widget with built-in scheduling and thresholds. schema.prisma:1752's Monitor packs view/filters/metric (the comment literally says mirrors DashboardWidget) together with windowMs/cadenceMs/thresholdOperator/alertThreshold. On the worker, monitorQueue.ts's MonitorProcessor.process runs ②③④ and only emits ⑤ when a line is crossed.
Watch the aggregate, page only on crossing (threshold and window from the monitor config; values illustrative): a monitor doesn't watch single scores; it aggregates a metric (error rate, avg cost, avg score…) over a rolling window and fires an alert only when it crosses your threshold, to webhook/Slack/GitHub dispatch. It turns "grep the logs later" into "it pages you on crossing" — part of how Part 5 upgrades "able to score" into "able to act on scores".
whatview / filters / metric
The same query trio as a dashboard widget: view decides whether to watch OBSERVATIONS or SCORES_NUMERIC / SCORES_CATEGORICAL (so it can directly monitor the scores Part 5 produced), filters scope the data, metric defines what to compute (avg score, error rate, P95 latency). A monitor is "one dashboard curve + one alarm line".
howwindow / cadence
windowMs is how far back to look (a sliding window, e.g. the past hour); cadenceMs is how often to measure (e.g. every 5 minutes). The two are decoupled: you can measure every 5 minutes looking back 1 hour—the window gives smoothing, the cadence gives timeliness.
whenthreshold / severity
thresholdOperator (GT/GTE/LT/LTE/EQ/NEQ) + alertThreshold (required) + warningThreshold (optional two-tier) decide which severity the value falls into. The state machine then decides whether to actually send an alert this time (next two sections).
The severity state machine: from a value to OK / WARNING / ALERT
Once the metric value is computed, computeSeverity maps it to a severity. The logic is clean: first check whether the value exists (handle "no data"), then compare by operator against the alert line, then the warning line—hit the alert line → ALERT, hit the warning line → WARNING, neither → OK.
Six-state severity: MonitorSeverity = { PAUSED, UNKNOWN, NO_DATA, OK, WARNING, ALERT } (schema.prisma:1736). computeSeverity first handles null per the noData mode, then compares the alert / warning thresholds by operator. Alert is checked before warning—highest wins, ensuring the most severe state takes priority.
config
values
role
thresholdOperator
GT / GTE / LT / LTE / EQ / NEQ
how to compare: error rate uses GT "exceeds", success rate uses LT "falls below"
alertThreshold / warningThreshold
two lines (warning optional)
two-tier warning: yellow card then red card, giving you a buffer
// null value: decide per the monitor's noData modeif (value === null) {
if (mode === "SUBSTITUTE_ZERO") value = 0; // treat as 0 and continueelse if (mode === "LAST_SEVERITY") return prevSeverity; // carry over lastelsereturn"NO_DATA"; // flag no data
}
// present value: check alert line first, then warning line — highest winsif (matches(value, operator, alertThreshold)) return"ALERT";
if (warningThreshold !== null &&
matches(value, operator, warningThreshold)) return"WARNING";
return"OK";
// matches: GT→value>th, LT→value<th, … (six operators)
Grounding the abstract config in real scenarios, three typical monitors look like this—note the first two watch the scores Part 5 produced:
helpfulness average dropped
view=SCORES_NUMERIC, metric=avg "helpfulness", operator=LT, warning 0.75 / alert 0.70. Window 1 hour, measured every 5 minutes. The first sentinel for quality regression.
toxicity pass-rate below target
view=SCORES_NUMERIC (a boolean score's pass-rate), metric=share "non-toxic", operator=LT, alert 0.95. A safety red line alerts the moment it's crossed.
error rate spiking
view=OBSERVATIONS, metric=share of errors, operator=GT, alert 5%. No score needed—watches Lesson 13's observation status directly, so an upstream wobble shows immediately.
Alert only on change, and decouple delivery away
Computing a severity does not mean an alert must fire. If you compute every 5 minutes and push each time because it's "still ALERT", your Slack drowns. applyStateMachine fixes this: it compares the previous and current severity and, as a rule, only emits on a state change (OK→ALERT emits, ALERT→ALERT doesn't); if the anomaly persists, it tops up per the renotify policy. And a PAUSED monitor is skipped outright—not overwriting user intent.
Two layers of restraint: applyStateMachine (processor/applyStateMachine.ts) decides emit by "did severity change + renotify", and PAUSED doesn't overwrite user intent; when it does fire, monitorQueue.ts only publishes the alert to the WebhookQueue, leaving the automation system (Trigger→Action) to decide Slack vs webhook.
processor/applyStateMachine.ts · worker/src/queues/monitorQueue.tswhether to send + how
// whether: skip if paused; else decide emit by "severity change + renotify"if (prev.severity === "PAUSED") return { emit: false, … }; // don't overwrite user intentconst severityChanged = prev.severity !== next.severity;
const emit = shouldEmit({ prev, next, prevAlertedAt, renotify, … }); // true only on change or due// how: a monitor doesn't send Slack directly, only publishes to WebhookQueue (decoupled)const processor = new MonitorProcessor(prisma, async (input) => {
await webhookQueue.add(QueueName.WebhookQueue, { payload: input, … });
});
await processor.process(job.data.payload, new Date()); // compute→severity→(if needed) publish
This path pushes Part 5 to its finish: from Lesson 28's "what a score looks like", through Lessons 29–32's "four ways to produce a score", to this lesson's "watch those scores/metrics and call someone the moment a line is crossed". Evaluation finally closes the loop into action—you no longer merely can see quality; the moment quality drops, the system sees it for you and notifies you at once. And the "publish to WebhookQueue, let automation deliver" step dovetails into Part 7's automation and integrations (Lesson 44)—monitoring is just one of many "event sources that can trigger automation".
🎯 Design trade-off
Why reuse the dashboard's query shape instead of inventing one? Because "a monitored metric" and "a dashboarded metric" are the same thing—avg score, error rate, P95 latency. Having Monitor's view/filters/metric directly mirror DashboardWidget means: any curve you can draw on a dashboard can become a self-watching alert in one click; the underlying ClickHouse query machinery is written once, reused twice. Pull and push share one metric definition—a rare "saves code and matches intuition" design.
Why alert only on a "state change" rather than every time a line is crossed? Because alert fatigue is a monitoring system's number-one killer. If error rate stays high and "still alerting" pings every 5 minutes, engineers soon mute the whole channel—and real new problems get buried. "Emit on change + renotify top-ups" makes every alert carry information (the state really changed), neither missing nor spamming. Making severity a state machine rather than a per-tick instantaneous value is precisely so it can ask the key question: "did it change from last time".
Why not send Slack directly, but publish to the WebhookQueue first? Again the belief that runs through this whole guide—decoupling. A monitor's job is "decide whether to alert"; it shouldn't care "how the alert reaches a human" (Slack? webhook? email? on-call?). Publishing alerts to one queue, with the automation system (Lesson 44's Trigger→Action) deciding the channel, means delivery can evolve independently, be retried, and connect to any new channel—while the monitor changes not a line. Decision separate from delivery, the same architectural taste as Lesson 30's "creation separate from execution" and Lesson 12's "one entry, many producers".
🎯 Key points · Part 5 wrap-up
A monitor = active evaluation: turn quality monitoring from "you pull a dashboard" into "it pushes an alert". It computes a metric over the recent windowMs per cadenceMs, alerting when a line is crossed.
Reuses the dashboard query shape: view/filters/metric (the comment literally says "mirrors DashboardWidget")—any chartable metric can become an alert; view includes SCORES_NUMERIC/CATEGORICAL, so it can directly monitor the scores Part 5 produced.
Severity state machine: computeSeverity compares the alert line then the warning line by operator (GT/GTE/LT/LTE/EQ/NEQ), yielding OK/WARNING/ALERT; null is handled per noData mode (zero/carry-over/flag); plus PAUSED/UNKNOWN lifecycle states.
Alert only on change: applyStateMachine compares previous and current severity, as a rule emitting only on change, topping up persistent anomalies per renotify, never overwriting PAUSED—curing alert fatigue.
Decoupled delivery: a monitor only publishes alerts to the WebhookQueue, with automation (Lesson 44's Trigger→Action) forwarding to Slack / webhook. Decision separate from delivery, echoing Lesson 30's "creation/execution split".
Part 5 closes the loop: from "the score model" (28) → "four ways to produce a score" (29–32) → "watch scores/metrics and alert actively" (33). Evaluation graduates from "can see quality" to "gets notified the moment quality drops"—truly becoming action.
🧪 Self-test - think about the design
1. What is the essential relationship between a Langfuse monitor and a dashboard?
a monitor reuses the dashboard widget's query shape (view/filters/metric, the source comment says mirrors DashboardWidget), differing in that a scheduler computes it on a cadence and compares against thresholds — pull (you look) becomes push (it calls you)
a dashboard is a monitor's alert history
a monitor is a read-only snapshot of a dashboard
they use entirely different query engines
Show answer & explanation click to expand
Answer: A. Monitor's view/filters/metric directly mirror DashboardWidget—any curve you can draw on a dashboard becomes a self-watching alert in one click, with the ClickHouse query machinery written once. The difference is only that a monitor adds ① scheduling (per cadenceMs) + ③④ thresholds and a state machine. Pull and push share one metric definition: saves code and matches intuition.
2. A monitor stays in ALERT. It computes every 5 minutes but does not send an alert every 5 minutes. What implements this restraint, and what problem does it solve?
applyStateMachine: as a rule emit only on a severity 'change' (OK→ALERT sends, ALERT→ALERT doesn't), topping up persistent anomalies per the renotify period — curing alert fatigue
it sends every time, relying on Slack to dedupe
it randomly drops some alerts
it only sends during work hours
Show answer & explanation click to expand
Answer: A. Alert fatigue is a monitoring system's top killer: a persistent ALERT pinging every 5 minutes makes engineers mute the channel and bury real issues. applyStateMachine compares prev/next severity, considering emit only when severityChanged, and tops up per renotify; PAUSED is skipped to not overwrite user intent. Making severity a state machine rather than an instantaneous value is exactly to ask 'did it change from last time'.
3. After deciding to alert, a monitor doesn't call the Slack API directly but publishes the alert to the WebhookQueue, with the automation system forwarding it. Why this design?
decision separate from delivery (decoupling): the monitor only decides 'whether to alert' and needn't know each delivery channel; delivery can evolve/retry/add channels independently with the monitor unchanged — echoing Lesson 30's create/execute split
Slack can't be called directly
to archive the alerts
the WebhookQueue is faster than calling Slack directly
Show answer & explanation click to expand
Answer: A. A monitor's job is deciding whether a line is crossed, not coupling to 'how the alert reaches a human' (Slack/webhook/email/on-call). Publishing to one WebhookQueue, with automation (Lesson 44's Trigger→Action) choosing the channel, lets delivery evolve/retry/add channels independently while the monitor stays put. Same decoupling taste as Lesson 30's create/execute split and Lesson 12's one-entry-many-producers.
💭 Open questions (no single right answer - just think or try)
Part 5 is now complete: the score model (28) → four ways to produce scores (29–32) → active monitoring and alerting (33). Reviewing the whole part, Langfuse turns 'evaluation' from a static 'can score, can query' system into a dynamic closed loop that 'auto-evaluates, alerts actively, and calibrates AI with human scores'. If you designed an evaluation system for your own LLM app, how would you combine these five capabilities (human/LLM-judge/code/monitoring)? Which quality signals would you continuously monitor and alert on, and why?