🪢 Langfuse 图解教程Langfuse Visual Guide 第五部分 · 评估与评分Part 5 · Evaluation & Scoring 31 / 55
第五部分 · 评估与评分Part 5 · Evaluation & Scoring

代码 evalCode-based evaluation

🧭 读前Before you start
先读 第30课 的 eval 流水线,本课讲用代码而非 LLM 来评估。
Read Lesson 30 on the eval pipeline; this lesson covers evaluating with code instead of an LLM.

不是所有「好不好」都得请 LLM 来判。很多质量信号是客观、确定的:回答是不是合法 JSON?有没有泄露邮箱地址?长度超没超限?字段齐不齐?这些根本不需要一个会胡思乱想的裁判——写一段代码算就行,确定、快、还不花 LLM 的钱。这就是代码 eval(code-based evaluation)。 但这里藏着一个尖锐的问题:评估器的代码是用户写的,平台要去执行别人的代码——这是天大的安全风险。所以这一课的主角是两样东西:一个把「在哪跑、怎么跑」抽象掉的 dispatcher(派发器),和一个把用户代码关起来跑的沙箱。

📋 生活类比
上一课的 LLM-as-judge 像请一位人类评委:主观、有判断力,但每次可能给不同的分、还得付出场费。代码 eval 则像用一把标准量规 / 卡尺:客观、同一个零件量一百次都是同一个读数、几乎不要钱。 量规适合量「尺寸合不合格」这种确定的事,评委适合判「设计美不美」这种主观的事——两者互补,不是替代。 但有个麻烦:这把卡尺是别人寄来的,你不敢直接插到主控机上用(万一它内藏恶意指令?)。稳妥做法是放进一间隔离的检验室(沙箱)里跑,门一关、还掐断电话线(禁网络)——它只能量你递进去的零件、把读数递出来,碰不到外面任何东西。

两种裁判:确定的代码 vs 概率的 LLM

第 29、31 课其实是同一个 score 的两种生产方式。它们都接在第 30 课那条调度流水线后面——区别只在「执行」那一步:LLM-as-judge 调一个模型,代码 eval 跑一段函数。选哪种,取决于你要评的东西是主观还是客观:

一条 trace,两种评判,殊途同归 一条 trace 的输入/输出待评的回答 LLM-as-judge(第 29 课)概率的 · 主观/语义 · 花 LLM 钱有用性 / 语气 / 是否答非所问 代码 eval(本课)确定的 · 客观/规则 · 几乎免费合法 JSON / 含邮箱 / 长度 / 精确匹配 都产出 source=EVAL 的 score
互补,不是替代:能用规则说清的(格式、长度、关键词、精确匹配)交给代码 eval——确定、快、免费;说不清、要语义理解的(有用性、语气、相关性)交给 LLM-as-judge。两者都接第 30 课调度流水线,只是「执行」步不同。
示例:一个代码评估函数产出 score defexact_match(output, expected): ok = output.strip() == expected.strip() return{ "name": "exact_match", "value": 1 if ok else 0, "dataType": "NUMERIC", } # 也可:JSON 合法?正则匹配?长度阈值? 写回一条 score name=exact_match value=1 · dataType=NUMERIC source=EVAL 确定性 · 可复现 · 免费 · 毫秒级
不是所有评估都要 LLM(值为示例):很多判断是确定性的——精确匹配、JSON 是否合法、是否含某关键词、长度是否超限……一段普通代码就能算,返回 {{name, value, dataType}},以 source=EVAL 写回。相比 LLM 裁判(第 29 课),代码 eval 可复现、免费、毫秒级,适合有客观标准的题。
维度LLM-as-judge(第 29 课)代码 eval(本课)
评判方式调一个模型做语义判断跑一段你写的函数
确定性概率的(同输入可能不同分)确定的(同输入恒同分)
擅长主观:有用性、语气、相关性客观:合法 JSON、含 PII、长度、精确/正则匹配
成本/速度每次一次 LLM 调用,慢、花钱一次函数执行,快、几乎免费
风险模型会错判、有偏好要执行用户代码→必须沙箱隔离

dispatcher:把「在哪跑」抽象掉的策略模式

worker 不该关心用户代码到底跑在 AWS Lambda 还是本地——它只想说一句「把这段代码连同输入跑一下,给我分数」。这就是 dispatcher 接口:一个极简的契约 { name, dispatch(input) }。底下挂着两个实现,resolveConfiguredCodeEvalDispatcher() 按环境变量选一个——这是教科书式的策略模式。

一个接口,两种执行后端(策略模式) worker 执行步dispatcher.dispatch(payload) interface CodeEvalDispatcher{ name; dispatch(input): Promise<{scores}> } AwsLambdaCodeEvalDispatchername="aws-lambda" · 按语言调 Lambda 沙箱python→executor-python · TS→executor-node(生产) LocalCodeEvalDispatchername="insecure-local" · node vm 跑仅 TS/JS · 名字直说「不安全」,仅本地开发
策略模式:worker 只依赖 CodeEvalDispatcher 接口(codeEvalDispatcherTypes.ts:119-121)。resolveConfiguredCodeEvalDispatcher(codeEvalDispatchers.ts:13)按 env.LANGFUSE_CODE_EVAL_DISPATCHER 选 Lambda(生产,强隔离)或本地 vm(开发,名字就叫 insecure-local)。换后端不动 worker 一行。
packages/shared/src/server/evals/codeEvalDispatcherTypes.ts · codeEvalDispatchers.ts接口 + 选择
// 极简契约:名字 + 一个 dispatch 方法,返回 { scores:[…] }
export interface CodeEvalDispatcher {
  name: string;
  dispatch(input: DispatchInput): Promise<DispatchResult>;   // DispatchResult = { scores: 至少1条 }
}

// 按环境变量选实现——策略模式的「选择点」
export function resolveConfiguredCodeEvalDispatcher(): CodeEvalDispatcher | null {
  if (dispatcher === "insecure-local") return new LocalCodeEvalDispatcher();  // node vm,仅 TS/JS
  if (dispatcher === "aws-lambda") return new AwsLambdaCodeEvalDispatcher({
    functionNameByLanguage: { PYTHON: "…-python", TYPESCRIPT: "…-node" } });
}

沙箱的铁律:禁网络、限大小、限时

既然要跑用户代码,平台就得用一圈硬约束把它框死。这些约束不是 UI 提示,而是执行层强制的——超了就直接报特定错误码。最值得玩味的是「禁网络」:它一箭双雕,既挡住安全风险,又保住确定性。

执行层强制的硬约束:超了就直接报特定错误码 沙箱(执行层强制) evaluate(payload) node vm / Lambda 无 fetch / require 禁网络(一箭双雕) 防 SSRF/外泄 + 保确定性 限时 超时 → TIMEOUT 源码 ≤ 256KB → SOURCE_TOO_LARGE 输入 ≤ 5.5MB → PAYLOAD_TOO_LARGE 结果 ≤ 256KB → RESULT_TOO_LARGE 结果须 {scores:[≥1]} → INVALID_RESULT
铁律具体限制为什么
禁网络评估器代码不准发任何网络请求(TIMEOUT 错误里明说)① 安全:防数据外泄、防 SSRF ② 确定性:网络=不确定+可能永不返回
限时必须在配置的运行时限内跑完,否则 TIMEOUT防死循环 / 卡死占住沙箱
源码大小≤ 256 KB(SOURCE_TOO_LARGE)评估器应短小,不是塞个大程序进来
输入大小≤ 5.5 MB(PAYLOAD_TOO_LARGE,含源码+变量)限制单次投喂量,护住派发链路
结果大小≤ 256 KB(RESULT_TOO_LARGE)返回的是分数,不该是海量数据
结果格式必须 { scores: [≥1 条] },每条带 name/dataType/value 且类型匹配(INVALID_RESULT)不规整就没法归一成 score
codeEvalDispatcherTypes.ts · localCodeEvalDispatcher.ts限制常量 + 沙箱执行
// 三道大小红线(字节),执行层强制(codeEvalDispatcherTypes.ts:4-6)
CODE_EVAL_SOURCE_MAX_BYTES          = 256 * 1024;        // 源码 ≤256KB
CODE_EVAL_DISPATCH_PAYLOAD_MAX_BYTES = 5.5 * 1024 * 1024; // 输入 ≤5.5MB
CODE_EVAL_DISPATCH_RESULT_MAX_BYTES  = 256 * 1024;        // 结果 ≤256KB

// 本地派发器:用 node 的 vm 跑,带超时;名字诚实地叫 "insecure-local"
import * as vm from "node:vm";
const context = vm.createContext({ /* 受限的全局,无 fetch/require */ });
vm.runInContext("evaluate(payload)", context, { timeout });  // 跑用户的 evaluate(),结果须是 {scores}

跑完之后呢?和第 29 课完全一样:拿到的 {scores:[…]} 归一成 score、标 source=EVAL、经第 12 课摄取链路回流 scores 表。代码 eval 并没有另起炉灶——它只是把第 30 课调度流水线里「执行」那一步,从「调 LLM」换成「dispatch 到沙箱」。同一套调度、同一种 score、同一条回流,只是裁判从「概率的模型」换成了「确定的函数」。这就是为什么 Langfuse 能把两种评估塞进同一个 JobExecution 状态机里。

1

取变量

和裁判流水线同源:按变量映射从 trace/observation 取出 input/output 等列(extractObservationVariables)。

2

组 payload

把用户的评估器源码 + 取到的变量 + runtime.language 打包成 CodeEvalPayload,先过大小红线(源码≤256KB、整体≤5.5MB)。

3

dispatch 到沙箱

dispatcher.dispatch(payload):Lambda 按语言调 python/node 函数,或本地 vm 跑——禁网络、限时。

4

校验结果

返回必须是 {scores:[≥1]},逐条校验 name/dataType/value 且类型匹配、结果≤256KB(parseDispatchResult)。

5

归一 + 回流

和第 29 课同款:标 source=EVAL,经摄取链路写回 scores 表,工单标 COMPLETED。

🎯 设计取舍
为什么要 dispatcher 这层抽象,而不是直接调 Lambda? 因为「在哪跑用户代码」是个会变的决策:云上用 Lambda(强隔离、按需扩容),自托管的人可能没有 Lambda、只想本地跑。把执行后端藏在 {name, dispatch} 接口后,worker 的逻辑对后端一无所知——换实现、加新沙箱(比如未来的 WASM runtime),都不动调用方一行。这正是策略模式的价值:让「怎么做」可替换,而「做什么」稳定。

为什么本地派发器要起名 "insecure-local"? 这是一处难得的诚实。Node 的 vm 模块不是真正的安全沙箱——有经验的人能从 vm context 里逃逸。Langfuse 不藏着掖着,直接把「不安全」写进名字,等于在源码里贴了张警告条:这条路只配本地开发,生产请用 Lambda 那种真隔离。命名即文档,把风险摆在最显眼处。

为什么「禁网络」这条铁律最关键? 它一刀解决两个问题。安全上:用户代码一旦能联网,就能把你递进去的 trace 数据(可能含敏感信息)偷偷外传,或拿沙箱当跳板打内网(SSRF)。确定性上:评估的灵魂是「同输入恒同分」,而网络请求天生不确定——接口会抖、会超时、会返回不同结果,甚至「可能永不返回」(源码原话)。掐断网络,等于同时焊死了安全口子和不确定性源头。一条约束,两重收益——好的限制往往如此。
🎯 本课要点
  • 代码 eval = 确定性评估:用一段函数算分,适合客观/规则类信号(合法 JSON、含 PII、长度、精确/正则匹配);与 LLM-as-judge(概率/主观)互补,不是替代。
  • 同一条流水线:它接在第 30 课调度流水线后,只把「执行」步从「调 LLM」换成「dispatch 到沙箱」;结果同样归一成 source=EVAL 的 score 回流——同一个 JobExecution 状态机。
  • dispatcher 策略模式:worker 只依赖 {name, dispatch} 接口;resolveConfiguredCodeEvalDispatcher 按环境变量选 AwsLambda(生产,按语言调 python/node 沙箱)或本地 vm。换后端不动调用方。
  • 本地派发器叫 "insecure-local":node 的 vm 不是真沙箱,名字直说「不安全」——只配本地开发,生产用 Lambda 真隔离。命名即警告。
  • 沙箱铁律:禁网络、限时、源码 ≤256KB、输入 ≤5.5MB、结果 ≤256KB、结果须是 {scores:[≥1]}。其中「禁网络」一箭双雕——既防数据外泄/SSRF,又保住「同输入恒同分」的确定性。

🧪 自测 · 想一想为什么这么设计

1. Langfuse 同时有 LLM-as-judge(第 29 课)和代码 eval(本课)两种评估。下面哪种说法最准确地刻画了它们的关系?
  1. 两者用完全独立的调度与存储
  2. 代码 eval 只能产出布尔分
  3. 代码 eval 是 LLM-as-judge 的升级替代品
  4. 互补:代码 eval 确定、适合客观/规则信号(合法 JSON、含 PII、长度);LLM-as-judge 概率、适合主观/语义信号(有用性、语气)。两者都接同一条调度流水线,只是「执行」步不同
看答案与解析 点击展开
答案:D。两者是同一个 score 的两种生产方式,都挂在第 30 课调度流水线后,区别只在「执行」那一步:LLM-as-judge 调模型、代码 eval 跑函数。结果都归一成 source=EVAL 的 score 回流同一张表、走同一个 JobExecution 状态机。能用规则说清的交给代码 eval(确定、免费),要语义理解的交给 LLM-as-judge。
2. 代码 eval 要执行用户写的代码。Langfuse 的本地派发器在源码里直接命名为 "insecure-local"。这个命名透露了什么?
  1. 本地只支持 Python
  2. 本地运行速度慢
  3. 它会泄露环境变量
  4. 诚实的警告:它用 node 的 vm 模块,而 vm 不是真正的安全沙箱(可被逃逸);命名即文档,明示这条路只配本地开发,生产要用 Lambda 那种真隔离
看答案与解析 点击展开
答案:D。node 的 vm 模块能创建隔离 context 跑代码,但业界共识是它不足以安全运行不可信代码(有已知逃逸手法)。Langfuse 不粉饰,把 insecure 写进 dispatcher 的 name,相当于在源码里贴警告条:本地开发可用,生产请切到 AwsLambdaCodeEvalDispatcher(ephemeral、强隔离、按语言调 python/node 函数)。
3. 代码 eval 的沙箱有一条铁律:评估器代码不准发任何网络请求。为什么这条「禁网络」被认为一箭双雕?
  1. 既保安全(用户代码联网就能外泄你递进去的 trace 数据、或拿沙箱打内网 SSRF),又保确定性(网络天生不确定、会抖会超时甚至可能永不返回,破坏「同输入恒同分」)
  2. 只是为了省带宽
  3. 让评估跑得更快
  4. 防止评估器调用 LLM
看答案与解析 点击展开
答案:A。安全上:一旦能联网,用户代码可把含敏感信息的 trace 数据偷传出去,或以沙箱为跳板访问内网服务(SSRF)。确定性上:评估的灵魂是「同输入恒同分」,而网络请求会抖、超时、返回不同结果,源码原话甚至说「可能永不返回」。掐断网络同时焊死了安全口子和不确定性源头——一条约束,两重收益。
💭 发散思考(没有标准答案,动手或动脑想想)
  • 想象你要给一个客服机器人写 5 个评估器。哪些你会用代码 eval(确定/规则),哪些用 LLM-as-judge(概率/语义)?请各举几个,并说说为什么——以及如果某个信号「看起来客观但其实有灰区」(比如「回答是否礼貌」),你会怎么权衡?

Not every "is it good" needs an LLM to judge. Many quality signals are objective and deterministic: is the answer valid JSON? did it leak an email address? is it over the length limit? are the fields complete? These need no daydreaming judge—just a piece of code to compute, deterministic, fast, and free of LLM cost. This is code-based evaluation. But there's a sharp problem here: the evaluator code is written by users, and the platform must execute someone else's code—an enormous security risk. So this lesson's protagonists are two things: a dispatcher that abstracts away "where and how it runs", and a sandbox that runs user code under lock and key.

📋 Analogy
Last lesson's LLM-as-judge is like hiring a human judge: subjective, with judgment, but may score differently each time and charges an appearance fee. Code eval is like using a standard gauge / caliper: objective, the same reading a hundred times on the same part, nearly free. A gauge suits deterministic things like "is the dimension within spec", a judge suits subjective things like "is the design beautiful"—complementary, not substitutes. But there's a catch: this caliper was mailed in by someone else, and you dare not plug it straight into the control machine (what if it hides malicious instructions?). The safe move is to run it in an isolated inspection room (sandbox), door shut, phone line cut (no network)—it can only measure the part you hand in and hand the reading back, touching nothing outside.

Two judges: deterministic code vs probabilistic LLM

Lessons 29 and 31 are really two ways of producing the same score. Both hang off Lesson 30's scheduling pipeline—the only difference is the "execution" step: LLM-as-judge calls a model, code eval runs a function. Which to pick depends on whether what you're evaluating is subjective or objective:

one trace, two judges, same destination a trace's input/outputthe answer to judge LLM-as-judge (Lesson 29)probabilistic · subjective/semantic · costs LLMhelpfulness / tone / off-topic? code eval (this lesson)deterministic · objective/rule · nearly freevalid JSON / has email / length / exact match both produce a source=EVAL score
Complementary, not substitutes: what rules can express (format, length, keywords, exact match) goes to code eval—deterministic, fast, free; what needs semantic understanding (helpfulness, tone, relevance) goes to LLM-as-judge. Both attach to Lesson 30's scheduling pipeline, differing only at the "execution" step.
Example: a code eval function produces a score defexact_match(output, expected): ok = output.strip() == expected.strip() return{ "name": "exact_match", "value": 1 if ok else 0, "dataType": "NUMERIC", } # or: valid JSON? regex match? length threshold? writes one score name=exact_match value=1 · dataType=NUMERIC source=EVAL deterministic · reproducible · free · milliseconds
Not every eval needs an LLM (values illustrative): many judgments are deterministic — exact match, valid JSON, contains a keyword, within a length limit… plain code computes them, returns {{name, value, dataType}}, written back with source=EVAL. Versus an LLM judge (Lesson 29), code eval is reproducible, free and millisecond-fast — ideal where there's an objective criterion.
dimensionLLM-as-judge (Lesson 29)code eval (this lesson)
how it judgescalls a model for semantic judgmentruns a function you wrote
determinismprobabilistic (same input may differ)deterministic (same input, same score)
good atsubjective: helpfulness, tone, relevanceobjective: valid JSON, has PII, length, exact/regex match
cost/speedone LLM call each, slow, costs moneyone function run, fast, nearly free
riskthe model misjudges, has biasesmust execute user code → must sandbox

The dispatcher: a strategy pattern that abstracts away "where it runs"

The worker shouldn't care whether user code runs on AWS Lambda or locally—it just wants to say "run this code with this input and give me scores". That is the dispatcher interface: a minimal contract { name, dispatch(input) }. Two implementations hang beneath it, and resolveConfiguredCodeEvalDispatcher() picks one by env var—a textbook strategy pattern.

one interface, two execution backends (strategy pattern) worker exec stepdispatcher.dispatch(payload) interface CodeEvalDispatcher{ name; dispatch(input): Promise<{scores}> } AwsLambdaCodeEvalDispatchername="aws-lambda" · per-language Lambda sandboxpython→executor-python · TS→executor-node (prod) LocalCodeEvalDispatchername="insecure-local" · runs on node vmTS/JS only · name admits "insecure", local dev only
Strategy pattern: the worker depends only on the CodeEvalDispatcher interface (codeEvalDispatcherTypes.ts:119-121). resolveConfiguredCodeEvalDispatcher (codeEvalDispatchers.ts:13) picks by env.LANGFUSE_CODE_EVAL_DISPATCHER: Lambda (production, strong isolation) or local vm (dev, literally named insecure-local). Swapping backends changes not a line of the worker.
packages/shared/src/server/evals/codeEvalDispatcherTypes.ts · codeEvalDispatchers.tsinterface + selection
// minimal contract: a name + one dispatch method returning { scores:[…] }
export interface CodeEvalDispatcher {
  name: string;
  dispatch(input: DispatchInput): Promise<DispatchResult>;   // DispatchResult = { scores: at least 1 }
}

// pick the impl by env var — the strategy pattern's "selection point"
export function resolveConfiguredCodeEvalDispatcher(): CodeEvalDispatcher | null {
  if (dispatcher === "insecure-local") return new LocalCodeEvalDispatcher();  // node vm, TS/JS only
  if (dispatcher === "aws-lambda") return new AwsLambdaCodeEvalDispatcher({
    functionNameByLanguage: { PYTHON: "…-python", TYPESCRIPT: "…-node" } });
}

The sandbox's iron rules: no network, size caps, time limit

Since it runs user code, the platform must box it in with hard constraints. These aren't UI hints—they're enforced at the execution layer: exceed one and you get a specific error code. The most intriguing is "no network": it kills two birds, blocking security risk and preserving determinism.

enforced at the execution layer: exceed one → a specific error code sandbox (enforced) evaluate(payload) node vm / Lambda no fetch / require no network (two birds) blocks SSRF/exfil + determinism time limit over → TIMEOUT source ≤ 256KB → SOURCE_TOO_LARGE input ≤ 5.5MB → PAYLOAD_TOO_LARGE result ≤ 256KB → RESULT_TOO_LARGE result must be {scores:[≥1]} → INVALID_RESULT
iron rulethe limitwhy
no networkevaluator code may make no network requests (stated in the TIMEOUT error)① security: prevent data exfiltration & SSRF ② determinism: network = nondeterministic + may never return
time limitmust finish within the configured runtime, else TIMEOUTprevent infinite loops / hangs holding the sandbox
source size≤ 256 KB (SOURCE_TOO_LARGE)an evaluator should be small, not a whole program
input size≤ 5.5 MB (PAYLOAD_TOO_LARGE, source + variables)cap per-run feed, protect the dispatch path
result size≤ 256 KB (RESULT_TOO_LARGE)it returns scores, not bulk data
result shapemust be { scores: [≥1] }, each with name/dataType/value of matching type (INVALID_RESULT)malformed can't normalize into a score
codeEvalDispatcherTypes.ts · localCodeEvalDispatcher.tslimit constants + sandbox run
// three size red-lines (bytes), enforced at the execution layer (codeEvalDispatcherTypes.ts:4-6)
CODE_EVAL_SOURCE_MAX_BYTES          = 256 * 1024;        // source ≤256KB
CODE_EVAL_DISPATCH_PAYLOAD_MAX_BYTES = 5.5 * 1024 * 1024; // input ≤5.5MB
CODE_EVAL_DISPATCH_RESULT_MAX_BYTES  = 256 * 1024;        // result ≤256KB

// local dispatcher: runs on node's vm with a timeout; honestly named "insecure-local"
import * as vm from "node:vm";
const context = vm.createContext({ /* restricted globals, no fetch/require */ });
vm.runInContext("evaluate(payload)", context, { timeout });  // run user's evaluate(), result must be {scores}

And after it runs? Exactly like Lesson 29: the returned {scores:[…]} normalizes into scores tagged source=EVAL and flows back through Lesson 12's ingestion path into the scores table. Code eval starts no new machinery—it just swaps Lesson 30's "execution" step from "call the LLM" to "dispatch to the sandbox". Same scheduling, same score, same flow-back, only the judge changes from "a probabilistic model" to "a deterministic function". That's why Langfuse fits both evaluation kinds into the same JobExecution state machine.

1

extract variables

Same origin as the judge pipeline: pull input/output etc. from the trace/observation per the variable mapping (extractObservationVariables).

2

build payload

Pack the user's evaluator source + extracted variables + runtime.language into a CodeEvalPayload, first checking size red-lines (source ≤256KB, total ≤5.5MB).

3

dispatch to sandbox

dispatcher.dispatch(payload): Lambda invokes the per-language python/node function, or the local vm runs it—no network, time-limited.

4

validate result

The return must be {scores:[≥1]}, each score's name/dataType/value validated with matching type, result ≤256KB (parseDispatchResult).

5

normalize + flow back

Just like Lesson 29: tag source=EVAL, write back to the scores table via the ingestion path, mark the ticket COMPLETED.

🎯 Design trade-off
Why the dispatcher abstraction instead of calling Lambda directly? Because "where to run user code" is a decision that changes: cloud uses Lambda (strong isolation, on-demand scaling), while self-hosters may lack Lambda and just want to run locally. Hiding the execution backend behind a {name, dispatch} interface keeps the worker's logic oblivious to the backend—swapping impls or adding a new sandbox (say a future WASM runtime) changes not a line of the caller. That's the value of the strategy pattern: make "how" replaceable while "what" stays stable.

Why name the local dispatcher "insecure-local"? A rare bit of honesty. Node's vm module is not a real security sandbox—the experienced can escape a vm context. Langfuse doesn't hide it; it writes "insecure" right into the name, like a warning sticker in the source: this path is fit only for local dev; for production use real isolation like Lambda. Naming as documentation, putting the risk where it's most visible.

Why is "no network" the most crucial iron rule? It solves two problems at a stroke. On security: once user code can reach the network, it could quietly exfiltrate the trace data you fed it (possibly sensitive) or use the sandbox as a springboard into your internal network (SSRF). On determinism: the soul of evaluation is "same input, same score", yet network requests are inherently nondeterministic—endpoints flake, time out, return different results, even "may never return" (the source's own words). Cutting the network welds shut both the security hole and the source of nondeterminism. One constraint, two payoffs—good restrictions often are.
🎯 Key points
  • Code eval = deterministic evaluation: a function computes the score, suited to objective/rule signals (valid JSON, has PII, length, exact/regex match); complementary to LLM-as-judge (probabilistic/subjective), not a substitute.
  • Same pipeline: it attaches to Lesson 30's scheduling pipeline, swapping only the "execution" step from "call the LLM" to "dispatch to the sandbox"; results likewise normalize into source=EVAL scores that flow back—the same JobExecution state machine.
  • Dispatcher strategy pattern: the worker depends only on the {name, dispatch} interface; resolveConfiguredCodeEvalDispatcher picks by env var AwsLambda (production, per-language python/node sandbox) or local vm. Swapping backends doesn't touch the caller.
  • The local dispatcher is named "insecure-local": node's vm isn't a real sandbox, and the name says so—fit only for local dev; production uses real Lambda isolation. Naming as warning.
  • Sandbox iron rules: no network, time limit, source ≤256KB, input ≤5.5MB, result ≤256KB, result must be {scores:[≥1]}. "No network" kills two birds—blocking exfiltration/SSRF and preserving "same input, same score" determinism.

🧪 Self-test - think about the design

1. Langfuse has both LLM-as-judge (Lesson 29) and code eval (this lesson). Which statement most accurately characterizes their relationship?
  1. they use entirely separate scheduling and storage
  2. code eval can only produce boolean scores
  3. code eval is an upgraded replacement for LLM-as-judge
  4. complementary: code eval is deterministic, for objective/rule signals (valid JSON, has PII, length); LLM-as-judge is probabilistic, for subjective/semantic signals (helpfulness, tone). Both attach to the same scheduling pipeline, differing only at the 'execution' step
Show answer & explanation click to expand
Answer: D. Both are two ways of producing the same score, hanging off Lesson 30's pipeline, differing only at the 'execution' step: LLM-as-judge calls a model, code eval runs a function. Both normalize into source=EVAL scores flowing into the same table via the same JobExecution state machine. Rule-expressible goes to code eval (deterministic, free); semantic goes to LLM-as-judge.
2. Code eval executes user-written code. Langfuse's local dispatcher is named "insecure-local" right in the source. What does this naming reveal?
  1. local only supports Python
  2. local execution is slow
  3. it leaks environment variables
  4. an honest warning: it uses node's vm module, and vm is not a real security sandbox (escapable); naming as documentation, signaling this path is for local dev only, with production needing real isolation like Lambda
Show answer & explanation click to expand
Answer: D. Node's vm can create an isolated context to run code, but the consensus is it's insufficient for safely running untrusted code (known escapes exist). Langfuse doesn't gloss over it—writing insecure into the dispatcher's name is a source-level warning: fine for local dev, switch to AwsLambdaCodeEvalDispatcher (ephemeral, strong isolation, per-language python/node functions) in production.
3. The code-eval sandbox has an iron rule: evaluator code may make no network requests. Why is this 'no network' considered to kill two birds?
  1. it preserves security (networked user code could exfiltrate the trace data you fed it, or use the sandbox for SSRF into your internal network) and determinism (network is inherently nondeterministic—flaky, timing out, even may-never-return, breaking 'same input, same score')
  2. just to save bandwidth
  3. to make eval run faster
  4. to stop the evaluator from calling an LLM
Show answer & explanation click to expand
Answer: A. Security: once networked, user code can smuggle out sensitive trace data or use the sandbox as a springboard to internal services (SSRF). Determinism: the soul of eval is 'same input, same score', yet network requests flake, time out, return different results—the source even says 'may never return'. Cutting the network welds shut both the security hole and the nondeterminism source — one constraint, two payoffs.
💭 Open questions (no single right answer - just think or try)
  • Imagine writing 5 evaluators for a customer-service bot. Which would you do with code eval (deterministic/rule) and which with LLM-as-judge (probabilistic/semantic)? Give a few of each and say why—and if a signal 'looks objective but has a gray zone' (e.g. 'is the reply polite'), how would you weigh it?