Not every "is it good" needs an LLM to judge. Many quality signals are objective and deterministic: is the answer valid JSON? did it leak an email address? is it over the length limit? are the fields complete? These need no daydreaming judge—just a piece of code to compute, deterministic, fast, and free of LLM cost. This is code-based evaluation.
But there's a sharp problem here: the evaluator code is written by users, and the platform must execute someone else's code—an enormous security risk. So this lesson's protagonists are two things: a dispatcher that abstracts away "where and how it runs", and a sandbox that runs user code under lock and key.
📋 Analogy
Last lesson's LLM-as-judge is like hiring a human judge: subjective, with judgment, but may score differently each time and charges an appearance fee. Code eval is like using a standard gauge / caliper: objective, the same reading a hundred times on the same part, nearly free.
A gauge suits deterministic things like "is the dimension within spec", a judge suits subjective things like "is the design beautiful"—complementary, not substitutes.
But there's a catch: this caliper was mailed in by someone else, and you dare not plug it straight into the control machine (what if it hides malicious instructions?). The safe move is to run it in an isolated inspection room (sandbox), door shut, phone line cut (no network)—it can only measure the part you hand in and hand the reading back, touching nothing outside.
Two judges: deterministic code vs probabilistic LLM
Lessons 29 and 31 are really two ways of producing the same score. Both hang off Lesson 30's scheduling pipeline—the only difference is the "execution" step: LLM-as-judge calls a model, code eval runs a function. Which to pick depends on whether what you're evaluating is subjective or objective:
Complementary, not substitutes: what rules can express (format, length, keywords, exact match) goes to code eval—deterministic, fast, free; what needs semantic understanding (helpfulness, tone, relevance) goes to LLM-as-judge. Both attach to Lesson 30's scheduling pipeline, differing only at the "execution" step.
Not every eval needs an LLM (values illustrative): many judgments are deterministic — exact match, valid JSON, contains a keyword, within a length limit… plain code computes them, returns {{name, value, dataType}}, written back with source=EVAL. Versus an LLM judge (Lesson 29), code eval is reproducible, free and millisecond-fast — ideal where there's an objective criterion.
objective: valid JSON, has PII, length, exact/regex match
cost/speed
one LLM call each, slow, costs money
one function run, fast, nearly free
risk
the model misjudges, has biases
must execute user code → must sandbox
The dispatcher: a strategy pattern that abstracts away "where it runs"
The worker shouldn't care whether user code runs on AWS Lambda or locally—it just wants to say "run this code with this input and give me scores". That is the dispatcher interface: a minimal contract { name, dispatch(input) }. Two implementations hang beneath it, and resolveConfiguredCodeEvalDispatcher() picks one by env var—a textbook strategy pattern.
Strategy pattern: the worker depends only on the CodeEvalDispatcher interface (codeEvalDispatcherTypes.ts:119-121). resolveConfiguredCodeEvalDispatcher (codeEvalDispatchers.ts:13) picks by env.LANGFUSE_CODE_EVAL_DISPATCHER: Lambda (production, strong isolation) or local vm (dev, literally named insecure-local). Swapping backends changes not a line of the worker.
// minimal contract: a name + one dispatch method returning { scores:[…] }export interfaceCodeEvalDispatcher {
name: string;
dispatch(input: DispatchInput): Promise<DispatchResult>; // DispatchResult = { scores: at least 1 }
}
// pick the impl by env var — the strategy pattern's "selection point"export functionresolveConfiguredCodeEvalDispatcher(): CodeEvalDispatcher | null {
if (dispatcher === "insecure-local") return new LocalCodeEvalDispatcher(); // node vm, TS/JS onlyif (dispatcher === "aws-lambda") return new AwsLambdaCodeEvalDispatcher({
functionNameByLanguage: { PYTHON: "…-python", TYPESCRIPT: "…-node" } });
}
The sandbox's iron rules: no network, size caps, time limit
Since it runs user code, the platform must box it in with hard constraints. These aren't UI hints—they're enforced at the execution layer: exceed one and you get a specific error code. The most intriguing is "no network": it kills two birds, blocking security risk and preserving determinism.
iron rule
the limit
why
no network
evaluator code may make no network requests (stated in the TIMEOUT error)
① security: prevent data exfiltration & SSRF ② determinism: network = nondeterministic + may never return
time limit
must finish within the configured runtime, else TIMEOUT
prevent infinite loops / hangs holding the sandbox
source size
≤ 256 KB (SOURCE_TOO_LARGE)
an evaluator should be small, not a whole program
input size
≤ 5.5 MB (PAYLOAD_TOO_LARGE, source + variables)
cap per-run feed, protect the dispatch path
result size
≤ 256 KB (RESULT_TOO_LARGE)
it returns scores, not bulk data
result shape
must be { scores: [≥1] }, each with name/dataType/value of matching type (INVALID_RESULT)
malformed can't normalize into a score
codeEvalDispatcherTypes.ts · localCodeEvalDispatcher.tslimit constants + sandbox run
// three size red-lines (bytes), enforced at the execution layer (codeEvalDispatcherTypes.ts:4-6)
CODE_EVAL_SOURCE_MAX_BYTES = 256 * 1024; // source ≤256KB
CODE_EVAL_DISPATCH_PAYLOAD_MAX_BYTES = 5.5 * 1024 * 1024; // input ≤5.5MB
CODE_EVAL_DISPATCH_RESULT_MAX_BYTES = 256 * 1024; // result ≤256KB// local dispatcher: runs on node's vm with a timeout; honestly named "insecure-local"import * as vm from"node:vm";
const context = vm.createContext({ /* restricted globals, no fetch/require */ });
vm.runInContext("evaluate(payload)", context, { timeout }); // run user's evaluate(), result must be {scores}
And after it runs? Exactly like Lesson 29: the returned {scores:[…]} normalizes into scores tagged source=EVAL and flows back through Lesson 12's ingestion path into the scores table. Code eval starts no new machinery—it just swaps Lesson 30's "execution" step from "call the LLM" to "dispatch to the sandbox". Same scheduling, same score, same flow-back, only the judge changes from "a probabilistic model" to "a deterministic function". That's why Langfuse fits both evaluation kinds into the same JobExecution state machine.
1
extract variables
Same origin as the judge pipeline: pull input/output etc. from the trace/observation per the variable mapping (extractObservationVariables).
2
build payload
Pack the user's evaluator source + extracted variables + runtime.language into a CodeEvalPayload, first checking size red-lines (source ≤256KB, total ≤5.5MB).
3
dispatch to sandbox
dispatcher.dispatch(payload): Lambda invokes the per-language python/node function, or the local vm runs it—no network, time-limited.
4
validate result
The return must be {scores:[≥1]}, each score's name/dataType/value validated with matching type, result ≤256KB (parseDispatchResult).
5
normalize + flow back
Just like Lesson 29: tag source=EVAL, write back to the scores table via the ingestion path, mark the ticket COMPLETED.
🎯 Design trade-off
Why the dispatcher abstraction instead of calling Lambda directly? Because "where to run user code" is a decision that changes: cloud uses Lambda (strong isolation, on-demand scaling), while self-hosters may lack Lambda and just want to run locally. Hiding the execution backend behind a {name, dispatch} interface keeps the worker's logic oblivious to the backend—swapping impls or adding a new sandbox (say a future WASM runtime) changes not a line of the caller. That's the value of the strategy pattern: make "how" replaceable while "what" stays stable.
Why name the local dispatcher "insecure-local"? A rare bit of honesty. Node's vm module is not a real security sandbox—the experienced can escape a vm context. Langfuse doesn't hide it; it writes "insecure" right into the name, like a warning sticker in the source: this path is fit only for local dev; for production use real isolation like Lambda. Naming as documentation, putting the risk where it's most visible.
Why is "no network" the most crucial iron rule? It solves two problems at a stroke. On security: once user code can reach the network, it could quietly exfiltrate the trace data you fed it (possibly sensitive) or use the sandbox as a springboard into your internal network (SSRF). On determinism: the soul of evaluation is "same input, same score", yet network requests are inherently nondeterministic—endpoints flake, time out, return different results, even "may never return" (the source's own words). Cutting the network welds shut both the security hole and the source of nondeterminism. One constraint, two payoffs—good restrictions often are.
🎯 Key points
Code eval = deterministic evaluation: a function computes the score, suited to objective/rule signals (valid JSON, has PII, length, exact/regex match); complementary to LLM-as-judge (probabilistic/subjective), not a substitute.
Same pipeline: it attaches to Lesson 30's scheduling pipeline, swapping only the "execution" step from "call the LLM" to "dispatch to the sandbox"; results likewise normalize into source=EVAL scores that flow back—the same JobExecution state machine.
Dispatcher strategy pattern: the worker depends only on the {name, dispatch} interface; resolveConfiguredCodeEvalDispatcher picks by env var AwsLambda (production, per-language python/node sandbox) or local vm. Swapping backends doesn't touch the caller.
The local dispatcher is named "insecure-local": node's vm isn't a real sandbox, and the name says so—fit only for local dev; production uses real Lambda isolation. Naming as warning.
Sandbox iron rules: no network, time limit, source ≤256KB, input ≤5.5MB, result ≤256KB, result must be {scores:[≥1]}. "No network" kills two birds—blocking exfiltration/SSRF and preserving "same input, same score" determinism.
🧪 Self-test - think about the design
1. Langfuse has both LLM-as-judge (Lesson 29) and code eval (this lesson). Which statement most accurately characterizes their relationship?
they use entirely separate scheduling and storage
code eval can only produce boolean scores
code eval is an upgraded replacement for LLM-as-judge
complementary: code eval is deterministic, for objective/rule signals (valid JSON, has PII, length); LLM-as-judge is probabilistic, for subjective/semantic signals (helpfulness, tone). Both attach to the same scheduling pipeline, differing only at the 'execution' step
Show answer & explanation click to expand
Answer: D. Both are two ways of producing the same score, hanging off Lesson 30's pipeline, differing only at the 'execution' step: LLM-as-judge calls a model, code eval runs a function. Both normalize into source=EVAL scores flowing into the same table via the same JobExecution state machine. Rule-expressible goes to code eval (deterministic, free); semantic goes to LLM-as-judge.
2. Code eval executes user-written code. Langfuse's local dispatcher is named "insecure-local" right in the source. What does this naming reveal?
local only supports Python
local execution is slow
it leaks environment variables
an honest warning: it uses node's vm module, and vm is not a real security sandbox (escapable); naming as documentation, signaling this path is for local dev only, with production needing real isolation like Lambda
Show answer & explanation click to expand
Answer: D. Node's vm can create an isolated context to run code, but the consensus is it's insufficient for safely running untrusted code (known escapes exist). Langfuse doesn't gloss over it—writing insecure into the dispatcher's name is a source-level warning: fine for local dev, switch to AwsLambdaCodeEvalDispatcher (ephemeral, strong isolation, per-language python/node functions) in production.
3. The code-eval sandbox has an iron rule: evaluator code may make no network requests. Why is this 'no network' considered to kill two birds?
it preserves security (networked user code could exfiltrate the trace data you fed it, or use the sandbox for SSRF into your internal network) and determinism (network is inherently nondeterministic—flaky, timing out, even may-never-return, breaking 'same input, same score')
just to save bandwidth
to make eval run faster
to stop the evaluator from calling an LLM
Show answer & explanation click to expand
Answer: A. Security: once networked, user code can smuggle out sensitive trace data or use the sandbox as a springboard to internal services (SSRF). Determinism: the soul of eval is 'same input, same score', yet network requests flake, time out, return different results—the source even says 'may never return'. Cutting the network welds shut both the security hole and the nondeterminism source — one constraint, two payoffs.
💭 Open questions (no single right answer - just think or try)
Imagine writing 5 evaluators for a customer-service bot. Which would you do with code eval (deterministic/rule) and which with LLM-as-judge (probabilistic/semantic)? Give a few of each and say why—and if a signal 'looks objective but has a gray zone' (e.g. 'is the reply polite'), how would you weigh it?