Baseten details Lumina, its self-refining LLM-judge system
Baseten has described Lumina, an evaluation engine that builds binary LLM-as-judge checks from observed model failures and refines them with customer input, giving it a reward signal for post-training open-source models.
Baseten described Lumina, an internal evaluation system meant to learn to judge model outputs the way a customer's subject-matter experts would. The company says static, off-the-shelf evaluations hide where models actually fail. Its view is that LLM-as-judge setups need outside information, and in practice that means the customer's own domain knowledge.
Lumina starts with a phase Baseten calls "discovery." A generic error-annotation meta-prompt is run over outputs from a range of models, from base models to state-of-the-art reasoning models, often on tens of thousands of examples. An annotation model, "usually Opus," then flags where each output fell short. Those errors are clustered into a hierarchical taxonomy, and each cluster becomes a binary PASS/FAIL check with a written rationale. Example checks include "does this medical note use correct tooth notation throughout?" Baseten avoids 1–10 scales, which it says LLMs struggle to calibrate. Instead, the binary results are added up into a quasi-Likert score, such as 18 of 25 checks passed.
The checks are then refined in a loop. Baseten aims for three properties: low variance across repeated runs, clear separation between acceptable and unacceptable outputs, and, most importantly, agreement with the customer. Lumina flags ambiguous edge cases, turns them into questions, sends them to customers in batches, and feeds the answers back into the pipeline.
Baseten says the resulting judges act as reward models for RL, as filters for SFT data through rejection sampling, and as signals for its own methods, iSFT and RGT. It uses them to help open-source models outperform closed generalist models. The post gives no benchmark results, and the available text ends before covering how Lumina handles gaps between the criteria and task context.