Baseten: Training loss predicts LLM-judge scores
Baseten reports that fine-tuning loss closely tracks LLM-as-judge pass rates on a clinical scribing task, suggesting teams could track loss instead of running costly evaluation suites at every step.
Inference company Baseten reports that training loss reliably predicts evaluation scores on a non-verifiable, domain-specific task. Prior work on loss and performance has mostly used benchmarks with ground-truth answers, such as maths, coding and multiple choice.
For the test, Baseten used a clinical scribe task. It trained 15 configurations: five Qwen-3 model sizes up to 32B parameters, each on three dataset sizes ranging from 1,293 to 4,673 examples. All runs used LoRA with fixed hyperparameters (rank 64, alpha 32, constant learning rate 2e-4). An ensemble of five LLM-judge prompts scored the outputs pass/fail on template adherence, medical inference, terminology, completeness and hallucinations. Baseten says it tuned the prompts to match the scribe customer's medical knowledge team.
Across all runs, final training loss explained 74% of the variance in pass rate (R² = 0.739, Pearson r = −0.86). Each 0.1 drop in loss corresponded to an 18.5-point rise in pass rate. With dataset size held fixed, R² rose to 0.904, 0.930 and 0.978. The largest-data runs showed signs of optimisation problems, with some ending at higher loss than medium-data runs.
The correlation held for both judges tested. Gemini-2.5-Flash and GPT-5-mini differed 2.8× in average pass rate (26.7% vs 9.5%) but had nearly identical correlations with loss (r = −0.86 and −0.87). Baseten reads this as a sign that the two judges measure the same underlying quality at different levels of strictness.
Baseten calls the study "admittedly a toy-scale exploration." It says it has seen the same pattern in production with up to a million training examples and 235B-parameter models, but published no data for those runs. The material provided cuts off partway through its analysis of model-size and data scaling.