LLM-as-a-judge evals, 50,000 grades per job
Score model outputs, agent transcripts, and RAG answers with a judge model, in bulk. Upload one JSONL file of grading prompts, get back a JSONL of verdicts, and pay half the realtime per-token rate.
- Judge 50k outputs per job
- Rubric prompts in JSONL
- Failed rows isolated
- 50% off realtime
Built for this shape of work.
Independent records, no user waiting, and a real budget: the profile the 50% batch discount is priced for.
Evals are the canonical batch workload
Nothing about grading yesterday's outputs needs a streaming connection. A judge run is thousands of independent requests with no latency requirement, which is exactly the shape the 50% batch discount is priced for.
Reproducible eval runs
One input file, one output file, one job id. Re-running an eval after a prompt change means re-submitting the same JSONL, and diffing two result files gives you the regression report.
A bad row never sinks the run
Malformed records and model errors land in a separate error file with their custom_id. The other 49,990 grades complete, and you re-submit only the failures.
One JSONL file, one job.
Each line is a complete OpenAI-compatible request with a custom_id that is echoed on the matching output row. Up to 50,000 records and 200MB per job.
{"custom_id":"eval-001","method":"POST","url":"/v1/chat/completions","body":{"model":"openrelay/gemma-4-31b","messages":[{"role":"system","content":"You are an evaluation judge. Score the answer 1-5 against the rubric and reply as JSON: {\"score\", \"rationale\"}."},{"role":"user","content":"Rubric: factual accuracy.\nQuestion: When did the Apollo 11 mission land?\nAnswer to grade: July 20, 1969."}]}}
{"custom_id":"eval-002","method":"POST","url":"/v1/chat/completions","body":{"model":"openrelay/gemma-4-31b","messages":[{"role":"system","content":"You are an evaluation judge. Given two answers, reply with the better one: A or B, plus one sentence of rationale."},{"role":"user","content":"Question: Summarize the refund policy.\nA: ...\nB: ..."}]}}from openai import OpenAI
client = OpenAI(
base_url="https://inference.openrelay.inc/v1",
api_key="or_••••••••", # same SDK, new base URL
)
# grade 50,000 model outputs against a rubric overnight
f = client.files.create(file=open("batch.jsonl", "rb"), purpose="batch")
batch = client.batches.create(
input_file_id=f.id,
endpoint="/v1/chat/completions",
completion_window="24h", # billed at 50% of realtime
)
# poll: validating → in_progress → completed
batch = client.batches.retrieve(batch.id)
print(batch.status, batch.request_counts)
# JSONL of {custom_id, response}; failures land in error_file_id
results = client.files.content(batch.output_file_id)The right models for this job.
Batch rates are 50% off the realtime per-token catalog rates, per 1M tokens.
See the full catalogGemma 4 31B
Google · 256K context
openrelay/gemma-4-31b
$0.495 / $0.745
batch input / output per 1M
The batch judge: rubric scoring, pairwise comparison, and agent-trajectory review, with {score, verdict, rationale} as JSON per row. 128K context per request fits long transcripts without chunking.
What separates a good run from a re-run.
Prompt and file patterns learned from real jobs, so the first submission is the one that counts.
Pin the judge, version the rubric
Put the rubric in the system message and keep it identical across the whole file. When the rubric changes, that is a new eval version: new file, new job, comparable result sets.
Ask for JSON verdicts
A judge that answers in prose needs a second parsing pass. Constrain the output shape in the prompt ({score, verdict, rationale}) so the results file is machine-readable as-is.
Use pairwise comparison for subtle regressions
Absolute 1-5 scores drift between runs. Putting the old and new answer in one prompt and asking A-or-B is more stable, and each comparison is still just one JSONL line.
Judge agent trajectories, not final answers alone
For agent evaluation, paste the full tool-call transcript and ask the judge whether each step was justified. Long transcripts are cheap when the job is billed at batch rates.
Common questions.
What is LLM-as-a-judge evaluation?
LLM-as-a-judge uses a strong model to score another model's outputs against a rubric: correctness, helpfulness, safety, or a pairwise better-of-two verdict. It replaces slow human review for the bulk of eval work, with humans auditing a sample. Because every grade is an independent request, judge runs map directly onto a batch API.
How do I run LLM evals in batch instead of an eval framework?
Eval frameworks loop over your dataset calling a realtime API. With batch you emit the same grading prompts as JSONL lines, submit one job, and collect verdicts from the output file. You keep your dataset and rubric code and drop the rate-limit handling, retry logic, and the 2x price.
How much does it cost to grade 50,000 outputs?
Cost is per token at half the realtime rate. A typical grading prompt of about 700 input and 150 output tokens on Gemma 4 31B runs at $0.495 per 1M input and $0.745 per 1M output tokens in batch, about $23 for the full 50,000 rows.
Which model should I use as a judge?
Use a model at least as capable as the one being judged. In batch that is Gemma 4 31B, which handles rubric scoring and pairwise verdicts and fits 128K tokens of transcript per request. To grade outputs from a model stronger than Gemma 4 31B, run the judge on GLM 5.2 through the realtime API instead.
Can I evaluate agents with a batch job?
Yes. Serialize each agent trajectory (messages, tool calls, results) into one grading prompt per line. The judge scores step quality and final outcome. This is the standard pattern for agent evaluation at scale, and it pairs with the agentic batch API for running the agent steps themselves.
More batch workloads
Synthetic data and distillation, a million examples per night
Generate synthetic training data and distill models with batch LLM jobs: instruction pairs, teacher outputs, augmentations, at 50% off realtime.
Classify a million documents with a prompt, not a training run
Classify documents, tickets, and text at scale with an LLM batch API.
Free text in, your schema out
Extract structured JSON from documents, emails, and free text with batch LLM jobs.
Running agent pipelines instead of flat request files? See the agentic batch API.
Ship the first job today.
Grab an API key, upload a JSONL file, and run open models at half the realtime cost. No contract, no minimums. Add a card and make a deposit to start.