Summarize the entire corpus, one JSONL line per document
Meeting notes, support threads, filings, call transcripts: summarization backlogs grow faster than anyone reads. A batch job turns the pile into consistent, structured summaries by tomorrow morning.
- Consistent format across docs
- 128K context per request
- Structured JSON summaries
- 50% off realtime
Built for this shape of work.
Independent records, no user waiting, and a real budget: the profile the 50% batch discount is priced for.
Nobody is waiting on summary #38,412
Interactive summarization is a chat feature. Corpus summarization is a pipeline: high volume, zero latency requirement, and a real budget. The 50% batch discount exists for exactly this split.
Uniform output across every document
One system prompt defines the summary shape (length, sections, JSON fields) for the whole file. Uniformity is what makes summaries loadable into search indexes, CRMs, and review tools.
Long documents fit natively
Gemma 4 31B covers 128K tokens per batch request, so the map step handles most documents whole, and the reduce step fits a few hundred per-document summaries in one request.
One JSONL file, one job.
Each line is a complete OpenAI-compatible request with a custom_id that is echoed on the matching output row. Up to 50,000 records and 200MB per job.
{"custom_id":"call-2026-07-30-a","method":"POST","url":"/v1/chat/completions","body":{"model":"openrelay/gemma-4-31b","messages":[{"role":"system","content":"Summarize the transcript as JSON: {\"tldr\" (2 sentences), \"decisions\": [], \"action_items\": [{\"owner\", \"task\"}], \"risks\": []}."},{"role":"user","content":"[00:00] Dana: Let's start with the Q3 capacity plan..."}]}}
{"custom_id":"thread-99120","method":"POST","url":"/v1/chat/completions","body":{"model":"openrelay/gemma-4-31b","messages":[{"role":"system","content":"Summarize this support thread in 3 sentences: problem, resolution, follow-up owed."},{"role":"user","content":"Customer: Our webhook deliveries stopped after..."}]}}from openai import OpenAI
client = OpenAI(
base_url="https://inference.openrelay.inc/v1",
api_key="or_••••••••", # same SDK, new base URL
)
# summarize 20,000 call transcripts into structured briefs
f = client.files.create(file=open("batch.jsonl", "rb"), purpose="batch")
batch = client.batches.create(
input_file_id=f.id,
endpoint="/v1/chat/completions",
completion_window="24h", # billed at 50% of realtime
)
# poll: validating → in_progress → completed
batch = client.batches.retrieve(batch.id)
print(batch.status, batch.request_counts)
# JSONL of {custom_id, response}; failures land in error_file_id
results = client.files.content(batch.output_file_id)The right models for this job.
Batch rates are 50% off the realtime per-token catalog rates, per 1M tokens.
See the full catalogGemma 4 31B
Google · 256K context
openrelay/gemma-4-31b
$0.495 / $0.745
batch input / output per 1M
128K context in batch covers most documents whole, and the same model runs the reduce step over the per-document summaries.
What separates a good run from a re-run.
Prompt and file patterns learned from real jobs, so the first submission is the one that counts.
Specify length in sentences, structure in JSON
'Summarize' alone produces wildly variable output. 'Three sentences: problem, resolution, follow-up' or an explicit JSON schema keeps 20,000 summaries interchangeable.
Map-reduce with two batch jobs
Job one summarizes each document. Job two feeds those summaries to a long-context model for the collection-level brief. Both are plain JSONL files; the second is generated from the first's output file.
Keep the source id in custom_id
custom_id is echoed on every output row. Encoding the document key there (crm-4471, call-2026-07-30-a) means results join back to your system of record with no lookup table.
Extract while you summarize
The same request can return the summary plus entities, sentiment, or action items as separate JSON fields. One pass over the corpus instead of three.
Common questions.
What is the best way to summarize thousands of documents with AI?
Batch, not a loop over a realtime API. Emit one JSONL record per document with a system prompt that pins the summary format, submit the file, and download structured summaries within the 24h window. You skip rate limits and retries and pay half the per-token rate.
How much does bulk document summarization cost?
Batch bills at half the online rate. On Gemma 4 31B a 3,000-token document with a 200-token summary costs about $0.0016, so a 100,000-document corpus lands around $160.
How do I summarize documents longer than the context window?
Most documents fit in 128K tokens whole. For the ones that do not, chunk by section, summarize chunks in the same job, then run a short second job that merges each document's chunk summaries.
Can the summaries follow my exact format?
Yes. The system prompt is the format contract: field names, section order, length limits, tone. Ask for the JSON object only and validate each row after the run; re-submit the few that fail to parse as a small follow-up job.
More batch workloads
OCR the whole archive, priced per token instead of per page
OCR thousands of PDFs, scans, invoices, and receipts with DeepSeek-OCR 2 through a batch API.
Sentiment for every review, with the why attached
Run sentiment analysis over reviews, surveys, and tickets in bulk with an LLM batch API.
Free text in, your schema out
Extract structured JSON from documents, emails, and free text with batch LLM jobs.
Running agent pipelines instead of flat request files? See the agentic batch API.
Ship the first job today.
Grab an API key, upload a JSONL file, and run open models at half the realtime cost. No contract, no minimums. Add a card and make a deposit to start.