Getting reliable JSON output from local LLMs with Ollama
Ollama’s JSON output is usually accurate, but in batch jobs, a single malformed response can halt the entire workflow. Here is how to build an unbreakable pipeline using schema definitions, validation, and a bounded retry loop.
The problem: “almost” JSON
The model produces valid JSON most of the time, yet an occasional introductory sentence, a missing quote, or an unexpected field name will break your pipeline. In batch processing, that small failure rate quickly turns into hundreds of errors. And syntax isn’t the only issue: valid JSON can still contain wrong field names or unexpected values.
Grouping the errors encountered when getting Ollama JSON output by type makes it easier to put a specific safeguard in place for each one:
| Error type | Example | Safeguard |
|---|---|---|
| Syntax error | Introductory sentence at the start, missing quote | format parameter and parse check |
| Missing field | One of the required fields is absent | required definition in schema and validation |
| Invalid value | A category name not on the list | Fixed options with enum |
| Semantic error | Valid JSON, but wrong classification | Few-shot prompt, sample checks, manual review queue |
Making the schema part of the request
Passing a JSON schema directly to Ollama’s format parameter forces the model to generate output that matches that structure. Passing only "json" enforces valid syntax but not the fields; passing a schema locks down both field names and data types.
{
"model": "llama3.1:8b",
"format": {
"type": "object",
"properties": {
"kategori": { "type": "string" },
"etiketler": { "type": "array", "items": { "type": "string" } }
},
"required": ["kategori", "etiketler"]
},
"stream": false
}Briefly explain the schema in the prompt as well: describing what each field means and listing allowed values (such as valid categories) makes it much easier for the model to pick the right value.
Keep the schema as small as possible. Using fixed-choice fields (enum) instead of free text reduces the chance of hallucinated values. Prefer flat fields over deeply nested structures; if needed, split the task into two separate requests.
Validation and retry loop
- Always run the response through a JSON Schema validator, even when you pass a schema.
- If validation fails, feed the error message back to the model and retry up to two times.
- On the third failure, move the record to a manual review queue without stopping the pipeline.
- Track the error rate; a spike usually signals a model or prompt change.
Keeping the temperature between 0 and 0.2 for structured output tasks significantly lowers the error rate.
Log the raw response, model name, and prompt version for every request. When an error occurs, these logs let you quickly tell whether the issue came from the model, the prompt, or the input data.
The example below sends a schema to Ollama’s /api/chat endpoint, validates the response, and retries up to two times with the error message. It uses Python’s jsonschema package for validation.
import json, requests
from jsonschema import validate, ValidationError
SCHEMA = {
"type": "object",
"properties": {
"kategori": {"type": "string", "enum": ["baski", "tarama", "diger"]},
"etiketler": {"type": "array", "items": {"type": "string"}},
},
"required": ["kategori", "etiketler"],
}
def classify(text, tries=3):
messages = [{"role": "user", "content": text}]
for _ in range(tries):
r = requests.post("http://localhost:11434/api/chat", json={
"model": "llama3.1:8b", "messages": messages,
"format": SCHEMA, "stream": False,
"options": {"temperature": 0},
}).json()
raw = r["message"]["content"]
try:
data = json.loads(raw)
validate(data, SCHEMA)
return data
except (json.JSONDecodeError, ValidationError) as e:
messages += [{"role": "assistant", "content": raw},
{"role": "user", "content": f"Geçersiz yanıt: {e}. Yalnızca şemaya uygun JSON döndür."}]
return None # elle inceleme kuyruğunaKeeping batch jobs running
- Process each record independently so a single failure doesn’t affect the rest.
- Save successful results immediately so the job can resume where it left off if interrupted.
- Write failed records to a separate table along with the raw response and error message.
- Generate a brief summary report at the end of the job showing success, retry, and manual review counts.
Whenever you change the prompt or model, re-run the same small test set. Comparing the error rate on this set shows the impact of the change without inspecting records one by one.
Choosing the model and quantization
Smaller models and lower quantization levels struggle more often to follow a schema. I covered the differences between levels in my note on comparing Q4, Q5, and Q8 in Ollama. Here is an example where I used this setup for print failure diagnosis: 3D Printing Desktop AI Assistant.
Conclusion
With the combination of a schema, validation, and bounded retries, Ollama JSON output becomes reliable enough for production. To integrate local models into your workflow, take a look at my local AI setup service.
