Getting reliable JSON output from local LLMs with Ollama

Mehmet Akar 4 min read
JSON schema validation workflow with Ollama

Ollama’s JSON output is usually accurate, but in batch jobs, a single malformed response can halt the entire workflow. Here is how to build an unbreakable pipeline using schema definitions, validation, and a bounded retry loop.

The problem: “almost” JSON

The model produces valid JSON most of the time, yet an occasional introductory sentence, a missing quote, or an unexpected field name will break your pipeline. In batch processing, that small failure rate quickly turns into hundreds of errors. And syntax isn’t the only issue: valid JSON can still contain wrong field names or unexpected values.

Grouping the errors encountered when getting Ollama JSON output by type makes it easier to put a specific safeguard in place for each one:

Error typeExampleSafeguard
Syntax errorIntroductory sentence at the start, missing quoteformat parameter and parse check
Missing fieldOne of the required fields is absentrequired definition in schema and validation
Invalid valueA category name not on the listFixed options with enum
Semantic errorValid JSON, but wrong classificationFew-shot prompt, sample checks, manual review queue

Making the schema part of the request

Passing a JSON schema directly to Ollama’s format parameter forces the model to generate output that matches that structure. Passing only "json" enforces valid syntax but not the fields; passing a schema locks down both field names and data types.

ollama/request.json
{
  "model": "llama3.1:8b",
  "format": {
    "type": "object",
    "properties": {
      "kategori": { "type": "string" },
      "etiketler": { "type": "array", "items": { "type": "string" } }
    },
    "required": ["kategori", "etiketler"]
  },
  "stream": false
}

Briefly explain the schema in the prompt as well: describing what each field means and listing allowed values (such as valid categories) makes it much easier for the model to pick the right value.

Keep the schema as small as possible. Using fixed-choice fields (enum) instead of free text reduces the chance of hallucinated values. Prefer flat fields over deeply nested structures; if needed, split the task into two separate requests.

Validation and retry loop

  • Always run the response through a JSON Schema validator, even when you pass a schema.
  • If validation fails, feed the error message back to the model and retry up to two times.
  • On the third failure, move the record to a manual review queue without stopping the pipeline.
  • Track the error rate; a spike usually signals a model or prompt change.
NOTE

Keeping the temperature between 0 and 0.2 for structured output tasks significantly lowers the error rate.

Log the raw response, model name, and prompt version for every request. When an error occurs, these logs let you quickly tell whether the issue came from the model, the prompt, or the input data.

The example below sends a schema to Ollama’s /api/chat endpoint, validates the response, and retries up to two times with the error message. It uses Python’s jsonschema package for validation.

classify.py
import json, requests
from jsonschema import validate, ValidationError

SCHEMA = {
    "type": "object",
    "properties": {
        "kategori": {"type": "string", "enum": ["baski", "tarama", "diger"]},
        "etiketler": {"type": "array", "items": {"type": "string"}},
    },
    "required": ["kategori", "etiketler"],
}

def classify(text, tries=3):
    messages = [{"role": "user", "content": text}]
    for _ in range(tries):
        r = requests.post("http://localhost:11434/api/chat", json={
            "model": "llama3.1:8b", "messages": messages,
            "format": SCHEMA, "stream": False,
            "options": {"temperature": 0},
        }).json()
        raw = r["message"]["content"]
        try:
            data = json.loads(raw)
            validate(data, SCHEMA)
            return data
        except (json.JSONDecodeError, ValidationError) as e:
            messages += [{"role": "assistant", "content": raw},
                         {"role": "user", "content": f"Geçersiz yanıt: {e}. Yalnızca şemaya uygun JSON döndür."}]
    return None  # elle inceleme kuyruğuna

Keeping batch jobs running

  1. Process each record independently so a single failure doesn’t affect the rest.
  2. Save successful results immediately so the job can resume where it left off if interrupted.
  3. Write failed records to a separate table along with the raw response and error message.
  4. Generate a brief summary report at the end of the job showing success, retry, and manual review counts.
TIP

Whenever you change the prompt or model, re-run the same small test set. Comparing the error rate on this set shows the impact of the change without inspecting records one by one.

Choosing the model and quantization

Smaller models and lower quantization levels struggle more often to follow a schema. I covered the differences between levels in my note on comparing Q4, Q5, and Q8 in Ollama. Here is an example where I used this setup for print failure diagnosis: 3D Printing Desktop AI Assistant.

Conclusion

With the combination of a schema, validation, and bounded retries, Ollama JSON output becomes reliable enough for production. To integrate local models into your workflow, take a look at my local AI setup service.

More notes

All notes →