Ditching OpenAI: Hosting Ollama and Structuring JSON Outputs with Local SLMs for Zero-Cost Agentic Workflows

Run local SLMs with Ollama for zero-cost agentic workflows. Learn Pydantic schema enforcement, real hardware requirements, and production best practices.

Running AI agents with paid APIs quickly becomes expensive, especially with complex agentic workflows involving multiple tool calls and reasoning steps. The practical alternative in 2026 is running strong Small Language Models locally with Ollama and enforcing strict structured JSON outputs.

This gives you zero recurring cost, complete data privacy, and predictable latency.

I recently migrated several internal agents from OpenAI to local models. The monthly bill went from $60+ down to nearly zero (only electricity), while keeping acceptable performance for automation and tool-calling tasks.

Setting Up Ollama

Ollama makes local deployment straightforward. For agentic workflows, I recommend:

• phi4:14b — excellent reasoning speed/quality balance
• qwen2.5:14b — best multilingual performance
• llama3.2:3b — for very fast, lightweight tasks

Enforcing Schema via Pydantic and Ollama

Simply asking for JSON is unreliable. The robust way is to combine Pydantic with Ollama’s native structured output support.

Here is a production-ready pattern:

import ollama
from pydantic import BaseModel, Field
from typing import Dict, Any

class ActionPayload(BaseModel):
    tool_name: str = Field(description="The specific tool function to trigger")
    arguments: Dict[str, Any] = Field(description="Key-value arguments required by the tool")
    confidence_score: float = Field(description="Confidence rating between 0.0 and 1.0")
    reasoning: str = Field(description="The internal logic behind this action step")

try:
    response = ollama.chat(
        model='phi4:14b',
        messages=[{
            'role': 'user',
            'content': 'The database CPU utilization is at 96% and queries are queuing. Decide on a mitigation step.'
        }],
        format=ActionPayload.model_json_schema(),   # Force strict schema
        options={'temperature': 0.1}
    )

    validated_action = ActionPayload.model_validate_json(response['message']['content'])
    print(validated_action.model_dump_json(indent=2))

except Exception as e:
    print(f"Schema validation failed: {e}")
    # Add retry logic here

Production Considerations

When deploying Ollama in dockerized environments, always set OLLAMA_NUM_PARALLEL (e.g. 4) according to your expected concurrent load. Monitor VRAM usage carefully — larger models can cause CUDA out-of-memory errors that crash the entire server if you share the GPU with other services. Enforce hard limits in your docker-compose resource allocation.

Hardware & VRAM Allocation Cheat Sheet

To avoid painful OOM crashes, use this practical mapping:

• Llama 3.2 3B → ~3.2 GB VRAM → Entry-level VPS or MacBook Air (fast classification)
• Mistral-Nemo 12B → ~10 GB VRAM → RTX 4060 Ti 16GB (good context extraction)
• Phi-4 14B → ~12 GB VRAM → RTX 4080 16GB / Mac Studio (strong reasoning)
• Qwen 2.5 14B → ~12 GB VRAM → RTX 4080 16GB (best multilingual tool calling)

Always leave 2–3 GB buffer for context overhead.

Final Thoughts

Ditching OpenAI for local SLMs with Ollama is now a mature and highly practical choice. With proper schema enforcement using Pydantic and careful hardware planning, you can build reliable zero-cost agentic systems. For more details on structured outputs, check the Ollama Python Library documentation. You can also explore available models on the Ollama Library.