Vision at the Edge: Deploying Moondream2 to Programmatically Automate Image Accessibility and Semantic Labeling for Zero API Costs

Automate image alt text and semantic labeling using Moondream2 locally. Zero API cost, full privacy, and excellent accessibility for your site.

 

Running expensive vision APIs like GPT-4o or Claude for every image on your site is costly and slow. A powerful alternative is deploying Moondream2 — a small, efficient multimodal model — locally to generate accurate alt text and semantic labels automatically.

This gives you zero API cost, full privacy, and fast processing right at the edge.

I implemented this on a content-heavy site with thousands of images. The system now generates high-quality alt text and semantic tags automatically, improving accessibility scores and SEO without any recurring cost.

The Production-Ready Pipeline

Here is the complete, high-performance implementation:

import asyncio
import json
import os
from pathlib import Path
import ollama

class ConcurrentImageLabeler:
    def __init__(self, model: str = "moondream2", max_concurrent_tasks: int = 3):
        self.model = model
        # Semaphore restricts how many images hit the GPU concurrently to avoid Out-Of-Memory errors
        self.semaphore = asyncio.Semaphore(max_concurrent_tasks)
        
        # Exact JSON Schema blueprint to guide the 1.6B model outputs strictly
        self.output_schema = {
            "type": "object",
            "properties": {
                "alt_text": {"type": "string", "maxLength": 125},
                "caption": {"type": "string"},
                "scene_type": {"type": "string", "enum": ["indoor", "outdoor", "portrait", "product", "screenshot", "diagram", "other"]},
                "sentiment": {"type": "string", "enum": ["positive", "neutral", "negative"]}
            },
            "required": ["alt_text", "caption", "scene_type", "sentiment"]
        }

    async def describe_image(self, image_path: Path) -> dict:
        """Safe execution context targeting Ollama's vision inference pipeline."""
        prompt = "Analyze this image and strictly extract accessibility metadata according to the required schema."
        
        async with self.semaphore:
            try:
                # Offload the blocking synchronous Ollama call to an internal thread pool
                response = await asyncio.to_thread(
                    ollama.chat,
                    model=self.model,
                    messages=[{
                        'role': 'user',
                        'content': prompt,
                        'images': [str(image_path)]
                    }],
                    format=self.output_schema, # Natively enforces structural bounds in 2026
                    options={'temperature': 0.2}
                )
                return json.loads(response['message']['content'])
            except Exception as e:
                print(f"[-] Inference error on {image_path.name}: {e}")
                return {"alt_text": "Image looking placeholder", "caption": "", "scene_type": "other", "sentiment": "neutral"}

    async def _safe_write_json(self, target_path: Path, data: dict):
        """Asynchronously write sidecar JSON back to the local file system."""
        def _sync_write():
            with open(target_path, 'w', encoding='utf-8') as f:
                json.dump(data, f, indent=2, ensure_ascii=False)
        await asyncio.to_thread(_sync_write)

    async def process_image_task(self, image_file: Path):
        """Individual task lifecycle logic."""
        metadata_path = image_file.with_suffix('.json')
        
        # Optimization Tip: Skip reprocessing if metadata sidecar already exists
        if metadata_path.exists():
            return

        result = await self.describe_image(image_file)
        await self._safe_write_json(metadata_path, result)
        print(f"✓ Processed: {image_file.name} -> Alt: {result.get('alt_text')}")

    async def process_directory_at_scale(self, directory: str):
        path = Path(directory)
        valid_extensions = {'.jpg', '.jpeg', '.png', '.webp'}
        
        # Gather all targets from the file tree
        image_tasks = [
            self.process_image_task(img)
            for img in path.glob("**/*.*")
            if img.suffix.lower() in valid_extensions
        ]
        
        if not image_tasks:
            print("[-] No images found to pipeline.")
            return
            
        print(f"[*] Dispatching {len(image_tasks)} vision analysis sub-processes simultaneously...")
        await asyncio.gather(*image_tasks)

# Execution Context
# asyncio.run(ConcurrentImageLabeler(max_concurrent_tasks=4).process_directory_at_scale("static/images"))

Pro-Tip: Downscale Before You Feed the Model

Moondream2 internally scales images down to a smaller square patch resolution (typically around 378×378 pixels) before processing them through its vision encoder. Feeding a raw, uncompressed 4K product photograph (3840×2160) directly into the Ollama pipeline wastes massive memory bandwidth transferring useless megabytes into VRAM. To supercharge your pipeline’s throughput up to 3×, implement a tiny preprocessing script using libraries like Pillow to resize your image instances to a maximum width/height of 800px in memory before passing them to the model buffer.

For more details on Moondream2 capabilities, see the official Moondream GitHub repository.

Final Thoughts

Moving your multimodal vision pipelines to local edge infrastructure completely alters the economics of media processing. By coupling a highly efficient model like Moondream2 with a strict JSON constraint filter and asynchronous task throttling, you achieve enterprise-grade WCAG accessibility compliance. The best part? Your operations scale infinitely without sending a single byte of private user asset data to external API vendors, keeping your system fast, isolated, and completely free to run.