Local RAG Without LangChain: Building a Dependency-Free Vector Search and Context Injection Pipeline in 50 Lines of Python

Build a clean, dependency-free local RAG pipeline in Python without LangChain. Fast vector search with proper chunking and relevance filtering. 

Most RAG (Retrieval-Augmented Generation) tutorials immediately push heavy frameworks like LangChain or LlamaIndex. These tools add dozens of dependencies, hidden abstraction layers, and make debugging difficult when things go wrong. You can build a clean, fast, and fully transparent local RAG pipeline using only a few lightweight libraries and standard Python.

This approach gives you complete control, better performance, and much easier debugging. I built this exact pipeline for a private company knowledge base assistant that runs entirely locally with Ollama.

The Complete Pipeline

Here is the full, production-capable code:

import os
import json
from pathlib import Path
from sentence_transformers import SentenceTransformer
import numpy as np
from typing import List, Dict

class SimpleRAG:
    def __init__(self, docs_dir: str = "knowledge", model_name: str = "all-MiniLM-L6-v2"):
        self.model = SentenceTransformer(model_name)
        self.docs_dir = Path(docs_dir)
        self.chunks: List[Dict[str, str]] = []
        self.embeddings = None
        self.load_and_chunk_documents()

    def _chunk_text(self, text: str, chunk_size: int = 150, overlap: int = 30) -> List[str]:
        """Simple sliding window chunker to respect model token limits (256 tokens for all-MiniLM-L6-v2)."""
        words = text.split()
        chunks = []
        stride = chunk_size - overlap
        for i in range(0, len(words), stride):
            chunk = " ".join(words[i:i + chunk_size])
            if chunk:
                chunks.append(chunk)
        return chunks

    def load_and_chunk_documents(self):
        """Load documents and split them into overlapping chunks before embedding."""
        if not self.docs_dir.exists():
            print(f"Warning: Directory '{self.docs_dir}' not found.")
            return

        for file_path in self.docs_dir.glob("**/*.*"):
            if file_path.suffix in ['.md', '.txt', '.html']:
                try:
                    with open(file_path, 'r', encoding='utf-8') as f:
                        raw_text = f.read().strip()
                        if not raw_text:
                            continue
                        
                        file_chunks = self._chunk_text(raw_text)
                        for idx, chunk in enumerate(file_chunks):
                            self.chunks.append({
                                "content": chunk,
                                "source": f"{file_path.relative_to(self.docs_dir)} [Chunk {idx+1}]"
                            })
                except Exception as e:
                    print(f"Error reading {file_path}: {e}")

        if self.chunks:
            texts = [c["content"] for c in self.chunks]
            self.embeddings = self.model.encode(texts, normalize_embeddings=True)
            print(f"Successfully indexed {len(self.chunks)} chunks from {self.docs_dir}")

    def search(self, query: str, top_k: int = 5, min_score: float = 0.35) -> List[dict]:
        """Semantic search with relevance filtering."""
        if self.embeddings is None or not self.chunks:
            return []

        query_embedding = self.model.encode(query, normalize_embeddings=True)
        similarities = self.embeddings @ query_embedding.T

        top_indices = np.argsort(similarities)[::-1]
        
        results = []
        for idx in top_indices:
            score = float(similarities[idx])
            if score < min_score or len(results) >= top_k:
                continue
            results.append({
                "content": self.chunks[idx]["content"],
                "source": self.chunks[idx]["source"],
                "score": round(score, 4)
            })
        return results

    def get_context(self, query: str, top_k: int = 4, min_score: float = 0.35) -> str:
        results = self.search(query, top_k, min_score)
        if not results:
            return "No relevant context found in local knowledge base."
        
        return "\n\n".join([
            f"--- Source: {r['source']} (Relevance: {r['score']}) ---\n{r['content']}"
            for r in results
        ])

The Mathematical Trick: Why We Don't Need Scikit-Learn

The retrieval step uses a single efficient line for cosine similarity:

similarities = self.embeddings @ query_embedding.T

This works because we normalize all embeddings (normalize_embeddings=True). The cosine similarity formula simplifies to a simple dot product when vectors have unit length. For more details, see the excellent explanation in the Sentence-Transformers documentation.

How to Use It

rag = SimpleRAG(docs_dir="knowledge")

query = "How does our payment retry logic work when the gateway times out?"
context = rag.get_context(query, top_k=4)

prompt = f"""Use the following context to answer the question accurately:

{context}

Question: {query}
Answer:"""

# Send to Ollama, LM Studio, or any local LLM

Production Tips and Best Practices

• Adjust chunk_size and overlap based on your model’s context window.
• Increase min_score (e.g. 0.45) for higher precision in production.
• Consider hybrid search (keyword + semantic) for better recall on technical terms.
• Cache query embeddings for repeated questions.

Final Thoughts

Before blindly installing packages that drag hundreds of transitive dependencies into your environment, ask yourself if you really need a complete orchestrator. By spending a short time writing a bare-metal implementation, you eliminate vendor lock-in, bypass dependency hell, and gain total transparency over chunking strategies, similarity thresholds, and vector math.This minimal RAG pipeline has served me extremely well for private knowledge bases and internal tools. Sometimes the simplest solution is also the most powerful.