A retrieval-augmented generation engine where embeddings, retrieval and generation all run on a self-hosted model. Built for confidential documents that can't touch a cloud API.
Legal contracts, medical records, financial statements, internal HR files — the moment they hit a third-party API, you've lost control of them. PrivateRAG keeps every byte on hardware you own.
No API keys, no outbound requests. Works air-gapped. Your corpus stays on the box.
Runs on a ~$300 consumer GPU. No metered billing, no surprise invoices at scale.
Answers cite their sources and refuse when the answer isn't in your documents.
Structured extraction constrained at the sampler level — the output parses every time.
Measured end-to-end on a single NVIDIA RTX 4060 (8 GB), warm model. Retrieval uses local nomic-embed-text embeddings with cosine search.
| Model | First token | Generation | Structured extract |
|---|---|---|---|
| qwen2.5:3b (q4) | ~0.2 s | ~95 tok/s | ~2.0 s |
| qwen2.5:7b (q4) | ~0.25 s | ~46 tok/s | ~3.5 s |
Retrieval < 150 ms per query · extraction verified against a JSON-Schema (invoice → typed line items, totals, dates).
┌───────────────────── your machine ─────────────────────┐ │ │ docs ──┼──► chunk ──► nomic-embed-text ──► vector store │ │ │ │ query ─┼──► nomic-embed-text ──► cosine top-k ──┘ │ │ │ │ │ ▼ │ │ qwen2.5 (Ollama) ──► streamed answer │ │ + cited sources │ └───────────────────────────────────────────────────────────┘ no network egress — ever
Requires Ollama and Python 3.10+. The models download once; after that it's fully offline.
# pull the models (one-time) ollama pull qwen2.5:3b ollama pull nomic-embed-text # start the engine cd engine && pip install -r requirements.txt uvicorn server:app --port 8017 # open http://localhost:8017 — watch the egress counter stay at 0