VectorDB

A local-first vector database and Retrieval-Augmented Generation (RAG) system, built with FastAPI, three different nearest-neighbor search algorithms, and a fully local LLM stack via Ollama.

Everything runs on your machine — no cloud vector store, no external API keys, no data leaving your computer.

---

Project Overview

VectorDB lets you:

1. Store text as vectors (via a local embedding model) and search them using three different nearest-neighbor algorithms — Brute Force, KD-Tree, and HNSW.
2. Upload documents, have them automatically chunked and embedded, and ask natural-language questions answered using a local LLM grounded in your own documents (RAG).
3. Visualize your vector space in 2D using PCA.

A single-page React frontend (loaded via CDN, no build step) is served directly from the FastAPI backend.

---

Features

• 🗄️ Local Vector Database — add arbitrary text, get it embedded and stored instantly.
• 🔍 Three Search Algorithms — Brute Force (exact), KD-Tree, and HNSW (approximate, production-grade).
• 📐 Multiple Distance Metrics — Cosine, Euclidean, and Manhattan.
• 📊 PCA Visualization — see your 768-dimensional vectors projected into 2D.
• 📚 Multi-Document Store — upload multiple documents, each tracked by a source name.
• 🔎 Source Filtering — restrict RAG search to one or more specific documents.
• 🤖 RAG-based Question Answering — ask questions answered only from your documents' content, powered by a local LLM.
• 🖥️ Built-in Frontend — a dark-themed single-page UI with no separate build/deploy step.
• ⚡ 100% Local — embeddings and generation run through Ollama on localhost.

---

Tech Stack

Layer	Technology
API Framework	FastAPI + Uvicorn
Embeddings & LLM	Ollama (nomic-embed-text, llama3.2)
Exact Search	Brute-force NumPy distance computation
Tree-based Search	scipy.spatial.KDTree
Approximate Search	usearch (HNSW index)
Dimensionality Reduction	scikit-learn PCA
Data Validation	Pydantic
Frontend	React 18 (via CDN) + Babel standalone

---

Project Architecture

The application follows a simple, flat, beginner-friendly module layout — no routers, services, or dependency-injection layers. Each file has a single, clear responsibility, which keeps the whole codebase easy to read top-to-bottom and easy to explain in an interview.

Request → FastAPI endpoint (main.py)
             │
             ├─→ OllamaClient (ollama_client.py)   → embeddings / text generation
             │
             ├─→ VectorDB (vector_db.py)
             │        └─→ BruteForceSearch / KDTreeSearch / HNSWSearch (search_algorithms.py)
             │
             └─→ DocumentDB (document_db.py)        → chunking, RAG retrieval

• main.py owns the FastAPI app, middleware, and every route. It only orchestrates — it calls into the other modules rather than containing business logic itself.
• vector_db.py and document_db.py own their respective in-memory data stores.
• search_algorithms.py contains the three interchangeable nearest-neighbor strategies that VectorDB delegates to.
• ollama_client.py isolates all outbound HTTP calls to the local Ollama server, plus a small text-cleaning helper shared by the embedding and generation paths.
• models.py centralizes every Pydantic request schema.

---

Folder Structure

Vector-DB/
│
├── main.py                # FastAPI app, middleware, routes, startup
├── ollama_client.py        # OllamaClient class + clean_text() helper
├── search_algorithms.py    # BruteForceSearch, KDTreeSearch, HNSWSearch
├── vector_db.py            # VectorDB (unifies the three search algorithms)
├── document_db.py          # DocumentDB (chunking + RAG retrieval)
├── models.py                # All Pydantic request models
├── requirements.txt
├── README.md
├── .gitignore
└── index.html               # Frontend (served at "/")

---

Requirements

• Python 3.10+
• Ollama installed and running locally
• The following Ollama models pulled locally:

  ollama pull nomic-embed-text
  ollama pull llama3.2

---

Installation

1. Clone the repository:

   git clone https://github.com/Piyush1228/Vector-DB.git
   cd Vector-DB

2. (Recommended) create and activate a virtual environment:

   python -m venv venv
   source venv/bin/activate        # macOS/Linux
   venv\Scripts\activate           # Windows

3. Install dependencies:

   pip install -r requirements.txt

4. Make sure Ollama is running in the background:

   ollama serve

---

Running the Project

Start the FastAPI server:

python main.py

You should see:

Starting VectorDB server v2.0...
API docs : http://localhost:8000/docs
Frontend : http://localhost:8000

• Open http://localhost:8000 for the frontend.
• Open http://localhost:8000/docs for interactive Swagger API docs.

---

API Endpoints

Health

Method	Endpoint	Description
GET	/health	Server, Ollama, and storage status

Vector Database

Method	Endpoint	Description
POST	/vectors/add	Embed and store a piece of text with a label
POST	/vectors/search	Search stored vectors (choose algorithm/metric)
GET	/vectors/visualize	Get 2D PCA-projected points
DELETE	/vectors/clear	Remove all stored vectors

Document Database (RAG)

Method	Endpoint	Description
GET	/documents/list	List all stored documents and chunk counts
POST	/documents/add	Add (or replace) a document — chunks & embeds it
DELETE	/documents/{source}	Delete a single document by its source name
DELETE	/documents	Delete all documents
POST	/documents/query	Ask a question, answered via RAG over your documents

Frontend

Method	Endpoint	Description
GET	/	Serves the index.html UI

---

Search Algorithms Used

Algorithm	Complexity	Accuracy	Notes
Brute Force	O(N)	100% exact	Scans every stored vector; simplest baseline.
KD-Tree	~O(log N)	Exact	Binary space-partitioning tree via scipy.spatial.KDTree.
HNSW	~O(log N)	Approximate	Graph-based index via usearch; same technique used by production vector databases like Pinecone, Chroma, and Weaviate.

All three algorithms support Cosine, Euclidean, and Manhattan distance metrics (KD-Tree and Brute Force natively; HNSW is configured for cosine similarity).

---

RAG Workflow

1. Add a document — its text is cleaned, split into overlapping word chunks (150 words per chunk, 20-word overlap), and each chunk is embedded via nomic-embed-text.
2. Chunks are indexed in a shared HNSW index, tagged with their source document name.
3. Ask a question — the question is embedded and used to retrieve the top-k most relevant chunks (optionally filtered to specific source documents).
4. Context is assembled from the retrieved chunks and injected into a prompt template.
5. llama3.2 generates an answer, grounded strictly in the retrieved context — if the answer isn't in the documents, the model is instructed to say so.

---

Screenshots

Add screenshots of the running frontend here:

• Vector DB page
• RAG Pipeline page
• 2D PCA Visualization
• "How It Works" page

---

Future Improvements

• Persist vectors/documents to disk (currently all in-memory, reset on restart).
• Add authentication for multi-user deployments.
• Support streaming LLM responses in the frontend.
• Add unit tests for each module.
• Support swapping in other embedding/generation models via configuration.
• Add pagination for large document/chunk lists.

---

License

This project is provided as-is for educational and personal use.