Skip to content

feat: first working engine with the RAG course techniques - #4

Merged
TWadi merged 17 commits into
mainfrom
twadi/rag-engine
Nov 2, 2026
Merged

TWadi merged 17 commits into
mainfrom
twadi/rag-engine

Conversation

@TWadi

@TWadi TWadi commented Nov 2, 2026

Copy link
Copy Markdown
Collaborator

What

The first working engine: a command-line RAG built with the techniques from the course we just finished, on LangChain, running fully locally on a laptop CPU.

uv sync
ollama pull qwen3:4b-instruct-2507-q4_K_M
cd examples
uv run ore ingest docs
uv run ore ask "Why does my pump keep turning itself off?"
Course technique Where Switch
Recursive, Markdown and semantic chunking chunking.py [chunking] strategy
Multilingual embeddings in Chroma store.py, factory.py [embeddings] model
Similarity, score threshold, MMR, metadata filters retrieval.py [retrieval] search_type, --source
BM25 + vectors fused with reciprocal rank fusion retrieval.py, fusion.py vector_weight, keyword_weight, rrf_k
Multi-query (RAG-Fusion) fusion.py, pipeline.py --multi-query
Cross-encoder reranking rerank.py [rerank], --no-rerank
Conversational follow-ups conversation.py ore chat
Cited answers, checked; abstains without evidence answer.py [answer] min_rerank_score
Retrieval evaluation evaluation.py ore eval

Also:

  • Ingestion is idempotent per content hash. A changed file replaces its old chunks.
  • Changing the chunking or embedding configuration asks for --rebuild, so an index is never mixed.
  • Every stage records a trace (--trace).
  • The configuration is a validated, immutable ore.toml.

Default models. All are free, local and permissively licensed (C-02):

  • intfloat/multilingual-e5-small (MIT) for embeddings.
  • cross-encoder/mmarco-mMiniLMv2-L12-H384-v1 (Apache-2.0) for reranking.
  • qwen3:4b-instruct-2507 (Apache-2.0) through Ollama for answers.

Two notes on the defaults:

  • Ollama's plain qwen3:4b tag is now the 2507 thinking release. It can't switch its reasoning off, and on our CPU that cost about 3 minutes per answer, so the default is the Instruct release.
  • Reasoning is configurable ([llm] think), and any <think> text is hidden from the output and never parsed for citations.

ADR-0006 records the choice of Python and LangChain, and how it fits ADR-0002 and ADR-0003:

  • LangChain's interfaces are the ports.
  • RetrievalPipeline keeps the fixed slots.
  • The sunsetting langchain-community and langchain-experimental packages are confined to four adapter modules, and a test enforces it.

Builds on the design docs merged in #3.

Why

It is step 2 of the roadmap: ingest → hybrid retrieval → reranking → cited answer, plus the evaluation harness.

How it was tested

  • Unit and integration tests: 78 tests, 98% coverage, with small fake models (no downloads in CI).
  • Lint and formatting: ruff check and ruff format --check are clean.
  • CI: a new workflow runs lint and the tests with an 80% coverage gate.
  • Evaluation on the example set: 12 questions over 10 files, in English, German and one in French. These are real models on CPU, k = 5.
Configuration Hit rate MRR Recall
Hybrid, no reranking 1.00 0.83 1.00
Hybrid + reranking (default) 1.00 0.88 0.96
MMR + reranking 1.00 0.88 0.96

Reranking moves the right file up the list. The recall dip is the French question with two expected files, where one fell to 6th.

  • End to end on CPU with real models:
    • "Why does my pump keep turning itself off?" gave a correct answer citing protection.md. The reranker lifted that file from 4th to 1st.
    • "Wie viel kostet der Drucksensor?" was answered in German: "Der Drucksensor PS-12 kostet 39 Euro [1]", citing service.md.
    • "Who founded the AquaFlow company?" correctly gave "I don't know based on the provided documents."

Known limits (next PRs)

  • BM25 has no stop-word lists yet. Words like "does" can push an unrelated chunk up. The reranker fixes the final order, but keyword search could be better.
  • CJK text is tokenised as whole runs for BM25.
  • Citation checking verifies that a cited source exists, not yet that it supports the statement. The support checker in ADR-0005 comes later.
  • There is no API server, web UI, knowledge graph or scanned-page handling yet.

Checklist

  • Tests cover the change
  • Docs updated (README, docs/, or a new ADR for architectural decisions)
  • No secrets, keys or private data in the diff

A CLI engine on LangChain (ADR-0006) that runs locally on a laptop CPU:
- ingest Markdown, text and PDF with recursive, Markdown-aware or semantic chunking; idempotent per content hash
- hybrid retrieval: Chroma vectors (similarity, threshold, MMR, metadata filters) + BM25, fused with reciprocal rank fusion
- multi-query expansion fused with RRF, cross-encoder reranking, history-aware follow-ups
- cited answers from a local Qwen3 4B through Ollama; citations parsed and checked, abstains without evidence
- per-stage trace, retrieval evaluation (hit rate, MRR, recall@k) with a config fingerprint
- example corpus (English and German) with an evaluation set; CI with lint and an 80% coverage gate
…fline mode

- workspace: categories and sections, each section a set of folders with its own index
- library: background indexing with progress (one job at a time), pruning of deleted files while keeping
  documents of unplugged folders, answers scoped to a section or fused across a category (RRF, one rerank)
- ore serve: FastAPI on 127.0.0.1 with a per-launch bearer token, loopback Host check and CORS for the app
  origin; answers stream as NDJSON; errors are RFC 9457 problem details
- offline mode: telemetry off, no progress bars, HF_HUB_OFFLINE when every model is on disk
- llm provider 'openai' for OpenAI-compatible local servers such as llama.cpp's llama-server
- ADR-0007: WPF + WebView2 + React desktop app over the local engine API
- WPF shell (.NET 10) hosting a React 19 + Tailwind 4 interface in WebView2, in the AI Engineering Arena's
  sticker-arcade theme (light and dark, fonts bundled for offline use)
- sidebar of categories and sections; per section: folders, background indexing with progress, documents and a chat;
  a category chat searches all its sections; citations as pills and source cards that open the file
- the app starts ore serve itself (bundled Python when installed, uv in a checkout), ties it to a job object and
  routes every API call through an EngineProxy that adds the token and allows only the engine's API paths, so the
  page never sees the token or talks to a local port
- conversations kept per section on this computer; follow-up questions use the recent history
- ore serve: --data-dir and --warm; python -m open_rag_engine
- tests: 36 .NET (engine start, ready line, locator, proxy, safe file opening), 13 web, engine suite; Windows CI job
TWadi and others added 14 commits November 2, 2026 19:04
The standard Windows title bar is replaced by one drawn in the Arena's top-bar style: the RAG sticker, the app name,
where you are (category > section) and Arena window buttons. It is a WebView2 drag region, so dragging, double-click
to maximise and the window menu still work; WindowChrome keeps resizing from a thin border, rounded corners on
Windows 11, and the frame follows light or dark mode.
The global keyboard-focus outline was unlayered CSS, so it beat the text area's outline-none and drew a rectangle
inside the rounded box. It now lives in the base layer, and the whole box shows focus with a cobalt border and shadow.
A bundled engine loads its models from models_dir/<org>--<name> under the same model names, so the index
fingerprint and existing indexes stay valid. --llm-url points the engine at the llama-server the app starts.
- chat: a late reply after switching section can no longer land in, or be saved under, another conversation; saving is
  debounced and done outside the state update; the chat resets if the engine restarts (+ regression tests)
- a connection lost mid-answer becomes a 503 instead of an unhandled exception; every forwarded call and question now
  always gets an answer for its id; library calls time out instead of hanging
- opening sources: only local document paths (no UNC, device or alternate-stream paths), checked off the UI thread;
  explorer.exe by full path
- engine lifecycle: start and restart are serialised, deliberate stops aren't reported as crashes, a start that times
  out stops the process, only the current engine can report a crash, and unexpected start errors show a message
- WebView2: content-security policy on the built page (connect-src 'none'), DenyCors mapping, all permissions denied,
  no host objects; HttpClient bypasses system proxies
- engine: Stop now really stops generation (the next token raises once the client is gone); models load once under a
  lock, so the warm-up and a first question can't load them twice
- proxy: exact allow-list match (\z), questions only via the stream, problem bodies built with a JSON serialiser
- deleting a section or category also deletes its saved conversation; checkout discovery is Debug-only
The Python .gitignore template ignored every lib/ folder, so desktop/web/src/lib
(the engine client, the WebView2 bridge, chat storage, the mock engine and
their tests) never reached the repository and a fresh checkout couldn't build
the interface. The rule now covers only the root lib/ and lib64/.
installer/build.py stages a self-contained Open RAG and runs Inno Setup:
the published app, a uv-managed CPython 3.12 with the engine and its locked
dependencies (CPU torch), the embedding and reranker models from the Hugging
Face cache, the Qwen3 GGUF from Ollama (digest-checked) and llama.cpp's
llama-server (pinned b12052, sha256-checked). The setup installs per user,
keeps the library on uninstall and comes in parts below GitHub's 2 GiB limit.

The app starts the bundled llama-server on a free loopback port with its own
key and points the engine at it with --llm-url. ChildHost replaces EngineHost
and runs both processes, waiting on ORE_READY or a /health probe. Logs rotate
at 5 MB. installer/smoke_test.py checks a stage or an installed app end to end.

The window also opens the credits' GitHub and LinkedIn links, and only those
(ExternalLinks).
The sidebar says who developed Open RAG and opens a Credits sheet with both
authors' GitHub and LinkedIn and the source code. Links open in the default
browser through the window, which accepts only the links on its own list; a
test checks the sheet's links against it.
feat: Windows installer with a bundled language model, plus credits
feat: Open RAG, a Windows desktop app with categories and sections
feat: local API for the desktop app, with sections, categories and offline mode
reset() deleted the chroma folder after delete_collection(). Chroma caches its
client per folder and keeps the database open, so on Linux, where the delete
succeeds, the next write hit a deleted file (SQLite 1032, readonly database)
and CI's rebuild test failed on every run. Windows refused the delete, which
hid it. delete_collection() already removes the vectors.

Also ignores Vite's cache folder.
Leads with what the app does and how to install it from the release, shows the
interface in dark and light mode and the credits, explains the answer pipeline
and the desktop processes with two diagrams, and updates the roadmap. The
screenshots use demo folders (our RAG lesson scripts and the example pump
manual) with the real engine and the bundled model.
@TWadi TWadi self-assigned this Nov 2, 2026
@TWadi
TWadi merged commit 8d001b3 into main Nov 2, 2026
2 checks passed
@TWadi
TWadi deleted the twadi/rag-engine branch November 2, 2026 22:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants