A question-answering assistant over a small machine-learning study guide. It finds the passages closest in meaning to a question, extracts an answer from them, reports a confidence score and its sources, and declines to answer when confidence is low.
Built for the LLMs and Generative AI sprint of the TripleTen AI/ML Bootcamp.
Question
│
▼
Sentence Transformer (all-MiniLM-L6-v2) → question embedding
│
▼
ChromaDB → the 3 most similar passages
│
▼
DistilBERT QA model → answer span extracted from those passages
│
▼
Confidence check (threshold 0.3) → answer, or a fallback message
│
▼
Answer + confidence + source documents
This follows the retrieval-augmented pattern, with one difference worth stating plainly: the final step extracts an answer span from the retrieved text. It does not generate new text with a large language model. Retrieval works the same way a generative system's would.
- 8 documents, 2,533 words in total, on machine-learning fundamentals: AI vs ML vs deep learning vs generative AI, supervised and unsupervised learning, regression, train-test split, mean squared error, and interpreting results with the empirical rule.
- Chunking: about 600 characters per chunk with 100 characters of overlap, which produced 38 chunks. The splitter works sentence by sentence, so a chunk never ends mid-sentence, and each chunk is prefixed with its document title for source attribution.
- Why 600 / 100: large enough to keep one concept together and to fit the QA model's input limit; the overlap protects answers that fall on a chunk boundary.
| Layer | Tool |
|---|---|
| Embeddings | Sentence Transformers (all-MiniLM-L6-v2) |
| Vector store | ChromaDB |
| Question answering | DistilBERT (distilbert-base-cased-distilled-squad) via Hugging Face transformers |
| Environment | Google Colab notebook |
15 test questions in four categories, with a confidence threshold of 0.3:
| Category | Questions | Answered with confidence | Average confidence |
|---|---|---|---|
| Simple facts | 5 | 3 | 49% |
| Detailed explanations | 5 | 0 | 5% |
| Comparison or complex | 2 | 0 | 3% |
| Out-of-scope edge cases | 3 | 0 (correct: all declined) | 0.1% |
- It declines when it should. All three out-of-scope questions ("What is the capital of France?") scored near zero and got the fallback message. Nothing was invented.
- Direct factual questions. "What are the main categories of Machine Learning?" returned "Supervised Learning and Unsupervised Learning" at 97% confidence, with the right source.
- Sources are always shown, so an answer can be checked.
- Explanations and comparisons. The system answered none of the seven "explain" or "compare" questions. Retrieval generally returned passages from the right documents; the bottleneck is the extractive model, which can only return a short span and cannot combine ideas across passages.
- High confidence is not always a good answer. "What is RMSE used for?" returned "business contexts" at 97% confidence: a span from the right passage, but not a useful answer.
- Sensitive to phrasing. Rewording a question can change both retrieval and confidence.
- Replace the extractive step with a generative LLM that is instructed to answer only from the retrieved passages, keeping the confidence fallback.
- Add hybrid search (keyword plus vector) for very specific queries.
- Chunk by document structure (headings, paragraphs) rather than character count.
- Measure retrieval and answer quality separately, so it is clear which stage fails.
Open RAG_Project_ArushiBhaskar_Final.ipynb in Google Colab and run all cells. The first cell installs the dependencies; no API key is needed.
Building each stage by hand (chunking, embeddings, vector search, answer extraction, confidence handling) made it clear that a RAG system fails in different ways at different stages, and that a refusal is a feature, not a failure.