Skip to content

hardening: align RAG security, CI and portfolio claims - #36

Merged
brunovicco merged 17 commits into
mainfrom
hardening/portfolio-review-2026-09
Sep 17, 2026
Merged

brunovicco merged 17 commits into
mainfrom
hardening/portfolio-review-2026-09

Conversation

@brunovicco

@brunovicco brunovicco commented Sep 17, 2026 •

Copy link
Copy Markdown
Owner

Summary

Hardening pass focused on making RAGForge's public claims match its implemented behavior and strengthening the evidence expected from a senior/staff RAG engineering portfolio.

Security

  • bind local Postgres/OpenSearch ports to 127.0.0.1
  • explicitly mark the Compose profile as development-only
  • add a RAG-specific threat model covering indirect prompt injection, corpus poisoning, vector/embedding manipulation, synthetic-evidence contamination, citation laundering, judge risk, data disclosure, cost/availability and supply chain
  • harden answer generation so retrieved evidence is explicitly treated as untrusted data
  • preserve the invariant that answer generation consumes authoritative source_text, not synthetic retrieval_text
  • add unit tests for those trust-boundary invariants
  • refresh transitive dependency security floors and regenerate uv.lock with uv
  • keep only explicit pip-audit exceptions for advisories currently reported without a fixed release

Engineering / CI

  • pin actions/checkout@v7 to its immutable commit SHA
  • add a credential-free integration job that starts real pgvector + OpenSearch and tests both adapters
  • configure the OpenSearch kernel prerequisite explicitly on GitHub-hosted runners
  • keep provider-backed integration tests opt-in

Architecture / documentation

  • replace generic scaffold language in docs/ARCHITECTURE.md with the actual RAGForge evaluation architecture
  • fix docs/DEVELOPMENT.md so it reflects the real FastAPI container runtime
  • clarify the distinction between captured write-through LLM calls and the still-planned deterministic replay engine
  • align README.md and README.pt-BR.md
  • scope the v0.1 SAC conclusion to the published sample instead of presenting it as a universal recommendation
  • add the missing confidence-interval limitation to the benchmark report
  • align privacy documentation with the real loopback-bound local infrastructure

CI evidence

Final head fe3e70d52afa72fb61d41a760c376733747b5075:

  • quality: passed — lock check, Ruff lint/format, architecture/governance guards, mypy, 556 unit tests, coverage gate, Bandit and pip-audit
  • integration: passed — real pgvector + OpenSearch startup, pgvector extension initialization, adapter integration tests and cleanup

The dependency audit initially surfaced newly published advisories in transitive packages. Packages with available fixes were upgraded through declared security floors and a regenerated lock file; unresolved advisories remain explicitly documented rather than silently hidden.

Deliberately not included

  • no hosted-provider calls in CI
  • no benchmark result regeneration
  • no dataset changes
  • no claim that prompt wording alone solves indirect prompt injection; live adversarial attack-success evaluation remains future work
  • no deterministic make bench replay implementation in this PR

@brunovicco
brunovicco merged commit 7148774 into main Sep 17, 2026
2 checks passed
@brunovicco
brunovicco deleted the hardening/portfolio-review-2026-09 branch September 17, 2026 04:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant