Add dynamic service registration + GitHub Actions reindex example - #99
Merged
Conversation
…xample config.yaml doesn't scale past a handful of repos — indexing hundreds means hundreds of hand-maintained entries. POST /reindex now accepts a repo definition inline (github_repo/github_ref/root/exclude alongside service), registering it in a new Qdrant-backed ServiceRegistry instead of requiring a config.yaml entry. Registered services persist across restarts, resolve through the same load_effective_services() used everywhere config.yaml services are looked up (config.yaml wins on name collisions), and show up in index_stats/list_indexed_services like any other service. Includes examples/github-actions/reindex-on-merge.yml, a drop-in workflow that self-registers a repo and reindexes it on every merge to main/master — no central config edit needed. Also, since config.yaml is now genuinely optional: - load_services() treats a missing/empty config.yaml as zero services instead of crashing, and raises a clear error (instead of a cryptic IsADirectoryError) if CONFIG_PATH resolves to a directory — the failure mode of a Docker bind mount pointing at a config.yaml that doesn't exist. - docker-compose.yaml no longer mounts config.yaml by default; a new docker-compose.config-yaml.yml overlay (via `-with-config` Makefile targets) adds it back for curated/static setups. - Dockerfile no longer requires config.yaml to exist at build time. Fixed get_code_context, which only ever resolved services from config.yaml and would 404 dynamically-registered ones despite them being indexed. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #83.
config.yamlis a static, curated list — fine for a handful of repos, but indexing hundreds this way means hundreds of hand-maintained entries. This adds a second, self-service way to tell semcode about a repo, and a ready-to-use GitHub Actions workflow built on it.POST /reindexacceptsgithub_repo/github_ref/root/excludeinline (alongside the existingservice/force). Providinggithub_reporegistersservicewith that repo definition — persisted in a new Qdrant collection (service_registry, seeserver/store/service_registry.py) rather than a file, so it's safe to write from an unauthenticated request and survives restarts.load_effective_services()(config.yaml∪ registry) everywhere a service name is looked up —IndexPipeline,GitHistoryPipeline,get_code_context,index_stats.config.yamlalways wins on a name collision, so a stray/malicious registration can never silently repoint a curated service.index_statsnow lists "From config.yaml" and "Registered dynamically via API" separately./reindexis unauthenticated the same way it already was forconfig.yamlservices.examples/github-actions/reindex-on-merge.yml: drop-in workflow for the indexed repo's own CI. On push tomain/masterit self-registers (default service name =owner-repo, collision-safe across orgs) and reindexes in the same call — no central file to edit, no server restart.config.yamlis now genuinely optional:Settings.load_services()treats a missing or emptyconfig.yamlas zero services instead of raisingFileNotFoundError.CONFIG_PATHresolves to a directory (the classic Docker bind-mount-of-a-missing-file footgun), it now raises a clear, actionable error instead of a crypticIsADirectoryError.Dockerfileno longer bakesconfig.yamlin at build time.docker-compose.yamlno longer mountsconfig.yamlby default;docker-compose.config-yaml.ymlis a small overlay that adds it back, wired up via new-with-configMakefile targets, for anyone who wants curated services alongside or instead of dynamic registration.get_code_contextonly ever resolved services fromconfig.yaml, so it would report a dynamically-registered service as "no longer in config.yaml" even while it was actively indexed. Fixed to use the same merged resolution as everything else.docs/configuration.mdgets a full "Dynamic Service Registration" section;docs/README.md,docs/ingestion.md,docs/retrieval-rrf.md, and the rootREADME.mdupdated to describe both paths and how they interact.Test plan
uv run pytest -q— 289 passeduvx ruff check ./uvx ruff format --check .— cleandocker compose -f docker-compose.yaml -f docker-compose.config-yaml.yml configmerges the config.yaml volume mount correctly on top of the base filePOST /reindexwithgithub_repofor a repo not inconfig.yaml, confirm it indexes and shows up inlist_indexed_services/index_stats, and survives a container restart🤖 Generated with Claude Code