feat(discovery): key-gated aggregators — Adzuna + USAJOBS - #34
Merged
Conversation
Model the key-gated aggregator providers (PROJECT.md §5.4-B) that the Adzuna/USAJOBS adapters will consume, mirroring the [ai.backends] shape: per-provider `_Base` models with defaulted keyring *handle* strings (never the secret itself). - AdzunaConfig (enabled / app_id_handle / app_key_handle / country) — query-string auth. - UsajobsConfig (enabled / email / api_key_handle) — header auth; the non-secret registering email lives in config, the key in the keychain. - AggregatorsConfig groups them; Config gains an `aggregators` field. All defaulted (a missing config still yields a valid Config) and forward-compatible (`extra="ignore"`), so the section loads before the adapters that read it land. Re-exported from atlas.config for tests. No secret ever appears here — only handles.
Introduce the key-gated aggregator path (PROJECT.md §5.4-B) and the first
provider that uses it — Adzuna (query-string auth).
- base.py: add `requires_key` to the AggregatorAdapter Protocol (free feeds
set False, key-gated True).
- __init__.py: a builder registry replacing the bare instance tuple —
`build_aggregator(name, *, config, store)` resolves a key-gated adapter's
credentials from the keychain and returns None (an inactive source) when
disabled or keyless; `validate_aggregator` / `aggregator_requires_key` are
the pure name checks the CLI uses without a store.
- adzuna.py: AdzunaAdapter (constructed with resolved app_id/app_key/country
by build_adzuna) GETs the search API with credentials in the query string,
normalizes {results:[...]}, and applies the shared matches_search.
- poller.py: run_aggregator_poll gains `config` + `store` params; a key-gated
source that builds to None is skipped as *inactive* — reported via the new
DiscoveryOutcome.inactive (defaulted, so ATS/JSON round-trips are unchanged)
rather than silently dropped or counted as a failure.
- cli: discover and the daemon tick thread config.aggregators + the secret
store into the aggregator poll; discover surfaces the inactive count; source
add points key-gated providers at `atlas source key`.
100% line+branch on the aggregator package + poller; mypy --strict incl. win32.
The second key-gated provider (PROJECT.md §5.4-B), exercising header auth — the reason the Fetcher seam carries a `headers` param. - usajobs.py: UsajobsAdapter GETs the USAJOBS search API with the credential in an `Authorization-Key` header + the registering email as `User-Agent` (both resolved by build_usajobs from config + keychain), parses the nested SearchResult.SearchResultItems shape (tolerating a list-valued ApplyURI and a missing UserArea/Details), and applies the shared matches_search. - build_usajobs returns None (inactive) when disabled, no email, or no key. - Registered on the builder registry; AGGREGATOR_TYPES is now four. 100% line+branch; mypy --strict incl. win32.
The first secret-writing path in the codebase (every SecretStore use so far was read-only). `atlas source key <aggregator>` prompts for each credential with hidden input — never echoed, never in shell history — and writes it to the OS keychain under the handle from config, so a key-gated source can be activated without secrets ever touching config or the command line. - registry: credential_prompts(aggregator, config) returns the (label, handle) pairs a key-gated provider stores (Adzuna: app id + app key; USAJOBS: api key), keeping per-provider credential knowledge in the package. - command: validates the name is a key-gated aggregator (unknown / free → exit 1), then prompts + SecretStore.set per credential; a ConfigError (e.g. no keychain) exits 1 cleanly. The secret is never printed or logged. 100% line+branch; mypy --strict incl. win32; full suite green (1061 tests).
Surface each registered aggregator's configuration health so a user can see at a glance which job sources are usable and which need a key. - doctor.py: build_aggregator_health(config, store) reports per provider — a free feed is "active"; a key-gated one is "active" (enabled + keyed), "needs API key" (enabled, keyless), or "disabled" — mirroring the backend build->availability->detail shape and never leaking the secret. A new DoctorReport.aggregators field (defaulted, JSON round-trippable) carries it, rendered as a second Rich table; `healthy` stays AI-only since aggregators are optional. - doctor command builds the aggregator health and attaches it to the report. 100% line+branch; mypy --strict incl. win32; full suite green (1065 tests).
Tick the key-gated aggregator work: STATUS.md gains a "What has landed" entry and an updated phase-progress row, and its "Next up" pointer moves to multiple profiles / IPC (key-gated aggregators are done). PROJECT.md §15 notes Adzuna + USAJOBS + the [aggregators] config / source key landed, and §10's config example gains an [aggregators] block. CHANGELOG records the two adapters, the config section, atlas source key, and the doctor aggregator report under Unreleased.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Adds the key-gated aggregators the free RemoteOK/Remotive feeds deferred (PROJECT.md §5.4-B; the top "Next up" item in
docs/STATUS.md) — the two credential-requiring providers the design names: Adzuna (query-string auth) and USAJOBS (header auth). A key-gated source resolves its credentials from the OS keychain and is shown as "needs API key" (inactive, never failing silently) until a key is stored.Key commits
feat(config): add [aggregators] config section—AdzunaConfig/UsajobsConfig/AggregatorsConfig, mirroring[ai.backends](enable flags + non-secret params + keyring handles, never the secret).feat(aggregators): add key-gating seam + Adzuna adapter—AggregatorAdapter.requires_key, abuild_aggregator(name, *, config, store)builder registry (resolves credentials, returnsNone= inactive when disabled/keyless),adzuna.py, and the poller refactor (config/storeparams,DiscoveryOutcome.inactive, inactive-skip) wired intoatlas discover+ the daemon.feat(aggregators): add USAJOBS adapter—usajobs.py, header auth (the reason theFetcherseam carriesheaders), nestedSearchResultparsing.feat(cli): add atlas source key for storing aggregator credentials— the first secret-writing path: hidden prompts →SecretStore.set.feat(cli): report aggregator sources in atlas doctor— a second table (active / needs API key / disabled).docs: record key-gated aggregators (Adzuna, USAJOBS)— STATUS/PROJECT/CHANGELOG.Design notes
base.pydocstring's documented intent andbuild_openrouter_provider): the poll builds the adapter with resolved credentials or getsNone.search(...)'s signature is unchanged; the secret never touches it and is never logged.JobSource(type="aggregator")rows; credentials live in the keychain, non-secret params in config.DiscoveryOutcome.inactiveand surfaced byatlas discover+atlas doctor, distinct from a failure.atlas source keyprompts with hidden input (never echoed / in shell history) rather than flags.Fetcher).Follow-ups (out of scope)
The free HN "Who is hiring" / arbeitnow aggregators drop into the same registry later.
Testing
ruff format --check+ruff check— cleanmypy --stricton Linux and win32 — no issues (271 files each)pytest --cov=atlas --cov-branch --cov-fail-under=100— 1065 passed, 100% line + branchAdapters are exercised offline via recorded fixtures through a
FakeFetcher; credential resolution / storage via aFakeKeyring; thesource keyhidden prompts viaCliRunner(input=...). No live network or real keychain in the suite.Refs PROJECT.md §5.4-B, §15 (Phase 2).