feat(discovery): aggregator adapters + saved keyword searches - #33
Merged
Conversation
Introduce the second job-discovery strategy (PROJECT.md §5.4-B): an
AggregatorAdapter Protocol + registry paralleling the ATS adapters, plus
the two free/no-key reference implementations.
- base.py: the AggregatorAdapter Protocol — search(spec, *, fetcher,
timeout_s) fetches a feed/API through the injected Fetcher and returns
DiscoveredPostings, each carrying its own company (an aggregator spans
many companies, unlike an ATS board). No detect() hook — an aggregator
source is named, not a pasted URL to classify.
- structure.py: SavedSearch (query / location / remote), the per-profile
spec serialized into the aggregator source config.
- filters.py: matches_search — the shared, deterministic query/location/
remote filter applied after normalization, so every aggregator filters
identically and the logic is unit-tested once.
- remoteok.py / remotive.py: the RemoteOK raw-array feed and the Remotive
wrapped {jobs:[...]} API, best-effort skipping the feed's legal notice
and malformed rows.
- __init__.py: the registry (AGGREGATOR_TYPES, get_aggregator →
UnknownAggregatorError).
Key-gated aggregators (Adzuna/USAJOBS) are a documented fast-follow; the
interface stays minimal so they drop in without an unused, untestable
credential branch here.
100% line+branch on the package; mypy --strict incl. win32.
Add the persistence and poll layer for the aggregator strategy, paralleling the ATS path but keyed on a per-profile saved search rather than a company board. - repository.py: AGGREGATOR_SOURCE_TYPE + get/get_or_create/list_enabled aggregator-source helpers, deduping a saved search by the (aggregator, normalized search, profile_id) triple held in the source config (no new column, no migration — JobSource already carries type / config / profile_id / enabled / last_polled_at). - service.py: add_saved_search (idempotent get-or-create) and persist_aggregated — like persist_discovered, but an aggregator spans many companies, so it get-or-creates a company per posting from the posting's own company (falling back to a placeholder when none is named); the two-tier dedup (source external id, then cross-source apply-URL hash) is unchanged. - poller.py: run_aggregator_poll, the best-effort-per-source counterpart to run_discovery_poll (unknown provider / unusable response / fetch failure is counted and skipped, never fatal). 100% line+branch on the three modules; mypy --strict incl. win32.
Expose the aggregator strategy on the CLI and wire its poll into the discovery flow (PROJECT.md §9). - atlas source add <aggregator> --query [--location] [--remote/--onsite]: validates the aggregator against the registry (unknown → exit 1 listing the supported providers), resolves the active profile (none → exit 1 with an atlas init hint), and saves the per-profile search (re-add is a no-op). - atlas source list: a Rich table (or --json) of saved searches over the new pure build_saved_search_report / render_saved_searches in atlas.cli.discovery, mirroring the watchlist split. - atlas discover and the daemon tick now run run_aggregator_poll after the ATS poll and before scoring; discover reports the combined counts, so newly-found aggregator postings are scored on the same pass. 100% line+branch; mypy --strict incl. win32; full suite green (1012 tests).
Tick the Phase 2 "Aggregator adapters + saved keyword searches" roadmap item (PROJECT.md §15), move STATUS.md's "Next up" pointer to key-gated aggregators / multiple profiles / IPC, add a "What has landed" entry and update the phase-progress table, and record the change under Unreleased in the changelog.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Adds the second job-discovery strategy — aggregator keyword searches — alongside the existing per-company ATS watchlist (PROJECT.md §5.4-B, the top unchecked Phase 2 roadmap item). Where an ATS adapter watchlists one company's board and pulls every job on it, an aggregator runs a per-profile saved search (keywords + location + filters) against a job feed/API and returns postings from many companies. This closes the second half of Journey B: the daemon now finds jobs from keyword searches, not only from watchlisted boards.
Scope (agreed up front): the framework + two free/no-key adapters (RemoteOK + Remotive). Key-gated aggregators (Adzuna/USAJOBS) are a documented fast-follow that drops into the same registry.
Key commits
feat(aggregators): add aggregator adapter framework + RemoteOK/Remotive— theAggregatorAdapterProtocol + registry,SavedSearch, the sharedmatches_searchfilter, and the two adapters (RemoteOK raw-array feed; Remotive?search=API), plusUnknownAggregatorError.feat(aggregators): persist + poll aggregator saved searches—AGGREGATOR_SOURCE_TYPE+ aggregator-source repository helpers,add_saved_search/persist_aggregated(get-or-creates a company per posting), andrun_aggregator_poll(best-effort per source).feat(cli): add atlas source add|list and poll aggregators in discover— theatlas sourcecommand group and the aggregator poll wired intoatlas discover+ the daemon tick (after ATS, before scoring).docs: record aggregator adapters + saved keyword searches— STATUS/PROJECT/CHANGELOG.Design notes
JobSource(type="aggregator")— the table already carriestype/config/profile_id/enabled/last_polled_at. Dedup is by the(aggregator, normalized search, profile_id)triple.Fetcherboundary (so the suite stays hermetic — no live HTTP),DiscoveredPosting/ScrapedPosting,dedupe_hash_for+ the two-tier dedup, andcreate_job_posting. No new dependency (both feeds are plain JSON).detect(url)(a source is named, not a pasted URL) and each posting carries its own company.Follow-ups (out of scope here)
Key-gated aggregators (Adzuna app id + key, USAJOBS email + key) and the HN "Who is hiring" / arbeitnow sources — each a new module + one registry entry, key-gated ones inactive until a key is pasted (§5.4-B). A
[aggregators]config section +requires_keyseam land with them.Testing
uv run ruff format --check .+uv run ruff check .— cleanuv run mypy --strict .anduv run mypy --strict --platform win32 .— no issues (267 files each)uv run pytest --cov=atlas --cov-branch --cov-fail-under=100— 1012 passed, 100% line + branchAdapters are exercised offline via recorded feed fixtures through a
FakeFetcher; the poll/service/repository tests run over the in-memorydb_enginewith an injected fixed clock; the CLI tests use Typer'sCliRunnerwith a shared in-memory engine and a monkeypatched fetcher.Refs PROJECT.md §5.4-B, §15 (Phase 2).