feat(discovery): company watchlist + Greenhouse ATS discovery - #30
Merged
Conversation
…URL detection The first slice of Phase 2 company-watchlist discovery (PROJECT.md §5.4-A): an extensible AtsAdapter Protocol + registry and the Greenhouse adapter, with no discovery persistence or CLI yet (those follow). - atlas.discovery.structure.DiscoveredPosting reuses the scraper's ScrapedPosting by composition, adding only the ATS's external_id. - atlas.discovery.ats.base.AtsAdapter: a runtime-checkable Protocol with a pure offline detect(url) hook and a list_postings() that fetches through the injected scrape Fetcher (no new network boundary). - atlas.discovery.ats.greenhouse.GreenhouseAdapter: detects the board token from the boards/job-boards/embed/subdomain URL forms and normalizes the public boards JSON API (content=true), unescaping the HTML description via the reused extract_main_text and skipping malformed jobs best-effort. - atlas.discovery.ats registry: get_adapter (UnknownAtsError on miss), detect_ats(url) -> (ats_type, board_token), and ATS_TYPES for messaging. - atlas.discovery.errors: DiscoveryError + UnknownAtsError. Tests drive detection over URL strings and list_postings through a FakeFetcher replaying a recorded board payload; 100% line+branch on the discovery package, mypy --strict clean (incl. win32). No new dependency.
The persistence layer beneath the discovery poll (PROJECT.md §5.4, §6): thin repository functions over an open session plus the two domain operations, with no poller or CLI yet. - repository: an ATS board is a JobSource(type="ats") whose config JSON carries ats_type / board_token / company_id (no new column, so no migration). get_ats_source / get_or_create_ats_source dedup by (ats_type, board_token); list_enabled_ats_sources feeds the poll (excludes disabled + the shared url source); stamp_last_polled_at; get_posting_by_source_external is the stable per-source re-poll key. - service.add_watchlist_company: get-or-create the Company (recording ats_type / ats_board_ref / domain) + the ATS source, idempotent. - service.persist_discovered: insert new postings, deduplicating first by (source, external_id) and then by the normalized-apply-URL dedupe_hash (cross-source collapse), reusing the scraper's dedupe_hash_for and create_job_posting. Clock injected for deterministic fetched_at. Tested over the in-memory db_engine; 100% line+branch on the discovery package, mypy --strict clean (incl. win32).
The discovery counterpart to the daemon's scoring poll (PROJECT.md §4.1, §5.4): run_discovery_poll, a pure function over an open session returning a DiscoveryOutcome (sources_polled / discovered / skipped / failed_sources). For each enabled ATS source it resolves the provider's adapter, lists the board through the injected fetcher, persists the new postings, and stamps last_polled_at. Polling is best-effort per source — an unknown provider, an unusable board response, or a fetch failure is counted (failed_sources) and skipped rather than aborting the batch, mirroring run_scoring_poll's handling of a MatchingError. The scoring poll picks up the new unscored postings on its own pass, so the two compose without coupling. Tested over the in-memory db_engine with a FakeFetcher: discovers + persists + stamps, re-poll is a no-op, a failing source is skipped while a good one still polls, an unknown provider is skipped without fetching, and an empty watchlist makes zero fetch calls. 100% line+branch, mypy --strict clean (incl. win32).
The scriptable surface for the watchlist + discovery poll (PROJECT.md §9): - atlas company add <url>: auto-detects the ATS provider + board token from the URL (no --ats flag) via detect_ats, then watchlists the company and its board. An optional --name overrides the token-derived display name; an unrecognized URL exits 1 naming the supported providers. Re-adding is a no-op. - atlas company list: renders the watchlisted boards (company, ATS, board, enabled, last polled) or --json; an empty list hints at 'atlas company add'. - atlas discover: runs one discovery poll now over the injected default_fetcher and reports the DiscoveryOutcome (or --json). Discovery is AI-free — it hints at 'atlas score' / the daemon to score the newly-found postings. The pure build_watchlist_report / render_* split lives in atlas.cli.discovery (mirroring atlas.cli.scrape), tested against db_engine; the commands are tested through the CliRunner with a shared in-memory engine and a FakeFetcher replaying a recorded board (no network). Full suite green at 100% line+branch, mypy --strict clean (incl. win32).
Wire the discovery poll into the daemon's scheduled work (PROJECT.md §4.1): the bound job now runs run_discovery_poll over the ATS watchlist first, then run_scoring_poll. Each runs in its own short session_scope, so discovery commits its new postings before the scoring poll's list_unscored_postings picks them up on the same tick — the two compose without coupling. The daemon-start test gains a case that seeds a watchlisted board + an injected FakeFetcher and drives the bound job, asserting the posting is discovered and persisted (the empty-watchlist case still makes no fetch, so the suite stays hermetic). Full suite green at 100% line+branch, mypy --strict clean (incl. win32).
Update the docs for the discovery slice (Definition of Done, AGENTS.md §8): - STATUS.md: bump 'Last updated' and 'Current phase'; add the 'What has landed' entry for atlas.discovery; move the 'Next up' pointer to the remaining ATS adapters (Lever/Ashby/Workday), aggregators, the Discover queue, and the IPC surface; tick the Phase-2 progress row. - PROJECT.md §15: mark Greenhouse done under the watchlist item, noting the remaining adapters drop into the same registry. - CHANGELOG.md (Unreleased/Added): the watchlist + Greenhouse discovery, the atlas company / atlas discover commands, and discover-before-score in the daemon. - README.md: a 'Discover jobs from company ATS boards' section (atlas company add|list, atlas discover) and the updated daemon paragraph (discover→score).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Phase 2's first real discovery source (PROJECT.md §5.4-A). Until now the daemon's
scheduled work was only a scoring poll over postings you'd pasted in by hand with
atlas add <url>— it couldn't find anything. This PR adds a company watchlist and anextensible ATS adapter layer, ships the Greenhouse adapter, and wires a discovery poll
into the daemon before the scoring poll, so Atlas now genuinely discovers jobs on a
schedule and scores them on the same pass.
Scope was deliberately kept to one clean vertical slice: the full extensible interface +
registry, but only the Greenhouse adapter. Lever/Ashby/Workday are fast-follow PRs — each
is a new module in
atlas.discovery.atsimplementing the sameAtsAdapterProtocol plus oneentry in the registry tuple, with no interface change.
Key commits
feat(discovery): add ATS adapter interface + Greenhouse adapter with URL detection— theAtsAdapterProtocol (pure offlinedetect(url)+list_postingsover the reused scrapeFetcher), the registry (get_adapter/detect_ats/ATS_TYPES), and the Greenhouseboards-API adapter +
DiscoveredPosting(reusesScrapedPostingby composition).feat(discovery): add watchlist and discovery repository + service— an ATS board is aJobSource(type="ats")whose config carriesats_type/board_token/company_id;add_watchlist_company+persist_discovered(dedup by external id, then by thenormalized-apply-URL
dedupe_hashshared withatlas add).feat(discovery): add the discovery poll—run_discovery_poll→DiscoveryOutcome,best-effort per source (mirrors
run_scoring_poll).feat(discovery): add atlas company and atlas discover CLI—company add <url>(URL auto-detects the ATS; unrecognized → exit 1),
company list,discover.feat(daemon): discover before scoring in the daemon poll— the bound job discovers, thenscores the fresh backlog.
docs: record company watchlist + Greenhouse discovery— STATUS/PROJECT/CHANGELOG/README.Migration
None. Every column used already exists (migration
e5b263970e7e):company.ats_type/ats_board_ref/domain,job_source.type/config/enabled/last_polled_at,job_posting.external_id. This is a purely behavioral feature, like the tracking PR.Testing
Fully hermetic (AGENTS.md §6.2): Greenhouse normalization + all URL-detection forms (and the
unrecognized-URL error) via
FakeFetcherand recorded board JSON; the poll over the in-memorydb_engine(discover/persist/stamp, re-poll no-op, a failing source skipped best-effort,unknown provider skipped without fetching, empty watchlist makes zero fetches); the CLI via
CliRunner+ pure build/render tests. Gates green:ruff format --check,ruff check,mypy --strict(incl.--platform win32), and 864 tests at 100% line+branch coverage. Nonew dependency.
Design notes for review
company_idlives inJobSource.configJSON (no FK) — chosen to avoid a migration andto match the existing single
type="url"source pattern; integrity held in code.--nameoverride, so the single-arg
atlas company add <url>works.atlas discoveris discovery-only (AI-free) so it runs without a configured backend; thedaemon is what chains discover → score.
profile_id=None) this slice; per-profile watchlists aredeferred to "multiple profiles fully wired" (§15). An explicit
--ats/--tokenoverride oncompany addis noted as a future enhancement.