diff --git a/CHANGELOG.md b/CHANGELOG.md index 8e80eae..9e37b6f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -9,6 +9,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Added +- **Lever, Ashby, and Workday ATS adapters** (`atlas.discovery.ats`): three new + discovery sources alongside Greenhouse (PROJECT.md §5.4-A), each a drop-in on the + existing registry — the daemon poll, watchlist service, and `atlas company add` + are generic over the `ats_type` string, so no downstream code changed. Each + adapter's `detect(url)` recognizes both the public board URL and the raw API URL: + **Lever** (`jobs.lever.co/` / `api.lever.co/v0/postings/`, a raw JSON + array); **Ashby** (`jobs.ashbyhq.com/` / + `api.ashbyhq.com/posting-api/job-board/`, deriving the external id from the + job's URL since Ashby exposes none, and skipping unlisted postings); and + **Workday** (`..myworkdayjobs.com` — a POST-based, paginated CxS API, + with a compound `::` board reference). Known limitations: Lever + polls the US base only, and Workday apply URLs omit any locale segment. +- **POST support on the `Fetcher` seam** (`atlas.scrape.fetcher`): the `Fetcher` / + `BrowserFetcher` protocols and `default_fetcher` gained optional, GET-defaulted + `method` / `json_body` / `headers` params so an ATS adapter can issue a JSON POST + (Workday's CxS API) over the same boundary. Backward compatible — existing GET + callers are unchanged; the test `FakeFetcher` now records these and can replay a + sequence of pages for pagination. - **Discover screen** (`atlas.tui.screens.discover`): the TUI's ranked queue of scored postings (PROJECT.md §8 screen #2) — the piece that makes the daemon's discovery/scoring work visible and actionable, closing Journey B (background diff --git a/README.md b/README.md index 3895145..a2c0b4a 100644 --- a/README.md +++ b/README.md @@ -135,13 +135,14 @@ atlas discover # poll the watchlist now for new postings (--json) ``` `atlas company add` auto-detects the ATS provider and board token from a -careers/board URL (e.g. `https://boards.greenhouse.io/`) — **Greenhouse** -is supported today, with Lever/Ashby/Workday to follow — and adds the company to -your watchlist; an unrecognized URL is refused with the list of supported -providers. `atlas discover` runs one poll now, fetching each enabled board, -normalizing and de-duplicating its postings (against both what discovery and -`atlas add` already saved), and saving the new ones; it's AI-free and points you -at `atlas score` (or the daemon) to score them. +careers/board URL — **Greenhouse** (`boards.greenhouse.io/`), **Lever** +(`jobs.lever.co/`), **Ashby** (`jobs.ashbyhq.com/`), and **Workday** +(`..myworkdayjobs.com/`) are supported, and each provider's raw +API URL is accepted too — then adds the company to your watchlist; an unrecognized +URL is refused with the list of supported providers. `atlas discover` runs one poll +now, fetching each enabled board, normalizing and de-duplicating its postings +(against both what discovery and `atlas add` already saved), and saving the new +ones; it's AI-free and points you at `atlas score` (or the daemon) to score them. Run background work with the daemon: diff --git a/docs/PROJECT.md b/docs/PROJECT.md index 54995b3..cae288e 100644 --- a/docs/PROJECT.md +++ b/docs/PROJECT.md @@ -1106,14 +1106,15 @@ The document specs everything; build order is phased. Each phase is independentl (`poll_interval_minutes`), and `atlas daemon start|stop|status`. **Remaining:** the **IPC surface** (Unix socket / Windows named pipe) for the TUI to trigger work + stream progress, and wiring the poll to real discovery sources once the adapters below land.)* -- [ ] **Company watchlist** + ATS adapters (Greenhouse, Lever, Ashby, Workday). - *(**Greenhouse ✅** — an extensible `AtsAdapter` Protocol + registry - (`atlas.discovery.ats`) with URL-based provider detection; the Greenhouse boards-API - adapter; a watchlist on the existing `company`/`job_source` tables (no migration); - `run_discovery_poll` (best-effort per source, dedup by external id + apply-URL hash), - wired into the daemon before the scoring poll; and `atlas company add|list` / - `atlas discover`. **Remaining:** the Lever/Ashby/Workday adapters (drop into the same - registry) and URL auto-detection for them.)* +- [x] **Company watchlist** + ATS adapters (Greenhouse, Lever, Ashby, Workday). *(✅ an + extensible `AtsAdapter` Protocol + registry (`atlas.discovery.ats`) with URL-based provider + detection; a watchlist on the existing `company`/`job_source` tables (no migration); + `run_discovery_poll` (best-effort per source, dedup by external id + apply-URL hash), wired + into the daemon before the scoring poll; and `atlas company add|list` / `atlas discover`. + **All four adapters** land: Greenhouse (boards API) and Lever (raw-array postings API) and + Ashby (job-board API, id derived from the job URL) over GET; Workday (per-tenant CxS API) over + a POST-with-pagination extension of the `Fetcher` seam, with a compound `::` + board reference. Follow-ups: Lever EU region and Workday apply-URL locale.)* - [ ] **Aggregator** adapters + saved keyword searches. - [x] Dedup + scored **Discover** queue in the TUI. *(✅ `DiscoverScreen` — a ranked queue of scored postings (`matching.repository.list_scored_postings`) with tailor / dismiss / save / diff --git a/docs/STATUS.md b/docs/STATUS.md index dec714c..7a0a5e8 100644 --- a/docs/STATUS.md +++ b/docs/STATUS.md @@ -9,18 +9,16 @@ > whenever a roadmap item lands, tick it here and move the "Next up" pointer. A stale > STATUS.md is a bug. -- **Last updated:** 2026-08-05 (**the TUI Discover queue landed** — `DiscoverScreen` - ranks scored postings by fit with tailor / dismiss / save / open-URL actions and - drill-in, backed by `list_scored_postings`, a new `JobPosting.queue_status` - (migration), and a `UrlOpener` seam. Journey B — background discovery → review → - tailor — is now closed end-to-end. Earlier the same phase: the daemon + scheduler - and the company watchlist + Greenhouse ATS discovery. **Next:** Lever/Ashby/Workday - adapters, aggregators, multiple profiles, and the daemon IPC surface) +- **Last updated:** 2026-08-05 (**the Lever, Ashby, and Workday ATS adapters landed** + — all four ATS providers the roadmap names now discover jobs, each a drop-in on the + registry; Workday required extending the `Fetcher` seam to POST + paginate. Earlier + the same phase: the daemon + scheduler, the Greenhouse adapter + watchlist, and the + TUI Discover queue. **Next:** aggregator adapters, multiple profiles, and the daemon + IPC surface) - **Current phase:** Phase 2 — Discovery & background 🚧 (daemon + scheduler ✅; - company watchlist + Greenhouse ATS discovery ✅; **scored Discover queue in the - TUI ✅** — `DiscoverScreen` + `list_scored_postings` + `queue_status` + - `UrlOpener`; **more ATS adapters, aggregators, multiple profiles, IPC next**). - Phase 1 core loop ✅ complete. + company watchlist + **all four ATS adapters ✅** (Greenhouse · Lever · Ashby · + Workday); scored Discover queue in the TUI ✅; **aggregators, multiple profiles, + IPC next**). Phase 1 core loop ✅ complete. - **Design source of truth:** [`docs/PROJECT.md`](./PROJECT.md) — especially the [phased roadmap](./PROJECT.md#15-phased-roadmap). - **Working agreement:** [`AGENTS.md`](../AGENTS.md) (branching, commits, tests, PR flow). @@ -30,28 +28,27 @@ ## ▶ Next up (do this next) **Phase 2 (Discovery & background) is well underway.** The **daemon + scheduler**, the -**company watchlist + Greenhouse ATS discovery**, and now the **scored Discover queue in the -TUI** have landed (see "What has landed"). The daemon discovers postings from watchlisted ATS -boards and scores them; `DiscoverScreen` (press `w`) presents them ranked by fit with tailor / -dismiss / save / open-URL actions — so Journey B (background discovery → review → tailor) works -end-to-end. +**company watchlist + all four ATS adapters** (Greenhouse, Lever, Ashby, Workday), and the +**scored Discover queue in the TUI** have landed (see "What has landed"). The daemon discovers +postings from watchlisted ATS boards across every supported provider and scores them; +`DiscoverScreen` (press `w`) presents them ranked by fit with tailor / dismiss / save / open-URL +actions — so Journey B (background discovery → review → tailor) works end-to-end. **Do these next to grow discovery (PROJECT.md §4.1, §5.4, §15):** -1. **More ATS adapters** — **Lever**, **Ashby**, **Workday**. Each is a new module in - `atlas.discovery.ats` implementing the `AtsAdapter` Protocol (a `detect(url)` hook + a - `list_postings` that fetches the board's public API through the injected `Fetcher`) plus one - entry in the `_ADAPTERS` registry tuple — no interface change. Greenhouse - (`ats/greenhouse.py`) is the worked example. -2. **Aggregator adapters** + saved keyword searches — the second discovery strategy (RemoteOK, +1. **Aggregator adapters** + saved keyword searches — the second discovery strategy (RemoteOK, Remotive, Adzuna, …), key-gated ones inactive until a key is pasted. A new `atlas.discovery.aggregators` package alongside `ats/`, feeding the same `persist_discovered`. -3. **Multiple profiles** fully wired (the Discover queue and scoring are single-active-profile +2. **Multiple profiles** fully wired (the Discover queue and scoring are single-active-profile today). Add the "owned by" claim convention (PROJECT.md §4.1) and a `busy_timeout` PRAGMA now that the daemon writes discovery rows while the TUI reads. -4. **IPC surface** — a local socket (Unix domain / Windows named pipe) so the TUI can trigger +3. **IPC surface** — a local socket (Unix domain / Windows named pipe) so the TUI can trigger "poll/tailor now" and stream progress. Follow the `platform/opener.py` seam (a Protocol + `sys.platform`-dispatched, pragma'd transport); a pure `handle_request` is the tested core. -5. **Desktop notifications** (`desktop-notifier`, §5.16) for new high-fit matches / deadlines. +4. **Desktop notifications** (`desktop-notifier`, §5.16) for new high-fit matches / deadlines. + +**Smaller ATS follow-ups (optional):** Lever EU-region boards (encode the region in `board_ref`); +Workday apply-URL locale (carry an optional locale in `board_ref`); SmartRecruiters / Recruitee / +Personio adapters (drop into the same registry). > **Deferred Phase-1 depth (optional, revisit as needed — not blockers for Phase 2):** > **PR 2b — tailoring depth**: `honesty_validate` traceability (§11), AI-phrase scrub (§5.7 @@ -104,6 +101,27 @@ The Phase-0 AI-provider checklist below is retained as historical reference. ## ✅ What has landed +Phase 2 · Lever, Ashby & Workday ATS adapters — `atlas.discovery.ats` (all four roadmap ATS +providers now discover jobs): + +- Three new adapters join Greenhouse on the registry, each a drop-in (a new module + one + `_ADAPTERS` entry — the poller/service/CLI stay generic over the `ats_type` string). Each + `detect(url)` accepts both the public board URL and the raw API URL. **Lever** + (`ats/lever.py`): `GET api.lever.co/v0/postings/?mode=json`, a **raw JSON array**. + **Ashby** (`ats/ashby.py`): `GET api.ashbyhq.com/posting-api/job-board/`; Ashby jobs + carry no id, so the external id is derived from the job URL, and unlisted postings are skipped. + **Workday** (`ats/workday.py`): the per-tenant **CxS API**, a POST that paginates — a compound + `::` board ref (detect emits it, list_postings parses it), offset paging to a + `_MAX_PAGES` cap with a warning if capped (no silent truncation). +- The `Fetcher` seam (`atlas.scrape.fetcher`) gained optional GET-defaulted + `method`/`json_body`/`headers` so Workday can POST over the same boundary — backward compatible + (existing GET callers unchanged); the test `FakeFetcher` records them and can replay a page + sequence for pagination. +- 959 tests at 100% line+branch; `mypy --strict` incl. win32; no new dependency. Each adapter + mirrors the Greenhouse test suite plus its own quirks (Lever raw-array, Ashby id-from-URL + + isListed filter, Workday POST body / pagination / cap-warning / compound-ref parsing). + Follow-ups: Lever EU region, Workday apply-URL locale. + Phase 2 · Scored Discover queue — `atlas.tui.screens.discover` (PROJECT.md §8 screen #2 — makes the discovery/scoring pipeline visible + actionable, closing Journey B): @@ -711,6 +729,6 @@ high-level state. |---|---|---| | 0 | Foundations (hygiene/CI · scaffold · config/DB/logging · AI providers) | ✅ **complete** — hygiene/CI · scaffold · config/keyring · data layer (SQLModel/SQLite WAL/Alembic) · logging · AI provider abstraction (core contract · CLI + API backends · failover · `atlas doctor` · capability probe) | | 1 | Core loop (onboarding · resume · scrape · scoring · tailoring · tracking · TUI) | ✅ **complete** — onboarding · master resume · paste-URL scrape · fit scoring · tailoring + cover letter + rendering · application tracking (state machine + CLI) · full TUI (Dashboard · Applications/Kanban · Application detail · Posting detail · Tailor workspace with background action workers). Optional depth (PR-2b tailoring / interactive editing) deferred | -| 2 | Discovery & background (daemon · ATS · aggregators · Discover queue) | 🚧 in progress — daemon + scheduler ✅ (APScheduler · `[discovery]` config · PID-file lifecycle · score-backlog poll · `atlas daemon start\|stop\|status`); company watchlist + Greenhouse ATS discovery ✅ (`atlas.discovery` registry + adapter · `run_discovery_poll` discover→score in the daemon · `atlas company add\|list` · `atlas discover`); **scored Discover queue in the TUI ✅** (`DiscoverScreen` ranked by fit · tailor/dismiss/save/open-URL · `list_scored_postings` · `queue_status` migration · `UrlOpener` seam); more ATS adapters (Lever/Ashby/Workday), aggregators, multiple profiles, IPC surface next | +| 2 | Discovery & background (daemon · ATS · aggregators · Discover queue) | 🚧 in progress — daemon + scheduler ✅ (APScheduler · `[discovery]` config · PID-file lifecycle · score-backlog poll · `atlas daemon start\|stop\|status`); company watchlist + **all four ATS adapters ✅** (Greenhouse · Lever · Ashby · Workday, on the `atlas.discovery.ats` registry · `run_discovery_poll` discover→score in the daemon · `atlas company add\|list` · `atlas discover`); scored Discover queue in the TUI ✅ (`DiscoverScreen` · `list_scored_postings` · `queue_status` · `UrlOpener`); aggregators, multiple profiles, IPC surface next | | 3 | Scheduling & status intelligence (CalDAV · email scan · Q&A drafting) | ⬜ not started | | 4 | Polish & depth (analytics · more adapters · scraping · DOCX · encryption) | ⬜ not started | diff --git a/src/atlas/discovery/ats/__init__.py b/src/atlas/discovery/ats/__init__.py index c23f41a..f6fd5cf 100644 --- a/src/atlas/discovery/ats/__init__.py +++ b/src/atlas/discovery/ats/__init__.py @@ -15,14 +15,22 @@ from __future__ import annotations +from atlas.discovery.ats.ashby import AshbyAdapter from atlas.discovery.ats.base import AtsAdapter from atlas.discovery.ats.greenhouse import GreenhouseAdapter +from atlas.discovery.ats.lever import LeverAdapter +from atlas.discovery.ats.workday import WorkdayAdapter from atlas.discovery.errors import UnknownAtsError __all__ = ["ATS_TYPES", "AtsAdapter", "detect_ats", "get_adapter"] #: Every registered ATS adapter, tried in order by :func:`detect_ats`. -_ADAPTERS: tuple[AtsAdapter, ...] = (GreenhouseAdapter(),) +_ADAPTERS: tuple[AtsAdapter, ...] = ( + GreenhouseAdapter(), + LeverAdapter(), + AshbyAdapter(), + WorkdayAdapter(), +) #: The registered provider names (sorted), for help text and error messages. ATS_TYPES: tuple[str, ...] = tuple(sorted(adapter.ats_type for adapter in _ADAPTERS)) diff --git a/src/atlas/discovery/ats/ashby.py b/src/atlas/discovery/ats/ashby.py new file mode 100644 index 0000000..97c3cc1 --- /dev/null +++ b/src/atlas/discovery/ats/ashby.py @@ -0,0 +1,149 @@ +"""The Ashby ATS adapter (PROJECT.md §5.4-A). + +Ashby exposes a public, unauthenticated **Job Posting API**: an organization's +board is identified by a *job board name* and its published postings are listed at +``https://api.ashbyhq.com/posting-api/job-board/{name}``, returning +``{"apiVersion": ..., "jobs": [...]}``. + +Detection is a pure, offline URL classifier covering both the public board URL and +the raw API URL the user might paste: + +- ``https://jobs.ashbyhq.com/`` — the name is the first path segment; +- ``https://api.ashbyhq.com/posting-api/job-board/`` — the name is the + segment after ``job-board``. + +Ashby job objects carry **no top-level id**, so the external id is derived from the +job's ``jobUrl`` (its last path segment is a UUID), falling back to ``applyUrl``. +Unlisted postings (``isListed`` false) are skipped. The listing fetch goes through +the injected :class:`~atlas.scrape.fetcher.Fetcher`, so the whole adapter runs +offline in tests (AGENTS.md §6.2). +""" + +from __future__ import annotations + +import html +import json +from typing import TYPE_CHECKING, Any +from urllib.parse import urlsplit + +from atlas.discovery.errors import DiscoveryError +from atlas.discovery.structure import DiscoveredPosting +from atlas.scrape.extract import extract_main_text +from atlas.scrape.structure import ScrapedPosting + +if TYPE_CHECKING: + from atlas.scrape.fetcher import Fetcher + +__all__ = ["AshbyAdapter"] + +#: Base URL of Ashby's public Job Posting API. +_API_BASE = "https://api.ashbyhq.com/posting-api/job-board" + +#: The public board host; the name is the first path segment. +_BOARD_HOST = "jobs.ashbyhq.com" + +#: The API host; the name is the path segment after ``job-board``. +_API_HOST = "api.ashbyhq.com" + + +class AshbyAdapter: + """Adapter for Ashby's public Job Posting API.""" + + ats_type = "ashby" + + def detect(self, url: str) -> str | None: + """Return the Ashby job-board name in ``url``, or ``None``. + + Pure and offline — see the module docstring for the recognized URL forms. + """ + parts = urlsplit(url.strip()) + host = (parts.hostname or "").lower() + segments = [segment for segment in parts.path.split("/") if segment] + if host == _BOARD_HOST: + return segments[0] if segments else None + if host == _API_HOST: + # .../posting-api/job-board/: the name follows "job-board". + if "job-board" in segments: + index = segments.index("job-board") + remainder = segments[index + 1 :] + return remainder[0] if remainder else None + return None + return None + + def list_postings( + self, board_ref: str, *, fetcher: Fetcher, timeout_s: int + ) -> list[DiscoveredPosting]: + """Fetch and normalize every listed posting on the Ashby board ``board_ref``. + + Raises: + DiscoveryError: If the response is not JSON or lacks a ``jobs`` list. + FetchError: Propagated from the fetcher when the board can't be fetched. + """ + url = f"{_API_BASE}/{board_ref}?includeCompensation=false" + result = fetcher(url, timeout_s=timeout_s) + try: + payload = json.loads(result.body) + except json.JSONDecodeError as exc: + raise DiscoveryError( + f"Ashby board {board_ref!r} returned a non-JSON response." + ) from exc + jobs = payload.get("jobs") if isinstance(payload, dict) else None + if not isinstance(jobs, list): + raise DiscoveryError(f"Ashby board {board_ref!r} returned no 'jobs' list.") + discovered: list[DiscoveredPosting] = [] + for job in jobs: + posting = _normalize_job(job) + if posting is not None: + discovered.append(posting) + return discovered + + +def _external_id(job: dict[str, Any]) -> str | None: + """Derive a stable external id from a job's ``jobUrl`` (else ``applyUrl``). + + Ashby jobs have no top-level id; the ``jobUrl``'s last path segment is a UUID. + Returns ``None`` when neither URL yields a non-empty segment. + """ + for key in ("jobUrl", "applyUrl"): + raw = job.get(key) + if not raw: + continue + segment = urlsplit(str(raw)).path.rstrip("/").rsplit("/", 1)[-1] + if segment: + return segment + return None + + +def _normalize_job(job: Any) -> DiscoveredPosting | None: + """Map one Ashby job onto a :class:`DiscoveredPosting`. + + Returns ``None`` (skipping the job) when the object is not a dict, is unlisted + (``isListed`` is ``False``), has no derivable external id, or is missing a + title or an apply URL — so a single malformed/unlisted job never fails the + board. + """ + if not isinstance(job, dict): + return None + if job.get("isListed") is False: + return None + external_id = _external_id(job) + title = job.get("title") + apply_url = job.get("applyUrl") or job.get("jobUrl") + if external_id is None or not title or not apply_url: + return None + plain = job.get("descriptionPlain") + description = ( + plain if plain else extract_main_text(html.unescape(job.get("descriptionHtml") or "")) + ) + return DiscoveredPosting( + external_id=external_id, + posting=ScrapedPosting( + title=title, + apply_url=apply_url, + location=job.get("location"), + employment_type=job.get("employmentType"), + remote_type=job.get("workplaceType"), + description=description, + posted_at=job.get("publishedAt"), + ), + ) diff --git a/src/atlas/discovery/ats/lever.py b/src/atlas/discovery/ats/lever.py new file mode 100644 index 0000000..7363d1b --- /dev/null +++ b/src/atlas/discovery/ats/lever.py @@ -0,0 +1,136 @@ +"""The Lever ATS adapter (PROJECT.md §5.4-A). + +Lever exposes a public, unauthenticated **Postings API**: a company's board is +identified by a *site* token and its published postings are listed at +``https://api.lever.co/v0/postings/{site}?mode=json``, which returns a **raw JSON +array** of postings (not a wrapped object). + +Detection is a pure, offline URL classifier covering both the public board URL and +the raw API URL the user might paste: + +- ``https://jobs.lever.co/`` (and the EU host ``jobs.eu.lever.co``) — the + site is the first path segment; +- ``https://api.lever.co/v0/postings/`` (and ``api.eu.lever.co``) — the site + is the segment after ``postings``. + +The listing fetch goes through the injected :class:`~atlas.scrape.fetcher.Fetcher` +so the whole adapter runs offline in tests with a :class:`FakeFetcher` (AGENTS.md +§6.2). + +**Known limitation:** ``list_postings`` always polls the US base (``api.lever.co``). +An EU-only board added via an ``eu`` URL still watchlists (``detect`` returns the +bare site token) but may 404 on poll; encoding the region in ``board_ref`` is a +follow-up. +""" + +from __future__ import annotations + +import html +import json +from typing import TYPE_CHECKING, Any +from urllib.parse import urlsplit + +from atlas.discovery.errors import DiscoveryError +from atlas.discovery.structure import DiscoveredPosting +from atlas.scrape.extract import extract_main_text +from atlas.scrape.structure import ScrapedPosting + +if TYPE_CHECKING: + from atlas.scrape.fetcher import Fetcher + +__all__ = ["LeverAdapter"] + +#: Base URL of Lever's public Postings API (US). +_API_BASE = "https://api.lever.co/v0/postings" + +#: Public board hosts (US + EU); the site is the first path segment. +_BOARD_HOSTS = frozenset({"jobs.lever.co", "jobs.eu.lever.co"}) + +#: API hosts (US + EU); the site is the path segment after ``postings``. +_API_HOSTS = frozenset({"api.lever.co", "api.eu.lever.co"}) + + +class LeverAdapter: + """Adapter for Lever's public Postings API.""" + + ats_type = "lever" + + def detect(self, url: str) -> str | None: + """Return the Lever site token in ``url``, or ``None``. + + Pure and offline — see the module docstring for the recognized URL forms. + """ + parts = urlsplit(url.strip()) + host = (parts.hostname or "").lower() + segments = [segment for segment in parts.path.split("/") if segment] + if host in _BOARD_HOSTS: + return segments[0] if segments else None + if host in _API_HOSTS: + # .../v0/postings/: the token follows the "postings" segment. + if "postings" in segments: + index = segments.index("postings") + remainder = segments[index + 1 :] + return remainder[0] if remainder else None + return None + return None + + def list_postings( + self, board_ref: str, *, fetcher: Fetcher, timeout_s: int + ) -> list[DiscoveredPosting]: + """Fetch and normalize every posting on the Lever board ``board_ref``. + + Raises: + DiscoveryError: If the response is not JSON or is not a list. + FetchError: Propagated from the fetcher when the board can't be fetched. + """ + url = f"{_API_BASE}/{board_ref}?mode=json" + result = fetcher(url, timeout_s=timeout_s) + try: + payload = json.loads(result.body) + except json.JSONDecodeError as exc: + raise DiscoveryError( + f"Lever board {board_ref!r} returned a non-JSON response." + ) from exc + if not isinstance(payload, list): + raise DiscoveryError(f"Lever board {board_ref!r} did not return a postings list.") + discovered: list[DiscoveredPosting] = [] + for job in payload: + posting = _normalize_job(job) + if posting is not None: + discovered.append(posting) + return discovered + + +def _normalize_job(job: Any) -> DiscoveredPosting | None: + """Map one Lever posting onto a :class:`DiscoveredPosting`. + + Returns ``None`` (skipping the posting) when the object is not a dict or is + missing the fields Atlas requires — an id, a title (``text``), and an apply + URL — so a single malformed posting never fails the whole board. + """ + if not isinstance(job, dict): + return None + job_id = job.get("id") + title = job.get("text") + apply_url = job.get("hostedUrl") or job.get("applyUrl") + if not job_id or not title or not apply_url: + return None + categories = job.get("categories") + if not isinstance(categories, dict): + categories = {} + plain = job.get("descriptionPlain") + description = plain if plain else extract_main_text(html.unescape(job.get("description") or "")) + posted_at = job.get("createdAt") + return DiscoveredPosting( + external_id=str(job_id), + posting=ScrapedPosting( + title=title, + apply_url=apply_url, + location=categories.get("location"), + employment_type=categories.get("commitment"), + team=categories.get("team"), + remote_type=job.get("workplaceType"), + description=description, + posted_at=str(posted_at) if posted_at is not None else None, + ), + ) diff --git a/src/atlas/discovery/ats/workday.py b/src/atlas/discovery/ats/workday.py new file mode 100644 index 0000000..1f33edd --- /dev/null +++ b/src/atlas/discovery/ats/workday.py @@ -0,0 +1,204 @@ +"""The Workday ATS adapter (PROJECT.md §5.4-A). + +Workday hosts each customer on a per-tenant, per-datacenter domain +(``..myworkdayjobs.com``) and exposes its careers site through the +**CxS API** — unlike the other adapters this is a **POST** with a JSON body and +**pagination**: + +``POST https://..myworkdayjobs.com/wday/cxs///jobs`` +with body ``{"appliedFacets": {}, "limit": N, "offset": M, "searchText": ""}`` and +``Accept: application/json``. Each page returns ``{"total": N, "jobPostings": [...]}``. + +Because the registry stores a single ``board_ref`` string but Workday needs three +values, :meth:`detect` produces a **compound** ``"::"`` token +that :meth:`list_postings` parses back apart. + +Detection covers the public board URL and the raw CxS API URL: + +- ``https://..myworkdayjobs.com//`` — the tenant is the + first host label, the datacenter the second (a ``wd``-prefixed label), and the + site the last path segment (a leading ``xx-XX`` locale is skipped); +- ``https://..myworkdayjobs.com/wday/cxs///jobs`` — the + tenant/site come from the path after ``cxs``. + +**Known limitation:** the apply URL omits any locale segment; a tenant that +requires one may 404 — carrying an optional locale in ``board_ref`` is a follow-up. +""" + +from __future__ import annotations + +import json +import logging +import re +from typing import TYPE_CHECKING, Any +from urllib.parse import urlsplit + +from atlas.discovery.errors import DiscoveryError +from atlas.discovery.structure import DiscoveredPosting +from atlas.scrape.structure import ScrapedPosting + +if TYPE_CHECKING: + from atlas.scrape.fetcher import Fetcher + +__all__ = ["WorkdayAdapter"] + +_LOGGER = logging.getLogger(__name__) + +#: Workday's public jobs domain suffix. +_HOST_SUFFIX = ".myworkdayjobs.com" + +#: A Workday datacenter host label (e.g. ``wd5``). +_DATACENTER = re.compile(r"^wd\d+$") + +#: A locale path segment (e.g. ``en-US``) to skip when finding the site. +_LOCALE = re.compile(r"^[a-z]{2}-[A-Z]{2}$") + +#: Postings per CxS page, and the page cap (→ ≤200 postings) to bound the poll. +_LIMIT = 20 +_MAX_PAGES = 10 + + +class WorkdayAdapter: + """Adapter for Workday's per-tenant CxS careers API.""" + + ats_type = "workday" + + def detect(self, url: str) -> str | None: + """Return the compound ``"::"`` token in ``url``, or ``None``. + + Pure and offline — see the module docstring for the recognized URL forms. + """ + parts = urlsplit(url.strip()) + host = (parts.hostname or "").lower() + if not host.endswith(_HOST_SUFFIX): + return None + labels = host.split(".") + if len(labels) < 2 or not _DATACENTER.match(labels[1]): + return None + tenant, datacenter = labels[0], labels[1] + segments = [segment for segment in parts.path.split("/") if segment] + # Raw CxS API URL: .../wday/cxs///jobs. + if "cxs" in segments: + index = segments.index("cxs") + remainder = segments[index + 1 :] + # remainder = [, , "jobs"?]; need at least tenant + site. + if len(remainder) < 2: + return None + site = remainder[1] + else: + # Board URL: //; the site is the last non-empty segment. + usable = [segment for segment in segments if not _LOCALE.match(segment)] + if not usable: + return None + site = usable[-1] + return f"{tenant}:{datacenter}:{site}" + + def list_postings( + self, board_ref: str, *, fetcher: Fetcher, timeout_s: int + ) -> list[DiscoveredPosting]: + """Fetch and normalize every posting on the Workday board ``board_ref``. + + Pages through the CxS API until all postings are collected or the page cap + is hit (logging a warning if capped, per AGENTS.md — no silent truncation). + + Raises: + DiscoveryError: If ``board_ref`` is malformed or a page response is not + usable JSON. + FetchError: Propagated from the fetcher when the board can't be fetched. + """ + tenant, datacenter, site = _parse_board_ref(board_ref) + base = f"https://{tenant}.{datacenter}{_HOST_SUFFIX}" + endpoint = f"{base}/wday/cxs/{tenant}/{site}/jobs" + discovered: list[DiscoveredPosting] = [] + total = 0 + for page in range(_MAX_PAGES): + body = { + "appliedFacets": {}, + "limit": _LIMIT, + "offset": page * _LIMIT, + "searchText": "", + } + result = fetcher( + endpoint, + timeout_s=timeout_s, + method="POST", + json_body=body, + headers={"Accept": "application/json"}, + ) + total, jobs = _parse_page(result.body, board_ref) + if not jobs: + break + for job in jobs: + posting = _normalize_job(job, base) + if posting is not None: + discovered.append(posting) + if len(discovered) >= total: + break + else: + # Ran the full page range without collecting everything (the loop + # breaks as soon as len >= total), so the board was capped — surface it + # rather than silently truncating (AGENTS.md). + _LOGGER.warning( + "Workday board %r capped at %d of %d postings.", + board_ref, + len(discovered), + total, + ) + return discovered + + +def _parse_board_ref(board_ref: str) -> tuple[str, str, str]: + """Split a compound ``"::"`` board ref into its three parts. + + Raises: + DiscoveryError: If ``board_ref`` is not exactly three colon-separated parts. + """ + parts = board_ref.split(":") + if len(parts) != 3 or not all(parts): + raise DiscoveryError(f"Malformed Workday board reference {board_ref!r}.") + tenant, datacenter, site = parts + return tenant, datacenter, site + + +def _parse_page(raw: str, board_ref: str) -> tuple[int, list[Any]]: + """Parse one CxS page body into ``(total, jobPostings)``. + + Raises: + DiscoveryError: If the body is not JSON, or lacks an integer ``total`` and a + ``jobPostings`` list. + """ + try: + payload = json.loads(raw) + except json.JSONDecodeError as exc: + raise DiscoveryError(f"Workday board {board_ref!r} returned a non-JSON response.") from exc + total = payload.get("total") if isinstance(payload, dict) else None + jobs = payload.get("jobPostings") if isinstance(payload, dict) else None + if not isinstance(total, int) or not isinstance(jobs, list): + raise DiscoveryError(f"Workday board {board_ref!r} returned an unexpected response.") + return total, jobs + + +def _normalize_job(job: Any, base: str) -> DiscoveredPosting | None: + """Map one Workday jobPosting onto a :class:`DiscoveredPosting`. + + Returns ``None`` (skipping the posting) when the object is not a dict or has no + ``externalPath`` (needed for both the apply URL and the external id). + """ + if not isinstance(job, dict): + return None + external_path = job.get("externalPath") + title = job.get("title") + if not external_path or not title: + return None + # The JR id is the trailing "_JR123" of the external path; fall back to the + # whole path when there is no underscore. + external_id = str(external_path).rsplit("_", 1)[-1] + return DiscoveredPosting( + external_id=external_id, + posting=ScrapedPosting( + title=title, + apply_url=f"{base}{external_path}", + location=job.get("locationsText"), + posted_at=job.get("postedOn"), + ), + ) diff --git a/src/atlas/scrape/fetcher.py b/src/atlas/scrape/fetcher.py index 5278dcc..0c685af 100644 --- a/src/atlas/scrape/fetcher.py +++ b/src/atlas/scrape/fetcher.py @@ -17,12 +17,15 @@ from __future__ import annotations from dataclasses import dataclass -from typing import Protocol, runtime_checkable +from typing import TYPE_CHECKING, Any, Protocol, runtime_checkable import httpx from atlas.scrape.errors import FetchError +if TYPE_CHECKING: + from collections.abc import Mapping + __all__ = [ "BrowserFetcher", "FetchResult", @@ -58,9 +61,21 @@ class Fetcher(Protocol): Implementations must follow redirects, decode the body as text, and raise :class:`~atlas.scrape.errors.FetchError` on a network failure or a non-success HTTP status. + + The default is a plain ``GET``; ``method`` / ``json_body`` / ``headers`` are + optional so an ATS adapter can issue a JSON ``POST`` (e.g. Workday's CxS API) + over the same seam. Existing callers pass none of these and are unaffected. """ - def __call__(self, url: str, *, timeout_s: int) -> FetchResult: + def __call__( + self, + url: str, + *, + timeout_s: int, + method: str = "GET", + json_body: Mapping[str, Any] | None = None, + headers: Mapping[str, str] | None = None, + ) -> FetchResult: """Fetch ``url`` and return its :class:`FetchResult`.""" @@ -73,11 +88,26 @@ class BrowserFetcher(Protocol): fallback and consulted only when a static fetch looks JS-rendered. """ - def __call__(self, url: str, *, timeout_s: int) -> FetchResult: + def __call__( + self, + url: str, + *, + timeout_s: int, + method: str = "GET", + json_body: Mapping[str, Any] | None = None, + headers: Mapping[str, str] | None = None, + ) -> FetchResult: """Render ``url`` in a browser and return its :class:`FetchResult`.""" -def default_fetcher(url: str, *, timeout_s: int) -> FetchResult: # pragma: no cover +def default_fetcher( + url: str, + *, + timeout_s: int, + method: str = "GET", + json_body: Mapping[str, Any] | None = None, + headers: Mapping[str, str] | None = None, +) -> FetchResult: # pragma: no cover """Fetch ``url`` over HTTP with ``httpx``, following redirects. This thin network boundary carries ``# pragma: no cover`` because the default @@ -86,15 +116,22 @@ def default_fetcher(url: str, *, timeout_s: int) -> FetchResult: # pragma: no c statuses and transport errors are normalized to :class:`~atlas.scrape.errors.FetchError`. + A plain ``GET`` by default; pass ``method="POST"`` with a ``json_body`` for an + ATS API that requires it (Workday). ``headers`` are merged on top of the + default ``User-Agent``. + Raises: FetchError: On a transport error or a non-success HTTP status. """ + merged_headers = {"User-Agent": _USER_AGENT, **(headers or {})} try: - response = httpx.get( + response = httpx.request( + method, url, follow_redirects=True, timeout=timeout_s, - headers={"User-Agent": _USER_AGENT}, + headers=merged_headers, + json=json_body, ) response.raise_for_status() except httpx.HTTPError as exc: diff --git a/tests/conftest.py b/tests/conftest.py index c2f29a3..2081b05 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -12,7 +12,7 @@ from collections.abc import Callable, Iterator, Mapping, Sequence from dataclasses import dataclass from pathlib import Path -from typing import TYPE_CHECKING, Protocol +from typing import TYPE_CHECKING, Any, Protocol import pytest from sqlmodel import SQLModel @@ -213,17 +213,25 @@ class FetchCall: url: str timeout_s: int + method: str = "GET" + json_body: Mapping[str, Any] | None = None + headers: Mapping[str, str] | None = None class FakeFetcher: """A scripted, offline :class:`~atlas.scrape.fetcher.Fetcher` for tests. Returns ``result`` for every call, or raises ``raises`` (e.g. a - :class:`~atlas.scrape.errors.FetchError`) instead. Every invocation is - recorded on :attr:`calls` so tests assert on the url and timeout the scraper - passed — without any real HTTP request (AGENTS.md §6.2). Its shape also - satisfies the ``BrowserFetcher`` protocol, so the same double stands in for - the JS-render fallback seam. + :class:`~atlas.scrape.errors.FetchError`) instead. Every invocation — including + the ``method`` / ``json_body`` / ``headers`` a POST adapter passes — is recorded + on :attr:`calls` so tests assert on exactly what the scraper sent, without any + real HTTP request (AGENTS.md §6.2). Its shape also satisfies the + ``BrowserFetcher`` protocol, so the same double stands in for the JS-render seam. + + Pass ``results`` (a sequence of :class:`FetchResult`) instead of a single + ``result`` to replay different responses per call — the paginated Workday poll + needs one page per offset. The sequence is consumed in order; calling past its + end raises, so a test that under-supplies pages fails loudly. """ def __init__( @@ -231,18 +239,42 @@ def __init__( result: FetchResult | None = None, *, raises: BaseException | None = None, + results: list[FetchResult] | None = None, ) -> None: - """Store the scripted result or exception to replay.""" + """Store the scripted single result / result sequence / exception to replay.""" self._result = result self._raises = raises + self._results = results + self._index = 0 self.calls: list[FetchCall] = [] - def __call__(self, url: str, *, timeout_s: int) -> FetchResult: - """Record the call and return the scripted result, or raise.""" - self.calls.append(FetchCall(url=url, timeout_s=timeout_s)) + def __call__( + self, + url: str, + *, + timeout_s: int, + method: str = "GET", + json_body: Mapping[str, Any] | None = None, + headers: Mapping[str, str] | None = None, + ) -> FetchResult: + """Record the call and return the next scripted result, or raise.""" + self.calls.append( + FetchCall( + url=url, + timeout_s=timeout_s, + method=method, + json_body=json_body, + headers=headers, + ) + ) if self._raises is not None: raise self._raises - assert self._result is not None, "FakeFetcher needs a result or a raises" + if self._results is not None: + assert self._index < len(self._results), "FakeFetcher ran out of scripted results" + result = self._results[self._index] + self._index += 1 + return result + assert self._result is not None, "FakeFetcher needs a result, results, or a raises" return self._result @@ -254,6 +286,7 @@ def __call__( result: FetchResult | None = ..., *, raises: BaseException | None = ..., + results: list[FetchResult] | None = ..., ) -> FakeFetcher: """Build a :class:`FakeFetcher` from a scripted result or error.""" ... @@ -267,8 +300,9 @@ def factory( result: FetchResult | None = None, *, raises: BaseException | None = None, + results: list[FetchResult] | None = None, ) -> FakeFetcher: - return FakeFetcher(result, raises=raises) + return FakeFetcher(result, raises=raises, results=results) return factory diff --git a/tests/test_discovery_ashby.py b/tests/test_discovery_ashby.py new file mode 100644 index 0000000..856c5aa --- /dev/null +++ b/tests/test_discovery_ashby.py @@ -0,0 +1,151 @@ +"""Tests for the Ashby adapter in :mod:`atlas.discovery.ats.ashby`. + +Detection is pure over URL strings; listing is exercised offline through a +scripted :class:`FakeFetcher` replaying a recorded payload — no real HTTP +(AGENTS.md §6.2). +""" + +from __future__ import annotations + +import json + +import pytest + +from atlas.discovery.ats.ashby import AshbyAdapter +from atlas.discovery.errors import DiscoveryError +from atlas.scrape.errors import FetchError +from atlas.scrape.fetcher import FetchResult +from tests.conftest import FakeFetcher + +_TIMEOUT = 30 + +_UUID_1 = "11111111-1111-1111-1111-111111111111" +_UUID_2 = "22222222-2222-2222-2222-222222222222" +_UUID_3 = "33333333-3333-3333-3333-333333333333" + +#: A recorded Ashby job-board payload — jobs carry no top-level id. +_BOARD = { + "apiVersion": "1", + "jobs": [ + { + "title": "Backend Engineer", + "location": "Remote - US", + "employmentType": "FullTime", + "workplaceType": "Remote", + "descriptionHtml": "<p>Build & run services.</p>", + "descriptionPlain": "Build & run services.", + "publishedAt": "2026-02-01T09:00:00Z", + "jobUrl": f"https://jobs.ashbyhq.com/acme/{_UUID_1}", + "applyUrl": f"https://jobs.ashbyhq.com/acme/{_UUID_1}/application", + "isListed": True, + }, + { + # No descriptionPlain → strip descriptionHtml; no applyUrl → jobUrl is + # the apply target AND the external-id source. + "title": "ML Engineer", + "location": "NYC", + "descriptionHtml": "<p>Ship models.</p>", + "jobUrl": f"https://jobs.ashbyhq.com/acme/{_UUID_2}", + }, + { + # jobUrl reduces to an empty segment → the id falls through to applyUrl, + # whose last path segment is the UUID. + "title": "Data Engineer", + "jobUrl": "https://jobs.ashbyhq.com/", + "applyUrl": f"https://jobs.ashbyhq.com/acme/{_UUID_3}", + }, + # Unlisted → skipped. + { + "title": "Hidden role", + "jobUrl": "https://jobs.ashbyhq.com/acme/hidden", + "isListed": False, + }, + # No jobUrl/applyUrl → no derivable id → skipped. + {"title": "No urls"}, + # Not an object → skipped. + "not-a-job", + ], +} + + +def _result(body: str) -> FetchResult: + return FetchResult( + url="https://api.ashbyhq.com/posting-api/job-board/acme?includeCompensation=false", + status_code=200, + content_type="application/json", + body=body, + ) + + +@pytest.mark.parametrize( + ("url", "token"), + [ + ("https://jobs.ashbyhq.com/acme", "acme"), + ("https://jobs.ashbyhq.com/acme/some-job", "acme"), + ("HTTPS://Jobs.AshbyHQ.com/Acme/", "Acme"), + ("https://api.ashbyhq.com/posting-api/job-board/acme", "acme"), + ("https://api.ashbyhq.com/posting-api/job-board/acme?includeCompensation=true", "acme"), + ], +) +def test_detect_recognizes_urls(url: str, token: str) -> None: + assert AshbyAdapter().detect(url) == token + + +@pytest.mark.parametrize( + "url", + [ + "https://jobs.lever.co/acme", # a different ATS + "https://example.com/careers", # not ashby + "https://jobs.ashbyhq.com/", # no name segment + "https://api.ashbyhq.com/posting-api/job-board", # no name after job-board + "https://api.ashbyhq.com/other/path", # no job-board segment + ], +) +def test_detect_rejects_unrecognized_urls(url: str) -> None: + assert AshbyAdapter().detect(url) is None + + +def test_list_postings_normalizes_jobs() -> None: + fetcher = FakeFetcher(_result(json.dumps(_BOARD))) + postings = AshbyAdapter().list_postings("acme", fetcher=fetcher, timeout_s=_TIMEOUT) + # Unlisted, no-urls, and non-dict jobs are skipped. + assert [p.external_id for p in postings] == [_UUID_1, _UUID_2, _UUID_3] + first = postings[0] + assert first.posting.title == "Backend Engineer" + assert first.posting.apply_url == f"https://jobs.ashbyhq.com/acme/{_UUID_1}/application" + assert first.posting.location == "Remote - US" + assert first.posting.employment_type == "FullTime" + assert first.posting.remote_type == "Remote" + assert first.posting.description == "Build & run services." # descriptionPlain + assert first.posting.posted_at == "2026-02-01T09:00:00Z" + second = postings[1] + # No applyUrl → jobUrl is both the apply target and the id source. + assert second.posting.apply_url == f"https://jobs.ashbyhq.com/acme/{_UUID_2}" + assert second.posting.description == "Ship models." # HTML stripped + assert fetcher.calls[0].url == ( + "https://api.ashbyhq.com/posting-api/job-board/acme?includeCompensation=false" + ) + + +def test_list_postings_empty_board() -> None: + fetcher = FakeFetcher(_result(json.dumps({"apiVersion": "1", "jobs": []}))) + assert AshbyAdapter().list_postings("acme", fetcher=fetcher, timeout_s=_TIMEOUT) == [] + + +def test_list_postings_non_json_raises() -> None: + fetcher = FakeFetcher(_result("nope")) + with pytest.raises(DiscoveryError, match="non-JSON"): + AshbyAdapter().list_postings("acme", fetcher=fetcher, timeout_s=_TIMEOUT) + + +@pytest.mark.parametrize("body", ['{"apiVersion": "1"}', "[]", '{"jobs": "nope"}']) +def test_list_postings_missing_jobs_raises(body: str) -> None: + fetcher = FakeFetcher(_result(body)) + with pytest.raises(DiscoveryError, match="no 'jobs' list"): + AshbyAdapter().list_postings("acme", fetcher=fetcher, timeout_s=_TIMEOUT) + + +def test_list_postings_propagates_fetch_error() -> None: + fetcher = FakeFetcher(raises=FetchError("boom")) + with pytest.raises(FetchError): + AshbyAdapter().list_postings("acme", fetcher=fetcher, timeout_s=_TIMEOUT) diff --git a/tests/test_discovery_ats_registry.py b/tests/test_discovery_ats_registry.py index 3ceb8c3..93720c9 100644 --- a/tests/test_discovery_ats_registry.py +++ b/tests/test_discovery_ats_registry.py @@ -5,27 +5,45 @@ import pytest from atlas.discovery.ats import ATS_TYPES, detect_ats, get_adapter +from atlas.discovery.ats.ashby import AshbyAdapter from atlas.discovery.ats.greenhouse import GreenhouseAdapter +from atlas.discovery.ats.lever import LeverAdapter +from atlas.discovery.ats.workday import WorkdayAdapter from atlas.discovery.errors import UnknownAtsError -def test_ats_types_lists_greenhouse() -> None: - assert ATS_TYPES == ("greenhouse",) +def test_ats_types_lists_registered_providers() -> None: + assert ATS_TYPES == ("ashby", "greenhouse", "lever", "workday") -def test_get_adapter_resolves_greenhouse() -> None: +def test_get_adapter_resolves_each_provider() -> None: assert isinstance(get_adapter("greenhouse"), GreenhouseAdapter) + assert isinstance(get_adapter("lever"), LeverAdapter) + assert isinstance(get_adapter("ashby"), AshbyAdapter) + assert isinstance(get_adapter("workday"), WorkdayAdapter) def test_get_adapter_unknown_raises() -> None: + # smartrecruiters is a documented-but-unregistered provider (PROJECT.md §5.4-A). with pytest.raises(UnknownAtsError) as excinfo: - get_adapter("lever") - assert excinfo.value.ats_type == "lever" + get_adapter("smartrecruiters") + assert excinfo.value.ats_type == "smartrecruiters" assert "greenhouse" in str(excinfo.value) -def test_detect_ats_classifies_greenhouse_url() -> None: - assert detect_ats("https://boards.greenhouse.io/acme") == ("greenhouse", "acme") +@pytest.mark.parametrize( + ("url", "expected"), + [ + ("https://boards.greenhouse.io/acme", ("greenhouse", "acme")), + ("https://jobs.lever.co/acme", ("lever", "acme")), + ("https://api.lever.co/v0/postings/acme", ("lever", "acme")), + ("https://jobs.ashbyhq.com/acme", ("ashby", "acme")), + ("https://api.ashbyhq.com/posting-api/job-board/acme", ("ashby", "acme")), + ("https://acme.wd5.myworkdayjobs.com/careers", ("workday", "acme:wd5:careers")), + ], +) +def test_detect_ats_classifies_urls(url: str, expected: tuple[str, str]) -> None: + assert detect_ats(url) == expected def test_detect_ats_returns_none_for_unrecognized_url() -> None: diff --git a/tests/test_discovery_lever.py b/tests/test_discovery_lever.py new file mode 100644 index 0000000..01ffdd5 --- /dev/null +++ b/tests/test_discovery_lever.py @@ -0,0 +1,132 @@ +"""Tests for the Lever adapter in :mod:`atlas.discovery.ats.lever`. + +Detection is pure over URL strings; listing is exercised offline through a +scripted :class:`FakeFetcher` replaying a recorded raw-array payload — no real +HTTP (AGENTS.md §6.2). +""" + +from __future__ import annotations + +import json + +import pytest + +from atlas.discovery.ats.lever import LeverAdapter +from atlas.discovery.errors import DiscoveryError +from atlas.scrape.errors import FetchError +from atlas.scrape.fetcher import FetchResult +from tests.conftest import FakeFetcher + +_TIMEOUT = 30 + +#: A recorded Lever postings payload — a raw JSON ARRAY (not wrapped). +_POSTINGS = [ + { + "id": "abc-123", + "text": "Backend Engineer", + "hostedUrl": "https://jobs.lever.co/acme/abc-123", + "applyUrl": "https://jobs.lever.co/acme/abc-123/apply", + "categories": {"location": "Remote - US", "commitment": "Full-time", "team": "Platform"}, + "workplaceType": "remote", + "descriptionPlain": "Build reliable services.", + "createdAt": 1737000000000, + }, + { + # No descriptionPlain → falls back to stripping the HTML description; + # applyUrl is used when hostedUrl is absent; and no categories at all + # exercises the missing-categories guard (location/team stay None). + "id": "def-456", + "text": "ML Engineer", + "applyUrl": "https://jobs.lever.co/acme/def-456/apply", + "description": "<p>Train & ship models.</p>", + }, + # Malformed: missing id/text/apply-url — skipped, not fatal. + {"text": "No id here"}, + # Not even an object — skipped. + "not-a-posting", +] + + +def _result(body: str) -> FetchResult: + return FetchResult( + url="https://api.lever.co/v0/postings/acme?mode=json", + status_code=200, + content_type="application/json", + body=body, + ) + + +@pytest.mark.parametrize( + ("url", "token"), + [ + ("https://jobs.lever.co/acme", "acme"), + ("https://jobs.lever.co/acme/abc-123", "acme"), + ("https://jobs.eu.lever.co/acme", "acme"), + ("HTTPS://Jobs.Lever.CO/Acme/", "Acme"), + ("https://api.lever.co/v0/postings/acme", "acme"), + ("https://api.lever.co/v0/postings/acme?mode=json", "acme"), + ("https://api.eu.lever.co/v0/postings/acme", "acme"), + ], +) +def test_detect_recognizes_urls(url: str, token: str) -> None: + assert LeverAdapter().detect(url) == token + + +@pytest.mark.parametrize( + "url", + [ + "https://boards.greenhouse.io/acme", # a different ATS + "https://example.com/careers", # not lever + "https://jobs.lever.co/", # no site segment + "https://api.lever.co/v0/postings", # no site after "postings" + "https://api.lever.co/v0/other", # no "postings" segment + ], +) +def test_detect_rejects_unrecognized_urls(url: str) -> None: + assert LeverAdapter().detect(url) is None + + +def test_list_postings_normalizes_raw_array() -> None: + fetcher = FakeFetcher(_result(json.dumps(_POSTINGS))) + postings = LeverAdapter().list_postings("acme", fetcher=fetcher, timeout_s=_TIMEOUT) + assert [p.external_id for p in postings] == ["abc-123", "def-456"] + first = postings[0] + assert first.posting.title == "Backend Engineer" + assert first.posting.apply_url == "https://jobs.lever.co/acme/abc-123" # hostedUrl preferred + assert first.posting.location == "Remote - US" + assert first.posting.employment_type == "Full-time" + assert first.posting.team == "Platform" + assert first.posting.remote_type == "remote" + assert first.posting.description == "Build reliable services." + assert first.posting.posted_at == "1737000000000" + second = postings[1] + assert ( + second.posting.apply_url == "https://jobs.lever.co/acme/def-456/apply" + ) # applyUrl fallback + assert second.posting.description == "Train & ship models." # HTML unescaped + stripped + assert second.posting.location is None # no categories → location stays None + assert fetcher.calls[0].url == "https://api.lever.co/v0/postings/acme?mode=json" + assert fetcher.calls[0].timeout_s == _TIMEOUT + + +def test_list_postings_empty_board() -> None: + fetcher = FakeFetcher(_result("[]")) + assert LeverAdapter().list_postings("acme", fetcher=fetcher, timeout_s=_TIMEOUT) == [] + + +def test_list_postings_non_json_raises() -> None: + fetcher = FakeFetcher(_result("nope")) + with pytest.raises(DiscoveryError, match="non-JSON"): + LeverAdapter().list_postings("acme", fetcher=fetcher, timeout_s=_TIMEOUT) + + +def test_list_postings_non_list_raises() -> None: + fetcher = FakeFetcher(_result('{"postings": []}')) + with pytest.raises(DiscoveryError, match="postings list"): + LeverAdapter().list_postings("acme", fetcher=fetcher, timeout_s=_TIMEOUT) + + +def test_list_postings_propagates_fetch_error() -> None: + fetcher = FakeFetcher(raises=FetchError("boom")) + with pytest.raises(FetchError): + LeverAdapter().list_postings("acme", fetcher=fetcher, timeout_s=_TIMEOUT) diff --git a/tests/test_discovery_poller.py b/tests/test_discovery_poller.py index 3724885..2412f13 100644 --- a/tests/test_discovery_poller.py +++ b/tests/test_discovery_poller.py @@ -8,7 +8,7 @@ import json from datetime import UTC, datetime -from typing import TYPE_CHECKING +from typing import TYPE_CHECKING, Any from atlas.db import session_scope from atlas.db.models import JobSource @@ -20,6 +20,8 @@ from tests.conftest import FakeFetcher if TYPE_CHECKING: + from collections.abc import Mapping + from sqlalchemy.engine import Engine _POLLED = datetime(2026, 8, 4, 9, 0, tzinfo=UTC) @@ -113,7 +115,15 @@ class _SequencedFetcher: def __init__(self) -> None: self.calls = 0 - def __call__(self, url: str, *, timeout_s: int) -> FetchResult: + def __call__( + self, + url: str, + *, + timeout_s: int, + method: str = "GET", + json_body: Mapping[str, Any] | None = None, + headers: Mapping[str, str] | None = None, + ) -> FetchResult: self.calls += 1 if "broken" in url: raise FetchError("boom") @@ -128,14 +138,19 @@ def __call__(self, url: str, *, timeout_s: int) -> FetchResult: def test_poll_skips_unknown_provider(db_engine: Engine) -> None: # A source whose provider has no adapter → UnknownAtsError (a DiscoveryError) - # is caught and counted, never fetched. + # is caught and counted, never fetched. smartrecruiters is documented but not + # yet registered (PROJECT.md §5.4-A). with session_scope(db_engine) as session: company = get_or_create_company(session, name="Acme") assert company.id is not None session.add( JobSource( type="ats", - config={"ats_type": "lever", "board_token": "acme", "company_id": company.id}, + config={ + "ats_type": "smartrecruiters", + "board_token": "acme", + "company_id": company.id, + }, ) ) session.flush() diff --git a/tests/test_discovery_workday.py b/tests/test_discovery_workday.py new file mode 100644 index 0000000..5d47f48 --- /dev/null +++ b/tests/test_discovery_workday.py @@ -0,0 +1,190 @@ +"""Tests for the Workday adapter in :mod:`atlas.discovery.ats.workday`. + +Detection is pure over URL strings; listing (a paginated POST) is exercised +offline through a scripted :class:`FakeFetcher` — the ``results`` sequence replays +one page per offset — so no real HTTP happens (AGENTS.md §6.2). +""" + +from __future__ import annotations + +import json +import logging +from typing import Any + +import pytest + +from atlas.discovery.ats.workday import WorkdayAdapter +from atlas.discovery.errors import DiscoveryError +from atlas.scrape.errors import FetchError +from atlas.scrape.fetcher import FetchResult +from tests.conftest import FakeFetcher + +_TIMEOUT = 30 + + +def _page(jobs: list[Any], total: int) -> FetchResult: + return FetchResult( + url="https://nvidia.wd5.myworkdayjobs.com/wday/cxs/nvidia/careers/jobs", + status_code=200, + content_type="application/json", + body=json.dumps({"total": total, "jobPostings": jobs}), + ) + + +def _job(path: str, title: str = "Engineer") -> dict[str, object]: + return { + "title": title, + "externalPath": path, + "locationsText": "Santa Clara, CA", + "postedOn": "Posted 3 Days Ago", + } + + +@pytest.mark.parametrize( + ("url", "token"), + [ + # Board URLs. + ("https://nvidia.wd5.myworkdayjobs.com/careers", "nvidia:wd5:careers"), + ("https://nvidia.wd5.myworkdayjobs.com/en-US/careers", "nvidia:wd5:careers"), + ("https://nvidia.wd5.myworkdayjobs.com/en-US/careers/", "nvidia:wd5:careers"), + # Raw CxS API URL. + ( + "https://nvidia.wd5.myworkdayjobs.com/wday/cxs/nvidia/careers/jobs", + "nvidia:wd5:careers", + ), + ], +) +def test_detect_recognizes_urls(url: str, token: str) -> None: + assert WorkdayAdapter().detect(url) == token + + +@pytest.mark.parametrize( + "url", + [ + "https://jobs.lever.co/acme", # a different ATS + "https://example.com/careers", # not workday + "https://nvidia.careers.myworkdayjobs.com/careers", # 2nd label isn't wd\d+ + "https://nvidia.wd5.myworkdayjobs.com/en-US", # only a locale, no site + "https://nvidia.wd5.myworkdayjobs.com/wday/cxs/nvidia", # no site after tenant + ], +) +def test_detect_rejects_unrecognized_urls(url: str) -> None: + assert WorkdayAdapter().detect(url) is None + + +def test_list_postings_posts_and_normalizes() -> None: + jobs = [ + _job("/job/Santa-Clara/Senior-Engineer_JR100"), + _job("/job/Remote/Staff-Engineer_JR200"), + ] + fetcher = FakeFetcher(results=[_page(jobs, total=2)]) + postings = WorkdayAdapter().list_postings( + "nvidia:wd5:careers", fetcher=fetcher, timeout_s=_TIMEOUT + ) + assert [p.external_id for p in postings] == ["JR100", "JR200"] + first = postings[0] + assert first.posting.title == "Engineer" + assert first.posting.apply_url == ( + "https://nvidia.wd5.myworkdayjobs.com/job/Santa-Clara/Senior-Engineer_JR100" + ) + assert first.posting.location == "Santa Clara, CA" + assert first.posting.posted_at == "Posted 3 Days Ago" + # The request is a POST with the CxS body + Accept header. + call = fetcher.calls[0] + assert call.method == "POST" + assert call.headers == {"Accept": "application/json"} + assert call.json_body == {"appliedFacets": {}, "limit": 20, "offset": 0, "searchText": ""} + assert call.url == "https://nvidia.wd5.myworkdayjobs.com/wday/cxs/nvidia/careers/jobs" + + +def test_list_postings_external_id_without_underscore() -> None: + # An externalPath with no "_JR" suffix falls back to the whole trailing segment. + fetcher = FakeFetcher(results=[_page([_job("/job/plain-path")], total=1)]) + postings = WorkdayAdapter().list_postings( + "nvidia:wd5:careers", fetcher=fetcher, timeout_s=_TIMEOUT + ) + assert postings[0].external_id == "/job/plain-path" + + +def test_list_postings_paginates() -> None: + page1 = _page([_job(f"/job/a_JR{i}") for i in range(20)], total=25) + page2 = _page([_job(f"/job/b_JR{i}") for i in range(20, 25)], total=25) + fetcher = FakeFetcher(results=[page1, page2]) + postings = WorkdayAdapter().list_postings( + "nvidia:wd5:careers", fetcher=fetcher, timeout_s=_TIMEOUT + ) + assert len(postings) == 25 + # Two pages fetched, at offsets 0 and 20. + assert [call.json_body["offset"] for call in fetcher.calls] == [0, 20] # type: ignore[index] + + +def test_list_postings_stops_on_empty_page() -> None: + # total claims more than a page, but the second page comes back empty → stop. + page1 = _page([_job(f"/job/a_JR{i}") for i in range(20)], total=99) + page2 = _page([], total=99) + fetcher = FakeFetcher(results=[page1, page2]) + postings = WorkdayAdapter().list_postings( + "nvidia:wd5:careers", fetcher=fetcher, timeout_s=_TIMEOUT + ) + assert len(postings) == 20 + assert len(fetcher.calls) == 2 + + +def test_list_postings_caps_and_warns(caplog: pytest.LogCaptureFixture) -> None: + # total far exceeds the cap; every page is full → capped at _MAX_PAGES with a warning. + full_page = _page([_job(f"/job/a_JR{i}") for i in range(20)], total=1000) + fetcher = FakeFetcher(results=[full_page] * 10) + with caplog.at_level(logging.WARNING, logger="atlas.discovery.ats.workday"): + postings = WorkdayAdapter().list_postings( + "nvidia:wd5:careers", fetcher=fetcher, timeout_s=_TIMEOUT + ) + assert len(postings) == 200 # _MAX_PAGES * _LIMIT + assert len(fetcher.calls) == 10 + assert any("capped at 200 of 1000" in record.message for record in caplog.records) + + +def test_list_postings_skips_malformed_jobs() -> None: + # A single-page board (total=1) whose page also carries malformed entries: the + # missing-title, non-dict, and missing-externalPath jobs are all skipped. + jobs: list[Any] = [ + _job("/job/ok_JR1"), + {"title": "no path"}, + "not-a-dict", + {"locationsText": "x"}, + ] + fetcher = FakeFetcher(results=[_page(jobs, total=1)]) + postings = WorkdayAdapter().list_postings( + "nvidia:wd5:careers", fetcher=fetcher, timeout_s=_TIMEOUT + ) + assert [p.external_id for p in postings] == ["JR1"] + + +def test_list_postings_malformed_board_ref_raises() -> None: + fetcher = FakeFetcher(results=[_page([], total=0)]) + with pytest.raises(DiscoveryError, match="Malformed Workday board reference"): + WorkdayAdapter().list_postings("bad-ref", fetcher=fetcher, timeout_s=_TIMEOUT) + + +def test_list_postings_non_json_raises() -> None: + body = FetchResult(url="u", status_code=200, content_type=None, body="") + fetcher = FakeFetcher(results=[body]) + with pytest.raises(DiscoveryError, match="non-JSON"): + WorkdayAdapter().list_postings("nvidia:wd5:careers", fetcher=fetcher, timeout_s=_TIMEOUT) + + +@pytest.mark.parametrize( + "body", + ['{"jobPostings": []}', '{"total": 5}', '{"total": "x", "jobPostings": []}', "[]"], +) +def test_list_postings_unexpected_shape_raises(body: str) -> None: + fetcher = FakeFetcher( + results=[FetchResult(url="u", status_code=200, content_type=None, body=body)] + ) + with pytest.raises(DiscoveryError, match="unexpected response"): + WorkdayAdapter().list_postings("nvidia:wd5:careers", fetcher=fetcher, timeout_s=_TIMEOUT) + + +def test_list_postings_propagates_fetch_error() -> None: + fetcher = FakeFetcher(raises=FetchError("boom")) + with pytest.raises(FetchError): + WorkdayAdapter().list_postings("nvidia:wd5:careers", fetcher=fetcher, timeout_s=_TIMEOUT) diff --git a/tests/test_scrape_fetcher.py b/tests/test_scrape_fetcher.py index 2eb8875..15b86c9 100644 --- a/tests/test_scrape_fetcher.py +++ b/tests/test_scrape_fetcher.py @@ -44,3 +44,44 @@ def test_fake_fetcher_satisfies_both_protocols() -> None: fetcher = FakeFetcher(FetchResult(url="u", status_code=200, content_type=None, body="")) assert isinstance(fetcher, Fetcher) assert isinstance(fetcher, BrowserFetcher) + + +def test_fake_fetcher_get_call_records_defaults() -> None: + result = FetchResult(url="https://x.test", status_code=200, content_type=None, body="ok") + fetcher = FakeFetcher(result) + fetcher("https://x.test", timeout_s=30) + call = fetcher.calls[0] + assert call.method == "GET" + assert call.json_body is None + assert call.headers is None + + +def test_fake_fetcher_records_post_method_body_and_headers() -> None: + result = FetchResult(url="https://x.test", status_code=200, content_type=None, body="{}") + fetcher = FakeFetcher(result) + body = {"limit": 20, "offset": 0} + got = fetcher( + "https://x.test", + timeout_s=30, + method="POST", + json_body=body, + headers={"Accept": "application/json"}, + ) + assert got is result + call = fetcher.calls[0] + assert call.method == "POST" + assert call.json_body == body + assert call.headers == {"Accept": "application/json"} + + +def test_fake_fetcher_replays_results_sequence_in_order() -> None: + pages = [ + FetchResult(url="u", status_code=200, content_type=None, body="page1"), + FetchResult(url="u", status_code=200, content_type=None, body="page2"), + ] + fetcher = FakeFetcher(results=pages) + assert fetcher("u", timeout_s=30).body == "page1" + assert fetcher("u", timeout_s=30).body == "page2" + # Exhausting the sequence fails loudly rather than silently repeating. + with pytest.raises(AssertionError, match="ran out of scripted results"): + fetcher("u", timeout_s=30)