Resolve the company behind a lead, and export a LinkedIn-ready audience - #8
Open
pallaoro wants to merge 10 commits into
Open
Resolve the company behind a lead, and export a LinkedIn-ready audience#8pallaoro wants to merge 10 commits into
pallaoro wants to merge 10 commits into
Conversation
…ence The LinkedIn Matched Audiences company upload asks for industry, city, state, zip and the company page URL. This app sources people, so it held none of them and shipped those five columns permanently blank; city and country were filled only when a sourcing agent happened to write a parseable free-text location, so a lead without one exported seven of ten columns empty. Adds a `companies` table keyed on the normalized domain — both the store and the cache, because unlike a person's work email a company record is the thing worth keeping, and enriching an account once serves every lead sourced at it afterwards. It deliberately does not expire on the 90-day contact rule: people change jobs in weeks, a company's HQ city and industry do not. Both exports LEFT JOIN it, so a company nobody has enriched still exports with blank columns rather than vanishing from the audience. The vendor's structured value wins over one parsed from free text, but never blanks a name we already had. The contact export takes one column from the join too — LinkedIn uses country as a matching hint, and the account's country is the right answer for a lead whose own location was empty. Provider adapters land next; this is the shape they write into.
… field Sibling of the person runner, deliberately separate rather than more EnrichFields. The two have different economics: a person waterfall buys one value per call, so a field is the right unit; a firmographic API returns industry, HQ address, ticker and headcount together for one credit. Six EnrichFields would make the abstraction itself cost five extra credits per company. What it does share is everything that is not the call shape: the store is injected exactly as EnrichCache is, so ordering and spend logic stay testable without a database; attempts land in the same ledger under a new `company` value, so "what did this run cost me?" keeps one answer; and the order lives in the existing waterfall_config table under its own row rather than a second table with its own route and UI. Two rules differ from the person side, both deliberate. There is no `verified` concept — no vendor grades an industry the way a finder grades deliverability — so the first hit wins outright and nothing keeps looking past it. And a 200 carrying an entirely empty record counts as a miss: storing it would poison the cache for six months with a row that fills no column and stop the vendor behind it from ever being asked. Companies are resolved from finishLead, the one point a lead is actually done, so a lead paused on a callback is enriched once when it lands rather than on every pass. Per lead rather than per distinct domain, which is safe because the batch enriches sequentially: the store write lands before the next lead starts, so the second lead at a company reads the row instead of buying it again. Registry is empty in this commit; adapters land next.
One card per vendor, not one per subject. People Data Labs and Apollo
resolve a person and an organization from the same key, so a vendor
already in the person registry gains "company" on its existing row rather
than a second row asking for the same secret — two cards for one key is
how a key gets pasted into one and not the other. A firmographic-only
vendor gets its own row.
The order goes through the same PUT /api/waterfall/{field} and the same
waterfall_config table under a "company" row, so there is no second
route, table or settings screen to keep in sync.
The response reports the company staleness window as its own number:
a user reading "90 days" next to a company row would be told something
untrue about when it gets re-bought.
… one Firmographic coverage is uneven by an order of magnitude, and the registry is about to get much wider than the three near-identical vendors it holds today. Findymail returns no ticker and no postal code; Surfe returns a ticker but no city; Snov returns neither a LinkedIn page nor a country. Under first-hit-wins, whichever of those a user happens to rank highest decides the whole record — and the store then trusts that half-empty row for six months, which is the exact failure the companies table was added to fix. So the runner merges instead: each answer fills the gaps the ones before it left, earliest-in-order winning per field. The earlier objection to merging was cost — that it multiplies every company by the number of vendors configured. That is answered by bounding the search rather than by dropping the merge. The run stops the moment COMPANY_ESSENTIAL is complete, so one full-coverage key still costs exactly one call, and a vendor whose entire documented coverage is already filled is skipped without being called. Essential is the four fields the audience upload and the UI read; a ticker is absent for most companies, so chasing one would spend a credit at every configured vendor on nearly every private company. That skip needs each adapter to declare what its vendor can return at all, so `covers` is required on CompanyProvider rather than optional: a vendor that overstates its coverage costs a credit for nothing, one that understates it is never called, and an optional field would default a new adapter into one of those two silently.
… do not The company registry held three of the nineteen vendors in the person registry. Every remaining one was checked against its own published contract, and the endpoint probed live: eleven have a firmographic API and now have an adapter; five do not, and that is a finding rather than a gap. No company endpoint exists at Skrapp or Anymail Finder (person-only APIs), at Zeliq (its own docs index lists exactly three endpoints: credit balance, enrich phone, enrich email), or at Dropcontact, whose company fields only ever come back bundled with a contact enrichment that requires a contact identifier. Kaspr keys every lookup on a person's profile URL. None of them is "planned" — there is nothing to wait for. Three of the eleven were mapped from a LIVE response rather than the docs, and one of the three is why that mattered: Findymail's published schema stops at name / domain / size / industry / linkedin_url, while the endpoint also returns city, region and country. An adapter written from the reference would have thrown away three of the LinkedIn upload's columns. The rest are docs-verified with the nesting written out, because the nesting is what a plausible guess gets wrong and a wrong guess is invisible: RocketReach puts the location under `address` and the page under `links.linkedin`, Forager returns `search_results` with the industry as an object and the postal code spelled `postcode`, LeadMagic calls the LinkedIn page `b2b_profile_url`, ContactOut keys its response by domain and hands back a bare vanity, and Tomba documents its own response two ways — both of which are read, and flagged, since there is no key to settle it. Each of those returns a row of undefineds if read the obvious way, which is a silently blank export. Datagma reuses the endpoint the phone lookup already calls, with the company flags on and the priced financial block off. Surfe and Snov answer out of band and are polled in-band by the shared helper, not paused. Snov is last in the default order because it is the only vendor here that charges per request rather than per match, so a miss costs a credit — and the adapter records that rather than reporting a free miss. The new mapping tests caught two adapters billing a credit for an empty 200: People Data Labs, which reads its record flat at the root and so has no envelope key whose absence means "no match", and RocketReach. Both now guard on the name.
…t serve it The README described two waterfalls and the app now runs three. Adds the vendor table, the per-vendor coverage each one actually fills, and the reason the company runner fills gaps where the other two stop at the first answer — with the bound that keeps that from costing a credit at every vendor. Names the five vendors with no company API explicitly rather than leaving them absent. An unexplained absence reads as an oversight and invites the next person to go looking for an endpoint that is not there.
live.test.ts proves the invocation contract by calling every adapter with a bogus key: the vendor rejects the key, so host, path, version and auth header are right. A rejected request never returns a record, so every line of response mapping stays unexercised — an adapter that reads the wrong field name passes that check, passes the stubbed fixtures (written from the same wrong reading), and returns an empty record in production. This runs the same adapters against the real vendor with the real key. The company half needs no configuration and carries no personal data: its subject is a company, and stripe.com is in every firmographic database worth an adapter. The person half takes its lead from LIVE_MAPPING_LEAD and skips without one, because no real person belongs hardcoded in an open-source repo that queries fourteen contact-data vendors with them. It asserts what a live payload can prove and a fixture cannot: that a hit mapped at least one field, that nothing came back outside the covers set the runner spends money on, and that headcounts, founding years and LinkedIn URLs are the shape the schema claims. Coverage gaps are printed, not failed — one company cannot tell a vendor's missing ticker from Stripe simply being private. Verified against the three keys on hand: 8/8 live hits, findymail 6/6 declared company fields, prospeo 8/8, wiza 9/10 (Stripe has no ticker).
The two are halves of one question and belong in one place: whether a vendor is reachable, and whether what it sends back is understood.
…nly contract-accurate The email table's 'Verified?' column was about the vendor's product — whether it grades an address as deliverable — but sat next to a paragraph saying response mapping is 'covered by tests', and the two read as one claim: that every adapter had been checked. Renamed the column to what it measures, and split the testing claim into its two honest halves. Every adapter has a fixture and a live endpoint check; only Findymail, Prospeo and Wiza have been answered by a real vendor with a real key, because a fixture transcribed from the docs by the same reading that wrote the adapter agrees with it by construction.
Both runners already refused to use a hit that carried no value — settle() guards on it, and the company runner called it 'a miss dressed as a hit'. But both wrote the ledger row before that check, so it was recorded as a hit with credits spent. That is the runtime signature of response-mapping drift, and it was indistinguishable from a vendor that worked, in the one place a user would ever look. Recording it as a miss would be worse than wrong: a miss means the vendor has no record, which is normal and costs nothing to be told. This means it had one, charged for it, and our field names no longer match theirs — our bug, not their gap. So it gets its own outcome rather than borrowing one that already means something else. No migration: the column is TEXT with no constraint. This matters more than it looks. Thirty-three adapters cannot be kept honest by whoever happens to hold keys; nobody holds all of them. Every install does hold some, so the ledger is what turns the installed base into the drift detector. Both cases are covered by tests that fail without the fix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The app enriched people and stopped there. A Matched Audiences upload wants the
company — industry, HQ, LinkedIn page, ticker — so every company column in the
export was going out empty. This adds a third waterfall that fills them.
What lands
providers/company.ts) — its own runner, its own cacheclock and its own ledger field, alongside the email and phone waterfalls.
companiestable stores firmographics per company rather than per lead, so alist of 200 people at 30 accounts costs 30 lookups and not 200.
Labs, Prospeo, LeadMagic, Datagma, Tomba, Forager, Findymail, ContactOut,
Surfe, Snov. One key per vendor still serves every waterfall it can.
verified value because one email is one email. A company record is ten
columns, and no vendor fills all ten every time, so this one keeps going for
the columns still open — bounded, so it cannot walk the whole registry
spending a credit per vendor on a record that is already nearly complete.
company (Match Audience) and contact upload formats.
The five that are absent on purpose
Skrapp, Anymail Finder, Zeliq, Dropcontact and Kaspr publish no company or
firmographic endpoint. Named explicitly in the README and
.dev.vars.example,because an unexplained absence reads as an oversight and sends the next person
looking for an endpoint that is not there.
Verification
npx tsc --noEmitclean;npx vitest run→ 252 passed, 45 skipped.LIVE_PROVIDER_CHECK=1→ 45/45. Every adapter, person and company, was calledagainst the real vendor with a bad key and got far enough to be rejected on
the key — which is what proves the host, path, API version and auth header
are right, and is the half a stubbed test cannot see.
Response mapping is stub-tested throughout; it is verified against real
responses only for the three vendors we hold keys for (Findymail, Wiza,
Prospeo).