feat(datasource intercom): contacts, companies and the promotion of the relations (lot 4) - #386
Conversation
Publishes the two collections phase 2 is walked through, and turns the denormalized contact columns of lot 1 into a relation now that their target exists. Contacts read through the one endpoint of the API that sorts, so the measured table gained a sortable flag and the cursor tier a server-side order; the match-all predicate Tickets already sent is what routes a list view asking for an order through the search. Companies read through the third pagination tier, by offset, which is the one place the window a list view asks for maps onto what Intercom takes. In exchange it is looked up rather than searched, and anything past the two published lookups is refused by name. Contact and company custom attributes are typed from GET /data_attributes and carry api_writable for the lot that writes, though every column here is read-only. Resolving a relation condition is now bounded on the cursor tier: the target is read one record past what a group may hold, rather than walking a whole collection to refuse the fan-out afterwards. contact_email and contact_ids give way to contact_id and the contact relation, one readable label plus navigation, which is the rule lot 2.5 set for the ticket labels. No saved view can rest on them: both were already refused server-side. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
What an operator needs before meeting any of it in production: the offset tier and why companies escape the cursor walker, the four lookups and the two of them that name a column, what a merge does to a row, which relation traversals are refused and which are a spec row the probe has yet to confirm. Corrects two statements lot 4 makes partly false: there is offset pagination, on companies alone, and one endpoint does sort. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A one-to-many is served by the agent listing the target on the key, so what the relation really rests on is the endpoint filtering a contact id -- measured on conversations, a spec row on tickets. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
13 new issues
|
![qltysh[bot]](https://avatars.githubusercontent.com/in/890766?s=60&v=4){kind=small}
| # collections that read them never send one, and the parameter is here for | ||
| # the one that does. `{ field:, ascending: }`, translated to Intercom's own | ||
| # spelling on the way out. | ||
| def search_page(path, query:, per_page:, starting_after: nil, list_key: 'data', params: {}, sort: nil) |
| # cursor. `POST /companies/list` is the only one, and it is what lets the | ||
| # Companies collection answer page 7 of a list view with one request | ||
| # instead of walking six pages to reach it. Intercom counts pages from 1. | ||
| def offset_page(path, page:, per_page:, params: {}, list_key: 'data') |
| # `params` is what narrows the endpoint rather than what pages it: | ||
| # `/data_attributes` answers the attributes of contacts and those of | ||
| # companies under `?model=`, and both are read whole. | ||
| def fetch_all(path, list_key: 'data', params: {}, boot: false) |
| body.is_a?(Hash) && !body[list_key].is_a?(Array) && !body['data'].is_a?(Array) && body.key?('id') | ||
| end | ||
|
|
||
| def collect_pages(path, list_key:, params:, boot:) |
| # tickets -- Intercom exposes no `GET /tickets` at all -- so the endpoint | ||
| # and its shape belong to the collection, while walking it does not. | ||
| def read_page(per_page:, cursor:, query: nil) | ||
| def read_page(per_page:, cursor:, query: nil, sort: nil) |
| client.search_page(search_endpoint.path, query: query, per_page: size, starting_after: cursor, | ||
| params: read_params, list_key: list_key) | ||
| params: read_params, list_key: list_key, sort: sort) | ||
| end |
| # Count only, and never a group: `total_count` is exact on every listing, | ||
| # while grouping the page in hand would answer a fraction as if it were | ||
| # the whole. | ||
| def aggregate(_caller, filter, aggregation, _limit = nil) |
| page += 1 | ||
| end | ||
|
|
||
| records |
| raise unless e.status == 404 | ||
|
|
||
| nil | ||
| end |
| add(union, name, column, definition) | ||
| end | ||
|
|
||
| def add(union, name, column, definition) |
![macroscopeapp[bot]](https://avatars.githubusercontent.com/in/900172?s=60&v=4){kind=small}
| records.concat(answer.records) | ||
| read += 1 | ||
| break if last_page?(answer, page) || (wanted && records.size >= wanted) | ||
| break if cap_reached?(read, records.size) |
There was a problem hiding this comment.
🟠 High collections/offset_collection.rb:148
Unpaginated reads return at most MAX_COLLECTED_PAGES pages, so collections larger than 1,500 records silently omit the remaining records; this affects unpaginated reads and relation resolution despite requesting all matches. collect_pages applies cap_reached? even when wanted is nil. Remove this cap so an unwindowed read continues until last_page?.
- break if cap_reached?(read, records.size)🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @packages/forest_admin_datasource_intercom/lib/forest_admin_datasource_intercom/collections/offset_collection.rb around line 148:
Unpaginated reads return at most `MAX_COLLECTED_PAGES` pages, so collections larger than 1,500 records silently omit the remaining records; this affects unpaginated reads and relation resolution despite requesting all matches. `collect_pages` applies `cap_reached?` even when `wanted` is `nil`. Remove this cap so an unwindowed read continues until `last_page?`.
| column = column_name_for(name) | ||
| return if column.empty? | ||
|
|
||
| add(union, name, column, definition) |
There was a problem hiding this comment.
🟠 High schema/data_attributes_introspector.rb:81
A custom attribute whose normalized name matches a native Contact or Company field (for example company_count, plan_name, user_count, name, id, or owner) is returned by collect and later added a second time, so add_field raises Field ... already defined in collection and prevents the entire datasource from booting. Filter attributes against each collection's already-declared fields before publishing them, or otherwise rename conflicting columns.
Also found in 2 other location(s)
packages/forest_admin_datasource_intercom/lib/forest_admin_datasource_intercom/datasource.rb:44
model_attributes('contact')passes arbitrary custom-attribute column names intoContactduring boot. A custom attribute such ascompany_count(or a name normalized to it) collides with a native contact column;Contact#define_schemathen callsadd_fieldtwice, and the toolkit raisesField company_count already defined in collection, preventing the entire datasource from booting rather than omitting the conflicting custom attribute.
packages/forest_admin_datasource_intercom/lib/forest_admin_datasource_intercom/datasource.rb:45
model_attributes('company')can return a custom attribute whose name is also a schema field, for exampleplan_nameoruser_count.Company#define_schemaadds its built-in field before iterating these attributes, so the duplicateadd_fieldraises and a workspace with that otherwise valid custom attribute cannot initialize the datasource.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @packages/forest_admin_datasource_intercom/lib/forest_admin_datasource_intercom/schema/data_attributes_introspector.rb around line 81:
A custom attribute whose normalized name matches a native `Contact` or `Company` field (for example `company_count`, `plan_name`, `user_count`, `name`, `id`, or `owner`) is returned by `collect` and later added a second time, so `add_field` raises `Field ... already defined in collection` and prevents the entire datasource from booting. Filter attributes against each collection's already-declared fields before publishing them, or otherwise rename conflicting columns.
Also found in 2 other location(s):
- packages/forest_admin_datasource_intercom/lib/forest_admin_datasource_intercom/datasource.rb:44 -- `model_attributes('contact')` passes arbitrary custom-attribute column names into `Contact` during boot. A custom attribute such as `company_count` (or a name normalized to it) collides with a native contact column; `Contact#define_schema` then calls `add_field` twice, and the toolkit raises `Field company_count already defined in collection`, preventing the entire datasource from booting rather than omitting the conflicting custom attribute.
- packages/forest_admin_datasource_intercom/lib/forest_admin_datasource_intercom/datasource.rb:45 -- `model_attributes('company')` can return a custom attribute whose name is also a schema field, for example `plan_name` or `user_count`. `Company#define_schema` adds its built-in field before iterating these attributes, so the duplicate `add_field` raises and a workspace with that otherwise valid custom attribute cannot initialize the datasource.
| operators: ['=', '!='] | ||
| source: spec | ||
| owner_id: | ||
| field: owner_id |
There was a problem hiding this comment.
🟠 High query/search_fields.yml:456
Contact owner_id filters are rejected because this schema serializes the value as a JSON string, while Intercom's Contacts Search API requires an integer. Change the field type to number so direct and relation-derived owner_id/owner:name filters use the accepted type.
| field: owner_id | |
| type: number |
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @packages/forest_admin_datasource_intercom/lib/forest_admin_datasource_intercom/query/search_fields.yml around line 456:
Contact `owner_id` filters are rejected because this schema serializes the value as a JSON string, while Intercom's Contacts Search API requires an integer. Change the field type to `number` so direct and relation-derived `owner_id`/`owner:name` filters use the accepted type.
| documents that jumping to page N is unsupported, so reaching page 20 costs 20 sequential requests. | ||
| The walk is capped at 50 pages / 7 500 records and every truncation is logged, naming the window | ||
| it stopped in. `POST /companies/list` is the exception and takes a page number, which is why | ||
| companies escape the walker and its caps entirely. |
There was a problem hiding this comment.
🟢 Low forest_admin_datasource_intercom/README.md:206
Company reads are still truncated at MAX_COLLECTED_PAGES (10), so callers following this README can receive an incomplete collection despite the claim that companies escape the caps. Update the documentation to describe the actual limit.
| companies escape the walker and its caps entirely. | |
| `POST /companies/list` is the exception and takes a page number, but company reads remain subject to `MAX_COLLECTED_PAGES` (10). |
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @packages/forest_admin_datasource_intercom/README.md around line 206:
Company reads are still truncated at `MAX_COLLECTED_PAGES` (10), so callers following this README can receive an incomplete collection despite the claim that companies escape the caps. Update the documentation to describe the actual limit.
| # for, and `/contacts/search` does not sort on an id anyway. | ||
| def server_sort(filter) | ||
| clauses = Array(filter&.sort) | ||
| return nil if clauses.empty? || default_pk_sort?(clauses) |
There was a problem hiding this comment.
🟡 Medium collections/cursor_collection.rb:391
id IN [...] requests return records in input-ID order, but server_sort marks sortable fields as honoured and emits no warning. Thus id IN [id_b, id_a] sorted by email silently returns [id_b, id_a] without applying the requested order. Treat ID lookups as unsortable and warn instead.
return nil if clauses.empty? || default_pk_sort?(clauses)
+ if id_lookup(filter)
+ warn_ignored_sort(clauses)
+ return nil
+ end🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @packages/forest_admin_datasource_intercom/lib/forest_admin_datasource_intercom/collections/cursor_collection.rb around line 391:
`id IN [...]` requests return records in input-ID order, but `server_sort` marks sortable fields as honoured and emits no warning. Thus `id IN [id_b, id_a]` sorted by `email` silently returns `[id_b, id_a]` without applying the requested order. Treat ID lookups as unsortable and warn instead.
Lot 1 paid for the parts of every ticket and published five columns derived from them, never the exchange itself. It costs no request: Intercom returns the parts in the search response and offers no way to ask it not to, so the page pays for them whatever the projection says. The builder moves out of Conversation into a shared module, with two hooks for what the two resources do differently: where their parts live, and whether anything opens the thread before them. A ticket has no source object, so its thread starts on its first part. An empty list means an empty thread here, where a conversation read from a listing carries no parts at all and keeps a nil that reads as unknown. Ticket reads now carry display_as=plaintext, which lot 1 sent on conversations alone: the bodies are HTML written by end customers, and there was no body on a ticket row until now. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
Added after review of the read surface: the thread of a ticket is now published, as the same Lot 1 paid for the parts of every ticket — they are what caps the page at 25, Intercom offering no field selection — and published five columns derived from them, never the exchange itself. So this costs no request: the parts are in the response whatever the projection says, and building the thread out of them is what the projection guards.
Worth knowing before scoping a role: the internal notes of the team are in the thread, next to what the customer was told. That is what a thread is on Intercom, and publishing half of it would be the more surprising answer — it is now stated in the README for both collections.
|
| .merge('id' => stringify_id(attrs['id']), 'redacted' => attrs['redacted']) | ||
| end | ||
|
|
||
| def entry(part_type:, created_at:, author:, body:, attachments:) |
| endpoint does not read. Filter on `last_reply_at` or on the state | ||
| instead. |
There was a problem hiding this comment.
🟡 Medium query/search_fields.yml:367
The timeline refusal directs operators to last_reply_at and state, but both are also refused for tickets, so following either recommendation produces another unsupported-filter error. Replace those references with an accurate statement that the search endpoint has no searchable equivalent.
| endpoint does not read. Filter on `last_reply_at` or on the state | |
| instead. | |
| endpoint does not expose a searchable equivalent for this value. |
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @packages/forest_admin_datasource_intercom/lib/forest_admin_datasource_intercom/query/search_fields.yml around lines 367-368:
The `timeline` refusal directs operators to `last_reply_at` and `state`, but both are also refused for tickets, so following either recommendation produces another unsupported-filter error. Replace those references with an accurate statement that the search endpoint has no searchable equivalent.
| # What comes before the parts. A conversation opens on its `source`, which | ||
| # is not a part at all; a ticket opens on its first part like any other | ||
| # event, so there is nothing to prepend. | ||
| def opening_entry(_attrs) = nil |
There was a problem hiding this comment.
🟠 High collections/timeline.rb:30
Ticket projections that include timeline raise NoMethodError instead of returning ticket rows because build_timeline calls parts_of(attrs), but Ticket does not provide that hook. Define a default empty parts_of implementation in the shared module so conversation-specific overrides remain supported.
def opening_entry(_attrs) = nil
+ def parts_of(_attrs) = []🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @packages/forest_admin_datasource_intercom/lib/forest_admin_datasource_intercom/collections/timeline.rb around line 30:
Ticket projections that include `timeline` raise `NoMethodError` instead of returning ticket rows because `build_timeline` calls `parts_of(attrs)`, but `Ticket` does not provide that hook. Define a default empty `parts_of` implementation in the shared module so conversation-specific overrides remain supported.
Measured on a real workspace: a custom contact attribute named `id`. The toolkit refuses a field declared twice, so the agent did not boot -- and had it accepted the column, the serializer would have written the attribute over the record's own key. Tickets already guarded against this since lot 1. The guard moves to a shared module the three collections use, and Contacts and Companies declare their relations before registering these columns, so a name landing on a relation is skipped the same way. Same module fixes a second thing on the two new collections: a custom date arrives as epoch seconds like every other Intercom date, and a Date column handed an integer renders as one. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
| # read from a listing carries no parts at all and its timeline stays nil, | ||
| # which reads as unknown. | ||
| def embed_timeline(records, rows, projection) | ||
| return unless projection.include?('timeline') |
There was a problem hiding this comment.
🟠 High collections/ticket.rb:155
A full-record Ticket#list(..., nil) returns timeline as nil even when ticket parts are present, so the declared timeline is silently omitted. BaseCollection#project represents an empty projection as all columns, but embed_timeline only checks for an explicit timeline; treat an empty projection as requesting it too.
🚀 Reply "fix it for me" or copy this AI Prompt for your agent:
In file @packages/forest_admin_datasource_intercom/lib/forest_admin_datasource_intercom/collections/ticket.rb around line 155:
A full-record `Ticket#list(..., nil)` returns `timeline` as `nil` even when ticket parts are present, so the declared timeline is silently omitted. `BaseCollection#project` represents an empty projection as all columns, but `embed_timeline` only checks for an explicit `timeline`; treat an empty projection as requesting it too.
1 participant
Lot 4 of PRD-1120, onto the integration branch. Read-only: every write on contacts and companies is lot 4b (PRD-1152).
Contacts and Companies are published, and the denormalized contact columns of lot 1 become a relation now that their target exists — which is what turns a ticket into a customer, their account, and everything they opened before.
Contacts — the one collection Intercom sorts
Cursor-paginated like conversations and tickets, with two routes of its own:
id IN [...], which this endpoint answers and no other does — a hundred at a time, instead of one request per record. It is what makes a related list of contacts affordable;company_id equals XreadsGET /companies/{id}/contacts./contacts/searchfilters no company field, so without that route the one relation an ops team walks the most would have been a refusal. Bare equality only: anandalso carrying a scope names a narrower set than the account does.POST /contacts/searchis the only endpoint of the API that takes asortand applies it, so this is the only collection with sortable columns. That needed asortableflag on the measured table, an order in the client, and — for a list view that asks for an order and carries no condition — the match-all predicate Tickets already send, the listing endpoint sorting nothing. A sort on any other column, or on two at once, is reported in the log: Intercom takes a single{ field, order }, and honouring the first of two clauses would order the page by something nobody asked for.The measured date restrictions (
>=,<=,!=andINrefused where the other two endpoints take them) land in the table. A Date column publishes the two bounds alone everywhere in this datasource anyway, so what the criterion really guarantees is that nothing wider reaches the wire — asserted, not assumed.A merged contact disappears from the listing and from the search: the row reads as gone rather than as an error.
Companies — a third pagination tier
POST /companies/listpaginates by offset, which is what a list view asks for: page 7 is one request, no cursor walked, no cap, no truncation warning. It is the one place R1 does not apply. A window straddling two pages is served exactly, by reading the page it lands in and the next.In exchange there is no search endpoint at all. Two lookups are published —
nameandcompany_id, the identifier the customer's own system gave the account — and everything past them is refused by name. Filtering by tag or by segment belongs with lot 5, which adds those collections.GET /companies/scrollis deliberately rejected: one open scroll per app, expiring in a minute, cannot serve concurrent list views.The relations, and the two decisions the ticket left open
Many-to-one plus one-to-many, not a join collection. The precedent is already shipped: lot 2.5's
state/previous_stateare navigable and their traversal refused by name, and PRD-1120 sanctions that outcome explicitly. A join collection would have been the honest cardinality of a group conversation at the price of three collections of plumbing in the interface.IntercomConversation.contact,IntercomTicket.contactIntercomContact.ownerIntercomContact.companyIntercomContact.conversations/.tickets,IntercomCompany.contactsThe denormalized columns give way, by the rule lot 2.5 set for the ticket labels — one readable label plus the relation, not two ways to read one fact.
contact_emailandcontact_idsare gone,contact_id(a foreign key rather than a Json blob no filter could reach) andcontact_count/contact_namestay. Nothing can rest on the two that go: both were already refused server-side, and the "ops in production" milestone has not been reached.What actually dimensioned the lot, and how it is answered
Relations#matching_idslisted the target with no bound, which is exact and cheap against a tier-C collection and ruinous against Contacts: a full cursor walk, then anorof thousands of ids, to refuse the fan-out afterwards.The read is now bounded to one record past what a group may hold. Sixteen is all it takes to know fifteen will not fit, so the refusal says "more than 15" rather than a count it deliberately did not go and measure. Tier C keeps its unbounded read — every record is already in hand, and cutting it short would drop ids its in-memory
incarries for free.This also answers, on the cursor side, the thread left open on #385 about relation conditions resolved against a truncated target.
Everything else
GET /data_attributes?model=contact|companytypes the custom attributes at boot, on the boot connection, degrading to no attribute column rather than to a failed boot.api_writableis read and carried on the introspected attribute — not on the column, everything here being read-only — so lot 4b does not re-introspect. The attributes ship display-only: they are filtered by name rather than by a per-type id, so R7 does not apply, but which operators Intercom answers on each data type has not been measured and this package publishes no filter it has not seen work.README: the third tier, the four lookups and the two that name a column, what a merge does to a row, which traversals are refused. It also corrects two statements this lot makes partly false — there is offset pagination, on companies alone, and one endpoint does sort.
Test plan
527 examples, 0 failures, 100% line coverage (1918/1918),rubocopclean at the repository root.Left to the probe, in the table rather than in an assumption
/tickets/searchfilters oncontact_ids— if it does not, the row moves to the refusals andIntercomTicket.contactstays navigable without being filterable;/contacts/searchanswers on a custom attribute.Unrelated and still open from #385:
Utils::Collection.aggregate_relationcounts through rows whose target resolved to nil, so/count-relatedover-reports wherelist_relationalready compacts. It lives in the toolkit and is mergeable onmainindependently.🤖 Generated with Claude Code
Note
Add IntercomContact and IntercomCompany collections and promote relations
IntercomContactandIntercomCompanycollections with schemas, relations, and serializers in contact.rb and company.rbOffsetCollectionbase class in offset_collection.rb for offset-paginated endpoints, whileCursorCollectiongains sorting and exact count supportCustomAttributesandTimelinemodules in custom_attributes.rb and timeline.rb, replacing local logic inTicketandConversationoffset_page,lookup_page, and sort clause translation to the API client.rbContactIdentityschema in contact_identity.rb replaces thecontact_idsJSON column with a scalarcontact_idand removescontact_email; cursor collections now forward supported sorts instead of ignoring them; contact bulk id reads truncate atMAX_IDS_READ(300)Macroscope summarized 6880e2c.