feat(datasource intercom): relations of the reference tier (lot 2.5)

packages/forest_admin_datasource_intercom/README.md

@@ -60,7 +60,15 @@ beats not running.
 
 A read-only token is enough, and is what to recommend for this lot. A permission the token lacks
 costs **columns or a collection, never the boot of the agent**: the ticket-type introspection
-degrades to no attribute column, and a collection whose endpoint answers 403 fails its own page.
+degrades to no attribute column, a collection whose endpoint answers 403 fails its own page, and a
+token that cannot read `/admins` or `/teams` leaves the `admin_names` / `team_names` column empty
+rather than failing the page it is on.
+
+A **relation is the exception**, and it is worth knowing before scoping a token: resolving one reads
+the target endpoint, and that read is not guarded the way the names above are. A token denied
+`/admins` fails any page projecting `admin_assignee:name`, and fails the related list behind
+`IntercomTeam#admins` — the failure lands on the collection being read, not on the one that was
+denied. Scope the token to the endpoints in the table below, or to none of them.
 
 ## Collections
 
@@ -70,12 +78,13 @@ degrades to no attribute column, and a collection whose endpoint answers 403 fai
 | `IntercomTicket` | `POST /tickets/search`, `GET /tickets/{id}` | cursor | yes, exactly |
 | `IntercomAdmin` | `GET /admins` | read whole | yes, exactly |
 | `IntercomTeam` | `GET /teams` | read whole | yes, exactly |
+| `IntercomTeamMembership` | `GET /teams` | read whole | yes, exactly |
 | `IntercomTicketType` | `GET /ticket_types` | read whole | yes, exactly |
 | `IntercomTicketState` | `GET /ticket_states` | read whole | yes, exactly |
 
 Two tiers, and they behave differently on purpose.
 
-**Read whole** — admins, teams, ticket types, ticket states. Their endpoints answer in one response,
+**Read whole** — admins, teams, team memberships, ticket types, ticket states. Their endpoints answer in one response,
 so filtering, sorting, paging and counting them in memory is *exact*: the records in hand are every
 record Intercom holds. These are the only collections that can be filtered, sorted and grouped in
 this lot, and the only ones a chart may group by. The cost is bandwidth, not correctness.
@@ -86,6 +95,66 @@ nothing is filtered or sorted in memory. Three routes and no fourth: no conditio
 Intercom's search DSL and walked through the search endpoint. What the translation cannot express is
 **refused by name** — see [Filtering](#filtering).
 
+## Relations
+
+Intercom joins nothing: a ticket carries an assignee id, and the teammate behind it is a second read
+of a second endpoint. What makes eight relations affordable is that every collection on the far end
+is read whole in one request — so a relation resolves for a **whole page at the price of one read**,
+never one read per row. The price is per target *collection*, not per relation: a ticket's `state`
+and `previous_state` are one read of `/ticket_states`, over the ids both of them name.
+
+*Exactly*, with one bound worth naming: "read whole" is what the endpoint answers, and `fetch_all`
+stops after [`MAX_COLLECTED_PAGES`](lib/forest_admin_datasource_intercom/client.rb) pages if Intercom
+paginates one of these on its own — it logs when it does. A workspace whose `/admins` or `/teams`
+runs past that cap resolves the relations pointing at the records it dropped as empty. The figure is
+sized for reference collections, which is what every target here is.
+
+| Collection | Relation | Target | Filterable through |
+| --- | --- | --- | --- |
+| `IntercomConversation` | `admin_assignee`, `closed_by` | `IntercomAdmin` | yes |
+| `IntercomConversation` | `team_assignee` | `IntercomTeam` | yes |
+| `IntercomTicket` | `admin_assignee` | `IntercomAdmin` | yes |
+| `IntercomTicket` | `team_assignee` | `IntercomTeam` | yes |
+| `IntercomTicket` | `ticket_type` | `IntercomTicketType` | yes |
+| `IntercomTicket` | `state`, `previous_state` | `IntercomTicketState` | **no** — read and navigate only |
+| `IntercomTeam` | `admins` | `IntercomAdmin` | no (many-to-many) |
+| `IntercomAdmin` | `teams` | `IntercomTeam` | no (many-to-many) |
+| `IntercomTeamMembership` | `team`, `admin` | `IntercomTeam`, `IntercomAdmin` | yes |
+
+Every one of them is **read-only**: this lot writes nothing, and Intercom exposes no endpoint that
+writes a team membership at all.
+
+**`IntercomTeamMembership` exists because Intercom's does not.** The workspace carries the
+membership on the team (`admin_ids`) and on the teammate (`team_ids`) both and exposes no resource
+for the pair, while a many-to-many needs a collection to travel through. It is synthesized from
+`GET /teams`, one record per pair, keyed `teamId:adminId`. Without it, both sides read as an array of
+ids nobody can click.
+
+Two consequences of travelling through it are worth knowing. A **related list of teammates is
+ordered by the membership, not by the teammate**: the agent hands the through collection the columns
+of the collection the relation reaches, so an order on `name` or `email` cannot be resolved there and
+is logged rather than silently dropped. And a `admin_ids` entry naming a teammate `/admins` does not
+answer -- one who left, one outside the token's reach -- **drops out of the related list** instead of
+appearing as an empty row.
+
+Alongside it, a team names its teammates (`admin_names`) and a teammate its teams (`team_names`) on
+the row itself, so a list view reads without a join. **Those replace the arrays of ids** the first
+lots published: one readable form plus a relation to navigate, rather than two ways to read one fact.
+They are read only when a projection asks for them, and a token that cannot read the other side
+costs the column and nothing else — never the page, and never the relation.
+
+The same rule settled the ticket labels: `state_label` and `ticket_type_name` stay on the row,
+`state_category` and `state_external_label` are gone — they are a hop away, on the `state` relation,
+and neither was ever filterable, so no segment, scope or saved filter could rest on them.
+
+A relation reads its target **undecorated**, so a permission scope or a segment defined on the target
+does not narrow what a relation resolves — the same way a native datasource joins a table without
+applying the scopes of the collection mapped to it.
+
+One semantic worth stating plainly: **a row whose foreign key is null matches no relation filter**,
+the way a join drops it, negated filters included. A ticket with no assignee is not "assigned to
+someone other than Marie".
+
 ## What the API cannot do, and what this does about it
 
 Where Forest asks for something Intercom has no equivalent for, this datasource **refuses with a
@@ -200,6 +269,9 @@ refuses a search by name.
   `last_responder_name`, `last_responder_type`. They exist nowhere in Intercom; `/tickets/search`
   filters none of them and ignores a sort on them without a word;
 - **the account of a ticket** — `company_id`, refused by the endpoint itself with `invalid_field`;
+- **the state of a ticket** — the measured table carries no filter on a state id, so `state_id`,
+  `previous_state_id` and the `state` relation are read and navigated rather than filtered. Whether
+  the endpoint filters one at all is one of the probe's open questions;
 - **the ticket attributes** — filtered as `ticket_attribute.{id}`, and the same attribute carries a
   different id per ticket type, so a union column has no single id to translate to. See
   [Tickets](#tickets);
@@ -212,6 +284,39 @@ refuses a search by name.
 - **group-by**, on either cursor collection: there is no aggregate endpoint, and grouping over the
   pages a walk collected would look exact while answering a fraction.
 
+### Through a relation
+
+A relation is published filterable as soon as *any* column of its target is — the agent decides that,
+not this datasource — so the interface offers `admin_assignee:name` the moment the relation exists.
+What Intercom is really filtered on is the foreign key: the **target says which of its records
+match**, over every record it holds rather than over a page, and the ids it names become the
+condition the search carries.
+
+That is exact, and it has three visible edges:
+
+- Intercom takes no membership operator on these fields, so several matches become **one equality per
+  match**, inside an `OR` — which counts against the fifteen conditions a group allows. A relation
+  condition matching more records than that is refused by name rather than sent and answered with a
+  400 naming neither the limit nor the filter that hit it. That `OR` is **inlined into a parent that
+  aggregates the same way**, so it costs no level of nesting where it does not have to: the two
+  levels Intercom allows are spent on the filter that was written, not on the expansion of a
+  relation. Where inlining it would take the parent past fifteen conditions it stays nested, width
+  being the scarcer of the two.
+- A condition the target matched **no record** with names no row, and the DSL cannot say so: the
+  search is skipped entirely rather than sent as a filter that would come back with everything.
+- A relation whose foreign key the endpoint does not filter — the ticket `state` — is refused with a
+  message saying which of the two it is: the relation is there to be read and navigated. Whether
+  `/tickets/search` filters a state id at all is one of the probe's open questions; the answer lands
+  in the table, not in an assumption.
+
+On the collections read whole the same condition costs nothing: they filter in memory, so the ids go
+in as a plain membership and none of the DSL's limits apply.
+
+A **many-to-many is published unfilterable** — `admins` and `teams` — and a condition written on one
+anyway, in a scope or a segment, is refused before it reaches this datasource: the agent's own
+validator answers a 400 naming the field and its type. Filter on a column of the collection next
+door instead.
+
 ### The limits of a search, checked before the request leaves
 
 Intercom nests a search **two levels** deep and takes **fifteen conditions per group**. Past either
@@ -327,7 +432,7 @@ Everything else is read when a collection is listed, so an agent boots whatever
 | Lot | What it brings |
 | --- | --- |
 | 3 | Writes and business actions: reply, close, snooze, reopen, assign, tag, convert |
-| 4 | Contacts and companies, and the relations promoted from today's denormalized columns |
+| 4 | Contacts and companies, and the relations towards them promoted from today's denormalized columns |
 | 5 | Notes, tags, segments |
 | 6 | Bounded group-by and the reporting export |
 

packages/forest_admin_datasource_intercom/lib/forest_admin_datasource_intercom/collections/admin.rb

@@ -25,11 +25,37 @@ def serialize(admin)
           'away_mode_enabled' => attrs['away_mode_enabled'],
           'away_mode_reassign' => attrs['away_mode_reassign'],
           'has_inbox_seat' => attrs['has_inbox_seat'],
-          'team_ids' => Array(attrs['team_ids']).map { |id| stringify_id(id) } }
+          # Not a column: it is what the names below are read from, and what the
+          # membership collection turns into the relation.
+          'team_ids' => Array(attrs['team_ids']).map { |id| stringify_id(id) },
+          'team_names' => nil }
+      end
+
+      # The teams the teammate belongs to, by name, and only when a projection
+      # asked for them: one read of `/teams` for the whole page. A token that
+      # cannot read them costs the column and nothing else.
+      def enrich(records, rows, projection)
+        return unless column_asked?(projection, 'team_names')
+
+        names = team_names
+        records.each_with_index do |record, index|
+          rows[index]['team_names'] = Array(record['team_ids']).filter_map { |id| names[id] }
+        end
       end
 
       private
 
+      def team_names
+        client.fetch_all('teams', list_key: 'teams')
+              .to_h { |team| [stringify_id(team['id']), team['name']] }
+      rescue APIError => e
+        ForestAdminDatasourceIntercom.logger.warn(
+          "[forest_admin_datasource_intercom] #{name} could not read the teams of the workspace (HTTP " \
+          "#{e.status || "-"}); the names are left empty. The relation to IntercomTeam is unaffected."
+        )
+        {}
+      end
+
       def define_schema
         add_column('id', 'String', is_primary_key: true)
         add_column('name', 'String')
@@ -41,11 +67,14 @@ def define_schema
         add_column('away_mode_enabled', 'Boolean')
         add_column('away_mode_reassign', 'Boolean')
         add_column('has_inbox_seat', 'Boolean')
-        # A list, so neither filterable nor sortable. It stays a plain column
-        # rather than a relation: Intercom carries the membership on the admin
-        # and on the team both, so declaring it twice would give the schema two
-        # sides of a many-to-many with no join collection to hold it.
-        add_column('team_ids', 'Json')
+        # A list, so neither filterable nor sortable -- as the array of ids it
+        # replaces was. It reads the teammate without a join; the relation below
+        # is what navigates it, through the membership collection that gives the
+        # many-to-many the join Intercom does not expose.
+        add_column('team_names', 'Json')
+        add_many_to_many('teams', foreign_collection: 'IntercomTeam',
+                                  through_collection: 'IntercomTeamMembership',
+                                  origin_key: 'admin_id', foreign_key: 'team_id')
       end
     end
   end

packages/forest_admin_datasource_intercom/lib/forest_admin_datasource_intercom/collections/base_collection.rb

@@ -4,11 +4,15 @@ module Collections
     # record is narrowed to the projection asked for, and how a window is cut
     # out of records already in hand.
     #
-    # Read-only for now. The writes and the business actions arrive with lot 3,
-    # and the relations with lot 4, once Contacts and Companies exist -- a
-    # relation whose target collection is missing is a schema the agent refuses
-    # to boot on.
+    # Read-only for now: the writes and the business actions arrive with lot 3.
+    # The relations towards Contacts and Companies wait for lot 4, those two
+    # collections not existing yet -- a relation whose target collection is
+    # missing is a schema the agent refuses to boot on. The relations between the
+    # collections this datasource already serves are declared and answered here,
+    # through `Relations`.
     class BaseCollection < ForestAdminDatasourceToolkit::Collection
+      include Relations
+
       ColumnSchema = ForestAdminDatasourceToolkit::Schema::ColumnSchema
       Operators = ForestAdminDatasourceToolkit::Components::Query::ConditionTree::Operators
       Equivalent = ForestAdminDatasourceToolkit::Components::Query::ConditionTree::ConditionTreeEquivalent
@@ -27,14 +31,34 @@ def client
 
       def define_schema = raise(NotImplementedError, "#{self.class} did not implement define_schema")
 
-      # A record narrowed to what was asked for. A projection naming a field the
-      # record does not carry yields nil rather than nothing at all: the agent
-      # asked for a column, and an absent key would read as a record missing it.
+      # A record narrowed to the columns that were asked for. A projection naming
+      # a field the record does not carry yields nil rather than nothing at all:
+      # the agent asked for a column, and an absent key would read as a record
+      # missing it.
+      #
+      # A path through a relation is not a column and is skipped here: it is
+      # answered by `embed_relations`, which nests a whole row under the relation
+      # name once the page is in hand.
+      #
+      # No projection at all asks for every column -- which is not the same as
+      # every key a serialized record happens to carry: a couple of them hold
+      # what a column is read *from*, the ids behind a membership for one, and
+      # publishing those would show the operator the plumbing.
       def project(record, projection)
-        fields = Array(projection)
-        return record if fields.empty?
+        asked = Array(projection).map(&:to_s)
+        return record.slice(*column_names) if asked.empty?
+
+        asked.reject { |field| field.include?(':') }.to_h { |field| [field, record[field]] }
+      end
+
+      # Whether a projection asks for a column. No projection at all asks for
+      # every declared column, which is how `project` reads it -- an enrichment
+      # guarded on the column being named would leave nil the very column the
+      # projection publishes.
+      def column_asked?(projection, column)
+        asked = Array(projection).map(&:to_s)
 
-        fields.to_h { |field| [field, record[field]] }
+        asked.empty? || asked.include?(column)
       end
 
       # The window a list view asked for, cut out of records already in hand.
@@ -54,6 +78,10 @@ def page_window(records, filter)
         records[offset, limit] || []
       end
 
+      def column_names
+        @column_names ||= fields.select { |_, field| field.is_a?(ColumnSchema) }.keys
+      end
+
       # The timezone in-memory date comparisons are evaluated in. The caller's,
       # since that is whose "today" the filter was written against.
       def timezone_for(caller)

packages/forest_admin_datasource_intercom/lib/forest_admin_datasource_intercom/collections/conversation.rb

@@ -10,7 +10,7 @@ module Collections
     # customers, and rendering third-party HTML inside Forest is neither safe nor
     # useful (R10).
     # Long by line count only: most of it declares the columns, one call each.
-    class Conversation < CursorCollection
+    class Conversation < CursorCollection # rubocop:disable Metrics/ClassLength
       include ContactIdentity
       include Conversation::Serializer
       include Conversation::Timeline
@@ -69,18 +69,38 @@ def define_schema
         add_column('updated_at', 'Date')
         add_column('waiting_since', 'Date')
         add_column('snoozed_until', 'Date')
-        add_column('admin_assignee_id', 'String')
-        add_column('team_assignee_id', 'String')
-        # The conversation carries its company as a whole object, so the account
-        # name is free here -- unlike on a ticket, which carries the id alone.
-        add_column('company_id', 'String')
-        add_column('company_name', 'String')
+        define_assignment_columns
         define_contact_columns
         define_source_columns
         define_statistics_columns
         add_column('tag_names', 'Json')
         add_column('ai_agent_participated', 'Boolean')
         add_column('timeline', 'Json')
+        define_relations
+      end
+
+      # Who the conversation sits with, and who closed it. All three targets are
+      # read whole in one request, and `/conversations/search` takes a filter on
+      # each of the three keys -- so these relations can be read, navigated and
+      # filtered through alike.
+      #
+      # No relation towards the company: a conversation carries its account as a
+      # whole object, so the name is already on the row, and the Companies
+      # collection arrives with lot 4.
+      def define_relations
+        add_many_to_one('admin_assignee', foreign_collection: 'IntercomAdmin', foreign_key: 'admin_assignee_id')
+        add_many_to_one('team_assignee', foreign_collection: 'IntercomTeam', foreign_key: 'team_assignee_id')
+        add_many_to_one('closed_by', foreign_collection: 'IntercomAdmin', foreign_key: 'closed_by_id')
+      end
+
+      # Who the conversation sits with, and which account it belongs to. The
+      # conversation carries its company as a whole object, so the account name
+      # is free here -- unlike on a ticket, which carries the id alone.
+      def define_assignment_columns
+        add_column('admin_assignee_id', 'String')
+        add_column('team_assignee_id', 'String')
+        add_column('company_id', 'String')
+        add_column('company_name', 'String')
       end
 
       # The contact identity is denormalized onto the row rather than declared as