diff --git a/AGENTS.md b/AGENTS.md index 1175d9b9..a6428b08 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -25,14 +25,14 @@ - `packages/utils` - `@proxy/utils`, the Result helpers shared by every package: `Do`, `requirePresent` and `parseSchema` (`src/parse.ts`), and `httpRequest`, the fetch wrapper that returns a `FetchError` union instead of throwing (`src/fetch.ts`). Also `ApiError`/`ApiErr` and `wrapDb` (`src/api-error.ts`), the error every server-side function returns, and the session cookie's name and parsing (`src/session-cookie.ts`) - `packages/integrations` - `@proxy/integrations`, the catalog the api and the web app share: each integration's actions, collections and fields (`src/catalog.ts`), the agents a login can be made for (`src/agent-providers.ts`), the email providers' servers (`src/email-providers.ts`), and access (`src/access.ts`): per action of a connection, an agent's own setting, else the connection's default, an action counting only with the one it `requires`, and tools (`src/tools.ts`): what an agent can do with a connection, generated from the catalog, one per list, open, write and command of each collection (`emails_list`, `emails_archive`), each with the action it needs and a JSON Schema of its params. A custom MCP server (`mcp` in the catalog, `custom`) has no actions of its own: `integrationFor(connection)` (`src/mcp-tools.ts`) builds its integration from the tools the connection keeps (`McpTool`, as the server lists them), one action per tool named by the tool, its risk from the tool's annotations (low when it only reads, medium when what it changes can be undone, high when it can't, and `unknown`, shown without a label, when it says nothing), and `integrationTools` passes them on as `call` tools with the server's own input schema (`ServerTool`, beside the catalog's `CollectionTool`). The web app adds the icons (`src/lib/integrations.ts`, whose `integrationOf(connection)` is what every page uses) - `packages/db` - `@proxy/db`: Prisma 6 on Postgres. `src/index.ts` is the client; each module (`auth.ts`, `user.ts`, `organization.ts`, `connection.ts`, `info.ts`, `agent.ts`, `audit.ts`) exports functions returning `Result` through `wrapDb`. A connection's `tools` column holds a custom MCP server's tools as JSON, and `name` what the person called it; both null for every other integration. `relationMode = "prisma"`: no foreign keys in the database, so Prisma runs the cascades and every relation column has its own index. Locally the database is Prisma's embedded Postgres (PGlite, `scripts/dev-db.ts`), which needs `connection_limit=1&pgbouncer=true` in `DATABASE_URL` -- `packages/api` - `@proxy/api`: Express on Bun. Routes are `handleXRoute` functions mounted in `src/server/create-api-router.ts`; JSON routes return `Result` through `withAuthResult`, which puts the signed-in user on `req.user`. Every environment variable is in `src/utils/env-schema.ts` with its docs, which the docs' Environment variables page is built from; add new ones there. Personal Agent Proxy is one origin everywhere: when `packages/web/dist` exists, as in the Docker image, the api serves the web app's build too (`src/server/serve-web.ts`), and every address the api builds starts from `APP_URL`. Browser-called writes take `requireBrowserOrigin` (CSRF); routes that read a cookie have no CORS. `src/connections/` lists, deletes and sets defaults on an organization's connections, connects a mailbox with an app password after signing in to its IMAP and SMTP servers (`email/`), and connects Granola, Notion and Linear by signing in to their MCP servers (`mcp/`, shared by all three: OAuth with PKCE against the server's authorization server, a client registered for each sign-in so nothing is set up beforehand, the handshake in an encrypted cookie, `mcpConnectRoutes` making both routes on the sign-in flow every OAuth integration shares, `connect-flow.ts`; tokens are refreshed as they run out, one refresh at a time per connection on each server, read again first so ones another call or server saved meanwhile are used rather than refreshed twice, and saved only over the ones they replace, whatever else the credential holds kept beside them, by `connections/token-session.ts`, which Google's connection will share; each tool call is one stateless `tools/call`, `mcp-client.ts`, which also lists a server's tools and, for a server that may keep a session (`handshake`), initializes one around each call as the spec asks). Every request to an MCP server, its metadata or its authorization server goes through `utils/checked-request.ts`, which checks the address as a database's host is checked (`utils/host-check.ts`: nothing private unless `ALLOW_PRIVATE_HOSTS`, and `https` unless the address is private) before each request, since a custom server and its endpoints are wherever it said, and never follows a redirect. A custom MCP server is connected by its address (`custom/`): `POST /api/connections/mcp` with `{name, url, token?, returnTo?}` (the name is the person's, kept through the sign-in in its cookie, since a server says nothing about itself that would do) asks the server for its tools, connects a server that answers at once, with no sign-in or the token typed in as a bearer token (as written when it has its scheme in front, `Basic …`), and for one that turns a request without a token down begins its sign-in there, under the browser's origin check, answering `{kind: 'signIn', url}` with the sign-in page the browser goes to: `mcp-discovery.ts` finds the authorization server from the server's protected resource metadata (the `WWW-Authenticate` header's `resource_metadata`, else the well-known paths), its endpoints from that server's metadata (else `/authorize`, `/token` and `/register` under it) and the scopes the server's own metadata names, and the sign-in, begun with `signInFlow` (`connect-flow.ts`, which `mcpConnectRoutes` builds on for the servers known in advance), keeps the endpoints in the handshake cookie and in the credential beside the tokens. The connection's account is the address without its scheme (`mcp.example.com/mcp`), connecting the same address again replaces its credential and tools, a new connection's defaults are the tools that say they only read, `POST /api/connections/:id/tools` asks the server for its tools again (replacing the tools drops the defaults and agents' settings for tools the server no longer lists, `updateConnectionTools`), and a tool is run as it is (`custom-mcp.ts`): its params go to the server as given, a tool that fails answers the agent with the server's own text as a `validation_error`, and the run is logged under the collection `tools` by the tool's name, with its params as JSON, cut to 200 characters. Granola's notes and transcripts are on by default, Notion's Read pages and Linear's Read issues alone; a Notion or Linear sign-in is for one workspace, and a Notion one lasts at most 180 days, however often it is refreshed. `src/agents/` makes, revokes and deletes agent logins (the password is sent once, only its argon2id hash is kept) and sets an agent's own settings per action of a connection where they differ from the default. `src/records/` reads and writes a collection's records through the integration's `Connector` (`connectors.ts`): Information's live in the database (`info-connector.ts`), a mailbox's are fetched over IMAP on every request (`email-connector.ts`), which merges the inbox and the folders the server marks as Drafts and Sent into one list (ids start with the folder: `inbox-…`, `drafts-…`, `sent-…`) and runs the commands: flags, moves to the folders the server marks as Archive (Gmail's All Mail) or Trash, and sending over SMTP, a draft or a new email, filing a copy in Sent where the server doesn't (Gmail does). Received emails are never edited or deleted for good; only drafts are. Granola's notes are fetched from its MCP server on every request (`granola-connector.ts`; the reading of the XML-like text Granola answers in is in `connections/granola/granola-mcp.ts`): its Notes and Transcripts collections are the same meetings by Granola's meeting id, and since Granola's tools have no pages, a page is 30 days of meetings; one that finds none skips back to the newest meeting before it, so the list ends where the meetings do. Granola's dates and attendees are shown as Granola writes them. Notion's workspace is fetched from its MCP server the same way (`notion-connector.ts`, the reading of its answers in `connections/notion/notion-mcp.ts`), as the person who signed in, as a tree in one nested collection: Notion has no list of every page, so the top is their Recents and a search (Notion's own, AI search where the workspace has it) finds the rest, one page of 50; inside a page are the pages and databases in its content, inside a database its saved views (`view-…` ids), or the rows of its only one, and inside a view its rows, paged with Notion's cursor (view queries have no plan quota; Notion's row and SQL queries do). A page reads as Notion-flavored Markdown, a row's properties beside it as "Name: value" lines (`notion-properties.ts`), and only a page's, database's or view's id is ever sent to Notion's fetch; a new page is a private draft at the top, or goes inside the page, database or view it is made in, a row's title in the database's title property; an edit sends the title, the properties written differently and the content, each only where it changed, and pages are never deleted. Notion's MCP server has no tool that deletes or trashes a page. Linear's issues are fetched from its MCP server the same way (`linear-connector.ts`, the reading of its answers in `connections/linear/linear-mcp.ts`), as the person who signed in, by their identifier (`ENG-123`): a list is the issues updated last first, 50 to a page with Linear's cursor, narrowed by the type of status (`?statusType=Started`) and searched in titles and descriptions, and an issue opens with its comments, oldest first. Teams, statuses, people, projects and labels are written by name, as Linear shows them and finds them, its refusals passed on to the agent; an edit sends only the fields written differently, an empty one clearing it. Linear's MCP server has no tool that archives or deletes an issue. Google is connected by signing in to Google with the app's own Google client, the one Google sign-in uses (`GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET`, a second redirect URI `/api/connections/google/callback`), asking for every scope at once, Gmail, Drive, Docs, Sheets and Slides, with offline access (`connections/google/`: `google-oauth.ts` the sign-in and refresh, with the client's secret, keeping the refresh token Google doesn't send again; `google-client.ts` the calls with a bearer token and Google's errors, a disabled API passing Google's own message on, which names the API and where to enable it); the credential holds the tokens, the scopes Google granted, since the consent screen lets a person untick some, and the address; the connection keeps Google's id for the account (`Connection.providerAccountId`, its `sub`), so signing in again renews the same connection, and its address, even after the address changed (`findAccountConnection`, by address for one without an id and for the MCP integrations). Each action needs one of the scopes allowing it (`google/scopes.ts`), so an action whose scope wasn't granted is unavailable: `connections/unavailable.ts`, the one place the rest of the api asks, reads it from the credential on every request (`unavailable` on a connection; one that can't be read leaves every action unavailable rather than failing the list), no default or agent setting can turn it on, `runAgentTool` refuses it saying to sign in to Google again, and the connection page shows its checkbox disabled with a "Sign in again" link (`?account=` opening Google on the account with `login_hint`). An integration's `disclosure`, Google's Limited Use line, is shown above a connection's permissions. Disconnecting a Google connection revokes its grant at Google first (`connections/disconnect.ts`), unless another connection in any organization is for the same account, since a grant covers all of the client's tokens for it, and deletes it even when Google can't be reached. Its Drive is one nested collection, `files` (`records/google-files.ts`, behind `google-connector.ts`, which dispatches on the collection), fetched live from Drive's and Sheets' APIs (`drive-api.ts`, `sheets-api.ts`): at the top four places, My Drive, Shared with me, Shared drives (`drive-…` ids) and Recent, inside them and inside folders their files, folders first, 50 to a page with Drive's page token, and a search is Drive's full-text one, anywhere or inside the folder opened; the trail is walked up a file's parents to My Drive, its shared drive, or, where the account can't see further up, Shared with me. A document opens as Markdown and a presentation as text, as Drive exports them, up to its 10 MB export cap, beyond which opening it says so; a spreadsheet opens into its tabs (`sheet--`) and a tab into its rows (`row---`), 50 to a page, the page token the next row's number, each row's values "Header: value" lines under the tab's first row, a column without a header going by its letter (`Column C`); a page past the tab's last row is empty and a row past it not found without asking Sheets, which refuses a range outside the grid. Drive isn't written to yet; writes, and Gmail, come later. PostgreSQL and MySQL (MariaDB too) are two integrations on one code path: a database is connected with its host, port, database, user, password and encryption (`connections/sql/`: `route.ts` makes both routes per dialect, `sql-client.ts` opens one connection per request with `Bun.sql`, to the address `utils/host-check.ts` resolves the host to, refusing private addresses unless `ALLOW_PRIVATE_HOSTS` is set, and `sql-statement.ts` checks an agent's query is one statement that reads), and `records/sql-connector.ts` reads and writes it through `records/sql-dialect.ts`, where the two differ: the tables are one nested collection, a table holding its rows, a row read and written as "column: value" lines and found again by its primary key (ids `public.users:42`, `pairs:3,4`; a table without a key is never written to), and a query, a second collection that is only ever run, goes through a read-only transaction with a time limit, a cursor on Postgres and `sql_select_limit` on MySQL, 200 rows at most, answered as JSON lines. MySQL's read-only transaction doesn't stop DDL, so the statement check is what keeps a query to reading; the docs tell people to make a read-only database user as well. Lists take `?search=` (only where the collection has a `searchHint`; Granola has none), `?filter=` (the collection's `filterField`, such as `?filter=Draft`), `?parent=` (only in a `nested` collection: the list inside a record, answered with its `trail`, the records above it; a record made there goes in it) and `?page=` (`list-query.ts`) and answer a page, newest first (a mailbox's 50 emails, Granola's 30 days), with the next page's token; a mailbox's token holds where each folder goes on, so paging through the merged list shows every email once, and its search uses Gmail's own syntax on Gmail (`X-GM-EXT-1`) and IMAP text search elsewhere. Agents sign in at `/agent-auth/login` with a cookie of their own (`src/agent-auth/`); `src/agent-side/` serves them: the connections an agent can use, the tools it has with each, and `runAgentTool` (`tools.ts`), the one place a tool runs, which checks it against the agent's access as it is now, validates its params against the tool's schema (an edit changes only the fields given), calls the connector, or a custom server's tool as it is, answering `{text}`, and logs it, denied ones too, to the activity log that `src/activity/` lists. Over HTTP that is `/api/agent/me`, `GET /api/agent/connections/:id/tools` and `POST /api/agent/connections/:id/tools/:tool` with `{params}`; `src/mcp/` serves the same over MCP at `/mcp` (Streamable HTTP without sessions, JSON answers, no stream): four tools whatever the login may do, `list_connections` (the entry point), `list_tools`, `read` (only tools that read, marked read-only) and `run`, and server instructions naming the agent's connections. MCP clients sign in with OAuth 2.1 (`src/oauth/`), Personal Agent Proxy being its own authorization server: metadata under `/.well-known/`, Dynamic Client Registration at `/oauth/register` (open to anyone, each registration first deleting the clients that registered over a day ago and never finished a sign-in), `/oauth/authorize` sealing the request into the web app's `/connect-agent` page, where the person chooses the agent login the client works as or makes one, and `/oauth/token` with PKCE (S256 only) and refresh tokens replaced on every use; codes and tokens are kept as their SHA-256, and revoking an agent ends its clients' tokens. These endpoints take any origin (CORS), since none reads a cookie; credentials and Information records are stored encrypted with `ENCRYPTION_KEY` (`src/utils/secret-crypto.ts`). Has its own `.prettierrc.json` with a 200-column print width +- `packages/api` - `@proxy/api`: Express on Bun. Routes are `handleXRoute` functions mounted in `src/server/create-api-router.ts`; JSON routes return `Result` through `withAuthResult`, which puts the signed-in user on `req.user`. Every environment variable is in `src/utils/env-schema.ts` with its docs, which the docs' Environment variables page is built from; add new ones there. Personal Agent Proxy is one origin everywhere: when `packages/web/dist` exists, as in the Docker image, the api serves the web app's build too (`src/server/serve-web.ts`), and every address the api builds starts from `APP_URL`. Browser-called writes take `requireBrowserOrigin` (CSRF); routes that read a cookie have no CORS. `src/connections/` lists, deletes and sets defaults on an organization's connections, connects a mailbox with an app password after signing in to its IMAP and SMTP servers (`email/`), and connects Granola, Notion and Linear by signing in to their MCP servers (`mcp/`, shared by all three: OAuth with PKCE against the server's authorization server, a client registered for each sign-in so nothing is set up beforehand, the handshake in an encrypted cookie, `mcpConnectRoutes` making both routes on the sign-in flow every OAuth integration shares, `connect-flow.ts`; tokens are refreshed as they run out, one refresh at a time per connection on each server, read again first so ones another call or server saved meanwhile are used rather than refreshed twice, and saved only over the ones they replace, whatever else the credential holds kept beside them, by `connections/token-session.ts`, which Google's connection will share; each tool call is one stateless `tools/call`, `mcp-client.ts`, which also lists a server's tools and, for a server that may keep a session (`handshake`), initializes one around each call as the spec asks). Every request to an MCP server, its metadata or its authorization server goes through `utils/checked-request.ts`, which checks the address as a database's host is checked (`utils/host-check.ts`: nothing private unless `ALLOW_PRIVATE_HOSTS`, and `https` unless the address is private) before each request, since a custom server and its endpoints are wherever it said, and never follows a redirect. A custom MCP server is connected by its address (`custom/`): `POST /api/connections/mcp` with `{name, url, token?, returnTo?}` (the name is the person's, kept through the sign-in in its cookie, since a server says nothing about itself that would do) asks the server for its tools, connects a server that answers at once, with no sign-in or the token typed in as a bearer token (as written when it has its scheme in front, `Basic …`), and for one that turns a request without a token down begins its sign-in there, under the browser's origin check, answering `{kind: 'signIn', url}` with the sign-in page the browser goes to: `mcp-discovery.ts` finds the authorization server from the server's protected resource metadata (the `WWW-Authenticate` header's `resource_metadata`, else the well-known paths), its endpoints from that server's metadata (else `/authorize`, `/token` and `/register` under it) and the scopes the server's own metadata names, and the sign-in, begun with `signInFlow` (`connect-flow.ts`, which `mcpConnectRoutes` builds on for the servers known in advance), keeps the endpoints in the handshake cookie and in the credential beside the tokens. The connection's account is the address without its scheme (`mcp.example.com/mcp`), connecting the same address again replaces its credential and tools, a new connection's defaults are the tools that say they only read, `POST /api/connections/:id/tools` asks the server for its tools again (replacing the tools drops the defaults and agents' settings for tools the server no longer lists, `updateConnectionTools`), and a tool is run as it is (`custom-mcp.ts`): its params go to the server as given, a tool that fails answers the agent with the server's own text as a `validation_error`, and the run is logged under the collection `tools` by the tool's name, with its params as JSON, cut to 200 characters. Granola's notes and transcripts are on by default, Notion's Read pages and Linear's Read issues alone; a Notion or Linear sign-in is for one workspace, and a Notion one lasts at most 180 days, however often it is refreshed. `src/agents/` makes, revokes and deletes agent logins (the password is sent once, only its argon2id hash is kept) and sets an agent's own settings per action of a connection where they differ from the default. `src/records/` reads and writes a collection's records through the integration's `Connector` (`connectors.ts`): Information's live in the database (`info-connector.ts`), a mailbox's are fetched over IMAP on every request (`email-connector.ts`), which merges the inbox and the folders the server marks as Drafts and Sent into one list (ids start with the folder: `inbox-…`, `drafts-…`, `sent-…`) and runs the commands: flags, moves to the folders the server marks as Archive (Gmail's All Mail) or Trash, and sending over SMTP, a draft or a new email, filing a copy in Sent where the server doesn't (Gmail does). Received emails are never edited or deleted for good; only drafts are. Granola's notes are fetched from its MCP server on every request (`granola-connector.ts`; the reading of the XML-like text Granola answers in is in `connections/granola/granola-mcp.ts`): its Notes and Transcripts collections are the same meetings by Granola's meeting id, and since Granola's tools have no pages, a page is 30 days of meetings; one that finds none skips back to the newest meeting before it, so the list ends where the meetings do. Granola's dates and attendees are shown as Granola writes them. Notion's workspace is fetched from its MCP server the same way (`notion-connector.ts`, the reading of its answers in `connections/notion/notion-mcp.ts`), as the person who signed in, as a tree in one nested collection: Notion has no list of every page, so the top is their Recents and a search (Notion's own, AI search where the workspace has it) finds the rest, one page of 50; inside a page are the pages and databases in its content, inside a database its saved views (`view-…` ids), or the rows of its only one, and inside a view its rows, paged with Notion's cursor (view queries have no plan quota; Notion's row and SQL queries do). A page reads as Notion-flavored Markdown, a row's properties beside it as "Name: value" lines (`notion-properties.ts`), and only a page's, database's or view's id is ever sent to Notion's fetch; a new page is a private draft at the top, or goes inside the page, database or view it is made in, a row's title in the database's title property; an edit sends the title, the properties written differently and the content, each only where it changed, and pages are never deleted. Notion's MCP server has no tool that deletes or trashes a page. Linear's issues are fetched from its MCP server the same way (`linear-connector.ts`, the reading of its answers in `connections/linear/linear-mcp.ts`), as the person who signed in, by their identifier (`ENG-123`): a list is the issues updated last first, 50 to a page with Linear's cursor, narrowed by the type of status (`?statusType=Started`) and searched in titles and descriptions, and an issue opens with its comments, oldest first. Teams, statuses, people, projects and labels are written by name, as Linear shows them and finds them, its refusals passed on to the agent; an edit sends only the fields written differently, an empty one clearing it. Linear's MCP server has no tool that archives or deletes an issue. Google is connected by signing in to Google with the app's own Google client, the one Google sign-in uses (`GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET`, a second redirect URI `/api/connections/google/callback`), asking for every scope at once, Gmail, Drive, Docs, Sheets and Slides, with offline access (`connections/google/`: `google-oauth.ts` the sign-in and refresh, with the client's secret, keeping the refresh token Google doesn't send again; `google-client.ts` the calls with a bearer token and Google's errors, a disabled API passing Google's own message on, which names the API and where to enable it); the credential holds the tokens, the scopes Google granted, since the consent screen lets a person untick some, and the address; the connection keeps Google's id for the account (`Connection.providerAccountId`, its `sub`), so signing in again renews the same connection, and its address, even after the address changed (`findAccountConnection`, by address for one without an id and for the MCP integrations). Each action needs one of the scopes allowing it (`google/scopes.ts`), so an action whose scope wasn't granted is unavailable: `connections/unavailable.ts`, the one place the rest of the api asks, reads it from the credential on every request (`unavailable` on a connection; one that can't be read leaves every action unavailable rather than failing the list), no default or agent setting can turn it on, `runAgentTool` refuses it saying to sign in to Google again, and the connection page shows its checkbox disabled with a "Sign in again" link (`?account=` opening Google on the account with `login_hint`). An integration's `disclosure`, Google's Limited Use line, is shown above a connection's permissions. Disconnecting a Google connection revokes its grant at Google first (`connections/disconnect.ts`), unless another connection in any organization is for the same account, since a grant covers all of the client's tokens for it, and deletes it even when Google can't be reached. Its Gmail is the `emails` collection, the same fields and commands as a mailbox's under the Google action ids (`readEmails` … `sendEmails`, both built by `emailsCollection` in the catalog), fetched live from Gmail's API (`records/google-emails.ts` over `connections/google/gmail-api.ts`): one list, 50 to a page with Gmail's page token, all mail but Spam and Trash, or one folder, each a label (Inbox `INBOX`, Draft `DRAFT`, Sent `SENT`, Trash `TRASH`, Spam `SPAM`) or, for Archive, a search for what is in none of them; an email's labels say its folder, so ids are Gmail's message ids, a draft's `draft-`, the id Gmail edits, deletes and sends it by and which stays as it is edited. `messages.list` gives only ids, so each is asked for with its headers, 8 at once, again twice when Gmail answers busy; Gmail's per-user quota of 6,000 units a minute makes a page cost about 1,000. A search is Gmail's own. An email opens as its text part, or its HTML part as text (read in one pass, `htmlToText`), with its attachments' names. Marking takes `UNREAD` off or puts it on, starring `STARRED`, archiving takes `INBOX` off, and Move to Trash is `messages.trash`, never `messages.delete`: nothing is deleted for good but drafts, the only email edited or deleted, an edit keeping the draft's Cc, Bcc and thread, its recipients' names where they are written as they read and its HTML where its text is, and refusing a draft with attachments, which it would drop; a draft is edited, deleted and sent by its message id too. Email goes out from the connection's own address, a draft with `drafts.send` and a new one with `messages.send`, written by `MailComposer` as the IMAP connector writes it; Gmail files it in Sent. Its Drive is one nested collection, `files` (`records/google-files.ts`, behind `google-connector.ts`, which dispatches on the collection), fetched live from Drive's and Sheets' APIs (`drive-api.ts`, `sheets-api.ts`): at the top four places, My Drive, Shared with me, Shared drives (`drive-…` ids) and Recent, inside them and inside folders their files, folders first, 50 to a page with Drive's page token, and a search is Drive's full-text one, anywhere or inside the folder opened; the trail is walked up a file's parents to My Drive, its shared drive, or, where the account can't see further up, Shared with me. A document opens as Markdown and a presentation as text, as Drive exports them, up to its 10 MB export cap, beyond which opening it says so; a spreadsheet opens into its tabs (`sheet--`) and a tab into its rows (`row---`), 50 to a page, the page token the next row's number, each row's values "Header: value" lines under the tab's first row, a column without a header going by its letter (`Column C`); a page past the tab's last row is empty and a row past it not found without asking Sheets, which refuses a range outside the grid. Drive isn't written to yet; its writes come later. PostgreSQL and MySQL (MariaDB too) are two integrations on one code path: a database is connected with its host, port, database, user, password and encryption (`connections/sql/`: `route.ts` makes both routes per dialect, `sql-client.ts` opens one connection per request with `Bun.sql`, to the address `utils/host-check.ts` resolves the host to, refusing private addresses unless `ALLOW_PRIVATE_HOSTS` is set, and `sql-statement.ts` checks an agent's query is one statement that reads), and `records/sql-connector.ts` reads and writes it through `records/sql-dialect.ts`, where the two differ: the tables are one nested collection, a table holding its rows, a row read and written as "column: value" lines and found again by its primary key (ids `public.users:42`, `pairs:3,4`; a table without a key is never written to), and a query, a second collection that is only ever run, goes through a read-only transaction with a time limit, a cursor on Postgres and `sql_select_limit` on MySQL, 200 rows at most, answered as JSON lines. MySQL's read-only transaction doesn't stop DDL, so the statement check is what keeps a query to reading; the docs tell people to make a read-only database user as well. Lists take `?search=` (only where the collection has a `searchHint`; Granola has none), `?filter=` (the collection's `filterField`, such as `?filter=Draft`), `?parent=` (only in a `nested` collection: the list inside a record, answered with its `trail`, the records above it; a record made there goes in it) and `?page=` (`list-query.ts`) and answer a page, newest first (a mailbox's 50 emails, Granola's 30 days), with the next page's token; a mailbox's token holds where each folder goes on, so paging through the merged list shows every email once, and its search uses Gmail's own syntax on Gmail (`X-GM-EXT-1`) and IMAP text search elsewhere. Agents sign in at `/agent-auth/login` with a cookie of their own (`src/agent-auth/`); `src/agent-side/` serves them: the connections an agent can use, the tools it has with each, and `runAgentTool` (`tools.ts`), the one place a tool runs, which checks it against the agent's access as it is now, validates its params against the tool's schema (an edit changes only the fields given), calls the connector, or a custom server's tool as it is, answering `{text}`, and logs it, denied ones too, to the activity log that `src/activity/` lists. Over HTTP that is `/api/agent/me`, `GET /api/agent/connections/:id/tools` and `POST /api/agent/connections/:id/tools/:tool` with `{params}`; `src/mcp/` serves the same over MCP at `/mcp` (Streamable HTTP without sessions, JSON answers, no stream): four tools whatever the login may do, `list_connections` (the entry point), `list_tools`, `read` (only tools that read, marked read-only) and `run`, and server instructions naming the agent's connections. MCP clients sign in with OAuth 2.1 (`src/oauth/`), Personal Agent Proxy being its own authorization server: metadata under `/.well-known/`, Dynamic Client Registration at `/oauth/register` (open to anyone, each registration first deleting the clients that registered over a day ago and never finished a sign-in), `/oauth/authorize` sealing the request into the web app's `/connect-agent` page, where the person chooses the agent login the client works as or makes one, and `/oauth/token` with PKCE (S256 only) and refresh tokens replaced on every use; codes and tokens are kept as their SHA-256, and revoking an agent ends its clients' tokens. These endpoints take any origin (CORS), since none reads a cookie; credentials and Information records are stored encrypted with `ENCRYPTION_KEY` (`src/utils/secret-crypto.ts`). Has its own `.prettierrc.json` with a 200-column print width - `skills/` - Agent skills, symlinked into `.claude/skills` and `.codex/skills` by `bun run setup` ## The app Personal Agent Proxy holds the integrations and personal information an AI agent needs, so every agent company does not have to rebuild them and you do not have to trust each one's security. A human connects services and hands out agent logins; an agent signs in with one and works through what it was given. Everything is served by `packages/api`; the web app has no data of its own. -- **Model**: an integration from the catalog (`@proxy/integrations`: Email over IMAP and SMTP, Google's Drive, Docs, Sheets and Slides over Google's APIs, one sign-in for a Google account, Granola's meeting notes, Notion's pages and Linear's issues over their MCP servers, PostgreSQL and MySQL over their own protocols) has the actions an agent can be allowed with a connection of it (Read, Archive, Write drafts, Send) and collections, each a list of fields with the actions reading and writing it need, and the commands it offers beyond editing (Mark as read, Archive, Send), each needing one action. A command runs on a `record`, where its `where` picks out which (Archive only in the inbox, Send only on a draft), or on values typed in (`new`: sending a new email). A mailbox is one collection, its folder a field the agent side filters by. A `nested` collection is a tree, as Notion's pages are: a record can hold others of the collection (`hasChildren`), listed inside it with the list tool's `parent` and made inside it with the create tool's, and access is the same anywhere in it. A collection without a `read` action is never listed or opened, only run, as a database's queries are, and a `new` command marked `readOnly` counts as a read for MCP clients. Integrations join the catalog once they work end to end. Any other service with an MCP server is connected as a custom MCP server: the server's tools are the actions, each allowed on its own, and there are no collections; an agent runs a tool with the params the server describes and gets back what it answered. A connection is a connected integration and its account, owned by the organization; an integration can be connected more than once, such as two mailboxes. Information is the built-in `info` connection (addresses, payment cards, notes), made with the organization, so access and the agent side treat it like any other. Nothing is copied from a provider: every request goes to it live. Fields marked `system` are set by the provider, never typed in. +- **Model**: an integration from the catalog (`@proxy/integrations`: Email over IMAP and SMTP, Google's Gmail, Drive, Docs, Sheets and Slides over Google's APIs, one sign-in for a Google account, Granola's meeting notes, Notion's pages and Linear's issues over their MCP servers, PostgreSQL and MySQL over their own protocols) has the actions an agent can be allowed with a connection of it (Read, Archive, Write drafts, Send) and collections, each a list of fields with the actions reading and writing it need, and the commands it offers beyond editing (Mark as read, Archive, Send), each needing one action. A command runs on a `record`, where its `where` picks out which (Archive only in the inbox, Send only on a draft), or on values typed in (`new`: sending a new email). A mailbox is one collection, its folder a field the agent side filters by. A `nested` collection is a tree, as Notion's pages are: a record can hold others of the collection (`hasChildren`), listed inside it with the list tool's `parent` and made inside it with the create tool's, and access is the same anywhere in it. A collection without a `read` action is never listed or opened, only run, as a database's queries are, and a `new` command marked `readOnly` counts as a read for MCP clients. Integrations join the catalog once they work end to end. Any other service with an MCP server is connected as a custom MCP server: the server's tools are the actions, each allowed on its own, and there are no collections; an agent runs a tool with the params the server describes and gets back what it answered. A connection is a connected integration and its account, owned by the organization; an integration can be connected more than once, such as two mailboxes. Information is the built-in `info` connection (addresses, payment cards, notes), made with the organization, so access and the agent side treat it like any other. Nothing is copied from a provider: every request goes to it live. Fields marked `system` are set by the provider, never typed in. - **Access** is one list of actions per connection, and every action the catalog lists can be set: what Personal Agent Proxy can do with the provider is the limit. Two layers: the connection's default (a `ConnectionDefault` row per action that is on) and an agent's own setting, on or off, where it differs (`AgentGrant`). An agent gets its own setting, else the default as it is now; an action that `requires` another counts only with it (Archive needs Read; Send needs nothing, so an agent can send without reading the mailbox), and an action a sign-in didn't cover, as Google's consent screen allows, is unavailable whatever is ticked. A collection's `writes` name the action creating, editing and deleting need, and `editable` which records they apply to (drafts). An agent login is for one agent from `AGENT_PROVIDERS`; each organization can have one login per company, named by it, and any number of custom MCP agents (`custom`, for any MCP client not in the list), each named by the person and told apart by their names (`Agent` is unique by organization, provider and name). It has a generated username and a password that is shown once. Every request on the agent side is an audit entry, allowed or denied, with what a list searched for. - **Accounts**: a human signs in with Google or a magic link (`src/auth/` in the api); the sign-in page says that continuing accepts the terms and the privacy policy, on the website, and a sign-up records it (`User.termsAcceptedAt`); accounts from before the notice have none until they are asked. A session is a `Session` row and an httpOnly cookie; the web app's `/api/me` loader on the human side's route redirects to `/login?callbackUrl=…` without one. Every user gets one `Organization` at sign-up, created with the user. `ALLOWED_SIGNUP_EMAILS` limits who can sign up; the login page asks `/auth/methods` whether to offer Google. In development the login page has a Create dev user button; in development, or without `RESEND_KEY`, the api prints magic links instead of emailing them. Cookies are secure when `APP_URL` is HTTPS. A new organization first goes through the welcome flow (`/welcome`, outside the human side's layout; the human side redirects there until `Organization.onboardedAt` is set, which `POST /api/me/onboarded` does when the flow is finished or skipped): step one is `AgentSetup` (`src/components/agent-setup.tsx`, which New agent and setting an agent up again use too): it picks the agent from `AGENT_PROVIDERS`, the choice in the address (`?agent=claude`) so the browser's back button returns to it, and does what its `connects` says, an `mcp` agent (Claude, Claude Code, Claude Desktop, Hermes Agent, ChatGPT, Poke) shown the server address and that agent's steps (Claude Code and Hermes Agent use terminal commands) while the page polls for the login its sign-in makes, and over http, as in development, only a `local` agent (Claude Code, Claude Desktop, Hermes Agent) can be picked, the rest greyed out with a pointer to self-hosting on a server with a domain, a `login` agent given a login at once, as a message to paste to it with the sign-in page, username and password, or, for a `vault` agent (Instinct), which won't take a password in chat, as three steps: a message asking it for a link to its vault, the login to fill in on the page it sends, and a message telling it to sign in; step two (`/welcome/connections`) is the catalog, a plus per integration with how many accounts are connected: a mailbox goes to `/welcome/connections/email`, the form on a page with Continue once signed in, and a sign-in started with `?returnTo=welcome` comes back to `/welcome/connections/:id`, the new connection's permissions (its defaults, saved as they are ticked) with Confirm back to the catalog; a database goes to `/welcome/connections/postgres` or `/welcome/connections/mysql` the same way as a mailbox. Organizations from before the flow were marked done by its migration. - **Human side** behind `/login`: `/connections` (a right click on a row renames the connection, `PUT /api/connections/:id/name`, an empty name being none of its own, which a custom MCP server can't have, or deletes it, each behind a dialog), `/connections/new` (the catalog, a plus per integration with how many accounts are connected: `/connections/new/email` connects a mailbox with an app password, with a Test connection that signs in and saves nothing, `/connections/new/postgres` and `/connections/new/mysql` a database the same way, `/connections/new/mcp` a custom MCP server under a name of the person's, by its address, with an access token where the server takes one instead of a sign-in (`ConnectMcpForm`, which sends the browser to the server's sign-in when the api says it asks for one), and Google, Granola, Notion and Linear are connected by signing in to them; an integration needing a client the server doesn't have (Google without `GOOGLE_CLIENT_ID` and `GOOGLE_CLIENT_SECRET`, which `/auth/methods` reports) is greyed out with a pointer to the docs, and its sign-in, reached anyway, comes back with `?error=google-unconfigured` saying the same; `ConnectEmailForm`, `ConnectDatabaseForm`, `ConnectMcpForm` and `IntegrationCatalog` serve the welcome flow too, at `/welcome/connections/`), `/connections/:id` (titled by the name the person gave it where they did, as the list and the agent side are, the defaults under Permissions, with how many agents differ per action, a custom MCP server's tools being its actions, with a button that asks the server for them again, and a "Connection added successfully" when `?added` says it was just connected), `/info`, `/agents`, `/agents/new` (the welcome flow's agent step, among the companies without a login yet and Custom MCP agent, on to the agent's page), `/agents/:id/setup` (the same for one agent, to hand it a new password or add the connector again; the only place a password shows, once), `/agents/:id` (what the agent can reach and did, the MCP clients working as it when there are any; access laid out like an editor's settings: one checkbox per action of each connection, the only control for it, each connection folded by default to a row saying Default or Changed; a setting of the agent's own is marked with a bar, a button that makes it the connection's default for every agent, and one that resets it, and a filter and Changed only narrow the list; `ConnectionAccess` serves both pages), `/activity` (filters in the URL, so the agent and connection pages link to their part of the log; a custom server's tool reads "Ran Search docs with {…}"). Each page has a loader in `src/loaders.ts` and revalidates after a change. Layout follows the spanish repo: the shadcn sidebar as a rail, an `h-12` header, one `max-w-3xl` column, one-line `RowList` rows. Type has one size, `text-sm`, in rows, summaries and hints alike, except column headers (`RowHeader`), which are `text-xs`: the thing itself in the normal colour, what is said about it (column headers, times, summaries, descriptions) in `text-muted-foreground`, and `font-medium` only for page and section titles. diff --git a/packages/api/src/connections/google/gmail-api.test.ts b/packages/api/src/connections/google/gmail-api.test.ts new file mode 100644 index 00000000..d91aab11 --- /dev/null +++ b/packages/api/src/connections/google/gmail-api.test.ts @@ -0,0 +1,148 @@ +import {describe, expect, mock, test} from 'bun:test'; +import {Ok} from 'ts-results-es'; + +import type {MessagePart} from './gmail-api'; + +// The Gmail calls go through the Google client, which reads the environment and the database on +// import; only the reading of Gmail's answers is tested here. +mock.module('@proxy/db/connection', () => ({updateConnectionCredential: mock(), getConnectionCredential: mock(async () => Ok(null))})); + +mock.module('../../utils/env', () => ({env: {NODE_ENV: 'production', APP_URL: 'https://app.example.com', GOOGLE_CLIENT_ID: 'client-1', GOOGLE_CLIENT_SECRET: 'secret-1'}})); + +const {attachmentNames, folderOf, folderQuery, htmlToText, mapWithLimit, messageBody} = await import('./gmail-api'); + +const base64url = (text: string, encoding: BufferEncoding = 'utf8'): string => Buffer.from(text, encoding).toString('base64url'); + +function textPart(mimeType: string, text: string, contentType = `${mimeType}; charset="UTF-8"`): MessagePart { + return {mimeType, filename: '', headers: [{name: 'Content-Type', value: contentType}], body: {size: text.length, data: base64url(text)}}; +} + +function attachment(filename: string): MessagePart { + return {mimeType: 'application/pdf', filename, headers: [{name: 'Content-Disposition', value: `attachment; filename="${filename}"`}], body: {size: 1200, attachmentId: 'att-1'}}; +} + +describe('folderOf', () => { + test('a message is in the folder of its label, the inbox before sent, Trash before a draft', () => { + expect(folderOf(['UNREAD', 'IMPORTANT', 'CATEGORY_PERSONAL', 'INBOX'])).toBe('Inbox'); + expect(folderOf(['SENT'])).toBe('Sent'); + expect(folderOf(['SENT', 'INBOX', 'UNREAD'])).toBe('Inbox'); + expect(folderOf(['DRAFT'])).toBe('Draft'); + expect(folderOf(['DRAFT', 'TRASH'])).toBe('Trash'); + expect(folderOf(['SPAM', 'UNREAD'])).toBe('Spam'); + }); + + test('one in none of the folders is archived', () => { + expect(folderOf(['IMPORTANT', 'Label_12'])).toBe('Archive'); + expect(folderOf([])).toBe('Archive'); + }); +}); + +describe('folderQuery', () => { + test('a folder is its label, with the search as it is', () => { + expect(folderQuery('Inbox', 'from:sam')).toEqual({labelIds: ['INBOX'], q: 'from:sam', includeSpamTrash: false}); + expect(folderQuery('Draft', null)).toEqual({labelIds: ['DRAFT'], q: null, includeSpamTrash: false}); + expect(folderQuery('Sent', null)).toEqual({labelIds: ['SENT'], q: null, includeSpamTrash: false}); + }); + + test('Trash and Spam are asked for, since Gmail leaves them out otherwise', () => { + expect(folderQuery('Trash', null)).toEqual({labelIds: ['TRASH'], q: null, includeSpamTrash: true}); + expect(folderQuery('Spam', 'invoice')).toEqual({labelIds: ['SPAM'], q: 'invoice', includeSpamTrash: true}); + }); + + test('the archive is a search for what is in no folder, the search in parentheses beside it', () => { + expect(folderQuery('Archive', null)).toEqual({labelIds: [], q: '-in:inbox -in:sent -in:drafts -in:trash -in:spam', includeSpamTrash: false}); + expect(folderQuery('Archive', 'from:sam OR from:alex').q).toBe('-in:inbox -in:sent -in:drafts -in:trash -in:spam (from:sam OR from:alex)'); + }); + + test('without a folder, all mail but Spam and Trash', () => { + expect(folderQuery(null, null)).toEqual({labelIds: [], q: null, includeSpamTrash: false}); + }); +}); + +describe('messageBody', () => { + test('the plain text part of an alternative, before the HTML one', () => { + const payload: MessagePart = { + mimeType: 'multipart/alternative', + filename: '', + parts: [textPart('text/plain', 'Hi Alex,\n\nSee you Friday.\n'), textPart('text/html', '

Hi Alex,

See you Friday.

')], + }; + + expect(messageBody(payload)).toBe('Hi Alex,\n\nSee you Friday.'); + }); + + test('a message with only HTML reads as its text, with links and blocks kept', () => { + const html = + '

Your order has shipped & is on its way.

Track it here.
  • One
  • Two
'; + + expect(messageBody(textPart('text/html', html))).toBe('Your order has shipped & is on its way.\nTrack it here (https://example.com/track/1).\n\n- One\n- Two'); + }); + + test('nested parts are walked, a text part in the charset it names, and the attachments listed after', () => { + const latin = {mimeType: 'text/plain', filename: '', headers: [{name: 'Content-Type', value: 'text/plain; charset=ISO-8859-1'}], body: {size: 12, data: base64url('Grüße, Sam', 'latin1')}}; + const payload: MessagePart = { + mimeType: 'multipart/mixed', + filename: '', + parts: [{mimeType: 'multipart/alternative', filename: '', parts: [latin, textPart('text/html', '

Grüße

')]}, attachment('invoice.pdf'), attachment('receipt.pdf')], + }; + + expect(messageBody(payload)).toBe('Grüße, Sam\n\nAttachments: invoice.pdf, receipt.pdf'); + expect(attachmentNames(payload)).toEqual(['invoice.pdf', 'receipt.pdf']); + }); + + test('a text attachment is not the body', () => { + const payload: MessagePart = {mimeType: 'multipart/mixed', filename: '', parts: [{...textPart('text/plain', 'notes'), filename: 'notes.txt'}, textPart('text/html', '

The body

')]}; + + expect(messageBody(payload)).toBe('The body\n\nAttachments: notes.txt'); + }); + + test('a message without parts is empty', () => { + expect(messageBody(undefined)).toBe(''); + }); + + test("a part's line endings read as new lines in text, and as spaces in HTML", () => { + expect(messageBody(textPart('text/plain', 'Hi Alex,\r\n\r\nSee you Friday.\r\n'))).toBe('Hi Alex,\n\nSee you Friday.'); + expect(messageBody(textPart('text/html', '

Hi\r\nAlex,

\r\n

See you\nFriday.

'))).toBe('Hi Alex,\nSee you Friday.'); + }); +}); + +test('htmlToText finds the end of a hidden element after a letter that lowercases to two', () => { + expect(htmlToText('

İstanbul

after

')).toBe('İstanbul\nafter'); + expect(htmlToText(`

${'İ'.repeat(12)} before

after

more

`)).toBe(`${'İ'.repeat(12)} before\nafter\nmore`); +}); + +test('htmlToText decodes entities only once the tags are gone', () => { + expect(htmlToText('

<b> is bold — ☺

')).toBe(' is bold — ☺'); +}); + +test('htmlToText reads a number Unicode has no character for as the replacement character', () => { + expect(htmlToText('

a�b�c�d

')).toBe('a\uFFFDb\uFFFDc\uFFFDd'); +}); + +test('htmlToText keeps the text of unclosed tags, and of a link that never ends', () => { + expect(htmlToText('

Hi there', open + 4); + at = end === -1 ? html.length : end + 3; + continue; + } + const close = html.indexOf('>', open); + if (close === -1) { + parts.push(sourceText(html.slice(open))); + break; + } + const tag = html.slice(open, close + 1); + at = close + 1; + const [, slash = '', rawName = ''] = /^<(\/?)([a-z][a-z0-9]*)/i.exec(tag) ?? []; + const name = rawName.toLowerCase(); + const closing = slash === '/'; + if (!closing && HIDDEN.includes(name)) { + const end = indexOfTag(html, `', end); + at = endClose === -1 ? html.length : endClose + 1; + continue; + } + if (name === 'a' && !closing) { + const href = /\shref\s*=\s*"([^"]*)"/i.exec(tag)?.[1]; + link = href === undefined ? null : {href, start: parts.length}; + continue; + } + if (name === 'a' && closing && link) { + const label = parts.splice(link.start).join('').trim(); + parts.push(label === '' || label === link.href ? link.href : `${label} (${link.href})`); + link = null; + continue; + } + parts.push(tagText(name, closing)); + } + return decodeEntities(parts.join('')) + .replace(/[ \t\u00a0]+/g, ' ') + .replace(/ ?\n ?/g, '\n') + .replace(/\n{3,}/g, '\n\n') + .trim(); +} + +// Text between tags: a line break in the HTML's source is a space, as a browser shows it. +function sourceText(text: string): string { + return text.replace(/\r\n?|\n/g, ' '); +} + +// Where a tag next starts from `from`, whatever its case. The email isn't lowercased whole to look, +// since a letter can lowercase to more than one, as İ does, which would move everything after it. +function indexOfTag(html: string, tag: string, from: number): number { + const pattern = new RegExp(tag, 'ig'); + pattern.lastIndex = from; + return pattern.exec(html)?.index ?? -1; +} + +/** A message's HTML part, as its sender wrote it, or null for one without. */ +export function messageHtml(payload: MessagePart | undefined): string | null { + const html = payload ? findText(payload, 'text/html') : null; + return html ? decodePart(html) : null; +} + +/** A message's text: its plain text part, else its HTML part as text, then the names of its attachments. */ +export function messageBody(payload: MessagePart | undefined): string { + if (!payload) { + return ''; + } + const plain = findText(payload, 'text/plain'); + const html = plain ? null : findText(payload, 'text/html'); + const text = plain ? decodePart(plain).replace(/\r\n?/g, '\n').trim() : html ? htmlToText(decodePart(html)) : ''; + const attachments = attachmentNames(payload); + if (attachments.length === 0) { + return text; + } + return `${text}\n\nAttachments: ${attachments.join(', ')}`.trim(); +} + +// ---- Calls ----------------------------------------------------------------------------------- + +/** Runs `work` on every item, at most `limit` at once, the results in the items' order. */ +export async function mapWithLimit(items: T[], limit: number, work: (item: T) => Promise): Promise { + const results = new Array(items.length); + // One iterator shared by every worker, so each item is taken once. + const queue = items.entries(); + const worker = async (): Promise => { + for (const [index, item] of queue) { + results[index] = await work(item); + } + }; + await Promise.all(Array.from({length: Math.min(limit, items.length)}, worker)); + return results; +} + +/** One page of message ids, newest first, in a folder and matching a search, from the page a previous one pointed at. */ +export async function listMessageIds(accessToken: string, query: FolderQuery, pageToken: string | null): Promise, ApiError>> { + const params = {maxResults: String(PAGE_SIZE), labelIds: query.labelIds, q: query.q, includeSpamTrash: query.includeSpamTrash ? 'true' : null, pageToken}; + const list = await googleRequest(accessToken, gmailUrl('/messages', params), messageListSchema); + return list.map(({messages = [], nextPageToken}) => ({items: messages.map(({id}) => id), nextPage: nextPageToken ?? null})); +} + +/** One page of drafts, newest first, matching a search. */ +export async function listDrafts(accessToken: string, search: string | null, pageToken: string | null, pageSize = PAGE_SIZE): Promise, ApiError>> { + const list = await googleRequest(accessToken, gmailUrl('/drafts', {maxResults: String(pageSize), q: search, pageToken}), draftListSchema); + return list.map(({drafts = [], nextPageToken}) => ({items: drafts.map(({id, message}) => ({draftId: id, messageId: message.id})), nextPage: nextPageToken ?? null})); +} + +// How many pages of drafts are looked through for a draft's id before giving up: 5,000 drafts. +const MAX_DRAFT_PAGES = 10; + +/** + * The draft ids of messages that are drafts, by message id. Gmail's messages don't say which + * draft they are, so the drafts are listed until every one is found. + */ +export async function findDraftIds(accessToken: string, messageIds: string[]): Promise, ApiError>> { + const found = new Map(); + let pageToken: string | null = null; + for (let page = 0; page < MAX_DRAFT_PAGES && found.size < messageIds.length; page++) { + const drafts = await listDrafts(accessToken, null, pageToken, DRAFTS_PAGE_SIZE); + if (drafts.isErr()) { + return drafts; + } + for (const {draftId, messageId} of drafts.value.items) { + if (messageIds.includes(messageId)) { + found.set(messageId, draftId); + } + } + pageToken = drafts.value.nextPage; + if (!pageToken) { + break; + } + } + return Ok(found); +} + +// Waits before asking again a message Gmail answered busy, or failed on for a moment. +const RETRY_DELAYS_MS = [250, 1000]; + +const busySchema = z.object({kind: z.literal('http'), status: z.number().refine((status) => status === 429 || status >= 500)}); + +function isBusy(error: ApiError): boolean { + return error.kind === 'provider_unreachable' && busySchema.safeParse(error.cause).success; +} + +// A call asked again, twice at most, while Gmail answers it is busy, so one of a page's messages +// turned down in the burst doesn't fail the page. +async function retried(call: () => Promise>): Promise> { + let result = await call(); + for (const delay of RETRY_DELAYS_MS) { + if (result.isOk() || !isBusy(result.error)) { + return result; + } + await Bun.sleep(delay); + result = await call(); + } + return result; +} + +/** A message with what a list shows of it: its labels, date and the headers in `LIST_HEADERS`. */ +function getMessageSummary(accessToken: string, messageId: string): Promise> { + return retried(() => googleRequest(accessToken, gmailUrl(`/messages/${messageId}`, {format: 'metadata', metadataHeaders: LIST_HEADERS}), messageSchema)); +} + +/** A message's labels alone, to tell a draft from other email. */ +export function getMessageLabels(accessToken: string, messageId: string): Promise> { + return googleRequest(accessToken, gmailUrl(`/messages/${messageId}`, {format: 'minimal'}), messageSchema); +} + +/** The summaries of several messages, at most a few asked for at once, in the order of their ids. */ +export function getMessageSummaries(accessToken: string, messageIds: string[]): Promise>> { + return mapWithLimit(messageIds, CONCURRENCY, (messageId) => getMessageSummary(accessToken, messageId)); +} + +/** A message whole: its headers and every part. */ +export function getMessage(accessToken: string, messageId: string): Promise> { + return googleRequest(accessToken, gmailUrl(`/messages/${messageId}`, {format: 'full'}), messageSchema); +} + +/** Adds and removes labels on a message, as marking, starring and archiving do. */ +export function modifyMessage(accessToken: string, messageId: string, change: {add?: string[]; remove?: string[]}): Promise> { + return googleWrite(accessToken, 'POST', gmailUrl(`/messages/${messageId}/modify`), {addLabelIds: change.add ?? [], removeLabelIds: change.remove ?? []}, messageSchema); +} + +/** Moves a message to Trash, where Gmail keeps it 30 days. Nothing here deletes a message for good. */ +export function trashMessage(accessToken: string, messageId: string): Promise> { + return googleWrite(accessToken, 'POST', gmailUrl(`/messages/${messageId}/trash`), {}, messageSchema); +} + +/** Sends a message; Gmail files it in Sent itself. */ +export function sendMessage(accessToken: string, message: {raw: string; threadId?: string}): Promise> { + return googleWrite(accessToken, 'POST', gmailUrl('/messages/send'), message, messageSchema); +} + +/** A draft whole, its message with every part. */ +export function getDraft(accessToken: string, draftId: string): Promise> { + return googleRequest(accessToken, gmailUrl(`/drafts/${draftId}`, {format: 'full'}), draftSchema); +} + +export function createDraft(accessToken: string, message: {raw: string; threadId?: string}): Promise> { + return googleWrite(accessToken, 'POST', gmailUrl('/drafts'), {message}, draftSchema); +} + +/** Replaces a draft's message; the draft keeps its id, its message gets a new one. */ +export function updateDraft(accessToken: string, draftId: string, message: {raw: string; threadId?: string}): Promise> { + return googleWrite(accessToken, 'PUT', gmailUrl(`/drafts/${draftId}`), {id: draftId, message}, draftSchema); +} + +/** Deletes a draft for good, as Gmail's own Discard does. */ +export function deleteDraft(accessToken: string, draftId: string): Promise> { + return googleDelete(accessToken, gmailUrl(`/drafts/${draftId}`)); +} + +/** Sends a draft as it is saved; it leaves the drafts, and its message is filed in Sent. */ +export function sendDraft(accessToken: string, draftId: string): Promise> { + return googleWrite(accessToken, 'POST', gmailUrl('/drafts/send'), {id: draftId}, messageSchema); +} diff --git a/packages/api/src/connections/google/google-client.test.ts b/packages/api/src/connections/google/google-client.test.ts index 4bfb1eca..eec0c6c8 100644 --- a/packages/api/src/connections/google/google-client.test.ts +++ b/packages/api/src/connections/google/google-client.test.ts @@ -84,6 +84,15 @@ describe('googleError', () => { }); }); + test('a token without the scope a call takes says to sign in again', async () => { + const {googleError} = await import('./google-client'); + + expect(googleError(httpError(403, {code: 403, message: 'Request had insufficient authentication scopes.', errors: [{reason: 'insufficientPermissions'}]}))).toMatchObject({ + kind: 'validation_error', + message: "Google didn't allow this at sign-in. Sign in to Google again to allow it.", + }); + }); + test("any other refusal is passed on in Google's words", async () => { const {googleError} = await import('./google-client'); diff --git a/packages/api/src/connections/google/google-client.ts b/packages/api/src/connections/google/google-client.ts index 0c347e8b..a7ff5d8a 100644 --- a/packages/api/src/connections/google/google-client.ts +++ b/packages/api/src/connections/google/google-client.ts @@ -40,6 +40,10 @@ function readGoogleError(error: Extract): {reasons: const RATE_LIMITED = ['rateLimitExceeded', 'userRateLimitExceeded', 'RATE_LIMIT_EXCEEDED']; +// What Google says of a call whose token lacks the scope it takes, as sending a draft does with +// only `gmail.send` granted. +const SCOPE_MISSING = ['insufficientPermissions', 'ACCESS_TOKEN_SCOPE_INSUFFICIENT']; + /** What one of Google's errors means: a token to refresh, a file not there, a refusal to pass on, or Google out of reach. */ export function googleError(error: FetchError): ApiError { if (error.kind !== 'http') { @@ -66,6 +70,9 @@ export function googleError(error: FetchError): ApiError { if (reasons.some((reason) => RATE_LIMITED.includes(reason))) { return ApiErr.providerUnreachable(error); } + if (reasons.some((reason) => SCOPE_MISSING.includes(reason))) { + return ApiErr.validationError("Google didn't allow this at sign-in. Sign in to Google again to allow it."); + } // Any other refusal, such as a file the account may not open or a query Google can't read, in // Google's own words. if (message !== '') { @@ -74,8 +81,12 @@ export function googleError(error: FetchError): ApiError { return ApiErr.providerUnreachable(error); } +function authHeaders(accessToken: string): Record { + return {authorization: `Bearer ${accessToken}`, accept: 'application/json'}; +} + function authorized(accessToken: string): RequestInit { - return {headers: {authorization: `Bearer ${accessToken}`, accept: 'application/json'}}; + return {headers: authHeaders(accessToken)}; } /** A GET to one of Google's APIs, its answer read with `schema`. */ @@ -90,6 +101,19 @@ export async function googleText(accessToken: string, url: URL): Promise(accessToken: string, method: 'POST' | 'PUT', url: URL, body: object, schema: S): Promise, ApiError>> { + const init = {method, headers: {...authHeaders(accessToken), 'content-type': 'application/json'}, body: JSON.stringify(body)}; + const answer = await httpRequest(url, init, {schema, timeoutMs: TIMEOUT_MS}); + return answer.mapErr(googleError); +} + +/** A DELETE, whose answer is empty. */ +export async function googleDelete(accessToken: string, url: URL): Promise> { + const answer = await httpRequest(url, {...authorized(accessToken), method: 'DELETE'}, {timeoutMs: TIMEOUT_MS}); + return answer.mapErr(googleError); +} + /** The app's Google client, which signing in and refreshing both need. */ export function requireGoogleClient(): Result { const client = googleOAuthClient(); diff --git a/packages/api/src/connections/google/scopes.ts b/packages/api/src/connections/google/scopes.ts index 63d5870e..df944e27 100644 --- a/packages/api/src/connections/google/scopes.ts +++ b/packages/api/src/connections/google/scopes.ts @@ -10,9 +10,20 @@ import type {GoogleCredential} from './google-oauth'; const scope = (name: string): string => `https://www.googleapis.com/auth/${name}`; +// Gmail's reading, labelling, Trash and drafts all take `gmail.modify`; sending takes it too, or +// `gmail.send` alone. +const GMAIL = [scope('gmail.modify')]; + // Any one of an action's scopes is enough. Drive's export and the Docs and Sheets APIs all take // `drive`, so it alone covers reading files. const ACTION_SCOPES: Partial> = { + readEmails: GMAIL, + markEmails: GMAIL, + flagEmails: GMAIL, + archiveEmails: GMAIL, + writeEmails: GMAIL, + trashEmails: GMAIL, + sendEmails: [scope('gmail.send'), scope('gmail.modify')], readFiles: [scope('drive')], }; diff --git a/packages/api/src/connections/route.test.ts b/packages/api/src/connections/route.test.ts index 2bb170e3..25aa41a6 100644 --- a/packages/api/src/connections/route.test.ts +++ b/packages/api/src/connections/route.test.ts @@ -2,6 +2,7 @@ import {beforeEach, describe, expect, mock, spyOn, test} from 'bun:test'; import {Err, Ok, type Result} from 'ts-results-es'; import {ApiErr, type ApiError} from '@proxy/utils'; +import {findIntegration} from '@proxy/integrations'; const listConnectionsWithCredential = mock(); const getConnectionWithCredential = mock(); @@ -101,9 +102,10 @@ describe('handleListConnectionsRoute', () => { const {handleListConnectionsRoute} = await import('./route'); const {connections} = (await handleListConnectionsRoute(makeRequest())).unwrap(); + const every = findIntegration('google')?.actions.map((action) => action.id); expect(connections.map((connection) => [connection.id, connection.unavailable])).toEqual([ ['conn-1', []], - ['conn-2', ['readFiles']], + ['conn-2', every ?? []], ]); }); diff --git a/packages/api/src/connections/unavailable.test.ts b/packages/api/src/connections/unavailable.test.ts index d6b9320f..d51c4d7d 100644 --- a/packages/api/src/connections/unavailable.test.ts +++ b/packages/api/src/connections/unavailable.test.ts @@ -44,6 +44,14 @@ describe('unavailableActions', () => { expect(unavailableActions(google(ALL_SCOPES.filter((scope) => !scope.endsWith('/drive'))))).toEqual(['readFiles']); }); + test('a Google sign-in with only gmail.send leaves every Gmail action but Send out', async () => { + const {unavailableActions} = await import('./unavailable'); + + const withoutModify = ALL_SCOPES.filter((scope) => !scope.endsWith('/gmail.modify')); + expect(unavailableActions(google(withoutModify))).toEqual(['readEmails', 'markEmails', 'flagEmails', 'archiveEmails', 'writeEmails', 'trashEmails']); + expect(unavailableActions(google(withoutModify.filter((scope) => !scope.endsWith('/gmail.send'))))).toContain('sendEmails'); + }); + test('every other integration covers all its actions, whatever its credential', async () => { const {unavailableActions} = await import('./unavailable'); diff --git a/packages/api/src/records/google-connector.ts b/packages/api/src/records/google-connector.ts index 1e95a78c..5dcbb04d 100644 --- a/packages/api/src/records/google-connector.ts +++ b/packages/api/src/records/google-connector.ts @@ -3,26 +3,47 @@ import {Err, type Result} from 'ts-results-es'; import {ApiErr, type ApiError} from '@proxy/utils'; import type {Connector, RecordTarget} from './connector'; +import {gmailConnector} from './google-emails'; import {getFilesRecord, listFilesTree} from './google-files'; /** * A Google account, fetched live from Google's APIs with the connection's tokens, one collection - * per product: `files`, Drive with Docs, Sheets and Slides (`google-files.ts`). Nothing is written - * yet: the collections offer no writes, and the connector turns any down. + * per product: `emails`, Gmail (`google-emails.ts`), and `files`, Drive with Docs, Sheets and + * Slides (`google-files.ts`). Files aren't written yet: that collection offers no writes, and the + * connector turns any down. */ -function unknownCollection(target: RecordTarget): Promise> { - return Promise.resolve(Err(ApiErr.notFound('collection', target.collection.id))); -} - function notWritten(): Promise> { return Promise.resolve(Err(ApiErr.validationError('Files in Google Drive can’t be changed here yet'))); } -export const googleConnector: Connector = { - list: (target, query) => (target.collection.id === 'files' ? listFilesTree(target, query) : unknownCollection(target)), - get: (target, recordId) => (target.collection.id === 'files' ? getFilesRecord(target, recordId) : unknownCollection(target)), +const filesConnector: Connector = { + list: listFilesTree, + get: getFilesRecord, create: notWritten, update: notWritten, remove: notWritten, }; + +const COLLECTIONS: Partial> = {emails: gmailConnector, files: filesConnector}; + +function unknownCollection(target: RecordTarget): Promise> { + return Promise.resolve(Err(ApiErr.notFound('collection', target.collection.id))); +} + +const unknownConnector: Connector = {list: unknownCollection, get: unknownCollection, create: unknownCollection, update: unknownCollection, remove: unknownCollection}; + +function connectorOf(target: RecordTarget): Connector { + return COLLECTIONS[target.collection.id] ?? unknownConnector; +} + +export const googleConnector: Connector = { + list: (target, query) => connectorOf(target).list(target, query), + get: (target, recordId) => connectorOf(target).get(target, recordId), + create: (target, values, parent) => connectorOf(target).create(target, values, parent), + update: (target, recordId, values) => connectorOf(target).update(target, recordId, values), + remove: (target, recordId) => connectorOf(target).remove(target, recordId), + // Only email has commands. + commands: gmailConnector.commands, + newCommands: gmailConnector.newCommands, +}; diff --git a/packages/api/src/records/google-emails.test.ts b/packages/api/src/records/google-emails.test.ts new file mode 100644 index 00000000..cfed4b0b --- /dev/null +++ b/packages/api/src/records/google-emails.test.ts @@ -0,0 +1,435 @@ +import {beforeEach, describe, expect, mock, spyOn, test} from 'bun:test'; +import {Ok, type Result} from 'ts-results-es'; + +import type {ApiError} from '@proxy/utils'; +import {findCollection, findIntegration, type Collection} from '@proxy/integrations'; + +mock.module('@proxy/db/connection', () => ({updateConnectionCredential: mock(), getConnectionCredential: mock(async () => Ok(null))})); + +mock.module('../observability/log', () => ({ + log: {debug: mock(), info: mock(), warn: mock(), error: mock()}, + serializeError: mock(() => ({})), +})); + +mock.module('../utils/env', () => ({env: {NODE_ENV: 'production', APP_URL: 'https://app.example.com', GOOGLE_CLIENT_ID: 'client-1', GOOGLE_CLIENT_SECRET: 'secret-1'}})); + +mock.module('../utils/secret-crypto', () => ({ + encryptSecret: (plaintext: string): Result => Ok(`enc:${plaintext}`), + decryptSecret: (stored: string): Result => Ok(stored.replace(/^enc:/, '')), +})); + +function emails(): Collection { + const integration = findIntegration('google'); + const found = integration && findCollection(integration, 'emails'); + if (!found) { + throw new Error('No emails collection'); + } + return found; +} + +const CREDENTIAL = {accessToken: 'access-1', refreshToken: 'refresh-1', expiresAt: new Date(Date.now() + 3_600_000).toISOString(), scopes: [], email: 'alex@example.com'}; + +const target = { + connection: {id: 'google-1', integrationId: 'google', account: 'alex@example.com', createdAt: new Date(), tools: null, name: null, defaults: [], credential: `enc:${JSON.stringify(CREDENTIAL)}`}, + collection: emails(), +}; + +const query = (changes: object = {}) => ({search: null, page: null, filter: null, parent: null, ...changes}); + +const base64url = (text: string): string => Buffer.from(text).toString('base64url'); + +type Header = {name: string; value: string}; +type Part = {mimeType: string; filename: string; headers?: Header[]; body: {size: number; data?: string; attachmentId?: string}; parts?: Part[]}; +type Message = {id: string; threadId: string; labelIds: string[]; internalDate: string; payload: Part}; + +function message(id: string, labelIds: string[], headers: Record, parts: Part[] = []): Message { + return { + id, + threadId: `thread-${id}`, + labelIds, + internalDate: String(Date.parse('2026-10-01T09:00:00Z')), + payload: {mimeType: 'multipart/alternative', filename: '', headers: Object.entries(headers).map(([name, value]) => ({name, value})), body: {size: 0}, parts}, + }; +} + +const text = (body: string): Part => ({ + mimeType: 'text/plain', + filename: '', + headers: [{name: 'Content-Type', value: 'text/plain; charset="UTF-8"'}], + body: {size: body.length, data: base64url(body)}, +}); +const html = (body: string): Part => ({mimeType: 'text/html', filename: '', headers: [{name: 'Content-Type', value: 'text/html; charset="UTF-8"'}], body: {size: body.length, data: base64url(body)}}); + +// A mailbox as Gmail answers it: an unread email from Sam in the inbox, one sent, and a draft. +function mailbox(): Record { + return { + m1: message('m1', ['UNREAD', 'IMPORTANT', 'INBOX'], {Subject: 'Q3 planning', From: '"Sam Lee" ', To: 'Alex Kim ', Date: 'Thu, 1 Oct 2026 09:00:00 +0000'}, [ + text('Agenda attached.\n'), + html('

Agenda attached.

'), + ]), + m2: message('m2', ['SENT'], {Subject: 'Re: dinner', From: 'Alex Kim ', To: 'Robin '}, [html('

See you Friday at 8.

')]), + m3: message( + 'm3', + ['DRAFT'], + {Subject: 'Offer', From: 'alex@example.com', To: 'Robin ', Cc: 'sam@example.com', 'In-Reply-To': '', References: ''}, + [text('Draft text')], + ), + }; +} + +type Call = {method: string; url: URL; body: Record | null}; + +const gmail = { + calls: [] as Call[], + messages: mailbox(), + drafts: {r1: 'm3'} as Record, + answer: (_call: Call): Response | null => null, +}; + +function notFound(): Response { + return Response.json({error: {code: 404, message: 'Requested entity was not found.', errors: [{reason: 'notFound'}]}}, {status: 404}); +} + +function draftAnswer(draftId: string): Response { + const messageId = gmail.drafts[draftId]; + const found = messageId ? gmail.messages[messageId] : undefined; + return found ? Response.json({id: draftId, message: found}) : notFound(); +} + +function answer(call: Call): Response { + const custom = gmail.answer(call); + if (custom) { + return custom; + } + const path = call.url.pathname.replace('/gmail/v1/users/me', ''); + if (path === '/messages' && call.method === 'GET') { + return Response.json({messages: Object.keys(gmail.messages).map((id) => ({id, threadId: `thread-${id}`})), nextPageToken: 'page-2', resultSizeEstimate: 3}); + } + if (path === '/drafts' && call.method === 'GET') { + return Response.json({drafts: Object.entries(gmail.drafts).map(([id, messageId]) => ({id, message: {id: messageId, threadId: `thread-${messageId}`}}))}); + } + if (path === '/drafts' && call.method === 'POST') { + gmail.drafts.r2 = 'm4'; + gmail.messages.m4 = message('m4', ['DRAFT'], {Subject: 'New', From: 'alex@example.com', To: 'robin@example.com'}, [text('Hello')]); + return Response.json({id: 'r2', message: {id: 'm4', threadId: 'thread-m4', labelIds: ['DRAFT']}}); + } + if (path === '/drafts/send') { + return Response.json({id: 'm9', threadId: 'thread-m9', labelIds: ['SENT']}); + } + if (path === '/messages/send') { + return Response.json({id: 'm8', threadId: 'thread-m8', labelIds: ['SENT']}); + } + const draft = /^\/drafts\/([^/]+)$/.exec(path)?.[1]; + if (draft && call.method === 'DELETE') { + return gmail.drafts[draft] ? new Response(null, {status: 204}) : notFound(); + } + if (draft) { + return draftAnswer(draft); + } + const [, id, verb] = /^\/messages\/([^/]+)(?:\/(modify|trash))?$/.exec(path) ?? []; + const found = id ? gmail.messages[id] : undefined; + if (!found) { + return notFound(); + } + if (verb === 'modify') { + const {addLabelIds = [], removeLabelIds = []} = call.body as {addLabelIds?: string[]; removeLabelIds?: string[]}; + found.labelIds = [...found.labelIds.filter((label) => !removeLabelIds.includes(label)), ...addLabelIds]; + } + if (verb === 'trash') { + found.labelIds = [...found.labelIds, 'TRASH']; + } + return Response.json(found); +} + +spyOn(globalThis, 'fetch').mockImplementation((async (input: string | URL | Request, init?: RequestInit) => { + const url = new URL(typeof input === 'string' ? input : input instanceof URL ? input.href : input.url); + const body = typeof init?.body === 'string' ? (JSON.parse(init.body) as Record) : null; + const call = {method: init?.method ?? 'GET', url, body}; + gmail.calls.push(call); + return answer(call); +}) as typeof fetch); + +function calls(method: string, path: string): Call[] { + return gmail.calls.filter((call) => call.method === method && call.url.pathname === `/gmail/v1/users/me${path}`); +} + +// The RFC 2822 text a raw message holds, its headers unfolded. +function rawText(body: Record | null | undefined): string { + const raw = (body?.raw as string | undefined) ?? ''; + return Buffer.from(raw, 'base64url') + .toString('utf8') + .replace(/\r\n[ \t]+/g, ' '); +} + +beforeEach(() => { + gmail.calls = []; + gmail.messages = mailbox(); + gmail.drafts = {r1: 'm3'}; + gmail.answer = () => null; +}); + +async function connector() { + const {googleConnector} = await import('./google-connector'); + return googleConnector; +} + +describe('list', () => { + test('a folder is its label, the search Gmail’s own, from the page token given, each message read for its headers', async () => { + const page = (await (await connector()).list(target, query({filter: 'Inbox', search: 'from:sam', page: 'page-1'}))).unwrap(); + + const [list] = calls('GET', '/messages'); + expect(Object.fromEntries(list?.url.searchParams ?? [])).toEqual({maxResults: '50', labelIds: 'INBOX', q: 'from:sam', pageToken: 'page-1'}); + expect(calls('GET', '/messages/m1')[0]?.url.searchParams.getAll('metadataHeaders')).toEqual(['Subject', 'From', 'To']); + expect(calls('GET', '/messages/m1')[0]?.url.searchParams.get('format')).toBe('metadata'); + expect(page.nextPage).toBe('page-2'); + expect(page.records[0]).toEqual({ + id: 'm1', + values: {folder: 'Inbox', from: 'Sam Lee ', to: 'Alex Kim ', subject: 'Q3 planning', date: '2026-10-01T09:00:00.000Z', status: 'Unread'}, + updatedAt: '2026-10-01T09:00:00.000Z', + }); + }); + + test('without a folder, every email, a draft by its draft id with its addresses bare', async () => { + const page = (await (await connector()).list(target, query())).unwrap(); + + expect(calls('GET', '/messages')[0]?.url.searchParams.has('labelIds')).toBe(false); + expect(page.records.map((record) => [record.id, record.values.folder])).toEqual([ + ['m1', 'Inbox'], + ['m2', 'Sent'], + ['draft-r1', 'Draft'], + ]); + expect(page.records[2]?.values.to).toBe('robin@example.com'); + expect(page.records[1]?.values.status).toBe(''); + }); + + test('the archive is a search, Trash and Spam asked for by their labels', async () => { + const c = await connector(); + await c.list(target, query({filter: 'Archive', search: 'invoice'})); + await c.list(target, query({filter: 'Trash'})); + + const [archive, trash] = calls('GET', '/messages'); + expect(archive?.url.searchParams.get('q')).toBe('-in:inbox -in:sent -in:drafts -in:trash -in:spam (invoice)'); + expect(archive?.url.searchParams.has('labelIds')).toBe(false); + expect(trash?.url.searchParams.get('labelIds')).toBe('TRASH'); + expect(trash?.url.searchParams.get('includeSpamTrash')).toBe('true'); + }); + + test('drafts are listed as drafts, searched and paged by Gmail', async () => { + gmail.answer = (call) => (call.url.pathname.endsWith('/drafts') ? Response.json({drafts: [{id: 'r1', message: {id: 'm3', threadId: 'thread-m3'}}], nextPageToken: 'drafts-2'}) : null); + const page = (await (await connector()).list(target, query({filter: 'Draft', search: 'offer'}))).unwrap(); + + expect(Object.fromEntries(calls('GET', '/drafts')[0]?.url.searchParams ?? [])).toEqual({maxResults: '50', q: 'offer'}); + expect(calls('GET', '/messages')).toEqual([]); + expect(page.records.map((record) => record.id)).toEqual(['draft-r1']); + expect(page.nextPage).toBe('drafts-2'); + }); + + test('a message Gmail turns down as busy is asked for again, so the page still lists it', async () => { + let busy = 1; + gmail.answer = (call) => { + if (call.url.pathname.endsWith('/messages/m2') && busy > 0) { + busy -= 1; + return Response.json({error: {code: 429, message: 'Too many concurrent requests for user.', errors: [{reason: 'rateLimitExceeded'}]}}, {status: 429}); + } + return null; + }; + const page = (await (await connector()).list(target, query())).unwrap(); + + expect(page.records.map((record) => record.id)).toEqual(['m1', 'm2', 'draft-r1']); + expect(calls('GET', '/messages/m2')).toHaveLength(2); + }); + + test('a draft moved to Trash is not looked for among the drafts', async () => { + gmail.messages.m3 = {...message('m3', ['DRAFT', 'TRASH'], {Subject: 'Offer'}), threadId: 'thread-m3'}; + const page = (await (await connector()).list(target, query({filter: 'Trash'}))).unwrap(); + + expect(page.records.find((record) => record.id === 'm3')?.values.folder).toBe('Trash'); + expect(calls('GET', '/drafts')).toEqual([]); + }); + + test('the last page has no next one, and an email gone before it was read is left out', async () => { + gmail.answer = (call) => (call.url.pathname.endsWith('/messages') ? Response.json({messages: [{id: 'm1'}, {id: 'gone'}]}) : null); + const page = (await (await connector()).list(target, query({page: 'page-2'}))).unwrap(); + + expect(page.records.map((record) => record.id)).toEqual(['m1']); + expect(page.nextPage).toBeNull(); + }); +}); + +describe('get', () => { + test('an email opens with its text part', async () => { + const record = (await (await connector()).get(target, 'm1')).unwrap(); + + expect(calls('GET', '/messages/m1')[0]?.url.searchParams.get('format')).toBe('full'); + expect(record.values.body).toBe('Agenda attached.'); + expect(record.values.subject).toBe('Q3 planning'); + }); + + test('an email with only HTML opens as its text', async () => { + const record = (await (await connector()).get(target, 'm2')).unwrap(); + + expect(record.values.body).toBe('See you Friday at 8.'); + expect(record.values.folder).toBe('Sent'); + }); + + test('a draft opens by its draft id, and by its message id is given its draft id', async () => { + const c = await connector(); + const byDraft = (await c.get(target, 'draft-r1')).unwrap(); + const byMessage = (await c.get(target, 'm3')).unwrap(); + + expect(byDraft.id).toBe('draft-r1'); + expect(byDraft.values).toMatchObject({folder: 'Draft', to: 'robin@example.com', body: 'Draft text'}); + expect(byMessage.id).toBe('draft-r1'); + }); + + test('an email Gmail doesn’t have is not found, and an id that isn’t Gmail’s never reaches it', async () => { + const c = await connector(); + + expect((await c.get(target, 'missing')).unwrapErr()).toMatchObject({kind: 'not_found', resource: 'record', id: 'missing'}); + expect((await c.get(target, '../labels')).unwrapErr().kind).toBe('not_found'); + expect(calls('GET', '/messages/../labels')).toEqual([]); + expect(gmail.calls.map((call) => call.url.pathname)).toEqual(['/gmail/v1/users/me/messages/missing']); + }); +}); + +describe('drafts', () => { + test('a new draft is written from the connection’s address and read back by its draft id', async () => { + const record = (await (await connector()).create(target, {to: 'robin@example.com', subject: 'New', body: 'Hello'}, null)).unwrap(); + + const raw = rawText(calls('POST', '/drafts')[0]?.body?.message as Record); + expect(raw).toContain('From: alex@example.com'); + expect(raw).toContain('To: robin@example.com'); + expect(raw).toContain('Subject: New'); + expect(raw).toContain('Hello'); + expect(record).toMatchObject({id: 'draft-r2', values: {folder: 'Draft', subject: 'New'}}); + }); + + test('an edit writes the fields given over the rest, keeping the draft’s Cc and thread', async () => { + const record = (await (await connector()).update(target, 'draft-r1', {subject: 'Offer, revised'})).unwrap(); + + const [put] = calls('PUT', '/drafts/r1'); + const sent = put?.body as {id: string; message: {threadId: string}}; + expect(sent.id).toBe('r1'); + expect(sent.message.threadId).toBe('thread-m3'); + const raw = rawText(sent.message); + expect(raw).toContain('Subject: Offer, revised'); + expect(raw).toContain('To: Robin '); + expect(raw).toContain('Cc: sam@example.com'); + expect(raw).toContain('In-Reply-To: '); + expect(raw).toContain('Draft text'); + expect(record.id).toBe('draft-r1'); + }); + + test('an edit with every field, as the agent side sends one, keeps the names and the HTML of what is unchanged', async () => { + const m3 = gmail.messages.m3; + m3?.payload.parts?.push(html('

Draft text

')); + await (await connector()).update(target, 'draft-r1', {to: 'robin@example.com', subject: 'Offer, revised', body: 'Draft text'}); + + const raw = rawText((calls('PUT', '/drafts/r1')[0]?.body as {message: Record}).message); + expect(raw).toContain('To: Robin '); + expect(raw).toContain('Content-Type: text/html'); + expect(raw).toContain('

Draft text

'); + }); + + test('an edit to the text writes it as text alone, and to the recipients as written', async () => { + const m3 = gmail.messages.m3; + m3?.payload.parts?.push(html('

Draft text

')); + await (await connector()).update(target, 'draft-r1', {to: 'sam@example.com', subject: 'Offer', body: 'New text'}); + + const raw = rawText((calls('PUT', '/drafts/r1')[0]?.body as {message: Record}).message); + expect(raw).toContain('To: sam@example.com'); + expect(raw).toContain('New text'); + expect(raw).not.toContain('text/html'); + }); + + test('a draft with attachments isn’t edited, and an email that isn’t a draft never is', async () => { + const c = await connector(); + const m3 = gmail.messages.m3; + m3?.payload.parts?.push({mimeType: 'application/pdf', filename: 'offer.pdf', body: {size: 10, attachmentId: 'a1'}}); + + expect((await c.update(target, 'draft-r1', {subject: 'x'})).unwrapErr()).toMatchObject({kind: 'validation_error', message: expect.stringContaining('attachments')}); + expect((await c.update(target, 'm1', {subject: 'x'})).unwrapErr()).toMatchObject({kind: 'validation_error'}); + expect((await c.remove(target, 'm1')).unwrapErr()).toMatchObject({kind: 'validation_error'}); + expect(calls('PUT', '/drafts/r1')).toEqual([]); + }); + + test('deleting a draft deletes it at Gmail, and a missing one is not found', async () => { + const c = await connector(); + + expect((await c.remove(target, 'draft-r1')).isOk()).toBe(true); + expect(calls('DELETE', '/drafts/r1')).toHaveLength(1); + expect((await c.remove(target, 'draft-r9')).unwrapErr().kind).toBe('not_found'); + }); +}); + +describe('commands', () => { + test.each([ + ['markRead', {addLabelIds: [], removeLabelIds: ['UNREAD']}], + ['markUnread', {addLabelIds: ['UNREAD'], removeLabelIds: []}], + ['flag', {addLabelIds: ['STARRED'], removeLabelIds: []}], + ['unflag', {addLabelIds: [], removeLabelIds: ['STARRED']}], + ['archive', {addLabelIds: [], removeLabelIds: ['INBOX']}], + ])('%s changes the labels and reads the email back', async (command, change) => { + const run = (await connector()).commands?.[command]; + const record = (await run?.(target, 'm1'))?.unwrap(); + + expect(calls('POST', '/messages/m1/modify')[0]?.body).toEqual(change); + expect(record?.id).toBe('m1'); + }); + + test('starring shows in the status, archiving moves the email to the archive', async () => { + const c = await connector(); + + expect((await c.commands?.flag?.(target, 'm1'))?.unwrap()?.values.status).toBe('Unread, Starred'); + expect((await c.commands?.archive?.(target, 'm1'))?.unwrap()?.values).toMatchObject({folder: 'Archive', status: ''}); + }); + + test('moving to Trash is Gmail’s trash, never a delete', async () => { + const record = (await (await connector()).commands?.trash?.(target, 'm1'))?.unwrap(); + + expect(calls('POST', '/messages/m1/trash')).toHaveLength(1); + expect(gmail.calls.some((call) => call.method === 'DELETE')).toBe(false); + expect(record?.values.folder).toBe('Trash'); + }); + + test('a command on a missing email is not found', async () => { + expect((await (await connector()).commands?.markRead?.(target, 'gone'))?.unwrapErr().kind).toBe('not_found'); + }); + + test('sending a draft sends it as saved', async () => { + const sent = await (await connector()).commands?.send?.(target, 'draft-r1'); + + expect(sent?.unwrap()).toBeNull(); + expect(calls('POST', '/drafts/send')[0]?.body).toEqual({id: 'r1'}); + }); + + test('a draft named by its message id is sent by its draft id', async () => { + const sent = await (await connector()).commands?.send?.(target, 'm3'); + + expect(sent?.unwrap()).toBeNull(); + expect(calls('POST', '/drafts/send')[0]?.body).toEqual({id: 'r1'}); + expect(((await (await connector()).commands?.send?.(target, 'm1'))?.unwrapErr() as {message?: string}).message).toBe('Only drafts can be sent'); + }); + + test('a draft without a recipient isn’t sent', async () => { + gmail.messages.m3 = message('m3', ['DRAFT'], {Subject: 'Offer', From: 'alex@example.com'}, [text('Draft text')]); + const sent = await (await connector()).commands?.send?.(target, 'draft-r1'); + + expect(sent?.unwrapErr()).toMatchObject({kind: 'validation_error', message: 'The email has no recipient'}); + expect(calls('POST', '/drafts/send')).toEqual([]); + }); + + test('a new email goes out from the connection’s address', async () => { + const c = await connector(); + const sent = await c.newCommands?.sendNew?.(target, {to: 'robin@example.com, sam@example.com', subject: 'Hello there', body: 'Line one'}); + + expect(sent?.unwrap()).toBeNull(); + const raw = rawText(calls('POST', '/messages/send')[0]?.body); + expect(raw).toContain('From: alex@example.com'); + expect(raw).toContain('To: robin@example.com, sam@example.com'); + expect(raw).toContain('Subject: Hello there'); + expect(raw).toMatch(/Message-ID: <[0-9a-f-]+@proxy>/); + expect(raw).toContain('Line one'); + expect((await c.newCommands?.sendNew?.(target, {to: ' ', subject: 'x', body: 'y'}))?.unwrapErr().kind).toBe('validation_error'); + }); +}); diff --git a/packages/api/src/records/google-emails.ts b/packages/api/src/records/google-emails.ts new file mode 100644 index 00000000..60399464 --- /dev/null +++ b/packages/api/src/records/google-emails.ts @@ -0,0 +1,378 @@ +import {randomUUID} from 'node:crypto'; + +import addressparser from 'nodemailer/lib/addressparser'; +import MailComposer from 'nodemailer/lib/mail-composer'; +import type Mail from 'nodemailer/lib/mailer'; +import {Err, Ok, Result} from 'ts-results-es'; + +import {ApiErr, type ApiError, Do, requirePresent} from '@proxy/utils'; + +import {callGoogle} from '../connections/google/google-client'; +import { + attachmentNames, + createDraft, + deleteDraft, + findDraftIds, + folderOf, + folderQuery, + getDraft, + getMessage, + getMessageLabels, + getMessageSummaries, + GMAIL_ID, + headerOf, + listDrafts, + listMessageIds, + messageBody, + messageHtml, + modifyMessage, + rawMessage, + sendDraft, + sendMessage, + trashMessage, + updateDraft, + type GmailMessage, + type MessagePart, +} from '../connections/google/gmail-api'; +import type {CommandRunner, Connector, DataRecord, ListQuery, RecordPage, RecordTarget, RecordValues} from './connector'; + +/** + * Gmail as the `emails` collection, fetched live, the same email as any mailbox's: one list, 50 to + * a page with Gmail's own page token, newest first. Without a folder it is all mail but Spam and + * Trash, as Gmail's All Mail is; a folder is one of Gmail's labels (Inbox, Draft, Sent, Trash, + * Spam), or for the archive what has none of them. A search is Gmail's own, as in its search box. + * + * A record's id is Gmail's message id, except a draft's, which is `draft-`: Gmail edits, + * deletes and sends drafts by their draft id, which stays as the draft is edited while its message + * id doesn't. The folder needn't be in the id, since an email's labels say which it is in. + * + * Marking read or unread takes the UNREAD label off or puts it on, starring the STARRED label, + * archiving takes INBOX off, and moving to Trash is Gmail's trash, which keeps the email 30 days. + * Nothing is deleted for good but drafts. Email goes out from the connection's own address, a draft + * as it is saved or a new one written here; Gmail files it in Sent. + */ + +type EmailRef = {kind: 'message'; messageId: string} | {kind: 'draft'; draftId: string}; + +function parseRef(recordId: string): EmailRef | null { + const draft = /^draft-([A-Za-z0-9_-]+)$/.exec(recordId); + if (draft?.[1]) { + return {kind: 'draft', draftId: draft[1]}; + } + if (GMAIL_ID.test(recordId)) { + return {kind: 'message', messageId: recordId}; + } + return null; +} + +function requireRef(recordId: string): Result { + const ref = parseRef(recordId); + if (!ref) { + return Err(ApiErr.notFound('record', recordId)); + } + return Ok(ref); +} + +// Drafts are edited, deleted and sent, by their draft ids; every other email only marked, starred +// or moved. A draft listed by its message id, when its draft id wasn't found among the drafts +// Gmail lists, is looked for again. +function requireDraft(token: string, recordId: string, refusal: string): Promise> { + return Do(async ($) => { + const ref = $(requireRef(recordId)); + if (ref.kind === 'draft') { + return ref.draftId; + } + const message = $((await getMessageLabels(token, ref.messageId)).mapErr(notFoundAs(recordId))); + if (!isDraft(message)) { + return $(Err(ApiErr.validationError(refusal))); + } + const draftId = $(await findDraftIds(token, [message.id])).get(message.id); + return $(requirePresent(draftId, ApiErr.validationError(refusal))); + }); +} + +function requireMessage(recordId: string): Result { + return requireRef(recordId).andThen((ref) => (ref.kind === 'message' ? Ok(ref.messageId) : Err(ApiErr.validationError('Only for Inbox')))); +} + +// Gmail says an id it never made is invalid rather than missing. +function notFoundAs(recordId: string) { + return (error: ApiError): ApiError => { + if (error.kind === 'not_found' || (error.kind === 'validation_error' && error.message.includes('Invalid id value'))) { + return ApiErr.notFound('record', recordId); + } + return error; + }; +} + +// Received email keeps the sender's name; a draft's addresses are bare, as they are typed. +function formatAddresses(header: string, withNames: boolean): string { + return addressparser(header, {flatten: true}) + .map(({name, address}) => (withNames && name && address ? `${name} <${address}>` : address || name)) + .filter(Boolean) + .join(', '); +} + +// "Unread, Starred", for an email in the inbox. +function status(labelIds: string[]): string { + const seen = labelIds.includes('UNREAD') ? 'Unread' : 'Read'; + if (labelIds.includes('STARRED')) { + return `${seen}, Starred`; + } + return seen; +} + +function dateOf(message: GmailMessage): string { + return message.internalDate ? new Date(Number(message.internalDate)).toISOString() : ''; +} + +function summaryValues(message: GmailMessage): RecordValues { + const labelIds = message.labelIds ?? []; + const folder = folderOf(labelIds); + const withNames = folder !== 'Draft'; + return { + folder, + from: formatAddresses(headerOf(message.payload, 'From'), withNames), + to: formatAddresses(headerOf(message.payload, 'To'), withNames), + subject: headerOf(message.payload, 'Subject'), + date: dateOf(message), + status: folder === 'Inbox' ? status(labelIds) : '', + }; +} + +function toRecord(recordId: string, message: GmailMessage, values: RecordValues = summaryValues(message)): DataRecord { + return {id: recordId, values, updatedAt: dateOf(message) || new Date(0).toISOString()}; +} + +function fullRecord(recordId: string, message: GmailMessage): DataRecord { + return toRecord(recordId, message, {...summaryValues(message), body: messageBody(message.payload)}); +} + +// A draft in the drafts, not one moved to Trash, which Gmail doesn't list among them. +function isDraft(message: GmailMessage): boolean { + return folderOf(message.labelIds ?? []) === 'Draft'; +} + +// A message a list found but that was gone by the time it was opened is left out. +function present(messages: Array>): Result, ApiError> { + const kept: Array = []; + for (const message of messages) { + if (message.isErr() && message.error.kind !== 'not_found') { + return message; + } + kept.push(message.isOk() ? message.value : null); + } + return Ok(kept); +} + +/** A page of drafts, by their draft ids, matching the search. */ +function listDraftsPage(token: string, query: ListQuery): Promise> { + return Do(async ($) => { + const page = $(await listDrafts(token, query.search, query.page)); + const messages = $( + present( + await getMessageSummaries( + token, + page.items.map(({messageId}) => messageId), + ), + ), + ); + const records = page.items.flatMap(({draftId}, index) => { + const message = messages[index]; + return message ? [toRecord(`draft-${draftId}`, message)] : []; + }); + return {records, nextPage: page.nextPage}; + }); +} + +/** A page of email in a folder, or all of it, matching the search; drafts in it by their draft ids. */ +function listPage(target: RecordTarget, query: ListQuery): Promise> { + return callGoogle(target.connection, (token) => + Do(async ($) => { + if (query.filter === 'Draft') { + return $(await listDraftsPage(token, query)); + } + const page = $(await listMessageIds(token, folderQuery(query.filter, query.search), query.page)); + const messages = $(present(await getMessageSummaries(token, page.items))).flatMap((message) => (message ? [message] : [])); + const drafts = messages.filter(isDraft).map(({id}) => id); + const draftIds = drafts.length > 0 ? $(await findDraftIds(token, drafts)) : new Map(); + const records = messages.map((message) => { + const draftId = draftIds.get(message.id); + return toRecord(draftId ? `draft-${draftId}` : message.id, message); + }); + return {records, nextPage: page.nextPage}; + }), + ); +} + +function readDraft(token: string, recordId: string, draftId: string): Promise> { + return Do(async ($) => { + const draft = $((await getDraft(token, draftId)).mapErr(notFoundAs(recordId))); + return fullRecord(`draft-${draft.id}`, draft.message); + }); +} + +// A message opened by its message id that turns out to be a draft is given its draft's id. +function readMessage(token: string, recordId: string, messageId: string): Promise> { + return Do(async ($) => { + const message = $((await getMessage(token, messageId)).mapErr(notFoundAs(recordId))); + const draftId = isDraft(message) ? $(await findDraftIds(token, [message.id])).get(message.id) : undefined; + return fullRecord(draftId ? `draft-${draftId}` : message.id, message); + }); +} + +function readRecord(token: string, recordId: string): Promise> { + return Do(async ($) => { + const ref = $(requireRef(recordId)); + if (ref.kind === 'draft') { + return $(await readDraft(token, recordId, ref.draftId)); + } + return $(await readMessage(token, recordId, ref.messageId)); + }); +} + +// What a draft keeps of itself when it is written again, beyond the fields: who else it goes to, +// and the email it answers, so a reply stays in its thread. +type Kept = {cc: string; bcc: string; inReplyTo: string; references: string}; + +function keptOf(payload: MessagePart | undefined): Kept { + return {cc: headerOf(payload, 'Cc'), bcc: headerOf(payload, 'Bcc'), inReplyTo: headerOf(payload, 'In-Reply-To'), references: headerOf(payload, 'References')}; +} + +/** + * An email from the connection's address, as the IMAP connector writes one, as RFC 2822 text, with + * an HTML version beside the text where one is kept. + */ +async function compose(from: string, values: RecordValues, kept: Kept | null, html: string | null = null): Promise> { + const options: Mail.Options = { + from, + to: values.to, + subject: values.subject, + text: values.body, + ...(html !== null ? {html} : {}), + messageId: `<${randomUUID()}@proxy>`, + ...(kept?.cc ? {cc: kept.cc} : {}), + ...(kept?.bcc ? {bcc: kept.bcc} : {}), + ...(kept?.inReplyTo ? {inReplyTo: kept.inReplyTo} : {}), + ...(kept?.references ? {references: kept.references} : {}), + }; + const message = new MailComposer(options).compile(); + // A draft keeps its Bcc until it is sent; Gmail takes it out then. + message.keepBcc = true; + return (await Result.wrapAsync(() => message.build())).mapErr((cause) => ApiErr.internalError(cause)); +} + +function createDraftRecord(target: RecordTarget, values: RecordValues): Promise> { + return callGoogle(target.connection, (token) => + Do(async ($) => { + const source = $(await compose(target.connection.account, values, null)); + const draft = $(await createDraft(token, rawMessage(source, null))); + return $(await readDraft(token, `draft-${draft.id}`, draft.id)); + }), + ); +} + +/** + * Writes a draft again with the fields given, the rest as they were, keeping its Cc, Bcc and + * thread. A draft with attachments is refused: written again here, it would lose them. Recipients + * written as they read, bare, keep their names, and the HTML version, as Gmail's own editor + * writes one, is kept while the text is as it reads; text written differently goes out as text. + */ +function updateDraftRecord(target: RecordTarget, recordId: string, values: RecordValues): Promise> { + return callGoogle(target.connection, (token) => + Do(async ($) => { + const draftId = $(await requireDraft(token, recordId, 'Only drafts can be edited or deleted')); + const draft = $((await getDraft(token, draftId)).mapErr(notFoundAs(recordId))); + const {payload, threadId} = draft.message; + if (attachmentNames(payload).length > 0) { + return $(Err(ApiErr.validationError('This draft has attachments, which editing it here would drop; edit it in Gmail'))); + } + const current = {to: headerOf(payload, 'To'), subject: headerOf(payload, 'Subject'), body: messageBody(payload)}; + const written = {...current, ...values}; + const to = written.to === formatAddresses(current.to, false) ? current.to : written.to; + const html = written.body === current.body ? messageHtml(payload) : null; + const source = $(await compose(target.connection.account, {...written, to}, keptOf(payload), html)); + $(await updateDraft(token, draftId, rawMessage(source, threadId ?? null))); + return $(await readDraft(token, recordId, draftId)); + }), + ); +} + +function removeDraft(target: RecordTarget, recordId: string): Promise> { + return callGoogle(target.connection, (token) => + Do(async ($) => { + const draftId = $(await requireDraft(token, recordId, 'Only drafts can be edited or deleted')); + $((await deleteDraft(token, draftId)).mapErr(notFoundAs(recordId))); + }), + ); +} + +// Changes an email's labels, as marking, starring and archiving do, and reads it back. +function relabel(change: {add?: string[]; remove?: string[]}): CommandRunner { + return (target, recordId) => + callGoogle(target.connection, (token) => + Do(async ($) => { + const messageId = $(requireMessage(recordId)); + $((await modifyMessage(token, messageId, change)).mapErr(notFoundAs(recordId))); + return $(await readMessage(token, recordId, messageId)); + }), + ); +} + +// Moves an email to Trash and reads it back, there. +const trash: CommandRunner = (target, recordId) => + callGoogle(target.connection, (token) => + Do(async ($) => { + const messageId = $(requireMessage(recordId)); + $((await trashMessage(token, messageId)).mapErr(notFoundAs(recordId))); + return $(await readMessage(token, recordId, messageId)); + }), + ); + +function requireRecipient(to: string, cc: string, bcc: string): Result { + if (`${to}${cc}${bcc}`.trim() === '') { + return Err(ApiErr.validationError('The email has no recipient')); + } + return Ok(undefined); +} + +// Sends a draft as it is saved; it leaves the drafts. +const send: CommandRunner = (target, recordId) => + callGoogle(target.connection, (token) => + Do(async ($) => { + const draftId = $(await requireDraft(token, recordId, 'Only drafts can be sent')); + const {payload} = $((await getDraft(token, draftId)).mapErr(notFoundAs(recordId))).message; + $(requireRecipient(headerOf(payload, 'To'), headerOf(payload, 'Cc'), headerOf(payload, 'Bcc'))); + $((await sendDraft(token, draftId)).mapErr(notFoundAs(recordId))); + return null; + }), + ); + +function sendNew(target: RecordTarget, values: RecordValues): Promise> { + return callGoogle(target.connection, (token) => + Do(async ($) => { + $(requireRecipient(values.to ?? '', '', '')); + const source = $(await compose(target.connection.account, values, null)); + $(await sendMessage(token, rawMessage(source, null))); + return null; + }), + ); +} + +export const gmailConnector: Connector = { + list: listPage, + get: (target, recordId) => callGoogle(target.connection, (token) => readRecord(token, recordId)), + create: createDraftRecord, + update: updateDraftRecord, + remove: removeDraft, + commands: { + markRead: relabel({remove: ['UNREAD']}), + markUnread: relabel({add: ['UNREAD']}), + flag: relabel({add: ['STARRED']}), + unflag: relabel({remove: ['STARRED']}), + archive: relabel({remove: ['INBOX']}), + trash, + send, + }, + newCommands: {sendNew}, +}; diff --git a/packages/db/scripts/seed.ts b/packages/db/scripts/seed.ts index f6c5bc7e..6bf1a2d9 100644 --- a/packages/db/scripts/seed.ts +++ b/packages/db/scripts/seed.ts @@ -204,8 +204,8 @@ const tracker = await connect({ defaults: ['read', 'create'], }); -// Made-up tokens, like Linear's: opening a file fails. -const drive = await connect({ +// Made-up tokens, like Linear's: opening a file or an email fails. +const googleAccount = await connect({ integrationId: 'google', account: 'demo.user@gmail.com', credential: encrypt( @@ -239,7 +239,7 @@ const connections: [Connection, ...Connection[]] = [ meetings, wiki, tracker, - drive, + googleAccount, ]; // ---- Information records -------------------------------------------------------------------- @@ -386,6 +386,7 @@ const agents = [ grants: [ grant(personal, {trash: true, write: false}), grant(work, {mark: true, flag: true, archive: true}), + grant(googleAccount, {readEmails: true, markEmails: true, archiveEmails: true}), grant(info, {readAddresses: false, readNotes: false}), grant(meetings, {readTranscripts: false}), grant(wiki, {read: false}), @@ -413,7 +414,7 @@ function requestsOf(collection: Collection): [string, ...string[]] { ]; } -// Titles for the mailboxes' emails; Information's come from the records seeded above. +// Titles for the mailboxes' emails, and Gmail's; Information's come from the records seeded above. const EMAIL_TITLES: [string, ...string[]] = [ 'Your order has shipped', 'Q3 planning — agenda', @@ -473,7 +474,7 @@ const SEARCHES: [string, ...string[]] = [ ]; function recordTitle(connection: Connection, collectionId: string): string { - if (connection.integrationId === 'email') { + if (connection.integrationId === 'email' || collectionId === 'emails') { return pick(EMAIL_TITLES); } diff --git a/packages/docs/content/connections/email/gmail.mdx b/packages/docs/content/connections/email/gmail.mdx index 63b0e484..36b3e96c 100644 --- a/packages/docs/content/connections/email/gmail.mdx +++ b/packages/docs/content/connections/email/gmail.mdx @@ -3,8 +3,8 @@ title: Gmail description: Make an app password for a Gmail or Google Workspace mailbox. --- -For Drive, Docs and Sheets as well, connect the account with [Google](/connections/google). Gmail -through that connection, for accounts that can't make app passwords, is coming. +Connecting the account with [Google](/connections/google) instead gives agents its Gmail without +an app password, for accounts that can't make one, and Drive, Docs and Sheets as well. diff --git a/packages/docs/content/connections/google.mdx b/packages/docs/content/connections/google.mdx index 38b8a75a..c2ec5e00 100644 --- a/packages/docs/content/connections/google.mdx +++ b/packages/docs/content/connections/google.mdx @@ -1,11 +1,12 @@ --- title: Google -description: Connect a Google account for agents to read Drive, Docs, Sheets and Slides. +description: Connect a Google account for agents to work with Gmail and read Drive, Docs, Sheets and Slides. --- -Agents work with the files the Google account can open, as the person who signs in. Every request -goes to Google live; nothing is copied. One sign-in covers Gmail, Drive, Docs, Sheets and Slides, -and one connection holds them all. Today agents can read files; Gmail and writing come later. +Agents work with the account's email and the files it can open, as the person who signs in. Every +request goes to Google live; nothing is copied. One sign-in covers Gmail, Drive, Docs, Sheets and +Slides, and one connection holds them all. Gmail needs no app password, so it works for accounts +that can't make one. Today agents can read files; writing them comes later. ## Before you start: a Google Cloud OAuth client @@ -90,7 +91,10 @@ See [Environment variables](/self-hosting/environment). Cloud projects made since May 2026 have Google's [standard quotas](https://developers.google.com/workspace/tools-safety), far above what a few -people use. +people use, except Gmail's: each account gets 6,000 units a minute, and a page of 50 emails costs +about 1,000 of them, since each email in it is asked for on its own. An agent paging through more +than a few lists a minute is told the provider could not be reached, and can try again a minute +later. ## Connect Google @@ -110,8 +114,9 @@ allowed at sign-in", until the person signs in again from the connection's page. ### Review the defaults -What agents are allowed starts with Read files. Turn it off for every agent on the connection's -page, or only for one agent on the agent's page. A line above the permissions says what that means +What agents are allowed starts with Read files alone, as a new mailbox gives agents no access to +its email. Turn actions on or off for every agent on the connection's page, or only for one agent +on the agent's page. A line above the permissions says what that means for the account: what is ticked is what agents can read and do with it, and nothing from Google is used to train a model. @@ -119,6 +124,25 @@ used to train a model. ## What agents can do +Agents see Gmail as one list, the newest first, 50 to a page: all mail but Spam and Trash, as +Gmail's All Mail is, or one folder of it, the inbox, drafts, sent mail, the archive, Trash or Spam. +A search is Gmail's own, as in its search box, such as `from:sam has:attachment newer_than:7d`. You +can allow each action separately: + +- **Read email** lists, opens and searches email. An email opens as its text, or its HTML as text + where it has none, with the names of its attachments. +- **Mark read or unread**, **Star**, **Archive** and **Move email to Trash** act on messages in + the inbox, and require Read email. Archived email stays in All Mail, and Gmail keeps Trash 30 + days before deleting it; nothing is deleted for good. +- **Write drafts** creates, edits and deletes drafts, and requires Read email. An edit keeps the + draft's Cc, Bcc and thread, and its formatting while its text is unchanged; text written anew + goes out as plain text. A draft with attachments isn't edited, since it would lose them. + Received messages are never edited or deleted. +- **Send** sends a new email or a saved draft from the account's own address, and Gmail files it + in Sent. It does not require Read email, so an agent can send without seeing the mailbox. + +For Drive, Docs, Sheets and Slides: + - **Read files** browses and searches Drive and opens files. At the top are four places: My Drive, Shared with me, Shared drives and Recent. My Drive, folders and shared drives list their files folders first, Shared with me the last shared first, and Recent the files the account opened, @@ -129,8 +153,8 @@ used to train a model. values as `Header: value` lines, the headers from the tab's first row. Other files, shortcuts included, open as their name, kind, owner, last change and link. -Gmail, and creating and editing files, are coming. The sign-in already asks for them, so no new -sign-in will be needed. +Creating and editing files is coming. The sign-in already asks for it, so no new sign-in will be +needed. ## Troubleshooting diff --git a/packages/docs/content/index.mdx b/packages/docs/content/index.mdx index 65e7acd4..cb35de7f 100644 --- a/packages/docs/content/index.mdx +++ b/packages/docs/content/index.mdx @@ -11,7 +11,7 @@ log, allowed or denied. The connection guides explain the integrations available today: [Information](/connections/information) for details stored in Personal Agent Proxy, [Email](/connections/email) for IMAP and SMTP mailboxes, -[Google](/connections/google) for Drive, Docs, Sheets and Slides, +[Google](/connections/google) for Gmail, Drive, Docs, Sheets and Slides, [Granola](/connections/granola) for meeting notes and transcripts, [Notion](/connections/notion) for pages and databases, [Linear](/connections/linear) for issues, and [PostgreSQL](/connections/postgres) or [MySQL](/connections/mysql) for a database's tables and diff --git a/packages/integrations/src/catalog.ts b/packages/integrations/src/catalog.ts index e8da63ab..c2eacc6b 100644 --- a/packages/integrations/src/catalog.ts +++ b/packages/integrations/src/catalog.ts @@ -4,6 +4,149 @@ const INBOX: Condition = {field: 'folder', values: ['Inbox']}; export const INFO_INTEGRATION_ID = 'info'; +// The actions an email collection's reads, writes and commands need, by what they do. +type EmailActions = { + read: string; + mark: string; + flag: string; + archive: string; + write: string; + trash: string; + send: string; +}; + +// What flagging is called: Flag on any mailbox, Star on Gmail. +type FlagWords = {flag: string; unflag: string; flagged: string; unflagged: string}; + +const FLAG: FlagWords = { + flag: 'Flag', + unflag: 'Unflag', + flagged: 'Flagged', + unflagged: 'Unflagged', +}; +const STAR: FlagWords = { + flag: 'Star', + unflag: 'Unstar', + flagged: 'Starred', + unflagged: 'Unstarred', +}; + +/** + * A mailbox's email as one collection, the same for any mailbox and for Gmail, so agents see one + * kind of email: the same fields and commands, under each integration's own action ids. Drafts + * are the only email edited or deleted; commands act on email in the inbox. + */ +function emailsCollection(args: { + actions: EmailActions; + folders: string[]; + searchHint: string; + flag: FlagWords; +}): Collection { + const {actions, flag} = args; + return { + id: 'emails', + name: 'Email', + singular: 'email', + titleField: 'subject', + summaryField: 'from', + filterField: 'folder', + searchHint: args.searchHint, + read: actions.read, + createLabel: 'Save as draft', + writes: {create: actions.write, update: actions.write, delete: actions.write}, + editable: {field: 'folder', values: ['Draft']}, + commands: [ + { + id: 'markRead', + on: 'record', + label: 'Mark as read', + action: actions.mark, + where: INBOX, + done: 'Marked {} as read', + tried: 'mark {} as read', + }, + { + id: 'markUnread', + on: 'record', + label: 'Mark as unread', + action: actions.mark, + where: INBOX, + done: 'Marked {} as unread', + tried: 'mark {} as unread', + }, + { + id: 'flag', + on: 'record', + label: flag.flag, + action: actions.flag, + where: INBOX, + done: `${flag.flagged} {}`, + tried: `${flag.flag.toLowerCase()} {}`, + }, + { + id: 'unflag', + on: 'record', + label: flag.unflag, + action: actions.flag, + where: INBOX, + done: `${flag.unflagged} {}`, + tried: `${flag.unflag.toLowerCase()} {}`, + }, + { + id: 'archive', + on: 'record', + label: 'Archive', + action: actions.archive, + where: INBOX, + done: 'Archived {}', + tried: 'archive {}', + }, + { + id: 'trash', + on: 'record', + label: 'Move to Trash', + action: actions.trash, + where: INBOX, + done: 'Moved {} to Trash', + tried: 'move {} to Trash', + }, + { + id: 'send', + on: 'record', + label: 'Send', + action: actions.send, + where: {field: 'folder', values: ['Draft']}, + done: 'Sent {}', + tried: 'send {}', + }, + { + id: 'sendNew', + on: 'new', + label: 'Send', + action: actions.send, + done: 'Sent {}', + tried: 'send {}', + }, + ], + fields: [ + { + key: 'folder', + label: 'Folder', + type: 'select', + options: args.folders, + system: true, + }, + {key: 'from', label: 'From', type: 'email', system: true}, + {key: 'to', label: 'To', type: 'email'}, + {key: 'subject', label: 'Subject', type: 'text'}, + {key: 'date', label: 'Date', type: 'datetime', system: true}, + // "Unread, Flagged" ("Unread, Starred" on Gmail), for an email in the inbox. + {key: 'status', label: 'Status', type: 'text', system: true}, + {key: 'body', label: 'Body', type: 'longtext'}, + ], + }; +} + // Any mailbox over IMAP and SMTP, signed in with an app password. The password can't be limited, // so access is only ever narrowed in Personal Agent Proxy. The inbox, drafts and sent mail are one // list, each email saying which folder it is in. Received emails are never edited or deleted for @@ -64,109 +207,21 @@ const email: Integration = { }, ], collections: [ - { - id: 'emails', - name: 'Email', - singular: 'email', - titleField: 'subject', - summaryField: 'from', - filterField: 'folder', + emailsCollection({ + actions: { + read: 'read', + mark: 'mark', + flag: 'flag', + archive: 'archive', + write: 'write', + trash: 'trash', + send: 'send', + }, + folders: ['Inbox', 'Draft', 'Sent'], searchHint: 'On Gmail, Gmail’s own search syntax works, such as from:sam has:attachment newer_than:7d. Other mailboxes match the words in the sender, recipients, subject and text.', - read: 'read', - createLabel: 'Save as draft', - writes: {create: 'write', update: 'write', delete: 'write'}, - editable: {field: 'folder', values: ['Draft']}, - commands: [ - { - id: 'markRead', - on: 'record', - label: 'Mark as read', - action: 'mark', - where: INBOX, - done: 'Marked {} as read', - tried: 'mark {} as read', - }, - { - id: 'markUnread', - on: 'record', - label: 'Mark as unread', - action: 'mark', - where: INBOX, - done: 'Marked {} as unread', - tried: 'mark {} as unread', - }, - { - id: 'flag', - on: 'record', - label: 'Flag', - action: 'flag', - where: INBOX, - done: 'Flagged {}', - tried: 'flag {}', - }, - { - id: 'unflag', - on: 'record', - label: 'Unflag', - action: 'flag', - where: INBOX, - done: 'Unflagged {}', - tried: 'unflag {}', - }, - { - id: 'archive', - on: 'record', - label: 'Archive', - action: 'archive', - where: INBOX, - done: 'Archived {}', - tried: 'archive {}', - }, - { - id: 'trash', - on: 'record', - label: 'Move to Trash', - action: 'trash', - where: INBOX, - done: 'Moved {} to Trash', - tried: 'move {} to Trash', - }, - { - id: 'send', - on: 'record', - label: 'Send', - action: 'send', - where: {field: 'folder', values: ['Draft']}, - done: 'Sent {}', - tried: 'send {}', - }, - { - id: 'sendNew', - on: 'new', - label: 'Send', - action: 'send', - done: 'Sent {}', - tried: 'send {}', - }, - ], - fields: [ - { - key: 'folder', - label: 'Folder', - type: 'select', - options: ['Inbox', 'Draft', 'Sent'], - system: true, - }, - {key: 'from', label: 'From', type: 'email', system: true}, - {key: 'to', label: 'To', type: 'email'}, - {key: 'subject', label: 'Subject', type: 'text'}, - {key: 'date', label: 'Date', type: 'datetime', system: true}, - // "Unread, Flagged", from the flags of an email in the inbox. - {key: 'status', label: 'Status', type: 'text', system: true}, - {key: 'body', label: 'Body', type: 'longtext'}, - ], - }, + flag: FLAG, + }), ], }; @@ -511,10 +566,13 @@ const mysql = sqlIntegration('mysql', 'MySQL', 'Tables and queries in a MySQL or // A Google account, signed in to once for everything Personal Agent Proxy reads and writes there: // Gmail, Drive, Docs, Sheets and Slides. One connection holds every product, so action ids carry -// the product, as Information's `readCards` and `writeCards` do. Drive is one tree: My Drive, -// Shared with me, Shared drives and Recent at the top, folders holding files, a spreadsheet holding -// its tabs and a tab its rows. Documents read as Markdown, presentations as text, rows as "Header: -// value" lines. Nothing is copied: every request goes to Google. +// the product, as Information's `readCards` and `writeCards` do. Gmail is the same email as any +// mailbox's, its labels as folders: all mail but Spam and Trash in one list, narrowed to the inbox, +// drafts, sent mail, the archive, Trash or Spam. Drafts are the only email edited or deleted; +// nothing else is deleted for good, only moved to Trash. Drive is one tree: My Drive, Shared with +// me, Shared drives and Recent at the top, folders holding files, a spreadsheet holding its tabs and +// a tab its rows. Documents read as Markdown, presentations as text, rows as "Header: value" lines. +// Nothing is copied: every request goes to Google. const google: Integration = { id: 'google', name: 'Google', @@ -523,6 +581,40 @@ const google: Integration = { disclosure: 'What is ticked here is what your agents can read and do with this Google account through Personal Agent Proxy, each time you ask them to. Personal Agent Proxy keeps only the sign-in and the activity log; nothing from Google is used to train a model.', actions: [ + { + id: 'readEmails', + label: 'Read email', + description: 'List, search and open email in Gmail, drafts and sent mail included', + risk: 'low', + }, + { + id: 'markEmails', + label: 'Mark read or unread', + description: 'Mark emails in the inbox read or unread', + risk: 'low', + requires: 'readEmails', + }, + { + id: 'flagEmails', + label: 'Star', + description: 'Star and unstar emails in the inbox', + risk: 'low', + requires: 'readEmails', + }, + { + id: 'archiveEmails', + label: 'Archive', + description: 'Move emails out of the inbox, keeping them in All Mail', + risk: 'medium', + requires: 'readEmails', + }, + { + id: 'writeEmails', + label: 'Write drafts', + description: 'Create, edit and delete drafts', + risk: 'medium', + requires: 'readEmails', + }, { id: 'readFiles', label: 'Read files', @@ -530,8 +622,38 @@ const google: Integration = { 'Browse and search Drive, and open files, documents, spreadsheets and presentations', risk: 'low', }, + { + id: 'trashEmails', + label: 'Move email to Trash', + description: 'Move emails to Trash, where Gmail keeps them 30 days before deleting them', + risk: 'high', + requires: 'readEmails', + }, + { + id: 'sendEmails', + label: 'Send', + description: 'Send email from this address, new or from a draft', + risk: 'high', + reachesOthers: true, + }, ], collections: [ + emailsCollection({ + actions: { + read: 'readEmails', + mark: 'markEmails', + flag: 'flagEmails', + archive: 'archiveEmails', + write: 'writeEmails', + trash: 'trashEmails', + send: 'sendEmails', + }, + // The archive is what is in none of the others: All Mail without the inbox. + folders: ['Inbox', 'Draft', 'Sent', 'Archive', 'Trash', 'Spam'], + searchHint: + 'Gmail’s own search syntax, as in Gmail’s search box, such as from:sam has:attachment newer_than:7d. Inside a folder, only what is in it.', + flag: STAR, + }), { id: 'files', name: 'Files', diff --git a/packages/integrations/src/tools.test.ts b/packages/integrations/src/tools.test.ts index 90f4d8ef..d6af1d28 100644 --- a/packages/integrations/src/tools.test.ts +++ b/packages/integrations/src/tools.test.ts @@ -199,8 +199,46 @@ describe('postgres and mysql', () => { describe('google', () => { const google = integration('google'); - test('files are a tree that is listed and opened, never written to yet', () => { - expect(integrationTools(google).map((tool) => tool.name)).toEqual(['files_list', 'files_get']); + test('email has the same tools as any mailbox’s, and files are listed and opened, never written to yet', () => { + expect(integrationTools(google).map((tool) => tool.name)).toEqual([ + ...integrationTools(email).map((tool) => tool.name), + 'files_list', + 'files_get', + ]); + }); + + test('email’s tools need the Gmail actions, and a list takes its folders and Gmail’s search', () => { + const list = findTool(google, 'emails_list'); + + expect(list?.action).toBe('readEmails'); + expect(list?.inputSchema.properties?.folder?.enum).toEqual([ + 'Inbox', + 'Draft', + 'Sent', + 'Archive', + 'Trash', + 'Spam', + ]); + expect(list?.inputSchema.properties?.search?.description).toContain('Gmail'); + expect(findTool(google, 'emails_flag')).toMatchObject({ + action: 'flagEmails', + description: expect.stringMatching(/^Star:/), + }); + expect(findTool(google, 'emails_send_new')).toMatchObject({ + action: 'sendEmails', + risk: 'high', + reachesOthers: true, + }); + expect(findTool(google, 'emails_trash')).toMatchObject({ + action: 'trashEmails', + risk: 'high', + where: {field: 'folder', values: ['Inbox']}, + }); + const update = findTool(google, 'emails_update'); + expect(update?.kind === 'update' ? update.where : null).toEqual({ + field: 'folder', + values: ['Draft'], + }); }); test('a list takes the folder to list inside and a search', () => {