diff --git a/AGENTS.md b/AGENTS.md index a6428b08..41e6f590 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -25,19 +25,19 @@ - `packages/utils` - `@proxy/utils`, the Result helpers shared by every package: `Do`, `requirePresent` and `parseSchema` (`src/parse.ts`), and `httpRequest`, the fetch wrapper that returns a `FetchError` union instead of throwing (`src/fetch.ts`). Also `ApiError`/`ApiErr` and `wrapDb` (`src/api-error.ts`), the error every server-side function returns, and the session cookie's name and parsing (`src/session-cookie.ts`) - `packages/integrations` - `@proxy/integrations`, the catalog the api and the web app share: each integration's actions, collections and fields (`src/catalog.ts`), the agents a login can be made for (`src/agent-providers.ts`), the email providers' servers (`src/email-providers.ts`), and access (`src/access.ts`): per action of a connection, an agent's own setting, else the connection's default, an action counting only with the one it `requires`, and tools (`src/tools.ts`): what an agent can do with a connection, generated from the catalog, one per list, open, write and command of each collection (`emails_list`, `emails_archive`), each with the action it needs and a JSON Schema of its params. A custom MCP server (`mcp` in the catalog, `custom`) has no actions of its own: `integrationFor(connection)` (`src/mcp-tools.ts`) builds its integration from the tools the connection keeps (`McpTool`, as the server lists them), one action per tool named by the tool, its risk from the tool's annotations (low when it only reads, medium when what it changes can be undone, high when it can't, and `unknown`, shown without a label, when it says nothing), and `integrationTools` passes them on as `call` tools with the server's own input schema (`ServerTool`, beside the catalog's `CollectionTool`). The web app adds the icons (`src/lib/integrations.ts`, whose `integrationOf(connection)` is what every page uses) - `packages/db` - `@proxy/db`: Prisma 6 on Postgres. `src/index.ts` is the client; each module (`auth.ts`, `user.ts`, `organization.ts`, `connection.ts`, `info.ts`, `agent.ts`, `audit.ts`) exports functions returning `Result` through `wrapDb`. A connection's `tools` column holds a custom MCP server's tools as JSON, and `name` what the person called it; both null for every other integration. `relationMode = "prisma"`: no foreign keys in the database, so Prisma runs the cascades and every relation column has its own index. Locally the database is Prisma's embedded Postgres (PGlite, `scripts/dev-db.ts`), which needs `connection_limit=1&pgbouncer=true` in `DATABASE_URL` -- `packages/api` - `@proxy/api`: Express on Bun. Routes are `handleXRoute` functions mounted in `src/server/create-api-router.ts`; JSON routes return `Result` through `withAuthResult`, which puts the signed-in user on `req.user`. Every environment variable is in `src/utils/env-schema.ts` with its docs, which the docs' Environment variables page is built from; add new ones there. Personal Agent Proxy is one origin everywhere: when `packages/web/dist` exists, as in the Docker image, the api serves the web app's build too (`src/server/serve-web.ts`), and every address the api builds starts from `APP_URL`. Browser-called writes take `requireBrowserOrigin` (CSRF); routes that read a cookie have no CORS. `src/connections/` lists, deletes and sets defaults on an organization's connections, connects a mailbox with an app password after signing in to its IMAP and SMTP servers (`email/`), and connects Granola, Notion and Linear by signing in to their MCP servers (`mcp/`, shared by all three: OAuth with PKCE against the server's authorization server, a client registered for each sign-in so nothing is set up beforehand, the handshake in an encrypted cookie, `mcpConnectRoutes` making both routes on the sign-in flow every OAuth integration shares, `connect-flow.ts`; tokens are refreshed as they run out, one refresh at a time per connection on each server, read again first so ones another call or server saved meanwhile are used rather than refreshed twice, and saved only over the ones they replace, whatever else the credential holds kept beside them, by `connections/token-session.ts`, which Google's connection will share; each tool call is one stateless `tools/call`, `mcp-client.ts`, which also lists a server's tools and, for a server that may keep a session (`handshake`), initializes one around each call as the spec asks). Every request to an MCP server, its metadata or its authorization server goes through `utils/checked-request.ts`, which checks the address as a database's host is checked (`utils/host-check.ts`: nothing private unless `ALLOW_PRIVATE_HOSTS`, and `https` unless the address is private) before each request, since a custom server and its endpoints are wherever it said, and never follows a redirect. A custom MCP server is connected by its address (`custom/`): `POST /api/connections/mcp` with `{name, url, token?, returnTo?}` (the name is the person's, kept through the sign-in in its cookie, since a server says nothing about itself that would do) asks the server for its tools, connects a server that answers at once, with no sign-in or the token typed in as a bearer token (as written when it has its scheme in front, `Basic …`), and for one that turns a request without a token down begins its sign-in there, under the browser's origin check, answering `{kind: 'signIn', url}` with the sign-in page the browser goes to: `mcp-discovery.ts` finds the authorization server from the server's protected resource metadata (the `WWW-Authenticate` header's `resource_metadata`, else the well-known paths), its endpoints from that server's metadata (else `/authorize`, `/token` and `/register` under it) and the scopes the server's own metadata names, and the sign-in, begun with `signInFlow` (`connect-flow.ts`, which `mcpConnectRoutes` builds on for the servers known in advance), keeps the endpoints in the handshake cookie and in the credential beside the tokens. The connection's account is the address without its scheme (`mcp.example.com/mcp`), connecting the same address again replaces its credential and tools, a new connection's defaults are the tools that say they only read, `POST /api/connections/:id/tools` asks the server for its tools again (replacing the tools drops the defaults and agents' settings for tools the server no longer lists, `updateConnectionTools`), and a tool is run as it is (`custom-mcp.ts`): its params go to the server as given, a tool that fails answers the agent with the server's own text as a `validation_error`, and the run is logged under the collection `tools` by the tool's name, with its params as JSON, cut to 200 characters. Granola's notes and transcripts are on by default, Notion's Read pages and Linear's Read issues alone; a Notion or Linear sign-in is for one workspace, and a Notion one lasts at most 180 days, however often it is refreshed. `src/agents/` makes, revokes and deletes agent logins (the password is sent once, only its argon2id hash is kept) and sets an agent's own settings per action of a connection where they differ from the default. `src/records/` reads and writes a collection's records through the integration's `Connector` (`connectors.ts`): Information's live in the database (`info-connector.ts`), a mailbox's are fetched over IMAP on every request (`email-connector.ts`), which merges the inbox and the folders the server marks as Drafts and Sent into one list (ids start with the folder: `inbox-…`, `drafts-…`, `sent-…`) and runs the commands: flags, moves to the folders the server marks as Archive (Gmail's All Mail) or Trash, and sending over SMTP, a draft or a new email, filing a copy in Sent where the server doesn't (Gmail does). Received emails are never edited or deleted for good; only drafts are. Granola's notes are fetched from its MCP server on every request (`granola-connector.ts`; the reading of the XML-like text Granola answers in is in `connections/granola/granola-mcp.ts`): its Notes and Transcripts collections are the same meetings by Granola's meeting id, and since Granola's tools have no pages, a page is 30 days of meetings; one that finds none skips back to the newest meeting before it, so the list ends where the meetings do. Granola's dates and attendees are shown as Granola writes them. Notion's workspace is fetched from its MCP server the same way (`notion-connector.ts`, the reading of its answers in `connections/notion/notion-mcp.ts`), as the person who signed in, as a tree in one nested collection: Notion has no list of every page, so the top is their Recents and a search (Notion's own, AI search where the workspace has it) finds the rest, one page of 50; inside a page are the pages and databases in its content, inside a database its saved views (`view-…` ids), or the rows of its only one, and inside a view its rows, paged with Notion's cursor (view queries have no plan quota; Notion's row and SQL queries do). A page reads as Notion-flavored Markdown, a row's properties beside it as "Name: value" lines (`notion-properties.ts`), and only a page's, database's or view's id is ever sent to Notion's fetch; a new page is a private draft at the top, or goes inside the page, database or view it is made in, a row's title in the database's title property; an edit sends the title, the properties written differently and the content, each only where it changed, and pages are never deleted. Notion's MCP server has no tool that deletes or trashes a page. Linear's issues are fetched from its MCP server the same way (`linear-connector.ts`, the reading of its answers in `connections/linear/linear-mcp.ts`), as the person who signed in, by their identifier (`ENG-123`): a list is the issues updated last first, 50 to a page with Linear's cursor, narrowed by the type of status (`?statusType=Started`) and searched in titles and descriptions, and an issue opens with its comments, oldest first. Teams, statuses, people, projects and labels are written by name, as Linear shows them and finds them, its refusals passed on to the agent; an edit sends only the fields written differently, an empty one clearing it. Linear's MCP server has no tool that archives or deletes an issue. Google is connected by signing in to Google with the app's own Google client, the one Google sign-in uses (`GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET`, a second redirect URI `/api/connections/google/callback`), asking for every scope at once, Gmail, Drive, Docs, Sheets and Slides, with offline access (`connections/google/`: `google-oauth.ts` the sign-in and refresh, with the client's secret, keeping the refresh token Google doesn't send again; `google-client.ts` the calls with a bearer token and Google's errors, a disabled API passing Google's own message on, which names the API and where to enable it); the credential holds the tokens, the scopes Google granted, since the consent screen lets a person untick some, and the address; the connection keeps Google's id for the account (`Connection.providerAccountId`, its `sub`), so signing in again renews the same connection, and its address, even after the address changed (`findAccountConnection`, by address for one without an id and for the MCP integrations). Each action needs one of the scopes allowing it (`google/scopes.ts`), so an action whose scope wasn't granted is unavailable: `connections/unavailable.ts`, the one place the rest of the api asks, reads it from the credential on every request (`unavailable` on a connection; one that can't be read leaves every action unavailable rather than failing the list), no default or agent setting can turn it on, `runAgentTool` refuses it saying to sign in to Google again, and the connection page shows its checkbox disabled with a "Sign in again" link (`?account=` opening Google on the account with `login_hint`). An integration's `disclosure`, Google's Limited Use line, is shown above a connection's permissions. Disconnecting a Google connection revokes its grant at Google first (`connections/disconnect.ts`), unless another connection in any organization is for the same account, since a grant covers all of the client's tokens for it, and deletes it even when Google can't be reached. Its Gmail is the `emails` collection, the same fields and commands as a mailbox's under the Google action ids (`readEmails` … `sendEmails`, both built by `emailsCollection` in the catalog), fetched live from Gmail's API (`records/google-emails.ts` over `connections/google/gmail-api.ts`): one list, 50 to a page with Gmail's page token, all mail but Spam and Trash, or one folder, each a label (Inbox `INBOX`, Draft `DRAFT`, Sent `SENT`, Trash `TRASH`, Spam `SPAM`) or, for Archive, a search for what is in none of them; an email's labels say its folder, so ids are Gmail's message ids, a draft's `draft-`, the id Gmail edits, deletes and sends it by and which stays as it is edited. `messages.list` gives only ids, so each is asked for with its headers, 8 at once, again twice when Gmail answers busy; Gmail's per-user quota of 6,000 units a minute makes a page cost about 1,000. A search is Gmail's own. An email opens as its text part, or its HTML part as text (read in one pass, `htmlToText`), with its attachments' names. Marking takes `UNREAD` off or puts it on, starring `STARRED`, archiving takes `INBOX` off, and Move to Trash is `messages.trash`, never `messages.delete`: nothing is deleted for good but drafts, the only email edited or deleted, an edit keeping the draft's Cc, Bcc and thread, its recipients' names where they are written as they read and its HTML where its text is, and refusing a draft with attachments, which it would drop; a draft is edited, deleted and sent by its message id too. Email goes out from the connection's own address, a draft with `drafts.send` and a new one with `messages.send`, written by `MailComposer` as the IMAP connector writes it; Gmail files it in Sent. Its Drive is one nested collection, `files` (`records/google-files.ts`, behind `google-connector.ts`, which dispatches on the collection), fetched live from Drive's and Sheets' APIs (`drive-api.ts`, `sheets-api.ts`): at the top four places, My Drive, Shared with me, Shared drives (`drive-…` ids) and Recent, inside them and inside folders their files, folders first, 50 to a page with Drive's page token, and a search is Drive's full-text one, anywhere or inside the folder opened; the trail is walked up a file's parents to My Drive, its shared drive, or, where the account can't see further up, Shared with me. A document opens as Markdown and a presentation as text, as Drive exports them, up to its 10 MB export cap, beyond which opening it says so; a spreadsheet opens into its tabs (`sheet--`) and a tab into its rows (`row---`), 50 to a page, the page token the next row's number, each row's values "Header: value" lines under the tab's first row, a column without a header going by its letter (`Column C`); a page past the tab's last row is empty and a row past it not found without asking Sheets, which refuses a range outside the grid. Drive isn't written to yet; its writes come later. PostgreSQL and MySQL (MariaDB too) are two integrations on one code path: a database is connected with its host, port, database, user, password and encryption (`connections/sql/`: `route.ts` makes both routes per dialect, `sql-client.ts` opens one connection per request with `Bun.sql`, to the address `utils/host-check.ts` resolves the host to, refusing private addresses unless `ALLOW_PRIVATE_HOSTS` is set, and `sql-statement.ts` checks an agent's query is one statement that reads), and `records/sql-connector.ts` reads and writes it through `records/sql-dialect.ts`, where the two differ: the tables are one nested collection, a table holding its rows, a row read and written as "column: value" lines and found again by its primary key (ids `public.users:42`, `pairs:3,4`; a table without a key is never written to), and a query, a second collection that is only ever run, goes through a read-only transaction with a time limit, a cursor on Postgres and `sql_select_limit` on MySQL, 200 rows at most, answered as JSON lines. MySQL's read-only transaction doesn't stop DDL, so the statement check is what keeps a query to reading; the docs tell people to make a read-only database user as well. Lists take `?search=` (only where the collection has a `searchHint`; Granola has none), `?filter=` (the collection's `filterField`, such as `?filter=Draft`), `?parent=` (only in a `nested` collection: the list inside a record, answered with its `trail`, the records above it; a record made there goes in it) and `?page=` (`list-query.ts`) and answer a page, newest first (a mailbox's 50 emails, Granola's 30 days), with the next page's token; a mailbox's token holds where each folder goes on, so paging through the merged list shows every email once, and its search uses Gmail's own syntax on Gmail (`X-GM-EXT-1`) and IMAP text search elsewhere. Agents sign in at `/agent-auth/login` with a cookie of their own (`src/agent-auth/`); `src/agent-side/` serves them: the connections an agent can use, the tools it has with each, and `runAgentTool` (`tools.ts`), the one place a tool runs, which checks it against the agent's access as it is now, validates its params against the tool's schema (an edit changes only the fields given), calls the connector, or a custom server's tool as it is, answering `{text}`, and logs it, denied ones too, to the activity log that `src/activity/` lists. Over HTTP that is `/api/agent/me`, `GET /api/agent/connections/:id/tools` and `POST /api/agent/connections/:id/tools/:tool` with `{params}`; `src/mcp/` serves the same over MCP at `/mcp` (Streamable HTTP without sessions, JSON answers, no stream): four tools whatever the login may do, `list_connections` (the entry point), `list_tools`, `read` (only tools that read, marked read-only) and `run`, and server instructions naming the agent's connections. MCP clients sign in with OAuth 2.1 (`src/oauth/`), Personal Agent Proxy being its own authorization server: metadata under `/.well-known/`, Dynamic Client Registration at `/oauth/register` (open to anyone, each registration first deleting the clients that registered over a day ago and never finished a sign-in), `/oauth/authorize` sealing the request into the web app's `/connect-agent` page, where the person chooses the agent login the client works as or makes one, and `/oauth/token` with PKCE (S256 only) and refresh tokens replaced on every use; codes and tokens are kept as their SHA-256, and revoking an agent ends its clients' tokens. These endpoints take any origin (CORS), since none reads a cookie; credentials and Information records are stored encrypted with `ENCRYPTION_KEY` (`src/utils/secret-crypto.ts`). Has its own `.prettierrc.json` with a 200-column print width +- `packages/api` - `@proxy/api`: Express on Bun. Routes are `handleXRoute` functions mounted in `src/server/create-api-router.ts`; JSON routes return `Result` through `withAuthResult`, which puts the signed-in user on `req.user`. Every environment variable is in `src/utils/env-schema.ts` with its docs, which the docs' Environment variables page is built from; add new ones there. Personal Agent Proxy is one origin everywhere: when `packages/web/dist` exists, as in the Docker image, the api serves the web app's build too (`src/server/serve-web.ts`), and every address the api builds starts from `APP_URL`. Browser-called writes take `requireBrowserOrigin` (CSRF); routes that read a cookie have no CORS. `src/connections/` lists, deletes and sets defaults on an organization's connections, connects a mailbox with an app password after signing in to its IMAP and SMTP servers (`email/`), and connects Granola, Notion and Linear by signing in to their MCP servers (`mcp/`, shared by all three: OAuth with PKCE against the server's authorization server, a client registered for each sign-in so nothing is set up beforehand, the handshake in an encrypted cookie, `mcpConnectRoutes` making both routes on the sign-in flow every OAuth integration shares, `connect-flow.ts`; tokens are refreshed as they run out, one refresh at a time per connection on each server, read again first so ones another call or server saved meanwhile are used rather than refreshed twice, and saved only over the ones they replace, whatever else the credential holds kept beside them, by `connections/token-session.ts`, which Google's connection will share; each tool call is one stateless `tools/call`, `mcp-client.ts`, which also lists a server's tools and, for a server that may keep a session (`handshake`), initializes one around each call as the spec asks). Every request to an MCP server, its metadata or its authorization server goes through `utils/checked-request.ts`, which checks the address as a database's host is checked (`utils/host-check.ts`: nothing private unless `ALLOW_PRIVATE_HOSTS`, and `https` unless the address is private) before each request, since a custom server and its endpoints are wherever it said, and never follows a redirect. A custom MCP server is connected by its address (`custom/`): `POST /api/connections/mcp` with `{name, url, token?, returnTo?}` (the name is the person's, kept through the sign-in in its cookie, since a server says nothing about itself that would do) asks the server for its tools, connects a server that answers at once, with no sign-in or the token typed in as a bearer token (as written when it has its scheme in front, `Basic …`), and for one that turns a request without a token down begins its sign-in there, under the browser's origin check, answering `{kind: 'signIn', url}` with the sign-in page the browser goes to: `mcp-discovery.ts` finds the authorization server from the server's protected resource metadata (the `WWW-Authenticate` header's `resource_metadata`, else the well-known paths), its endpoints from that server's metadata (else `/authorize`, `/token` and `/register` under it) and the scopes the server's own metadata names, and the sign-in, begun with `signInFlow` (`connect-flow.ts`, which `mcpConnectRoutes` builds on for the servers known in advance), keeps the endpoints in the handshake cookie and in the credential beside the tokens. The connection's account is the address without its scheme (`mcp.example.com/mcp`), connecting the same address again replaces its credential and tools, a new connection's defaults are the tools that say they only read, `POST /api/connections/:id/tools` asks the server for its tools again (replacing the tools drops the defaults and agents' settings for tools the server no longer lists, `updateConnectionTools`), and a tool is run as it is (`custom-mcp.ts`): its params go to the server as given, a tool that fails answers the agent with the server's own text as a `validation_error`, and the run is logged under the collection `tools` by the tool's name, with its params as JSON, cut to 200 characters. Granola's notes and transcripts are on by default, Notion's Read pages and Linear's Read issues alone; a Notion or Linear sign-in is for one workspace, and a Notion one lasts at most 180 days, however often it is refreshed. `src/agents/` makes, revokes and deletes agent logins (the password is sent once, only its argon2id hash is kept) and sets an agent's own settings per action of a connection where they differ from the default. `src/records/` reads and writes a collection's records through the integration's `Connector` (`connectors.ts`): Information's live in the database (`info-connector.ts`), a mailbox's are fetched over IMAP on every request (`email-connector.ts`), which merges the inbox and the folders the server marks as Drafts and Sent into one list (ids start with the folder: `inbox-…`, `drafts-…`, `sent-…`) and runs the commands: flags, moves to the folders the server marks as Archive (Gmail's All Mail) or Trash, and sending over SMTP, a draft or a new email, filing a copy in Sent where the server doesn't (Gmail does). Received emails are never edited or deleted for good; only drafts are. Granola's notes are fetched from its MCP server on every request (`granola-connector.ts`; the reading of the XML-like text Granola answers in is in `connections/granola/granola-mcp.ts`): its Notes and Transcripts collections are the same meetings by Granola's meeting id, and since Granola's tools have no pages, a page is 30 days of meetings; one that finds none skips back to the newest meeting before it, so the list ends where the meetings do. Granola's dates and attendees are shown as Granola writes them. Notion's workspace is fetched from its MCP server the same way (`notion-connector.ts`, the reading of its answers in `connections/notion/notion-mcp.ts`), as the person who signed in, as a tree in one nested collection: Notion has no list of every page, so the top is their Recents and a search (Notion's own, AI search where the workspace has it) finds the rest, one page of 50; inside a page are the pages and databases in its content, inside a database its saved views (`view-…` ids), or the rows of its only one, and inside a view its rows, paged with Notion's cursor (view queries have no plan quota; Notion's row and SQL queries do). A page reads as Notion-flavored Markdown, a row's properties beside it as "Name: value" lines (`notion-properties.ts`), and only a page's, database's or view's id is ever sent to Notion's fetch; a new page is a private draft at the top, or goes inside the page, database or view it is made in, a row's title in the database's title property; an edit sends the title, the properties written differently and the content, each only where it changed, and pages are never deleted. Notion's MCP server has no tool that deletes or trashes a page. Linear's issues are fetched from its MCP server the same way (`linear-connector.ts`, the reading of its answers in `connections/linear/linear-mcp.ts`), as the person who signed in, by their identifier (`ENG-123`): a list is the issues updated last first, 50 to a page with Linear's cursor, narrowed by the type of status (`?statusType=Started`) and searched in titles and descriptions, and an issue opens with its comments, oldest first. Teams, statuses, people, projects and labels are written by name, as Linear shows them and finds them, its refusals passed on to the agent; an edit sends only the fields written differently, an empty one clearing it. Linear's MCP server has no tool that archives or deletes an issue. Google is connected by signing in to Google with the app's own Google client, the one Google sign-in uses (`GOOGLE_CLIENT_ID`, `GOOGLE_CLIENT_SECRET`, a second redirect URI `/api/connections/google/callback`), asking for every scope at once, Gmail, Drive, Docs, Sheets and Slides, with offline access (`connections/google/`: `google-oauth.ts` the sign-in and refresh, with the client's secret, keeping the refresh token Google doesn't send again; `google-client.ts` the calls with a bearer token and Google's errors, a disabled API passing Google's own message on, which names the API and where to enable it); the credential holds the tokens, the scopes Google granted, since the consent screen lets a person untick some, and the address; the connection keeps Google's id for the account (`Connection.providerAccountId`, its `sub`), so signing in again renews the same connection, and its address, even after the address changed (`findAccountConnection`, by address for one without an id and for the MCP integrations). Each action needs one of the scopes allowing it (`google/scopes.ts`), so an action whose scope wasn't granted is unavailable: `connections/unavailable.ts`, the one place the rest of the api asks, reads it from the credential on every request (`unavailable` on a connection; one that can't be read leaves every action unavailable rather than failing the list), no default or agent setting can turn it on, `runAgentTool` refuses it saying to sign in to Google again, and the connection page shows its checkbox disabled with a "Sign in again" link (`?account=` opening Google on the account with `login_hint`). An integration's `disclosure`, Google's Limited Use line, is shown above a connection's permissions. Disconnecting a Google connection revokes its grant at Google first (`connections/disconnect.ts`), unless another connection in any organization is for the same account, since a grant covers all of the client's tokens for it, and deletes it even when Google can't be reached. Its Gmail is the `emails` collection, the same fields and commands as a mailbox's under the Google action ids (`readEmails` … `sendEmails`, both built by `emailsCollection` in the catalog), fetched live from Gmail's API (`records/google-emails.ts` over `connections/google/gmail-api.ts`): one list, 50 to a page with Gmail's page token, all mail but Spam and Trash, or one folder, each a label (Inbox `INBOX`, Draft `DRAFT`, Sent `SENT`, Trash `TRASH`, Spam `SPAM`) or, for Archive, a search for what is in none of them; an email's labels say its folder, so ids are Gmail's message ids, a draft's `draft-`, the id Gmail edits, deletes and sends it by and which stays as it is edited. `messages.list` gives only ids, so each is asked for with its headers, 8 at once, again twice when Gmail answers busy; Gmail's per-user quota of 6,000 units a minute makes a page cost about 1,000. A search is Gmail's own. An email opens as its text part, or its HTML part as text (read in one pass, `htmlToText`), with its attachments' names. Marking takes `UNREAD` off or puts it on, starring `STARRED`, archiving takes `INBOX` off, and Move to Trash is `messages.trash`, never `messages.delete`: nothing is deleted for good but drafts, the only email edited or deleted, an edit keeping the draft's Cc, Bcc and thread, its recipients' names where they are written as they read and its HTML where its text is, and refusing a draft with attachments, which it would drop; a draft is edited, deleted and sent by its message id too. Email goes out from the connection's own address, a draft with `drafts.send` and a new one with `messages.send`, written by `MailComposer` as the IMAP connector writes it; Gmail files it in Sent. Its Drive is one nested collection, `files` (`records/google-files.ts`, behind `google-connector.ts`, which dispatches on the collection), fetched live from Drive's and Sheets' APIs (`drive-api.ts`, `sheets-api.ts`): at the top four places, My Drive, Shared with me, Shared drives (`drive-…` ids, of the kind Shared drive, which is never edited or trashed) and Recent, inside them and inside folders their files, folders first, 50 to a page with Drive's page token, and a search is Drive's full-text one, anywhere or inside the folder opened; the trail is walked up a file's parents to My Drive, its shared drive, or, where the account can't see further up, Shared with me. A document opens as Markdown and a presentation as text, as Drive exports them, up to its 10 MB export cap, beyond which its content says the file is too large, so it can still be renamed or trashed; a spreadsheet opens into its tabs (`sheet--`) and a tab into its rows (`row---`), 50 to a page, the page token the next row's number, each row's values "Header: value" lines under the tab's first row, a header repeated or a column without one going by its letter (`Status (E)`); a page past the tab's last row is empty and a row past it not found without asking Sheets, which refuses a range outside the grid. Folders, documents and spreadsheets are made in My Drive (`root`, where one made without a parent goes too), a folder or a shared drive, never in Shared with me, Shared drives or Recent, the record's `kind` saying which, the one select a create writes and an edit can't change: a document is uploaded as Markdown and a spreadsheet as CSV with the Google type Drive converts it to (a multipart upload, `drive-api.ts`), and a row made inside a tab goes after the last of its rows holding anything (`values.append` on the whole tab, written again from column A where Sheets started it further right), its "Header: value" lines put under the headers with `readValueLines`. An edit renames a file or a tab (`updateSheetProperties`), replaces a document's content whole by uploading Markdown again, only where it is written differently from the record as `runAgentTool` read it (`before`, passed to every connector's update), which drops its comments and suggestions, and changes a row only in the cells written differently (`values:batchUpdate`); a presentation's and a spreadsheet's content are only read, and the places and shared drives aren't changed. Files go to Trash with the `trash` command (`trashed: true`), never `files.delete`, and tabs and rows don't. A second collection, `ranges` (`records/google-ranges.ts`), is only run, as a database's queries are: Read (`readOnly`, Read files) and Write (Edit files) take a spreadsheet's id or link and a range in A1 notation, checked against a strict pattern before it reaches Google's address, and read or write its values as CSV (`records/csv.ts`; a read answers 1000 rows at most), written as if typed in, so a formula stays one, and answered as Sheets shows them afterwards. Every files action takes the `drive` scope, the Sheets API's writes included. `google-connector.ts` runs a command on the collection it was run on, since `emails` and `files` both have `trash`. PostgreSQL and MySQL (MariaDB too) are two integrations on one code path: a database is connected with its host, port, database, user, password and encryption (`connections/sql/`: `route.ts` makes both routes per dialect, `sql-client.ts` opens one connection per request with `Bun.sql`, to the address `utils/host-check.ts` resolves the host to, refusing private addresses unless `ALLOW_PRIVATE_HOSTS` is set, and `sql-statement.ts` checks an agent's query is one statement that reads), and `records/sql-connector.ts` reads and writes it through `records/sql-dialect.ts`, where the two differ: the tables are one nested collection, a table holding its rows, a row read and written as "column: value" lines and found again by its primary key (ids `public.users:42`, `pairs:3,4`; a table without a key is never written to), and a query, a second collection that is only ever run, goes through a read-only transaction with a time limit, a cursor on Postgres and `sql_select_limit` on MySQL, 200 rows at most, answered as JSON lines. MySQL's read-only transaction doesn't stop DDL, so the statement check is what keeps a query to reading; the docs tell people to make a read-only database user as well. Lists take `?search=` (only where the collection has a `searchHint`; Granola has none), `?filter=` (the collection's `filterField`, such as `?filter=Draft`), `?parent=` (only in a `nested` collection: the list inside a record, answered with its `trail`, the records above it; a record made there goes in it) and `?page=` (`list-query.ts`) and answer a page, newest first (a mailbox's 50 emails, Granola's 30 days), with the next page's token; a mailbox's token holds where each folder goes on, so paging through the merged list shows every email once, and its search uses Gmail's own syntax on Gmail (`X-GM-EXT-1`) and IMAP text search elsewhere. Agents sign in at `/agent-auth/login` with a cookie of their own (`src/agent-auth/`); `src/agent-side/` serves them: the connections an agent can use, the tools it has with each, and `runAgentTool` (`tools.ts`), the one place a tool runs, which checks it against the agent's access as it is now, validates its params against the tool's schema (an edit changes only the fields given), calls the connector, or a custom server's tool as it is, answering `{text}`, and logs it, denied ones too, to the activity log that `src/activity/` lists. Over HTTP that is `/api/agent/me`, `GET /api/agent/connections/:id/tools` and `POST /api/agent/connections/:id/tools/:tool` with `{params}`; `src/mcp/` serves the same over MCP at `/mcp` (Streamable HTTP without sessions, JSON answers, no stream): four tools whatever the login may do, `list_connections` (the entry point), `list_tools`, `read` (only tools that read, marked read-only) and `run`, and server instructions naming the agent's connections. MCP clients sign in with OAuth 2.1 (`src/oauth/`), Personal Agent Proxy being its own authorization server: metadata under `/.well-known/`, Dynamic Client Registration at `/oauth/register` (open to anyone, each registration first deleting the clients that registered over a day ago and never finished a sign-in), `/oauth/authorize` sealing the request into the web app's `/connect-agent` page, where the person chooses the agent login the client works as or makes one, and `/oauth/token` with PKCE (S256 only) and refresh tokens replaced on every use; codes and tokens are kept as their SHA-256, and revoking an agent ends its clients' tokens. These endpoints take any origin (CORS), since none reads a cookie; credentials and Information records are stored encrypted with `ENCRYPTION_KEY` (`src/utils/secret-crypto.ts`). Has its own `.prettierrc.json` with a 200-column print width - `skills/` - Agent skills, symlinked into `.claude/skills` and `.codex/skills` by `bun run setup` ## The app Personal Agent Proxy holds the integrations and personal information an AI agent needs, so every agent company does not have to rebuild them and you do not have to trust each one's security. A human connects services and hands out agent logins; an agent signs in with one and works through what it was given. Everything is served by `packages/api`; the web app has no data of its own. -- **Model**: an integration from the catalog (`@proxy/integrations`: Email over IMAP and SMTP, Google's Gmail, Drive, Docs, Sheets and Slides over Google's APIs, one sign-in for a Google account, Granola's meeting notes, Notion's pages and Linear's issues over their MCP servers, PostgreSQL and MySQL over their own protocols) has the actions an agent can be allowed with a connection of it (Read, Archive, Write drafts, Send) and collections, each a list of fields with the actions reading and writing it need, and the commands it offers beyond editing (Mark as read, Archive, Send), each needing one action. A command runs on a `record`, where its `where` picks out which (Archive only in the inbox, Send only on a draft), or on values typed in (`new`: sending a new email). A mailbox is one collection, its folder a field the agent side filters by. A `nested` collection is a tree, as Notion's pages are: a record can hold others of the collection (`hasChildren`), listed inside it with the list tool's `parent` and made inside it with the create tool's, and access is the same anywhere in it. A collection without a `read` action is never listed or opened, only run, as a database's queries are, and a `new` command marked `readOnly` counts as a read for MCP clients. Integrations join the catalog once they work end to end. Any other service with an MCP server is connected as a custom MCP server: the server's tools are the actions, each allowed on its own, and there are no collections; an agent runs a tool with the params the server describes and gets back what it answered. A connection is a connected integration and its account, owned by the organization; an integration can be connected more than once, such as two mailboxes. Information is the built-in `info` connection (addresses, payment cards, notes), made with the organization, so access and the agent side treat it like any other. Nothing is copied from a provider: every request goes to it live. Fields marked `system` are set by the provider, never typed in. +- **Model**: an integration from the catalog (`@proxy/integrations`: Email over IMAP and SMTP, Google's Gmail, Drive, Docs, Sheets and Slides over Google's APIs, one sign-in for a Google account, Granola's meeting notes, Notion's pages and Linear's issues over their MCP servers, PostgreSQL and MySQL over their own protocols) has the actions an agent can be allowed with a connection of it (Read, Archive, Write drafts, Send) and collections, each a list of fields with the actions reading and writing it need, and the commands it offers beyond editing (Mark as read, Archive, Send), each needing one action. A command runs on a `record`, where its `where` picks out which (Archive only in the inbox, Send only on a draft), or on values typed in (`new`: sending a new email). A mailbox is one collection, its folder a field the agent side filters by. A `nested` collection is a tree, as Notion's pages are: a record can hold others of the collection (`hasChildren`), listed inside it with the list tool's `parent` and made inside it with the create tool's, and access is the same anywhere in it. A collection without a `read` action is never listed or opened, only run, as a database's queries and a spreadsheet's ranges are, and a `new` command marked `readOnly` counts as a read for MCP clients. Integrations join the catalog once they work end to end. Any other service with an MCP server is connected as a custom MCP server: the server's tools are the actions, each allowed on its own, and there are no collections; an agent runs a tool with the params the server describes and gets back what it answered. A connection is a connected integration and its account, owned by the organization; an integration can be connected more than once, such as two mailboxes. Information is the built-in `info` connection (addresses, payment cards, notes), made with the organization, so access and the agent side treat it like any other. Nothing is copied from a provider: every request goes to it live. Fields marked `system` are set by the provider, never typed in. - **Access** is one list of actions per connection, and every action the catalog lists can be set: what Personal Agent Proxy can do with the provider is the limit. Two layers: the connection's default (a `ConnectionDefault` row per action that is on) and an agent's own setting, on or off, where it differs (`AgentGrant`). An agent gets its own setting, else the default as it is now; an action that `requires` another counts only with it (Archive needs Read; Send needs nothing, so an agent can send without reading the mailbox), and an action a sign-in didn't cover, as Google's consent screen allows, is unavailable whatever is ticked. A collection's `writes` name the action creating, editing and deleting need, and `editable` which records they apply to (drafts). An agent login is for one agent from `AGENT_PROVIDERS`; each organization can have one login per company, named by it, and any number of custom MCP agents (`custom`, for any MCP client not in the list), each named by the person and told apart by their names (`Agent` is unique by organization, provider and name). It has a generated username and a password that is shown once. Every request on the agent side is an audit entry, allowed or denied, with what a list searched for. - **Accounts**: a human signs in with Google or a magic link (`src/auth/` in the api); the sign-in page says that continuing accepts the terms and the privacy policy, on the website, and a sign-up records it (`User.termsAcceptedAt`); accounts from before the notice have none until they are asked. A session is a `Session` row and an httpOnly cookie; the web app's `/api/me` loader on the human side's route redirects to `/login?callbackUrl=…` without one. Every user gets one `Organization` at sign-up, created with the user. `ALLOWED_SIGNUP_EMAILS` limits who can sign up; the login page asks `/auth/methods` whether to offer Google. In development the login page has a Create dev user button; in development, or without `RESEND_KEY`, the api prints magic links instead of emailing them. Cookies are secure when `APP_URL` is HTTPS. A new organization first goes through the welcome flow (`/welcome`, outside the human side's layout; the human side redirects there until `Organization.onboardedAt` is set, which `POST /api/me/onboarded` does when the flow is finished or skipped): step one is `AgentSetup` (`src/components/agent-setup.tsx`, which New agent and setting an agent up again use too): it picks the agent from `AGENT_PROVIDERS`, the choice in the address (`?agent=claude`) so the browser's back button returns to it, and does what its `connects` says, an `mcp` agent (Claude, Claude Code, Claude Desktop, Hermes Agent, ChatGPT, Poke) shown the server address and that agent's steps (Claude Code and Hermes Agent use terminal commands) while the page polls for the login its sign-in makes, and over http, as in development, only a `local` agent (Claude Code, Claude Desktop, Hermes Agent) can be picked, the rest greyed out with a pointer to self-hosting on a server with a domain, a `login` agent given a login at once, as a message to paste to it with the sign-in page, username and password, or, for a `vault` agent (Instinct), which won't take a password in chat, as three steps: a message asking it for a link to its vault, the login to fill in on the page it sends, and a message telling it to sign in; step two (`/welcome/connections`) is the catalog, a plus per integration with how many accounts are connected: a mailbox goes to `/welcome/connections/email`, the form on a page with Continue once signed in, and a sign-in started with `?returnTo=welcome` comes back to `/welcome/connections/:id`, the new connection's permissions (its defaults, saved as they are ticked) with Confirm back to the catalog; a database goes to `/welcome/connections/postgres` or `/welcome/connections/mysql` the same way as a mailbox. Organizations from before the flow were marked done by its migration. - **Human side** behind `/login`: `/connections` (a right click on a row renames the connection, `PUT /api/connections/:id/name`, an empty name being none of its own, which a custom MCP server can't have, or deletes it, each behind a dialog), `/connections/new` (the catalog, a plus per integration with how many accounts are connected: `/connections/new/email` connects a mailbox with an app password, with a Test connection that signs in and saves nothing, `/connections/new/postgres` and `/connections/new/mysql` a database the same way, `/connections/new/mcp` a custom MCP server under a name of the person's, by its address, with an access token where the server takes one instead of a sign-in (`ConnectMcpForm`, which sends the browser to the server's sign-in when the api says it asks for one), and Google, Granola, Notion and Linear are connected by signing in to them; an integration needing a client the server doesn't have (Google without `GOOGLE_CLIENT_ID` and `GOOGLE_CLIENT_SECRET`, which `/auth/methods` reports) is greyed out with a pointer to the docs, and its sign-in, reached anyway, comes back with `?error=google-unconfigured` saying the same; `ConnectEmailForm`, `ConnectDatabaseForm`, `ConnectMcpForm` and `IntegrationCatalog` serve the welcome flow too, at `/welcome/connections/`), `/connections/:id` (titled by the name the person gave it where they did, as the list and the agent side are, the defaults under Permissions, with how many agents differ per action, a custom MCP server's tools being its actions, with a button that asks the server for them again, and a "Connection added successfully" when `?added` says it was just connected), `/info`, `/agents`, `/agents/new` (the welcome flow's agent step, among the companies without a login yet and Custom MCP agent, on to the agent's page), `/agents/:id/setup` (the same for one agent, to hand it a new password or add the connector again; the only place a password shows, once), `/agents/:id` (what the agent can reach and did, the MCP clients working as it when there are any; access laid out like an editor's settings: one checkbox per action of each connection, the only control for it, each connection folded by default to a row saying Default or Changed; a setting of the agent's own is marked with a bar, a button that makes it the connection's default for every agent, and one that resets it, and a filter and Changed only narrow the list; `ConnectionAccess` serves both pages), `/activity` (filters in the URL, so the agent and connection pages link to their part of the log; a custom server's tool reads "Ran Search docs with {…}"). Each page has a loader in `src/loaders.ts` and revalidates after a change. Layout follows the spanish repo: the shadcn sidebar as a rail, an `h-12` header, one `max-w-3xl` column, one-line `RowList` rows. Type has one size, `text-sm`, in rows, summaries and hints alike, except column headers (`RowHeader`), which are `text-xs`: the thing itself in the normal colour, what is said about it (column headers, times, summaries, descriptions) in `text-muted-foreground`, and `font-medium` only for page and section titles. - **MCP**: an agent that speaks MCP (Poke, Claude, Claude Code, Claude Desktop, Hermes Agent, ChatGPT, or any other as a Custom MCP agent) adds `/mcp` as a custom connector and signs in; the person picks which agent login it works as on `/connect-agent`, or makes one there: a client whose name points at no company's agent is offered a new custom agent named after it, which the person can rename. Setting up a custom agent waits for one that wasn't there before to sign in, since there can be several. The agent page shows the server's address and the MCP clients working as the login, each with a Disconnect; the activity log marks requests that came in over MCP (`AuditEntry.via`). -- **Agent side** behind `/agent/login`, with its own cookie: `/agent` lists what the login can do with each connection and the lists it can open, `/agent/:connection/tool/:tool` one of a custom MCP server's tools, a form of its params built from the tool's schema and what it answered below (the home page lists the tools the login may run), `/agent/:connection/:collection` a collection's records (newest first, 50 to a page with an Older link, narrowed by its `filterField`, such as the folder, in the address as the list tool takes it, `?folder=Draft`, and searched where the collection has a `searchHint`, which says what the search takes; in a nested collection `?parent=` opens it inside a record, with the records above it as breadcrumbs, a record holding others opening into them, and New making one there), `…/new` and `…/:record` the generic `RecordForm` and `RecordFields`, with a button per command the agent may run on the record, Send beside Save as draft on a new email, and a command that answers something, as running a query does, shown under the form (loaders in `src/agent-loaders.ts`). Every page runs a tool (`toolName` in `@proxy/integrations`) through the api. It is built for a model driving a browser: no sidebar, no icon-only buttons, nothing that only shows on hover. Lists it can neither read nor add to are left out, and one it can only add to (sending without Read) opens on its form; asking for one anyway shows the api's refusal, which it logs as denied. +- **Agent side** behind `/agent/login`, with its own cookie: `/agent` lists what the login can do with each connection and the lists it can open, `/agent/:connection/tool/:tool` one of a custom MCP server's tools, a form of its params built from the tool's schema and what it answered below (the home page lists the tools the login may run), `/agent/:connection/:collection` a collection's records (newest first, 50 to a page with an Older link, narrowed by its `filterField`, such as the folder, in the address as the list tool takes it, `?folder=Draft`, and searched where the collection has a `searchHint`, which says what the search takes; in a nested collection `?parent=` opens it inside a record, with the records above it as breadcrumbs, a record holding others opening into them, and New making one there), `…/new` and `…/:record` the generic `RecordForm` and `RecordFields`, with a button per command the agent may run on the record, Send beside Save as draft on a new email, Write beside Read on a range, and a command that answers something, as running a query does, shown under the form (loaders in `src/agent-loaders.ts`). Every page runs a tool (`toolName` in `@proxy/integrations`) through the api. It is built for a model driving a browser: no sidebar, no icon-only buttons, nothing that only shows on hover. Lists it can neither read nor add to are left out, and one it can only add to (sending without Read) opens on its form; asking for one anyway shows the api's refusal, which it logs as denied. - **Testing email locally**: any IMAP server with TLS on 993 and SMTP on 465 or 587 works through Other. A self-signed one needs `NODE_TLS_REJECT_UNAUTHORIZED=0` on the api, never in production. The server's hostname needs a dot and can't be an IP address: `localhost4.localdomain4` reaches 127.0.0.1 on most Linux machines. ## Code Style diff --git a/packages/api/src/agent-side/tools.test.ts b/packages/api/src/agent-side/tools.test.ts index 873058be..c1158c36 100644 --- a/packages/api/src/agent-side/tools.test.ts +++ b/packages/api/src/agent-side/tools.test.ts @@ -269,7 +269,7 @@ describe('runAgentTool', () => { const result = await runAgentTool(caller, 'mail-1', 'emails_update', {id: draft.id, subject: 'Re: Q4 planning'}); expect(result.isOk()).toBe(true); - expect(connector.update.mock.calls[0]?.slice(1)).toEqual([draft.id, {to: 'sam@example.com', subject: 'Re: Q4 planning', body: 'Draft body'}]); + expect(connector.update.mock.calls[0]?.slice(1)).toEqual([draft.id, {to: 'sam@example.com', subject: 'Re: Q4 planning', body: 'Draft body'}, draft]); expect(logAgentRequest.mock.calls[0]?.[0]).toMatchObject({action: 'update', recordTitle: 'Re: Q4 planning'}); }); diff --git a/packages/api/src/agent-side/tools.ts b/packages/api/src/agent-side/tools.ts index e1bafc66..dbb9e2c0 100644 --- a/packages/api/src/agent-side/tools.ts +++ b/packages/api/src/agent-side/tools.ts @@ -304,7 +304,7 @@ function update(run: ToolRun, params: Record): Promise !field.system).map((field) => [field.key, before.values[field.key] ?? ''])); const values = $(parseRecordValues(collection, {...current, ...fieldsOf(params)})); - const record = $(await run.connector.update(run.target, recordId, values)); + const record = $(await run.connector.update(run.target, recordId, values, before)); $(await log(run, 'allowed', recordTitle(collection, record.values))); return {record}; }); diff --git a/packages/api/src/connections/google/drive-api.ts b/packages/api/src/connections/google/drive-api.ts index b5e9bea7..2d3688e7 100644 --- a/packages/api/src/connections/google/drive-api.ts +++ b/packages/api/src/connections/google/drive-api.ts @@ -1,17 +1,22 @@ +import {randomUUID} from 'node:crypto'; + import type {Result} from 'ts-results-es'; import {z} from 'zod'; import type {ApiError} from '@proxy/utils'; -import {googleRequest, googleText} from './google-client'; +import {googleRequest, googleText, googleUpload, googleWrite} from './google-client'; /** * The Drive v3 calls the `files` collection makes, and the reading of their answers. Every call - * takes shared drives too. + * takes shared drives too. A file is uploaded as Markdown or CSV with the Google type it is to + * become, and Drive converts it. */ const DRIVE = 'https://www.googleapis.com/drive/v3'; +const UPLOAD = 'https://www.googleapis.com/upload/drive/v3'; + export const PAGE_SIZE = 50; /** Drive's ids, as every id an agent sends is checked against before it reaches an address or a query. */ @@ -55,8 +60,8 @@ const driveListSchema = z.object({drives: z.array(sharedDriveSchema), nextPageTo export type Page = {items: T[]; nextPage: string | null}; -function driveUrl(path: string, params: Record): URL { - const url = new URL(`${DRIVE}${path}`); +function driveUrl(path: string, params: Record, base: string = DRIVE): URL { + const url = new URL(`${base}${path}`); for (const [name, value] of Object.entries(params)) { url.searchParams.set(name, value); } @@ -117,3 +122,57 @@ export async function listSharedDrives(accessToken: string, search: string | nul export function getSharedDrive(accessToken: string, driveId: string): Promise> { return googleRequest(accessToken, driveUrl(`/drives/${driveId}`, {fields: 'id,name'}), sharedDriveSchema); } + +/** A new file's name, the Google type it is to be, and the folder it goes in. */ +export type NewFile = {name: string; mimeType: string; parent: string}; + +/** What a file is uploaded as: its type, such as `text/markdown`, and its text. */ +export type Upload = {type: string; body: string}; + +// Text uploaded as the UTF-8 it is sent as, so Drive reads letters beyond ASCII as they are. +function textType(type: string): string { + return `${type}; charset=UTF-8`; +} + +// A file's metadata and its content in one request, as Drive's multipart upload takes them. +function multipart(metadata: object, upload: Upload): Upload { + const boundary = `proxy-${randomUUID()}`; + const body = [ + `--${boundary}`, + 'Content-Type: application/json; charset=UTF-8', + '', + JSON.stringify(metadata), + `--${boundary}`, + `Content-Type: ${textType(upload.type)}`, + '', + upload.body, + `--${boundary}--`, + ].join('\r\n'); + return {type: `multipart/related; boundary=${boundary}`, body}; +} + +/** Makes a file in a folder: empty, or from what is uploaded, which Drive converts to the file's type. */ +export function createFile(accessToken: string, file: NewFile, upload: Upload | null): Promise> { + const metadata = {name: file.name, mimeType: file.mimeType, parents: [file.parent]}; + const params = {fields: FILE_FIELDS, supportsAllDrives: 'true'}; + if (upload === null) { + return googleWrite(accessToken, 'POST', driveUrl('/files', params), metadata, driveFileSchema); + } + return googleUpload(accessToken, 'POST', driveUrl('/files', {...params, uploadType: 'multipart'}, UPLOAD), multipart(metadata, upload), driveFileSchema); +} + +/** Changes a file's metadata, such as its name, or puts it in Trash. */ +export function updateFile(accessToken: string, fileId: string, changes: {name?: string; trashed?: true}): Promise> { + return googleWrite(accessToken, 'PATCH', driveUrl(`/files/${fileId}`, {fields: FILE_FIELDS, supportsAllDrives: 'true'}), changes, driveFileSchema); +} + +/** Replaces all of a file's content with what is uploaded, which Drive converts to the file's own type. */ +export function replaceContent(accessToken: string, fileId: string, upload: Upload): Promise> { + return googleUpload( + accessToken, + 'PATCH', + driveUrl(`/files/${fileId}`, {uploadType: 'media', fields: FILE_FIELDS, supportsAllDrives: 'true'}, UPLOAD), + {...upload, type: textType(upload.type)}, + driveFileSchema, + ); +} diff --git a/packages/api/src/connections/google/google-client.ts b/packages/api/src/connections/google/google-client.ts index a7ff5d8a..46f720f6 100644 --- a/packages/api/src/connections/google/google-client.ts +++ b/packages/api/src/connections/google/google-client.ts @@ -38,6 +38,9 @@ function readGoogleError(error: Extract): {reasons: return {reasons, message}; } +/** What reading a file past Google's export limit says, where its content would be. */ +export const TOO_LARGE_TO_READ = 'This file is too large to read; Google exports at most 10 MB'; + const RATE_LIMITED = ['rateLimitExceeded', 'userRateLimitExceeded', 'RATE_LIMIT_EXCEEDED']; // What Google says of a call whose token lacks the scope it takes, as sending a draft does with @@ -61,7 +64,7 @@ export function googleError(error: FetchError): ApiError { const {reasons, message} = readGoogleError(error); if (reasons.includes('exportSizeLimitExceeded')) { - return ApiErr.validationError('This file is too large to read; Google exports at most 10 MB'); + return ApiErr.validationError(TOO_LARGE_TO_READ); } if (reasons.includes('accessNotConfigured') || reasons.includes('SERVICE_DISABLED') || /has not been used|is disabled/.test(message)) { return ApiErr.validationError(message); @@ -101,9 +104,16 @@ export async function googleText(accessToken: string, url: URL): Promise(accessToken: string, method: 'POST' | 'PUT', url: URL, body: object, schema: S): Promise, ApiError>> { - const init = {method, headers: {...authHeaders(accessToken), 'content-type': 'application/json'}, body: JSON.stringify(body)}; +type WriteMethod = 'POST' | 'PUT' | 'PATCH'; + +/** A POST, PUT or PATCH to one of Google's APIs with a JSON body, its answer read with `schema`. */ +export function googleWrite(accessToken: string, method: WriteMethod, url: URL, body: object, schema: S): Promise, ApiError>> { + return googleUpload(accessToken, method, url, {type: 'application/json', body: JSON.stringify(body)}, schema); +} + +/** A write whose body is of its own type, as a file uploaded to Drive is, its answer read with `schema`. */ +export async function googleUpload(accessToken: string, method: WriteMethod, url: URL, content: {type: string; body: string}, schema: S): Promise, ApiError>> { + const init = {method, headers: {...authHeaders(accessToken), 'content-type': content.type}, body: content.body}; const answer = await httpRequest(url, init, {schema, timeoutMs: TIMEOUT_MS}); return answer.mapErr(googleError); } diff --git a/packages/api/src/connections/google/scopes.ts b/packages/api/src/connections/google/scopes.ts index df944e27..61696d83 100644 --- a/packages/api/src/connections/google/scopes.ts +++ b/packages/api/src/connections/google/scopes.ts @@ -14,8 +14,10 @@ const scope = (name: string): string => `https://www.googleapis.com/auth/${name} // `gmail.send` alone. const GMAIL = [scope('gmail.modify')]; -// Any one of an action's scopes is enough. Drive's export and the Docs and Sheets APIs all take -// `drive`, so it alone covers reading files. +// Any one of an action's scopes is enough. Drive's export and uploads and the Sheets API take +// `drive` for reading and writing alike, so it alone covers every file action. +const DRIVE = [scope('drive')]; + const ACTION_SCOPES: Partial> = { readEmails: GMAIL, markEmails: GMAIL, @@ -24,7 +26,10 @@ const ACTION_SCOPES: Partial> = { writeEmails: GMAIL, trashEmails: GMAIL, sendEmails: [scope('gmail.send'), scope('gmail.modify')], - readFiles: [scope('drive')], + readFiles: DRIVE, + createFiles: DRIVE, + editFiles: DRIVE, + trashFiles: DRIVE, }; /** The actions a Google sign-in didn't cover, in the catalog's order; one the map leaves out is never covered. */ diff --git a/packages/api/src/connections/google/sheets-api.ts b/packages/api/src/connections/google/sheets-api.ts index 9b64ba9b..50d8c37b 100644 --- a/packages/api/src/connections/google/sheets-api.ts +++ b/packages/api/src/connections/google/sheets-api.ts @@ -1,13 +1,14 @@ import type {Result} from 'ts-results-es'; import {z} from 'zod'; -import type {ApiError} from '@proxy/utils'; +import {ApiErr, type ApiError, Do, requirePresent} from '@proxy/utils'; -import {googleRequest} from './google-client'; +import {googleRequest, googleWrite} from './google-client'; /** - * The Sheets v4 calls the `files` collection makes, and the reading of their answers: a - * spreadsheet's tabs, and the values in ranges of one. + * The Sheets v4 calls the `files` and `ranges` collections make, and the reading of their answers: + * a spreadsheet's tabs, and the values in ranges of one, read as Sheets shows them and written as + * if typed in, so a number stays a number and `=SUM(B2:B9)` is a formula. */ const SHEETS = 'https://sheets.googleapis.com/v4/spreadsheets'; @@ -52,9 +53,96 @@ export async function getSheets(accessToken: string, spreadsheetId: string): Pro ); } +// A tab's title as A1 notation quotes it: `'Alex''s notes'`. +function quotedTitle(title: string): string { + return `'${title.replace(/'/g, "''")}'`; +} + /** Rows `from` to `to` of a tab, every column, as A1 notation writes them: `'Q1 Sales'!A2:ZZ51`. */ export function rowsRange(title: string, from: number, to: number): string { - return `'${title.replace(/'/g, "''")}'!A${from}:${LAST_COLUMN}${to}`; + return `${quotedTitle(title)}!A${from}:${LAST_COLUMN}${to}`; +} + +/** One cell of a tab, its column counting from 0: `'Q1 Sales'!C5`. */ +export function cellRange(title: string, column: number, row: number): string { + return `${quotedTitle(title)}!${columnLetters(column)}${row}`; +} + +// The values in a range, the range percent-encoded in the path, and what is done with them, such as `:append`. +function valuesUrl(spreadsheetId: string, range: string, params: Record = {}, verb = ''): URL { + const url = new URL(`${SHEETS}/${spreadsheetId}/values/${encodeURIComponent(range)}${verb}`); + for (const [name, value] of Object.entries(params)) { + url.searchParams.set(name, value); + } + return url; +} + +// What a write answers: the range it changed, and, where asked for, the values there now. +const updatedSchema = z.object({updatedRange: z.string().optional(), updatedRows: z.number().optional(), updatedData: valueRangeSchema.optional()}); + +const appendSchema = z.object({updates: z.object({updatedRange: z.string()})}); + +/** The values in a range, as Sheets shows them, and the range itself as Sheets writes it. */ +export async function getValues(accessToken: string, spreadsheetId: string, range: string): Promise> { + const answer = await googleRequest(accessToken, valuesUrl(spreadsheetId, range), valueRangeSchema.extend({range: z.string()})); + return answer.map((read) => ({range: read.range, values: read.values ?? []})); +} + +/** Writes values into a range, as if typed in, and answers the range written and the values there now. */ +export async function putValues(accessToken: string, spreadsheetId: string, range: string, values: string[][]): Promise> { + const url = valuesUrl(spreadsheetId, range, {valueInputOption: 'USER_ENTERED', includeValuesInResponse: 'true', responseValueRenderOption: 'FORMATTED_VALUE'}); + const answer = await googleWrite(accessToken, 'PUT', url, {range, majorDimension: 'ROWS', values}, updatedSchema); + return answer.map((written) => ({range: written.updatedRange ?? range, values: written.updatedData?.values ?? []})); +} + +// Where a write's answer says it wrote: the part after the last "!", since a tab's title can hold +// one too, as in `'a!B7'!A12:C12`. +const writtenSchema = /^\$?([A-Z]+)\$?(\d+)(?::\$?([A-Z]+)\$?\d+)?$/; + +function writtenCells(range: string): Result<{column: number; row: number; lastColumn: number}, ApiError> { + const [, first = '', row = '', last] = writtenSchema.exec(range.slice(range.lastIndexOf('!') + 1)) ?? []; + return requirePresent(row || null, ApiErr.validationError(`Google answered a range that names no row: ${range}`)).map(() => ({ + column: columnIndex(first), + row: Number(row), + lastColumn: columnIndex(last ?? first), + })); +} + +/** + * Adds a row under the last of a tab's rows that hold anything, as if typed in, and answers the + * row's number. Sheets appends after the last block of rows in the range it is given, starting at + * that block's first column, so the whole tab is given; where its last block doesn't start in + * column A, the row is written again from A. + */ +export async function appendRow(accessToken: string, spreadsheetId: string, title: string, cells: string[]): Promise> { + const url = valuesUrl(spreadsheetId, quotedTitle(title), {valueInputOption: 'USER_ENTERED', insertDataOption: 'INSERT_ROWS'}, ':append'); + return Do(async ($) => { + const {updates} = $(await googleWrite(accessToken, 'POST', url, {majorDimension: 'ROWS', values: [cells]}, appendSchema)); + const written = $(writtenCells(updates.updatedRange)); + if (written.column === 0) { + return written.row; + } + // The row's own cells from A, and the ones written past them emptied. + const width = Math.max(cells.length, written.lastColumn + 1); + const row = Array.from({length: width}, (_, column) => cells[column] ?? ''); + const range = `${quotedTitle(title)}!A${written.row}:${columnLetters(width - 1)}${written.row}`; + $(await putValues(accessToken, spreadsheetId, range, [row])); + return written.row; + }); +} + +/** Writes each value into its own cell, as if typed in, the cells around them as they are. */ +export async function updateCells(accessToken: string, spreadsheetId: string, cells: {range: string; value: string}[]): Promise> { + const data = cells.map(({range, value}) => ({range, majorDimension: 'ROWS', values: [[value]]})); + const answer = await googleWrite(accessToken, 'POST', new URL(`${SHEETS}/${spreadsheetId}/values:batchUpdate`), {valueInputOption: 'USER_ENTERED', data}, z.object({})); + return answer.map(() => undefined); +} + +/** Renames a tab. */ +export async function renameSheet(accessToken: string, spreadsheetId: string, sheetId: number, title: string): Promise> { + const request = {updateSheetProperties: {properties: {sheetId, title}, fields: 'title'}}; + const answer = await googleWrite(accessToken, 'POST', new URL(`${SHEETS}/${spreadsheetId}:batchUpdate`), {requests: [request]}, z.object({})); + return answer.map(() => undefined); } /** The values in each range, in the order asked, one list of cells a row. */ @@ -67,6 +155,11 @@ export async function getRanges(accessToken: string, spreadsheetId: string, rang return batch.map(({valueRanges}) => valueRanges.map((range) => range.values ?? [])); } +/** A column's position counting from 0, from its letters: 0 for A, 26 for AA. */ +export function columnIndex(letters: string): number { + return [...letters].reduce((index, letter) => index * 26 + (letter.charCodeAt(0) - 64), 0) - 1; +} + /** A column's letters, from its position counting from 0: A, Z, AA. */ export function columnLetters(index: number): string { const letter = String.fromCharCode(65 + (index % 26)); diff --git a/packages/api/src/connections/route.test.ts b/packages/api/src/connections/route.test.ts index 25aa41a6..ca77caad 100644 --- a/packages/api/src/connections/route.test.ts +++ b/packages/api/src/connections/route.test.ts @@ -115,7 +115,15 @@ describe('handleListConnectionsRoute', () => { const {handleListConnectionsRoute} = await import('./route'); const [, connection] = (await handleListConnectionsRoute(makeRequest())).unwrap().connections; - expect(connection).toEqual({id: 'conn-2', integrationId: 'google', account: 'alex@example.com', defaults: [], tools: null, name: null, unavailable: ['readFiles']}); + expect(connection).toEqual({ + id: 'conn-2', + integrationId: 'google', + account: 'alex@example.com', + defaults: [], + tools: null, + name: null, + unavailable: ['readFiles', 'createFiles', 'editFiles', 'trashFiles'], + }); }); test('a user without an organization is signed out', async () => { @@ -277,7 +285,7 @@ describe('handleSetConnectionDefaultsRoute', () => { const {handleSetConnectionDefaultsRoute} = await import('./route'); const result = await handleSetConnectionDefaultsRoute(makeRequest({connectionId: 'conn-2'}, {actions: {readFiles: false}})); - expect(result.unwrap().unavailable).toEqual(['readFiles']); + expect(result.unwrap().unavailable).toEqual(['readFiles', 'createFiles', 'editFiles', 'trashFiles']); expect(setConnectionDefaults).toHaveBeenCalledWith({connectionId: 'conn-2', actions: {readFiles: false}}); }); diff --git a/packages/api/src/connections/unavailable.test.ts b/packages/api/src/connections/unavailable.test.ts index d51c4d7d..fb5718c4 100644 --- a/packages/api/src/connections/unavailable.test.ts +++ b/packages/api/src/connections/unavailable.test.ts @@ -38,10 +38,10 @@ describe('unavailableActions', () => { expect(unavailableActions(google(ALL_SCOPES))).toEqual([]); }); - test('a Google sign-in without Drive leaves Read files out', async () => { + test('a Google sign-in without Drive leaves every files action out', async () => { const {unavailableActions} = await import('./unavailable'); - expect(unavailableActions(google(ALL_SCOPES.filter((scope) => !scope.endsWith('/drive'))))).toEqual(['readFiles']); + expect(unavailableActions(google(ALL_SCOPES.filter((scope) => !scope.endsWith('/drive'))))).toEqual(['readFiles', 'createFiles', 'editFiles', 'trashFiles']); }); test('a Google sign-in with only gmail.send leaves every Gmail action but Send out', async () => { diff --git a/packages/api/src/records/connector.ts b/packages/api/src/records/connector.ts index 1a0d2a43..3e122347 100644 --- a/packages/api/src/records/connector.ts +++ b/packages/api/src/records/connector.ts @@ -49,7 +49,9 @@ export type Connector = { get: (target: RecordTarget, recordId: string) => Promise>; // In a nested collection, inside `parent` when one is given. create: (target: RecordTarget, values: RecordValues, parent: string | null) => Promise>; - update: (target: RecordTarget, recordId: string, values: RecordValues) => Promise>; + // `before` is the record as the caller read it, for a connector to tell what was written + // differently from it without reading it again. + update: (target: RecordTarget, recordId: string, values: RecordValues, before?: DataRecord) => Promise>; remove: (target: RecordTarget, recordId: string) => Promise>; // By the catalog's command id: the record commands, and those on values typed in. commands?: Partial>; diff --git a/packages/api/src/records/csv.test.ts b/packages/api/src/records/csv.test.ts new file mode 100644 index 00000000..b8079bf1 --- /dev/null +++ b/packages/api/src/records/csv.test.ts @@ -0,0 +1,44 @@ +import {expect, test} from 'bun:test'; + +import {csvOf, parseCsv} from './csv'; + +test('values with commas, quotes and line breaks are quoted, and read back as they were', () => { + const rows = [ + ['Item', 'Note'], + ['Rent', 'Paid, on time'], + ['Say "hi"', 'Two\nlines'], + ['', ''], + ]; + const text = csvOf(rows); + + expect(text).toBe('Item,Note\nRent,"Paid, on time"\n"Say ""hi""","Two\nlines"\n,'); + expect(parseCsv(text).unwrap()).toEqual(rows); +}); + +test('Windows line breaks and a line break at the end end rows, and an empty text is no rows', () => { + expect(parseCsv('a,b\r\nc,d\r\n').unwrap()).toEqual([ + ['a', 'b'], + ['c', 'd'], + ]); + expect(parseCsv('').unwrap()).toEqual([]); +}); + +test('a line with nothing on it is no row, so it empties nothing', () => { + expect(parseCsv('a\n\nb\n\n').unwrap()).toEqual([['a'], ['b']]); + expect(parseCsv('a,b\r\n\r\nc,d').unwrap()).toEqual([ + ['a', 'b'], + ['c', 'd'], + ]); + expect(parseCsv('\n').unwrap()).toEqual([]); + expect(parseCsv('a,\n').unwrap()).toEqual([['a', '']]); +}); + +test('an empty value in quotes is a row of its own, so one cell can be emptied', () => { + expect(parseCsv('""').unwrap()).toEqual([['']]); + expect(parseCsv('a\n""\n').unwrap()).toEqual([['a'], ['']]); +}); + +test('a quote inside an unquoted value is kept, and one never closed is turned down', () => { + expect(parseCsv('5" pipe,2').unwrap()).toEqual([['5" pipe', '2']]); + expect(parseCsv('"open,2').isErr()).toBe(true); +}); diff --git a/packages/api/src/records/csv.ts b/packages/api/src/records/csv.ts new file mode 100644 index 00000000..19914732 --- /dev/null +++ b/packages/api/src/records/csv.ts @@ -0,0 +1,72 @@ +import {Err, Ok, type Result} from 'ts-results-es'; + +import {ApiErr, type ApiError} from '@proxy/utils'; + +/** + * Rows of values as CSV, one row a line, as a spreadsheet's ranges are read and written: a value + * holding a comma, a quote or a line break in quotes, a quote in it doubled. + */ + +function csvValue(value: string): string { + return /[",\r\n]/.test(value) ? `"${value.replace(/"/g, '""')}"` : value; +} + +export function csvOf(rows: string[][]): string { + return rows.map((row) => row.map(csvValue).join(',')).join('\n'); +} + +/** + * The rows CSV gives, a line break inside quotes kept in its value, and a line with nothing on it, + * as a second line break at the end, no row; an unclosed quote is turned down. + */ +export function parseCsv(text: string): Result { + const rows: string[][] = []; + let row: string[] = []; + let value = ''; + let quoted = false; + // Whether the line read so far has anything on it, even an empty value in quotes, as `""` is. + let started = false; + for (let index = 0; index < text.length; index += 1) { + const char = text[index]; + if (!quoted && (char === '\n' || char === '\r')) { + if (char === '\r' && text[index + 1] === '\n') { + index += 1; + } + if (started) { + rows.push([...row, value]); + } + row = []; + value = ''; + started = false; + continue; + } + started = true; + if (quoted && char === '"' && text[index + 1] === '"') { + value += '"'; + index += 1; + continue; + } + if (char === '"' && (quoted || value === '')) { + quoted = !quoted; + continue; + } + if (quoted) { + value += char; + continue; + } + if (char === ',') { + row.push(value); + value = ''; + continue; + } + value += char; + } + if (quoted) { + return Err(ApiErr.validationError('The CSV has a quote that is never closed')); + } + // The last row, unless a line break already ended it. + if (started) { + rows.push([...row, value]); + } + return Ok(rows); +} diff --git a/packages/api/src/records/google-connector.test.ts b/packages/api/src/records/google-connector.test.ts index 57380cc1..37f85245 100644 --- a/packages/api/src/records/google-connector.test.ts +++ b/packages/api/src/records/google-connector.test.ts @@ -226,7 +226,7 @@ describe('googleConnector.list', () => { expect(recent.trail).toEqual([{id: 'recent', title: 'Recent'}]); }); - test('shared drives are listed as folders, and inside one its own files', async () => { + test('shared drives are listed as such, and inside one its own files', async () => { const {googleConnector} = await import('./google-connector'); drive.answer = (url) => { if (url.pathname === '/drive/v3/drives') { @@ -241,7 +241,7 @@ describe('googleConnector.list', () => { const drives = (await googleConnector.list(target, query({parent: 'shared-drives'}))).unwrap(); const inside = (await googleConnector.list(target, query({parent: 'drive-0ATeamDrive'}))).unwrap(); - expect(drives.records).toEqual([{id: 'drive-0ATeamDrive', values: {name: 'Team', kind: 'Folder'}, updatedAt: '', hasChildren: true}]); + expect(drives.records).toEqual([{id: 'drive-0ATeamDrive', values: {name: 'Team', kind: 'Shared drive'}, updatedAt: '', hasChildren: true}]); const [list] = requested('/drive/v3/files'); expect(list?.searchParams.get('q')).toBe("'0ATeamDrive' in parents and trashed=false"); expect(list?.searchParams.get('corpora')).toBe('drive'); @@ -419,11 +419,11 @@ describe('googleConnector.get', () => { expect(requested('/v4/spreadsheets/1Budget/values:batchGet')).toHaveLength(0); }); - test('a file too large to export says so', async () => { + test('a file too large to export says so in its content, and opens otherwise', async () => { const {googleConnector} = await import('./google-connector'); - const result = await googleConnector.get(target, '1Huge'); + const record = (await googleConnector.get(target, '1Huge')).unwrap(); - expect(result.isErr() && result.error).toEqual({kind: 'validation_error', statusCode: 400, message: 'This file is too large to read; Google exports at most 10 MB'}); + expect(record.values).toMatchObject({name: 'Everything ever written', kind: 'Document', content: 'This file is too large to read; Google exports at most 10 MB'}); }); test("a disabled API is Google's own message, saying where to enable it", async () => { @@ -444,10 +444,3 @@ describe('googleConnector.get', () => { expect(result.isErr() && result.error).toMatchObject({kind: 'not_found', resource: 'record', id: '1Gone'}); }); }); - -test('files are not written to yet', async () => { - const {googleConnector} = await import('./google-connector'); - const result = await googleConnector.update(target, '1Plan', {name: 'New name'}); - - expect(result.isErr() && result.error.kind).toBe('validation_error'); -}); diff --git a/packages/api/src/records/google-connector.ts b/packages/api/src/records/google-connector.ts index 5dcbb04d..dcbd1441 100644 --- a/packages/api/src/records/google-connector.ts +++ b/packages/api/src/records/google-connector.ts @@ -2,30 +2,33 @@ import {Err, type Result} from 'ts-results-es'; import {ApiErr, type ApiError} from '@proxy/utils'; -import type {Connector, RecordTarget} from './connector'; +import type {CommandRunner, Connector, NewCommandRunner, RecordTarget} from './connector'; import {gmailConnector} from './google-emails'; -import {getFilesRecord, listFilesTree} from './google-files'; +import {createFilesRecord, getFilesRecord, listFilesTree, trashFilesRecord, updateFilesRecord} from './google-files'; +import {rangesConnector} from './google-ranges'; /** * A Google account, fetched live from Google's APIs with the connection's tokens, one collection - * per product: `emails`, Gmail (`google-emails.ts`), and `files`, Drive with Docs, Sheets and - * Slides (`google-files.ts`). Files aren't written yet: that collection offers no writes, and the - * connector turns any down. + * per product: `emails`, Gmail (`google-emails.ts`), `files`, Drive with Docs, Sheets and Slides + * (`google-files.ts`), and `ranges`, a spreadsheet's values in A1 notation (`google-ranges.ts`). + * Commands go to the collection they were run on, since two can share an id, as Trash does. */ -function notWritten(): Promise> { - return Promise.resolve(Err(ApiErr.validationError('Files in Google Drive can’t be changed here yet'))); +// Files are moved to Trash, never deleted: the collection offers no delete. +function notDeleted(): Promise> { + return Promise.resolve(Err(ApiErr.validationError('Files in Google Drive are moved to Trash, never deleted here'))); } const filesConnector: Connector = { list: listFilesTree, get: getFilesRecord, - create: notWritten, - update: notWritten, - remove: notWritten, + create: createFilesRecord, + update: updateFilesRecord, + remove: notDeleted, + commands: {trash: trashFilesRecord}, }; -const COLLECTIONS: Partial> = {emails: gmailConnector, files: filesConnector}; +const COLLECTIONS: Partial> = {emails: gmailConnector, files: filesConnector, ranges: rangesConnector}; function unknownCollection(target: RecordTarget): Promise> { return Promise.resolve(Err(ApiErr.notFound('collection', target.collection.id))); @@ -37,13 +40,33 @@ function connectorOf(target: RecordTarget): Connector { return COLLECTIONS[target.collection.id] ?? unknownConnector; } +const connectors = Object.values(COLLECTIONS).flatMap((connector) => (connector ? [connector] : [])); + +function noCommand(id: string): Promise> { + return Promise.resolve(Err(ApiErr.notFound('command', id))); +} + +// Every command id any collection has, each run by the collection's own. +const commands = Object.fromEntries( + [...new Set(connectors.flatMap((connector) => Object.keys(connector.commands ?? {})))].map((id): [string, CommandRunner] => [ + id, + (target, recordId) => connectorOf(target).commands?.[id]?.(target, recordId) ?? noCommand(id), + ]), +); + +const newCommands = Object.fromEntries( + [...new Set(connectors.flatMap((connector) => Object.keys(connector.newCommands ?? {})))].map((id): [string, NewCommandRunner] => [ + id, + (target, values) => connectorOf(target).newCommands?.[id]?.(target, values) ?? noCommand(id), + ]), +); + export const googleConnector: Connector = { list: (target, query) => connectorOf(target).list(target, query), get: (target, recordId) => connectorOf(target).get(target, recordId), create: (target, values, parent) => connectorOf(target).create(target, values, parent), - update: (target, recordId, values) => connectorOf(target).update(target, recordId, values), + update: (target, recordId, values, before) => connectorOf(target).update(target, recordId, values, before), remove: (target, recordId) => connectorOf(target).remove(target, recordId), - // Only email has commands. - commands: gmailConnector.commands, - newCommands: gmailConnector.newCommands, + commands, + newCommands, }; diff --git a/packages/api/src/records/google-files.test.ts b/packages/api/src/records/google-files.test.ts new file mode 100644 index 00000000..c4d78ffe --- /dev/null +++ b/packages/api/src/records/google-files.test.ts @@ -0,0 +1,481 @@ +import {beforeEach, describe, expect, mock, spyOn, test} from 'bun:test'; +import {Ok, type Result} from 'ts-results-es'; + +import type {ApiError} from '@proxy/utils'; +import {findCollection, findIntegration, type Collection} from '@proxy/integrations'; + +mock.module('@proxy/db/connection', () => ({updateConnectionCredential: mock(), getConnectionCredential: mock(async () => Ok(null))})); + +mock.module('../observability/log', () => ({ + log: {debug: mock(), info: mock(), warn: mock(), error: mock()}, + serializeError: mock(() => ({})), +})); + +mock.module('../utils/env', () => ({env: {NODE_ENV: 'production', APP_URL: 'https://app.example.com', GOOGLE_CLIENT_ID: 'client-1', GOOGLE_CLIENT_SECRET: 'secret-1'}})); + +mock.module('../utils/secret-crypto', () => ({ + encryptSecret: (plaintext: string): Result => Ok(`enc:${plaintext}`), + decryptSecret: (stored: string): Result => Ok(stored.replace(/^enc:/, '')), +})); + +function files(): Collection { + const integration = findIntegration('google'); + const found = integration && findCollection(integration, 'files'); + if (!found) { + throw new Error('No files collection'); + } + return found; +} + +// What a refusal says, or its kind where it says nothing of its own. +function messageOf(result: Result | undefined): string | null { + if (!result || result.isOk()) { + return null; + } + return 'message' in result.error ? result.error.message : result.error.kind; +} + +const CREDENTIAL = {accessToken: 'access-1', refreshToken: 'refresh-1', expiresAt: new Date(Date.now() + 3_600_000).toISOString(), scopes: [], email: 'alex@example.com'}; + +const target = { + connection: {id: 'google-1', integrationId: 'google', account: 'alex@example.com', createdAt: new Date(), tools: null, name: null, defaults: [], credential: `enc:${JSON.stringify(CREDENTIAL)}`}, + collection: files(), +}; + +const MY_DRIVE_ID = '0AFmyDriveRootUk9PVA'; + +const MIME = { + folder: 'application/vnd.google-apps.folder', + document: 'application/vnd.google-apps.document', + spreadsheet: 'application/vnd.google-apps.spreadsheet', + presentation: 'application/vnd.google-apps.presentation', +}; + +type File = {id: string; name: string; mimeType: string; parents: string[]; content: string}; + +// A Drive as Google keeps it, changed by the writes a test makes: Projects in My Drive, holding a +// plan, a budget whose tab Q1 has a header row and two rows, and a deck. +type State = {files: Record; grid: string[][]; made: number}; + +function fresh(): State { + const file = (id: string, name: string, mimeType: string, content = ''): File => ({id, name, mimeType, parents: id === '1Projects' ? [MY_DRIVE_ID] : ['1Projects'], content}); + return { + files: { + '1Projects': file('1Projects', 'Projects', MIME.folder), + '1Plan': file('1Plan', 'Launch plan', MIME.document, '# Launch plan\n\n- Ship it'), + '1Budget': file('1Budget', 'Budget 2026', MIME.spreadsheet), + '1Deck': file('1Deck', 'Board deck', MIME.presentation, 'Slide one'), + }, + grid: [ + ['Item', 'Amount', 'Status'], + ['Rent', '1,200', 'Paid'], + ['Food', '300', ''], + ], + made: 0, + }; +} + +let state = fresh(); + +type Request = {method: string; url: URL; type: string; body: string}; + +// Where Sheets puts an appended row: after the last rows holding anything, from the first column of +// the block they are in. +const google = {requests: [] as Request[], appendColumn: 'A', appendedTitle: 'Q1'}; + +function driveFile(file: File): object { + return { + id: file.id, + name: file.name, + mimeType: file.mimeType, + modifiedTime: '2026-10-08T08:00:00.000Z', + owners: [{displayName: 'Alex Kim'}], + webViewLink: `https://drive.google.com/open?id=${file.id}`, + parents: file.parents, + }; +} + +function notFound(): Response { + return Response.json({error: {code: 404, message: 'File not found.', errors: [{reason: 'notFound'}]}}, {status: 404}); +} + +// The values of rows `from` to `to`, A1 ranges as `rowsRange` writes them. +function rowsOf(range: string): string[][] { + const [, from = '1', to = '1'] = /!A(\d+):ZZ(\d+)$/.exec(range) ?? []; + return state.grid.slice(Number(from) - 1, Number(to)); +} + +function cellOf(range: string): {row: number; column: number} { + const [, letter = 'A', row = '1'] = /!([A-Z]+)(\d+)$/.exec(range) ?? []; + return {row: Number(row), column: letter.charCodeAt(0) - 65}; +} + +// The metadata and the content of a multipart upload. +function multipartOf(request: Request): {metadata: {name: string; mimeType: string; parents: string[]}; type: string; content: string} { + const boundary = /boundary=(.+)$/.exec(request.type)?.[1] ?? ''; + const [, first = '', second = ''] = request.body.split(`--${boundary}`); + const [, json = '{}'] = first.split('\r\n\r\n'); + const [head = '', ...content] = second.split('\r\n\r\n'); + return {metadata: JSON.parse(json.trim()), type: head.replace('\r\nContent-Type: ', ''), content: content.join('\r\n\r\n').replace(/\r\n$/, '')}; +} + +function make(name: string, mimeType: string, parents: string[], content: string): Response { + state.made += 1; + const file = {id: `1New${state.made}`, name, mimeType, parents, content}; + state.files[file.id] = file; + return Response.json(driveFile(file)); +} + +function answer(request: Request): Response { + const {method, url} = request; + const path = decodeURIComponent(url.pathname); + if (path === '/drive/v3/files/root') { + return Response.json({id: MY_DRIVE_ID}); + } + if (method === 'POST' && path === '/drive/v3/files') { + const {name, mimeType, parents} = JSON.parse(request.body); + return make(name, mimeType, parents, ''); + } + if (method === 'POST' && path === '/upload/drive/v3/files') { + const {metadata, content} = multipartOf(request); + return make(metadata.name, metadata.mimeType, metadata.parents, content); + } + const exported = /^\/drive\/v3\/files\/([^/]+)\/export$/.exec(path)?.[1]; + if (exported) { + return new Response(state.files[exported]?.content ?? ''); + } + const id = /^\/(?:upload\/)?drive\/v3\/files\/([^/]+)$/.exec(path)?.[1]; + const file = id ? state.files[id] : undefined; + if (id && !file) { + return notFound(); + } + if (file && method === 'PATCH' && path.startsWith('/upload/')) { + file.content = request.body; + return Response.json(driveFile(file)); + } + if (file && method === 'PATCH') { + Object.assign(file, JSON.parse(request.body)); + return Response.json(driveFile(file)); + } + if (file) { + return Response.json(driveFile(file)); + } + if (path === '/v4/spreadsheets/1Budget') { + return Response.json({sheets: [{properties: {sheetId: 0, title: 'Q1', gridProperties: {rowCount: 100, columnCount: 3}}}]}); + } + if (path === '/v4/spreadsheets/1Budget/values:batchGet') { + return Response.json({valueRanges: url.searchParams.getAll('ranges').map((range) => ({range, values: rowsOf(range)}))}); + } + if (method === 'POST' && path.endsWith(':append')) { + const [row = []] = JSON.parse(request.body).values; + const shift = google.appendColumn.charCodeAt(0) - 65; + state.grid.push([...Array.from({length: shift}, () => ''), ...row]); + const at = state.grid.length; + const last = String.fromCharCode(65 + shift + row.length - 1); + return Response.json({updates: {updatedRange: `${google.appendedTitle}!${google.appendColumn}${at}:${last}${at}`, updatedRows: 1}}); + } + if (method === 'PUT' && path.startsWith('/v4/spreadsheets/1Budget/values/')) { + const {range, values} = JSON.parse(request.body); + const [, row = '1'] = /!A(\d+):[A-Z]+\d+$/.exec(range) ?? []; + state.grid[Number(row) - 1] = values[0]; + return Response.json({updatedRange: range, updatedData: {range, values}}); + } + if (method === 'POST' && path === '/v4/spreadsheets/1Budget/values:batchUpdate') { + for (const {range, values} of JSON.parse(request.body).data) { + const {row, column} = cellOf(range); + const cells = state.grid[row - 1] ?? []; + cells[column] = values[0][0]; + state.grid[row - 1] = cells; + } + return Response.json({totalUpdatedCells: 1}); + } + if (method === 'POST' && path === '/v4/spreadsheets/1Budget:batchUpdate') { + return Response.json({replies: [{}]}); + } + return notFound(); +} + +spyOn(globalThis, 'fetch').mockImplementation((async (input: string | URL | Request, init?: RequestInit) => { + const url = new URL(typeof input === 'string' ? input : input instanceof URL ? input.href : input.url); + const headers = new Headers(init?.headers); + const request = {method: init?.method ?? 'GET', url, type: headers.get('content-type') ?? '', body: typeof init?.body === 'string' ? init.body : ''}; + google.requests.push(request); + return answer(request); +}) as typeof fetch); + +// The writes made: every request but a GET. +function writes(): Request[] { + return google.requests.filter((request) => request.method !== 'GET'); +} + +beforeEach(() => { + state = fresh(); + google.requests = []; + google.appendColumn = 'A'; + google.appendedTitle = 'Q1'; +}); + +describe('creating files', () => { + test('a folder is made inside the folder given, in shared drives too, and answered as opening it would', async () => { + const {googleConnector} = await import('./google-connector'); + const record = (await googleConnector.create(target, {name: 'Receipts', kind: 'Folder', content: ''}, '1Projects')).unwrap(); + + const [made] = writes(); + expect(made?.url.pathname).toBe('/drive/v3/files'); + expect(made?.url.searchParams.get('supportsAllDrives')).toBe('true'); + expect(JSON.parse(made?.body ?? '')).toEqual({name: 'Receipts', mimeType: MIME.folder, parents: ['1Projects']}); + expect(record).toMatchObject({id: '1New1', values: {name: 'Receipts', kind: 'Folder', content: ''}, hasChildren: true}); + }); + + test('without a parent, or in My Drive, a file goes at the top of My Drive', async () => { + const {googleConnector} = await import('./google-connector'); + await googleConnector.create(target, {name: 'Inbox', kind: 'Folder'}, null); + await googleConnector.create(target, {name: 'Inbox 2', kind: 'Folder'}, 'root'); + + expect(writes().map((request) => JSON.parse(request.body).parents)).toEqual([['root'], ['root']]); + }); + + test('a document is uploaded as Markdown and a spreadsheet as CSV, each with the Google type Drive converts it to', async () => { + const {googleConnector} = await import('./google-connector'); + const doc = (await googleConnector.create(target, {name: 'Notes', kind: 'Document', content: '# Notes\n\nIt’s a start.'}, '1Projects')).unwrap(); + await googleConnector.create(target, {name: 'Costs', kind: 'Spreadsheet', content: 'Item,Amount\nRent,"1,200"'}, 'drive-0ATeamDrive'); + + const [docUpload, sheetUpload] = writes(); + expect(docUpload?.url.pathname).toBe('/upload/drive/v3/files'); + expect(Object.fromEntries(docUpload?.url.searchParams ?? [])).toMatchObject({uploadType: 'multipart', supportsAllDrives: 'true'}); + expect(docUpload?.type).toMatch(/^multipart\/related; boundary=/); + expect(multipartOf(docUpload as Request)).toEqual({ + metadata: {name: 'Notes', mimeType: MIME.document, parents: ['1Projects']}, + type: 'text/markdown; charset=UTF-8', + content: '# Notes\n\nIt’s a start.', + }); + expect(multipartOf(sheetUpload as Request)).toEqual({ + metadata: {name: 'Costs', mimeType: MIME.spreadsheet, parents: ['0ATeamDrive']}, + type: 'text/csv; charset=UTF-8', + content: 'Item,Amount\nRent,"1,200"', + }); + expect(doc.values).toMatchObject({name: 'Notes', kind: 'Document', content: '# Notes\n\nIt’s a start.'}); + }); + + test('an empty document is made without an upload', async () => { + const {googleConnector} = await import('./google-connector'); + await googleConnector.create(target, {name: 'Blank', kind: 'Document', content: ''}, '1Projects'); + + expect(writes().map((request) => [request.url.pathname, JSON.parse(request.body).mimeType])).toEqual([['/drive/v3/files', MIME.document]]); + }); + + test('a row goes at the end of a tab, its lines under the headers they name, and opens as a row', async () => { + const {googleConnector} = await import('./google-connector'); + const record = (await googleConnector.create(target, {name: '', kind: '', content: 'Status: Due\nItem: Travel'}, 'sheet-1Budget-0')).unwrap(); + + const [append] = writes(); + expect(decodeURIComponent(append?.url.pathname ?? '')).toBe("/v4/spreadsheets/1Budget/values/'Q1':append"); + expect(Object.fromEntries(append?.url.searchParams ?? [])).toEqual({valueInputOption: 'USER_ENTERED', insertDataOption: 'INSERT_ROWS'}); + expect(JSON.parse(append?.body ?? '')).toEqual({majorDimension: 'ROWS', values: [['Travel', '', 'Due']]}); + expect(record).toMatchObject({id: 'row-1Budget-0-4', values: {name: 'Travel', kind: 'Row', content: 'Item: Travel\nStatus: Due'}}); + }); + + test('a row Sheets appends past column A, under the last block of rows, is written again from A', async () => { + google.appendColumn = 'B'; + const {googleConnector} = await import('./google-connector'); + const record = (await googleConnector.create(target, {content: 'Item: Travel\nStatus: Due'}, 'sheet-1Budget-0')).unwrap(); + + const [, rewrite] = writes(); + expect(rewrite?.method).toBe('PUT'); + expect(JSON.parse(rewrite?.body ?? '')).toEqual({range: "'Q1'!A4:D4", majorDimension: 'ROWS', values: [['Travel', '', 'Due', '']]}); + expect(record).toMatchObject({id: 'row-1Budget-0-4', values: {content: 'Item: Travel\nStatus: Due'}}); + }); + + test("the new row's number is read after the tab's title, whatever the title holds", async () => { + google.appendedTitle = "'a!B7'"; + const {googleConnector} = await import('./google-connector'); + const record = (await googleConnector.create(target, {content: 'Item: Travel'}, 'sheet-1Budget-0')).unwrap(); + + expect(record.id).toBe('row-1Budget-0-4'); + }); + + test('a header repeated is told apart by its column, and a row written back changes the cell it names', async () => { + state.grid = [ + ['Item', 'Status', 'Status'], + ['Rent', 'Open', 'Closed'], + ]; + const {googleConnector} = await import('./google-connector'); + const row = (await googleConnector.get(target, 'row-1Budget-0-2')).unwrap(); + expect(row.values.content).toBe('Item: Rent\nStatus: Open\nStatus (C): Closed'); + + await googleConnector.update(target, 'row-1Budget-0-2', {content: 'Item: Rent\nStatus: Open\nStatus (C): Done'}); + expect(JSON.parse(writes()[0]?.body ?? '').data).toEqual([{range: "'Q1'!C2", majorDimension: 'ROWS', values: [['Done']]}]); + }); + + test('a row given a name is turned down, since its name is its first value', async () => { + const {googleConnector} = await import('./google-connector'); + const result = await googleConnector.create(target, {name: 'Travel', content: 'Status: Due'}, 'sheet-1Budget-0'); + + expect(messageOf(result)).toBe('A row’s name is its first value: write it in content'); + expect(writes()).toEqual([]); + }); + + test('a row naming a header the tab lacks is turned down before anything is written', async () => { + const {googleConnector} = await import('./google-connector'); + const result = await googleConnector.create(target, {content: 'Colour: red'}, 'sheet-1Budget-0'); + + expect(messageOf(result)).toStartWith('“Colour: red” doesn’t start with one of the names: Item, Amount, Status'); + expect(writes()).toEqual([]); + }); + + test('nothing is made in the places without a folder of their own, in a spreadsheet, or in a file', async () => { + const {googleConnector} = await import('./google-connector'); + const folder = {name: 'New', kind: 'Folder', content: ''}; + const refusals = await Promise.all([ + googleConnector.create(target, folder, 'recent'), + googleConnector.create(target, folder, 'shared-with-me'), + googleConnector.create(target, folder, 'shared-drives'), + googleConnector.create(target, folder, '1Budget'), + googleConnector.create(target, folder, '1Plan'), + googleConnector.create(target, {name: 'New', kind: 'Presentation', content: ''}, '1Projects'), + googleConnector.create(target, {name: '', kind: 'Document', content: ''}, '1Projects'), + googleConnector.create(target, {name: 'New', kind: 'Folder', content: 'Something'}, '1Projects'), + ]); + + expect(refusals.map((result) => messageOf(result))).toEqual([ + 'Nothing can be made in Recent: make it in a folder, My Drive or a shared drive', + 'Nothing can be made in Shared with me: make it in a folder, My Drive or a shared drive', + 'Nothing can be made in Shared drives: make it in a folder, My Drive or a shared drive', + 'Rows go in one of the spreadsheet’s tabs: open it and pick one as parent', + 'A document holds nothing: make it in a folder', + 'Say what to make as kind: Folder, Document or Spreadsheet', + 'A new document needs a name', + 'A folder has no content', + ]); + expect(writes()).toEqual([]); + }); +}); + +describe('editing files', () => { + test('a file is renamed with its metadata alone', async () => { + const {googleConnector} = await import('./google-connector'); + const record = (await googleConnector.update(target, '1Deck', {name: 'Board deck v2', kind: 'Presentation', content: 'Slide one'})).unwrap(); + + const [rename] = writes(); + expect([rename?.method, rename?.url.pathname, JSON.parse(rename?.body ?? '')]).toEqual(['PATCH', '/drive/v3/files/1Deck', {name: 'Board deck v2'}]); + expect(rename?.url.searchParams.get('supportsAllDrives')).toBe('true'); + expect(writes()).toHaveLength(1); + expect(record.values.name).toBe('Board deck v2'); + }); + + test('a document’s content is replaced whole by uploading Markdown, and only when written differently', async () => { + const {googleConnector} = await import('./google-connector'); + await googleConnector.update(target, '1Plan', {name: 'Launch plan', content: '# Launch plan\n\n- Ship it'}); + expect(writes()).toEqual([]); + + const record = (await googleConnector.update(target, '1Plan', {name: 'Launch plan', content: '# Launch plan\n\n- Shipped'})).unwrap(); + + const [upload] = writes(); + expect([upload?.method, upload?.url.pathname, upload?.url.searchParams.get('uploadType'), upload?.type, upload?.body]).toEqual([ + 'PATCH', + '/upload/drive/v3/files/1Plan', + 'media', + 'text/markdown; charset=UTF-8', + '# Launch plan\n\n- Shipped', + ]); + expect(record.values.content).toBe('# Launch plan\n\n- Shipped'); + }); + + test('a presentation’s or a spreadsheet’s content, a kind or an empty name is turned down before anything is written', async () => { + const {googleConnector} = await import('./google-connector'); + const refusals = await Promise.all([ + googleConnector.update(target, '1Deck', {name: 'Board deck v2', content: 'Slide two'}), + googleConnector.update(target, '1Budget', {content: 'Q1: 3 rows'}), + googleConnector.update(target, '1Plan', {kind: 'Spreadsheet'}), + googleConnector.update(target, '1Plan', {name: ' '}), + googleConnector.update(target, 'root', {name: 'Mine'}), + ]); + + expect(refusals.map((result) => messageOf(result))).toEqual([ + 'A presentation’s content is only read here; its name can be changed', + 'A spreadsheet’s content is in its tabs: change its rows, or write a range', + 'What a file is can’t be changed: this is a Document', + 'A file needs a name', + 'The places at the top and shared drives aren’t changed here, only what is in them', + ]); + expect(writes()).toEqual([]); + }); + + test('a rename leaves the content alone when it is as it read, however it reads now', async () => { + const {googleConnector} = await import('./google-connector'); + const before = (await googleConnector.get(target, '1Plan')).unwrap(); + const plan = state.files['1Plan']; + if (plan) { + plan.content = '# Launch plan\n\n- Shipped, by someone else meanwhile'; + } + google.requests = []; + await googleConnector.update(target, '1Plan', {...before.values, name: 'Plan'}, before); + + expect(writes().map((request) => [request.method, request.url.pathname])).toEqual([['PATCH', '/drive/v3/files/1Plan']]); + expect(state.files['1Plan']?.content).toBe('# Launch plan\n\n- Shipped, by someone else meanwhile'); + }); + + test('a row changes only in the cells written differently, a cell written empty cleared', async () => { + const {googleConnector} = await import('./google-connector'); + const record = (await googleConnector.update(target, 'row-1Budget-0-3', {name: 'Food', kind: 'Row', content: 'Item: Food\nAmount: 350\nStatus: Paid'})).unwrap(); + + const [update] = writes(); + expect(update?.url.pathname).toBe('/v4/spreadsheets/1Budget/values:batchUpdate'); + expect(JSON.parse(update?.body ?? '')).toEqual({ + valueInputOption: 'USER_ENTERED', + data: [ + {range: "'Q1'!B3", majorDimension: 'ROWS', values: [['350']]}, + {range: "'Q1'!C3", majorDimension: 'ROWS', values: [['Paid']]}, + ], + }); + expect(record.values.content).toBe('Item: Food\nAmount: 350\nStatus: Paid'); + + await googleConnector.update(target, 'row-1Budget-0-2', {content: 'Status:'}); + expect(JSON.parse(writes()[1]?.body ?? '').data).toEqual([{range: "'Q1'!C2", majorDimension: 'ROWS', values: [['']]}]); + }); + + test('a row written as it reads changes nothing, and its name is its first value', async () => { + const {googleConnector} = await import('./google-connector'); + await googleConnector.update(target, 'row-1Budget-0-2', {name: 'Rent', content: 'Item: Rent\nAmount: 1,200\nStatus: Paid'}); + const renamed = await googleConnector.update(target, 'row-1Budget-0-2', {name: 'Mortgage'}); + + expect(messageOf(renamed)).toBe('A row’s name is its first value: change it in content'); + expect(writes()).toEqual([]); + }); + + test('a tab is renamed through the spreadsheet', async () => { + const {googleConnector} = await import('./google-connector'); + await googleConnector.update(target, 'sheet-1Budget-0', {name: 'Q1 2026', content: '100 rows, 3 columns'}); + + const [rename] = writes(); + expect(rename?.url.pathname).toBe('/v4/spreadsheets/1Budget:batchUpdate'); + expect(JSON.parse(rename?.body ?? '')).toEqual({requests: [{updateSheetProperties: {properties: {sheetId: 0, title: 'Q1 2026'}, fields: 'title'}}]}); + }); +}); + +describe('moving files to Trash', () => { + test('a file goes to Trash, never deleted, and leaves the tree', async () => { + const {googleConnector} = await import('./google-connector'); + const result = await googleConnector.commands?.trash?.({...target}, '1Plan'); + + expect(result?.unwrap()).toBeNull(); + expect(writes().map((request) => [request.method, request.url.pathname, JSON.parse(request.body)])).toEqual([['PATCH', '/drive/v3/files/1Plan', {trashed: true}]]); + }); + + test('places, tabs and rows are not files to trash, and deleting is refused', async () => { + const {googleConnector} = await import('./google-connector'); + const tab = await googleConnector.commands?.trash?.(target, 'sheet-1Budget-0'); + const removed = await googleConnector.remove(target, '1Plan'); + + expect(tab?.isErr() && tab.error.kind).toBe('validation_error'); + expect(messageOf(removed)).toBe('Files in Google Drive are moved to Trash, never deleted here'); + expect(writes()).toEqual([]); + }); + + test('a file that is gone is not found by its id', async () => { + const {googleConnector} = await import('./google-connector'); + const result = await googleConnector.commands?.trash?.(target, '1Gone'); + + expect(result?.isErr() && result.error).toMatchObject({kind: 'not_found', id: '1Gone'}); + }); +}); diff --git a/packages/api/src/records/google-files.ts b/packages/api/src/records/google-files.ts index 753020b9..45337913 100644 --- a/packages/api/src/records/google-files.ts +++ b/packages/api/src/records/google-files.ts @@ -3,6 +3,7 @@ import {Err, Ok, type Result} from 'ts-results-es'; import {ApiErr, type ApiError, Do} from '@proxy/utils'; import { + createFile, DRIVE_ID, driveQuoted, exportFile, @@ -14,14 +15,16 @@ import { listSharedDrives, MIME_TYPES, PAGE_SIZE, + replaceContent, + updateFile, type DriveFile, type FileParents, type SharedDrive, } from '../connections/google/drive-api'; -import {callGoogle} from '../connections/google/google-client'; -import {columnLetters, getRanges, getSheets, rowsRange, type Sheet} from '../connections/google/sheets-api'; -import type {Crumb, DataRecord, ListQuery, RecordPage, RecordTarget} from './connector'; -import {valueLines} from './value-lines'; +import {callGoogle, TOO_LARGE_TO_READ} from '../connections/google/google-client'; +import {appendRow, cellRange, columnLetters, getRanges, getSheets, renameSheet, rowsRange, updateCells, type Sheet} from '../connections/google/sheets-api'; +import type {CommandRunner, Crumb, DataRecord, ListQuery, RecordPage, RecordTarget, RecordValues} from './connector'; +import {readValueLines, valueLines} from './value-lines'; /** * A Google account's Drive as the `files` collection: a tree, fetched live. At the top are four @@ -32,7 +35,14 @@ import {valueLines} from './value-lines'; * A document opens as Markdown and a presentation as text, as Drive exports them, up to Drive's * 10 MB; a spreadsheet opens into its tabs, and a tab into its rows, 50 to a page, each row's values * as "Header: value" lines, the headers from the tab's first row. Other files open as what Drive - * knows of them, with a link. Nothing is written yet. + * knows of them, with a link. + * + * Folders, documents and spreadsheets are made inside a folder, My Drive or a shared drive: a + * document from Markdown and a spreadsheet from CSV, which Drive converts as it uploads them. A row + * is made inside a tab, its "Header: value" lines put under the headers. Any file can be renamed, + * and a tab; a document's content is replaced whole by uploading Markdown again, which drops its + * comments and suggestions; a row changes only in the cells written differently. A presentation's + * and a spreadsheet's content are only read. Files go to Trash, never deleted for good. */ const PLACES = [ @@ -144,7 +154,7 @@ function placeRecord(place: Place): DataRecord { } function driveRecord(drive: SharedDrive): DataRecord { - return {id: `drive-${drive.id}`, values: {name: drive.name, kind: 'Folder'}, updatedAt: '', hasChildren: true}; + return {id: `drive-${drive.id}`, values: {name: drive.name, kind: 'Shared drive'}, updatedAt: '', hasChildren: true}; } function sheetId(file: DriveFile, sheet: Sheet): string { @@ -169,9 +179,38 @@ function sheetRecord(file: DriveFile, sheet: Sheet): DataRecord { }; } +// The last column a row is read and written in: ZZ, as `rowsRange` reads. +const MAX_COLUMNS = 702; + +// What a row's values go by: the tab's header, or, for a column without one, its letter. +function columnName(headers: string[], column: number): string { + return headers[column] || `Column ${columnLetters(column)}`; +} + +/** + * Every column of a tab a row's lines can name, as far as its grid or its headers go, each name + * once, so a line is read back into the column it was written from: a header repeated, or the + * same as a column without one goes by, goes on with its column's letter, "Status (E)". + */ +function columnNames(headers: string[], sheet: Sheet): string[] { + const count = Math.min(Math.max(headers.length, sheet.columnCount), MAX_COLUMNS); + const names: string[] = []; + const taken = new Set(); + for (let column = 0; column < count; column += 1) { + let name = columnName(headers, column); + while (taken.has(name)) { + name = `${name} (${columnLetters(column)})`; + } + taken.add(name); + names.push(name); + } + return names; +} + // A row's values under the tab's headers; a column without one goes by its letter. function rowRecord(file: DriveFile, sheet: Sheet, headers: string[], row: number, cells: string[]): DataRecord { - const content = valueLines(cells.map((cell, column): [string, string] => [headers[column] || `Column ${columnLetters(column)}`, cell])); + const names = columnNames(headers, {...sheet, columnCount: Math.max(sheet.columnCount, cells.length)}); + const content = valueLines(cells.map((cell, column): [string, string] => [names[column] ?? columnName(headers, column), cell])); return { id: `row-${file.id}-${sheet.sheetId}-${row}`, // Rows have no time of their own; the spreadsheet's is when anything in it changed. @@ -401,15 +440,25 @@ export function listFilesTree(target: RecordTarget, query: ListQuery): Promise> { + const content = await exportFile(drive.token, fileId, type); + if (content.isErr() && content.error.kind === 'validation_error' && content.error.message === TOO_LARGE_TO_READ) { + return Ok(TOO_LARGE_TO_READ); + } + return content; +} + // A document as Markdown, a presentation as text, a spreadsheet's tabs one a line; other files have none. function contentOf(drive: Drive, file: DriveFile): Promise> { return Do(async ($) => { const kind = kindOf(file.mimeType); if (kind === 'Document') { - return $(await exportFile(drive.token, file.id, 'text/markdown')); + return $(await exported(drive, file.id, 'text/markdown')); } if (kind === 'Presentation') { - return $(await exportFile(drive.token, file.id, 'text/plain')); + return $(await exported(drive, file.id, 'text/plain')); } if (kind === 'Spreadsheet') { const sheets = $(await getSheets(drive.token, file.id)); @@ -456,3 +505,253 @@ export function getFilesRecord(target: RecordTarget, recordId: string): Promise< }), ); } + +// ---- Writing -------------------------------------------------------------------------------- + +// What a new file can be, the Google type it is made as, and what its content is uploaded as. +const MADE: Partial> = { + Folder: {mimeType: MIME_TYPES.folder, upload: null}, + Document: {mimeType: MIME_TYPES.document, upload: 'text/markdown'}, + Spreadsheet: {mimeType: MIME_TYPES.spreadsheet, upload: 'text/csv'}, +}; + +const KINDS_MADE = 'Folder, Document or Spreadsheet'; + +function refuse(message: string): Promise> { + return Promise.resolve(Err(ApiErr.validationError(message))); +} + +// A folder, document or spreadsheet, made in the folder with Drive's id `parentId`, answered as +// opening it would. +function makeFile(drive: Drive, parentId: string, values: RecordValues): Promise> { + return Do(async ($) => { + const kind = values.kind ?? ''; + const made = MADE[kind]; + if (!made) { + return $(Err(ApiErr.validationError(`Say what to make as kind: ${KINDS_MADE}`))); + } + const name = (values.name ?? '').trim(); + if (name === '') { + return $(Err(ApiErr.validationError(`A new ${kind.toLowerCase()} needs a name`))); + } + const content = values.content ?? ''; + if (made.upload === null && content.trim() !== '') { + return $(Err(ApiErr.validationError('A folder has no content'))); + } + const upload = made.upload === null || content === '' ? null : {type: made.upload, body: content}; + const file = $(await createFile(drive.token, {name, mimeType: made.mimeType, parent: parentId}, upload)); + return $(await getRef(drive, {kind: 'file', fileId: file.id})); + }); +} + +// The headers of a tab's first row, and every column its rows' lines can name. +function headersOf(drive: Drive, file: DriveFile, sheet: Sheet): Promise> { + return Do(async ($) => { + const [[headers = []] = []] = $(await getRanges(drive.token, file.id, [rowsRange(sheet.title, 1, 1)])); + return {headers, names: columnNames(headers, sheet)}; + }); +} + +// A new row at the end of a tab, its lines put under the headers they name. +function addRow(drive: Drive, ref: {fileId: string; sheetId: number}, values: RecordValues): Promise> { + return Do(async ($) => { + if ((values.kind ?? '') !== '' && values.kind !== 'Row') { + return $(Err(ApiErr.validationError('A tab holds rows: leave kind out, or make it Row'))); + } + if ((values.name ?? '') !== '') { + return $(Err(ApiErr.validationError('A row’s name is its first value: write it in content'))); + } + const {file, sheet} = $(await getSheet(drive, ref.fileId, ref.sheetId)); + const {headers, names} = $(await headersOf(drive, file, sheet)); + if (isBlank(headers)) { + return $(Err(ApiErr.validationError(`“${sheet.title}” has no headers in its first row; write a range instead`))); + } + const written = $(readValueLines(values.content ?? '', names)); + const cells = names.map((name) => written[name] ?? ''); + const last = cells.findLastIndex((cell) => cell !== ''); + if (last === -1) { + return $(Err(ApiErr.validationError('Write the row’s values in content, one "Header: value" a line'))); + } + const row = $(await appendRow(drive.token, file.id, sheet.title, cells.slice(0, last + 1))); + return $(await getRef(drive, {kind: 'row', fileId: file.id, sheetId: sheet.sheetId, row})); + }); +} + +function createIn(drive: Drive, ref: FileRef, values: RecordValues): Promise> { + if (ref.kind === 'place') { + if (ref.place.id !== MY_DRIVE.id) { + return refuse(`Nothing can be made in ${ref.place.name}: make it in a folder, My Drive or a shared drive`); + } + return makeFile(drive, MY_DRIVE.id, values); + } + if (ref.kind === 'drive') { + return makeFile(drive, ref.driveId, values); + } + if (ref.kind === 'sheet') { + return addRow(drive, ref, values); + } + if (ref.kind === 'row') { + return refuse('A row holds nothing: add rows to its tab'); + } + return Do(async ($) => { + const file = $(await getFile(drive.token, ref.fileId)); + const kind = kindOf(file.mimeType); + if (kind === 'Spreadsheet') { + return $(Err(ApiErr.validationError('Rows go in one of the spreadsheet’s tabs: open it and pick one as parent'))); + } + if (kind !== 'Folder') { + return $(Err(ApiErr.validationError(`A ${kind.toLowerCase()} holds nothing: make it in a folder`))); + } + return $(await makeFile(drive, file.id, values)); + }); +} + +/** + * A new file inside `parent`, My Drive when none is given: a folder, document or spreadsheet in a + * folder, My Drive or a shared drive, or a row in a tab. + */ +export function createFilesRecord(target: RecordTarget, values: RecordValues, parent: string | null): Promise> { + return withDrive(target, (drive) => + Do(async ($) => { + const where = parent ?? MY_DRIVE.id; + const ref = $(requireRef(where)); + return $((await createIn(drive, ref, values)).mapErr(notFoundAs(where))); + }), + ); +} + +// A kind written differently from the record's: what a record is can't be changed. +function requireKind(current: string, written: string | undefined): Result { + if (written === undefined || written === '' || written === current) { + return Ok(undefined); + } + return Err(ApiErr.validationError(`What a file is can’t be changed: this is a ${current}`)); +} + +// A new name, when it is written differently: none at all is turned down. +function renamed(current: string, written: string | undefined): Result { + if (written === undefined || written === current) { + return Ok(null); + } + if (written.trim() === '') { + return Err(ApiErr.validationError('A file needs a name')); + } + return Ok(written.trim()); +} + +// What can't be written into each kind of file but a document, and what can be done instead. +const CONTENT_REFUSALS: Partial> = { + Presentation: 'A presentation’s content is only read here; its name can be changed', + Spreadsheet: 'A spreadsheet’s content is in its tabs: change its rows, or write a range', + Folder: 'A folder has no content; its name can be changed', +}; + +const OTHER_FILE_REFUSAL = 'Only a document’s content can be written here; this file’s name can be changed'; + +/** + * A file renamed, and a document's content replaced, each only where it is written differently + * from how it read, `before`, so a rename doesn't upload the content as it was read over a change + * made since; without `before`, from how it reads now. + */ +function updateFileRef(drive: Drive, fileId: string, values: RecordValues, before: DataRecord | undefined): Promise> { + return Do(async ($) => { + const file = $(await getFile(drive.token, fileId)); + const kind = kindOf(file.mimeType); + $(requireKind(kind, values.kind)); + const name = $(renamed(file.name, values.name)); + const content = values.content ?? null; + const read = before?.values.content ?? $(await contentOf(drive, file)); + const replaced = content !== null && content !== read; + if (replaced && kind !== 'Document') { + return $(Err(ApiErr.validationError(CONTENT_REFUSALS[kind] ?? OTHER_FILE_REFUSAL))); + } + if (name !== null) { + $(await updateFile(drive.token, file.id, {name})); + } + if (replaced) { + $(await replaceContent(drive.token, file.id, {type: 'text/markdown', body: content})); + } + return $(await getRef(drive, {kind: 'file', fileId: file.id})); + }); +} + +// A tab renamed; its rows are changed one by one. +function updateSheet(drive: Drive, ref: {fileId: string; sheetId: number}, values: RecordValues): Promise> { + return Do(async ($) => { + const {file, sheet} = $(await getSheet(drive, ref.fileId, ref.sheetId)); + $(requireKind('Sheet', values.kind)); + if (values.content !== undefined && values.content !== sheetSize(sheet)) { + return $(Err(ApiErr.validationError('A tab’s content is its size: change its rows one by one, or write a range'))); + } + const name = $(renamed(sheet.title, values.name)); + if (name === null) { + return sheetRecord(file, sheet); + } + $(await renameSheet(drive.token, file.id, sheet.sheetId, name)); + return $(await getRef(drive, {kind: 'sheet', fileId: file.id, sheetId: sheet.sheetId})); + }); +} + +// Only the cells of the headers written differently from how they read change; one written empty +// is cleared, and one left out stays as it is. +function updateRow(drive: Drive, ref: {fileId: string; sheetId: number; row: number}, values: RecordValues): Promise> { + return Do(async ($) => { + const {file, sheet} = $(await getSheet(drive, ref.fileId, ref.sheetId)); + $(requireKind('Row', values.kind)); + const [[headers = []] = [], [cells = []] = []] = $(await getRanges(drive.token, file.id, [rowsRange(sheet.title, 1, 1), rowsRange(sheet.title, ref.row, ref.row)])); + if (isBlank(cells)) { + return $(Err(ApiErr.notFound('row'))); + } + const current = rowRecord(file, sheet, headers, ref.row, cells); + if (values.name !== undefined && values.name !== current.values.name) { + return $(Err(ApiErr.validationError('A row’s name is its first value: change it in content'))); + } + const names = columnNames(headers, sheet); + const written = $(readValueLines(values.content ?? '', names)); + const changed = names.flatMap((name, column) => { + const value = written[name]; + return value !== undefined && value !== (cells[column] ?? '') ? [{range: cellRange(sheet.title, column, ref.row), value}] : []; + }); + if (changed.length === 0) { + return current; + } + $(await updateCells(drive.token, file.id, changed)); + return $(await getRef(drive, {kind: 'row', fileId: file.id, sheetId: sheet.sheetId, row: ref.row})); + }); +} + +function updateRef(drive: Drive, ref: FileRef, values: RecordValues, before: DataRecord | undefined): Promise> { + if (ref.kind === 'place' || ref.kind === 'drive') { + return refuse('The places at the top and shared drives aren’t changed here, only what is in them'); + } + if (ref.kind === 'sheet') { + return updateSheet(drive, ref, values); + } + if (ref.kind === 'row') { + return updateRow(drive, ref, values); + } + return updateFileRef(drive, ref.fileId, values, before); +} + +/** A file, tab or row changed where it is written differently, answered as opening it would. */ +export function updateFilesRecord(target: RecordTarget, recordId: string, values: RecordValues, before?: DataRecord): Promise> { + return withDrive(target, (drive) => + Do(async ($) => { + const ref = $(requireRef(recordId)); + return $((await updateRef(drive, ref, values, before)).mapErr(notFoundAs(recordId))); + }), + ); +} + +/** Moves a file or folder to Trash, where Drive keeps it 30 days; it leaves the tree. */ +export const trashFilesRecord: CommandRunner = (target, recordId) => + withDrive(target, (drive) => + Do(async ($) => { + const ref = $(requireRef(recordId)); + if (ref.kind !== 'file') { + return $(Err(ApiErr.validationError('Only files and folders go to Trash, not places, shared drives, tabs or rows'))); + } + $((await updateFile(drive.token, ref.fileId, {trashed: true})).mapErr(notFoundAs(recordId))); + return null; + }), + ); diff --git a/packages/api/src/records/google-ranges.test.ts b/packages/api/src/records/google-ranges.test.ts new file mode 100644 index 00000000..873f1862 --- /dev/null +++ b/packages/api/src/records/google-ranges.test.ts @@ -0,0 +1,188 @@ +import {beforeEach, describe, expect, mock, spyOn, test} from 'bun:test'; +import {Ok, type Result} from 'ts-results-es'; + +import type {ApiError} from '@proxy/utils'; +import {findCollection, findIntegration, type Collection} from '@proxy/integrations'; + +mock.module('@proxy/db/connection', () => ({updateConnectionCredential: mock(), getConnectionCredential: mock(async () => Ok(null))})); + +mock.module('../observability/log', () => ({ + log: {debug: mock(), info: mock(), warn: mock(), error: mock()}, + serializeError: mock(() => ({})), +})); + +mock.module('../utils/env', () => ({env: {NODE_ENV: 'production', APP_URL: 'https://app.example.com', GOOGLE_CLIENT_ID: 'client-1', GOOGLE_CLIENT_SECRET: 'secret-1'}})); + +mock.module('../utils/secret-crypto', () => ({ + encryptSecret: (plaintext: string): Result => Ok(`enc:${plaintext}`), + decryptSecret: (stored: string): Result => Ok(stored.replace(/^enc:/, '')), +})); + +function ranges(): Collection { + const integration = findIntegration('google'); + const found = integration && findCollection(integration, 'ranges'); + if (!found) { + throw new Error('No ranges collection'); + } + return found; +} + +// What a refusal says, or its kind where it says nothing of its own. +function messageOf(result: Result | undefined): string | null { + if (!result || result.isOk()) { + return null; + } + return 'message' in result.error ? result.error.message : result.error.kind; +} + +const CREDENTIAL = {accessToken: 'access-1', refreshToken: 'refresh-1', expiresAt: new Date(Date.now() + 3_600_000).toISOString(), scopes: [], email: 'alex@example.com'}; + +const target = { + connection: {id: 'google-1', integrationId: 'google', account: 'alex@example.com', createdAt: new Date(), tools: null, name: null, defaults: [], credential: `enc:${JSON.stringify(CREDENTIAL)}`}, + collection: ranges(), +}; + +type Request = {method: string; url: URL; body: string}; + +const google = {requests: [] as Request[], answer: (_request: Request): Response | null => null}; + +spyOn(globalThis, 'fetch').mockImplementation((async (input: string | URL | Request, init?: RequestInit) => { + const url = new URL(typeof input === 'string' ? input : input instanceof URL ? input.href : input.url); + const request = {method: init?.method ?? 'GET', url, body: typeof init?.body === 'string' ? init.body : ''}; + google.requests.push(request); + const custom = google.answer(request); + if (custom) { + return custom; + } + if (request.method === 'PUT') { + const {range, values} = JSON.parse(request.body); + return Response.json({ + updatedRange: "'Q1 Sales'!A2:C3", + updatedRows: values.length, + updatedData: { + range, + values: [ + ['Rent', '1,200', '2,400'], + ['Food, drink', '300', '600'], + ], + }, + }); + } + return Response.json({ + range: "'Q1 Sales'!A1:C3", + majorDimension: 'ROWS', + values: [ + ['Item', 'Amount'], + ['Rent', 1200], + ['Say "hi"', ''], + ], + }); +}) as typeof fetch); + +beforeEach(() => { + google.requests = []; + google.answer = () => null; +}); + +const LINK = 'https://docs.google.com/spreadsheets/d/1BudgetSheet_x-9/edit?gid=0#gid=0'; + +describe('reading a range', () => { + test('a range of a spreadsheet given by its link answers its values as CSV', async () => { + const {googleConnector} = await import('./google-connector'); + const record = (await googleConnector.newCommands?.read?.(target, {spreadsheet: LINK, range: "'Q1 Sales'!A1:C3", values: ''}))?.unwrap(); + + const [request] = google.requests; + expect(request?.url.pathname).toBe("/v4/spreadsheets/1BudgetSheet_x-9/values/'Q1%20Sales'!A1%3AC3"); + expect(record?.values).toEqual({spreadsheet: '1BudgetSheet_x-9', range: "'Q1 Sales'!A1:C3", rows: '3 rows', result: 'Item,Amount\nRent,1200\n"Say ""hi""",'}); + }); + + test('a range of more than 1000 rows answers the first 1000, saying how many there are', async () => { + google.answer = () => Response.json({range: "'Q1 Sales'!A1:A1500", majorDimension: 'ROWS', values: Array.from({length: 1500}, (_, index) => [String(index + 1)])}); + const {googleConnector} = await import('./google-connector'); + const record = (await googleConnector.newCommands?.read?.(target, {spreadsheet: LINK, range: "'Q1 Sales'!A:A", values: ''}))?.unwrap(); + + expect(record?.values.rows).toBe('The first 1000 of 1500 rows: read the rest as a range starting below them'); + expect(record?.values.result?.split('\n')).toHaveLength(1000); + }); + + test('a spreadsheet is taken by its id or a Drive link, and a range without a tab or in an unquoted one', async () => { + const {googleConnector} = await import('./google-connector'); + await googleConnector.newCommands?.read?.(target, {spreadsheet: '1BudgetSheet', range: 'Sheet1!B:B'}); + await googleConnector.newCommands?.read?.(target, {spreadsheet: 'https://drive.google.com/open?id=1BudgetSheet', range: 'A1:C9'}); + await googleConnector.newCommands?.read?.(target, {spreadsheet: '1BudgetSheet', range: '$A$1:$F$20'}); + await googleConnector.newCommands?.read?.(target, {spreadsheet: '1BudgetSheet', range: 'Übersicht!A1'}); + + expect(google.requests.map((request) => decodeURIComponent(request.url.pathname))).toEqual([ + '/v4/spreadsheets/1BudgetSheet/values/Sheet1!B:B', + '/v4/spreadsheets/1BudgetSheet/values/A1:C9', + '/v4/spreadsheets/1BudgetSheet/values/$A$1:$F$20', + '/v4/spreadsheets/1BudgetSheet/values/Übersicht!A1', + ]); + }); + + test('a range that isn’t A1 notation, or a spreadsheet that isn’t Google’s, never reaches Google', async () => { + const {googleConnector} = await import('./google-connector'); + const refusals = await Promise.all([ + googleConnector.newCommands?.read?.(target, {spreadsheet: '1BudgetSheet', range: 'A1:C3?fields=*'}), + googleConnector.newCommands?.read?.(target, {spreadsheet: '1BudgetSheet', range: "'Q1'!A1/../../x"}), + googleConnector.newCommands?.read?.(target, {spreadsheet: '1BudgetSheet', range: ''}), + googleConnector.newCommands?.read?.(target, {spreadsheet: 'https://evil.example.com/d/1BudgetSheet', range: 'A1'}), + googleConnector.newCommands?.read?.(target, {spreadsheet: '../1BudgetSheet', range: 'A1'}), + ]); + + expect(refusals.map((result) => result?.isErr() && result.error.kind)).toEqual(['validation_error', 'validation_error', 'validation_error', 'validation_error', 'validation_error']); + expect(google.requests).toEqual([]); + }); + + test('Google turning a range down is passed on in its own words', async () => { + const {googleConnector} = await import('./google-connector'); + google.answer = () => Response.json({error: {code: 400, message: "Unable to parse range: 'Nope'!A1", status: 'INVALID_ARGUMENT'}}, {status: 400}); + const result = await googleConnector.newCommands?.read?.(target, {spreadsheet: '1BudgetSheet', range: "'Nope'!A1"}); + + expect(result?.isErr() && result.error).toEqual({kind: 'validation_error', statusCode: 400, message: "Google: Unable to parse range: 'Nope'!A1"}); + }); +}); + +describe('writing a range', () => { + test('CSV is written as if typed in, and the answer is what the range holds now', async () => { + const {googleConnector} = await import('./google-connector'); + const record = (await googleConnector.newCommands?.write?.(target, {spreadsheet: LINK, range: "'Q1 Sales'!A2", values: 'Rent,"1,200",=B2*2\n"Food, drink",300,=B3*2\n'}))?.unwrap(); + + const [request] = google.requests; + expect(request?.method).toBe('PUT'); + expect(Object.fromEntries(request?.url.searchParams ?? [])).toEqual({valueInputOption: 'USER_ENTERED', includeValuesInResponse: 'true', responseValueRenderOption: 'FORMATTED_VALUE'}); + expect(JSON.parse(request?.body ?? '')).toEqual({ + range: "'Q1 Sales'!A2", + majorDimension: 'ROWS', + values: [ + ['Rent', '1,200', '=B2*2'], + ['Food, drink', '300', '=B3*2'], + ], + }); + expect(record?.values).toEqual({spreadsheet: '1BudgetSheet_x-9', range: "'Q1 Sales'!A2:C3", rows: '2 rows', result: 'Rent,"1,200","2,400"\n"Food, drink",300,600'}); + }); + + test('an empty value in quotes empties the cell', async () => { + const {googleConnector} = await import('./google-connector'); + await googleConnector.newCommands?.write?.(target, {spreadsheet: LINK, range: 'A1', values: '""'}); + + expect(JSON.parse(google.requests[0]?.body ?? '').values).toEqual([['']]); + }); + + test('no values, or a quote never closed, is turned down', async () => { + const {googleConnector} = await import('./google-connector'); + const empty = await googleConnector.newCommands?.write?.(target, {spreadsheet: LINK, range: 'A1', values: ''}); + const unclosed = await googleConnector.newCommands?.write?.(target, {spreadsheet: LINK, range: 'A1', values: '"Rent,1200'}); + + expect(messageOf(empty)).toBe('Write the values as CSV, one row a line; "" empties a cell'); + expect(messageOf(unclosed)).toBe('The CSV has a quote that is never closed'); + expect(google.requests).toEqual([]); + }); + + test('ranges are never listed or opened', async () => { + const {googleConnector} = await import('./google-connector'); + const result = await googleConnector.get(target, 'range'); + + expect(result.isErr() && result.error.kind).toBe('validation_error'); + }); +}); diff --git a/packages/api/src/records/google-ranges.ts b/packages/api/src/records/google-ranges.ts new file mode 100644 index 00000000..f454b969 --- /dev/null +++ b/packages/api/src/records/google-ranges.ts @@ -0,0 +1,98 @@ +import {Err, Ok, type Result} from 'ts-results-es'; + +import {ApiErr, type ApiError, Do} from '@proxy/utils'; + +import {DRIVE_ID} from '../connections/google/drive-api'; +import {callGoogle} from '../connections/google/google-client'; +import {getValues, putValues} from '../connections/google/sheets-api'; +import type {Connector, DataRecord, NewCommandRunner, RecordValues} from './connector'; +import {csvOf, parseCsv} from './csv'; + +/** + * A spreadsheet's ranges as the `ranges` collection: never listed or opened, only read or written, + * a range at a time, in A1 notation with the tab, `'Q1 Sales'!A1:F20`. Values go in and come out + * as CSV, one row a line; they are read as Sheets shows them and written as if typed in. + */ + +// The spreadsheet in a link to it, `…/spreadsheets/d//edit`, or in Drive's `?id=` links. +function spreadsheetIdOf(text: string): string | null { + const trimmed = text.trim(); + if (DRIVE_ID.test(trimmed)) { + return trimmed; + } + const url = URL.parse(trimmed); + if (!url || url.protocol !== 'https:' || !/(^|\.)google\.com$/.test(url.hostname)) { + return null; + } + const id = /\/d\/([^/]+)/.exec(url.pathname)?.[1] ?? url.searchParams.get('id'); + return id && DRIVE_ID.test(id) ? id : null; +} + +function requireSpreadsheet(values: RecordValues): Result { + const id = spreadsheetIdOf(values.spreadsheet ?? ''); + if (id === null) { + return Err(ApiErr.validationError('Say which spreadsheet as spreadsheet: its id, or its link')); + } + return Ok(id); +} + +// A tab's title, quoted as A1 notation quotes it, a quote in it doubled, or one of letters, digits +// and underscores alone as it is. +const TAB = String.raw`(?:'(?:[^'\x00-\x1f]|''){1,100}'|[\p{L}\p{N}_]{1,100})`; +// A cell (B2, or $B$2 as a formula fixes it), a whole column (B) or a whole row (2). +const CELL = String.raw`(?:\$?[A-Za-z]{1,3}(?:\$?\d{1,7})?|\$?\d{1,7})`; +const A1 = new RegExp(String.raw`^(?:${TAB}!)?${CELL}(?::${CELL})?$|^${TAB}$`, 'u'); + +function requireRange(values: RecordValues): Result { + const range = (values.range ?? '').trim(); + if (!A1.test(range)) { + return Err(ApiErr.validationError(`Write the range in A1 notation, such as 'Sheet1'!A1:F20, or A1:F20 in the first tab, not “${range}”`)); + } + return Ok(range); +} + +// The rows a range answers at most, as a database's query answers 200: a whole tab can be far more +// than an agent can take in at once. +const MAX_ROWS = 1000; + +function rowsText(count: number): string { + if (count === 0) { + return 'No rows'; + } + if (count > MAX_ROWS) { + return `The first ${MAX_ROWS} of ${count} rows: read the rest as a range starting below them`; + } + return `${count} ${count === 1 ? 'row' : 'rows'}`; +} + +function rangeRecord(spreadsheet: string, range: string, values: string[][]): DataRecord { + return {id: 'range', values: {spreadsheet, range, rows: rowsText(values.length), result: csvOf(values.slice(0, MAX_ROWS))}, updatedAt: new Date().toISOString()}; +} + +/** A range's values, as CSV in `result`. */ +const read: NewCommandRunner = (target, values) => + Do(async ($) => { + const spreadsheet = $(requireSpreadsheet(values)); + const range = $(requireRange(values)); + const answer = $(await callGoogle(target.connection, (token) => getValues(token, spreadsheet, range))); + return rangeRecord(spreadsheet, answer.range, answer.values); + }); + +/** Writes `values` into a range and answers what it holds now, formulas worked out. */ +const write: NewCommandRunner = (target, values) => + Do(async ($) => { + const spreadsheet = $(requireSpreadsheet(values)); + const range = $(requireRange(values)); + const rows = $(parseCsv(values.values ?? '')); + if (rows.length === 0) { + return $(Err(ApiErr.validationError('Write the values as CSV, one row a line; "" empties a cell'))); + } + const answer = $(await callGoogle(target.connection, (token) => putValues(token, spreadsheet, range, rows))); + return rangeRecord(spreadsheet, answer.range, answer.values); + }); + +function onlyRun(): Promise> { + return Promise.resolve(Err(ApiErr.validationError('Ranges are only read or written, a range at a time'))); +} + +export const rangesConnector: Connector = {list: onlyRun, get: onlyRun, create: onlyRun, update: onlyRun, remove: onlyRun, newCommands: {read, write}}; diff --git a/packages/db/scripts/seed.ts b/packages/db/scripts/seed.ts index 6bf1a2d9..2f5d21b7 100644 --- a/packages/db/scripts/seed.ts +++ b/packages/db/scripts/seed.ts @@ -354,6 +354,7 @@ const agents = [ grant(work, {write: true, send: true}), grant(receipts, {read: true}), grant(newsletters, {read: false}), + grant(googleAccount, {createFiles: true, editFiles: true}), ], }), // Reads the cards no agent gets by default and files receipts; kept out of the work mailbox. @@ -463,6 +464,13 @@ const FILE_TITLES: [string, ...string[]] = [ 'Trip itinerary', ]; +// The ranges agents read and write in spreadsheets. +const RANGE_TITLES: [string, ...string[]] = [ + "'Q1 Sales'!A1:F20", + 'Budget!B2:B14', + "'Trip costs'!A1:D9", +]; + // What agents search for. const SEARCHES: [string, ...string[]] = [ 'invoice', @@ -490,6 +498,10 @@ function recordTitle(connection: Connection, collectionId: string): string { return pick(ISSUE_TITLES); } + if (collectionId === 'ranges') { + return pick(RANGE_TITLES); + } + if (connection.integrationId === 'google') { return pick(FILE_TITLES); } diff --git a/packages/docs/content/connections/google.mdx b/packages/docs/content/connections/google.mdx index c2ec5e00..a0dac1a4 100644 --- a/packages/docs/content/connections/google.mdx +++ b/packages/docs/content/connections/google.mdx @@ -1,12 +1,12 @@ --- title: Google -description: Connect a Google account for agents to work with Gmail and read Drive, Docs, Sheets and Slides. +description: Connect a Google account for agents to work with Gmail and Drive, Docs, Sheets and Slides. --- Agents work with the account's email and the files it can open, as the person who signs in. Every request goes to Google live; nothing is copied. One sign-in covers Gmail, Drive, Docs, Sheets and Slides, and one connection holds them all. Gmail needs no app password, so it works for accounts -that can't make one. Today agents can read files; writing them comes later. +that can't make one. ## Before you start: a Google Cloud OAuth client @@ -146,15 +146,34 @@ For Drive, Docs, Sheets and Slides: - **Read files** browses and searches Drive and opens files. At the top are four places: My Drive, Shared with me, Shared drives and Recent. My Drive, folders and shared drives list their files folders first, Shared with me the last shared first, and Recent the files the account opened, - in shared drives too, the last opened first, 50 to a page; trashed files are never listed. A search looks for words in - file names and contents, anywhere in Drive, or among the files in the folder opened. A document - opens as Markdown and a presentation as text, as Google exports them, up to Google's limit of - 10 MB. A spreadsheet opens into its tabs, and a tab into its rows, 50 to a page, each row's - values as `Header: value` lines, the headers from the tab's first row. Other files, shortcuts - included, open as their name, kind, owner, last change and link. - -Creating and editing files is coming. The sign-in already asks for it, so no new sign-in will be -needed. + in shared drives too, the last opened first, 50 to a page; trashed files are never listed. A + search looks for words in file names and contents, anywhere in Drive, or among the files in the + folder opened. A document opens as Markdown and a presentation as text, as Google exports them, + up to Google's limit of 10 MB; past it, the content says the file is too large, and the file can + still be renamed or moved to Trash. A spreadsheet opens into its tabs, and a tab into its rows, + 50 to a page, each row's values as `Header: value` lines, the headers from the tab's first row; a + header used twice, or a column without one, goes by its column's letter, `Status (E)`. Other + files, shortcuts included, open as their name, kind, owner, last change and link. +- **Create files** makes folders, documents and spreadsheets in My Drive, a folder or a shared + drive, and adds rows to a spreadsheet's tabs, and requires Read files. A document is written as + Markdown and a spreadsheet as CSV, and Google converts them as they are uploaded. A row is written + as `Header: value` lines, under the headers of the tab's first row, and goes after the last row + holding anything, from column A. +- **Edit files** renames files and a spreadsheet's tabs, replaces a document's content, and changes + rows, and requires Read files. A document's content is replaced whole with the Markdown written, + which drops its comments and suggestions, and only when it is written differently from how it + read, so renaming a document leaves its content alone. Shared drives aren't renamed or trashed. A row changes only in the values written differently, + so a formula in another of its cells stays. A presentation's and a spreadsheet's content are only + read. +- **Move files to Trash** moves files and folders to Drive's Trash, where Drive keeps them 30 days + before deleting them, and requires Read files. Nothing is deleted for good. + +Agents can also read and write a range of a spreadsheet, in A1 notation with its tab, such as +`'Q1 Sales'!A1:F20`, given the spreadsheet's link or id, which covers tabs without a header row. +A range without a tab is in the first one. Reading needs Read files and answers the values as +Sheets shows them, as CSV, one row a line, 1000 rows at most. +Writing needs Edit files and takes CSV the same way, each value as if typed into its cell, so +`=SUM(B2:B9)` is a formula, and `""` empties a cell; it answers what the range holds afterwards. ## Troubleshooting diff --git a/packages/integrations/src/catalog.ts b/packages/integrations/src/catalog.ts index c2eacc6b..3bd73cfe 100644 --- a/packages/integrations/src/catalog.ts +++ b/packages/integrations/src/catalog.ts @@ -564,6 +564,9 @@ const postgres = sqlIntegration( const mysql = sqlIntegration('mysql', 'MySQL', 'Tables and queries in a MySQL or MariaDB database'); +// What a Drive record can be beside a place: a spreadsheet holds its tabs (Sheet), a tab its rows. +const FILE_KINDS = ['Folder', 'Document', 'Spreadsheet', 'Presentation', 'Sheet', 'Row', 'File']; + // A Google account, signed in to once for everything Personal Agent Proxy reads and writes there: // Gmail, Drive, Docs, Sheets and Slides. One connection holds every product, so action ids carry // the product, as Information's `readCards` and `writeCards` do. Gmail is the same email as any @@ -571,8 +574,10 @@ const mysql = sqlIntegration('mysql', 'MySQL', 'Tables and queries in a MySQL or // drafts, sent mail, the archive, Trash or Spam. Drafts are the only email edited or deleted; // nothing else is deleted for good, only moved to Trash. Drive is one tree: My Drive, Shared with // me, Shared drives and Recent at the top, folders holding files, a spreadsheet holding its tabs and -// a tab its rows. Documents read as Markdown, presentations as text, rows as "Header: value" lines. -// Nothing is copied: every request goes to Google. +// a tab its rows. Documents read as Markdown, presentations as text, rows as "Header: value" lines; +// new documents are written as Markdown and new spreadsheets as CSV, which Drive converts, and a +// spreadsheet's ranges are read and written in A1 notation as CSV. Files are moved to Trash, never +// deleted. Nothing is copied: every request goes to Google. const google: Integration = { id: 'google', name: 'Google', @@ -622,6 +627,20 @@ const google: Integration = { 'Browse and search Drive, and open files, documents, spreadsheets and presentations', risk: 'low', }, + { + id: 'createFiles', + label: 'Create files', + description: 'Create folders, documents and spreadsheets, and add rows', + risk: 'medium', + requires: 'readFiles', + }, + { + id: 'editFiles', + label: 'Edit files', + description: 'Rename files, replace a document’s content and change rows and ranges', + risk: 'medium', + requires: 'readFiles', + }, { id: 'trashEmails', label: 'Move email to Trash', @@ -636,6 +655,14 @@ const google: Integration = { risk: 'high', reachesOthers: true, }, + { + id: 'trashFiles', + label: 'Move files to Trash', + description: + 'Move files and folders to Trash, where Drive keeps them 30 days before deleting them', + risk: 'high', + requires: 'readFiles', + }, ], collections: [ emailsCollection({ @@ -664,25 +691,35 @@ const google: Integration = { 'Words in file names and contents, anywhere in Drive, or inside the folder opened.', nested: true, read: 'readFiles', - writes: {}, + writes: {create: 'createFiles', update: 'editFiles'}, + // The four places at the top are always there, as they are. + editable: {field: 'kind', values: FILE_KINDS}, + commands: [ + { + id: 'trash', + on: 'record', + label: 'Move to Trash', + action: 'trashFiles', + // A spreadsheet's tabs and rows aren't files of their own. + where: { + field: 'kind', + values: FILE_KINDS.filter((kind) => kind !== 'Sheet' && kind !== 'Row'), + }, + done: 'Moved {} to Trash', + tried: 'move {} to Trash', + }, + ], fields: [ {key: 'name', label: 'Name', type: 'text'}, + // What a record is, and what a new one is to be: a folder, document or spreadsheet inside a + // folder or a place, or a row inside a tab, where it can be left out. It can't be changed. { key: 'kind', label: 'Kind', type: 'select', // A place is one of the four at the top: My Drive, Shared with me, Shared drives, Recent. - options: [ - 'Place', - 'Folder', - 'Document', - 'Spreadsheet', - 'Presentation', - 'Sheet', - 'Row', - 'File', - ], - system: true, + // A shared drive holds files as a folder does, but isn't renamed or trashed here. + options: ['Place', 'Shared drive', ...FILE_KINDS], }, {key: 'modified', label: 'Modified', type: 'datetime', system: true}, // Whoever owns the file in Drive; files in a shared drive have none. @@ -690,10 +727,54 @@ const google: Integration = { // The file in Drive. {key: 'link', label: 'Link', type: 'text', system: true}, // A document as Markdown, a presentation as text, a spreadsheet's tabs one a line with their - // size, a row's values one "Header: value" a line, the headers from the tab's first row. + // size, a row's values one "Header: value" a line, the headers from the tab's first row. A + // new document is written as Markdown and a new spreadsheet as CSV. Writing a document's + // content replaces all of it, dropping its comments and suggestions; a row's changes only + // the headers written differently. A presentation's and a spreadsheet's are only read. {key: 'content', label: 'Content', type: 'longtext'}, ], }, + // A range of a spreadsheet, read or written in A1 notation, as people who know Sheets think of + // one, and for tabs without a header row. Never listed or opened, only run. + { + id: 'ranges', + name: 'Ranges', + singular: 'range', + titleField: 'range', + writes: {}, + commands: [ + { + id: 'read', + on: 'new', + label: 'Read', + action: 'readFiles', + readOnly: true, + done: 'Read {}', + tried: 'read {}', + }, + { + id: 'write', + on: 'new', + label: 'Write', + action: 'editFiles', + done: 'Wrote {}', + tried: 'write {}', + }, + ], + fields: [ + // The spreadsheet's id, or its link. + {key: 'spreadsheet', label: 'Spreadsheet', type: 'text'}, + // A1 notation, with the tab or in the first one: 'Q1 Sales'!A1:F20, A1:F20. + {key: 'range', label: 'Range', type: 'text'}, + // What Write puts in the range, as CSV, one row a line, each value as if typed into the + // cell: =SUM(B2:B9) is a formula. + {key: 'values', label: 'Values', type: 'longtext'}, + // "12 rows", or how many of more were answered: at most 1000. + {key: 'rows', label: 'Rows', type: 'text', system: true}, + // The range's values as Sheets shows them, as CSV, one row a line: after Write, as written. + {key: 'result', label: 'Result', type: 'longtext', system: true}, + ], + }, ], }; diff --git a/packages/integrations/src/tools.test.ts b/packages/integrations/src/tools.test.ts index d6af1d28..96d5d306 100644 --- a/packages/integrations/src/tools.test.ts +++ b/packages/integrations/src/tools.test.ts @@ -199,11 +199,50 @@ describe('postgres and mysql', () => { describe('google', () => { const google = integration('google'); - test('email has the same tools as any mailbox’s, and files are listed and opened, never written to yet', () => { + test('email has the same tools as any mailbox’s, files are never deleted, only moved to Trash, and ranges are only run', () => { expect(integrationTools(google).map((tool) => tool.name)).toEqual([ ...integrationTools(email).map((tool) => tool.name), 'files_list', 'files_get', + 'files_create', + 'files_update', + 'files_trash', + 'ranges_read', + 'ranges_write', + ]); + }); + + test('a file is made from its kind, name and content, inside the folder given, and trashing leaves out tabs and rows', () => { + const create = findTool(google, 'files_create'); + + expect(create?.action).toBe('createFiles'); + expect(Object.keys(create?.inputSchema.properties ?? {})).toEqual([ + 'parent', + 'name', + 'kind', + 'content', + ]); + expect(findTool(google, 'files_update')).toMatchObject({action: 'editFiles', risk: 'medium'}); + const update = findTool(google, 'files_update'); + expect(update?.kind === 'update' ? update.where?.values : null).not.toContain('Place'); + expect(findTool(google, 'files_trash')).toMatchObject({ + action: 'trashFiles', + risk: 'high', + where: {field: 'kind', values: ['Folder', 'Document', 'Spreadsheet', 'Presentation', 'File']}, + }); + }); + + test('reading a range is a read, writing one needs Edit files', () => { + expect(findTool(google, 'ranges_read')).toMatchObject({ + action: 'readFiles', + readOnly: true, + kind: 'newCommand', + }); + expect(findTool(google, 'ranges_write')).toMatchObject({action: 'editFiles', readOnly: false}); + expect(Object.keys(findTool(google, 'ranges_write')?.inputSchema.properties ?? {})).toEqual([ + 'spreadsheet', + 'range', + 'values', ]); }); diff --git a/packages/web/src/pages/agent/agent-new-record-page.tsx b/packages/web/src/pages/agent/agent-new-record-page.tsx index 1bc5b11f..57dafcde 100644 --- a/packages/web/src/pages/agent/agent-new-record-page.tsx +++ b/packages/web/src/pages/agent/agent-new-record-page.tsx @@ -20,10 +20,11 @@ function parentTitleFromState(state: unknown): string | null { } // A new record, saved or run through a command such as Send, whichever the agent may do: Send and -// Save as draft side by side when it may do both. Without either the form isn't offered; the api -// would refuse and log it as denied. In a nested collection it goes inside `?parent=` when given, -// such as a page inside the one the agent was looking at. A command that answers something, as -// running a query answers its rows, shows it under the form, where the next one can be written. +// Save as draft side by side when it may do both, or two commands side by side, as Read and Write +// on a range. Without either the form isn't offered; the api would refuse and log it as denied. In +// a nested collection it goes inside `?parent=` when given, such as a page inside the one the agent +// was looking at. A command that answers something, as running a query answers its rows, shows it +// under the form, where the next one can be written. export function AgentNewRecordPage() { const navigate = useNavigate(); const [params] = useSearchParams(); @@ -32,7 +33,7 @@ export function AgentNewRecordPage() { const [error, setError] = useState(null); const [answer, setAnswer] = useState(null); const canCreate = target !== null && allows(target.collection, target.actions, 'create'); - const [command] = target ? newCommands(target.collection, target.actions) : []; + const [command, second] = target ? newCommands(target.collection, target.actions) : []; if (!target || (!canCreate && !command)) { return This login cannot add anything here.; } @@ -79,6 +80,24 @@ export function AgentNewRecordPage() { }); }; + // Beside a command, saving the record where the agent may, else the next command, as Write + // beside Read on a range. + const alternative = () => { + if (command && canCreate) { + return { + label: createLabel, + onSubmit: (values: Record) => void create(values), + }; + } + if (command && second) { + return { + label: second.label, + onSubmit: (values: Record) => void run(second, values), + }; + } + return undefined; + }; + return ( navigate(command && !canCreate ? '/agent' : listPath)} onSubmit={(values) => void (command ? run(command, values) : create(values))} - alternative={ - command && canCreate - ? {label: createLabel, onSubmit: (values) => void create(values)} - : undefined - } + alternative={alternative()} /> {answer && (