diff --git a/.claude/CLAUDE.md b/.claude/CLAUDE.md index 8cc24c55..f7fd2d3d 100644 --- a/.claude/CLAUDE.md +++ b/.claude/CLAUDE.md @@ -184,6 +184,7 @@ Always use the left column. Never use the right. | the Service (short for the PowerSync Service) | the service | | Sync Rules | sync rules | | Sync Streams | sync streams | +| Sync Config | sync config (except in code, e.g. `sync_config`, `sync-config.yaml`) | When referring to the PowerSync Service in shortened form, write "the Service" (capitalized), never "the service". Generic services (a Docker Compose service, a third-party service) stay lowercase. diff --git a/architecture/powersync-service.mdx b/architecture/powersync-service.mdx index 792266e0..230f786a 100644 --- a/architecture/powersync-service.mdx +++ b/architecture/powersync-service.mdx @@ -64,9 +64,11 @@ When a change occurs in the source database that affects a certain bucket (based The PowerSync Service persists the bucket state in durable storage: there is a pluggable storage layer for bucket data, and MongoDB and Postgres are currently supported as _bucket storage_ databases. The _bucket storage_ database is separate from the connection to your _source database_ (Postgres, MongoDB, MySQL, SQL Server or Convex). Our cloud-hosting offering (PowerSync Cloud) uses MongoDB Atlas as the _bucket storage_ database. +With [S3 object storage](/sync/advanced/storage-version-4#s3-object-storage) (Beta, requires storage version 4), larger blocks of prepared sync data are stored in S3 and synced to clients directly from there, while MongoDB keeps the metadata that locates them. Clients continue to connect only to the PowerSync Service. + Persisting the bucket state in a database is also part of how PowerSync achieves high scalability: it means that the PowerSync Service can have a low memory footprint even as you scale to very large volumes of synced data and users/clients. -The layout of that persisted data follows a storage version. That version can change when you deploy new Sync Streams or Sync Rules. That way, large internal collections can evolve without always requiring expensive upfront migrations across your entire dataset. In most cases you do not need to name the storage version in your sync config. The optional `config.storage_version` field is there for situations where you do need explicit control. Examples include preparing for a Service downgrade that only supports an older storage format, testing different storage formats in a non-production environment, or keeping bucket data on an older stable format while you change other parts of the sync config. For more details, see [Storage version](/sync/advanced/compatibility#storage-version). +The layout of that persisted data follows a storage version. That version can change when you deploy new Sync Streams or Sync Rules. That way, large internal collections can evolve without always requiring expensive upfront migrations across your entire dataset. In most cases you do not need to name the storage version in your Sync Config. The optional `config.storage_version` field is there for situations where you do need explicit control. Examples include preparing for a Service downgrade that only supports an older storage format, testing different storage formats in a non-production environment, or keeping bucket data on an older stable format while you change other parts of the Sync Config. For more details, see [Storage version](/sync/advanced/compatibility#storage-version). ## Replication from the Source Database @@ -85,7 +87,7 @@ When the PowerSync Service replicates data from the source database, it: ### Initial Replication vs. Incremental Replication -Whenever a new version of Sync Streams/Rules is deployed, initial replication takes place by means of taking a snapshot of all tables/collections they reference. +Whenever a new Sync Config is deployed, initial replication takes a snapshot of all tables or collections it references. With [incremental reprocessing](/sync/advanced/storage-version-4#incremental-reprocessing) (Beta), a Sync Streams deployment only snapshots the streams that were added or changed. Clients continue using the current copy of the data until the new one is ready. After that, data is incrementally replicated using a change data capture stream. The specific mechanism depends on the source database type: Postgres logical replication, MongoDB change streams, the MySQL binlog, SQL Server Change Data Capture, or Convex document deltas. diff --git a/client-sdks/advanced/pre-seeded-sqlite.mdx b/client-sdks/advanced/pre-seeded-sqlite.mdx index c7ecf54f..b431f768 100644 --- a/client-sdks/advanced/pre-seeded-sqlite.mdx +++ b/client-sdks/advanced/pre-seeded-sqlite.mdx @@ -22,7 +22,7 @@ If you're interested in seeing an end-to-end example, we've prepared a demo repo ### Generate a Scoped JWT Token In most cases you'd want to pre-seed the SQLite database with user specific data and not all data from the source database, as you normally would when using PowerSync. For this you would need to generate JWT tokens that include the necessary properties to satisfy the conditions of the queries in your Sync Streams (or legacy Sync Rules). -Let's say we have the following sync config: +Let's say we have the following Sync Config: diff --git a/client-sdks/advanced/sequential-id-mapping.mdx b/client-sdks/advanced/sequential-id-mapping.mdx index 04582ddd..d3d5aa57 100644 --- a/client-sdks/advanced/sequential-id-mapping.mdx +++ b/client-sdks/advanced/sequential-id-mapping.mdx @@ -187,7 +187,7 @@ can move on to updating your Sync Streams/Sync Rules to use the UUID column inst ## Update Sync Streams -As sequential IDs can only be created on the backend source database, we need to use UUIDs in the client. The sync config is updated to use the `uuid` column as the `id` column for the `lists` and `todos` tables, explicitly defining which columns to select so that `list_id` (the integer ID) is no longer exposed to the client. +As sequential IDs can only be created on the backend source database, we need to use UUIDs in the client. The Sync Config is updated to use the `uuid` column as the `id` column for the `lists` and `todos` tables, explicitly defining which columns to select so that `list_id` (the integer ID) is no longer exposed to the client. diff --git a/configuration/powersync-service/self-hosted-instances.mdx b/configuration/powersync-service/self-hosted-instances.mdx index a58a489c..bab0b0a3 100644 --- a/configuration/powersync-service/self-hosted-instances.mdx +++ b/configuration/powersync-service/self-hosted-instances.mdx @@ -466,7 +466,7 @@ The PowerSync Service requires a storage database to store the data and metadata - Storage version to use when deploying a sync configuration that does not specify a storage version. You typically do not need to change this. See [Storage Version](/sync/advanced/compatibility#storage-version) for the available versions and how they interact with the sync configuration's `storage_version` field. Available since Service version 1.26.0. + Storage version to use when deploying a Sync Config that does not specify a storage version. Set it to `4` to opt in to [storage version 4](/sync/advanced/storage-version-4) (Beta) for newly deployed Sync Configs. See [Storage Version](/sync/advanced/compatibility#storage-version) for how this interacts with the Sync Config's `storage_version` field. Available since Service version 1.26.0. #### MongoDB Storage @@ -502,7 +502,7 @@ storage: - Experimental support for storing large bucket data chunks in S3-compatible object storage instead of MongoDB. Available since Service version 1.24.0. Requires storage version 3, which is not the default. Set `storage_version: 3` under the `config` block of your sync config, as described in [Storage Version](/sync/advanced/compatibility#optional-config-storage_version). + Stores large blocks of bucket data in Amazon S3 or an S3-compatible object store instead of MongoDB. Requires [storage version 4](/sync/advanced/storage-version-4) (Beta), which is not the default. Sync Configs on version 2 keep all data in MongoDB even when this is configured. See [S3 object storage](/sync/advanced/storage-version-4#s3-object-storage) for setup guidance. Available since Service version 1.24.0. diff --git a/configuration/source-db/setup.mdx b/configuration/source-db/setup.mdx index 3f263e8b..462a9a4e 100644 --- a/configuration/source-db/setup.mdx +++ b/configuration/source-db/setup.mdx @@ -385,7 +385,7 @@ DocumentDB is not fully MongoDB-compatible. The following behaviors differ from - **Source database only.** DocumentDB is supported as a source database, not as PowerSync's [storage database](/architecture/powersync-service#bucket-storage) (especially relevant if you are self-hosting PowerSync). We don't currently plan to support it for storage. - **Post-images are not supported.** Set [Post Images](#post-images) to **Off** (`post_images: off`, the default); the `auto_configure` and `read_only` modes fail. This does not reduce correctness: DocumentDB always includes the full current document on update events, so updates and deletes still replicate correctly. - **Collection drop and rename are not replicated.** Already-synced rows remain in PowerSync storage under the old name. To recover, redeploy your Sync Streams to trigger a resync. Regular inserts, updates, and deletes are unaffected. -- **Large initial snapshots may not complete on legacy storage.** DocumentDB retains only a limited amount of change-feed history. On storage versions 1 and 2, a large or busy source can age its resume position out of that window before the initial snapshot finishes, and replication restarts from scratch. Use [Storage Version 3](/sync/advanced/compatibility#storage-version) or later, which consumes the change stream while the snapshot is still running. +- **Large initial snapshots may not complete on legacy storage.** DocumentDB keeps source changes for a limited time. On storage versions 1 and 2, PowerSync waits for the initial scan to finish before reading those changes. On a large or busy source, the earliest required changes can expire before the scan finishes, forcing PowerSync to start over. Use [storage version 4](/sync/advanced/compatibility#storage-version), which reads new changes while the initial scan is running. - **Rows of 15 MiB or larger are not synced.** Such rows are dropped with a logged error. This is a general [PowerSync limit](/resources/performance-and-limits), not DocumentDB-specific, but it is more reachable on DocumentDB because the source permits documents up to 16 MiB and every change event carries the full document. - **Large documents replicate very slowly.** DocumentDB delivers large change events much more slowly than standard MongoDB. The document still replicates correctly, but latency spikes around large documents (including updates that change only a small field), and a single large event delays every change behind it. If your workload includes large documents, validate replication latency against your cluster. - **Streaming updates can be delayed by up to roughly 1 second after an idle period.** DocumentDB does not currently support reliable long-polling on its change stream, so PowerSync adds a short, capped delay between polls when the stream is idle. This is a temporary workaround and is expected to be resolved in an upcoming Azure DocumentDB release. diff --git a/docs.json b/docs.json index 82b2af3c..85b2a447 100644 --- a/docs.json +++ b/docs.json @@ -230,6 +230,7 @@ "sync/advanced/client-id", "sync/advanced/case-sensitivity", "sync/advanced/compatibility", + "sync/advanced/storage-version-4", "sync/advanced/sync-data-by-time", "sync/advanced/schemas-and-connections", "sync/advanced/multiple-client-versions", diff --git a/integrations/neon.mdx b/integrations/neon.mdx index 20720b18..f5f9f67f 100644 --- a/integrations/neon.mdx +++ b/integrations/neon.mdx @@ -141,7 +141,7 @@ PowerSync uses logical replication to sync data from your Neon database. 1. In the PowerSync Dashboard, select your project and instance and go to the **Sync Streams** view (shown as **Sync Rules** if using legacy Sync Rules). -2. Edit the sync config in the editor and replace the contents with the below: +2. Edit the Sync Config in the editor and replace the contents with the below: @@ -190,8 +190,8 @@ PowerSync uses logical replication to sync data from your Neon database. -3. Click **"Validate"** and ensure there are no errors. This validates your sync config against your Postgres database. -4. Click **"Deploy"** to deploy your sync config. +3. Click **"Validate"** and ensure there are no errors. This validates your Sync Config against your Postgres database. +4. Click **"Deploy"** to deploy your Sync Config. - For additional information on PowerSync's Sync Streams, refer to the [Sync Streams](/sync/streams/overview) documentation. diff --git a/intro/setup-guide.mdx b/intro/setup-guide.mdx index cb4d76c0..71c7de6a 100644 --- a/intro/setup-guide.mdx +++ b/intro/setup-guide.mdx @@ -224,11 +224,11 @@ PowerSync is available as a cloud-hosted service (PowerSync Cloud) or can be sel powersync docker configure --database postgres --storage postgres ``` - Docker sets up Postgres for both the source database and bucket storage and creates `powersync/docker/docker-compose.yaml`. Other databases are supported as well, you will learn more about this in the next step. Before starting, replace `powersync/sync-config.yaml` with this minimal sync config: + Docker sets up Postgres for both the source database and bucket storage and creates `powersync/docker/docker-compose.yaml`. Other databases are supported as well, you will learn more about this in the next step. Before starting, replace `powersync/sync-config.yaml` with this minimal Sync Config: ```yaml config: - edition: 2 + edition: 3 streams: todos: @@ -567,18 +567,18 @@ streams: - Edit `powersync/sync-config.yaml` with your sync config, then validate and deploy to the linked Cloud instance: + Edit `powersync/sync-config.yaml` with your Sync Config, then validate and deploy to the linked Cloud instance: ```bash powersync validate powersync deploy ``` - This deploys your full config (connection, auth, and sync config). For subsequent sync-only changes, use `powersync deploy sync-config` instead. + This deploys your full config (connection, auth, and Sync Config). For subsequent sync-only changes, use `powersync deploy sync-config` instead. - Edit `powersync/sync-config.yaml` with your sync config. The default file has a placeholder (`SELECT * FROM todos`). Replace it with your actual table/collection names, then apply the changes: + Edit `powersync/sync-config.yaml` with your Sync Config. The default file has a placeholder (`SELECT * FROM todos`). Replace it with your actual table/collection names, then apply the changes: ```bash powersync validate diff --git a/maintenance-ops/compacting-buckets.mdx b/maintenance-ops/compacting-buckets.mdx index 4d8b8090..03356ff7 100644 --- a/maintenance-ops/compacting-buckets.mdx +++ b/maintenance-ops/compacting-buckets.mdx @@ -24,6 +24,8 @@ When using PowerSync Cloud, compact logs are available in the Dashboard's **Logs For self-hosted setups (PowerSync Open Edition & PowerSync Enterprise Self-Hosted Edition), the `compact` command in the Docker image can be used to compact all buckets. This can be run manually, or on a regular schedule using Kubernetes [CronJob](https://kubernetes.io/docs/concepts/workloads/controllers/cron-jobs/) or similar scheduling functionality. +On [storage version 4](/sync/advanced/storage-version-4), each run decides per bucket whether to merge newly added chunks, which is fast, or to run a full compaction that reads the whole bucket. Most runs are fast, so you can schedule `compact` to run every hour. + [Defragmenting](/maintenance-ops/compacting-buckets#defragmenting) may still be required. ## Background @@ -149,16 +151,10 @@ You should consider defragmenting your buckets when: ### Defragmenting Strategies -There are manual and automated approaches to defragmenting: +There are two mechanisms with different scopes: -1. **Manual Defragmentation** - - Use the [PowerSync Dashboard](https://dashboard.powersync.com/) to manually trigger defragmentation: Select your project and instance, go to the **Settings** view, and click the **Defragment** button in the "Compact operation history" section - - Best for one-time cleanup or after major data changes +1. **Update source rows, then compact.** Use this when you can identify the rows whose old `PUT` operations block compacting. You can target one table or subset of rows, but each update creates a new operation that clients must download. For recurring maintenance, schedule the updates with a tool such as `pg_cron`: -2. **Scheduled Defragmentation** - - Set up a cron job to regularly update rows - - Recommended for frequently updated tables or tables with large churn - - Example using `pg_cron`: ```sql -- Daily defragmentation for high-churn tables UPDATE audit_logs SET last_updated = now() @@ -168,11 +164,12 @@ There are manual and automated approaches to defragmenting: UPDATE users SET last_updated = now() WHERE last_updated < now() - interval '1 week'; ``` - - This will cause clients to re-sync each updated row, while preventing the number of operations from growing indefinitely. Depending on how often rows in the bucket are modified, the interval can be increased or decreased. + +2. **Force a full reprocess.** In the [PowerSync Dashboard](https://dashboard.powersync.com/), select your project and instance, open **Settings**, and use **Defragment** in the "Compact operation history" section. This rebuilds all sync data, even when [incremental reprocessing](/sync/advanced/storage-version-4#incremental-reprocessing) would otherwise apply, and clients download all their data again. Use it for one-time cleanup of the entire Sync Config. ### Defragmenting Trade-offs -Defragmenting + compacting as described above can significantly reduce the number of operations in a bucket, at the cost of existing clients needing to re-sync that data. When and how to do this depends on the specific use-case and data update patterns. +Both methods can significantly reduce the operation history, at the cost of existing clients downloading data again. Updating source rows and then compacting lets you control which rows are affected. The Dashboard action rebuilds and replaces the complete replication stream. Key considerations: 1. **Frequency**: More frequent defragmentation means fewer operations per sync but more frequent re-syncs @@ -181,10 +178,13 @@ Key considerations: ## Sync Streams Deployments -Whenever modifications to [Sync Streams](/sync/streams/overview) (or legacy [Sync Rules](/sync/rules/overview)) are deployed, all buckets are re-created from scratch. This has a similar effect to fully defragmenting and compacting all buckets. This was recommended as a workaround before explicit compacting became available ([released July 26, 2024](https://releases.powersync.com/announcements/bucket-compacting)). +Without incremental reprocessing, deploying [Sync Streams](/sync/streams/overview) recreates all buckets from scratch. This has the same effect as defragmenting and compacting every bucket, and clients download all their data again. With [incremental reprocessing](/sync/advanced/storage-version-4#incremental-reprocessing) (Beta), unchanged streams keep their buckets and operation history, so a deployment no longer clears their history. -Soon, we will use [incremental reprocessing](https://github.com/orgs/powersync-ja/discussions/349) to process changed definitions only. +Do not redeploy a Sync Config for routine history cleanup. Use compacting, targeted source-row updates, or the Dashboard's **Defragment** action instead. ## Technical Details -See the [documentation](https://github.com/powersync-ja/powersync-service/blob/main/docs/storage/compacting-operations.md) in the `powersync-service` repo for more technical details on compacting. \ No newline at end of file +See the `powersync-service` repo documentation: + +- [Compacting operations](https://github.com/powersync-ja/powersync-service/blob/main/docs/storage/compacting-operations.md) - How compacting rewrites operation history into `MOVE` and `CLEAR` operations. This applies to all storage versions. +- [Compaction design](https://github.com/powersync-ja/powersync-service/blob/main/docs/storage/v3-compaction-design.md) - How [storage version 4](/sync/advanced/storage-version-4) schedules chunk merges and full compactions per bucket. diff --git a/maintenance-ops/production-readiness-guide.mdx b/maintenance-ops/production-readiness-guide.mdx index ba159833..d3d4c902 100644 --- a/maintenance-ops/production-readiness-guide.mdx +++ b/maintenance-ops/production-readiness-guide.mdx @@ -292,7 +292,7 @@ During a snapshot, PowerSync warns when less than 50% of the WAL budget remains. #### Managing Replication Slots -Under normal operating conditions when new Sync Streams/Sync Rules are deployed to a PowerSync Service instance, a new replication slot will also be created and used for replication. The old replication slot from the previous version of the sync config will still remain, until reprocessing is completed, at which point the old replication slot will be removed by the PowerSync Service. +Under normal operating conditions when new Sync Streams/Sync Rules are deployed to a PowerSync Service instance, a new replication slot will also be created and used for replication. The old replication slot from the previous version of the Sync Config will still remain, until reprocessing is completed, at which point the old replication slot will be removed by the PowerSync Service. However, in some cases, a replication slot may remain without being used. Usually this happens when a PowerSync Service instance is de-provisioned, stopped intentionally or due to unexpected errors. This results in excessive disk usage due to the continued growth of the WAL. To check which replication slots used by a PowerSync Service are no longer active, the following query can be executed against the source Postgres database: diff --git a/maintenance-ops/self-hosting/diagnostics.mdx b/maintenance-ops/self-hosting/diagnostics.mdx index 80607242..dbd08baa 100644 --- a/maintenance-ops/self-hosting/diagnostics.mdx +++ b/maintenance-ops/self-hosting/diagnostics.mdx @@ -45,9 +45,9 @@ The response `data` object contains: **`connections`** — whether PowerSync can reach the configured source database and any connection-level errors. -**`active_sync_rules`** — the currently serving sync config (Sync Streams or Sync Rules). Contains a `connections[]` array with details about each replication connection including slot name, WAL status, and tables being replicated. Also includes an `errors[]` array for warnings or errors. +**`active_sync_rules`** — the currently serving Sync Config (Sync Streams or Sync Rules). Contains a `connections[]` array with details about each replication connection including slot name, WAL status, and tables being replicated. Also includes an `errors[]` array for warnings or errors. -**`deploying_sync_rules`** — only present while a new sync config is being deployed and the initial replication is in progress. PowerSync runs this process in parallel so clients continue to be served by the existing active config. Once initial replication completes, this section disappears and `active_sync_rules` updates. +**`deploying_sync_rules`** — only present while a new Sync Config is being deployed and the initial replication is in progress. PowerSync runs this process in parallel so clients continue to be served by the existing active config. Once initial replication completes, this section disappears and `active_sync_rules` updates. Each connection in `active_sync_rules.connections[]` includes: diff --git a/maintenance-ops/self-hosting/update-sync-rules.mdx b/maintenance-ops/self-hosting/update-sync-rules.mdx index 323d31b1..8c7f5cd2 100644 --- a/maintenance-ops/self-hosting/update-sync-rules.mdx +++ b/maintenance-ops/self-hosting/update-sync-rules.mdx @@ -3,23 +3,23 @@ title: "Update Sync Streams (Sync Config)" description: "Update Sync Streams/Sync Rules in a self-hosted PowerSync deployment." --- -There are three ways to update your sync config in a self-hosted deployment: +There are three ways to update your Sync Config in a self-hosted deployment: 1. **CLI** — Edit your config and apply with `powersync docker reset` 2. **Config file** — Update your config and restart the service 3. **API endpoint** — Deploy at runtime without restarting - During deployment, existing Sync Streams/Sync Rules continue serving clients while new sync config processes. Clients seamlessly transition once [initial replication](/architecture/powersync-service#initial-replication-vs-incremental-replication) completes. + During deployment, existing Sync Streams/Sync Rules continue serving clients while new Sync Config processes. Clients seamlessly transition once [initial replication](/architecture/powersync-service#initial-replication-vs-incremental-replication) completes. - Run `powersync validate` in the CLI before deploying to catch errors in your sync config without applying changes. + Run `powersync validate` in the CLI before deploying to catch errors in your Sync Config without applying changes. ## Option 1: CLI -If you set up PowerSync using the CLI (`powersync docker`), update your sync config and apply it without a full service restart: +If you set up PowerSync using the CLI (`powersync docker`), update your Sync Config and apply it without a full service restart: @@ -34,13 +34,13 @@ If you set up PowerSync using the CLI (`powersync docker`), update your sync con ```bash powersync docker reset ``` - This restarts the PowerSync Service and applies your updated sync config. + This restarts the PowerSync Service and applies your updated Sync Config. ## Option 2: Config File -Define your sync config in `service.yaml` either inline or via a separate file. See [Self-Hosted Instance Configuration](/configuration/powersync-service/self-hosted-instances) for the full config reference. +Define your Sync Config in `service.yaml` either inline or via a separate file. See [Self-Hosted Instance Configuration](/configuration/powersync-service/self-hosted-instances) for the full config reference. @@ -85,13 +85,13 @@ Define your sync config in `service.yaml` either inline or via a separate file. docker compose restart powersync ``` - Once the service starts up, it will load the updated sync config and begin processing it while continuing to serve the existing config until initial replication completes. + Once the service starts up, it will load the updated Sync Config and begin processing it while continuing to serve the existing config until initial replication completes. ## Option 3: Deploy via API -Deploy sync config at runtime without restarting. Useful for quick iterations during development. +Deploy Sync Config at runtime without restarting. Useful for quick iterations during development. The API is disabled when Sync Streams (or legacy Sync Rules) are defined in `service.yaml`. diff --git a/maintenance-ops/self-hosting/usage-reporting.mdx b/maintenance-ops/self-hosting/usage-reporting.mdx index bd7631e1..453b9ef3 100644 --- a/maintenance-ops/self-hosting/usage-reporting.mdx +++ b/maintenance-ops/self-hosting/usage-reporting.mdx @@ -22,10 +22,17 @@ Below are the data points collected every few minutes and associated with a rand | chunks\_replicated\_total | counter | | operations\_synced\_total | counter | | replication\_storage\_size\_bytes | gauge | +| attributed\_source\_records\_bytes | gauge | | operation\_storage\_size\_bytes | gauge | +| attributed\_bucket\_data\_bytes | gauge | | parameter\_storage\_size\_bytes | gauge | +| attributed\_parameter\_indexes\_bytes | gauge | +| object\_storage\_size\_bytes | gauge | +| attributed\_object\_storage\_bytes | gauge | | concurrent\_connections | gauge | +The four `attributed_*` gauges include `sync_config_id` and `sync_config_state` attributes. The Prometheus endpoint also includes `version_label` when the Sync Config has one. The Service removes `version_label` before sharing telemetry with PowerSync. + To scrape your self-hosted PowerSync Service metrics, please see the [Monitoring](/maintenance-ops/self-hosting/monitoring) docs page for more details. ### Opting Out diff --git a/resources/feature-status.mdx b/resources/feature-status.mdx index ce09dadd..4a75b996 100644 --- a/resources/feature-status.mdx +++ b/resources/feature-status.mdx @@ -60,6 +60,7 @@ Below is a summary of the current main PowerSync features and their release stat | Postgres | GA | | | | | **PowerSync Service** | | +| Storage Version 4 (incremental reprocessing, S3 object storage) | Beta | | Open Edition | GA | | Enterprise Self-Hosted | GA | | Sync Streams | GA | @@ -107,4 +108,4 @@ Below is a summary of the current main PowerSync features and their release stat | CLI | Beta | Also see: -- [PowerSync Roadmap](https://roadmap.powersync.com) \ No newline at end of file +- [PowerSync Roadmap](https://roadmap.powersync.com) diff --git a/resources/usage-and-billing/reducing-usage.mdx b/resources/usage-and-billing/reducing-usage.mdx index 6d7f7f55..a8a8cc7f 100644 --- a/resources/usage-and-billing/reducing-usage.mdx +++ b/resources/usage-and-billing/reducing-usage.mdx @@ -57,7 +57,7 @@ PowerSync processes source changes at the row level. Rewriting a row selected by ### Deployments -Deploying Sync Streams recreates the buckets. Existing clients then sync the new buckets, which can temporarily increase data synced even if the underlying rows did not change. Group related configuration changes when practical instead of deploying many small updates. +Deploying Sync Streams recreates the buckets, and existing clients download all their data again even if the underlying rows did not change. With [incremental reprocessing](/sync/advanced/storage-version-4#incremental-reprocessing) (Beta), clients download only the data for streams you added or changed. In both cases, group related changes when practical instead of deploying many small updates. ## Reducing Data Hosted diff --git a/resources/usage-and-billing/usage-and-billing-faq.mdx b/resources/usage-and-billing/usage-and-billing-faq.mdx index 9748208b..e17ed62f 100644 --- a/resources/usage-and-billing/usage-and-billing-faq.mdx +++ b/resources/usage-and-billing/usage-and-billing-faq.mdx @@ -174,7 +174,7 @@ Each connected `PowerSyncDatabase` counts as a concurrent client. If one user se Some events increase usage temporarily and don't indicate a problem: - New app installations download all their subscribed data, including the operations history, during the initial sync. Expect higher data synced when onboarding new groups of users. A large operations history makes these initial syncs bigger; see [More Operations Than Rows](#more-operations-than-rows). -- Deploying changes to [Sync Streams](/sync/streams/overview) recreates buckets, and existing clients sync the new buckets. Data hosted can also increase temporarily while the previous and new bucket data exist concurrently. See [Deployments](/resources/usage-and-billing/reducing-usage#deployments). [Incremental reprocessing](https://roadmap.powersync.com/c/85-more-efficient-sync-reprocessing) is planned to limit this work to changed definitions. +- Deploying changes to [Sync Streams](/sync/streams/overview) recreates buckets, and existing clients download the new buckets. Data hosted can also increase temporarily while the previous and new bucket data exist concurrently. With [incremental reprocessing](/sync/advanced/storage-version-4#incremental-reprocessing) (Beta), only the streams you added or changed are affected. See [Deployments](/resources/usage-and-billing/reducing-usage#deployments). - Defragmentation reduces what new installations download, but existing clients re-sync each touched row. ## Accident Forgiveness diff --git a/snippets/create-cloud-instance-cli.mdx b/snippets/create-cloud-instance-cli.mdx index c00b418d..168e5ac6 100644 --- a/snippets/create-cloud-instance-cli.mdx +++ b/snippets/create-cloud-instance-cli.mdx @@ -6,7 +6,7 @@ powersync login powersync init cloud ``` -This creates a `powersync/` directory with `service.yaml` (instance name, region, connection, auth) and `sync-config.yaml` (sync config). Edit `powersync/service.yaml` to set your instance name and region. +This creates a `powersync/` directory with `service.yaml` (instance name, region, connection, auth) and `sync-config.yaml` (Sync Config). Edit `powersync/service.yaml` to set your instance name and region. Then create the instance and link the directory to it: diff --git a/sync/advanced/compatibility.mdx b/sync/advanced/compatibility.mdx index ebad7620..6d74303a 100644 --- a/sync/advanced/compatibility.mdx +++ b/sync/advanced/compatibility.mdx @@ -1,6 +1,6 @@ --- title: "Compatibility" -description: "Configure compatibility editions and bucket storage format version in PowerSync's sync config." +description: "Configure compatibility editions and bucket storage format version in PowerSync's Sync Config." --- To ensure consistency, it is important that the PowerSync Service does not interpret the same source row in different ways after updating to a new version. @@ -10,7 +10,7 @@ At the same time, we want to fix bugs or other inaccuracies that have accumulate To make this trade‑off explicit, you choose whether to keep the existing behavior or turn on newer fixes that slightly change how data is processed. -Use the `config` block in your sync config YAML to choose the behavior. There are two ways to turn fixes on: +Use the `config` block in your Sync Config YAML to choose the behavior. There are two ways to turn fixes on: 1. Set an `edition` to enable the full set of fixes for that edition. This is the recommended approach for new projects. 2. Toggle individual options for more fine‑grained control. @@ -58,36 +58,42 @@ streams: ## Storage Version -The PowerSync Service stores replicated bucket data in [bucket storage](/architecture/powersync-service#bucket-storage). That data uses a storage version that can evolve when you deploy new Sync Streams. This versioning approach avoids large upfront migrations on existing bucket data when the Service introduces bigger storage changes. Each time your sync config is deployed and processed, the bucket data written for that deployment uses a specific storage version. +A storage version tells the PowerSync Service how to organize prepared sync data in the [bucket storage database](/architecture/powersync-service#bucket-storage). + +Changing the version does not rewrite the current data in place. When you next deploy the Sync Config, PowerSync prepares a new copy using the selected version. Clients continue using the current copy until the new one is ready. This avoids taking the instance offline for a bucket storage migration. ### Optional `config.storage_version` -You can pin the bucket storage version by setting it under the `config` block: +You can choose the bucket storage version in the `config` block: ```yaml config: edition: 3 - storage_version: 2 # version 2 (stable) and 3 (experimental) are currently supported + storage_version: 4 streams: todos: query: SELECT * FROM todos WHERE owner_id = auth.user_id() ``` -### When to Set `storage_version` Explicitly +When you omit `storage_version`, the PowerSync Service uses its default, which is version 2 in v1.26.0. Self-hosted deployments can change the default with `storage.default_storage_version` in `service.yaml`. On PowerSync Cloud, PowerSync manages the default. -In most deployments you can omit `storage_version`. The PowerSync Service then uses the latest stable storage version it supports. You should only set this field if you need more control, e.g.: +Set `storage_version` when you need to: -1. Service downgrade: If you need to run an older Service version that only supports up to a given storage version, deploy sync config with that `storage_version`, wait until reprocessing for that deployment has finished, then downgrade the Service. -2. Experiments: Opt into an experimental storage version in non-production environments. -3. Delaying a storage upgrade: Change other sync config while keeping bucket data on an older stable storage version until you are ready for the newer format. +- Use [storage version 4](/sync/advanced/storage-version-4), which is in Beta and enables incremental reprocessing and S3 object storage. +- Delay a storage upgrade. When the default moves to a newer version, pin `storage_version` to the version your data already uses. You can then keep deploying Sync Config changes without migrating the bucket data and making clients download everything again. Remove the pin when you are ready for the new format. +- Prepare for a Service downgrade. Select a version supported by the older Service, deploy the Sync Config, and wait for the new copy to finish before downgrading. -### Stable and Experimental Versions +### Available Versions -The Service distinguishes stable and experimental storage versions as follows: +| Version | Bucket storage | Status | +| --- | --- | --- | +| `1` | MongoDB or Postgres | Legacy format, retained for existing deployments. | +| `2` | MongoDB or Postgres | Stable. The default in v1.26.0. | +| `3` | MongoDB | Experimental. The unstable predecessor of version 4, with the same format. Do not use it in production. Deploy with version 4 instead. | +| `4` | MongoDB | Stable. Enables [incremental reprocessing and S3 object storage](/sync/advanced/storage-version-4) (Beta). | -- Even numbers (for example `2`, `4`) denote stable formats. Once a stable version is supported, newer Service releases are expected to keep supporting it until it is officially deprecated. -- Odd numbers (for example `3`) denote experimental formats. The layout may change without notice and support may be removed in a future release. Use odd versions only for testing, not production. +Version numbers follow a pattern. Even numbers are stable formats: they stay backwards compatible and later Service versions continue to support them. Stable makes no guarantee that a format is bug-free. Odd numbers are experimental formats: their layout can change without notice and support can be removed in a later release, so use them only for testing, never in production. ## Supported Fixes diff --git a/sync/advanced/storage-version-4.mdx b/sync/advanced/storage-version-4.mdx new file mode 100644 index 00000000..ed056610 --- /dev/null +++ b/sync/advanced/storage-version-4.mdx @@ -0,0 +1,184 @@ +--- +title: "Storage Version 4" +description: "Opt in to storage version 4 for faster sync, incremental reprocessing of Sync Streams changes, and S3 object storage." +--- + +Storage version 4 is a new version of the format the PowerSync Service uses to store the data it syncs to clients. It is in [Beta](/resources/feature-status) as of PowerSync Service v1.26.0. + +Compared to version 2, it provides: + +- Faster sync and faster reprocessing after a deployment. +- [Incremental reprocessing](#incremental-reprocessing): a Sync Streams deployment reprocesses only the streams you added or changed. Clients no longer download all their data again after every deployment. +- [S3 object storage](#s3-object-storage): larger blocks of synced data move from the storage database to S3. This removes the main bottleneck when many clients sync at once or sync large amounts of data. + +## Availability + +The PowerSync Cloud and self-hosted columns below apply during the Beta only. Once storage version 4 is generally available, it is the default for all instances and you no longer need to opt in. S3 object storage is then also enabled on all PowerSync Cloud instances. Self-hosted deployments configure it themselves. + +| | Source database | Sync Config | PowerSync Cloud (Beta) | Self-hosted (Beta) | +| --- | --- | --- | --- | --- | +| Storage version 4 | Any | Sync Streams or Sync Rules | Free plan: automatic. Other plans: [opt in](#opt-in). | [Opt in](#opt-in) | +| Incremental reprocessing | MongoDB | Sync Streams | Included with version 4 | Included with version 4 | +| S3 object storage | Any | Sync Streams or Sync Rules | Enabled per instance by PowerSync on request | [Configure it yourself](#s3-object-storage) | + +Storage version 4 requires MongoDB as the [bucket storage database](/architecture/powersync-service#bucket-storage). PowerSync Cloud always uses MongoDB. Self-hosted deployments with Postgres bucket storage cannot use version 4. + +Incremental reprocessing for Postgres and other source databases is planned. See the [proposal](https://github.com/orgs/powersync-ja/discussions/349) for background. It is not supported for legacy [Sync Rules](/sync/rules/overview). If you still use Sync Rules, [migrate to Sync Streams](/sync/streams/migration). + +## Opt In + +Version 4 is not the default in PowerSync Service v1.26.0. Moving a Sync Config to version 4 runs like any other deployment: + +1. PowerSync reprocesses all data selected by your Sync Config in the background. The current version keeps serving clients, so there is no downtime. +2. When the new copy is ready, PowerSync switches to it. On PowerSync Cloud, this appears as a new deploy event in the PowerSync Dashboard. +3. Clients download their data again once, as after any deployment before version 4. On self-hosted deployments with many clients, scale out the API before the switch to absorb the re-sync. + +After this first deployment, later Sync Streams deployments use incremental reprocessing automatically when your instance meets its requirements. There is no separate setting. + +### PowerSync Cloud + +Free plan instances are upgraded automatically during the Beta. No action is needed. + +On other plans, add `storage_version: 4` to the `config` block of your Sync Config and deploy it: + +```yaml +config: + edition: 3 + storage_version: 4 + +streams: + todos: + query: SELECT * FROM todos WHERE owner_id = auth.user_id() +``` + +### Self-Hosted + +Add `storage_version: 4` to each Sync Config as shown above. Alternatively, set [`default_storage_version`](/configuration/powersync-service/self-hosted-instances#param-default-storage-version) for newly deployed Sync Configs in `service.yaml`: + +```yaml service.yaml +storage: + type: mongodb + uri: !env PS_MONGO_STORAGE_URI + default_storage_version: 4 +``` + +The default only applies to Sync Configs deployed after the change. Redeploy existing Sync Configs to move them to version 4. + +To move a Sync Config back to version 2, set `storage_version: 2` and deploy again. This is another full reprocess. + +## Incremental Reprocessing + +Incremental reprocessing requires a MongoDB source database, Sync Streams, and storage version 4. It is active whenever all three apply. + +Without it, every deployment reads all data selected by the Sync Config from your source database and prepares a complete new copy. Clients then download all their data again, even if only one stream changed. + +With incremental reprocessing, PowerSync compares the new Sync Config with the current one and reprocesses only the streams you added or changed. Unchanged streams keep their data on the PowerSync Service and on clients. Deployments finish faster, your source database does less work, and clients download only the data for affected streams they subscribe to. + +- Adding a stream reads only the data that stream selects. +- Removing a stream deletes its data without reading from the source database. +- Renaming a stream counts as removing it and adding a new one, so its data is rebuilt. +- Changing a stream's queries reprocesses that stream. Other streams are unaffected. + +The time saved depends on how your data is split across streams. If one stream selects most of your data, changing that stream still takes about as long as a full reprocess. + +PowerSync favors correctness over reuse. When it cannot confirm that a change leaves a stream's data unchanged, it rebuilds that stream. A deployment that reprocesses more than you expect is not an error. + +Event definitions for [Custom Write Checkpoints](/handling-writes/custom-write-checkpoints) follow the same rules. Unchanged events keep their data, and new or changed events are read again. + +### When PowerSync Reprocesses Everything + +Some changes start a full reprocess, after which clients download all their data again: + +- The first deployment on storage version 4. +- Changes to the `config` block of the Sync Config, such as `edition`, compatibility fixes, or `storage_version`. +- The **Defragment** action in the PowerSync Dashboard, which exists to rebuild all data. See [Defragmenting](/maintenance-ops/compacting-buckets#defragmenting). +- Replication failures, for example when PowerSync loses its position in the MongoDB change stream and has to start over. + +### Sync Config Versions and Replication Streams + +Every deployment creates a new Sync Config version. A replication stream is the copy of your data that PowerSync replicates from the source database and serves to clients. The PowerSync Dashboard shows both, for example as **Sync Streams 3 (246a)**: `3` is the Sync Config version and `246a` identifies the replication stream. In logs, the replication stream appears as a name ending in the same identifier, such as `powersync__1_246a`. + +Before storage version 4, every deployment also created a new replication stream, so the two always changed together. With incremental reprocessing they change independently: + +- An incremental deployment creates a new Sync Config version and keeps the replication stream. Only the version changes. +- A full reprocess creates a new replication stream. After **Defragment** or a replication failure only the identifier changes, because the Sync Config did not. After a change to the `config` block, both change. + +If the identifier changed, PowerSync reprocessed everything and clients download all their data again. + +### Checking What a Deployment Reprocessed + +After a deployment, the [instance logs](/maintenance-ops/monitoring-and-alerting#instance-logs) list what PowerSync reused and what it read again: + +``` +info: Incremental reprocessing sync config update: +Reused definitions: + - type=bucket_data, id=1, name=user_lists|0 +New definitions: + - type=bucket_data, id=6, name=user_tasks|0, tables=tasks +Definitions to drop after switching: + - type=bucket_data, id=5, name=user_tasks|0 +``` + +Reused definitions keep their data. New definitions are read from the source database. One stream can appear as several entries. Check this output when a deployment takes longer or reprocesses more than you expect. + +## S3 Object Storage + +Your instance keeps the data it syncs to clients in its bucket storage database, alongside everything else it needs to run. With S3 object storage, larger blocks of that data move to Amazon S3 or an S3-compatible object store, and the PowerSync Service syncs them to clients directly from there. Smaller blocks, and the metadata that locates each block, stay in MongoDB. + +This reduces load on the storage database: + +- Initial sync is faster, most noticeably for clients that sync large amounts of data and when many clients connect at the same time. +- The instance is more stable, because the storage database no longer does the heavy lifting. +- An instance can serve far more concurrent clients without compromising on performance. + +Clients connect only to the PowerSync Service and never to the object store, so no client changes are needed. If the object store becomes unreachable, sync is interrupted until it recovers. Clients reconnect and resume automatically. + +S3 object storage requires MongoDB bucket storage and storage version 4. It works with Sync Streams and legacy Sync Rules. + + + S3 object storage holds PowerSync's internal sync data. To store files uploaded by your app, use [Attachments](/client-sdks/advanced/attachments). + + +### PowerSync Cloud + +During the Beta, PowerSync enables S3 object storage per instance. [Contact us](/resources/contact-us) if you want it on your instance before we enable it for all instances. + +### Self-Hosted + + + + Create a bucket in the same region as the PowerSync Service. Use a dedicated bucket, or a unique `prefix` per PowerSync instance, so that instances never read or delete each other's files. Give the PowerSync Service permission to list the bucket and to read, write, and delete objects under the prefix. + + Leave object versioning off, or suspend it if the bucket already has it, and leave Object Lock off. PowerSync deletes files itself once they are no longer needed, so versioning keeps charging for old versions and locked objects cannot be cleaned up. Do not add an expiration lifecycle rule: an expired object may still be referenced by MongoDB, which breaks sync for that data. + + + Add `object_storage` to the `storage` section of `service.yaml`: + + ```yaml service.yaml + storage: + type: mongodb + uri: !env PS_MONGO_STORAGE_URI + default_storage_version: 4 + object_storage: + type: s3 + bucket: powersync-bucket-data + region: us-east-1 + prefix: production + + api: + parameters: + max_concurrent_connections: 1000 + ``` + + Without `access_key_id` and `secret_access_key`, PowerSync uses the AWS credentials available to the process, such as an IAM role. Raising [`max_concurrent_connections`](/configuration/powersync-service/self-hosted-instances#param-max-concurrent-connections) from its default of 200 is what lets each API process serve more clients. Without S3 object storage, the storage database limits how far you can raise it. For S3-compatible providers such as MinIO or Cloudflare R2, also set `endpoint`, and set `force_path_style: true` if the provider requires path-style requests. + + + Deploy your Sync Configs on storage version 4 as described in [Opt In](#opt-in). Sync Configs on version 2 keep all data in MongoDB, even when `object_storage` is configured. + + Once replication reaches a healthy checkpoint, confirm that objects appear under the prefix, run a test initial sync, and run `compact` once to surface permission errors early. + + + +The [configuration reference](/configuration/powersync-service/self-hosted-instances#param-object-storage) lists all `object_storage` options, including timeouts, request concurrency, and the size threshold below which blocks stay in MongoDB. + +Keep the scheduled [compact](/maintenance-ops/compacting-buckets) job running. It removes files that are no longer needed. The `teardown` command deletes PowerSync's files under the prefix before it drops the storage database. The `powersync_object_storage_size_bytes` [metric](/maintenance-ops/self-hosting/monitoring) reports how much object storage PowerSync uses. diff --git a/sync/streams/ctes.mdx b/sync/streams/ctes.mdx index 7a00c416..aace566a 100644 --- a/sync/streams/ctes.mdx +++ b/sync/streams/ctes.mdx @@ -3,7 +3,7 @@ title: "Common Table Expressions (CTEs)" description: "Reuse common query patterns across Sync Streams using Common Table Expressions (CTEs) to simplify configurations and reduce duplication." --- -When streams need reusable filtering logic, you can define it once in a Common Table Expression (CTE) and reference it in queries. CTEs can be defined at the stream level (scoped to one stream) or at the top level of your sync config (shared across all streams). This keeps stream definitions DRY and makes it easier to maintain. For the supported syntax of the `with` block and CTE rules, see [Supported SQL — CTE and WITH syntax](/sync/supported-sql#cte-and-with-syntax). +When streams need reusable filtering logic, you can define it once in a Common Table Expression (CTE) and reference it in queries. CTEs can be defined at the stream level (scoped to one stream) or at the top level of your Sync Config (shared across all streams). This keeps stream definitions DRY and makes it easier to maintain. For the supported syntax of the `with` block and CTE rules, see [Supported SQL — CTE and WITH syntax](/sync/supported-sql#cte-and-with-syntax). ## Why Use CTEs diff --git a/sync/streams/migration.mdx b/sync/streams/migration.mdx index 083a5799..8ba4a17e 100644 --- a/sync/streams/migration.mdx +++ b/sync/streams/migration.mdx @@ -41,7 +41,7 @@ If you want "sync everything upfront" behavior (like Sync Rules), set [`auto_sub - PowerSync Service v1.20.0+ (Cloud instances already meet this) - Latest SDK versions with [Rust-based sync client](https://releases.powersync.com/announcements/improved-sync-performance-in-our-client-sdks) (enabled by default on latest SDKs) -- `config: edition: 3` in your sync config +- `config: edition: 3` in your Sync Config @@ -103,7 +103,7 @@ You can generate a Sync Streams draft from your existing Sync Rules in two ways: 1. **Dashboard:** In the [PowerSync Dashboard](https://dashboard.powersync.com/), use the **Migrate to Sync Streams** button. It converts your Sync Rules into a Sync Streams draft that you can review before deploying. -2. **CLI:** Run `powersync migrate sync-rules` to produce a Sync Streams draft from your current sync config. +2. **CLI:** Run `powersync migrate sync-rules` to produce a Sync Streams draft from your current Sync Config. The output uses `auto_subscribe: true` by default, preserving your existing sync-everything-upfront behavior so no client-side changes are required when you first deploy. @@ -236,7 +236,7 @@ const page2 = await db.syncStream('posts', { page_number: 2 }).subscribe(); ## Client-Side Changes -After updating your sync config, update your client code to use subscriptions: +After updating your Sync Config, update your client code to use subscriptions: ```js // Before (Sync Rules with Client Parameters) diff --git a/sync/supported-sql.mdx b/sync/supported-sql.mdx index 870880d3..a99508eb 100644 --- a/sync/supported-sql.mdx +++ b/sync/supported-sql.mdx @@ -298,7 +298,7 @@ For how to use JOINs in your stream queries (when to use them, patterns, and exa Supported in Sync Streams only. Not available in Sync Rules. -Common Table Expressions (CTEs) can be defined in a `with:` block **inside a stream** (stream-level, scoped to that stream) or at the **top level** of the sync config (global, shared across all streams). Each CTE is a name and a single `SELECT` query. The following rules apply: +Common Table Expressions (CTEs) can be defined in a `with:` block **inside a stream** (stream-level, scoped to that stream) or at the **top level** of the Sync Config (global, shared across all streams). Each CTE is a name and a single `SELECT` query. The following rules apply: - **Stream-level CTEs take precedence over global CTEs.** If a stream defines a CTE with the same name as a global CTE, the stream-level definition is used within that stream. - **Global CTE names must not shadow source table names.** If a global CTE has the same name as a database table or collection, PowerSync reports a validation error. Stream-level CTE names are not subject to this restriction. diff --git a/tools/cli.mdx b/tools/cli.mdx index e011c14f..50b3926a 100644 --- a/tools/cli.mdx +++ b/tools/cli.mdx @@ -27,7 +27,7 @@ description: "Manage PowerSync Cloud and self-hosted instances from the command -The PowerSync CLI lets you manage PowerSync Service instances, deploy sync config (your Sync Streams or Sync Rules), generate client schemas, run diagnostics, and more. It is distributed as the [powersync](https://www.npmjs.com/package/powersync) npm package. +The PowerSync CLI lets you manage PowerSync Service instances, deploy Sync Config (your Sync Streams or Sync Rules), generate client schemas, run diagnostics, and more. It is distributed as the [powersync](https://www.npmjs.com/package/powersync) npm package. The CLI is currently in [beta](/resources/feature-status). We recommend it for @@ -38,7 +38,7 @@ The PowerSync CLI lets you manage PowerSync Service instances, deploy sync confi For a full step-by-step flow using the CLI, use the [Setup Guide](/intro/setup-guide): choose the **CLI (Cloud)** or **CLI (Self-Hosted)** tab in steps 2–5 to configure your instance, connect the - source database, deploy sync config, and generate development tokens. + source database, deploy Sync Config, and generate development tokens. The CLI was overhauled in version 0.9.0. The redesign is based on this [design proposal](https://docs.google.com/document/d/1iqpJF2gog2jB-ZWeN8TBEjcad8aBKNKbue2yJ21q_-s/edit). @@ -122,7 +122,7 @@ You can create instances, deploy and pull config, run all Cloud commands. ``` - Edit `powersync/service.yaml` (name, region, replication, auth) and sync config; use `!env` for secrets. + Edit `powersync/service.yaml` (name, region, replication, auth) and Sync Config; use `!env` for secrets. ```bash @@ -253,7 +253,7 @@ Then use the same commands as any self-hosted instance (`powersync status`, `pow | `powersync pull instance --instance-id=` | Download Cloud config to local files | | `powersync deploy` | Deploy full config to linked Cloud instance | | `powersync deploy service-config` | [Cloud] Deploy only service config | -| `powersync deploy sync-config` | [Cloud] Deploy only sync config | +| `powersync deploy sync-config` | [Cloud] Deploy only Sync Config | | `powersync validate` | Validate config and Sync Streams/Rules | | `powersync edit config` | Open Config Studio (Monaco editor) | | `powersync status` | Instance diagnostics (Cloud and self-hosted) | @@ -270,7 +270,7 @@ Run `powersync --help` or `powersync --help` for flags. Full [command ## Deploying From CI (e.g. GitHub Actions) -You can automate sync config (and full config) deployments using the CLI in CI. Use the config directory as the source of truth: keep `service.yaml` and `sync-config.yaml` in the repo (with secrets via `!env` and CI secrets), then run `powersync deploy` (or `powersync deploy sync-config`). +You can automate Sync Config (and full config) deployments using the CLI in CI. Use the config directory as the source of truth: keep `service.yaml` and `sync-config.yaml` in the repo (with secrets via `!env` and CI secrets), then run `powersync deploy` (or `powersync deploy sync-config`). **Secrets:** Set `PS_ADMIN_TOKEN` to your PowerSync personal access token. If the workflow does not use a linked directory, also set `INSTANCE_ID`. For self-hosted, `API_URL` can specify the PowerSync API base URL. @@ -280,7 +280,7 @@ You can automate sync config (and full config) deployments using the CLI in CI. href="https://github.com/powersync-community/github-actions-demo" horizontal > - Example: deploy sync config on push to main + Example: deploy Sync Config on push to main ## Migrating From the Previous CLI diff --git a/tools/local-development.mdx b/tools/local-development.mdx index 4175d7bd..97a6a03c 100644 --- a/tools/local-development.mdx +++ b/tools/local-development.mdx @@ -110,7 +110,7 @@ storage: # The port which the PowerSync API server will listen on port: 8080 -# Points to the sync config file +# Points to the Sync Config file sync_config: path: sync-config.yaml @@ -159,5 +159,5 @@ docker compose up - [PowerSync CLI](https://github.com/powersync-ja/powersync-cli) — open source CLI; use it to scaffold and run a Docker-based local stack - [self-host-demo](https://github.com/powersync-ja/self-host-demo) — complete working examples with Docker Compose - [Self-Hosted Instance Configuration](/configuration/powersync-service/self-hosted-instances) — full `service.yaml` reference -- [Sync Streams](/sync/streams/overview) — sync config syntax +- [Sync Streams](/sync/streams/overview) — Sync Config syntax - [Generate a Development Token](/intro/setup-guide#5-generate-a-development-token) — for testing without a full auth setup diff --git a/tools/powersync-dashboard.mdx b/tools/powersync-dashboard.mdx index fee3033a..1791b01b 100644 --- a/tools/powersync-dashboard.mdx +++ b/tools/powersync-dashboard.mdx @@ -69,7 +69,7 @@ When you navigate to a specific instance, you'll see a left sidebar with various - **Health** - Overview of its connection health, deploy history, replication status, and recently connected clients - **Database Connections** - Configure and manage the source database connection - **Client Auth** - Configure authentication settings -- **Sync Streams / Sync Rules** - Edit, validate, and deploy your sync config. +- **Sync Streams / Sync Rules** - Edit, validate, and deploy your Sync Config. - **Sync Test** - Test your Sync Streams (or legacy Sync Rules) - **Client SDK Setup** - Generate the [client-side schema](/intro/setup-guide#define-your-client-side-schema) based on your deployed [Sync Streams](/sync/streams/overview) or [Sync Rules](/sync/rules/overview) - **Write API** - Resources for exposing the write API endpoint @@ -87,7 +87,7 @@ In the top bar, you'll see a "Connect" button that provides quick access to your Here are some of the most common tasks you'll perform in the dashboard: -- **Edit and deploy Sync Streams / Sync Rules** - Select your project and instance and go to the **Sync Streams** (or legacy **Sync Rules**) view to edit your sync config, then click **"Validate"** and **"Deploy"** to deploy +- **Edit and deploy Sync Streams / Sync Rules** - Select your project and instance and go to the **Sync Streams** (or legacy **Sync Rules**) view to edit your Sync Config, then click **"Validate"** and **"Deploy"** to deploy - **Generate development token** - Navigate to the **Client Auth** and ensure the **Development tokens** setting is checked. Click the "Connect" button in the top bar and follow instructions to generate a [development token](/configuration/auth/development-tokens). - **Launch the Sync Diagnostics Client** - Navigate to the **Sync Test**, generate a development token and click "Launch" to launch the [Sync Diagnostics Client](/tools/diagnostics-client). - **Copy your instance URL** - Click **Connect** in the top bar and copy the instance URL from the dialog. @@ -130,7 +130,7 @@ A custom role is a set of permissions that you choose. Each permission covers on The access levels are as follows: - **View** gives read-only access to that area. The member can see it but cannot change anything. -- **Deploy** applies only to Instances. It gives View access plus the ability to deploy changes to an instance, such as updating the sync config and database connections. It does not allow creating or deleting instances. +- **Deploy** applies only to Instances. It gives View access plus the ability to deploy changes to an instance, such as updating the Sync Config and database connections. It does not allow creating or deleting instances. - **Manage** gives full access to that area, including creating, editing, and deleting. For Instances, it also includes deploying changes. @@ -142,7 +142,7 @@ The table below lists each permission, its available access levels, and what tho | Permission | Access levels | What it grants | | --- | --- | --- | | Projects | View, Manage | View projects. Manage adds creating, renaming, and deleting projects. | -| Instances | View, Deploy, Manage | View instances. Deploy adds deploying changes such as sync config and database connections. Manage adds creating and deleting instances. | +| Instances | View, Deploy, Manage | View instances. Deploy adds deploying changes such as Sync Config and database connections. Manage adds creating and deleting instances. | | Instance logs | View | View instance logs. | | Alert Rules | View, Manage | View Alert Rules. Manage adds creating, editing, and deleting them. | | Notification Rules | View, Manage | View Notification Rules. Manage adds creating, editing, deleting, and testing them. |