Repository navigation
chore(dev): run local dev, integration and e2e on DocumentDB #3375
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
08825ff
a7be35a
aecade3
e5de341
a4a8b46
a83c279
1909959
10ccf32
42087e9
03bb6b7
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -123,11 +123,15 @@ jobs: | |
| - name: Spin up docker services | ||
| run: | | ||
| docker buildx create --use --driver=docker-container | ||
| docker buildx bake -f ./docker-compose.ci.yml --set *.cache-to="type=gha,mode=max,scope=ci-integration" --set *.cache-from="type=gha,scope=ci-integration" --load | ||
| # One cache scope per image (a scope holds one manifest); db reuses the e2e job's documentdb scope. | ||
| docker buildx bake -f ./docker-compose.ci.yml \ | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 P2 — CI compiles DocumentDB from source on a cold cache inside 15–16 minute job limits On a cache miss (first run of this PR, eviction from the 10 GB GHA cache, or any change under docker/documentdb), each of the 4 integration shards ( Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond |
||
| --set otel-collector.cache-to="type=gha,mode=max,scope=ci-integration" --set otel-collector.cache-from="type=gha,scope=ci-integration" \ | ||
| --set db.cache-to="type=gha,mode=max,scope=documentdb" --set db.cache-from="type=gha,scope=documentdb" \ | ||
| --load | ||
| - name: Build dependencies | ||
| run: make ci-build | ||
| # Integration tests boot a MockServer on fixed ports and share a single | ||
| # MongoDB/ClickHouse database, so they must stay serial (--runInBand) | ||
| # DocumentDB/ClickHouse database, so they must stay serial (--runInBand) | ||
| # within a runner. We shard ACROSS runners instead: each shard is its own | ||
| # VM with its own Docker stack, and jest's --shard splits the test files | ||
| # so the shards run in parallel. | ||
|
|
@@ -146,6 +150,8 @@ jobs: | |
| - name: Run integration tests | ||
| run: | | ||
| docker compose -f ./docker-compose.ci.yml up -d --quiet-pull | ||
| # Wait on DocumentDB only: the collector's healthcheck needs an OpAMP server this job doesn't run. | ||
| docker compose -f ./docker-compose.ci.yml up -d --wait --wait-timeout 300 db | ||
| status=0 | ||
| yarn workspace @hyperdx/common-utils ci:int --shard=${{ matrix.shard }}/${{ strategy.job-total }} || status=1 | ||
| yarn workspace @hyperdx/api ci:int --shard=${{ matrix.shard }}/${{ strategy.job-total }} || status=1 | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -60,7 +60,7 @@ install-tools: | |
| # API server : 30100 + slot | ||
| # App (Next.js) : 30200 + slot | ||
| # OpAMP : 30300 + slot | ||
| # MongoDB : 30400 + slot | ||
| # DocumentDB : 30400 + slot | ||
| # ClickHouse HTTP : 30500 + slot | ||
| # ClickHouse Native : 30600 + slot | ||
| # OTel health : 30700 + slot | ||
|
|
@@ -158,7 +158,8 @@ dev-int: | |
| @echo "Using CI slot $(HDX_CI_SLOT) (project=$(HDX_CI_PROJECT) ch=$(HDX_CI_CH_PORT) mongo=$(HDX_CI_MONGO_PORT) api=$(HDX_CI_API_PORT))" | ||
| @mkdir -p $(HDX_CI_LOGS_DIR) | ||
| @bash scripts/ensure-dev-portal.sh | ||
| docker compose -p $(HDX_CI_PROJECT) -f ./docker-compose.ci.yml up -d | ||
| docker compose -p $(HDX_CI_PROJECT) -f ./docker-compose.ci.yml up -d --build | ||
| docker compose -p $(HDX_CI_PROJECT) -f ./docker-compose.ci.yml up -d --wait --wait-timeout 300 db | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. 🟡 P2 — A failed DocumentDB health wait in dev-int skips the The new Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond |
||
| bash -c 'set -o pipefail; npx nx run @hyperdx/api:dev:int $(FILE) 2>&1 | tee $(HDX_CI_LOGS_DIR)/api-int.log'; ret=$$?; \ | ||
| docker compose -p $(HDX_CI_PROJECT) -f ./docker-compose.ci.yml down -v; \ | ||
| $(call archive-int-logs); \ | ||
|
|
@@ -169,7 +170,8 @@ dev-int-common-utils: | |
| @echo "Using CI slot $(HDX_CI_SLOT) (project=$(HDX_CI_PROJECT) ch=$(HDX_CI_CH_PORT) mongo=$(HDX_CI_MONGO_PORT))" | ||
| @mkdir -p $(HDX_CI_LOGS_DIR) | ||
| @bash scripts/ensure-dev-portal.sh | ||
| docker compose -p $(HDX_CI_PROJECT) -f ./docker-compose.ci.yml up -d | ||
| docker compose -p $(HDX_CI_PROJECT) -f ./docker-compose.ci.yml up -d --build | ||
| docker compose -p $(HDX_CI_PROJECT) -f ./docker-compose.ci.yml up -d --wait --wait-timeout 300 db | ||
| bash -c 'set -o pipefail; npx nx run @hyperdx/common-utils:dev:int $(FILE) 2>&1 | tee $(HDX_CI_LOGS_DIR)/common-utils-int.log'; ret=$$?; \ | ||
| docker compose -p $(HDX_CI_PROJECT) -f ./docker-compose.ci.yml down -v; \ | ||
| $(call archive-int-logs); \ | ||
|
|
@@ -179,6 +181,7 @@ dev-int-common-utils: | |
| ci-int: | ||
| @mkdir -p $(HDX_CI_LOGS_DIR) | ||
| docker compose -p $(HDX_CI_PROJECT) -f ./docker-compose.ci.yml up -d --quiet-pull | ||
| docker compose -p $(HDX_CI_PROJECT) -f ./docker-compose.ci.yml up -d --wait --wait-timeout 300 db | ||
| bash -c 'set -o pipefail; npx nx run-many -t ci:int --parallel=false --output-style=stream 2>&1 | tee $(HDX_CI_LOGS_DIR)/ci-int.log'; ret=$$?; \ | ||
| docker compose -p $(HDX_CI_PROJECT) -f ./docker-compose.ci.yml down -v; \ | ||
| $(call archive-int-logs); \ | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -30,7 +30,18 @@ yarn dev # or equivalently: make dev | |
| ## Environment Configuration | ||
|
|
||
| - `.env.development`: Development environment variables | ||
| - Docker Compose manages ClickHouse, MongoDB, OTel Collector | ||
| - Docker Compose manages ClickHouse, DocumentDB, OTel Collector | ||
| - The metadata store is [DocumentDB](https://github.com/documentdb/documentdb) | ||
| (Postgres plus a Mongo wire-protocol gateway), built from | ||
|
Member
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I was trying to connect via postgres directly as well, but we dont expose the port. do you think its worth exposing or later problem?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Hmm, I'd say that we probably don't need the PG port exposed for now. Added a |
||
| `docker/documentdb/Dockerfile`. The API still talks to it through Mongoose. | ||
| To browse it with MongoDB Compass or `mongosh`, connect to | ||
| `mongodb://hyperdx:hyperdx@localhost:<HDX_DEV_MONGO_PORT>/?tls=true&tlsAllowInvalidCertificates=true&authMechanism=SCRAM-SHA-256`. | ||
| Everything works except Compass's Performance tab (the gateway doesn't | ||
| implement `serverStatus` or `top`). | ||
| For the Postgres underneath (documents are BSON rows in `documentdb_data`), | ||
| run `docker exec -it -u postgres hdx-dev-<slot>-db-1 psql -h /tmp -U postgres -d postgres`. | ||
| Dev data lives in `.volumes/docdb_dev_<slot>`; data from the old MongoDB | ||
| volumes (`.volumes/db_dev_<slot>`) is not migrated, so the dev stack starts empty. | ||
| - Hot reload enabled for all services in development | ||
|
|
||
| ## Worktree Isolation (Multi-Agent / Multi-Developer) | ||
|
|
@@ -59,7 +70,7 @@ integration tests (14320-40098) and E2E tests (20320-21399). | |
| | API server | 30100 | 30100 - 30199 | `HYPERDX_API_PORT` | | ||
| | App (Next.js) | 30200 | 30200 - 30299 | `HYPERDX_APP_PORT` | | ||
| | OpAMP | 30300 | 30300 - 30399 | `HYPERDX_OPAMP_PORT` | | ||
| | MongoDB | 30400 | 30400 - 30499 | `HDX_DEV_MONGO_PORT` | | ||
| | DocumentDB | 30400 | 30400 - 30499 | `HDX_DEV_MONGO_PORT` | | ||
| | ClickHouse HTTP | 30500 | 30500 - 30599 | `HDX_DEV_CH_HTTP_PORT` | | ||
| | ClickHouse Native | 30600 | 30600 - 30699 | `HDX_DEV_CH_NATIVE_PORT` | | ||
| | OTel health | 30700 | 30700 - 30799 | `HDX_DEV_OTEL_HEALTH_PORT` | | ||
|
|
@@ -173,15 +184,15 @@ Port mapping (base + slot): | |
| | Service | Default port (slot 0) | Variable | | ||
| | --------------- | --------------------- | ----------------- | | ||
| | ClickHouse HTTP | 18123 | HDX_CI_CH_PORT | | ||
| | MongoDB | 39999 | HDX_CI_MONGO_PORT | | ||
| | DocumentDB | 39999 | HDX_CI_MONGO_PORT | | ||
| | API test server | 19000 | HDX_CI_API_PORT | | ||
| | OpAMP | 14320 | HDX_CI_OPAMP_PORT | | ||
|
|
||
| **CI Testing Notes:** | ||
|
|
||
| - Uses separate Docker Compose configuration (`docker-compose.ci.yml`) | ||
| - Isolated test environment with unique `-p int-<slot>` project name | ||
| - Includes all necessary services (ClickHouse, MongoDB, OTel Collector) | ||
| - Includes all necessary services (ClickHouse, DocumentDB, OTel Collector) | ||
| - Tests run against real database instances for accurate integration testing | ||
|
|
||
| ### E2E Testing | ||
|
|
@@ -198,7 +209,7 @@ E2E port mapping (base + slot): | |
| | ClickHouse HTTP | 20500 | 20500 - 20599 | `HDX_E2E_CH_PORT` | | ||
| | ClickHouse Native | 20600 | 20600 - 20699 | `HDX_E2E_CH_NATIVE_PORT` | | ||
| | API server | 21000 | 21000 - 21099 | `HDX_E2E_API_PORT` | | ||
| | MongoDB | 21100 | 21100 - 21199 | `HDX_E2E_MONGO_PORT` | | ||
| | DocumentDB | 21100 | 21100 - 21199 | `HDX_E2E_MONGO_PORT` | | ||
| | App (local) | 21200 | 21200 - 21299 | `HDX_E2E_APP_LOCAL_PORT` | | ||
| | App (fullstack) | 21300 | 21300 - 21399 | `HDX_E2E_APP_PORT` | | ||
|
|
||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
🟡 P2 — Building the DocumentDB image from source on a cache miss probably won't fit the 16/15-minute CI job timeouts
The Dockerfile compiles the C extensions serially (
make install-documentdb, no -j) and runs a releasecargo buildof the gateway. Any change todocker/documentdb/**, or the GHA cache evicting the largedocumentdbscope (mode=max keeps the Rust and C builder layers), makes all 4 integration shards (timeout-minutes: 16) and all 4 e2e shards (.github/workflows/e2e-tests.yml,timeout-minutes: 15) cold-build at once before tests start. Fix: build the image once in a dedicated upstream job (or push it to GHCR) that the shardsneeds:/pull, or raise the timeouts to allow for a cold build.Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond