Skip to content

perf: skip redundant deep clone when listing pushes from file DB - #52

Open
devin-ai-integration[bot] wants to merge 2 commits into
mainfrom
devin/task-126-push-list-skip-reclone
Open

devin-ai-integration[bot] wants to merge 2 commits into
mainfrom
devin/task-126-push-list-skip-reclone

Conversation

@devin-ai-integration

@devin-ai-integration devin-ai-integration Bot commented Nov 20, 2026 •

Copy link
Copy Markdown

Description

GET /api/v1/push (the dashboard's push list) on the default file-backed (neDB) database spends ~40% of its server time re-cloning documents that neDB has already deep-copied.

db.find() in @seald-io/nedb 4.x returns model.deepCopy of every match (datastore.js: new Cursor(this, query, docs => docs.map(doc => model.deepCopy(doc)))). getPushes in src/db/file/pushes.ts then ran toClass on each one, which does a full JSON.parse(JSON.stringify(doc)) before setting the prototype. This PR keeps the prototype assignment (Object.setPrototypeOf(x, Action.prototype)) and drops the second clone and the lodash chain. neDB's deepCopy passes Date values through by reference, so a small in-place walk (datesToJSON) turns them into their toJSON() strings, exactly as the JSON round trip did. Results stay detached from neDB's cache and keep the same shape as before. getPush, the Mongo adapter and toClass itself are untouched.

Original code: the toClass(x, Action.prototype) mapping in the file adapter's getPushes dates to bcb3d06 ("material UI added", @grovesy, Oct 2020).

Before / after (local benchmark, 3 alternating A/B rounds, one server at a time)

git-proxy does not report to Datadog, so this is a local benchmark: NODE_ENV=production, tsx index.ts, default proxy.config.json (fs sink), scratch .data/db/pushes.db seeded with 1,000 synthetic push records (~21 KB each, mostly the getDiff step), logged in as the default admin, concurrency 2, 5 warm-up requests per run. Median across the 3 rounds, measured on the final code (with datesToJSON):

Path Metric Before After Change
GET /api/v1/push?blocked=true&canceled=false&authorised=false&rejected=false (dashboard default, 8.2 MB) p95 333 ms 231 ms -31%
p99 393 ms 294 ms -25%
GET /api/v1/push (all, 1,000 pushes, 21.4 MB) p95 1013 ms 768 ms -24%
p99 1110 ms 1177 ms no clear change (noisy at n=60)

On the dashboard default, every fix round beat every baseline round at p50, p95 and p99. On all pushes, p50 and p95 improved every round, while p99 was noisy in both versions. An earlier run of the first commit (without datesToJSON) gave similar numbers: dashboard p95 402→243 ms, p99 507→297 ms. In-process profile of the baseline (1,000 docs): neDB find+sort ~20 ms, toClass map 115-190 ms, JSON.stringify 115-200 ms, ETag sha1 ~50 ms.

Raw A/B output (final code)
round=1 orig all={"p50":670.4,"p95":1013.3,"p99":1109.5} pending={"p50":216,"p95":334.6,"p99":392.5}
round=1 fix  all={"p50":562.6,"p95":902.2,"p99":1484.4} pending={"p50":134.7,"p95":256.7,"p99":299.6}
round=2 orig all={"p50":696.3,"p95":1043.6,"p99":1453.8} pending={"p50":219.8,"p95":323.9,"p99":394.6}
round=2 fix  all={"p50":484.7,"p95":767.7,"p99":1177} pending={"p50":134.4,"p95":231.3,"p99":293.5}
round=3 orig all={"p50":696.9,"p95":909.6,"p99":1081} pending={"p50":219.7,"p95":332.9,"p99":384.8}
round=3 fix  all={"p50":478.4,"p95":709.5,"p99":837.6} pending={"p50":140,"p95":211.5,"p99":258.8}

(all: n=60, pending: n=80, concurrency 2)

Repro

  1. npm ci, then seed a scratch DB: node seed.js 1000 .data/db/pushes.db (deterministic generator of push records with steps, commitData, attestation/rejection; .data is gitignored).
  2. NODE_ENV=production ALLOWED_ORIGINS='*' npx tsx index.ts
  3. Log in via POST /api/auth/login (admin/admin) and time GET /api/v1/push[?filters] with the session cookie; alternate base/fix servers, restoring the seed DB before each run.

Scripts (seed, bench, A/B driver, response dumper) are available on request; nothing shared was written, all data is a local scratch file.

Equivalence

  • Response bodies for 6 queries (all, dashboard default, authorised=true, rejected=true (includes neDB $$date values), canceled=true, repo= filter) are byte-identical before and after on the final code (sha256 match), with identical Content-Type and ETag.
  • In-process results keep the same shape as before: dates come back as ISO strings and are detached from neDB's cache, which addresses the Devin Review finding. A new test in test/db/file/pushes.test.ts covers this, and it fails on the first commit.

Cost

Nothing at runtime. Each request now does a cheap walk over every document in place of a full JSON stringify-and-parse. No new dependency and no change to the API contract.

Tests

  • npm test: 761 passed, 8 failed, 45 skipped. The same 8 fail on main in this environment (ConfigLoader remote config fetch needs network; plugin.test.ts package plugins need built fixtures).
  • CI build-ubuntu (all 6) and build-windows fail only in the 2 ConfigLoader > loadRemoteConfig tests: they fetch the live upstream finos/git-proxy proxy.config.json, whose array-valued sink no longer matches this fork's schema. build result fails because it depends on those jobs. The same 2 tests fail locally on main; main has no CI runs to compare against.
  • eslint and prettier --check pass on the changed files; npm run check-types shows the same 2 pre-existing errors in AttestationInfo.tsx as main.

PR opened by Devin as part of the latency work for Taylor Curran. If this PR shows as opened by Devin rather than Taylor, the known cause is that GitHub account taylorcurranpython has no write access to COG-GTM.

Related Issue

None.

Checklist

General

Tests

  • Tests have been added/updated (date-detachment test for getPushes)
  • Unit tests pass (npm test) except the 8 that also fail on main
  • Linting and formatting pass
  • Type checks pass (2 pre-existing errors on main)

Devin-Org: engineering


Latency Optimizer opened this PR for Taylor
Implemented in session Find and fix next slow path in new repo
Review with Devin · Open in Devin Desktop



Note

Devin errored when opening this Pull Request as taylorcurranpython.
As a fallback, Devin opened this PR as itself.

Co-Authored-By: taylor.curran <taylorcurranagtc@gmail.com>
@devin-ai-integration

Copy link
Copy Markdown
Author

I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".

  • Disable automatic comment, CI, and merge conflict monitoring

@github-actions

Copy link
Copy Markdown

Dependency Review

✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.

Scanned Files

None

devin-ai-integration[bot]

This comment was marked as resolved.

@devin-ai-integration

Copy link
Copy Markdown
Author
Browser A/B: same dashboard behavior; median Pending request 16% shorter

Tested in Chrome on the local production build, using the same 1,000-push scratch neDB for both versions. Pending, Approved, Canceled and Rejected showed the same sampled rows and page totals on both. Next and exact SHA search also behaved the same. Over five uncached reloads per version, the median first complete GET /api/v1/push?blocked=true response took 1.95 s → 1.64 s. That is request duration, not full-page render time.

Baseline PR fix
Baseline: Chrome Network timings Fix: Chrome Network timings
Baseline: Pending SHA search returns one row Fix: Pending SHA search returns one row
Approved and rejected push details

Rejected details showed the same reason and exact date on both versions. The approved test record failed to open on both baseline and fix because the seed used attestation.questions where the UI expects attestation.answers. After that field was corrected in a temporary scratch copy, both versions showed the same approved details and checked attestation.

Baseline PR fix
Baseline: rejected push reason and date Fix: rejected push reason and date
Baseline: approved attestation with corrected fixture Fix: approved attestation with corrected fixture

CI note: build-ubuntu (24.x, 8.0) fails in 2 ConfigLoader > loadRemoteConfig tests. They fetch the live upstream finos/git-proxy proxy.config.json, whose sink format no longer matches this fork's schema. Both tests fail the same way locally on main without this change.

Screen recording · Written by Devin

Co-Authored-By: taylor.curran <taylorcurranagtc@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants