Repository navigation
perf: skip redundant deep clone when listing pushes from file DB - #52
Open
devin-ai-integration[bot] wants to merge 2 commits into
Open
devin-ai-integration[bot] wants to merge 2 commits into
devin-ai-integration[bot] wants to merge 2 commits into
Conversation
Co-Authored-By: taylor.curran <taylorcurranagtc@gmail.com>
Author
|
I'll fix CI failures and address comments from users with write access. I'll skip comments containing "(aside)".
|
Dependency Review✅ No vulnerabilities or license issues or OpenSSF Scorecard issues found.Scanned FilesNone |
Author
Co-Authored-By: taylor.curran <taylorcurranagtc@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
GET /api/v1/push(the dashboard's push list) on the default file-backed (neDB) database spends ~40% of its server time re-cloning documents that neDB has already deep-copied.db.find()in@seald-io/nedb4.x returnsmodel.deepCopyof every match (datastore.js:new Cursor(this, query, docs => docs.map(doc => model.deepCopy(doc)))).getPushesinsrc/db/file/pushes.tsthen rantoClasson each one, which does a fullJSON.parse(JSON.stringify(doc))before setting the prototype. This PR keeps the prototype assignment (Object.setPrototypeOf(x, Action.prototype)) and drops the second clone and the lodash chain. neDB'sdeepCopypassesDatevalues through by reference, so a small in-place walk (datesToJSON) turns them into theirtoJSON()strings, exactly as the JSON round trip did. Results stay detached from neDB's cache and keep the same shape as before.getPush, the Mongo adapter andtoClassitself are untouched.Original code: the
toClass(x, Action.prototype)mapping in the file adapter'sgetPushesdates to bcb3d06 ("material UI added", @grovesy, Oct 2020).Before / after (local benchmark, 3 alternating A/B rounds, one server at a time)
git-proxy does not report to Datadog, so this is a local benchmark:
NODE_ENV=production,tsx index.ts, defaultproxy.config.json(fs sink), scratch.data/db/pushes.dbseeded with 1,000 synthetic push records (~21 KB each, mostly thegetDiffstep), logged in as the defaultadmin, concurrency 2, 5 warm-up requests per run. Median across the 3 rounds, measured on the final code (withdatesToJSON):GET /api/v1/push?blocked=true&canceled=false&authorised=false&rejected=false(dashboard default, 8.2 MB)GET /api/v1/push(all, 1,000 pushes, 21.4 MB)On the dashboard default, every fix round beat every baseline round at p50, p95 and p99. On all pushes, p50 and p95 improved every round, while p99 was noisy in both versions. An earlier run of the first commit (without
datesToJSON) gave similar numbers: dashboard p95 402→243 ms, p99 507→297 ms. In-process profile of the baseline (1,000 docs): neDB find+sort ~20 ms,toClassmap 115-190 ms,JSON.stringify115-200 ms, ETag sha1 ~50 ms.Raw A/B output (final code)
(all: n=60, pending: n=80, concurrency 2)
Repro
npm ci, then seed a scratch DB:node seed.js 1000 .data/db/pushes.db(deterministic generator of push records with steps, commitData, attestation/rejection;.datais gitignored).NODE_ENV=production ALLOWED_ORIGINS='*' npx tsx index.tsPOST /api/auth/login(admin/admin) and timeGET /api/v1/push[?filters]with the session cookie; alternate base/fix servers, restoring the seed DB before each run.Scripts (seed, bench, A/B driver, response dumper) are available on request; nothing shared was written, all data is a local scratch file.
Equivalence
authorised=true,rejected=true(includes neDB$$datevalues),canceled=true,repo=filter) are byte-identical before and after on the final code (sha256 match), with identicalContent-Typeand ETag.test/db/file/pushes.test.tscovers this, and it fails on the first commit.Cost
Nothing at runtime. Each request now does a cheap walk over every document in place of a full JSON stringify-and-parse. No new dependency and no change to the API contract.
Tests
npm test: 761 passed, 8 failed, 45 skipped. The same 8 fail onmainin this environment (ConfigLoaderremote config fetch needs network;plugin.test.tspackage plugins need built fixtures).build-ubuntu(all 6) andbuild-windowsfail only in the 2ConfigLoader > loadRemoteConfigtests: they fetch the live upstreamfinos/git-proxyproxy.config.json, whose array-valuedsinkno longer matches this fork's schema.build resultfails because it depends on those jobs. The same 2 tests fail locally onmain;mainhas no CI runs to compare against.eslintandprettier --checkpass on the changed files;npm run check-typesshows the same 2 pre-existing errors inAttestationInfo.tsxasmain.PR opened by Devin as part of the latency work for Taylor Curran. If this PR shows as opened by Devin rather than Taylor, the known cause is that GitHub account taylorcurranpython has no write access to COG-GTM.
Related Issue
None.
Checklist
General
Tests
getPushes)npm test) except the 8 that also fail onmainmain)Devin-Org: engineering
Implemented in session Find and fix next slow path in new repo
Review with Devin · Open in Devin Desktop
Note
Devin errored when opening this Pull Request as taylorcurranpython.
As a fallback, Devin opened this PR as itself.