Repository navigation
fix(projectgraph): bound the link step on glob imports and stop it at the deadline - #61
Conversation
The lookup of a name in a unit reads the wildcard imports of its scope. The resolution of each import can walk a unit whose lookup reads the same imports again. src/sys.rs of time 0.1.45 has eight glob uses, and the resolving guard of resolveRust stops only the keys on the stack. So the work grew with the orders of the eight imports. The glob candidates of each Rust path made each step wider, and the method measure on the lettre_email project ran for more than an hour. wildcardModules now keeps the modules of the imports of each scope. A read of a scope that a resolution reads now gets the modules of the last whole pass, and the passes repeat until the modules stay the same. The table keeps no modules that read a scope further up the chain. The lettre_email project now links in less than one second. The case rust/glob-use-many copies the shape of src/sys.rs and does not end without the change. With projectgraph.Version at 18 and the new case out, the graph and result hashes of the cases are those of 16/18 and 16/18/5. The bump to 19 adds the rows for the new case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PFQbqPqd9xn1fT6hBsyoUm Signed-off-by: Claude <noreply@anthropic.com>
The table of the modules of the wildcard imports of each scope kept a list that a resolution gave under the resolving guard of a Rust path. app/src/prelude.rs has pub use core_lib::*. The link step resolved the path core_lib of that glob use, walked the unit of the file, and read its glob use again. The guard gave no module there, and the table kept the empty list for the whole build. A call of crate::prelude::helper then went to the external key crate//prelude//helper with no record, and a target under helper read not-found. wildcardModules now keeps the modules only while a resolution reads the scope. A read of that scope gets the modules of the last whole pass, and the passes repeat until the modules stay the same. The single file with many glob uses stays fast, as each nested read of the scope gets the modules of the pass at once. A fuzz of 1500 generated crates gives the same graph as the code before the table. The table gave 24 graphs that differ. The case rust/glob-use-workspace-crate fails with the table. With projectgraph.Version at 18 and the two new cases out, the graph and result hashes of the cases are those of 16/18 and 16/18/5. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PFQbqPqd9xn1fT6hBsyoUm Signed-off-by: Claude <noreply@anthropic.com>
…ends Build checked its context only between files. A cycle of glob uses across Rust files makes the resolutions of one file grow with the orders of the files, so a build with a deadline ran on for minutes or hours past it. A scan gives the link step a budget, and the budget did not stop it. Build now sets a stop flag when its context ends. Each step of a walk and each read of the wildcard imports of a scope end at once on the flag. Build checks the context again before it returns a graph, so the graph of a stopped build goes to no caller. TestBuildStopsAtTheDeadline links ten files that glob use one another with a deadline of 200 ms. The build ends in about 0.2 s. Without the flag the test ran for more than two minutes. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PFQbqPqd9xn1fT6hBsyoUm Signed-off-by: Claude <noreply@anthropic.com>
…d cut wildcardModules resolved the wildcard imports of a scope again on each read. src/sys.rs of time 0.1.45 has eight glob uses, and the link step of the lettre_email project read them 12,000 times in 1.6 s. A table now keeps the modules of a scope. The earlier table kept a list that the resolving guard of a Rust path had cut, and a target read not-found. So one check, uncut, decides both the store and the hit. A read records each path that it asks the resolving guard for. uncut holds when no such path is in progress, and no other cycle guard, MRO lookup or read of wildcard imports is in progress. At the end of a read the paths in progress are those from before the read, so a recorded path among them means that the guard cut the read. At a hit the same check means that a new read would get the same answers from the guard. Any other read resolves the imports as before and stores nothing. uncut compares only the paths of the resolving guard. The reads under any other guard cost less than 3% of the read time on the Rust corpus. The table starts after the alias step, because a read before it can miss a class. The graphs do not change. projectgraph.Version stays 19, and the hash rows of the cases stay as they are. - Fuzz, table against no table in one process: 3000 Rust, 3000 Java and 3000 Go generated projects, 0 diffs. 39 Rust seeds ran out of time and do not count. - Fuzz, this commit against its parent by graph hash: 2965 more Rust crates and workspaces, 0 diffs. 35 more seeds ran out of time. - A build that resolves again at each hit and compares the lists: 4.79 million hits in three builds of the 30 corpus projects, 0 mismatches. - Both method reports are byte-identical to those of the parent. - A table without the path check gives 36 diffs in 300 seeds and fails the case rust/glob-use-workspace-crate. Link time of seven Rust projects, best of three, in ms: project no table cut table this table postgres 822 627 763 native-tls 576 340 367 cookie 184 106 120 imageproc 645 423 496 tokio-postgres 773 633 709 lettre_email 1644 73 194 rusqlite_migration 124 113 144 sum 4768 2315 2793 Known limit: the tables of the build grow during the link step. A read at a later time can see more of them than the stored read did. The checks above found no such case. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PFQbqPqd9xn1fT6hBsyoUm Signed-off-by: Claude <noreply@anthropic.com>
…settle A read of the wildcard imports of a scope that a resolution reads now gets the modules of the last pass. A memo that the pass writes from that list can lack a module for the whole build. In the case rust/glob-use-enum-variants, the MRO of a type misses its trait, and the call k.area() gets a no-definition record and no edge. main resolves every import again on each read, so it finds the edge. Each memo write made while a read is in progress now logs an undo step. A pass whose modules change undoes its writes, and the next pass writes them again. A write of a settled pass stays, unless an outer pass undoes it. The log covers the names, MRO, external base and alias binding memos. The routes of Rust crates read no glob use, so they need no step. A memo that skips each write after a pass read gives the same graphs, but the Rust link time of seven corpus projects goes from 2.4 s to 6.7 s. With the undo log it is 2.3 s. Build also checks the context after finish. finish resolves the external keys of the units, and a stop inside it changed the External of a node with a nil error. The two new cases fail on the parent commit and pass on main. Without them the hash rows of version 19 stay as they were, so no other case changes. The rows of version 19 change for the new cases. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PFQbqPqd9xn1fT6hBsyoUm Signed-off-by: Claude <noreply@anthropic.com>
SafeDep Report SummaryNo dependency changes detected. Nothing to scan. This report is generated by SafeDep GitHub App |
Codex Review Summary: ✅ CompletedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Important Review skippedToo many files! This PR contains 125 files, which is 25 over the limit of 100. To get a review, reduce the PR to 100 files or fewer by splitting it into smaller PRs or changing its base branch. Upgrade to a paid plan to raise the limit. This review couldn't start because sufficient usage credits or metered capacity aren't available. Add credits or update usage-based reviews in the billing tab, then retry.
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #61 +/- ##
==========================================
- Coverage 76.61% 76.60% -0.01%
==========================================
Files 186 186
Lines 19449 19512 +63
==========================================
+ Hits 14900 14947 +47
- Misses 3657 3667 +10
- Partials 892 898 +6 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|



Summary
The link step on main takes 42 s on the Rust project lettre_email.
src/sys.rsof time 0.1.45 has eight glob uses. Each lookup of a name that the file does not define reads those glob uses, and each glob use resolves through the same file again. The work grows with the orders of the imports. A cycle of glob uses across files can keep the link step running for hours, and the context check between files does not stop it.This PR bounds that work, keeps the graphs sound, and stops the build soon after its deadline.
Changes
wildcardModulesresolves the wildcard imports of a scope in passes. A read of a scope that a resolution reads now gets the modules of the last pass. The passes repeat until the modules stay the same. 6b14f59 kept a table of the modules for the build. That table could keep a list that the resolving guard of a Rust path had cut, and a target read not-found. 7e6b8cc removes the table. The caserust/glob-use-workspace-crateguards this.context.AfterFuncsets astoppedflag. The walk and the glob reads end at once when it is set.Buildchecks the context afterfinish, so a stopped build returns the context error and never a part of a graph.TestBuildStopsAtTheDeadlinebuilds a cycle of glob uses across ten files. It ends in 0.2 s withcontext.DeadlineExceeded. Without the checks it runs for more than 120 s.uncutholds. A read records each Rust path that it asks the resolving guard for.uncutholds when no such path is in progress and no other cycle guard, MRO lookup or glob read is in progress. The same check decides the store and the hit.rust/glob-use-enum-variantsandjava/static-wildcard-nested-classfail without this commit and pass on main.projectgraph.Versiongoes to 19, because the edges of some graphs change. The golden files change only in the version field.Soundness checks
Timing
Link time on lettre_email goes from 42 s on main to 0.15 s.
Link time of six more Rust corpus projects, in ms, best of three. The two columns come from separate runs on a noisy machine.
vet needs no change. At the default
--reach importlevel only the indexer runs.projectgraph.Buildruns only at--reach function, and vet already mapscontext.DeadlineExceededfrom it to a budget finding.Known limits
uncut, or the table becomes unsound for it.b.memo.Test plan
go test ./.... modgraphTestBuildHoldsEachEdgeOfGoListfails on main too.golangci-lint run ./projectgraph/... ./test/projectgraph/...: 0 issues../langcode/ ./lang/grammar/ ./fileindex/... ./projectgraph/... ./internal/....go test -tags projectgraph -run 'TestPublicSets|TestGovulncheckTraces|TestVulnAgreement' ./test/projectgraph/.🤖 Generated with Claude Code
https://claude.ai/code/session_01PFQbqPqd9xn1fT6hBsyoUm