Related to #2099 / #2100
What's happening
When dataformCoreVersion is set in workflow_settings.yaml, compile() in cli/api/commands/compile.ts copies the entire project directory to a temp dir before installing core and compiling:
fs.copySync(resolvedProjectPath, temporaryProjectPath);
No filter is passed, so fs-extra's copySync copies everything in the project directory byte-for-byte — including directories that have nothing to do with the Dataform project itself (.venv/, .git/, unrelated build output, etc.), if they happen to live alongside definitions//includes/. This cost scales with total files/bytes under the project root, not with the size of the actual Dataform project, and it's paid on every single compile invocation.
Nothing else in the compile pipeline mitigates this — the .gitignore written by dataform init is only for git hygiene and isn't consulted here, and the later glob.sync("!(node_modules)/**/*.*", ...) call that enumerates source files happens after this copy has already paid the I/O cost.
Impact
On a repo where a Python .venv or similar directory sits next to the Dataform project (common in mixed-tooling repos, e.g. Dataform + a Python-based ELT/testing setup), this can turn a sub-second copy into many seconds, independent of the actual project's size — worsening the fixed overhead already reported in #2099.
Suggested fix
Pass a filter to copySync that skips, at minimum, node_modules (already excluded from the later glob) and .git, and ideally respects a .gitignore/.dataformignore in the project root if present — similar in spirit to how most file-watching/bundling tools (webpack, esbuild, etc.) avoid walking ignored directories by default. This wouldn't change the stateless-install guarantee (core is still freshly installed every time) — it just stops copying files that are never going to be require()'d anyway.
Happy to put up a PR for this if a maintainer can confirm the preferred approach (hardcoded excludes vs. .gitignore-aware).
Related to #2099 / #2100
What's happening
When
dataformCoreVersionis set inworkflow_settings.yaml,compile()incli/api/commands/compile.tscopies the entire project directory to a temp dir before installing core and compiling:No
filteris passed, sofs-extra'scopySynccopies everything in the project directory byte-for-byte — including directories that have nothing to do with the Dataform project itself (.venv/,.git/, unrelated build output, etc.), if they happen to live alongsidedefinitions//includes/. This cost scales with total files/bytes under the project root, not with the size of the actual Dataform project, and it's paid on every singlecompileinvocation.Nothing else in the compile pipeline mitigates this — the
.gitignorewritten bydataform initis only for git hygiene and isn't consulted here, and the laterglob.sync("!(node_modules)/**/*.*", ...)call that enumerates source files happens after this copy has already paid the I/O cost.Impact
On a repo where a Python
.venvor similar directory sits next to the Dataform project (common in mixed-tooling repos, e.g. Dataform + a Python-based ELT/testing setup), this can turn a sub-second copy into many seconds, independent of the actual project's size — worsening the fixed overhead already reported in #2099.Suggested fix
Pass a
filtertocopySyncthat skips, at minimum,node_modules(already excluded from the later glob) and.git, and ideally respects a.gitignore/.dataformignorein the project root if present — similar in spirit to how most file-watching/bundling tools (webpack, esbuild, etc.) avoid walking ignored directories by default. This wouldn't change the stateless-install guarantee (core is still freshly installed every time) — it just stops copying files that are never going to berequire()'d anyway.Happy to put up a PR for this if a maintainer can confirm the preferred approach (hardcoded excludes vs.
.gitignore-aware).