[AndrewBrown-USDA]

This PR introduces a caching system for Web Soil Survey ZIP archives in the downloadSSURGO() workflow, along with new utility functions for cache management. For #473

[Copilot]

Copilot review overview

🟡 Changes recommended

Critical download and cache-safety issues remain unresolved.

Review effort: Lite
Findings: 2 · 2 · 1

Open (5)

• Preserve cached archive until forced replacement download succeeds · New
• Handle partial failures in multi-area downloads · New
• Include fiscal year when retaining latest cache archives · New
• Resolve exdir only when extracting archives · New
• Document fiscal-year directory behavior for default exdir · New

What changed in this PR

Adds persistent fiscal-year-organized caching for Web Soil Survey ZIP downloads.

Changes:

• Adds cache management, listing, cleanup, and download utilities.
• Integrates caching with downloadSSURGO() and createSSURGO().
• Updates tests, documentation, exports, and release notes.

Unresolved issues include unsafe forced replacement, incomplete-download handling, cache deduplication, fallback behavior, and documentation corrections.

File	Description
vignettes/​local-ssurgo.Rmd	Documents persistent cache usage.
tests/​testthat/​test-downloadSSURGO-cache.R	Tests cache behavior and fallback scenarios.
R/​WSS_cache.R	Implements cache management and download utilities.
R/​createSSURGO.R	Integrates caching into download and database workflows.
NEWS.md	Records the new cache functionality.
NAMESPACE	Exports cache helper functions.
man/​list_WSS_cache.Rd	Documents cache inventory.
man/​downloadSSURGO.Rd	Documents updated download arguments.
man/​createSSURGO.Rd	Updates extraction argument documentation.
man/​clear_WSS_cache.Rd	Documents cache cleanup.

Files not reviewed (4)

• man/clear_WSS_cache.Rd: Generated file
• man/createSSURGO.Rd: Generated file
• man/downloadSSURGO.Rd: Generated file
• man/list_WSS_cache.Rd: Generated file

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[dylanbeaudette]

I'd like the other soilDB authors to weigh-in on the topic of accepting any genAI suggested changes.

[AndrewBrown-USDA]

I will note that having an agent review a PR is not the same as blindly accepting the proposed changes.

[Copilot]

Copilot review overview

🟡 Changes recommended

Unresolved cache reliability and extraction-isolation issues include a critical risk of mixing outputs across requests.

Review effort: Lite
Findings: 1

Open (1)

• Default extraction directory mixes outputs across calls · New

Resolved since last review (5)

• Handle partial failures in multi-area downloads
• Preserve cached archive until forced replacement download succeeds
• Resolve exdir only when extracting archives
• Include fiscal year when retaining latest cache archives
• Document fiscal-year directory behavior for default exdir

Files not reviewed (4)

• man/clear_WSS_cache.Rd: Generated file
• man/createSSURGO.Rd: Generated file
• man/downloadSSURGO.Rd: Generated file
• man/list_WSS_cache.Rd: Generated file

Previously missed (4)

In code that hasn't changed since last review

Failed initial downloads leave corrupt destination files

R/​WSS_cache.R:160

For a first download (force = FALSE and no existing destination), curl_download() writes directly to destfile. If the transfer fails after creating a partial file, the error path does not remove it because tmp_destfile equals destfile; the next call then returns that corrupt ZIP at line 153 without retrying. Always download to a temporary file and rename it into place only after success, as is already done for forced replacements.

Multiple fiscal years are matched incorrectly

R/​WSS_cache.R:219

fiscal_year is normalized element-wise, but == only works correctly for a single requested year; passing multiple years (for example list_WSS_cache(fiscal_year = c(2023, 2024))) recycles the shorter vector and can select the wrong rows. Use set membership for the normalized values so all requested fiscal years are honored.

Fallback cache selection does not support LIKE predicates

R/​createSSURGO.R:166

The documented WHERE form areasymbol LIKE 'KS%' cannot use this fallback: .extract_wss_areasymbols() returns KS%, but .wss_cache_select() treats areasymbols as exact values, so cached KS### entries are never selected when the remote metadata query fails. Please make the fallback evaluate supported WHERE predicates against the cache inventory (at least SQL LIKE), or limit this path to exact area-symbol inputs and report that limitation.

Document NULL exdir behavior in createSSURGO

R/​createSSURGO.R:308

Making exdir optional changes the public behavior, but the @param exdir documentation still describes only an explicitly supplied path and does not say that NULL selects the latest fiscal-year WSS cache directory (or errors when none exists). Update the roxygen text so the generated createSSURGO help documents this new default.

[Copilot]

Copilot review overview

🟡 Changes recommended

Critical API and area-filtering issues, plus unresolved cache fallback and replacement problems, block approval.

Review effort: Lite
Findings: 2 · 1

Open (3)

• Preserve positional API when adding areasymbols · New
• Match areasymbols in directories and filenames · New
• Preserve cached archive until replacement succeeds · New

Resolved since last review (1)

• Default extraction directory mixes outputs across calls

Files not reviewed (4)

• man/clear_WSS_cache.Rd: Generated file
• man/createSSURGO.Rd: Generated file
• man/downloadSSURGO.Rd: Generated file
• man/list_WSS_cache.Rd: Generated file

[AndrewBrown-USDA]

Test failures are unrelated to this PR (fetchSCAN() failures); those seem to be persisting so I will merge this shortly and focus on some improvements to SCAN testing in another PR

[StephenRoecker-USDA]

I'd like the other soilDB authors to weigh-in on the topic of accepting any genAI suggested changes.
@dylanbeaudette

I don't have a problem with genAI changes in general. I think all the same best practices apply. Targeted changes, presumed ownership of individual contributors functions, etc.. Where it gets sticky is functions that are reused internally by packages, such as uncode, that have to work. In such cases as usual it requires sufficient testing and eyeballs.

[Copilot]

🟡 Changes recommended

Fallback and filtering can select incorrect or incomplete surveys, and database writes can bypass the caller’s append setting.

1 open issue

• Restrict fallback parsing to supported positive area filters · New

4 previously missed issues

In code that hasn't changed since last review

Validate all requested areas are covered by fallback cache

R/​createSSURGO.R:168

When metadata lookup fails, requesting CA067 and CA077 succeeds with only CA067 if that is the only cached area. This bypasses the partial-download check and can produce an incomplete database. Check coverage of explicitly requested area symbols before accepting the fallback and report any missing symbols.

Select newest cached archive per area for latest_only

R/​createSSURGO.R:169

latest_only = TRUE keeps one ZIP per area and fiscal year. If CA067 has FY23 and FY24 archives, metadata lookup failure selects both: default extraction errors about multiple directories, while an explicit exdir extracts both versions into the same destination. Select the newest cached archive per area here, preserving the per-year behavior used by cache cleanup.

Match extraction paths relative to exdir

R/​createSSURGO.R:258

createSSURGO() passes full paths here, so every ancestor directory becomes an area identifier. With exdir = '/exports/CA067', filtering for CA067 also matches /exports/CA067/CA077/tabular/mapunit.txt through the common extraction root and imports an unrelated survey. Pass paths relative to exdir to the matcher, excluding the extraction root and its ancestors.

Prevent unintended appends after table initialization failure

R/​createSSURGO.R:712

i > 1 permits appending to tables that existed before this call even when append = FALSE. With a supplied connection, an existing table, and overwrite = FALSE, the first write fails inside try(), but later files are appended and committed anyway. Track successful table initialization within this invocation and auto-append only after it succeeds, or when append = TRUE is explicit.

3 resolved since last review

• Match areasymbols in directories and filenames
• Preserve positional API when adding areasymbols
• Preserve cached archive until replacement succeeds

4 files not reviewed

• man/clear_WSS_cache.Rd: Generated file
• man/createSSURGO.Rd: Generated file
• man/downloadSSURGO.Rd: Generated file
• man/list_WSS_cache.Rd: Generated file

_{Review effort: Balanced}