feat(connectors): Delta sink: production feasibility improvements for S3 storage

[mmmmxa]

Which issue does this PR address?

Closes #3839
Relates to #4105

Rationale

The current Delta connector implementation does not allow to properly set up a sink and does not provide user with helpful guidance on how to do so.

What changed?

Before:

• S3 storage type mandates access key and secret key, while this practice is discouraged by AWS.
• The existence of the destination delta table is not checked and docs don't mention that.
• aws_s3_endpoint_url may conflict with a region setting and its use is not necessary unless it's used with minio locally or with non AWS S3 compatible storage.
• Documentation does not help the user to set up the sink.
• The comment about the lock does not make sense for the uses of this connector, see connectors: Delta Lake Sink - allow concurrent writes across partitions #3839 for more info.

After:

• Stopped mandating the S3 access and secret keys. Mentioned the two access options in the docs. Suggested the sufficient policy configuration to be given to the instance.
• Checking the table existence after opening, wrote a test for it.
• Documented the usage of aws_s3_endpoint_url configuration setting.
• Added detailed information on how to set up the connector properly.
• Comment on the lock was deleted as an end to the discussion of the problem that was initially brought up.

Local Execution

• Passed
• Pre-commit hooks ran

AI Usage

1. Which tools? - Claude Code
2. Scope of usage? - generated some of the tests, Q&A, review of the branch
3. How did you verify the generated code works correctly? - performed a thorough review and ran the tests
4. Can you explain every line of the code if asked? - yes

[codecov]

Codecov Report

❌ Patch coverage is 78.72340% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 54.68%. Comparing base (362bd78) to head (6abff68).

Files with missing lines	Patch %	Lines
core/connectors/sinks/delta_sink/src/sink.rs	56.52%	10 Missing ⚠️

Additional details and impacted files

@@              Coverage Diff              @@
##             master    #4219       +/-   ##
=============================================
- Coverage     87.48%   54.68%   -32.80%     
  Complexity     1575     1575               
=============================================
  Files          1280     1277        -3     
  Lines        223241   180528    -42713     
  Branches     186604   143891    -42713     
=============================================
- Hits         195293    98722    -96571     
- Misses        23237    77240    +54003     
+ Partials       4711     4566      -145     

Components	Coverage Δ
Rust Core	46.99% <78.72%> (-41.56%)	⬇️
Java SDK	68.68% <ø> (ø)
C# SDK	77.46% <ø> (ø)
Python SDK	90.97% <ø> (ø)
PHP SDK	85.67% <ø> (ø)
Node SDK	96.44% <ø> (ø)
Go SDK	70.17% <ø> (ø)

Files with missing lines	Coverage Δ
core/connectors/sinks/delta_sink/src/storage.rs	99.54% <100.00%> (-0.46%)	⬇️
core/connectors/sinks/delta_sink/src/sink.rs	54.54% <56.52%> (+31.46%)	⬆️

... and 480 files with indirect coverage changes

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[mmmmxa]

/ready