Skip to content

feat(connectors): add OpenDAL sink - #4123

Open
George-Miao wants to merge 4 commits into
apache:masterfrom
George-Miao:master
Open

George-Miao wants to merge 4 commits into
apache:masterfrom
George-Miao:master

Conversation

@George-Miao

@George-Miao George-Miao commented Sep 10, 2026

Copy link
Copy Markdown

Which issue does this PR address?

Closes #4071

Rationale

OpenDAL provides an uniformed interface for writing to various services, which is useful for iggy to support multiple new sink types with a very small amount of change & maintainance effort.

What changed?

Added opendal-sink crate that implements a sink adapter for OpenDAL and integration test. I intentionally didn't touch the other part, e.g., CI. Those can be done in another follow-up PR.

Local Execution

  • Passed
  • Pre-commit hooks ran

AI Usage

Uses codex & omp for writing tests, reviewing the changes & write some part of the doc.

@github-actions

Copy link
Copy Markdown

Thanks for the PR. It is labeled S-waiting-on-review and queued for review.

Slash commands (own line, regular comment) move it around the queue:

  • /ready - back to S-waiting-on-review after addressing feedback
  • /author - flip to S-waiting-on-author while you finish changes
  • /request-review @user-or-team - request a reviewer
  • /pin - exempt the PR from the stale bot, /unpin to undo

See CONTRIBUTING.md for details.

@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Sep 10, 2026
@codecov

codecov Bot commented Sep 10, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 87.25341% with 84 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.39%. Comparing base (63362dc) to head (5afbe36).

Files with missing lines Patch % Lines
...ore/connectors/sinks/opendal_sink/src/formatter.rs 55.38% 57 Missing and 1 partial ⚠️
core/connectors/sinks/opendal_sink/src/lib.rs 94.65% 17 Missing and 7 partials ⚠️
core/connectors/sinks/opendal_sink/src/path.rs 97.50% 1 Missing and 1 partial ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #4123      +/-   ##
============================================
- Coverage     87.48%   87.39%   -0.10%     
- Complexity     1575     1578       +3     
============================================
  Files          1280     1283       +3     
  Lines        223192   223851     +659     
  Branches     186555   187214     +659     
============================================
+ Hits         195267   195633     +366     
- Misses        23229    23479     +250     
- Partials       4696     4739      +43     
Components Coverage Δ
Rust Core 88.56% <87.25%> (-0.01%) ⬇️
Java SDK 68.71% <ø> (+0.03%) ⬆️
C# SDK 77.42% <ø> (-0.07%) ⬇️
Python SDK 90.97% <ø> (ø)
PHP SDK 85.67% <ø> (ø)
Node SDK 94.74% <ø> (-1.69%) ⬇️
Go SDK 70.11% <ø> (+0.05%) ⬆️
Files with missing lines Coverage Δ
core/connectors/sinks/opendal_sink/src/path.rs 97.50% <97.50%> (ø)
core/connectors/sinks/opendal_sink/src/lib.rs 94.65% <94.65%> (ø)
...ore/connectors/sinks/opendal_sink/src/formatter.rs 55.38% <55.38%> (ø)

... and 47 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

Comment thread core/connectors/sinks/opendal_sink/src/lib.rs Outdated

@mattp5657 mattp5657 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 things:

  1. This connector is missing from the release/version-bump tooling. Absent from scripts/bump-version.sh, the connector_plugins default in _build_rust_artifacts.yml, and the connector list in edge-release.yml. It'll pass CI but won't ship in edge or tagged release artifacts, and a future version bump will silently skip it. Not unique to this PR (a handful of existing sinks are missing from these same lists too), but worth fixing here or in an immediate follow-up rather than adding a fourth thing to that pile.

  2. I think we could use more unit and integration test coverage, happy and unhappy paths for every function. Untested today: path.rs's render_template and sanitize_path_segment beyond one edge case, lib.rs's consume()/write_message() unhappy path, the retry policy past one attempt, and S3/GCS/Azure end-to-end (fs is the only backend any test exercises)

Comment thread core/connectors/sinks/opendal_sink/src/lib.rs
Comment thread core/connectors/sinks/opendal_sink/src/lib.rs Outdated
Comment thread core/connectors/sinks/opendal_sink/src/path.rs Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-review PR is waiting on a reviewer

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement OpenDAL connector

3 participants