Skip to content

Added support for rfc9394 (IMAP PARTIAL Extension for Paged SEARCH and FETCH) - #2012

Open
PretorianX wants to merge 1 commit into
jstedfast:masterfrom
PretorianX:imap-partial
Open

Added support for rfc9394 (IMAP PARTIAL Extension for Paged SEARCH and FETCH)#2012
PretorianX wants to merge 1 commit into
jstedfast:masterfrom
PretorianX:imap-partial

Conversation

@PretorianX

@PretorianX PretorianX commented Aug 11, 2026

Copy link
Copy Markdown

Implements the IMAP PARTIAL extension (RFC 9394) for paged SEARCH and FETCH, plus the closely-related PARTIAL sort return option from RFC 5267.

Closes #1687

API additions

  • PartialRange — a new struct representing an RFC 9394 partial-range (1:500 = oldest 500 results, -1:-100 = newest 100; both bounds non-zero, same sign, magnitudes ≤ 4294967295, sent verbatim without normalization).
  • IMailFolder/MailFolder/ImapFolder:
    • Search (SearchOptions options, SearchQuery query, PartialRange partial, CancellationToken) (+ async) → UID SEARCH RETURN (... PARTIAL first:last) <query>
    • Sort (SearchOptions options, SearchQuery query, IList<OrderBy> orderBy, PartialRange partial, CancellationToken) (+ async) → UID SORT RETURN (... PARTIAL first:last) (<order>) <charset> <query>
    • The MailFolder members are virtual (validate, then throw NotSupportedException) to avoid breaking third-party subclasses.
  • SearchResults.Partial (PartialRange?) — the range echoed back in the ESEARCH ... PARTIAL (<range> <uid-set|NIL>) return data. NILPartial is set and UniqueIds is empty.
  • IFetchRequest.Partial / FetchRequest.Partial (PartialRange?) — emits the RFC 9394 PARTIAL fetch modifier for UID FETCHes, combinable with CHANGEDSINCE/VANISHED in a single modifier list, e.g. UID FETCH 1:* (UID FLAGS MODSEQ) (CHANGEDSINCE 98305 VANISHED PARTIAL -1:-30).

Parser fix

The ESEARCH response parsers now handle the PARTIAL return data item (including the NIL payload). Previously a server sending * ESEARCH (TAG "…") UID PARTIAL (…) — reachable today via the raw IImapFolder.Search(string)/Sort(string) APIs — hit the unexpected-token path and threw ImapProtocolException.

Semantics: a PARTIAL result populates UniqueIds but intentionally does not set SearchResults.Count (a page length is not the total match count — combine with SearchOptions.Count for totals) and does not feed the Min/Max fallback.

Capability gating (review welcome)

  • Paged SEARCH requires PARTIAL (RFC 9394) or CONTEXT=SEARCH (RFC 5267 defines the same return option).
  • Paged SORT requires PARTIAL or any CONTEXT=*. RFC 5267 strictly grants the PARTIAL sort return option under CONTEXT=SORT, but real-world servers (Dovecot) share the return-option parser between SEARCH and SORT and only advertise CONTEXT=SEARCH while accepting PARTIAL for both — gating on CONTEXT=SORT alone would make the API unusable against Dovecot.
  • Negative ranges are an RFC 9394 addition and therefore require the PARTIAL capability specifically (verified against Dovecot 2.3, which replies BAD: PARTIAL range broken to PARTIAL -1:-10).
  • Fetch with a Partial range requires the PARTIAL capability, is only valid for UID-based fetches (NotSupportedException otherwise), and throws NotSupportedException if the UID set is too large to fit in a single UID FETCH command (the modifier would otherwise be re-applied per sub-command with different semantics).

Testing

  • Unit tests for PartialRange (validation/parsing/formatting/equality) and replay tests covering: positive/negative ranges, PARTIAL + COUNT/MIN/MAX, NIL responses, shorter-than-requested results, sort-order preservation, CHANGEDSINCE/VANISHED combination, raw-search parser tolerance, capability gating, SearchOptions.All + partial rejection (mutually exclusive per RFC 9394 §3.1), and the multi-command split guard.
  • Verified end-to-end against a live Dovecot 2.3 server (advertises ESEARCH ESORT CONTEXT=SEARCH, no PARTIAL) with a large mailbox: paged search/sort round-trips, echoed ranges, NIL handling and the negative-range rejection all behave as described, with the paged sort returning a small fraction of the bytes and time of RETURN (ALL).

Notes

  • rfc/rfc9394.txt was already vendored and RFCs.md already listed RFC 9394; the // TODO: rfc9208 and rfc9394 header comment in ImapCapabilities.cs was trimmed accordingly.
  • IMailFolder/IFetchRequest gain members, which is source-breaking for external implementers of those interfaces.

…d FETCH)

Adds a PartialRange struct and support for paging:
* SEARCH: Search (SearchOptions, SearchQuery, PartialRange) emits
  UID SEARCH RETURN (... PARTIAL first:last) and SearchResults.Partial
  carries the range echoed back by the ESEARCH response.
* FETCH: FetchRequest.Partial emits the PARTIAL fetch modifier for
  UID FETCH, combinable with CHANGEDSINCE in a single modifier list.
* SORT: Sort (SearchOptions, SearchQuery, IList<OrderBy>, PartialRange)
  emits UID SORT RETURN (... PARTIAL first:last) (rfc5267).

The ESEARCH response parser now also handles the PARTIAL return data
item (previously an unexpected-token ImapProtocolException, reachable
via the raw IImapFolder.Search/Sort APIs).

Capability gating: searches accept PARTIAL or CONTEXT=SEARCH; sorts
accept PARTIAL or CONTEXT=* (rfc5267 strictly wants CONTEXT=SORT, but
servers such as Dovecot share the return-option parser between SEARCH
and SORT and only advertise CONTEXT=SEARCH); negative ranges (an
rfc9394 addition) require the PARTIAL capability.

Fixes jstedfast#1687
@jstedfast

Copy link
Copy Markdown
Owner

I'll try to review this soon, I just haven't had a long enough stretch of time yet.

One of the reasons I hadn't implemented this yet is that I hadn't seen a server support it. Has that changed? It's ok if it hasn't, I'm just curious.

@jstedfast

Copy link
Copy Markdown
Owner

FWIW, just skimmed over it on my phone and it looks like it's got all the parts I would expect (all the right overloads, unit tests, doc comments, etc).

I feel pretty good about merging it, but will try to dive deeper in the next few days (I will for sure do it before I make any new releases).

@PretorianX

Copy link
Copy Markdown
Author

I'll try to review this soon, I just haven't had a long enough stretch of time yet.

One of the reasons I hadn't implemented this yet is that I hadn't seen a server support it. Has that changed? It's ok if it hasn't, I'm just curious.

This is supported by Dovecot CE 2.3+ Dovecot RFC 5267.

RFC 9394's negative form (-1:-50, newest-first) is therefore rejected with PARTIAL range broken.

This PR supporting negative ranges is a superset, forward-compatible when (if?) Dovecot adds them so both RFCs are supported.

On my 54,809-message mailbox the SORT response drops from 380,682 bytes to 465 bytes.
In production across 646k requests this cut IMAP timeouts on the folder-listing endpoint by
~28% (load-matched windows), with the >10 s sort tail down 35%.

@jstedfast

Copy link
Copy Markdown
Owner

Sounds good. I can definitely see this improving the ability to get faster search results for clients that build on this feature.

Are you writing an actual mail client or is this more of a utility app?

@PretorianX

Copy link
Copy Markdown
Author

Sounds good. I can definitely see this improving the ability to get faster search results for clients that build on this feature.

Are you writing an actual mail client or is this more of a utility app?

this is webmail app for internal company users (3k+ large mailboxes) based on dovecot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implement the IMAP PARTIAL Extension for Paged SEARCH and FETCH

2 participants