Added support for rfc9394 (IMAP PARTIAL Extension for Paged SEARCH and FETCH) - #2012
Added support for rfc9394 (IMAP PARTIAL Extension for Paged SEARCH and FETCH)#2012PretorianX wants to merge 1 commit into
Conversation
…d FETCH) Adds a PartialRange struct and support for paging: * SEARCH: Search (SearchOptions, SearchQuery, PartialRange) emits UID SEARCH RETURN (... PARTIAL first:last) and SearchResults.Partial carries the range echoed back by the ESEARCH response. * FETCH: FetchRequest.Partial emits the PARTIAL fetch modifier for UID FETCH, combinable with CHANGEDSINCE in a single modifier list. * SORT: Sort (SearchOptions, SearchQuery, IList<OrderBy>, PartialRange) emits UID SORT RETURN (... PARTIAL first:last) (rfc5267). The ESEARCH response parser now also handles the PARTIAL return data item (previously an unexpected-token ImapProtocolException, reachable via the raw IImapFolder.Search/Sort APIs). Capability gating: searches accept PARTIAL or CONTEXT=SEARCH; sorts accept PARTIAL or CONTEXT=* (rfc5267 strictly wants CONTEXT=SORT, but servers such as Dovecot share the return-option parser between SEARCH and SORT and only advertise CONTEXT=SEARCH); negative ranges (an rfc9394 addition) require the PARTIAL capability. Fixes jstedfast#1687
|
I'll try to review this soon, I just haven't had a long enough stretch of time yet. One of the reasons I hadn't implemented this yet is that I hadn't seen a server support it. Has that changed? It's ok if it hasn't, I'm just curious. |
|
FWIW, just skimmed over it on my phone and it looks like it's got all the parts I would expect (all the right overloads, unit tests, doc comments, etc). I feel pretty good about merging it, but will try to dive deeper in the next few days (I will for sure do it before I make any new releases). |
This is supported by Dovecot CE 2.3+ Dovecot RFC 5267. RFC 9394's negative form ( This PR supporting negative ranges is a superset, forward-compatible when (if?) Dovecot adds them so both RFCs are supported. On my 54,809-message mailbox the SORT response drops from 380,682 bytes to 465 bytes. |
|
Sounds good. I can definitely see this improving the ability to get faster search results for clients that build on this feature. Are you writing an actual mail client or is this more of a utility app? |
this is webmail app for internal company users (3k+ large mailboxes) based on dovecot. |
Implements the IMAP PARTIAL extension (RFC 9394) for paged SEARCH and FETCH, plus the closely-related PARTIAL sort return option from RFC 5267.
Closes #1687
API additions
PartialRange— a new struct representing an RFC 9394partial-range(1:500= oldest 500 results,-1:-100= newest 100; both bounds non-zero, same sign, magnitudes ≤ 4294967295, sent verbatim without normalization).IMailFolder/MailFolder/ImapFolder:Search (SearchOptions options, SearchQuery query, PartialRange partial, CancellationToken)(+ async) →UID SEARCH RETURN (... PARTIAL first:last) <query>Sort (SearchOptions options, SearchQuery query, IList<OrderBy> orderBy, PartialRange partial, CancellationToken)(+ async) →UID SORT RETURN (... PARTIAL first:last) (<order>) <charset> <query>MailFoldermembers arevirtual(validate, then throwNotSupportedException) to avoid breaking third-party subclasses.SearchResults.Partial(PartialRange?) — the range echoed back in theESEARCH ... PARTIAL (<range> <uid-set|NIL>)return data.NIL⇒Partialis set andUniqueIdsis empty.IFetchRequest.Partial/FetchRequest.Partial(PartialRange?) — emits the RFC 9394PARTIALfetch modifier for UID FETCHes, combinable withCHANGEDSINCE/VANISHEDin a single modifier list, e.g.UID FETCH 1:* (UID FLAGS MODSEQ) (CHANGEDSINCE 98305 VANISHED PARTIAL -1:-30).Parser fix
The ESEARCH response parsers now handle the
PARTIALreturn data item (including theNILpayload). Previously a server sending* ESEARCH (TAG "…") UID PARTIAL (…)— reachable today via the rawIImapFolder.Search(string)/Sort(string)APIs — hit the unexpected-token path and threwImapProtocolException.Semantics: a
PARTIALresult populatesUniqueIdsbut intentionally does not setSearchResults.Count(a page length is not the total match count — combine withSearchOptions.Countfor totals) and does not feed theMin/Maxfallback.Capability gating (review welcome)
PARTIAL(RFC 9394) orCONTEXT=SEARCH(RFC 5267 defines the same return option).PARTIALor anyCONTEXT=*. RFC 5267 strictly grants the PARTIAL sort return option underCONTEXT=SORT, but real-world servers (Dovecot) share the return-option parser between SEARCH and SORT and only advertiseCONTEXT=SEARCHwhile accepting PARTIAL for both — gating onCONTEXT=SORTalone would make the API unusable against Dovecot.PARTIALcapability specifically (verified against Dovecot 2.3, which repliesBAD: PARTIAL range brokentoPARTIAL -1:-10).Fetchwith aPartialrange requires thePARTIALcapability, is only valid for UID-based fetches (NotSupportedExceptionotherwise), and throwsNotSupportedExceptionif the UID set is too large to fit in a singleUID FETCHcommand (the modifier would otherwise be re-applied per sub-command with different semantics).Testing
PartialRange(validation/parsing/formatting/equality) and replay tests covering: positive/negative ranges,PARTIAL+COUNT/MIN/MAX,NILresponses, shorter-than-requested results, sort-order preservation,CHANGEDSINCE/VANISHEDcombination, raw-search parser tolerance, capability gating,SearchOptions.All+ partial rejection (mutually exclusive per RFC 9394 §3.1), and the multi-command split guard.ESEARCH ESORT CONTEXT=SEARCH, noPARTIAL) with a large mailbox: paged search/sort round-trips, echoed ranges, NIL handling and the negative-range rejection all behave as described, with the paged sort returning a small fraction of the bytes and time ofRETURN (ALL).Notes
rfc/rfc9394.txtwas already vendored andRFCs.mdalready listed RFC 9394; the// TODO: rfc9208 and rfc9394header comment inImapCapabilities.cswas trimmed accordingly.IMailFolder/IFetchRequestgain members, which is source-breaking for external implementers of those interfaces.