IPIP-0526: Historic Bitswap Provider Publishing API

[lidel]

This IPIP

• does not change any existing specifications
• documents the legacy PUT /routing/v1/providers/ endpoint that only accepts signed Bitswap provider records, originally introduced for index-provider/IPNI integration in 2022
• includes CSS styling for historic IPIP state

Why?

I spent multiple days over the years discussing this (e.g. ipni/index-provider#403), and explaining all the background and history always takes most of the time.

Now, there is a PR I can link to + this allows us to close all issues related to abandoned IPIP #378 and point here as a provisional spec reference if anyone needs to use or discuss this API for legacy reasons.

Should the be merged?

I'm fine either way.

The IPIP has a special state "historic" and clearly states is provided only for archival purposes.

• If we are not planning on standardizing this API, but there won't be anything better to replace it any time soon, we could merge it to have SOMETHING people can reference, but also be vary its provisional and limited in scope and support.

• If we are planning to create alternative any time soon (EOY 2026), it might be ok to keep it as a draft indefinitely, and close without merging when a modern replacement is defined by a different IPIP.

For now, I'm marking it as a draft, we can decide its fate in 2026.

cc @aschmahmann @gammazero

[github-actions]

🚀 Build Preview on IPFS ready

• 🔎 Commit: 83febb2
• 🔏 CID bafybeighz6y2czawqc4bz75tjl4bb6tmf2znblyrkr6lztudrpamzxt6hi
• 📦 Preview:

  • dweb.link

  • inbrowser.link

[Rinse12]

Adding a production use case, since the thread notes that nobody stepped up to make the HTTP provide API a real thing.

I maintain pkc-js, a pubsub-based social protocol built on kubo. Each community node publishes an IPNS record, subscribes to a pubsub topic for that community, and serves a large and constantly growing set of content blocks. Our readers are browsers and light nodes that resolve through delegated HTTP routers, so provider records in HTTP routers are what actually make our content retrievable. We have been running Routing.Type=custom with provide mapped to our own HTTP routers in production for a while. It works, with the caveat that maintainers correctly describe that path as experimental and untested: see ipfs/kubo#11089, where it silently regressed for two releases, precisely because there are no tests and no spec.

What we want to end up with is narrow: the DHT carries only the few keys tied to a community's IPNS and its pubsub swarm (the IPNS-over-pubsub rendezvous CID and the current root), and the HTTP routers carry everything else. Announcing every content block to the DHT is work that nothing in our system reads, and it scales with the whole pinset instead of with the handful of keys that need global reach. Kubo cannot express that split today and I have opened ipfs/kubo#11411 about it, but the config side is not really the blocker: the blocker is that HTTP providing has no spec, so it cannot become a supported path that such a feature could be built on.

Points we would like the IPIP to settle, from an implementer's perspective:

1. Batching and limits. How many CIDs per request, and how a server advertises its own limit. Kubo currently assumes 100 by default via MaxProvideBatchSize.
2. Record lifetime and refresh. Whether announcements expire, and if so how a client learns the TTL rather than guessing a reprovide interval. We re-announce on a fixed interval that is a guess, not a contract.
3. Address handling. What the server may do with the multiaddrs in an announcement, in particular whether it is allowed to filter or rewrite them. This has cost us real debugging time: routers that keep only /ip4-rooted addresses drop the /dns4-rooted AutoTLS WSS addresses, which are the only ones a browser can dial, so the announcement survives but becomes useless to exactly the clients we care about. We ended up running a rewriting proxy in front of the router to work around it. Whatever the spec decides here, please make it explicit rather than implementation-defined.
4. Authentication. Whether announcements are authenticated, and how, so operators can run routers that are not open write endpoints.
5. Idempotency and deletion. Whether re-announcing the same CID is a no-op refresh, and whether there is a way to withdraw an announcement.

Happy to be a test deployment for whatever shape this lands in, and to report back with results.

[lidel]

Thanks for these @Rinse12. The honest answer to most of them is "unspecified and left to the server": there are no protocol-level limits, no address-handling rules, no withdrawal, and nothing consumes the returned TTL; authentication is the record signature alone, with enforcement being server policy. Rather than answering at length here, I folded all five into the IPIP: see the new "Unspecified Behavior" section and the updated Signature and Response sections, which record what Kubo and index-provider actually do for each.

To be upfront about scope: with the time I have, documenting this API precisely, with test vectors, is as far as I can take it. Fixing the gaps needs an IPIP-378-style successor and someone with bandwidth to drive it; this document is meant to give them a running start, and your list is good input for that.

Could you give the IPIP a final review? If it looks sensible and accurate from your side, I'm fine merging it.
At least you will have a description of this old API in a linkable form.

[Rinse12]

Thanks @lidel, this is a much better outcome than I expected from the five questions. Folding them into an "Unspecified Behavior" section is more useful than answers in a thread, and the observed-behavior notes (non-atomic acceptance, the crash on a record missing Timestamp/AdvisoryTTL, RSA/ECDSA Peer IDs never verifying) are things we would otherwise have found the hard way.

I reviewed the document against our current code. Three items, then an offer.

1. The Known users section needs a correction, our end changed last week

The paragraph says that on our path the Signature is usually not even valid, because a local proxy rewrites Payload.Addrs after Kubo signs the record. That was accurate when you wrote it and is no longer accurate.

We removed that proxy on 2026-08-12 (pkc-js#262, pkc-js#263), because Kubo 0.43.0 fixed the bug it worked around: provider records PUT to a delegated HTTP router were losing the browser-dialable transports (AutoTLS /tls/ws, webrtc-direct). That was ipfs/kubo#11369, fixed by ipfs/kubo#11394. Thank you for that one.

Keeping the proxy after the fix turned out to be worse than removing it. It rebuilt the address set from ipfs id plus ipfs swarm addrs, and self swarm addrs carry no certhash, so the webrtc-direct and webtransport addrs it published were ones no browser could dial. A survey of our six production routers showed exactly that split: nodes still behind the proxy had certhash-less records sitting next to complete ones from natively announcing nodes.

So the current shape of this deployment is:

• pkc-js never constructs or signs a provider record. It calls routing.provide over the Kubo RPC and Kubo builds, signs and PUTs the record itself. Records on this path carry valid signatures.
• The routers still do not verify them, and still bound abuse by requiring announced addrs to match the request IP, so that part of your text stands.

Suggested edit: keep the description, put it in the past tense, and mark the 49f6b9c5 permalink as historic. If it is useful, the removal commit explains the reasoning in full: 08ca094d.

2. A second refresh trigger the document does not mention

Refresh cadence is framed entirely as TTL expiry. In our deployment the dominant trigger is address rotation, not expiry. WebRTC certhashes rotate and AutoTLS names get re-issued, and the API has no way to say "same keys, new addrs", so the only option is a full re-announce of the affected keys.

Concretely, we poll our own node once a minute for changes to its browser-dialable self addresses, and on a change we re-provide just the two pubsub-topic routing CIDs that a browser needs in order to bootstrap into a community. Content CIDs are deliberately excluded, since those are re-announced with fresh addresses on the next publish anyway.

Worth a sentence in the Refresh cadence bullet, because it is a case where a client re-announces far more often than any TTL would suggest, for a reason the API cannot express.

3. AdvisoryTTL in the response, and one normative SHOULD that cannot be followed

The Response Body section says AdvisoryTTL is "TTL in nanoseconds chosen by the server (not the value from the request)" and that "Clients SHOULD republish before this TTL expires."

That describes the reference server, but it is not what a client sees in the wild. Our router echoes the client's own requested value straight back:

resBody.ProvideResults.push({
  Schema: provider.Schema,
  Protocol: provider.Protocol,
  AdvisoryTTL: provider.Payload.AdvisoryTTL
})

Its actual retention is a fixed 24 hours with an hourly sweep, and the response never mentions that number. So a client obeying the SHOULD against our router would be republishing on a schedule it invented itself and got handed back. Combined with your own finding that the reference client discards the response value entirely, that SHOULD has no conforming implementation on either side.

Suggest scoping the sentence to the reference server, or making it non-normative, since a historic document describing observed behavior probably should not carry a SHOULD that nothing implements.

While in that area: it may be worth noting in the Address policy bullet that address rewriting and signature verification are in tension. A server that substitutes or filters Addrs (ours rewrites Kubo's unspecified 0.0.0.0 to the request IP) is storing a record that no longer matches the signature it was sent, which matters the moment anything wants to re-serve or forward the signed record rather than just index it.

On the successor

I would like to drive the IPIP-378-style successor. I cannot promise Shipyard-grade throughput, but we run both sides of this API in production (a Kubo-based client and our own server implementation), which is the position from which the gaps actually hurt, and we have the deployment to test against rather than only a spec to argue about.

One input for that work, from our server's TODO list: what it actually wants is to hand the client an announce interval and a minimum announce interval, evict peers on that basis, and reject clients that announce too often. That is the BitTorrent tracker model, and it is the missing half of the TTL story. This API has a server-to-client TTL field that nobody consumes, and no way at all for a server to state a cadence it will enforce. Whatever replaces it should carry that in the response and make it normative.

The other requirement we care about is on the client side and is not really a spec matter, but it shapes what the spec has to allow: we want the DHT to carry only the handful of keys tied to a community's IPNS and its pubsub swarm, with the HTTP routers carrying every content block. That is ipfs/kubo#11411, and it only becomes a supportable configuration once HTTP providing is a specified path rather than an experimental one.

Everything else in the document matches what I see in our code. The description of our router (stores only Keys, ID and Addrs, no signature verification, addrs must match the request IP) is accurate, as is the note that we run with the public DHT disabled, with the caveat above that the intent is selective DHT publishing rather than none. With the Known users correction applied I am happy for this to merge.