Add adaptive concurrency with -n -1

[folbricht]

Closes #430.

A fixed concurrency limits throughput to about N × chunk size / round-trip time. With the default of 10 and 64KiB chunks, a connection with a 50ms round trip tops out around 100 Mbit/s however fast it is.

Behavior

-n -1 on the command line, or "n": -1 in a store's options in the config, lets desync choose:

• Transfers to and from remote stores (HTTP, S3, GCS, OCI, SFTP, SSH) go through a LimitedStore with an adaptive limiter (NewAdaptiveStore), which adjusts the number of concurrent requests to each store while the command runs, up to 128. Commands run 128 workers when any of their stores is adaptive, and the limit decides how many of them have a request in flight. Local stores aren't limited.
• When only some stores are adaptive, from their options in the config, the other remote stores in the same command (a cache, further stores, failover group members) are held to their own concurrency with a fixed limiter (NewLimitedStore). Without an adaptive store, nothing is wrapped.
• Work bound by the CPU, chunking in make and hashing in verify and verify-index, uses runtime.GOMAXPROCS(0).

A plain number behaves as before. 0 and values below -1 are still rejected on the command line. In the config, 0 is the same as not setting n, which is why -1 rather than 0 is the value for this.

Store connection pools (HTTP and OCI idle connections, S3 transport, SFTP and SSH sessions) are sized for the maximum when the concurrency is adaptive. OCI prune, which doesn't adapt, runs 10 removals at a time with -1. With --verbose, every change of a limit is logged.

How the limit is picked

The limiter works like TCP BBR sizes its congestion window. It starts at 10 and doubles after each window of requests (at least 20, about one round trip) until throughput grows by less than 25% for three windows in a row. From then on the limit is twice the highest throughput of the last 10 windows times the lowest average latency seen, which is twice the number of requests the network and the store can process at once. The headroom lets throughput grow when the network or the store get faster, and the limit follows. A failed request, other than a missing chunk, halves the limit and discards the throughput history. Windows that don't reach the limit don't change it.

Latency alone isn't a reliable signal here. Responses sharing a link take longer well before the link is full, so a latency-based limit (TCP Vegas style) backs off early: about half the achievable throughput at a 20ms round trip.

Benchmark

extract of 256MB of random data (64KiB average chunks) from chunk-server, over loopback shaped with tc netem in a network namespace. Throughput in Mbit/s:

Link	-n 10	-n 100	-n -1	Final limit
1 Gbit/s, 20ms RTT	222	904	853	128
1 Gbit/s, 50ms RTT	98	662	646	128
1 Gbit/s, 100ms RTT	51	462	468	128
50 Mbit/s, 50ms RTT (64MB)	46	47	47	29
Loopback, no shaping	1930	2216	2155	~25

At a 100ms round trip, filling the link would take about 190 requests in flight, so both are bounded by the ceiling.

The limit settles higher than the link needs, since the lowest latency is first measured while connections are still being set up. That doesn't cost throughput, but keeps more connections open than necessary.

[RyuzakiKK]

I did some manual tests of these changes and the adaptive concurrency adjustments seem to be working as expected.