Repository navigation
Description
Describe the bug
When using clickhouse-go with clickhouse-cloud, we observe these errors happening during what we suspect to be auto-scaling events on the cloud side.
{"time":"2024-05-31T16:22:56.043609257Z","level":"INFO","source":{"function":"github.com/runreveal/lib/await.(*runner).Run.func1","file":"/home/runner/go/pkg/mod/github.com/runreveal/lib/await@v0.0.0-20240514034311-588661551148/await.go","line":110},"msg":"subroutine error: read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!click!house/ch-go@v0.61.5/proto/reader.go:62\n - read tcp 172.30.76.61:54124->35.82.252.60:9440: use of closed network connection"}
I believe it's because clickhouse proactively closes network connections that have encountered some kind of exception? It would be nice to have either a clearer error message here, or even better automatic reconnection / retries in the event of a failure here.
Getting this particular error bubbled up to the application isn't particularly useful. The only thing we can do is kill the client and recreate it before re-issuing the query because we can no longer trust the connection.
To be clear: we are not closing the connection on our end, so for whatever reason clickhouse server has to close the connection proactively, I think this library should be able to catch and handle that situation for a cleaner shutdown with a sentinel error value at the very least so we can make a more informed decision about how to handle it.
Steps to reproduce
- Create a clickhouse client connection to a clickhouse-cloud instance in Go.
- Trigger some scaling events while attempting to insert large batches.
- Observe the error
Expected behaviour
We get a better error returned or have options to configure retries, or both.
Error log
{"time":"2024-05-31T16:22:56.043609257Z","level":"INFO","source":{"function":"github.com/runreveal/lib/await.(*runner).Run.func1","file":"/home/runner/go/pkg/mod/github.com/runreveal/lib/await@v0.0.0-20240514034311-588661551148/await.go","line":110},"msg":"subroutine error: read:\n github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n /home/runner/go/pkg/mod/github.com/!click!house/ch-go@v0.61.5/proto/reader.go:62\n - read tcp 172.30.76.61:54124->35.82.252.60:9440: use of closed network connection"}
Configuration
Environment
- Client version: github.com/ClickHouse/clickhouse-go/v2 v2.23.2
- ch-go version: v0.61.5
- Language version: 1.22.3
- OS: Ubuntu Linux 22.04
ClickHouse server
- ClickHouse Server version: 24.2.2
- ClickHouse Server non-default settings, if any: clickhouse-cloud default settings.
CREATE TABLEstatements for tables involved:- Sample data for all these tables, use clickhouse-obfuscator if necessary
Activity
[-]Use of Closed Network Connection[/-][+]Unclear Use of Closed Network Connection Error[/+][-]Unclear Use of Closed Network Connection Error[/-][+]Unclear "Use of Closed Network Connection" Error[/+]fix(query): always close client.conn in cancelQuery (issue ClickHouse…
fix(query): always close client.conn in cancelQuery (issue ClickHouse…
charredlot commentedon Jun 24, 2024
not sure if it's the same issue, but ran into a similar problem with the chpool client getting stuck in a bad state when the server-side disconnects...made a PR here #409
fix(query): always close client.conn in cancelQuery (issue ClickHouse…
Merge pull request #409 from charredlot/close-conn
cwegener commentedon Jul 17, 2024
FWIW, I can see these errors when using a local ClickHouse server standalone instance.
polyglotAI-bot commentedon Nov 17, 2026
Recommendation: Close — already fixed
#409 fixes the ch-go part (after a dropped connection, the client stayed open and each next query failed with
broken pipe), first release v0.62.0. The messageuse of closed network connectioncomes from clickhouse-go: report a new case on a current release there.main: Fixed on main — the client stays open and broken on v0.61.5 (2 of 2 runs), not onmain.batch.Sendwatchdog).errors.Is(err, ch.ErrClosed)orIsClosed()before a retry;chpooldestroys a closed client onRelease(chpool/client.go:26-27).Analysis:
use of closed network connectionis Go'snet.ErrClosed: the client process closed its own socket, while a connection that the server or a proxy drops givesEOF,broken pipe, orconnection reset by peerin ch-go. In the report, clickhouse-go v2.23.2batch.Sendcloses the socket when the batch context ends (conn_batch.go:220-226at v2.23.2), so the pending read returns this raw error; on clickhouse-gomainit returnsctx.Err()(conn_process.go:119-121, ClickHouse/clickhouse-go#1389, first release v2.29.0). In ch-go v0.61.5,cancelQueryreturned beforeClosewhen its write failed; #409 always closes the client (query.go:45-48), and automatic reconnect is out of scope forch.Client(AGENTS.md:28).Reproduction: complete program and output
Output on
main(4c61f2c), ClickHouse 26.10.4.207:Output on v0.61.5 (the version in the report), same program and server:
Automated triage by Polyglot AI. A maintainer makes the final decision. Shortened on 2026-11-20: the edit history has the first version.