Skip to content

Unclear "Use of Closed Network Connection" Error #405

Open

Description

@abraithwaite

Describe the bug

When using clickhouse-go with clickhouse-cloud, we observe these errors happening during what we suspect to be auto-scaling events on the cloud side.

{"time":"2024-05-31T16:22:56.043609257Z","level":"INFO","source":{"function":"github.com/runreveal/lib/await.(*runner).Run.func1","file":"/home/runner/go/pkg/mod/github.com/runreveal/lib/await@v0.0.0-20240514034311-588661551148/await.go","line":110},"msg":"subroutine error: read:\n    github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n        /home/runner/go/pkg/mod/github.com/!click!house/ch-go@v0.61.5/proto/reader.go:62\n  - read tcp 172.30.76.61:54124->35.82.252.60:9440: use of closed network connection"}

I believe it's because clickhouse proactively closes network connections that have encountered some kind of exception? It would be nice to have either a clearer error message here, or even better automatic reconnection / retries in the event of a failure here.

Getting this particular error bubbled up to the application isn't particularly useful. The only thing we can do is kill the client and recreate it before re-issuing the query because we can no longer trust the connection.

To be clear: we are not closing the connection on our end, so for whatever reason clickhouse server has to close the connection proactively, I think this library should be able to catch and handle that situation for a cleaner shutdown with a sentinel error value at the very least so we can make a more informed decision about how to handle it.

Steps to reproduce

  1. Create a clickhouse client connection to a clickhouse-cloud instance in Go.
  2. Trigger some scaling events while attempting to insert large batches.
  3. Observe the error

Expected behaviour

We get a better error returned or have options to configure retries, or both.

Error log

{"time":"2024-05-31T16:22:56.043609257Z","level":"INFO","source":{"function":"github.com/runreveal/lib/await.(*runner).Run.func1","file":"/home/runner/go/pkg/mod/github.com/runreveal/lib/await@v0.0.0-20240514034311-588661551148/await.go","line":110},"msg":"subroutine error: read:\n    github.com/ClickHouse/ch-go/proto.(*Reader).ReadFull\n        /home/runner/go/pkg/mod/github.com/!click!house/ch-go@v0.61.5/proto/reader.go:62\n  - read tcp 172.30.76.61:54124->35.82.252.60:9440: use of closed network connection"}

Configuration

Environment

  • Client version: github.com/ClickHouse/clickhouse-go/v2 v2.23.2
  • ch-go version: v0.61.5
  • Language version: 1.22.3
  • OS: Ubuntu Linux 22.04

ClickHouse server

  • ClickHouse Server version: 24.2.2
  • ClickHouse Server non-default settings, if any: clickhouse-cloud default settings.
  • CREATE TABLE statements for tables involved:
  • Sample data for all these tables, use clickhouse-obfuscator if necessary

Activity

  1. changed the title [-]Use of Closed Network Connection[/-] [+]Unclear Use of Closed Network Connection Error[/+] on May 31, 2024
  2. changed the title [-]Unclear Use of Closed Network Connection Error[/-] [+]Unclear "Use of Closed Network Connection" Error[/+] on May 31, 2024
  3. charredlot commented on Jun 24, 2024

    @charredlot
    Contributor

    not sure if it's the same issue, but ran into a similar problem with the chpool client getting stuck in a bad state when the server-side disconnects...made a PR here #409

  4. added a commit that references this issue on Jul 1, 2024
  5. cwegener commented on Jul 17, 2024

    @cwegener

    FWIW, I can see these errors when using a local ClickHouse server standalone instance.

  6. polyglotAI-bot commented on Nov 17, 2026

    @polyglotAI-bot
    Collaborator

    Recommendation: Close — already fixed
    #409 fixes the ch-go part (after a dropped connection, the client stayed open and each next query failed with broken pipe), first release v0.62.0. The message use of closed network connection comes from clickhouse-go: report a new case on a current release there.

    Analysis: use of closed network connection is Go's net.ErrClosed: the client process closed its own socket, while a connection that the server or a proxy drops gives EOF, broken pipe, or connection reset by peer in ch-go. In the report, clickhouse-go v2.23.2 batch.Send closes the socket when the batch context ends (conn_batch.go:220-226 at v2.23.2), so the pending read returns this raw error; on clickhouse-go main it returns ctx.Err() (conn_process.go:119-121, ClickHouse/clickhouse-go#1389, first release v2.29.0). In ch-go v0.61.5, cancelQuery returned before Close when its write failed; #409 always closes the client (query.go:45-48), and automatic reconnect is out of scope for ch.Client (AGENTS.md:28).

    Reproduction: complete program and output
    // Issue 405: which error does ch-go return when the connection drops,
    // and can the caller use the client after that?
    package main
    
    import (
    	"context"
    	"errors"
    	"fmt"
    	"io"
    	"net"
    	"os"
    	"sync"
    	"time"
    
    	"github.com/ClickHouse/ch-go"
    	"github.com/ClickHouse/ch-go/proto"
    )
    
    // proxy forwards TCP to the server. drop closes all forwarded connections,
    // as a server or a load balancer does when it drops a connection.
    type proxy struct {
    	ln    net.Listener
    	mu    sync.Mutex
    	conns []net.Conn
    }
    
    func newProxy(target string) *proxy {
    	ln, err := net.Listen("tcp", "127.0.0.1:0")
    	if err != nil {
    		panic(err)
    	}
    	p := &proxy{ln: ln}
    	go func() {
    		for {
    			c, err := ln.Accept()
    			if err != nil {
    				return
    			}
    			u, err := net.Dial("tcp", target)
    			if err != nil {
    				_ = c.Close()
    				continue
    			}
    			p.mu.Lock()
    			p.conns = append(p.conns, c, u)
    			p.mu.Unlock()
    			go func() { _, _ = io.Copy(u, c) }()
    			go func() { _, _ = io.Copy(c, u) }()
    		}
    	}()
    	return p
    }
    
    func (p *proxy) drop() {
    	p.mu.Lock()
    	defer p.mu.Unlock()
    	for _, c := range p.conns {
    		_ = c.Close()
    	}
    	p.conns = nil
    }
    
    func report(name string, c *ch.Client, err error) {
    	fmt.Printf("%s: err=%v\n", name, err)
    	fmt.Printf("%s: errors.Is(io.EOF)=%v errors.Is(DeadlineExceeded)=%v IsClosed=%v\n",
    		name, errors.Is(err, io.EOF), errors.Is(err, context.DeadlineExceeded), c.IsClosed())
    	err = c.Do(context.Background(), ch.Query{Body: "SELECT 1"})
    	fmt.Printf("%s: next Do: err=%v errors.Is(ch.ErrClosed)=%v\n", name, err, errors.Is(err, ch.ErrClosed))
    }
    
    func main() {
    	ctx := context.Background()
    	addr := os.Getenv("CLICKHOUSE_ADDR")
    	if addr == "" {
    		addr = "localhost:9000"
    	}
    	const slowSelect = "SELECT sleepEachRow(0.5) FROM numbers(10) SETTINGS max_block_size = 1"
    
    	// Case 1: the connection drops while a SELECT streams.
    	{
    		p := newProxy(addr)
    		c, err := ch.Dial(ctx, ch.Options{Address: p.ln.Addr().String()})
    		if err != nil {
    			panic(err)
    		}
    		time.AfterFunc(700*time.Millisecond, p.drop)
    		var v proto.ColUInt8
    		err = c.Do(ctx, ch.Query{Body: slowSelect,
    			Result:   proto.Results{{Name: "sleepEachRow(0.5)", Data: &v}},
    			OnResult: func(context.Context, proto.Block) error { return nil }})
    		report("select-drop", c, err)
    	}
    
    	// Case 2: the connection drops during a large INSERT.
    	{
    		admin, err := ch.Dial(ctx, ch.Options{Address: addr})
    		if err != nil {
    			panic(err)
    		}
    		for _, q := range []string{"DROP TABLE IF EXISTS issue405",
    			"CREATE TABLE issue405 (v UInt64) ENGINE = MergeTree ORDER BY tuple()"} {
    			if err := admin.Do(ctx, ch.Query{Body: q}); err != nil {
    				panic(err)
    			}
    		}
    		p := newProxy(addr)
    		c, err := ch.Dial(ctx, ch.Options{Address: p.ln.Addr().String()})
    		if err != nil {
    			panic(err)
    		}
    		var v proto.ColUInt64
    		for i := 0; i < 100_000; i++ {
    			v.Append(uint64(i))
    		}
    		time.AfterFunc(300*time.Millisecond, p.drop)
    		err = c.Do(ctx, ch.Query{Body: "INSERT INTO issue405 VALUES",
    			Input:   proto.Input{{Name: "v", Data: &v}},
    			OnInput: func(context.Context) error { return nil }}) // send the same block forever
    		report("insert-drop", c, err)
    		_ = admin.Do(ctx, ch.Query{Body: "DROP TABLE issue405"})
    	}
    
    	// Case 3: the caller's context ends while a SELECT streams.
    	{
    		c, err := ch.Dial(ctx, ch.Options{Address: addr})
    		if err != nil {
    			panic(err)
    		}
    		qctx, cancel := context.WithTimeout(ctx, 700*time.Millisecond)
    		var v proto.ColUInt8
    		err = c.Do(qctx, ch.Query{Body: slowSelect,
    			Result:   proto.Results{{Name: "sleepEachRow(0.5)", Data: &v}},
    			OnResult: func(context.Context, proto.Block) error { return nil }})
    		cancel()
    		report("select-ctx-timeout", c, err)
    	}
    }

    Output on main (4c61f2c), ClickHouse 26.10.4.207:

    select-drop: err=packet: uvarint: read: read: EOF
    select-drop: errors.Is(io.EOF)=true errors.Is(DeadlineExceeded)=false IsClosed=true
    select-drop: next Do: err=client is closed errors.Is(ch.ErrClosed)=true
    insert-drop: err=send input: flush: writev tcp …->…: writev: broken pipe
    insert-drop: errors.Is(io.EOF)=false errors.Is(DeadlineExceeded)=false IsClosed=true
    insert-drop: next Do: err=client is closed errors.Is(ch.ErrClosed)=true
    select-ctx-timeout: err=context deadline exceeded
    select-ctx-timeout: errors.Is(io.EOF)=false errors.Is(DeadlineExceeded)=true IsClosed=true
    select-ctx-timeout: next Do: err=client is closed errors.Is(ch.ErrClosed)=true
    

    Output on v0.61.5 (the version in the report), same program and server:

    select-drop: err=packet: uvarint: read: read: EOF
    select-drop: errors.Is(io.EOF)=true errors.Is(DeadlineExceeded)=false IsClosed=true
    select-drop: next Do: err=client is closed errors.Is(ch.ErrClosed)=true
    insert-drop: err=send input: flush: write: write tcp …->…: write: broken pipe
    insert-drop: errors.Is(io.EOF)=false errors.Is(DeadlineExceeded)=false IsClosed=false
    insert-drop: next Do: err=flush: write: write tcp …->…: write: broken pipe errors.Is(ch.ErrClosed)=false
    select-ctx-timeout: err=context deadline exceeded
    select-ctx-timeout: errors.Is(io.EOF)=false errors.Is(DeadlineExceeded)=true IsClosed=true
    select-ctx-timeout: next Do: err=client is closed errors.Is(ch.ErrClosed)=true
    

    Automated triage by Polyglot AI. A maintainer makes the final decision. Shortened on 2026-11-20: the edit history has the first version.

  7. added
    connectivityArea: connection, TLS, pool, timeouts, cancellation, transport, compression
    on Nov 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingconnectivityArea: connection, TLS, pool, timeouts, cancellation, transport, compression

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions