Error getting data from disk cache in concurrent situation

[tanruixiang]

Describe this problem

see titile

---- disk_cache::test::test_disk_cache_multi_thread_fetch_same_block stdout ----
thread 'disk_cache::test::test_disk_cache_multi_thread_fetch_same_block' panicked at components/object_store/src/disk_cache.rs:1107:13:
assertion `left == right` failed
  left: b"i j k l m n o p q r s t u v w x y za b c d e f g\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0 p q r s t u v w x y"
 right: b"i j k l m n o p q r s t u v w x y za b c d e f g h i j k l m n o p q r s t u v w x y"
stack backtrace:
   0: rust_begin_unwind
             at /rustc/8550f15e148407159af401e02b1d9259762b3496/library/std/src/panicking.rs:617:5
   1: core::panicking::panic_fmt
             at /rustc/8550f15e148407159af401e02b1d9259762b3496/library/core/src/panicking.rs:67:14
   2: core::panicking::assert_failed_inner
   3: core::panicking::assert_failed
             at /rustc/8550f15e148407159af401e02b1d9259762b3496/library/core/src/panicking.rs:248:5
   4: object_store::disk_cache::test::test_disk_cache_multi_thread_fetch_same_block::{{closure}}
             at ./src/disk_cache.rs:1107:13
   5: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/8550f15e148407159af401e02b1d9259762b3496/library/core/src/future/future.rs:125:9
   6: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/8550f15e148407159af401e02b1d9259762b3496/library/core/src/future/future.rs:125:9
   7: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}::{{closure}}
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/scheduler/current_thread.rs:651:57
   8: tokio::runtime::coop::with_budget
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/coop.rs:107:5
   9: tokio::runtime::coop::budget
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/coop.rs:73:5
  10: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}::{{closure}}
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/scheduler/current_thread.rs:651:25
  11: tokio::runtime::scheduler::current_thread::Context::enter
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/scheduler/current_thread.rs:410:19
  12: tokio::runtime::scheduler::current_thread::CoreGuard::block_on::{{closure}}
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/scheduler/current_thread.rs:650:36
  13: tokio::runtime::scheduler::current_thread::CoreGuard::enter::{{closure}}
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/scheduler/current_thread.rs:729:68
  14: tokio::runtime::context::scoped::Scoped<T>::set
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/context/scoped.rs:40:9
  15: tokio::runtime::context::set_scheduler::{{closure}}
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/context.rs:176:26
  16: std::thread::local::LocalKey<T>::try_with
             at /rustc/8550f15e148407159af401e02b1d9259762b3496/library/std/src/thread/local.rs:270:16
  17: std::thread::local::LocalKey<T>::with
             at /rustc/8550f15e148407159af401e02b1d9259762b3496/library/std/src/thread/local.rs:246:9
  18: tokio::runtime::context::set_scheduler
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/context.rs:176:9
  19: tokio::runtime::scheduler::current_thread::CoreGuard::enter
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/scheduler/current_thread.rs:729:27
  20: tokio::runtime::scheduler::current_thread::CoreGuard::block_on
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/scheduler/current_thread.rs:638:19
  21: tokio::runtime::scheduler::current_thread::CurrentThread::block_on::{{closure}}
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/scheduler/current_thread.rs:175:28
  22: tokio::runtime::context::runtime::enter_runtime
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/context/runtime.rs:65:16
  23: tokio::runtime::scheduler::current_thread::CurrentThread::block_on
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/scheduler/current_thread.rs:167:9
  24: tokio::runtime::runtime::Runtime::block_on
             at /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.29.1/src/runtime/runtime.rs:311:47
  25: object_store::disk_cache::test::test_disk_cache_multi_thread_fetch_same_block
             at ./src/disk_cache.rs:1106:9
  26: object_store::disk_cache::test::test_disk_cache_multi_thread_fetch_same_block::{{closure}}
             at ./src/disk_cache.rs:[1066](https://github.com/CeresDB/ceresdb/actions/runs/6183763154/job/16786118811?pr=1214#step:8:1067):62
  27: core::ops::function::FnOnce::call_once
             at /rustc/8550f15e148407159af401e02b1d9259762b3496/library/core/src/ops/function.rs:250:5
  28: core::ops::function::FnOnce::call_once
             at /rustc/8550f15e148407159af401e02b1d9259762b3496/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

https://github.com/CeresDB/ceresdb/actions/runs/6183763154/job/16786118811?pr=1214

Server version

a7f09df

Steps to reproduce

CI

Expected behavior

No response

Additional Information

No response

[jiacai2050]

Sadly, this issue still exists 😅

https://github.com/apache/incubator-horaedb/actions/runs/8845827437/job/24290512689