Segmentation fault during blob_client::upload_block_blob_from_stream

[drorgy]

This issue looks similar to this issue from the azure-storage-cpp project: Azure/azure-storage-cpp#265
(in that issue it was mentioned that there were indeed segmentation fault errors in azure-storage-cpp version 3.0 which where fixed later on).

Valgrind output of the error I'm getting:

==8950== Process terminating with default action of signal 11 (SIGSEGV)
==8950== Access not within mapped region at address 0x4008
==8950== at 0xAF4500E: std::istream::read(char*, long) (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.22)
==8950== by 0x60C7F09: azure::storage_lite::CurlEasyRequest::read(char*, unsigned long, unsigned long, void*) (in /mnt/public/zdevfs2/yoav/src/dev/unittest/CppLinuxUnitTests/builds/libazure-storage-cpplite.so)
==8950== by 0x599F5A2: Curl_fillreadbuffer (in /mnt/public/zdevfs2/yoav/src/dev/unittest/CppLinuxUnitTests/builds/libcurl.so)
==8950== by 0x59A0783: Curl_readwrite (in /mnt/public/zdevfs2/yoav/src/dev/unittest/CppLinuxUnitTests/builds/libcurl.so)
==8950== by 0x59A9885: multi_runsingle (in /mnt/public/zdevfs2/yoav/src/dev/unittest/CppLinuxUnitTests/builds/libcurl.so)
==8950== by 0x59AA260: curl_multi_perform (in /mnt/public/zdevfs2/yoav/src/dev/unittest/CppLinuxUnitTests/builds/libcurl.so)
==8950== by 0x59A1CE9: curl_easy_perform (in /mnt/public/zdevfs2/yoav/src/dev/unittest/CppLinuxUnitTests/builds/libcurl.so)
==8950== by 0x60C66F7: azure::storage_lite::CurlEasyRequest::perform() (in /mnt/public/zdevfs2/yoav/src/dev/unittest/CppLinuxUnitTests/builds/libazure-storage-cpplite.so)
==8950== by 0x60C9021: azure::storage_lite::CurlEasyRequest::submit(std::function<void (int, azure::storage_lite::storage_istream, CURLcode)>, std::chrono::duration<long, std::ratio<1l, 1l> >) (in /mnt/public/zdevfs2/yoav/src/dev/unittest/CppLinuxUnitTests/builds/libazure-storage-cpplite.so)
==8950== by 0x60DDFF1: azure::storage_lite::async_executor::submit_helper(std::shared_ptr<std::promise<azure::storage_lite::storage_outcome > >, std::shared_ptr<azure::storage_lite::storage_outcome >, std::shared_ptrazure::storage_lite::storage_account, std::shared_ptrazure::storage_lite::storage_request_base, std::shared_ptrazure::storage_lite::http_base, std::shared_ptrazure::storage_lite::executor_context, std::shared_ptrazure::storage_lite::retry_context) (in /mnt/public/zdevfs2/yoav/src/dev/unittest/CppLinuxUnitTests/builds/libazure-storage-cpplite.so)
==8950== by 0x60E4CFB: azure::storage_lite::async_executor::submit(std::shared_ptrazure::storage_lite::storage_account, std::shared_ptrazure::storage_lite::storage_request_base, std::shared_ptrazure::storage_lite::http_base, std::shared_ptrazure::storage_lite::executor_context) (in /mnt/public/zdevfs2/yoav/src/dev/unittest/CppLinuxUnitTests/builds/libazure-storage-cpplite.so)
==8950== by 0x60D3100: azure::storage_lite::blob_client::upload_block_blob_from_stream(std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::__cxx11::basic_string<char, std::char_traits, std::allocator > const&, std::istream&, std::vector<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits, std::allocator >, std::__cxx11::basic_string<char, std::char_traits, std::allocator > > > > const&, unsigned long) (in /mnt/public/zdevfs2/yoav/src/dev/unittest/CppLinuxUnitTests/builds/libazure-storage-cpplite.so)

Thanks,
Dror

[Jinming-Hu]

drorgy Can you share a code snippet that can reproduce this issue?

[drorgy]

@JinmingHu-MSFT
Thanks, we are currently working on getting code snippet to reproduce this as code base it is reproduced in is big.

[drorgy]

We still don't have a code snippet yet, but I came across these links:
https://stackoverflow.com/questions/21887264/why-libcurl-needs-curlopt-nosignal-option-and-what-are-side-effects-when-it-is
https://curl.haxx.se/mail/lib-2010-01/0115.html

Could it be that the crash is because CURLOPT_NOSIGNAL is not set to 1? (curl_easy_setopt(handle, CURLOPT_NOSIGNAL, 1L); )
I checked the azure-storage-cpplite code and didn't find CURLOPT_NOSIGNAL anywhere.

Thanks,
Dror

[drorgy]

I just compiled after setting CURLOPT_NOSIGNAL. The segmentation faults don't happen but getting SIGPIPE and my requests to azure-storage are stuck.
SIGPIPEs should be ok - https://curl.haxx.se/libcurl/c/CURLOPT_NOSIGNAL.html (so the issue here is that the requests are stuck).
To see change made: #89

[Jinming-Hu]

Hi drorgy , can you confirm that this crash issue (SIGSEGV) is caused by DNS failure? I mean, can you double check that you're using neither c-ares nor threaded DNS?

Because the change in your PR involves behavior change of this SDK, which will affect all of our cpplite customers. So I want to be careful about this.

In addition about the SIGPIPE issue, if we don't enable CURLOPT_NOSIGNAL, the program will receive SIGPIPE and crash when there's something wrong with the TLS connection. If we enable CURLOPT_NOSIGNAL, what would happen instead?

[drorgy]

Hi Jinming,
I saw in another similar SDK that CURLOPT_NOSIGNAL is turned on.
See also this link: https://curl.haxx.se/libcurl/c/threadsafe.html
Currently the code I sent in PR is not complete as my program gets stuck (I configured SIGPIPE to be ignored in my program anyway). I think there is probably additional settings needed aside from CURLOPT_NOSIGNAL.
Dror