Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Memory retention discussion - not a bug (yet?) #28

Open
ghost opened this issue Mar 12, 2022 · 6 comments
Open

Memory retention discussion - not a bug (yet?) #28

ghost opened this issue Mar 12, 2022 · 6 comments

Comments

@ghost
Copy link

ghost commented Mar 12, 2022 •
edited by ghost
Loading

So, looking through valgrind and dr_memory reports, I'm seeing no explicit memory leaks (all allocated memory is still reachable). However, I am seeing what looks like a memory retention/bloat issue with easylogging++. It seems that every time the logger is used to write, it instantiates a new el::Logger object, which calls el::Logger::configure(), eventually resulting in an el::Configurations::unsafeSet() allocation that gets pushed onto a list.
Culprit seems to be here:

void Configurations::unsafeSet(Level level, ConfigurationType configurationType, const std::string& value) {
  Configuration* conf = RegistryWithPred<Configuration, Configuration::Predicate>::get(level, configurationType);
  if (conf == nullptr) {
    registerNew(new Configuration(level, configurationType, value)); 
  } else {
    conf->setValue(value);
  }

which does the push:

  virtual inline void registerNew(T_Ptr* ptr) ELPP_FINAL {
    this->list().push_back(ptr);
  }

I'm not sure if/when it ever gets popped back off of this stack. And this seems to happen repetitively. My guess is that there is some sort of cleanup that needs to be happening in the xcash/monero core code after each write (or somehow convincing easylogging++ not to instantiate a Logger object every single time).

For example, this MTRACE() macro from cryptonote_protocol_handler.inl seems to be one of the instigators:

  template<class t_core>
  bool t_cryptonote_protocol_handler<t_core>::kick_idle_peers()
  {
    MTRACE("Checking for idle peers...");
    std::vector<boost::uuids::uuid> kick_connections;

Still researching it, though...
@ghost
Copy link
Author

ghost commented Mar 12, 2022 •
edited by ghost
Loading

An example valgrind finding from running xcashd for around 10 minutes.
63 times it lost 48 bytes (3,024 / 63) so it will be a fairly slow memory growth

==313226== 3,024 bytes in 63 blocks are still reachable in loss record 757 of 792
==313226==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==313226==    by 0x5A0FB55: el::Configurations::unsafeSet(el::Level, el::ConfigurationType, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (easylogging++.cc:496)
==313226==    by 0x5A0FD6F: el::Configurations::unsafeSetGlobally(el::ConfigurationType, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)::{lambda()#1}::operator()() const (easylogging++.cc:524)
==313226==    by 0x5A210DC: std::_Function_handler<bool (), el::Configurations::unsafeSetGlobally(el::ConfigurationType, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)::{lambda()#1}>::_M_invoke(std::_Any_data const&) (std_function.h:285)
==313226==    by 0x5A26869: std::function<bool ()>::operator()() const (std_function.h:688)
==313226==    by 0x5A0CBE4: el::LevelHelper::forEachLevel(unsigned int*, std::function<bool ()> const&) (easylogging++.cc:117)
==313226==    by 0x5A0FE2B: el::Configurations::unsafeSetGlobally(el::ConfigurationType, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) (easylogging++.cc:523)
==313226==    by 0x5A0FBAE: el::Configurations::unsafeSet(el::Level, el::ConfigurationType, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (easylogging++.cc:501)
==313226==    by 0x5A0D671: el::Configurations::set(el::Level, el::ConfigurationType, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (easylogging++.cc:291)
==313226==    by 0x5A0D731: el::Configurations::set(el::Configuration*) (easylogging++.cc:301)
==313226==    by 0x5A0D3E9: el::Configurations::setFromBase(el::Configurations*) (easylogging++.cc:262)
==313226==    by 0x5A10925: el::Logger::configure(el::Configurations const&) (easylogging++.cc:606)
==313226==    by 0x5A10451: el::Logger::Logger(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, el::Configurations const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::shared_ptr<std::basic_fstream<char, std::char_traits<char> > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<std::basic_fstream<char, std::char_traits<char> > > > > >*) (easylogging++.cc:565)
==313226==    by 0x5A17A66: el::base::RegisteredLoggers::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) (easylogging++.cc:1847)
==313226==    by 0x5A1CC10: el::base::Writer::initializeLogger(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool) (easylogging++.cc:2573)
==313226==    by 0x5A1CA59: el::base::Writer::construct(int, char const*, ...) (easylogging++.cc:2565)
==313226==    by 0x4E22AC3: cryptonote::t_cryptonote_protocol_handler<cryptonote::core>::kick_idle_peers() (cryptonote_protocol_handler.inl:1304)
==313226==    by 0x4E21BD4: operator() (mem_fn_template.hpp:49)
==313226==    by 0x4E21BD4: operator()<bool, boost::_mfi::mf0<bool, cryptonote::t_cryptonote_protocol_handler<cryptonote::core> >, boost::_bi::list0> (bind.hpp:249)
==313226==    by 0x4E21BD4: operator() (bind.hpp:1294)
==313226==    by 0x4E21BD4: do_call<boost::_bi::bind_t<bool, boost::_mfi::mf0<bool, cryptonote::t_cryptonote_protocol_handler<cryptonote::core> >, boost::_bi::list1<boost::_bi::value<cryptonote::t_cryptonote_protocol_handler<cryptonote::core>*> > > > (math_helper.h:258)
==313226==    by 0x4E21BD4: cryptonote::t_cryptonote_protocol_handler<cryptonote::core>::on_idle() (cryptonote_protocol_handler.inl:1297)
==313226==    by 0x4E14AE5: operator() (mem_fn_template.hpp:49)
==313226==    by 0x4E14AE5: operator()<bool, boost::_mfi::mf0<bool, cryptonote::t_cryptonote_protocol_handler<cryptonote::core> >, boost::_bi::list0> (bind.hpp:249)
==313226==    by 0x4E14AE5: operator() (bind.hpp:1294)
==313226==    by 0x4E14AE5: epee::net_utils::boosted_tcp_server<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >::idle_callback_conext<boost::_bi::bind_t<bool, boost::_mfi::mf0<bool, cryptonote::t_cryptonote_protocol_handler<cryptonote::core> >, boost::_bi::list1<boost::_bi::value<cryptonote::t_cryptonote_protocol_handler<cryptonote::core>*> > > >::call_handler() (abstract_tcp_server2.h:263)
==313226==    by 0x4E596D1: epee::net_utils::boosted_tcp_server<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >::global_timer_handler(boost::shared_ptr<epee::net_utils::boosted_tcp_server<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >::idle_callback_conext_base>) (abstract_tcp_server2.h:280)

And another, this time from within tools::log_stack_trace() :

==313226== 3,024 bytes in 63 blocks are still reachable in loss record 765 of 792
==313226==    at 0x483BE63: operator new(unsigned long) (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==313226==    by 0x5A0FB55: el::Configurations::unsafeSet(el::Level, el::ConfigurationType, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (easylogging++.cc:496)
==313226==    by 0x5A0FD6F: el::Configurations::unsafeSetGlobally(el::ConfigurationType, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)::{lambda()#1}::operator()() const (easylogging++.cc:524)
==313226==    by 0x5A210DC: std::_Function_handler<bool (), el::Configurations::unsafeSetGlobally(el::ConfigurationType, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)::{lambda()#1}>::_M_invoke(std::_Any_data const&) (std_function.h:285)
==313226==    by 0x5A26869: std::function<bool ()>::operator()() const (std_function.h:688)
==313226==    by 0x5A0CBE4: el::LevelHelper::forEachLevel(unsigned int*, std::function<bool ()> const&) (easylogging++.cc:117)
==313226==    by 0x5A0FE2B: el::Configurations::unsafeSetGlobally(el::ConfigurationType, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) (easylogging++.cc:523)
==313226==    by 0x5A0FBAE: el::Configurations::unsafeSet(el::Level, el::ConfigurationType, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (easylogging++.cc:501)
==313226==    by 0x5A0D671: el::Configurations::set(el::Level, el::ConfigurationType, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&) (easylogging++.cc:291)
==313226==    by 0x5A0D731: el::Configurations::set(el::Configuration*) (easylogging++.cc:301)
==313226==    by 0x5A0D3E9: el::Configurations::setFromBase(el::Configurations*) (easylogging++.cc:262)
==313226==    by 0x5A10925: el::Logger::configure(el::Configurations const&) (easylogging++.cc:606)
==313226==    by 0x5A10451: el::Logger::Logger(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, el::Configurations const&, std::map<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >, std::shared_ptr<std::basic_fstream<char, std::char_traits<char> > >, std::less<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > >, std::allocator<std::pair<std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const, std::shared_ptr<std::basic_fstream<char, std::char_traits<char> > > > > >*) (easylogging++.cc:565)
==313226==    by 0x5A17A66: el::base::RegisteredLoggers::get(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool) (easylogging++.cc:1847)
==313226==    by 0x5A1CC10: el::base::Writer::initializeLogger(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool, bool) (easylogging++.cc:2573)
==313226==    by 0x5A1CA59: el::base::Writer::construct(int, char const*, ...) (easylogging++.cc:2565)
==313226==    by 0x55117A9: tools::log_stack_trace(char const*) (stack_trace.cpp:133)
==313226==    by 0x5512C06: __cxa_throw (stack_trace.cpp:95)
==313226==    by 0x59261DC: ??? (in /usr/lib/x86_64-linux-gnu/libboost_thread.so.1.71.0)
==313226==    by 0x489FC73: cryptonote::rpc::ZmqServer::serve() (zmq_server.cpp:87)

@zachhildreth
Copy link
Member

Hi @CygnusMiner
Great work on this!

If you can continue testing and find a code fix for the slow mem leak, we can do a bug bounty on this for you. Its because this is an important problem, and yes its a slow mem leak as it takes weeks for it to stall

@ghost
Copy link
Author

ghost commented Mar 14, 2022 •
edited by ghost
Loading

Ok, I've been squirreled by a separate finding from valgrind (below) - an actual memory leak this time :) - 2MB per allocate.

Tracing it down, it seems to be occurring here:

    // this isn't supposed to happen, but guard against it for now.
    if(hp_state == NULL)
        slow_hash_allocate_state();

The commenter seems to think that the hp_state thread pointer should already have been allocated, but the only deliberate allocation I see for the pointer is in miner.cpp and in blockchain.cpp, in both cases they specifically free the allocation after they're done with it (miner.cpp and blockchain.cpp).

This allocation seems to occur per thread so it should be a limited memory leak (unless threads are destroyed and recreated often in here - I don't know the Monero/Xcash code that well).

However, limited or not it looks like the best approach would be to remove the conditional:

    // this isn't supposed to happen, but guard against it for now.
    if(hp_state == NULL)

explicitly allocating it, and then call the slow_hash_free_state() after you're done with hp_state pointer, somewhere around here

Also, as an aside and just to apease the Paranoid OCD in me, wouldn't it be prudent to memset the block prior to freeing after each use in encrypting? Just to reduce the chance of any accidental loss of sensitive data from a forced core dump?

For reference, one of the findings from Valgrind:

==401653== 4,194,304 bytes in 2 blocks are definitely lost in loss record 60 of 61
==401653==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==401653==    by 0x558CAB3: slow_hash_allocate_state (slow-hash.c:689)
==401653==    by 0x558CFB6: cn_slow_hash (slow-hash.c:771)
==401653==    by 0x52DEC02: cn_slow_hash (hash.h:75)
==401653==    by 0x52DEC02: cryptonote::get_block_longhash(cryptonote::block const&, crypto::hash&, unsigned long) (cryptonote_format_utils.cpp:1035)
==401653==    by 0x52DEC8D: cryptonote::get_block_longhash(cryptonote::block const&, unsigned long) (cryptonote_format_utils.cpp:1062)
==401653==    by 0x50D6EC8: cryptonote::Blockchain::handle_block_to_main_chain(cryptonote::block const&, crypto::hash const&, cryptonote::block_verification_context&) (blockchain.cpp:3460)
==401653==    by 0x50E08C9: cryptonote::Blockchain::add_new_block(cryptonote::block const&, cryptonote::block_verification_context&) (blockchain.cpp:4332)
==401653==    by 0x5115055: cryptonote::core::add_new_block(cryptonote::block const&, cryptonote::block_verification_context&) (cryptonote_core.cpp:1333)
==401653==    by 0x5125D9E: cryptonote::core::handle_incoming_block(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, cryptonote::block_verification_context&, bool) (cryptonote_core.cpp:1380)
==401653==    by 0x4E70BC4: cryptonote::t_cryptonote_protocol_handler<cryptonote::core>::try_add_next_blocks(cryptonote::cryptonote_connection_context&) (cryptonote_protocol_handler.inl:1192)
==401653==    by 0x4E74C53: cryptonote::t_cryptonote_protocol_handler<cryptonote::core>::handle_response_get_objects(int, cryptonote::NOTIFY_RESPONSE_GET_OBJECTS::request&, cryptonote::cryptonote_connection_context&) (cryptonote_protocol_handler.inl:1057)
==401653==    by 0x2BF7E2: operator() (mem_fn_template.hpp:393)
==401653==    by 0x2BF7E2: operator()<int, boost::_mfi::mf3<int, cryptonote::t_cryptonote_protocol_handler<cryptonote::core>, int, cryptonote::NOTIFY_RESPONSE_GET_OBJECTS::request&, cryptonote::cryptonote_connection_context&>, boost::_bi::rrlist3<int&, boost::value_initialized<cryptonote::NOTIFY_RESPONSE_GET_OBJECTS::request>&, cryptonote::cryptonote_connection_context&> > (bind.hpp:453)
==401653==    by 0x2BF7E2: operator()<int&, boost::value_initialized<cryptonote::NOTIFY_RESPONSE_GET_OBJECTS::request>&, cryptonote::cryptonote_connection_context&> (bind.hpp:1330)
==401653==    by 0x2BF7E2: int epee::net_utils::buff_to_t_adapter<cryptonote::t_cryptonote_protocol_handler<cryptonote::core>, cryptonote::NOTIFY_RESPONSE_GET_OBJECTS::request, cryptonote::cryptonote_connection_context, boost::_bi::bind_t<int, boost::_mfi::mf3<int, cryptonote::t_cryptonote_protocol_handler<cryptonote::core>, int, cryptonote::NOTIFY_RESPONSE_GET_OBJECTS::request&, cryptonote::cryptonote_connection_context&>, boost::_bi::list4<boost::_bi::value<cryptonote::t_cryptonote_protocol_handler<cryptonote::core>*>, boost::arg<1>, boost::arg<2>, boost::arg<3> > > >(cryptonote::t_cryptonote_protocol_handler<cryptonote::core>*, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, boost::_bi::bind_t<int, boost::_mfi::mf3<int, cryptonote::t_cryptonote_protocol_handler<cryptonote::core>, int, cryptonote::NOTIFY_RESPONSE_GET_OBJECTS::request&, cryptonote::cryptonote_connection_context&>, boost::_bi::list4<boost::_bi::value<cryptonote::t_cryptonote_protocol_handler<cryptonote::core>*>, boost::arg<1>, boost::arg<2>, boost::arg<3> > >, cryptonote::cryptonote_connection_context&) (levin_abstract_invoke2.h:216)
==401653==    by 0x2BFCE0: int cryptonote::t_cryptonote_protocol_handler<cryptonote::core>::handle_invoke_map<cryptonote::cryptonote_connection_context>(bool, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, cryptonote::cryptonote_connection_context&, bool&) (cryptonote_protocol_handler.h:89)
==401653==    by 0x2C0262: int nodetool::node_server<cryptonote::t_cryptonote_protocol_handler<cryptonote::core> >::handle_invoke_map<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> >(bool, int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context>&, bool&) (net_node.h:148)
==401653==    by 0x2C0434: nodetool::node_server<cryptonote::t_cryptonote_protocol_handler<cryptonote::core> >::notify(int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context>&) (net_node.h:136)
==401653==    by 0x4E494E7: epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> >::handle_recv(void const*, unsigned long) (levin_protocol_handler_async.h:483)
==401653==    by 0x4E9549A: epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >::handle_read(boost::system::error_code const&, unsigned long) (abstract_tcp_server2.inl:340)
==401653==    by 0x4E812B8: call<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > >, const boost::system::error_code, long unsigned int> (mem_fn_template.hpp:271)
==401653==    by 0x4E812B8: operator()<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > > (mem_fn_template.hpp:286)
==401653==    by 0x4E812B8: operator()<boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::rrlist2<const boost::system::error_code&, long unsigned int const&> > (bind.hpp:398)
==401653==    by 0x4E812B8: operator()<const boost::system::error_code&, long unsigned int const&> (bind.hpp:1318)
==401653==    by 0x4E812B8: operator() (bind_handler.hpp:164)
==401653==    by 0x4E812B8: asio_handler_invoke<boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::system::error_code, long unsigned int> > (handler_invoke_hook.hpp:69)
==401653==    by 0x4E812B8: invoke<boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::system::error_code, long unsigned int>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > > (handler_invoke_helpers.hpp:37)
==401653==    by 0x4E812B8: asio_handler_invoke<boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::system::error_code, long unsigned int>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::system::error_code, long unsigned int> (bind_handler.hpp:207)
==401653==    by 0x4E812B8: invoke<boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::system::error_code, long unsigned int>, boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::system::error_code, long unsigned int> > (handler_invoke_helpers.hpp:37)
==401653==    by 0x4E812B8: void boost::asio::detail::strand_service::dispatch<boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::system::error_code, unsigned long> >(boost::asio::detail::strand_service::strand_impl*&, boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::system::error_code, unsigned long>&) (strand_service.hpp:62)
==401653==    by 0x4E821F0: operator()<boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::system::error_code, long unsigned int> > (io_context_strand.hpp:343)
==401653==    by 0x4E821F0: initiate<boost::asio::io_context::strand::initiate_dispatch, boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::system::error_code, long unsigned int>, boost::asio::io_context::strand*> (async_result.hpp:82)
==401653==    by 0x4E821F0: async_initiate<boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::system::error_code, long unsigned int>, void(), boost::asio::io_context::strand::initiate_dispatch, boost::asio::io_context::strand*> (async_result.hpp:257)
==401653==    by 0x4E821F0: dispatch<boost::asio::detail::binder2<boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::system::error_code, long unsigned int> > (io_context_strand.hpp:189)
==401653==    by 0x4E821F0: operator()<boost::system::error_code, long unsigned int> (wrapped_handler.hpp:99)
==401653==    by 0x4E821F0: operator() (bind_handler.hpp:164)
==401653==    by 0x4E821F0: operator() (wrapped_handler.hpp:191)
==401653==    by 0x4E821F0: asio_handler_invoke<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > > > (handler_invoke_hook.hpp:69)
==401653==    by 0x4E821F0: invoke<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > >, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > > (handler_invoke_helpers.hpp:37)
==401653==    by 0x4E821F0: asio_handler_invoke<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > >, boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > > (wrapped_handler.hpp:275)
==401653==    by 0x4E821F0: invoke<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > >, boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > > > (handler_invoke_helpers.hpp:37)
==401653==    by 0x4E821F0: complete<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > > > (handler_work.hpp:100)
==401653==    by 0x4E821F0: do_complete (completion_handler.hpp:70)
==401653==    by 0x4E821F0: void boost::asio::detail::strand_service::dispatch<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, unsigned long>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > > >(boost::asio::detail::strand_service::strand_impl*&, boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, unsigned long>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > >&) (strand_service.hpp:88)
==401653==    by 0x4E82C38: operator()<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > > > (io_context_strand.hpp:343)
==401653==    by 0x4E82C38: initiate<boost::asio::io_context::strand::initiate_dispatch, boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > >, boost::asio::io_context::strand*> (async_result.hpp:82)
==401653==    by 0x4E82C38: async_initiate<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > >, void(), boost::asio::io_context::strand::initiate_dispatch, boost::asio::io_context::strand*> (async_result.hpp:257)
==401653==    by 0x4E82C38: dispatch<boost::asio::detail::rewrapped_handler<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int>, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> > > > (io_context_strand.hpp:189)
==401653==    by 0x4E82C38: asio_handler_invoke<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int>, boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running> (wrapped_handler.hpp:232)
==401653==    by 0x4E82C38: invoke<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int>, boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running> > (handler_invoke_helpers.hpp:37)
==401653==    by 0x4E82C38: asio_handler_invoke<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int>, boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int> (bind_handler.hpp:207)
==401653==    by 0x4E82C38: invoke<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int>, boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int> > (handler_invoke_helpers.hpp:37)
==401653==    by 0x4E82C38: dispatch<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int>, std::allocator<void> > (io_object_executor.hpp:119)
==401653==    by 0x4E82C38: complete<boost::asio::detail::binder2<boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, const boost::system::error_code&, long unsigned int>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::system::error_code, long unsigned int> > (handler_work.hpp:72)
==401653==    by 0x4E82C38: boost::asio::detail::reactive_socket_recv_op<boost::asio::mutable_buffers_1, boost::asio::detail::wrapped_handler<boost::asio::io_context::strand, boost::_bi::bind_t<void, boost::_mfi::mf2<void, epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > >, boost::system::error_code const&, unsigned long>, boost::_bi::list3<boost::_bi::value<boost::shared_ptr<epee::net_utils::connection<epee::levin::async_protocol_handler<nodetool::p2p_connection_context_t<cryptonote::cryptonote_connection_context> > > > >, boost::arg<1> (*)(), boost::arg<2> (*)()> >, boost::asio::detail::is_continuation_if_running>, boost::asio::detail::io_object_executor<boost::asio::executor> >::do_complete(void*, boost::asio::detail::scheduler_operation*, boost::system::error_code const&, unsigned long) (reactive_socket_recv_op.hpp:123)

@ghost
Copy link
Author

ghost commented Mar 14, 2022

Also Valgrind is upset that you're testing an uninitialized array in get_network_block_database_hash()

@zachhildreth
Copy link
Member

Thanks for more work on this! I can go ahead and add the new find to the testnet and we can test it out in a few days. I will look more into that area as I am not so familiar with that particular section of that file.

For the memset I would think so, thats how dpops was built because of that risk, so I will need to look into that as well.

For the "xcash original" code part yes that makes sense looking at it now, so that would be good to fix as well.

Let keep this going with your findings on this page and I can reference them once we add them to the testnet

@ghost
Copy link
Author

ghost commented Mar 14, 2022 •
edited by ghost
Loading

There's also an issue on exit which I experienced during memory testing. It seems that a mutex variable is getting destroyed at daemon shutdown, followed by an attempt to unlock using that mutex. The result's a thrown boost::lock_error exception from mlocker::unlock() call.

We're exiting the daemon at this point, so it's not a big deal, but it's causing an abort prior to clean-up which is messing around with the memory checkers.

I hacked my way around it by throwing away the exception with a try/catch wrapper (below) but that's hardly a valid fix.

  void mlocker::unlock(void *ptr, size_t len)
  {
    try {
      size_t page_size = get_page_size();
      if (page_size == 0)
        return;
      CRITICAL_REGION_LOCAL(mutex());
      const size_t first = ((uintptr_t)ptr) / page_size;
      const size_t last = (((uintptr_t)ptr) + len - 1) / page_size;
      for (size_t page = first; page <= last; ++page)
        unlock_page(page);
      --num_locked_objects;
    } catch (const boost::lock_error& e) { // Something's gone wrong - return without activity
      std::cerr << "\nFailed Unlock!\n";
      return;
    }
  }

Mostly just making note of it here as an item to track down how to properly address the use of the mlocker after it's been destroyed.

Valgrind output follows:

==331274== 240 bytes in 1 blocks are possibly lost in loss record 458 of 852
==331274==    at 0x483B7F3: malloc (in /usr/lib/x86_64-linux-gnu/valgrind/vgpreload_memcheck-amd64-linux.so)
==331274==    by 0x5B131F3: __cxa_allocate_exception (in /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.28)
==331274==    by 0x1F36D6: void boost::throw_exception<boost::lock_error>(boost::lock_error const&) (throw_exception.hpp:70)
==331274==    by 0x48AA321: lock (mutex.hpp:64)
==331274==    by 0x48AA321: critical_region_t (syncobj.h:134)
==331274==    by 0x48AA321: epee::mlocker::get_page_size() (mlocker.cpp:98)
==331274==    by 0x48AA3F0: epee::mlocker::unlock(void*, unsigned long) (mlocker.cpp:130)
==331274==    by 0x557ED33: epee::mlocked<tools::scrubbed<crypto::ec_scalar> >::~mlocked() (mlocker.h:76)
==331274==    by 0x5E2200D: __cxa_finalize (cxa_finalize.c:83)
==331274==    by 0x556FB56: ??? (in /home/me/workspace/xcash/testnet/xcash-core/build/debug/src/crypto/libcncrypto.so)
==331274==    by 0x4011F5A: _dl_fini (dl-fini.c:138)
==331274==    by 0x5E218D6: __run_exit_handlers (exit.c:108)
==331274==    by 0x5E21A8F: exit (exit.c:139)
==331274==    by 0x5DFF0B9: (below main) (libc-start.c:342)

Oh, and also there's another uninitialized variable in wallet2.cpp height

        uint64_t height;
        if (!string_tools::get_xtype_from_string(height, fields[1]))
          continue;

        MINFO("Found segregation height via DNS: " << fields[0] << " fork height at " << height);
        uint64_t diff = height > current_height ? height - current_height : current_height - height;

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@zachhildreth