Crash multithreaded server #175

firstpilotpirks · 2017-12-12T13:13:32Z

Hi,
I am using rpclib for microservices of system. I made multi-thread client and server and found bug.

I have error:

terminate called after throwing an instance of 'std::system_error'
  what():  close: Bad file descriptor
[1]    27051 abort      ./server

or

./server
[1]    22191 segmentation fault  ./server

I created multi-thread server and multi-thread client.

Server

#include <iostream>
#include "rpc/server.h"

int main(int argc, char *argv[]) 
{
    rpc::server srv(8088);
    srv.bind("add", [](int a, int b)
    {
        std::cout << "add" << std::endl;
        return a + b;
    });

    srv.suppress_exceptions(true);
    srv.async_run(8);
    std::cin.ignore();

    return 0;
}

Client

#include <thread>
#include "rpc/client.h"

int main() 
{
    while(true)
    {
        std::vector<std::thread> thread_pool(10);
        for (auto & client_thread : thread_pool)
        {
            client_thread = std::thread([]()
                                        {
                                            rpc::client client("127.0.0.1", 8088);
                                            int result;
                                            result = client.call("add", 2, 3).as<int>();
                                        });
        }

        for (auto & client_thread : thread_pool)
        {
            if (client_thread.joinable())
            {
                client_thread.join();
            }
        }
    }

    return 0;
}

Build
CXX_FLAGS "-std=c++14 -lpthread -O2

(gdb) backtrace in policy

#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007ffff6fa519a in __GI_abort () at abort.c:89
#2  0x00007ffff78ceb85 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007ffff78cc956 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007ffff78cc9a1 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007ffff78f775e in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007ffff7bc3519 in start_thread (arg=0x7ffff5f6d700) at pthread_create.c:456
#7  0x00007ffff705ca5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

or

(gdb) backtrace in master

terminate called after throwing an instance of 'std::bad_weak_ptr'
  what():  bad_weak_ptr

Thread 4 "server" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff5f6d700 (LWP 15501)]
0x00005555555db50e in clmdep_asio::detail::epoll_reactor::deregister_descriptor(int, clmdep_asio::detail::epoll_reactor::descriptor_state*&, bool) ()
(gdb) backtrace 
#0  0x00005555555db50e in clmdep_asio::detail::epoll_reactor::deregister_descriptor(int, clmdep_asio::detail::epoll_reactor::descriptor_state*&, bool) ()
#1  0x00005555555ecc32 in clmdep_asio::detail::reactive_socket_service_base::close(clmdep_asio::detail::reactive_socket_service_base::base_implementation_type&, std::error_code&) ()
#2  0x00005555555ef788 in clmdep_asio::stream_socket_service<clmdep_asio::ip::tcp>::close(clmdep_asio::detail::reactive_socket_service<clmdep_asio::ip::tcp>::implementation_type&, std::error_code&) ()
#3  0x00005555555eece7 in clmdep_asio::basic_socket<clmdep_asio::ip::tcp, clmdep_asio::stream_socket_service<clmdep_asio::ip::tcp> >::close() ()
#4  0x00005555555e8272 in rpc::detail::server_session::close()::{lambda()#1}::operator()() const ()
#5  0x00005555555eab74 in void clmdep_asio::clmdep_asio_handler_invoke<rpc::detail::server_session::close()::{lambda()#1}>(rpc::detail::server_session::close()::{lambda()#1}&, ...) ()
#6  0x00005555555ea6ef in void clmdep_asio_handler_invoke_helpers::invoke<rpc::detail::server_session::close()::{lambda()#1}, {lambda()#1}>(rpc::detail::server_session::close()::{lambda()#1}&, {lambda()#1}&) ()
#7  0x00005555555e9e9b in clmdep_asio::detail::completion_handler<rpc::detail::server_session::close()::{lambda()#1}>::do_complete(clmdep_asio::detail::task_io_service*, clmdep_asio::detail::task_io_service_operation*, std::error_code const&, unsigned long) ()
#8  0x00005555555da43e in clmdep_asio::detail::task_io_service_operation::complete(clmdep_asio::detail::task_io_service&, std::error_code const&, unsigned long) ()
#9  0x00005555555ed4a5 in clmdep_asio::detail::strand_service::do_complete(clmdep_asio::detail::task_io_service*, clmdep_asio::detail::task_io_service_operation*, std::error_code const&, unsigned long) ()
#10 0x00005555555da43e in clmdep_asio::detail::task_io_service_operation::complete(clmdep_asio::detail::task_io_service&, std::error_code const&, unsigned long) ()
#11 0x00005555555dca4c in clmdep_asio::detail::task_io_service::do_run_one(clmdep_asio::detail::scoped_lock<clmdep_asio::detail::posix_mutex>&, clmdep_asio::detail::task_io_service_thread_info&, std::error_code const&) ()
#12 0x00005555555dc57e in clmdep_asio::detail::task_io_service::run(std::error_code&) ()
#13 0x00005555555dcce9 in clmdep_asio::io_service::run() ()
#14 0x00005555555d8697 in rpc::server::async_run(unsigned long)::{lambda()#1}::operator()() const ()
#15 0x00005555555d894b in std::_Function_handler<void (), rpc::server::async_run(unsigned long)::{lambda()#1}>::_M_invoke(std::_Any_data const&) ()
#16 0x00005555555e2a94 in std::function<void ()>::operator()() const ()
#17 0x00005555555e15da in void std::__invoke_impl<void, std::function<void ()>>(std::__invoke_other, std::function<void ()>&&) ()
#18 0x00005555555e0300 in std::__invoke_result<std::function<void ()>>::type std::__invoke<std::function<void ()>>(std::function<void ()>&&) ()
#19 0x00005555555e7e92 in decltype (__invoke((_S_declval<0ul>)())) std::thread::_Invoker<std::tuple<std::function<void ()> > >::_M_invoke<0ul>(std::_Index_tuple<0ul>) ()
#20 0x00005555555e7ceb in std::thread::_Invoker<std::tuple<std::function<void ()> > >::operator()() ()
#21 0x00005555555e7c0a in std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::function<void ()> > > >::_M_run() ()
#22 0x00007ffff78f772f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#23 0x00007ffff7bc3519 in start_thread (arg=0x7ffff5f6d700) at pthread_create.c:456
#24 0x00007ffff705ca5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

I tryed branches:

master
dev
policy
2.2.1

All have this bug.

Can you please tell me if there is any fix or workaround for this issue?

Thanks,
Alex.

sztomi · 2017-12-12T14:16:09Z

Sounds like server_session::close() needs to be more defensive about the current socket state. The reason it wasn't before is that I feel like checking the current state can only do so much when it can change asynchronously. I'll try to implement a reliable fix for this.

That said, can you describe what the use case is for running multiple threads like this? Wouldn't async_call be sufficient?

firstpilotpirks · 2017-12-12T15:01:35Z

I have several services on linux. One service is one program, and I have several physical mashine. I would like to use several programming language togethe (C++ with Python, maybe with Go ...),
and message-pack rpc is a good library for this. I would like to have about 30 services. I have a situation when there are more than 10 calls in one moment and I have segmentation fault. async_call
is good idea for multiple calls in one program but I have several programs.

Thanks for your reply.

sztomi · 2017-12-12T15:53:22Z

OK, so the client you listed here is an artificial example that can trigger the bug.

Until this gets fixed, I recommend structuring your clients so that they don't disconnect so frequently (because in your example they do). I'm curious if that's what happens in your real programs as well (i.e. do you get the same stack trace?)

firstpilotpirks · 2017-12-13T09:02:27Z

We are starting project and we are choosing a system communication of programs. We have two applicants for this role. (We used D-Bus in previous project, but it not work for network). We took Rpclib and Apache Thrift. We are making tests for these libraries now.

You are right. We will have one connection for several call for production but we are must making tests for all pismimetic scenarios.

About stack trace. I got three different trace after fall.

1

Thread 3 "server" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7ffff676e700 (LWP 1638)]
0x00005555555dfade in clmdep_asio::detail::epoll_reactor::deregister_descriptor(int, clmdep_asio::detail::epoll_reactor::descriptor_state*&, bool) ()
(gdb) backtrace 
#0  0x00005555555dfade in clmdep_asio::detail::epoll_reactor::deregister_descriptor(int, clmdep_asio::detail::epoll_reactor::descriptor_state*&, bool) ()
#1  0x00005555555e202f in clmdep_asio::detail::reactive_socket_service_base::destroy(clmdep_asio::detail::reactive_socket_service_base::base_implementation_type&) ()
#2  0x00005555555e6d5f in clmdep_asio::stream_socket_service<clmdep_asio::ip::tcp>::destroy(clmdep_asio::detail::reactive_socket_service<clmdep_asio::ip::tcp>::implementation_type&) ()
#3  0x00005555555e58bf in clmdep_asio::basic_io_object<clmdep_asio::stream_socket_service<clmdep_asio::ip::tcp>, true>::~basic_io_object() ()
#4  0x00005555555e47c8 in clmdep_asio::basic_socket<clmdep_asio::ip::tcp, clmdep_asio::stream_socket_service<clmdep_asio::ip::tcp> >::~basic_socket() ()
#5  0x00005555555e2e76 in clmdep_asio::basic_stream_socket<clmdep_asio::ip::tcp, clmdep_asio::stream_socket_service<clmdep_asio::ip::tcp> >::~basic_stream_socket() ()
#6  0x00005555555ebf62 in rpc::detail::async_writer::~async_writer() ()
#7  0x00005555555ec010 in rpc::detail::server_session::~server_session() ()
#8  0x00005555555ec030 in void __gnu_cxx::new_allocator<rpc::detail::server_session>::destroy<rpc::detail::server_session>(rpc::detail::server_session*) ()
#9  0x00005555555ebe95 in void std::allocator_traits<std::allocator<rpc::detail::server_session> >::destroy<rpc::detail::server_session>(std::allocator<rpc::detail::server_session>&, rpc::detail::server_session*) ()
#10 0x00005555555ebd07 in std::_Sp_counted_ptr_inplace<rpc::detail::server_session, std::allocator<rpc::detail::server_session>, (__gnu_cxx::_Lock_policy)2>::_M_dispose() ()
#11 0x00005555555d4dc6 in std::_Sp_counted_base<(__gnu_cxx::_Lock_policy)2>::_M_release() ()
#12 0x00005555555d3e5d in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::~__shared_count() ()
#13 0x00005555555e3334 in std::__shared_ptr<rpc::detail::server_session, (__gnu_cxx::_Lock_policy)2>::~__shared_ptr()
    ()
#14 0x00005555555e3350 in std::shared_ptr<rpc::detail::server_session>::~shared_ptr() ()
#15 0x00005555555ecbce in rpc::detail::server_session::do_read()::{lambda(std::error_code, unsigned long)#1}::~error_code() ()
#16 0x00005555555eefb8 in clmdep_asio::detail::rewrapped_handler<clmdep_asio::detail::binder2<clmdep_asio::detail::wrapped_handler<clmdep_asio::io_service::strand, rpc::detail::server_session::do_read()::{lambda(std::error_code, unsigned long)#1}, clmdep_asio::detail::is_continuation_if_running>, std::error_code, unsigned long>, {lambda(std::error_code, unsigned long)#1}>::~rewrapped_handler() ()
#17 0x00005555555ef680 in clmdep_asio::detail::completion_handler<clmdep_asio::detail::rewrapped_handler<clmdep_asio::detail::binder2<clmdep_asio::detail::wrapped_handler<clmdep_asio::io_service::strand, rpc::detail::server_session::do_read()::{lambda(std::error_code, unsigned long)#1}, clmdep_asio::detail::is_continuation_if_running>, std::error_code, unsigned long>, {lambda(std::error_code, unsigned long)#1}> >::do_complete(clmdep_asio::detail::task_io_service*, clmdep_asio::detail::task_io_service_operation*, std::error_code const&, unsigned long) ()
#18 0x00005555555ef3e9 in void clmdep_asio::detail::strand_service::dispatch<clmdep_asio::detail::rewrapped_handler<clmdep_asio::detail::binder2<clmdep_asio::detail::wrapped_handler<clmdep_asio::io_service::strand, rpc::detail::server_session::do_read()::{lambda(std::error_code, unsigned long)#1}, clmdep_asio::detail::is_continuation_if_running>, std::error_code, unsigned long>, {lambda(std::error_code, unsigned long)#1}> >(clmdep_asio::detail::strand_service::strand_impl*&, clmdep_asio::detail::rewrapped_handler<clmdep_asio::detail::binder2<clmdep_asio::detail::wrapped_handler<clmdep_asio::io_service::strand, rpc::detail::server_session::do_read()::{lambda(std::error_code, unsigned long)#1}, clmdep_asio::detail::is_continuation_if_running>, std::error_code, unsigned long>, {lambda(std::error_code, unsigned long)#1}>&) ()
#19 0x00005555555ef1d6 in clmdep_asio::async_result<clmdep_asio::handler_type<clmdep_asio::detail::rewrapped_handler<clmdep_asio::detail::binder2<clmdep_asio::detail::wrapped_handler<clmdep_asio::io_service::strand, rpc::detail::server_session::do_read()::{lambda(std::error_code, unsigned long)#1}, clmdep_asio::detail::is_continuation_if_running>, std:---Type <return> to continue, or q <return> to quit---
:error_code, unsigned long>, {lambda(std::error_code, unsigned long)#1}>, void ()>::type>::type clmdep_asio::io_service::strand::dispatch<clmdep_asio::detail::rewrapped_handler<clmdep_asio::detail::binder2<clmdep_asio::detail::wrapped_handler<clmdep_asio::io_service::strand, rpc::detail::server_session::do_read()::{lambda(std::error_code, unsigned long)#1}, clmdep_asio::detail::is_continuation_if_running>, std::error_code, unsigned long>, {lambda(std::error_code, unsigned long)#1}> >(clmdep_asio::handler_type&&) ()
#20 0x00005555555ef00c in void clmdep_asio::detail::clmdep_asio_handler_invoke<clmdep_asio::detail::binder2<clmdep_asio::detail::wrapped_handler<clmdep_asio::io_service::strand, rpc::detail::server_session::do_read()::{lambda(std::error_code, unsigned long)#1}, clmdep_asio::detail::is_continuation_if_running>, std::error_code, unsigned long>, clmdep_asio::io_service::strand, {lambda(std::error_code, unsigned long)#1}, rpc::detail::server_session::do_read()::{lambda(std::error_code, unsigned long)#1}>(clmdep_asio::detail::binder2<clmdep_asio::detail::wrapped_handler<clmdep_asio::io_service::strand, rpc::detail::server_session::do_read()::{lambda(std::error_code, unsigned long)#1}, clmdep_asio::detail::is_continuation_if_running>, std::error_code, unsigned long>&, clmdep_asio::detail::wrapped_handler<clmdep_asio::io_service::strand, {lambda(std::error_code, unsigned long)#1}, rpc::detail::server_session::do_read()::{lambda(std::error_code, unsigned long)#1}>*) ()
#21 0x00005555555eef40 in void clmdep_asio_handler_invoke_helpers::invoke<clmdep_asio::detail::binder2<clmdep_asio::detail::wrapped_handler<clmdep_asio::io_service::strand, rpc::detail::server_session::do_read()::{lambda(std::error_code, unsigned long)#1}, clmdep_asio::detail::is_continuation_if_running>, std::error_code, unsigned long>, clmdep_asio::detail::is_continuation_if_running>(clmdep_asio::detail::binder2<clmdep_asio::detail::wrapped_handler<clmdep_asio::io_service::strand, rpc::detail::server_session::do_read()::{lambda(std::error_code, unsigned long)#1}, clmdep_asio::detail::is_continuation_if_running>, std::error_code, unsigned long>&, clmdep_asio::detail::is_continuation_if_running&)
    ()
#22 0x00005555555eeba9 in clmdep_asio::detail::reactive_socket_recv_op<clmdep_asio::mutable_buffers_1, clmdep_asio::detail::wrapped_handler<clmdep_asio::io_service::strand, rpc::detail::server_session::do_read()::{lambda(std::error_code, unsigned long)#1}, clmdep_asio::detail::is_continuation_if_running> >::do_complete(clmdep_asio::detail::task_io_service*, clmdep_asio::detail::task_io_service_operation*, std::error_code const&, unsigned long) ()
#23 0x00005555555dea0e in clmdep_asio::detail::task_io_service_operation::complete(clmdep_asio::detail::task_io_service&, std::error_code const&, unsigned long) ()
#24 0x00005555555e0571 in clmdep_asio::detail::epoll_reactor::descriptor_state::do_complete(clmdep_asio::detail::task_io_service*, clmdep_asio::detail::task_io_service_operation*, std::error_code const&, unsigned long) ()
#25 0x00005555555dea0e in clmdep_asio::detail::task_io_service_operation::complete(clmdep_asio::detail::task_io_service&, std::error_code const&, unsigned long) ()
#26 0x00005555555e101c in clmdep_asio::detail::task_io_service::do_run_one(clmdep_asio::detail::scoped_lock<clmdep_asio::detail::posix_mutex>&, clmdep_asio::detail::task_io_service_thread_info&, std::error_code const&) ()
#27 0x00005555555e0b4e in clmdep_asio::detail::task_io_service::run(std::error_code&) ()
#28 0x00005555555e12b9 in clmdep_asio::io_service::run() ()
#29 0x00005555555dcd03 in rpc::server::async_run(unsigned long)::{lambda()#1}::operator()() const ()
#30 0x00005555555dcfb7 in std::_Function_handler<void (), rpc::server::async_run(unsigned long)::{lambda()#1}>::_M_invoke(std::_Any_data const&) ()
#31 0x00005555555e70ba in std::function<void ()>::operator()() const ()
#32 0x00005555555e5c00 in void std::__invoke_impl<void, std::function<void ()>>(std::__invoke_other, std::function<void ()>&&) ()
#33 0x00005555555e48fc in std::__invoke_result<std::function<void ()>>::type std::__invoke<std::function<void ()>>(std::function<void ()>&&) ()
#34 0x00005555555ec05c in decltype (__invoke((_S_declval<0ul>)())) std::thread::_Invoker<std::tuple<std::function<void ()> > >::_M_invoke<0ul>(std::_Index_tuple<0ul>) ()
#35 0x00005555555ebeb5 in std::thread::_Invoker<std::tuple<std::function<void ()> > >::operator()() ()
#36 0x00005555555ebdd4 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::function<void ()> > > >::_M_r---Type <return> to continue, or q <return> to quit---
un() ()
#37 0x00007ffff78f772f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#38 0x00007ffff7bc3519 in start_thread (arg=0x7ffff676e700) at pthread_create.c:456
#39 0x00007ffff705ca5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

2

Thread 37 "server" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffe574c700 (LWP 11875)]
0x00005555555dfade in clmdep_asio::detail::epoll_reactor::deregister_descriptor(int, clmdep_asio::detail::epoll_reactor::descriptor_state*&, bool) ()
(gdb) backtrace 
#0  0x00005555555dfade in clmdep_asio::detail::epoll_reactor::deregister_descriptor(int, clmdep_asio::detail::epoll_reactor::descriptor_state*&, bool) ()
#1  0x00005555555f0d3c in clmdep_asio::detail::reactive_socket_service_base::close(clmdep_asio::detail::reactive_socket_service_base::base_implementation_type&, std::error_code&) ()
#2  0x00005555555f388c in clmdep_asio::stream_socket_service<clmdep_asio::ip::tcp>::close(clmdep_asio::detail::reactive_socket_service<clmdep_asio::ip::tcp>::implementation_type&, std::error_code&) ()
#3  0x00005555555f2e0f in clmdep_asio::basic_socket<clmdep_asio::ip::tcp, clmdep_asio::stream_socket_service<clmdep_asio::ip::tcp> >::close() ()
#4  0x00005555555ec43c in rpc::detail::server_session::close()::{lambda()#1}::operator()() const ()
#5  0x00005555555eec7e in void clmdep_asio::clmdep_asio_handler_invoke<rpc::detail::server_session::close()::{lambda()#1}>(rpc::detail::server_session::close()::{lambda()#1}&, ...) ()
#6  0x00005555555ee7f9 in void clmdep_asio_handler_invoke_helpers::invoke<rpc::detail::server_session::close()::{lambda()#1}, {lambda()#1}>(rpc::detail::server_session::close()::{lambda()#1}&, {lambda()#1}&) ()
#7  0x00005555555edfc3 in clmdep_asio::detail::completion_handler<rpc::detail::server_session::close()::{lambda()#1}>::do_complete(clmdep_asio::detail::task_io_service*, clmdep_asio::detail::task_io_service_operation*, std::error_code const&, unsigned long) ()
#8  0x00005555555dea0e in clmdep_asio::detail::task_io_service_operation::complete(clmdep_asio::detail::task_io_service&, std::error_code const&, unsigned long) ()
#9  0x00005555555f15af in clmdep_asio::detail::strand_service::do_complete(clmdep_asio::detail::task_io_service*, clmdep_asio::detail::task_io_service_operation*, std::error_code const&, unsigned long) ()
#10 0x00005555555dea0e in clmdep_asio::detail::task_io_service_operation::complete(clmdep_asio::detail::task_io_service&, std::error_code const&, unsigned long) ()
#11 0x00005555555e101c in clmdep_asio::detail::task_io_service::do_run_one(clmdep_asio::detail::scoped_lock<clmdep_asio::detail::posix_mutex>&, clmdep_asio::detail::task_io_service_thread_info&, std::error_code const&) ()
#12 0x00005555555e0b4e in clmdep_asio::detail::task_io_service::run(std::error_code&) ()
#13 0x00005555555e12b9 in clmdep_asio::io_service::run() ()
#14 0x00005555555dcd03 in rpc::server::async_run(unsigned long)::{lambda()#1}::operator()() const ()
#15 0x00005555555dcfb7 in std::_Function_handler<void (), rpc::server::async_run(unsigned long)::{lambda()#1}>::_M_invoke(std::_Any_data const&) ()
#16 0x00005555555e70ba in std::function<void ()>::operator()() const ()
#17 0x00005555555e5c00 in void std::__invoke_impl<void, std::function<void ()>>(std::__invoke_other, std::function<void ()>&&) ()
#18 0x00005555555e48fc in std::__invoke_result<std::function<void ()>>::type std::__invoke<std::function<void ()>>(std::function<void ()>&&) ()
#19 0x00005555555ec05c in decltype (__invoke((_S_declval<0ul>)())) std::thread::_Invoker<std::tuple<std::function<void ()> > >::_M_invoke<0ul>(std::_Index_tuple<0ul>) ()
#20 0x00005555555ebeb5 in std::thread::_Invoker<std::tuple<std::function<void ()> > >::operator()() ()
#21 0x00005555555ebdd4 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<std::function<void ()> > > >::_M_run() ()
#22 0x00007ffff78f772f in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#23 0x00007ffff7bc3519 in start_thread (arg=0x7fffe574c700) at pthread_create.c:456
#24 0x00007ffff705ca5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

3

Thread 12 "server" received signal SIGABRT, Aborted.
[Switching to Thread 0x7ffff1f65700 (LWP 8502)]
__GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
51	../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
(gdb) backtrace 
#0  __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:51
#1  0x00007ffff6fa519a in __GI_abort () at abort.c:89
#2  0x00007ffff78ceb85 in __gnu_cxx::__verbose_terminate_handler() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3  0x00007ffff78cc956 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4  0x00007ffff78cc9a1 in std::terminate() () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5  0x00007ffff78f775e in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6  0x00007ffff7bc3519 in start_thread (arg=0x7ffff1f65700) at pthread_create.c:456
#7  0x00007ffff705ca5f in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:97

sztomi · 2017-12-13T09:15:36Z

It looks like the 1st one is different - I would be interested in a test program that can trigger this.

The 2nd one is what you originally reported.

The 3rd one just looks like the stack trace of the main thread, not where the actual crash happened.

firstpilotpirks · 2017-12-14T17:13:08Z

The test program that can trigger this it is a very simple program. I wrote this program in the first message (Server and Client). I can give you test program with other scenarios. Please tell me if you want test program with cmake file.

sztomi · 2017-12-17T10:50:07Z

@STPR if you have a cmake file, I'll take it, but I can write one myself if not.

firstpilotpirks · 2017-12-18T07:45:30Z

I mean, what it is one case from my test.
Can I add a new test case to your tests in to "rpclib/tests/rpc" ? I can do it in a few days.
If you don't mind. Where can I do "pull request"? I mean what branch(master, dev, experimental) ?

Sory for my bad English.

cmake file for the previous example:
Client

cmake_minimum_required(VERSION 2.8.9) 
project (rpc) 
 
add_executable(client client.cpp) 
 
set(CMAKE_CXX_FLAGS "-std=c++14 -lpthread -O2") 
 
include_directories("rpclib/include") 
LINK_DIRECTORIES("rpclib")
TARGET_LINK_LIBRARIES(client "rpclib/librpc.a")

Server

cmake_minimum_required(VERSION 2.8.9)
project (rpc)

add_executable(server server.cpp)

set(CMAKE_CXX_FLAGS "-std=c++14 -lpthread -O0 -g")

include_directories("rpclib/include") 
LINK_DIRECTORIES("rpclib")
TARGET_LINK_LIBRARIES(client "rpclib/librpc.a")

sztomi · 2017-12-18T10:47:31Z

Yes, go ahead, and thanks! Pull requests are preferred against the dev branch (unless the bug is only reproducible on some other branch but that's not the case here).

yunfengzh · 2017-12-20T05:37:16Z

I also encounter all cases, in my project, 8 clients send request to server (2 thread). branch is 2.2.1, ubuntu16.04.

And the most case is system::error (bad file descriptor).
That is, it seems server_session::async_writer::socket_ is closed twice.
(gdb) p *this $3 = {<clmdep_asio::basic_io_object<clmdep_asio::stream_socket_service<clmdep_asio::ip::tcp>, true>> = { implementation = {<clmdep_asio::detail::reactive_socket_service_base::base_implementation_type> = {socket_ = -1, state_ = 0 '\000', reactor_data_ = 0x0}, protocol_ = { family_ = 2}}, service_ = 0x7ae4f0}, <clmdep_asio::socket_base> = { static message_peek = 2, static message_out_of_band = 1, static message_do_not_route = 4, static message_end_of_record = 128, static max_connections = 128}, <No data fields>}

implementation.socket_ = -1, and implementation.reactor_data_ = 0.
When implementation.reactor_data_ = 0, SIGSEGV case is found.

In lib/rpc/detail/server_session.cc, I doubts
server_session::server_session(server *srv, RPCLIB_ASIO::io_service *io, RPCLIB_ASIO::ip::tcp::socket socket, std::shared_ptr<dispatcher> disp, bool suppress_exceptions) : async_writer(io, std::move(socket)),
std::move(socket) should be protected.

I also found clmdep_asio has been accpeted by boost project, so I hope you add testcases to make rpclib working with boost library.

sytelus · 2018-05-18T01:47:43Z

Just wondering about the state of this issue. We had a lot of users requesting this mode as well. If this is not fixed yet, is async_call a good workaround? Any major disadvantages with async_call vs. above approach?


        Added test for issue rpclib#175


        Added test for issue rpclib#175


        Added test for issue rpclib#175

firstpilotpirks changed the title ~~Crash multithread server~~ Crash multithreaded server Dec 12, 2017

sztomi added the bug label Dec 12, 2017

qchateau added a commit to qchateau/rpclib that referenced this issue Dec 11, 2018

Added test for issue rpclib#175

a0e6123

qchateau mentioned this issue Dec 11, 2018

Resolve multithread crash (#175) #225

Open

qchateau added a commit to qchateau/rpclib that referenced this issue Dec 11, 2018

Added test for issue rpclib#175

9742f8e

qchateau added a commit to qchateau/rpclib that referenced this issue Mar 19, 2019

Added test for issue rpclib#175

808fdd1

songxiaocheng mentioned this issue Dec 10, 2019

Fix Unreal crash by changing rpclib to a fork version microsoft/AirSim#2340

Merged

rpclib / rpclib

Crash multithreaded server #175

Crash multithreaded server #175

firstpilotpirks commented Dec 12, 2017

sztomi commented Dec 12, 2017

firstpilotpirks commented Dec 12, 2017

sztomi commented Dec 12, 2017

firstpilotpirks commented Dec 13, 2017

sztomi commented Dec 13, 2017

firstpilotpirks commented Dec 14, 2017

sztomi commented Dec 17, 2017

firstpilotpirks commented Dec 18, 2017

sztomi commented Dec 18, 2017

yunfengzh commented Dec 20, 2017 •

edited

sytelus commented May 18, 2018

rpclib / rpclib

Join GitHub today

Crash multithreaded server #175

Crash multithreaded server #175

Comments

firstpilotpirks commented Dec 12, 2017

sztomi commented Dec 12, 2017

firstpilotpirks commented Dec 12, 2017

sztomi commented Dec 12, 2017

firstpilotpirks commented Dec 13, 2017

sztomi commented Dec 13, 2017

firstpilotpirks commented Dec 14, 2017

sztomi commented Dec 17, 2017

firstpilotpirks commented Dec 18, 2017

sztomi commented Dec 18, 2017

yunfengzh commented Dec 20, 2017 • edited

sytelus commented May 18, 2018

yunfengzh commented Dec 20, 2017 •

edited