From: Ilya Maximets <i.maximets@samsung.com>
To: Yuanhan Liu <yuanhan.liu@linux.intel.com>
Cc: dev@dpdk.org, Huawei Xie <huawei.xie@intel.com>,
Dyasly Sergey <s.dyasly@samsung.com>,
Heetae Ahn <heetae82.ahn@samsung.com>,
Thomas Monjalon <thomas.monjalon@6wind.com>
Subject: Re: [dpdk-dev] [PATCH] vhost: fix connect hang in client mode
Date: Thu, 21 Jul 2016 14:14:59 +0300 [thread overview]
Message-ID: <5790AEB3.2010708@samsung.com> (raw)
In-Reply-To: <5790A5D4.1090703@samsung.com>
On 21.07.2016 13:37, Ilya Maximets wrote:
>
>
> On 21.07.2016 13:13, Yuanhan Liu wrote:
>> On Thu, Jul 21, 2016 at 12:45:32PM +0300, Ilya Maximets wrote:
>>> On 21.07.2016 12:37, Yuanhan Liu wrote:
>>>> On Thu, Jul 21, 2016 at 11:21:15AM +0300, Ilya Maximets wrote:
>>>>> If something abnormal happened to QEMU, 'connect()' can block calling
>>>>> thread (e.g. main thread of OVS) forever or for a really long time.
>>>>> This can break whole application or block the reconnection thread.
>>>>>
>>>>> Example with OVS:
>>>>>
>>>>> ovs_rcu(urcu2)|WARN|blocked 512000 ms waiting for main to quiesce
>>>>> (gdb) bt
>>>>> #0 connect () from /lib64/libpthread.so.0
>>>>> #1 vhost_user_create_client (vsocket=0xa816e0)
>>>>> #2 rte_vhost_driver_register
>>>>> #3 netdev_dpdk_vhost_user_construct
>>>>> #4 netdev_open (name=0xa664b0 "vhost1")
>>>>> [...]
>>>>> #11 main
>>>>>
>>>>> Fix that by setting non-blocking mode for client sockets for connection.
>>>>>
>>>>> Fixes: 64ab701c3d1e ("vhost: add vhost-user client mode")
>>>>
>>>> Thanks for spotting and fixing yet another bug!
>>>>
>>>>>
>>>>> +static int
>>>>> +vhost_user_connect_nonblock(int fd, struct sockaddr *un, size_t sz)
>>>>
>>>> I don't quite understand why this is needed: connect() with O_NONBLOCK
>>>> flag set is not enough?
>>>
>>> There is a little issue with non-blocking connect() call. Connection
>>> establishing may be started but '-1' returned with 'errno = EINPROGRESS'.
>>> In this case we must wait on fd until it will be available for writing.
>>> After that we need to check current status of connection using getsockopt().
>>>
>>> I don't sure that we're able to get such situation, but it's documented,
>>> and, I think, we should handle it.
>>>
>>> See 'man connect' for details.
>>
>> I see. Thanks.
>>
>> But basically, I don't like the way of introduing yet another
>> fdset here. I'm wondering we could leverage current fdset code
>> to achieve that. This might need some work though.
>>
>> So how about making it simple and stupid at this stage: sleep a
>> while (maybe 1ms, or maybe 1s) when that happens, and give up
>> when the connection is still not established?
>
> Hmm, how about this fixup:
> ------------------------------------------------------------------------------
> diff --git a/lib/librte_vhost/vhost_user/vhost-net-user.c b/lib/librte_vhost/vhost_user/vhost-net-user.c
> index 8626d13..b0f45e6 100644
> --- a/lib/librte_vhost/vhost_user/vhost-net-user.c
> +++ b/lib/librte_vhost/vhost_user/vhost-net-user.c
> @@ -537,18 +537,7 @@ vhost_user_connect_nonblock(int fd, struct sockaddr *un, size_t sz)
> errno = EINVAL;
>
> ret = connect(fd, un, sz);
> - if (ret == -1 && errno != EINPROGRESS)
> - return -1;
> - if (ret == 0)
> - goto connected;
> -
> - FD_ZERO(&fdset);
> - FD_SET(fd, &fdset);
> -
> - ret = select(fd + 1, NULL, &fdset, NULL, &tv);
> - if (!ret)
> - errno = ETIMEDOUT;
> - if (ret != 1)
> + if (ret < 0 && errno != EISCONN)
> return -1;
>
> ret = getsockopt(fd, SOL_SOCKET, SO_ERROR, &so_error, &len);
> @@ -558,7 +547,6 @@ vhost_user_connect_nonblock(int fd, struct sockaddr *un, size_t sz)
> return -1;
> }
>
> -connected:
> flags = fcntl(fd, F_GETFL, 0);
> if (flags < 0) {
> RTE_LOG(ERR, VHOST_CONFIG,
> ------------------------------------------------------------------------------
> ?
>
> We will not check the EINPROGRESS, but subsequent 'connect()' will return
> EISCONN if connection already established. getsockopt() is kept just in
> case. Subsequent 'connect()' will happen on the next iteration of
> reconnection cycle (1 second sleep).
I've sent v2 with this changes.
Best regards, Ilya Maximets.
next prev parent reply other threads:[~2016-07-21 11:15 UTC|newest]
Thread overview: 20+ messages / expand[flat|nested] mbox.gz Atom feed top
2016-07-21 8:21 Ilya Maximets
2016-07-21 9:37 ` Yuanhan Liu
2016-07-21 9:45 ` Ilya Maximets
2016-07-21 10:13 ` Yuanhan Liu
2016-07-21 10:37 ` Ilya Maximets
2016-07-21 11:14 ` Ilya Maximets [this message]
2016-07-21 11:40 ` Yuanhan Liu
2016-07-21 12:10 ` Ilya Maximets
2016-07-21 12:13 ` Ilya Maximets
2016-07-21 12:35 ` Yuanhan Liu
2016-07-21 12:42 ` Ilya Maximets
2016-07-21 12:58 ` Yuanhan Liu
2016-07-21 12:58 ` Ilya Maximets
2016-07-21 13:10 ` Yuanhan Liu
2016-07-21 11:12 ` [dpdk-dev] [PATCH v2] " Ilya Maximets
2016-07-21 13:19 ` [dpdk-dev] [PATCH v3] " Ilya Maximets
2016-07-21 13:35 ` Yuanhan Liu
2016-07-21 13:43 ` Ilya Maximets
2016-07-21 13:56 ` Yuanhan Liu
2016-07-21 22:21 ` Thomas Monjalon
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=5790AEB3.2010708@samsung.com \
--to=i.maximets@samsung.com \
--cc=dev@dpdk.org \
--cc=heetae82.ahn@samsung.com \
--cc=huawei.xie@intel.com \
--cc=s.dyasly@samsung.com \
--cc=thomas.monjalon@6wind.com \
--cc=yuanhan.liu@linux.intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).