DPDK patches and discussions
 help / color / mirror / Atom feed
From: Elad Nachman <eladv6@gmail.com>
To: Igor Ryzhov <iryzhov@nfware.com>
Cc: Ferruh Yigit <ferruh.yigit@intel.com>,
	Stephen Hemminger <stephen@networkplumber.org>,
	dev <dev@dpdk.org>
Subject: Re: [dpdk-dev] [PATCH 2/2] kni: fix rtnl deadlocks and race conditions v3
Date: Wed, 24 Feb 2021 15:33:05 +0200	[thread overview]
Message-ID: <CACXF7qn9DNK5Rfuug=FdQ044nkK+YAFG6jmpr0eqbwttzg4etw@mail.gmail.com> (raw)
In-Reply-To: <CAF+s_Fza_63+4jL9anOEuBPgt+0WdicaWu9JfUOmr3vK-Qkr2Q@mail.gmail.com>

Currently KNI has a lot of issues with deadlocks locking the code,
after this commit, they are gone, and the code runs properly without
crashing.
That was tested with over 100 restarts of the application, which
previously required a hard reset of the board.

I think this benefit overweights the complication of the code.

The function is called with rtnl locked because this is how the Linux
kernel is designed to work - it is not designed to work with deferral
to user-space mid-function.

To fix all such requests you need to reach an agreement with Linux
netdev, which is unlikely.

Calling user-space can be done asynchronously, as Ferruh asked, but
then you will always have to return success, even on failure, as Linux
kernel does not have a mechanism to asynchronously report on failure
for such system calls.

IMHO - weighting the non-reporting of failure versus how the code
looks (as it functions perfectly OK), I decided to go with
functionality.

FYI,

Elad.

On Wed, Feb 24, 2021 at 2:50 PM Igor Ryzhov <iryzhov@nfware.com> wrote:
>
> This looks more like a hack than an actual fix to me.
>
> After this commit:
> "ip link set up" is sent to the userspace with unlocked rtnl_lock
> "ip link set down" is sent to the userspace with locked rtnl_lock
>
> How is this really fixing anything? IMHO it only complicates the code.
> If talking with userspace under rtnl_lock is a problem, then we should fix all such requests, not only part of them.
> If it is not a problem, then I don't see any point in merging this.
>
> On Tue, Feb 23, 2021 at 4:45 PM Elad Nachman <eladv6@gmail.com> wrote:
>>
>> This part of the series includes my fixes for the issues reported
>> by Ferruh and Igor on top of part 1 of the patch series:
>>
>> A. KNI sync lock is being locked while rtnl is held.
>> If two threads are calling kni_net_process_request() ,
>> then the first one will take the sync lock, release rtnl lock then sleep.
>> The second thread will try to lock sync lock while holding rtnl.
>> The first thread will wake, and try to lock rtnl, resulting in a deadlock.
>> The remedy is to release rtnl before locking the KNI sync lock.
>> Since in between nothing is accessing Linux network-wise,
>> no rtnl locking is needed.
>>
>> B. There is a race condition in __dev_close_many() processing the
>> close_list while the application terminates.
>> It looks like if two vEth devices are terminating,
>> and one releases the rtnl lock, the other takes it,
>> updating the close_list in an unstable state,
>> causing the close_list to become a circular linked list,
>> hence list_for_each_entry() will endlessly loop inside
>> __dev_close_many() .
>> Since the description for the original patch indicate the
>> original motivation was bringing the device up,
>> I have changed kni_net_process_request() to hold the rtnl mutex
>> in case of bringing the device down since this is the path called
>> from __dev_close_many() , causing the corruption of the close_list.
>>
>> Signed-off-by: Elad Nachman <eladv6@gmail.com>
>> ---
>> v3:
>> * Include original patch and new patch as a series of patch, added a
>>   comment to the new patch
>> v2:
>> * rebuild the patch as increment from patch 64106
>> * fix comment and blank lines
>> ---
>>  kernel/linux/kni/kni_net.c | 29 +++++++++++++++++++++--------
>>  1 file changed, 21 insertions(+), 8 deletions(-)
>>
>> diff --git a/kernel/linux/kni/kni_net.c b/kernel/linux/kni/kni_net.c
>> index f0b6e9a8d..017e44812 100644
>> --- a/kernel/linux/kni/kni_net.c
>> +++ b/kernel/linux/kni/kni_net.c
>> @@ -110,9 +110,26 @@ kni_net_process_request(struct net_device *dev, struct rte_kni_request *req)
>>         void *resp_va;
>>         uint32_t num;
>>         int ret_val;
>> +       int req_is_dev_stop = 0;
>> +
>> +       /* For configuring the interface to down,
>> +        * rtnl must be held all the way to prevent race condition
>> +        * inside __dev_close_many() between two netdev instances of KNI
>> +        */
>> +       if (req->req_id == RTE_KNI_REQ_CFG_NETWORK_IF &&
>> +                       req->if_up == 0)
>> +               req_is_dev_stop = 1;
>>
>>         ASSERT_RTNL();
>>
>> +       /* Since we need to wait and RTNL mutex is held
>> +        * drop the mutex and hold reference to keep device
>> +        */
>> +       if (!req_is_dev_stop) {
>> +               dev_hold(dev);
>> +               rtnl_unlock();
>> +       }
>> +
>>         mutex_lock(&kni->sync_lock);
>>
>>         /* Construct data */
>> @@ -124,16 +141,8 @@ kni_net_process_request(struct net_device *dev, struct rte_kni_request *req)
>>                 goto fail;
>>         }
>>
>> -       /* Since we need to wait and RTNL mutex is held
>> -        * drop the mutex and hold refernce to keep device
>> -        */
>> -       dev_hold(dev);
>> -       rtnl_unlock();
>> -
>>         ret_val = wait_event_interruptible_timeout(kni->wq,
>>                         kni_fifo_count(kni->resp_q), 3 * HZ);
>> -       rtnl_lock();
>> -       dev_put(dev);
>>
>>         if (signal_pending(current) || ret_val <= 0) {
>>                 ret = -ETIME;
>> @@ -152,6 +161,10 @@ kni_net_process_request(struct net_device *dev, struct rte_kni_request *req)
>>
>>  fail:
>>         mutex_unlock(&kni->sync_lock);
>> +       if (!req_is_dev_stop) {
>> +               rtnl_lock();
>> +               dev_put(dev);
>> +       }
>>         return ret;
>>  }
>>
>> --
>> 2.17.1
>>

  reply	other threads:[~2021-02-24 13:33 UTC|newest]

Thread overview: 42+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-11-26 14:46 [dpdk-dev] [PATCH] kni: fix rtnl deadlocks and race conditions Elad Nachman
2021-02-19 18:41 ` Ferruh Yigit
2021-02-21  8:03   ` Elad Nachman
2021-02-22 15:58     ` Ferruh Yigit
2021-02-23 12:05 ` [dpdk-dev] [PATCH V2] kni: fix rtnl deadlocks and race conditions v2 Elad Nachman
2021-02-23 12:53   ` Ferruh Yigit
2021-02-23 13:44 ` [dpdk-dev] [PATCH 1/2] kni: fix rtnl deadlocks and race conditions v3 Elad Nachman
2021-02-23 13:45 ` [dpdk-dev] [PATCH 2/2] " Elad Nachman
2021-02-24 12:49   ` Igor Ryzhov
2021-02-24 13:33     ` Elad Nachman [this message]
2021-02-24 14:04       ` Igor Ryzhov
2021-02-24 14:06         ` Elad Nachman
2021-02-24 14:41           ` Igor Ryzhov
2021-02-24 14:56             ` Elad Nachman
2021-02-24 15:18               ` Igor Ryzhov
     [not found]                 ` <CACXF7qkhkzFc-=v=iiBzh2V7rLjk1U34VUfPbNrnYJND_0TKHQ@mail.gmail.com>
2021-02-24 16:31                   ` Igor Ryzhov
2021-02-24 15:54     ` Stephen Hemminger
2021-02-25 14:32 ` [dpdk-dev] [PATCH 1/2] kni: fix kernel deadlock when using mlx devices Elad Nachman
2021-02-25 14:32   ` [dpdk-dev] [PATCH 2/2] kni: fix rtnl deadlocks and race conditions v4 Elad Nachman
2021-02-25 21:01     ` Igor Ryzhov
2021-02-26 15:48       ` Stephen Hemminger
2021-02-26 17:43         ` Elad Nachman
2021-03-01  8:10           ` Igor Ryzhov
2021-03-01 16:38             ` Stephen Hemminger
2021-03-15 16:58               ` Ferruh Yigit
2021-03-01 20:27             ` Dan Gora
2021-03-01 21:26               ` Dan Gora
2021-03-02 16:44                 ` Elad Nachman
2021-03-15 17:17     ` Ferruh Yigit
2021-03-16 18:35       ` Elad Nachman
2021-03-16 18:42         ` Ferruh Yigit
2021-03-15 17:17   ` [dpdk-dev] [PATCH 1/2] kni: fix kernel deadlock when using mlx devices Ferruh Yigit
2021-03-29 14:36 ` [dpdk-dev] [PATCH v5 1/3] kni: refactor user request processing Ferruh Yigit
2021-03-29 14:36   ` [dpdk-dev] [PATCH v5 2/3] kni: support async user request Ferruh Yigit
2021-03-29 14:36   ` [dpdk-dev] [PATCH v5 3/3] kni: fix kernel deadlock when using mlx devices Ferruh Yigit
2021-04-09 14:56     ` [dpdk-dev] [dpdk-stable] " Ferruh Yigit
2021-04-12 14:35       ` Elad Nachman
2021-04-20 23:07         ` Thomas Monjalon
2021-04-23  8:41           ` Igor Ryzhov
2021-04-23  8:59             ` Ferruh Yigit
2021-04-23 12:43               ` Igor Ryzhov
2021-04-23 12:58                 ` Igor Ryzhov

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to='CACXF7qn9DNK5Rfuug=FdQ044nkK+YAFG6jmpr0eqbwttzg4etw@mail.gmail.com' \
    --to=eladv6@gmail.com \
    --cc=dev@dpdk.org \
    --cc=ferruh.yigit@intel.com \
    --cc=iryzhov@nfware.com \
    --cc=stephen@networkplumber.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).