From mboxrd@z Thu Jan 1 00:00:00 1970
Return-Path: <dev-bounces@dpdk.org>
Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124])
by inbox.dpdk.org (Postfix) with ESMTP id DABFAA034F;
Wed, 24 Feb 2021 13:50:01 +0100 (CET)
Received: from [217.70.189.124] (localhost [127.0.0.1])
by mails.dpdk.org (Postfix) with ESMTP id C6FBE16083C;
Wed, 24 Feb 2021 13:50:01 +0100 (CET)
Received: from mail-ej1-f52.google.com (mail-ej1-f52.google.com
[209.85.218.52]) by mails.dpdk.org (Postfix) with ESMTP id 029C74069B
for <dev@dpdk.org>; Wed, 24 Feb 2021 13:50:00 +0100 (CET)
Received: by mail-ej1-f52.google.com with SMTP id do6so2865756ejc.3
for <dev@dpdk.org>; Wed, 24 Feb 2021 04:50:00 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=nfware-com.20150623.gappssmtp.com; s=20150623;
h=mime-version:references:in-reply-to:from:date:message-id:subject:to
:cc; bh=qZl4sO5J/D1sXc7XCUgyVpghuAhaeZ8sEJxXWhjXOQ0=;
b=HEQ68iuZu1wtEcsp3mp0eZ+Ko1XmrRYMP8EsQwo6knvVgwYIxqN//uSg0ejXeOB+9b
jcK+1hobi/wP6UGFKLeGqOE9rfdx/vCcYVtGmHRtF+aQT6f7IFj5Puae9wDStBSjOYzE
hlbNC4qNW3/0VyEfQZuQ1g1RX14xMoqcSHHNk6jgPPgoz03oYH0HSFvuDE5OZwNFoRc7
wsLGD+zrlVCnqEWlxzh+euNTHtLwqDRzUBIjZV/+lzLhYx5oXaRhUF/nKCNnAPaJ7nHH
bbi7Yq8NmbYeEeBXX7UxYYA0Wcd8JSd8C4WSFl7U0C2auEQ06JPKecuOXNp+MLLtzJXy
syIw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20161025;
h=x-gm-message-state:mime-version:references:in-reply-to:from:date
:message-id:subject:to:cc;
bh=qZl4sO5J/D1sXc7XCUgyVpghuAhaeZ8sEJxXWhjXOQ0=;
b=XuB6aB24dhnTxgNANlRh1uiZE4HbgAwC0JBLH+POX60+tVQxdYGuChaS1ZKiM0wQeX
NaVOD84zHDc8lfeemSLTER94DngUsGu6vaXGlBlO4b318LpZ6oCu927uaM3nM+zO7FHE
pgU4agB1c70jGQtucpm0dbShLRycYWfJLkH6mTFSDTlaGTk6aLNUoiegOwaqn76UTl6I
onQFxU8HQB3Rzt0w/o0y4SZGMwyqGx8vAdExHU3x1UQNHw2R7vQiPbGFuCT5zKVmbEy/
edx/7Vs5R2iEqnAotOxv+3GuCh0W24MTipbXoijJdMZnIhpTWmVwW2ivk5y2a/gjYsBU
piQg==
X-Gm-Message-State: AOAM533esVYzYJyHua4xgRN25dAzXzN/tw3uXP20dCSgAoRKJITSTNFq
WAMmsg3Xakr9lwNArhSoU8iovCTRr0kEp4jP9dNlaA==
X-Google-Smtp-Source: ABdhPJwE5ru1p8RfgKz289OexWSZl/VlumqrGpjSxgzZgQBpmOyzzQtcPUQy4VgDu3Cx0oaaHj71+VjfDSQHlAuDhLU=
X-Received: by 2002:a17:907:98af:: with SMTP id
ju15mr20805180ejc.263.1614171000678;
Wed, 24 Feb 2021 04:50:00 -0800 (PST)
MIME-Version: 1.0
References: <20201126144613.4986-1-eladv6@gmail.com>
<20210223134504.699-1-eladv6@gmail.com>
In-Reply-To: <20210223134504.699-1-eladv6@gmail.com>
From: Igor Ryzhov <iryzhov@nfware.com>
Date: Wed, 24 Feb 2021 15:49:49 +0300
Message-ID: <CAF+s_Fza_63+4jL9anOEuBPgt+0WdicaWu9JfUOmr3vK-Qkr2Q@mail.gmail.com>
To: Elad Nachman <eladv6@gmail.com>
Cc: Ferruh Yigit <ferruh.yigit@intel.com>,
Stephen Hemminger <stephen@networkplumber.org>, dev <dev@dpdk.org>
Content-Type: text/plain; charset="UTF-8"
X-Content-Filtered-By: Mailman/MimeDel 2.1.29
Subject: Re: [dpdk-dev] [PATCH 2/2] kni: fix rtnl deadlocks and race
conditions v3
X-BeenThere: dev@dpdk.org
X-Mailman-Version: 2.1.29
Precedence: list
List-Id: DPDK patches and discussions <dev.dpdk.org>
List-Unsubscribe: <https://mails.dpdk.org/options/dev>,
<mailto:dev-request@dpdk.org?subject=unsubscribe>
List-Archive: <http://mails.dpdk.org/archives/dev/>
List-Post: <mailto:dev@dpdk.org>
List-Help: <mailto:dev-request@dpdk.org?subject=help>
List-Subscribe: <https://mails.dpdk.org/listinfo/dev>,
<mailto:dev-request@dpdk.org?subject=subscribe>
Errors-To: dev-bounces@dpdk.org
Sender: "dev" <dev-bounces@dpdk.org>
This looks more like a hack than an actual fix to me.
After this commit:
"ip link set up" is sent to the userspace with unlocked rtnl_lock
"ip link set down" is sent to the userspace with locked rtnl_lock
How is this really fixing anything? IMHO it only complicates the code.
If talking with userspace under rtnl_lock is a problem, then we should fix
all such requests, not only part of them.
If it is not a problem, then I don't see any point in merging this.
On Tue, Feb 23, 2021 at 4:45 PM Elad Nachman <eladv6@gmail.com> wrote:
> This part of the series includes my fixes for the issues reported
> by Ferruh and Igor on top of part 1 of the patch series:
>
> A. KNI sync lock is being locked while rtnl is held.
> If two threads are calling kni_net_process_request() ,
> then the first one will take the sync lock, release rtnl lock then sleep.
> The second thread will try to lock sync lock while holding rtnl.
> The first thread will wake, and try to lock rtnl, resulting in a deadlock.
> The remedy is to release rtnl before locking the KNI sync lock.
> Since in between nothing is accessing Linux network-wise,
> no rtnl locking is needed.
>
> B. There is a race condition in __dev_close_many() processing the
> close_list while the application terminates.
> It looks like if two vEth devices are terminating,
> and one releases the rtnl lock, the other takes it,
> updating the close_list in an unstable state,
> causing the close_list to become a circular linked list,
> hence list_for_each_entry() will endlessly loop inside
> __dev_close_many() .
> Since the description for the original patch indicate the
> original motivation was bringing the device up,
> I have changed kni_net_process_request() to hold the rtnl mutex
> in case of bringing the device down since this is the path called
> from __dev_close_many() , causing the corruption of the close_list.
>
> Signed-off-by: Elad Nachman <eladv6@gmail.com>
> ---
> v3:
> * Include original patch and new patch as a series of patch, added a
> comment to the new patch
> v2:
> * rebuild the patch as increment from patch 64106
> * fix comment and blank lines
> ---
> kernel/linux/kni/kni_net.c | 29 +++++++++++++++++++++--------
> 1 file changed, 21 insertions(+), 8 deletions(-)
>
> diff --git a/kernel/linux/kni/kni_net.c b/kernel/linux/kni/kni_net.c
> index f0b6e9a8d..017e44812 100644
> --- a/kernel/linux/kni/kni_net.c
> +++ b/kernel/linux/kni/kni_net.c
> @@ -110,9 +110,26 @@ kni_net_process_request(struct net_device *dev,
> struct rte_kni_request *req)
> void *resp_va;
> uint32_t num;
> int ret_val;
> + int req_is_dev_stop = 0;
> +
> + /* For configuring the interface to down,
> + * rtnl must be held all the way to prevent race condition
> + * inside __dev_close_many() between two netdev instances of KNI
> + */
> + if (req->req_id == RTE_KNI_REQ_CFG_NETWORK_IF &&
> + req->if_up == 0)
> + req_is_dev_stop = 1;
>
> ASSERT_RTNL();
>
> + /* Since we need to wait and RTNL mutex is held
> + * drop the mutex and hold reference to keep device
> + */
> + if (!req_is_dev_stop) {
> + dev_hold(dev);
> + rtnl_unlock();
> + }
> +
> mutex_lock(&kni->sync_lock);
>
> /* Construct data */
> @@ -124,16 +141,8 @@ kni_net_process_request(struct net_device *dev,
> struct rte_kni_request *req)
> goto fail;
> }
>
> - /* Since we need to wait and RTNL mutex is held
> - * drop the mutex and hold refernce to keep device
> - */
> - dev_hold(dev);
> - rtnl_unlock();
> -
> ret_val = wait_event_interruptible_timeout(kni->wq,
> kni_fifo_count(kni->resp_q), 3 * HZ);
> - rtnl_lock();
> - dev_put(dev);
>
> if (signal_pending(current) || ret_val <= 0) {
> ret = -ETIME;
> @@ -152,6 +161,10 @@ kni_net_process_request(struct net_device *dev,
> struct rte_kni_request *req)
>
> fail:
> mutex_unlock(&kni->sync_lock);
> + if (!req_is_dev_stop) {
> + rtnl_lock();
> + dev_put(dev);
> + }
> return ret;
> }
>
> --
> 2.17.1
>
>