From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 785E3A0548 for ; Sun, 28 Nov 2021 16:01:26 +0100 (CET) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 6B52640E5A; Sun, 28 Nov 2021 16:01:26 +0100 (CET) Received: from NAM10-MW2-obe.outbound.protection.outlook.com (mail-mw2nam10on2056.outbound.protection.outlook.com [40.107.94.56]) by mails.dpdk.org (Postfix) with ESMTP id 25DBA4275A for ; Sun, 28 Nov 2021 16:01:24 +0100 (CET) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=O5TPrCkc2JFoTw0PsScoA2NhB04LDWDSIzbUu6wp1gRvH3dSHQZ3gJPbS+A9h2UMlvwuEThG9U9XYwC5dicJgAANl709Ixu/ACv1eoDQKOrI0VzlYiG4V6qiVPTkAySrIa66OcnLdATOB6ObgLym+HoFi8PnYm9vcErOZORXJPtJ2ptX7zbAjvggnW4w3E6gBvtL/jsSZL59ubB8fs6NQ+1CPcXOPklrlL8cjA6XEN2KFLzqK3HTpuY58lMeKISxdtvrBfJehBujB9xz8rPpAYQv6lkZXpGxi3ZT/dGyr3rKGP7HaQ1Swj1ryshHxk0NA1Mq+k22EQy9TUqTtiDbyQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=61GJeE/iTqlqmkhqkNR+hBdSHriju3eLVmoXtPoiVEg=; b=ARaF8PbQ0HeGZIsRyx7zvH/t1HtJBcJkqRdYMSLJh3LKhxgRk5dx1m7lRt+5fW9BwCUBmWMoDxerCGzybqQZjMz8kPkgclByDX5BtQpKOWDp7ggEj6F8r6/adkEAPRFZYJ5U//szHq/NmxuTfHmRHFwW7gqUhOYbgzT9VoHLPA3W1+NSeK7rBmw5RVDSEVyN+74gKeco7p2o0iLKkTLwG+PRx3sHo2j19B3hmq/otUp2MEK9oUMwy1z7+L/Mvtd2Vp7oWsEigAlv7JItgF2uI3phrQPtOnSdp4I01s4gG1SaAxSlY1YTD2R5fvePmC+TtgD3b92WYQYWIRyu7eGIgg== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.34) smtp.rcpttodomain=nfware.com smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=quarantine pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=61GJeE/iTqlqmkhqkNR+hBdSHriju3eLVmoXtPoiVEg=; b=pXjq+L+t63N/hEi7BeQJeNywr39IuYsYcFC2Ad54DwWBf5WAeM7117UeioOW3nN5R0ES7kXRIpiFego5VLhMEMJDQ5hf0TM14Ji8x709KrhsySJGbUJOHs29I6zcTmNWnL2FbVQ1+6XKh1g4SH4KpetW8bem6QFhQ5sQloBban8H/2z2xyFJvqxnXXnDggj8cEWX3PrwFifSmjcyqgjl3s4Sdb5igzdZZbfCGrIet4cfjjnwQ2zggDSHKiwTuMqM0lXSlJadHyFbI4UwPa/Ds8lQogjcMUuuw/HUaiBCowXenscIUnXT6nGcvRRy8C0atcyQXRZDwMg14RE/KA0cNQ== Received: from BN9PR03CA0900.namprd03.prod.outlook.com (2603:10b6:408:13c::35) by DM5PR12MB1482.namprd12.prod.outlook.com (2603:10b6:4:a::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4734.22; Sun, 28 Nov 2021 15:01:21 +0000 Received: from BN8NAM11FT047.eop-nam11.prod.protection.outlook.com (2603:10b6:408:13c:cafe::5c) by BN9PR03CA0900.outlook.office365.com (2603:10b6:408:13c::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4734.23 via Frontend Transport; Sun, 28 Nov 2021 15:01:20 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.34) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.34 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.34; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.34) by BN8NAM11FT047.mail.protection.outlook.com (10.13.177.220) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4734.22 via Frontend Transport; Sun, 28 Nov 2021 15:01:20 +0000 Received: from nvidia.com (172.20.187.6) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Sun, 28 Nov 2021 15:01:08 +0000 From: Xueming Li To: Ferruh Yigit CC: Luca Boccassi , Igor Ryzhov , "dpdk stable" Subject: patch 'kni: restrict bifurcated device support' has been queued to stable release 20.11.4 Date: Sun, 28 Nov 2021 22:54:03 +0800 Message-ID: <20211128145423.3974892-60-xuemingl@nvidia.com> X-Mailer: git-send-email 2.34.0 In-Reply-To: <20211128145423.3974892-1-xuemingl@nvidia.com> References: <20211110063216.2744012-1-xuemingl@nvidia.com> <20211128145423.3974892-1-xuemingl@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL101.nvidia.com (172.20.187.10) To HQMAIL107.nvidia.com (172.20.187.13) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 2ea5f747-fe2f-496a-fbd0-08d9b27ff006 X-MS-TrafficTypeDiagnostic: DM5PR12MB1482: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:10000; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: pxt1ZJrIFzH1bxnyWk2GMJuJq2jK3R2AegebnZ1jKCw5YqXsMHwFE5TnOMhOE3D0oON8TNBuViHJlcL9AgWEM4NQXhCSesikaEKuoOeXbAZOABQp5RKY73CNwDiri5crSULrptsXyodch5NK2YyaBDndV+GF8xD8JwjVaQIfW/1PrU21bsjIm0L8a0yg1Ct58oUqS5PX3ikai1IrIIPdI1m1UaSMxbfkkesk8ozn4ZrTowSMh/ZVrBRAyChU/MoaS6wv7nF19QJj0e/gw95rFxxnU/xVop2W0tr8ywrkap/OvZLeGe+0TtoQqy68r6hM2eCHxkEkyy8r0HugtrG9TUWAzQ1N0qfD0EMugxIvMuAUkj/cyYZgdtEBGjKbiJddCmHSQRI8KNs3OSKUHRiP9vSZkqxgCkNc0fdGfSO+kttfebEpb7XNas7TnGQA/0zHUwEOa0oe9TMo5DTq9n+eE8JZGSqbQW7XotKw5Ra7kCY5a+MfIVf1KGCpUWfdL33wGwF4s1WPjU5DDrxoQLAP5my3f3O/R5SVxSsKegDu6kLubcZB2RMsUtfJZNCNscMeORUpS/jeFq1g/VMKuRY5CYMaXaVQ2DsdjQ/UYNHU3rr53F6E8N0nDwvZext0ip9uAy9r7VDBSAYOotkrRNej6vCGJpZUJHhkgVRK2vrOgRjyjuujKCwGmzyVLv42ow1dAGAd5l/+brSPJNDcXQM75apXUwz7J5sm5S2HTozTx0qOaCBvn1TXEPEFCLOa/U3QZI8U6AR0MmA971Zj+zggP0X4KvFyQxMHr9KQuSgusDrR4HA9WzNWuX2bACbiy0Kv5K3jI+brjNq/zXyuGvwBzA== X-Forefront-Antispam-Report: CIP:216.228.112.34; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid03.nvidia.com; CAT:NONE; SFS:(4636009)(36840700001)(46966006)(6916009)(55016003)(53546011)(36756003)(2616005)(47076005)(1076003)(83380400001)(2906002)(426003)(4001150100001)(356005)(336012)(86362001)(6666004)(36860700001)(54906003)(7636003)(26005)(5660300002)(70206006)(508600001)(7696005)(966005)(6286002)(316002)(82310400004)(8676002)(8936002)(4326008)(16526019)(186003)(70586007); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Nov 2021 15:01:20.4897 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 2ea5f747-fe2f-496a-fbd0-08d9b27ff006 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.34]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT047.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR12MB1482 X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org Hi, FYI, your patch has been queued to stable release 20.11.4 Note it hasn't been pushed to http://dpdk.org/browse/dpdk-stable yet. It will be pushed if I get no objections before 11/30/21. So please shout if anyone has objections. Also note that after the patch there's a diff of the upstream commit vs the patch applied to the branch. This will indicate if there was any rebasing needed to apply to the stable branch. If there were code changes for rebasing (ie: not only metadata diffs), please double check that the rebase was correctly done. Queued patches are on a temporary branch at: https://github.com/steevenlee/dpdk This queued commit can be viewed at: https://github.com/steevenlee/dpdk/commit/e811e57fe13bd9159e112471d5953160bd10fd8d Thanks. Xueming Li --- >From e811e57fe13bd9159e112471d5953160bd10fd8d Mon Sep 17 00:00:00 2001 From: Ferruh Yigit Date: Tue, 23 Nov 2021 16:46:17 +0000 Subject: [PATCH] kni: restrict bifurcated device support Cc: Xueming Li [ upstream commit a1b2558cdb6aff0c459c9cb11b382840b66d1d2b ] To enable bifurcated device support, rtnl_lock is released before calling userspace callbacks and asynchronous requests are enabled. But these changes caused more issues, like bug #809, #816. To reduce the scope of the problems, the bifurcated device support related changes are only enabled when it is requested explicitly with new 'enable_bifurcated' module parameter. And bifurcated device support is disabled by default. So the bifurcated device related problems are isolated and they can be fixed without impacting all use cases. Bugzilla ID: 816 Fixes: 631217c76135 ("kni: fix kernel deadlock with bifurcated device") Signed-off-by: Ferruh Yigit Acked-by: Igor Ryzhov --- .../prog_guide/kernel_nic_interface.rst | 28 +++++++++++++ kernel/linux/kni/kni_dev.h | 3 ++ kernel/linux/kni/kni_misc.c | 36 ++++++++++++++++ kernel/linux/kni/kni_net.c | 42 +++++++++++-------- 4 files changed, 92 insertions(+), 17 deletions(-) diff --git a/doc/guides/prog_guide/kernel_nic_interface.rst b/doc/guides/prog_guide/kernel_nic_interface.rst index 1ce03ec1a3..771c7d7fda 100644 --- a/doc/guides/prog_guide/kernel_nic_interface.rst +++ b/doc/guides/prog_guide/kernel_nic_interface.rst @@ -56,6 +56,12 @@ can be specified when the module is loaded to control its behavior: off Interfaces will be created with carrier state set to off. on Interfaces will be created with carrier state set to on. (charp) + parm: enable_bifurcated: Enable request processing support for + bifurcated drivers, which means releasing rtnl_lock before calling + userspace callback and supporting async requests (default=off): + on Enable request processing support for bifurcated drivers. + (charp) + Loading the ``rte_kni`` kernel module without any optional parameters is the typical way a DPDK application gets packets into and out of the kernel @@ -174,6 +180,28 @@ To set the default carrier state to *off*: If the ``carrier`` parameter is not specified, the default carrier state of KNI interfaces will be set to *off*. +.. _kni_bifurcated_device_support: + +Bifurcated Device Support +~~~~~~~~~~~~~~~~~~~~~~~~~ + +User callbacks are executed while kernel module holds the ``rtnl`` lock, this +causes a deadlock when callbacks run control commands on another Linux kernel +network interface. + +Bifurcated devices has kernel network driver part and to prevent deadlock for +them ``enable_bifurcated`` is used. + +To enable bifurcated device support: + +.. code-block:: console + + # insmod /kernel/linux/kni/rte_kni.ko enable_bifurcated=on + +Enabling bifurcated device support releases ``rtnl`` lock before calling +callback and locks it back after callback. Also enables asynchronous request to +support callbacks that requires rtnl lock to work (interface down). + KNI Creation and Deletion ------------------------- diff --git a/kernel/linux/kni/kni_dev.h b/kernel/linux/kni/kni_dev.h index c15da311ba..e8633486ee 100644 --- a/kernel/linux/kni/kni_dev.h +++ b/kernel/linux/kni/kni_dev.h @@ -34,6 +34,9 @@ /* Default carrier state for created KNI network interfaces */ extern uint32_t kni_dflt_carrier; +/* Request processing support for bifurcated drivers. */ +extern uint32_t bifurcated_support; + /** * A structure describing the private information for a kni device. */ diff --git a/kernel/linux/kni/kni_misc.c b/kernel/linux/kni/kni_misc.c index 2b464c4381..aae977c187 100644 --- a/kernel/linux/kni/kni_misc.c +++ b/kernel/linux/kni/kni_misc.c @@ -41,6 +41,10 @@ static uint32_t multiple_kthread_on; static char *carrier; uint32_t kni_dflt_carrier; +/* Request processing support for bifurcated drivers. */ +static char *enable_bifurcated; +uint32_t bifurcated_support; + #define KNI_DEV_IN_USE_BIT_NUM 0 /* Bit number for device in use */ static int kni_net_id; @@ -568,6 +572,22 @@ kni_parse_carrier_state(void) return 0; } +static int __init +kni_parse_bifurcated_support(void) +{ + if (!enable_bifurcated) { + bifurcated_support = 0; + return 0; + } + + if (strcmp(enable_bifurcated, "on") == 0) + bifurcated_support = 1; + else + return -1; + + return 0; +} + static int __init kni_init(void) { @@ -593,6 +613,13 @@ kni_init(void) else pr_debug("Default carrier state set to on.\n"); + if (kni_parse_bifurcated_support() < 0) { + pr_err("Invalid parameter for bifurcated support\n"); + return -EINVAL; + } + if (bifurcated_support == 1) + pr_debug("bifurcated support is enabled.\n"); + #ifdef HAVE_SIMPLIFIED_PERNET_OPERATIONS rc = register_pernet_subsys(&kni_net_ops); #else @@ -659,3 +686,12 @@ MODULE_PARM_DESC(carrier, "\t\ton Interfaces will be created with carrier state set to on.\n" "\t\t" ); + +module_param(enable_bifurcated, charp, 0644); +MODULE_PARM_DESC(enable_bifurcated, +"Enable request processing support for bifurcated drivers, " +"which means releasing rtnl_lock before calling userspace callback and " +"supporting async requests (default=off):\n" +"\t\ton Enable request processing support for bifurcated drivers.\n" +"\t\t" +); diff --git a/kernel/linux/kni/kni_net.c b/kernel/linux/kni/kni_net.c index 611719b5ee..29e5b9e21f 100644 --- a/kernel/linux/kni/kni_net.c +++ b/kernel/linux/kni/kni_net.c @@ -113,12 +113,14 @@ kni_net_process_request(struct net_device *dev, struct rte_kni_request *req) ASSERT_RTNL(); - /* If we need to wait and RTNL mutex is held - * drop the mutex and hold reference to keep device - */ - if (req->async == 0) { - dev_hold(dev); - rtnl_unlock(); + if (bifurcated_support) { + /* If we need to wait and RTNL mutex is held + * drop the mutex and hold reference to keep device + */ + if (req->async == 0) { + dev_hold(dev); + rtnl_unlock(); + } } mutex_lock(&kni->sync_lock); @@ -132,12 +134,14 @@ kni_net_process_request(struct net_device *dev, struct rte_kni_request *req) goto fail; } - /* No result available since request is handled - * asynchronously. set response to success. - */ - if (req->async != 0) { - req->result = 0; - goto async; + if (bifurcated_support) { + /* No result available since request is handled + * asynchronously. set response to success. + */ + if (req->async != 0) { + req->result = 0; + goto async; + } } ret_val = wait_event_interruptible_timeout(kni->wq, @@ -160,9 +164,11 @@ async: fail: mutex_unlock(&kni->sync_lock); - if (req->async == 0) { - rtnl_lock(); - dev_put(dev); + if (bifurcated_support) { + if (req->async == 0) { + rtnl_lock(); + dev_put(dev); + } } return ret; } @@ -207,8 +213,10 @@ kni_net_release(struct net_device *dev) /* Setting if_up to 0 means down */ req.if_up = 0; - /* request async because of the deadlock problem */ - req.async = 1; + if (bifurcated_support) { + /* request async because of the deadlock problem */ + req.async = 1; + } ret = kni_net_process_request(dev, &req); -- 2.34.0 --- Diff of the applied patch vs upstream commit (please double-check if non-empty: --- --- - 2021-11-28 22:41:06.307839066 +0800 +++ 0059-kni-restrict-bifurcated-device-support.patch 2021-11-28 22:41:03.400206198 +0800 @@ -1 +1 @@ -From a1b2558cdb6aff0c459c9cb11b382840b66d1d2b Mon Sep 17 00:00:00 2001 +From e811e57fe13bd9159e112471d5953160bd10fd8d Mon Sep 17 00:00:00 2001 @@ -4,0 +5,3 @@ +Cc: Xueming Li + +[ upstream commit a1b2558cdb6aff0c459c9cb11b382840b66d1d2b ] @@ -20 +22,0 @@ -Cc: stable@dpdk.org @@ -26 +27,0 @@ - doc/guides/rel_notes/release_21_11.rst | 6 +++ @@ -30 +31 @@ - 5 files changed, 98 insertions(+), 17 deletions(-) + 4 files changed, 92 insertions(+), 17 deletions(-) @@ -78,17 +78,0 @@ -diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst -index 4d8c59472a..fa2ce760d8 100644 ---- a/doc/guides/rel_notes/release_21_11.rst -+++ b/doc/guides/rel_notes/release_21_11.rst -@@ -75,6 +75,12 @@ New Features - operations. - * Added multi-process support. - -+* **Updated default KNI behavior on net devices control callbacks.** -+ -+ Updated KNI net devices control callbacks to run with ``rtnl`` kernel lock -+ held by default. A newly added ``enable_bifurcated`` KNI kernel module -+ parameter can be used to run callbacks with ``rtnl`` lock released. -+ - * **Added HiSilicon DMA driver.** - - The HiSilicon DMA driver provides device drivers for the Kunpeng's DMA devices. @@ -110 +94 @@ -index f4944e1ddf..f10dcd069d 100644 +index 2b464c4381..aae977c187 100644 @@ -124 +108 @@ -@@ -565,6 +569,22 @@ kni_parse_carrier_state(void) +@@ -568,6 +572,22 @@ kni_parse_carrier_state(void) @@ -147 +131 @@ -@@ -590,6 +610,13 @@ kni_init(void) +@@ -593,6 +613,13 @@ kni_init(void) @@ -161 +145 @@ -@@ -656,3 +683,12 @@ MODULE_PARM_DESC(carrier, +@@ -659,3 +686,12 @@ MODULE_PARM_DESC(carrier,