From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id D78E0A0548; Tue, 17 Aug 2021 15:46:58 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 1390A41210; Tue, 17 Aug 2021 15:45:39 +0200 (CEST) Received: from NAM11-BN8-obe.outbound.protection.outlook.com (mail-bn8nam11on2067.outbound.protection.outlook.com [40.107.236.67]) by mails.dpdk.org (Postfix) with ESMTP id D3C46411FD for ; Tue, 17 Aug 2021 15:45:36 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=apgH6yjrUKphL9UefQUE+FUATO5bC9u9Qm3ihJFDj5R/SoqxcbjmKk6TFHzTSQJ207Ga3FqU/hPig/CYyiIrkDgvd0euUyIjDUUgG7z/67DlPc0vOonlJC92lu7Z3WFheahXFvwKvfl/SQu28huJ0LknHEBDrB2OC0KapyfKYeNOeFcneUYqdyOkQnd7Y/oTteWWuN8wXAkrImpy/PZT0MRwPpnV1pOFRu8VeHAWtk5qDA/OeJYIMTaicTT4MNyp0Zei1/Z+Okks7L0cUXWLrq+vSUijXppH+uKV6IIiPon3ZOptah42axbHuGdbw+cGcRR6fUYkNZD2jlVlvbTjcw== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=izfhNgChbR4EUhI0CMTNwFcF8+C2jgLFhePBWlWkysQ=; b=CFaWaX7Kvy1xfA4jGFFGdnWowjdaL4RDe5DUlJp/8BsFPEhaU9xMy0vdfz2cUl0OI7NzmjFmpwu4kLruhMiNOkb0GiJPGEVgqJVcPMtxq+J/P4NrVv0BpHBv4LGK8LxNxbTnixs+IfSPegSDdl6T2cnGqXAf+xN374YT9JyZxOw+3lRSjnjgWH/Inqj05ctKF8As0d4RBPx9lwXvFi7j73uG9BGfnkGJ2ySZ3dacOfsrgtCAthDzSJ+et8rc+wsp/LOM+jtYyyulru5EK+egHjbwDpn4TA3rWTqnFjtC+gOeRpJR93NBMjWvKmRNFm1NHk/hjLX7MPN+qU3I6jDXPA== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.32) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=izfhNgChbR4EUhI0CMTNwFcF8+C2jgLFhePBWlWkysQ=; b=E8ZKkc9MIw1DgrfGsgG0BL50AWGtl2iWMWJNRDySOP0uXtG/X/Y5tl7FzksMk0lLnVdOdLRkH9q/V4y/HgnuBdoNU6T27DIhKh5+swRUg/eYh+BqpISKnC/ER5Ner0ub7x4KXtRZGSkNEp6kqJUI56NAnoAduE+7DdZmSmqMW4C3vvpE2ZLZDDFsEBtgpyvmD9Hci0Ia0UZ1RosLsSoIQqIPBcsKMGWI/AaWZMje06CKOG+1a9/WJv8sFkVIg6oYLxBnrcYQ4jsxnEIxB92H5/mPidYHKBAPVvJ8XjmQq6H+PF+ZqWwhvD927nJAf2y8CcIRuGr0arvw4hoa7pMN5w== Received: from MWHPR17CA0060.namprd17.prod.outlook.com (2603:10b6:300:93::22) by CY4PR1201MB2535.namprd12.prod.outlook.com (2603:10b6:903:db::15) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.15; Tue, 17 Aug 2021 13:45:35 +0000 Received: from CO1NAM11FT023.eop-nam11.prod.protection.outlook.com (2603:10b6:300:93:cafe::41) by MWHPR17CA0060.outlook.office365.com (2603:10b6:300:93::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4415.17 via Frontend Transport; Tue, 17 Aug 2021 13:45:35 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.32) smtp.mailfrom=nvidia.com; dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.32 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.32; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.32) by CO1NAM11FT023.mail.protection.outlook.com (10.13.175.35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4415.16 via Frontend Transport; Tue, 17 Aug 2021 13:45:35 +0000 Received: from DRHQMAIL107.nvidia.com (10.27.9.16) by HQMAIL109.nvidia.com (172.20.187.15) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 17 Aug 2021 06:45:34 -0700 Received: from nvidia.com (172.20.187.6) by DRHQMAIL107.nvidia.com (10.27.9.16) with Microsoft SMTP Server (TLS) id 15.0.1497.2; Tue, 17 Aug 2021 13:45:33 +0000 From: Michael Baum To: CC: Matan Azrad , Raslan Darawsheh , Viacheslav Ovsiienko Date: Tue, 17 Aug 2021 16:44:32 +0300 Message-ID: <20210817134441.1966618-13-michaelba@nvidia.com> X-Mailer: git-send-email 2.25.1 In-Reply-To: <20210817134441.1966618-1-michaelba@nvidia.com> References: <20210817134441.1966618-1-michaelba@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [172.20.187.6] X-ClientProxiedBy: HQMAIL111.nvidia.com (172.20.187.18) To DRHQMAIL107.nvidia.com (10.27.9.16) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: e2caf681-6833-40de-37e9-08d961854a32 X-MS-TrafficTypeDiagnostic: CY4PR1201MB2535: X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2201; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: H83G5SpaWPbWyapieaew36/gLaoCEGs13QvzCZB/rH/SrET8mFhajBqfxA3VV+N2sB0DvkrugrLq87rBfJd3AQAiqJ8Tnsq81pD1ldmR0vVKm8FSmBVaXv0Djqp9R1EFHKazn0UFJNKxKKasWZJulqy+wUdDsJKr18mzZ29B9iBc1v82mvzccODe1DKUWuHuKH7g5e2uWyIihIOpy0pZYI6VlKmEyomwERrmCjQT/fHpYGiicSPoJA9euf51yIv6rlznvTdRvxLUzMPDcY7ghdszO6/SSh79pzc3iOOGUOVyggUaPdrunBNnVPoWp63SeFaGmkM6uTleL/Nlt/s1dsh17dHJHsmz2JxBtJ62T6xpL/jzFaZJZdXKIEPYbPhrsI3qfNa10idV6hhQc9GImilQu4N8o4EFJQtiisQzpib4+xVCmJPJ8hv2bgAPLyrbXk7iNjMFAnYAZstgmC1V9hLIAQQ/fNjWwFZNApeAIdc0XqAzzd+OIPZh7+g287Lko5emd0GPylffoWsN6hT4is9MnvxZQuuuSod2gZmk5/QDQ82YrzQOR0Vo9Flr8H7jJfdzU7hXSu7GYjdwgcazDKgDhmYwpIC02dBenKnxvwPIEi3GOyI7VnYVImiWCyne+ulePtw3yoDLHJbW+1m5pxgrx5ot5rE5uIC4JnZotigxtL3UoJWMYLLDkbOBHwSr4JNLqETd+EhUu1Mj5i+Qvg== X-Forefront-Antispam-Report: CIP:216.228.112.32; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid01.nvidia.com; CAT:NONE; SFS:(4636009)(36840700001)(46966006)(47076005)(356005)(36756003)(1076003)(26005)(7636003)(83380400001)(8676002)(186003)(86362001)(54906003)(8936002)(16526019)(2906002)(7696005)(2616005)(107886003)(4326008)(82310400003)(36860700001)(6916009)(55016002)(70586007)(70206006)(508600001)(336012)(426003)(6286002)(5660300002)(316002); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 17 Aug 2021 13:45:35.1846 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: e2caf681-6833-40de-37e9-08d961854a32 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.32]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: CO1NAM11FT023.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR1201MB2535 Subject: [dpdk-dev] [RFC 12/21] common/mlx5: add ROCE disable in context device creation X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Add option to get IB device after disabling RoCE. It is relevant if there is vDPA class in device arguments list. Signed-off-by: Michael Baum --- drivers/common/mlx5/linux/mlx5_common_os.c | 126 ++++++++++++++++++++- 1 file changed, 125 insertions(+), 1 deletion(-) diff --git a/drivers/common/mlx5/linux/mlx5_common_os.c b/drivers/common/mlx5/linux/mlx5_common_os.c index 6f78897390..4a94865241 100644 --- a/drivers/common/mlx5/linux/mlx5_common_os.c +++ b/drivers/common/mlx5/linux/mlx5_common_os.c @@ -15,6 +15,7 @@ #include #include "mlx5_common.h" +#include "mlx5_nl.h" #include "mlx5_common_log.h" #include "mlx5_common_os.h" #include "mlx5_glue.h" @@ -39,6 +40,9 @@ const struct mlx5_glue *mlx5_glue; #define MLX5_TXDB_NCACHED 1 #define MLX5_TXDB_HEURISTIC 2 +#define MLX5_VDPA_MAX_RETRIES 20 +#define MLX5_VDPA_USEC 1000 + int mlx5_get_pci_addr(const char *dev_path, struct rte_pci_addr *pci_addr) { @@ -417,6 +421,123 @@ mlx5_glue_constructor(void) mlx5_glue = NULL; } +/* Try to disable ROCE by Netlink\Devlink. */ +static int +mlx5_nl_roce_disable(const char *addr) +{ + int nlsk_fd = mlx5_nl_init(NETLINK_GENERIC); + int devlink_id; + int enable; + int ret; + + if (nlsk_fd < 0) + return nlsk_fd; + devlink_id = mlx5_nl_devlink_family_id_get(nlsk_fd); + if (devlink_id < 0) { + ret = devlink_id; + DRV_LOG(DEBUG, + "Failed to get devlink id for ROCE operations by Netlink."); + goto close; + } + ret = mlx5_nl_enable_roce_get(nlsk_fd, devlink_id, addr, &enable); + if (ret) { + DRV_LOG(DEBUG, "Failed to get ROCE enable by Netlink: %d.", + ret); + goto close; + } else if (!enable) { + DRV_LOG(INFO, "ROCE has already disabled(Netlink)."); + goto close; + } + ret = mlx5_nl_enable_roce_set(nlsk_fd, devlink_id, addr, 0); + if (ret) + DRV_LOG(DEBUG, "Failed to disable ROCE by Netlink: %d.", ret); + else + DRV_LOG(INFO, "ROCE is disabled by Netlink successfully."); +close: + close(nlsk_fd); + return ret; +} + +/* Try to disable ROCE by sysfs. */ +static int +mlx5_sys_roce_disable(const char *addr) +{ + FILE *file_o; + int enable; + int ret; + + MKSTR(file_p, "/sys/bus/pci/devices/%s/roce_enable", addr); + file_o = fopen(file_p, "rb"); + if (!file_o) { + rte_errno = ENOTSUP; + return -ENOTSUP; + } + ret = fscanf(file_o, "%d", &enable); + if (ret != 1) { + rte_errno = EINVAL; + ret = EINVAL; + goto close; + } else if (!enable) { + ret = 0; + DRV_LOG(INFO, "ROCE has already disabled(sysfs)."); + goto close; + } + fclose(file_o); + file_o = fopen(file_p, "wb"); + if (!file_o) { + rte_errno = ENOTSUP; + return -ENOTSUP; + } + fprintf(file_o, "0\n"); + ret = 0; +close: + if (ret) + DRV_LOG(DEBUG, "Failed to disable ROCE by sysfs: %d.", ret); + else + DRV_LOG(INFO, "ROCE is disabled by sysfs successfully."); + fclose(file_o); + return ret; +} + +static int +mlx5_roce_disable(struct rte_device *dev) +{ + char pci_addr[PCI_PRI_STR_SIZE] = { 0 }; + + if (mlx5_dev_to_pci_str(dev, pci_addr, sizeof(pci_addr)) < 0) + return -rte_errno; + /* Firstly try to disable ROCE by Netlink and fallback to sysfs. */ + if (mlx5_nl_roce_disable(pci_addr) != 0 && + mlx5_sys_roce_disable(pci_addr) != 0) + return -rte_errno; + return 0; +} + +static struct ibv_device * +mlx5_vdpa_get_ibv_dev(struct rte_device *dev) +{ + struct ibv_device *ibv; + int retry; + + if (mlx5_roce_disable(dev) != 0) { + DRV_LOG(WARNING, "Failed to disable ROCE for \"%s\".", + dev->name); + return NULL; + } + /* Wait for the IB device to appear again after reload. */ + for (retry = MLX5_VDPA_MAX_RETRIES; retry > 0; --retry) { + ibv = mlx5_os_get_ibv_dev(dev); + if (ibv != NULL) + return ibv; + usleep(MLX5_VDPA_USEC); + } + DRV_LOG(ERR, + "Cannot get IB device after disabling RoCE for \"%s\", retries exceed %d.", + dev->name, MLX5_VDPA_MAX_RETRIES); + rte_errno = EAGAIN; + return NULL; +} + static int mlx5_config_doorbell_mapping_env(int dbnc) { @@ -471,7 +592,10 @@ mlx5_os_devx_open_device(struct mlx5_dev_ctx *dev_ctx, struct rte_device *dev, struct ibv_context *ctx = NULL; int dbmap_env; - ibv = mlx5_os_get_ibv_dev(dev); + if (classes & MLX5_CLASS_VDPA) + ibv = mlx5_vdpa_get_ibv_dev(dev); + else + ibv = mlx5_os_get_ibv_dev(dev); if (!ibv) return -rte_errno; DRV_LOG(INFO, "Dev information matches for device \"%s\".", ibv->name); -- 2.25.1