From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 05714A0C52; Tue, 28 Sep 2021 10:35:08 +0200 (CEST) Received: from [217.70.189.124] (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 46C4C410F7; Tue, 28 Sep 2021 10:24:21 +0200 (CEST) Received: from NAM02-DM3-obe.outbound.protection.outlook.com (mail-dm3nam07on2072.outbound.protection.outlook.com [40.107.95.72]) by mails.dpdk.org (Postfix) with ESMTP id 45217410ED for ; Tue, 28 Sep 2021 10:24:19 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=CArcnSe4CvZkwS1FzaNXfGz2isYa64cmOJKRd7CXOUYb7s6Z3N2lLlr0M3IhbUCF3AZjqUcDiufpsfHsCQKMWS9+i4MYB7I2pYN1dzVRAqqPlgCSbqRcSewD1PS0aHb8eyEgyqBOjHAzdTRPrI9GaekeihM9xCe73NxMSdiZ4yt8nfSgow7tcOL1uhFDnXonYNOoc3hp8p/IU1zk+A4Wy7DPviBa+pblvcEaXigXtSvMWwamVzNhUOUMOQZFvzj6p7FZa/BiQVLHHARUCuQfzmtY30v+cxO1N1WXfybP6EagET/nPDiF/dis2K915DLMySR6Y3rBNeMtE71Fi9HwAg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=kjY+WQNjP3LEubWpEENSi1UEU8roYt/WDebLNwHO/94=; b=kRTGeV2fGkH4xhHVfN/Yi8uNmkxVNAhJxp1olTMo8puPtA8MXvLqRDFs4jCgCE7UfAq176IyO05+gECZQpe7tCrAD3kct0LlAY/GcTtxkxDJJyBXPvotPZ/z9nM+C6dsV/QJ4Qqbmk8/ZaFzKBUeqgSjnlS5pa6RVW4zCe7vEdYGlefn3YhjvU9MBVk3GPkn7kBjDTWIE9wBUZwDDnnHNV9YHp4F3oy8GEIvQaNqqFme1LIozBlpB5zPXp7VphCkXaCg97GjGN+tGYRaoq2qrPCxSdOwGVbqnuKB72xLesKIgYw0cM7do8VAiBnNbwFoepR0FJRJdN0LCalcfmv50Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 216.228.112.35) smtp.rcpttodomain=dpdk.org smtp.mailfrom=nvidia.com; dmarc=pass (p=quarantine sp=none pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=kjY+WQNjP3LEubWpEENSi1UEU8roYt/WDebLNwHO/94=; b=W3pv2wOSMjk1e8n87VVqYUUqX+X8gvfNNNc3+jYbytEaascDZvwnKF1PC3hcDOTsIhkXEK3NSnUz5bdsp9mWsp0Qbad8mSvXJhPQumpywK0e+HCOYaMPdgMCUCMJX1lAQw+N7hXgkKqdxVVLX/H2jiKyIC6AZ623PKDTg3o8stXy6N9kLesDhD1pc0L9Z5woB5Xu8/K4wMBZJxhgNLX2PS/+q7Sz/NJuiN5XuRlND2R6c6IbhK41btp4e3QIwcV3Bz9KEacXyDw/CQOk33OcpOO3ttz9f4d3+mwcc3vNGR5dWB7jWyaBfVDXt4PFtkIzVkMYG+kzROFdw0OyBSlhgw== Received: from BN9PR03CA0167.namprd03.prod.outlook.com (2603:10b6:408:f4::22) by DM6PR12MB3674.namprd12.prod.outlook.com (2603:10b6:5:1c4::29) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4544.18; Tue, 28 Sep 2021 08:24:17 +0000 Received: from BN8NAM11FT053.eop-nam11.prod.protection.outlook.com (2603:10b6:408:f4:cafe::3e) by BN9PR03CA0167.outlook.office365.com (2603:10b6:408:f4::22) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4544.15 via Frontend Transport; Tue, 28 Sep 2021 08:24:17 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 216.228.112.35) smtp.mailfrom=nvidia.com; dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 216.228.112.35 as permitted sender) receiver=protection.outlook.com; client-ip=216.228.112.35; helo=mail.nvidia.com; Received: from mail.nvidia.com (216.228.112.35) by BN8NAM11FT053.mail.protection.outlook.com (10.13.177.209) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4544.13 via Frontend Transport; Tue, 28 Sep 2021 08:24:17 +0000 Received: from DRHQMAIL107.nvidia.com (10.27.9.16) by HQMAIL111.nvidia.com (172.20.187.18) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Tue, 28 Sep 2021 08:24:15 +0000 Received: from nvidia.com (172.20.187.5) by DRHQMAIL107.nvidia.com (10.27.9.16) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Tue, 28 Sep 2021 08:24:14 +0000 From: Rongwei Liu To: , , , CC: , Date: Tue, 28 Sep 2021 11:23:55 +0300 Message-ID: <20210928082355.1616087-3-rongweil@nvidia.com> X-Mailer: git-send-email 2.27.0 In-Reply-To: <20210928082355.1616087-1-rongweil@nvidia.com> References: <20210928082355.1616087-1-rongweil@nvidia.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [172.20.187.5] X-ClientProxiedBy: HQMAIL105.nvidia.com (172.20.187.12) To DRHQMAIL107.nvidia.com (10.27.9.16) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 06b635aa-105a-44c2-1fb3-08d982595d06 X-MS-TrafficTypeDiagnostic: DM6PR12MB3674: X-LD-Processed: 43083d15-7273-40c1-b7db-39efd9ccc17a,ExtAddr X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:1013; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: W5065Bg3W9yWoABmnWxGgp6YUhoQCPg+G9AyA36Q6BN2vpaXxKE/dMUBC2bIxdD53KbZ00XI+csZEl2tSBHoVMmy+PdDpr7K5LY2UHcwVLp4BSRzeKDSwBMoe5PGm/tP4VEs1dcEe2WhXU653xLurf0jxIXJrQ6+Nl8HCWhHMLVR02yqEa+whQBz5u/me5RUoKKWwpsx5Ul7C2KK2sjTxphassK+gtvJ1PedFCQBqBE0wIPdnJqn2cZ53EF/TVMAu7vXlk7ipQ1QzTU3Jc8NWWzj8qi+nDHeQzLVO8clXy38vg0nwiIe5lSfK0RTcFDRcrGp8WOc7xzgN1QXDqtdU+3rCVnqHC63WtRWL510lW+og1Oz9L3MtPgujAYSBTzvepHhqvRBi/0Mi/m+gNzWg185Fp9jrLatQIHwKHcp6MJPl1CNjJfPrn6Qh6Ribbc9yvPcDAJsP8tPfQdxj6cO25kHir+OaMWdCrSXCykq0ekPn+7NOfdjGHKnnDw4xqbKxNWpLA/+TVk26/IRvS45LizkFfPXJ5jD0TzwosjpuscJsy17+CPdSvdOdVRSuVc3dupbRoTo2a20mJYpjpEHGtK/fnbsbsIVdj3gjETS1BAmjrrq/fUEdHK8MMcrtDTgjkrv0Fd7pKLAnuyNlbLf2lDOT6Iq9l3Ttv5Jfh7VfmSEWPMNW7EtQOvzg9VgMKvMAFfI8A8MIw+lRJojQrEsNw== X-Forefront-Antispam-Report: CIP:216.228.112.35; CTRY:US; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:mail.nvidia.com; PTR:schybrid02.nvidia.com; CAT:NONE; SFS:(4636009)(46966006)(36840700001)(36756003)(6666004)(82310400003)(86362001)(8936002)(1076003)(356005)(8676002)(70206006)(70586007)(508600001)(107886003)(55016002)(7636003)(26005)(47076005)(426003)(6286002)(36860700001)(186003)(336012)(5660300002)(316002)(83380400001)(54906003)(16526019)(2616005)(4326008)(2906002)(7696005)(110136005); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 28 Sep 2021 08:24:17.2329 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 06b635aa-105a-44c2-1fb3-08d982595d06 X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[216.228.112.35]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: BN8NAM11FT053.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM6PR12MB3674 Subject: [dpdk-dev] [PATCH 2/2] net/mlx5: support socket direct mode bonding X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" In socket direct mode, it's possible to bind any two (maybe four in future) PCIe devices with IDs like xxxx:xx:xx.x and yyyy:yy:yy.y. Bonding member interfaces are unnecessary to have the same PCIe domain/bus/device ID anymore, Kernel driver uses "system_image_guid" to identify if devices can be bound together or not. Sysfs "phys_switch_id" is used to get "system_image_guid" of each network interface. OFED 5.4+ is required to support "phys_switch_id". Centos 8.1 needs to enable switch_dev mode to read "phys_switch_id". Signed-off-by: Rongwei Liu Acked-by: Viacheslav Ovsiienko viacheslavo@nvidia.com --- drivers/net/mlx5/linux/mlx5_os.c | 43 +++++++++++++++++++++++++------- 1 file changed, 34 insertions(+), 9 deletions(-) diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c index 3746057673..b8ce6cd7f3 100644 --- a/drivers/net/mlx5/linux/mlx5_os.c +++ b/drivers/net/mlx5/linux/mlx5_os.c @@ -2008,6 +2008,8 @@ mlx5_device_bond_pci_match(const struct ibv_device *ibv_dev, FILE *bond_file = NULL, *file; int pf = -1; int ret; + uint8_t cur_guid[32] = {0}; + uint8_t guid[32] = {0}; /* * Try to get master device name. If something goes @@ -2022,6 +2024,8 @@ mlx5_device_bond_pci_match(const struct ibv_device *ibv_dev, np = mlx5_nl_portnum(nl_rdma, ibv_dev->name); if (!np) return -1; + if (mlx5_get_device_guid(pci_dev, cur_guid, sizeof(cur_guid)) < 0) + return -1; /* * The Master device might not be on the predefined * port (not on port index 1, it is not garanted), @@ -2050,6 +2054,7 @@ mlx5_device_bond_pci_match(const struct ibv_device *ibv_dev, char tmp_str[IF_NAMESIZE + 32]; struct rte_pci_addr pci_addr; struct mlx5_switch_info info; + int ret; /* Process slave interface names in the loop. */ snprintf(tmp_str, sizeof(tmp_str), @@ -2080,15 +2085,6 @@ mlx5_device_bond_pci_match(const struct ibv_device *ibv_dev, tmp_str); break; } - /* Match PCI address, allows BDF0+pfx or BDFx+pfx. */ - if (pci_dev->domain == pci_addr.domain && - pci_dev->bus == pci_addr.bus && - pci_dev->devid == pci_addr.devid && - ((pci_dev->function == 0 && - pci_dev->function + owner == pci_addr.function) || - (pci_dev->function == owner && - pci_addr.function == owner))) - pf = info.port_name; /* Get ifindex. */ snprintf(tmp_str, sizeof(tmp_str), "/sys/class/net/%s/ifindex", ifname); @@ -2105,6 +2101,30 @@ mlx5_device_bond_pci_match(const struct ibv_device *ibv_dev, bond_info->ports[info.port_name].pci_addr = pci_addr; bond_info->ports[info.port_name].ifindex = ifindex; bond_info->n_port++; + /* + * Under socket direct mode, bonding will use + * system_image_guid as identification. + * After OFED 5.4, guid is readable (ret >= 0) under sysfs. + * All bonding slaves should have the same guid even if driver + * is using PCIe BDF. + */ + ret = mlx5_get_device_guid(&pci_addr, guid, sizeof(guid)); + if (ret < 0) + break; + else if (ret > 0) { + if (!memcmp(guid, cur_guid, sizeof(guid)) && + owner == info.port_name && + (owner != 0 || (owner == 0 && + !rte_pci_addr_cmp(pci_dev, &pci_addr)))) + pf = info.port_name; + } else if (pci_dev->domain == pci_addr.domain && + pci_dev->bus == pci_addr.bus && + pci_dev->devid == pci_addr.devid && + ((pci_dev->function == 0 && + pci_dev->function + owner == pci_addr.function) || + (pci_dev->function == owner && + pci_addr.function == owner))) + pf = info.port_name; } if (pf >= 0) { /* Get bond interface info */ @@ -2117,6 +2137,11 @@ mlx5_device_bond_pci_match(const struct ibv_device *ibv_dev, DRV_LOG(INFO, "PF device %u, bond device %u(%s)", ifindex, bond_info->ifindex, bond_info->ifname); } + if (owner == 0 && pf != 0) { + DRV_LOG(INFO, "PCIe instance %04x:%02x:%02x.%x isn't bonding master", + pci_dev->domain, pci_dev->bus, pci_dev->devid, + pci_dev->function); + } return pf; } -- 2.27.0