* [dpdk-dev] [PATCH v5 1/2] common/mlx5: support pcie device guid query
2021-10-26 8:48 [dpdk-dev] [PATCH v5 0/2] support socket direct mode bonding Rongwei Liu
@ 2021-10-26 8:48 ` Rongwei Liu
2021-10-26 8:48 ` [dpdk-dev] [PATCH v5 2/2] net/mlx5: support socket direct mode bonding Rongwei Liu
2021-10-26 11:27 ` [dpdk-dev] [PATCH v5 0/2] " Raslan Darawsheh
2 siblings, 0 replies; 5+ messages in thread
From: Rongwei Liu @ 2021-10-26 8:48 UTC (permalink / raw)
To: matan, viacheslavo, orika, thomas; +Cc: dev, rasland
sysfs entry "phys_switch_id" holds each PCIe device'
guid.
The devices which reside in the same physical NIC should
have the same guid.
Signed-off-by: Rongwei Liu <rongweil@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
drivers/common/mlx5/linux/mlx5_common_os.c | 40 ++++++++++++++++++++++
drivers/common/mlx5/linux/mlx5_common_os.h | 19 ++++++++++
2 files changed, 59 insertions(+)
diff --git a/drivers/common/mlx5/linux/mlx5_common_os.c b/drivers/common/mlx5/linux/mlx5_common_os.c
index 8db3fe790a..b516564b79 100644
--- a/drivers/common/mlx5/linux/mlx5_common_os.c
+++ b/drivers/common/mlx5/linux/mlx5_common_os.c
@@ -2,6 +2,7 @@
* Copyright 2020 Mellanox Technologies, Ltd
*/
+#include <sys/types.h>
#include <unistd.h>
#include <string.h>
#include <stdio.h>
@@ -704,3 +705,42 @@ mlx5_os_open_device(struct mlx5_common_device *cdev, uint32_t classes)
DRV_LOG(ERR, "Failed to open IB device \"%s\".", ibv->name);
return -rte_errno;
}
+int
+mlx5_get_device_guid(const struct rte_pci_addr *dev, uint8_t *guid, size_t len)
+{
+ char tmp[512];
+ char cur_ifname[IF_NAMESIZE + 1];
+ FILE *id_file;
+ DIR *dir;
+ struct dirent *ptr;
+ int ret;
+
+ if (guid == NULL || len < sizeof(u_int64_t) + 1)
+ return -1;
+ memset(guid, 0, len);
+ snprintf(tmp, sizeof(tmp), "/sys/bus/pci/devices/%04x:%02x:%02x.%x/net",
+ dev->domain, dev->bus, dev->devid, dev->function);
+ dir = opendir(tmp);
+ if (dir == NULL)
+ return -1;
+ /* Traverse to identify PF interface */
+ do {
+ ptr = readdir(dir);
+ if (ptr == NULL || ptr->d_type != DT_DIR) {
+ closedir(dir);
+ return -1;
+ }
+ } while (strchr(ptr->d_name, '.') || strchr(ptr->d_name, '_') ||
+ strchr(ptr->d_name, 'v'));
+ snprintf(cur_ifname, sizeof(cur_ifname), "%s", ptr->d_name);
+ closedir(dir);
+ snprintf(tmp + strlen(tmp), sizeof(tmp) - strlen(tmp),
+ "/%s/phys_switch_id", cur_ifname);
+ /* Older OFED like 5.3 doesn't support read */
+ id_file = fopen(tmp, "r");
+ if (!id_file)
+ return 0;
+ ret = fscanf(id_file, "%16s", guid);
+ fclose(id_file);
+ return ret;
+}
diff --git a/drivers/common/mlx5/linux/mlx5_common_os.h b/drivers/common/mlx5/linux/mlx5_common_os.h
index c2957f91ec..83066e752d 100644
--- a/drivers/common/mlx5/linux/mlx5_common_os.h
+++ b/drivers/common/mlx5/linux/mlx5_common_os.h
@@ -284,4 +284,23 @@ mlx5_os_free(void *addr)
void
mlx5_set_context_attr(struct rte_device *dev, struct ibv_context *ctx);
+/**
+ * This is used to query system_image_guid as describing in PRM.
+ *
+ * @param dev[in]
+ * Pointer to a device instance as PCIe id.
+ * @param guid[out]
+ * Pointer to the buffer to hold device guid.
+ * Guid is uint64_t and corresponding to 17 bytes string.
+ * @param len[in]
+ * Guid buffer length, 17 bytes at least.
+ *
+ * @return
+ * -1 if internal failure.
+ * 0 if OFED doesn't support.
+ * >0 if success.
+ */
+int
+mlx5_get_device_guid(const struct rte_pci_addr *dev, uint8_t *guid, size_t len);
+
#endif /* RTE_PMD_MLX5_COMMON_OS_H_ */
--
2.27.0
^ permalink raw reply [flat|nested] 5+ messages in thread* [dpdk-dev] [PATCH v5 2/2] net/mlx5: support socket direct mode bonding
2021-10-26 8:48 [dpdk-dev] [PATCH v5 0/2] support socket direct mode bonding Rongwei Liu
2021-10-26 8:48 ` [dpdk-dev] [PATCH v5 1/2] common/mlx5: support pcie device guid query Rongwei Liu
@ 2021-10-26 8:48 ` Rongwei Liu
2021-10-26 11:26 ` Raslan Darawsheh
2021-10-26 11:27 ` [dpdk-dev] [PATCH v5 0/2] " Raslan Darawsheh
2 siblings, 1 reply; 5+ messages in thread
From: Rongwei Liu @ 2021-10-26 8:48 UTC (permalink / raw)
To: matan, viacheslavo, orika, thomas; +Cc: dev, rasland
In socket direct mode, it's possible to bind any two (maybe four
in future) PCIe devices with IDs like xxxx:xx:xx.x and
yyyy:yy:yy.y. Bonding member interfaces are unnecessary to have
the same PCIe domain/bus/device ID anymore,
Kernel driver uses "system_image_guid" to identify if devices can
be bound together or not. Sysfs "phys_switch_id" is used to get
"system_image_guid" of each network interface.
OFED 5.4+ is required to support "phys_switch_id".
Signed-off-by: Rongwei Liu <rongweil@nvidia.com>
Acked-by: Viacheslav Ovsiienko <viacheslavo@nvidia.com>
---
doc/guides/nics/mlx5.rst | 4 +++
doc/guides/rel_notes/release_21_11.rst | 4 +++
drivers/net/mlx5/linux/mlx5_os.c | 43 ++++++++++++++++++++------
3 files changed, 42 insertions(+), 9 deletions(-)
diff --git a/doc/guides/nics/mlx5.rst b/doc/guides/nics/mlx5.rst
index 47709d93b3..45f44c97d7 100644
--- a/doc/guides/nics/mlx5.rst
+++ b/doc/guides/nics/mlx5.rst
@@ -468,6 +468,10 @@ Limitations
- TXQ affinity subjects to HW hash once enabled.
+- Bonding under socket direct mode
+
+ - Needs OFED 5.4+.
+
Statistics
----------
diff --git a/doc/guides/rel_notes/release_21_11.rst b/doc/guides/rel_notes/release_21_11.rst
index 1ccac87b73..2f46b27709 100644
--- a/doc/guides/rel_notes/release_21_11.rst
+++ b/doc/guides/rel_notes/release_21_11.rst
@@ -217,6 +217,10 @@ New Features
* Added PDCP short MAC-I support.
* Added raw vector datapath API support.
+* **Updated Mellanox mlx5 driver.**
+
+ * Added socket direct mode bonding support.
+
* **Updated NXP dpaa2_sec crypto PMD.**
* Added PDCP short MAC-I support.
diff --git a/drivers/net/mlx5/linux/mlx5_os.c b/drivers/net/mlx5/linux/mlx5_os.c
index 72bbb665cf..3deae861d5 100644
--- a/drivers/net/mlx5/linux/mlx5_os.c
+++ b/drivers/net/mlx5/linux/mlx5_os.c
@@ -1898,6 +1898,8 @@ mlx5_device_bond_pci_match(const char *ibdev_name,
FILE *bond_file = NULL, *file;
int pf = -1;
int ret;
+ uint8_t cur_guid[32] = {0};
+ uint8_t guid[32] = {0};
/*
* Try to get master device name. If something goes wrong suppose
@@ -1911,6 +1913,8 @@ mlx5_device_bond_pci_match(const char *ibdev_name,
np = mlx5_nl_portnum(nl_rdma, ibdev_name);
if (!np)
return -1;
+ if (mlx5_get_device_guid(pci_dev, cur_guid, sizeof(cur_guid)) < 0)
+ return -1;
/*
* The master device might not be on the predefined port(not on port
* index 1, it is not guaranteed), we have to scan all Infiniband
@@ -1938,6 +1942,7 @@ mlx5_device_bond_pci_match(const char *ibdev_name,
char tmp_str[IF_NAMESIZE + 32];
struct rte_pci_addr pci_addr;
struct mlx5_switch_info info;
+ int ret;
/* Process slave interface names in the loop. */
snprintf(tmp_str, sizeof(tmp_str),
@@ -1969,15 +1974,6 @@ mlx5_device_bond_pci_match(const char *ibdev_name,
tmp_str);
break;
}
- /* Match PCI address, allows BDF0+pfx or BDFx+pfx. */
- if (pci_dev->domain == pci_addr.domain &&
- pci_dev->bus == pci_addr.bus &&
- pci_dev->devid == pci_addr.devid &&
- ((pci_dev->function == 0 &&
- pci_dev->function + owner == pci_addr.function) ||
- (pci_dev->function == owner &&
- pci_addr.function == owner)))
- pf = info.port_name;
/* Get ifindex. */
snprintf(tmp_str, sizeof(tmp_str),
"/sys/class/net/%s/ifindex", ifname);
@@ -1994,6 +1990,30 @@ mlx5_device_bond_pci_match(const char *ibdev_name,
bond_info->ports[info.port_name].pci_addr = pci_addr;
bond_info->ports[info.port_name].ifindex = ifindex;
bond_info->n_port++;
+ /*
+ * Under socket direct mode, bonding will use
+ * system_image_guid as identification.
+ * After OFED 5.4, guid is readable (ret >= 0) under sysfs.
+ * All bonding members should have the same guid even if driver
+ * is using PCIe BDF.
+ */
+ ret = mlx5_get_device_guid(&pci_addr, guid, sizeof(guid));
+ if (ret < 0)
+ break;
+ else if (ret > 0) {
+ if (!memcmp(guid, cur_guid, sizeof(guid)) &&
+ owner == info.port_name &&
+ (owner != 0 || (owner == 0 &&
+ !rte_pci_addr_cmp(pci_dev, &pci_addr))))
+ pf = info.port_name;
+ } else if (pci_dev->domain == pci_addr.domain &&
+ pci_dev->bus == pci_addr.bus &&
+ pci_dev->devid == pci_addr.devid &&
+ ((pci_dev->function == 0 &&
+ pci_dev->function + owner == pci_addr.function) ||
+ (pci_dev->function == owner &&
+ pci_addr.function == owner)))
+ pf = info.port_name;
}
if (pf >= 0) {
/* Get bond interface info */
@@ -2006,6 +2026,11 @@ mlx5_device_bond_pci_match(const char *ibdev_name,
DRV_LOG(INFO, "PF device %u, bond device %u(%s)",
ifindex, bond_info->ifindex, bond_info->ifname);
}
+ if (owner == 0 && pf != 0) {
+ DRV_LOG(INFO, "PCIe instance %04x:%02x:%02x.%x isn't bonding owner",
+ pci_dev->domain, pci_dev->bus, pci_dev->devid,
+ pci_dev->function);
+ }
return pf;
}
--
2.27.0
^ permalink raw reply [flat|nested] 5+ messages in thread* Re: [dpdk-dev] [PATCH v5 0/2] support socket direct mode bonding
2021-10-26 8:48 [dpdk-dev] [PATCH v5 0/2] support socket direct mode bonding Rongwei Liu
2021-10-26 8:48 ` [dpdk-dev] [PATCH v5 1/2] common/mlx5: support pcie device guid query Rongwei Liu
2021-10-26 8:48 ` [dpdk-dev] [PATCH v5 2/2] net/mlx5: support socket direct mode bonding Rongwei Liu
@ 2021-10-26 11:27 ` Raslan Darawsheh
2 siblings, 0 replies; 5+ messages in thread
From: Raslan Darawsheh @ 2021-10-26 11:27 UTC (permalink / raw)
To: Rongwei Liu, Matan Azrad, Slava Ovsiienko, Ori Kam,
NBU-Contact-Thomas Monjalon
Cc: dev
Hi,
> -----Original Message-----
> From: Rongwei Liu <rongweil@nvidia.com>
> Sent: Tuesday, October 26, 2021 11:48 AM
> To: Matan Azrad <matan@nvidia.com>; Slava Ovsiienko
> <viacheslavo@nvidia.com>; Ori Kam <orika@nvidia.com>; NBU-Contact-
> Thomas Monjalon <thomas@monjalon.net>
> Cc: dev@dpdk.org; Raslan Darawsheh <rasland@nvidia.com>
> Subject: [PATCH v5 0/2] support socket direct mode bonding
>
> In socket direct mode, it's possible to bind any two (maybe four in the
> future) PCIe devices with IDs like xxxx:xx:xx.x and yyyy:yy:yy.y. Bonding
> member interfaces are unnecessary to have the same PCIe
> domain/bus/device ID anymore.
>
> Doesn't need to backport to DPDK 20.11
>
> v2: fix ci warnings.
> v3: add description in release_21_11.rst.
> v4: add description in mlx5.rst.
> v5: rebase on top of master-net-mlx
>
> Rongwei Liu (2):
> common/mlx5: support pcie device guid query
> net/mlx5: support socket direct mode bonding
>
> doc/guides/nics/mlx5.rst | 4 ++
> doc/guides/rel_notes/release_21_11.rst | 4 ++
> drivers/common/mlx5/linux/mlx5_common_os.c | 40
> ++++++++++++++++++++ drivers/common/mlx5/linux/mlx5_common_os.h
> | 19 ++++++++++
> drivers/net/mlx5/linux/mlx5_os.c | 43 +++++++++++++++++-----
> 5 files changed, 101 insertions(+), 9 deletions(-)
>
> --
> 2.27.0
Series applied to next-net-mlx,
Kindest regards,
Raslan Darawsheh
^ permalink raw reply [flat|nested] 5+ messages in thread