DPDK patches and discussions
 help / color / mirror / Atom feed
From: Tal Shnaiderman <talshn@nvidia.com>
To: dev@dpdk.org
Cc: thomas@monjalon.net, matan@nvidia.com, rasland@nvidia.com,
	ophirmu@nvidia.com
Subject: [dpdk-dev] [PATCH v2 18/35] net/mlx5/windows: initial probing implementation
Date: Mon, 28 Dec 2020 14:32:45 +0200	[thread overview]
Message-ID: <20201228123302.3608-19-talshn@nvidia.com> (raw)
In-Reply-To: <20201228123302.3608-1-talshn@nvidia.com>

This commit implements mlx5_os_pci_probe API under Windows. It does all
required initializations then it gets the PCI device list using glue API
get_device_list().  Next, all non MLX5 matched devices are filtered out.
The supported NIC types are: CONNECTX4VF, CONNECTX4LXVF, CONNECTX5VF,
CONNECTX5EXVF, CONNECTX5BFVF, CONNECTX6VF, MELLANOX_CONNECTX6DXVF.  Each
device in the list is assigned with default configuration parameters,
most of them are 0. The default dv_flow_en parameter value is 1 (which
means Windows match and action flows are based on DV code). Next for
each PCI device call mlx5_dev_spawn() to create an eth device (struct
rte_ethdev). The implementation of device spawn is in the follow up
commit.  Finally, the device list is free.

Signed-off-by: Tal Shnaiderman <talshn@nvidia.com>
Signed-off-by: Ophir Munk <ophirmu@nvidia.com>
Acked-by: Matan Azrad <matan@nvidia.com>
---
 drivers/net/mlx5/windows/mlx5_os.c | 267 +++++++++++++++++++++++++++++++++++++
 1 file changed, 267 insertions(+)

diff --git a/drivers/net/mlx5/windows/mlx5_os.c b/drivers/net/mlx5/windows/mlx5_os.c
index 14bd5bf39a..d9c402e942 100644
--- a/drivers/net/mlx5/windows/mlx5_os.c
+++ b/drivers/net/mlx5/windows/mlx5_os.c
@@ -9,6 +9,7 @@
 #include <stdlib.h>
 
 #include <rte_windows.h>
+#include <rte_ethdev_pci.h>
 
 #include <mlx5_glue.h>
 #include <mlx5_devx_cmds.h>
@@ -26,6 +27,65 @@
 #include "mlx5_mr.h"
 #include "mlx5_flow.h"
 
+static const char *MZ_MLX5_PMD_SHARED_DATA = "mlx5_pmd_shared_data";
+
+/* Spinlock for mlx5_shared_data allocation. */
+static rte_spinlock_t mlx5_shared_data_lock = RTE_SPINLOCK_INITIALIZER;
+
+/**
+ * Initialize shared data between primary and secondary process.
+ *
+ * A memzone is reserved by primary process and secondary processes attach to
+ * the memzone.
+ *
+ * @return
+ *   0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_init_shared_data(void)
+{
+	const struct rte_memzone *mz;
+	int ret = 0;
+
+	rte_spinlock_lock(&mlx5_shared_data_lock);
+	if (mlx5_shared_data == NULL) {
+		/* Allocate shared memory. */
+		mz = rte_memzone_reserve(MZ_MLX5_PMD_SHARED_DATA,
+					 sizeof(*mlx5_shared_data),
+					 SOCKET_ID_ANY, 0);
+		if (mz == NULL) {
+			DRV_LOG(ERR,
+				"Cannot allocate mlx5 shared data");
+			ret = -rte_errno;
+			goto error;
+		}
+		mlx5_shared_data = mz->addr;
+		memset(mlx5_shared_data, 0, sizeof(*mlx5_shared_data));
+		rte_spinlock_init(&mlx5_shared_data->lock);
+	}
+error:
+	rte_spinlock_unlock(&mlx5_shared_data_lock);
+	return ret;
+}
+
+/**
+ * PMD global initialization.
+ *
+ * Independent from individual device, this function initializes global
+ * per-PMD data structures distinguishing primary and secondary processes.
+ * Hence, each initialization is called once per a process.
+ *
+ * @return
+ *   0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+static int
+mlx5_init_once(void)
+{
+	if (mlx5_init_shared_data())
+		return -rte_errno;
+	return 0;
+}
+
 /**
  * Get mlx5 device attributes.
  *
@@ -147,6 +207,31 @@ mlx5_os_open_device(const struct mlx5_dev_spawn_data *spawn,
 	return err;
 }
 
+/**
+ * Spawn an Ethernet device from Verbs information.
+ *
+ * @param dpdk_dev
+ *   Backing DPDK device.
+ * @param spawn
+ *   Verbs device parameters (name, port, switch_info) to spawn.
+ * @param config
+ *   Device configuration parameters.
+ *
+ * @return
+ *   NULL pointer. Operation is not supported and rte_errno is set to ENOTSUP.
+ */
+static struct rte_eth_dev *
+mlx5_dev_spawn(struct rte_device *dpdk_dev,
+	       struct mlx5_dev_spawn_data *spawn,
+	       struct mlx5_dev_config *config)
+{
+	(void)dpdk_dev;
+	(void)spawn;
+	(void)config;
+	rte_errno = -ENOTSUP;
+	return NULL;
+}
+
 /**
  * This function should share events between multiple ports of single IB
  * device.  Currently it has no support under Windows.
@@ -339,6 +424,188 @@ mlx5_os_set_allmulti(struct rte_eth_dev *dev, int enable)
 	return -ENOTSUP;
 }
 
+/**
+ * DPDK callback to register a PCI device.
+ *
+ * This function spawns Ethernet devices out of a given PCI device.
+ *
+ * @param[in] pci_drv
+ *   PCI driver structure (mlx5_driver).
+ * @param[in] pci_dev
+ *   PCI device information.
+ *
+ * @return
+ *   0 on success, a negative errno value otherwise and rte_errno is set.
+ */
+int
+mlx5_os_pci_probe(struct rte_pci_driver *pci_drv __rte_unused,
+		  struct rte_pci_device *pci_dev)
+{
+	struct devx_device_bdf *devx_bdf_devs, *orig_devx_bdf_devs;
+	/*
+	 * Number of found IB Devices matching with requested PCI BDF.
+	 * nd != 1 means there are multiple IB devices over the same
+	 * PCI device and we have representors and master.
+	 */
+	unsigned int nd = 0;
+	/*
+	 * Number of found IB device Ports. nd = 1 and np = 1..n means
+	 * we have the single multiport IB device, and there may be
+	 * representors attached to some of found ports.
+	 * Currently not supported.
+	 * unsigned int np = 0;
+	 */
+
+	/*
+	 * Number of DPDK ethernet devices to Spawn - either over
+	 * multiple IB devices or multiple ports of single IB device.
+	 * Actually this is the number of iterations to spawn.
+	 */
+	unsigned int ns = 0;
+	/*
+	 * Bonding device
+	 *   < 0 - no bonding device (single one)
+	 *  >= 0 - bonding device (value is slave PF index)
+	 */
+	int bd = -1;
+	struct mlx5_dev_spawn_data *list = NULL;
+	struct mlx5_dev_config dev_config;
+	unsigned int dev_config_vf;
+	int ret;
+	uint32_t restore;
+
+	if (rte_eal_process_type() == RTE_PROC_SECONDARY) {
+		DRV_LOG(ERR, "Secondary process is not supported on Windows.");
+		return -ENOTSUP;
+	}
+	ret = mlx5_init_once();
+	if (ret) {
+		DRV_LOG(ERR, "unable to init PMD global data: %s",
+			strerror(rte_errno));
+		return -rte_errno;
+	}
+	errno = 0;
+	devx_bdf_devs = mlx5_glue->get_device_list(&ret);
+	orig_devx_bdf_devs = devx_bdf_devs;
+	if (!devx_bdf_devs) {
+		rte_errno = errno ? errno : ENOSYS;
+		DRV_LOG(ERR, "cannot list devices, is ib_uverbs loaded?");
+		return -rte_errno;
+	}
+	/*
+	 * First scan the list of all Infiniband devices to find
+	 * matching ones, gathering into the list.
+	 */
+	struct devx_device_bdf *devx_bdf_match[ret + 1];
+
+	while (ret-- > 0) {
+		if (pci_dev->addr.bus != devx_bdf_devs->bus_id ||
+		    pci_dev->addr.devid != devx_bdf_devs->dev_id ||
+		    pci_dev->addr.function != devx_bdf_devs->fnc_id) {
+			devx_bdf_devs++;
+			continue;
+		}
+
+		devx_bdf_match[nd++] = devx_bdf_devs;
+	}
+	devx_bdf_match[nd] = NULL;
+	if (!nd) {
+		/* No device matches, just complain and bail out. */
+		DRV_LOG(WARNING,
+			"no DevX device matches PCI device " PCI_PRI_FMT ","
+			" is DevX Configured?",
+			pci_dev->addr.domain, pci_dev->addr.bus,
+			pci_dev->addr.devid, pci_dev->addr.function);
+		rte_errno = ENOENT;
+		ret = -rte_errno;
+		goto exit;
+	}
+	/*
+	 * Now we can determine the maximal
+	 * amount of devices to be spawned.
+	 */
+	list = mlx5_malloc(MLX5_MEM_ZERO,
+			   sizeof(struct mlx5_dev_spawn_data),
+			   RTE_CACHE_LINE_SIZE, SOCKET_ID_ANY);
+	if (!list) {
+		DRV_LOG(ERR, "spawn data array allocation failure");
+		rte_errno = ENOMEM;
+		ret = -rte_errno;
+		goto exit;
+	}
+	memset(&list[ns].info, 0, sizeof(list[ns].info));
+	list[ns].max_port = 1;
+	list[ns].phys_port = 1;
+	list[ns].phys_dev = devx_bdf_match[ns];
+	list[ns].eth_dev = NULL;
+	list[ns].pci_dev = pci_dev;
+	list[ns].pf_bond = bd;
+	list[ns].ifindex = -1; /* Spawn will assign */
+	list[ns].info =
+		(struct mlx5_switch_info){
+			.master = 0,
+			.representor = 0,
+			.name_type = MLX5_PHYS_PORT_NAME_TYPE_UPLINK,
+			.port_name = 0,
+			.switch_id = 0,
+		};
+	/* Device specific configuration. */
+	switch (pci_dev->id.device_id) {
+	case PCI_DEVICE_ID_MELLANOX_CONNECTX4VF:
+	case PCI_DEVICE_ID_MELLANOX_CONNECTX4LXVF:
+	case PCI_DEVICE_ID_MELLANOX_CONNECTX5VF:
+	case PCI_DEVICE_ID_MELLANOX_CONNECTX5EXVF:
+	case PCI_DEVICE_ID_MELLANOX_CONNECTX5BFVF:
+	case PCI_DEVICE_ID_MELLANOX_CONNECTX6VF:
+	case PCI_DEVICE_ID_MELLANOX_CONNECTXVF:
+		dev_config_vf = 1;
+		break;
+	default:
+		dev_config_vf = 0;
+		break;
+	}
+	/* Default configuration. */
+	memset(&dev_config, 0, sizeof(struct mlx5_dev_config));
+	dev_config.vf = dev_config_vf;
+	dev_config.mps = 0;
+	dev_config.dbnc = MLX5_ARG_UNSET;
+	dev_config.rx_vec_en = 1;
+	dev_config.txq_inline_max = MLX5_ARG_UNSET;
+	dev_config.txq_inline_min = MLX5_ARG_UNSET;
+	dev_config.txq_inline_mpw = MLX5_ARG_UNSET;
+	dev_config.txqs_inline = MLX5_ARG_UNSET;
+	dev_config.vf_nl_en = 0;
+	dev_config.mr_ext_memseg_en = 1;
+	dev_config.mprq.max_memcpy_len = MLX5_MPRQ_MEMCPY_DEFAULT_LEN;
+	dev_config.mprq.min_rxqs_num = MLX5_MPRQ_MIN_RXQS;
+	dev_config.dv_esw_en = 0;
+	dev_config.dv_flow_en = 1;
+	dev_config.decap_en = 0;
+	dev_config.log_hp_size = MLX5_ARG_UNSET;
+	list[ns].eth_dev = mlx5_dev_spawn(&pci_dev->device,
+					  &list[ns],
+					  &dev_config);
+	if (!list[ns].eth_dev)
+		goto exit;
+	restore = list[ns].eth_dev->data->dev_flags;
+	rte_eth_copy_pci_info(list[ns].eth_dev, pci_dev);
+	/* Restore non-PCI flags cleared by the above call. */
+	list[ns].eth_dev->data->dev_flags |= restore;
+	rte_eth_dev_probing_finish(list[ns].eth_dev);
+	ret = 0;
+exit:
+	/*
+	 * Do the routine cleanup:
+	 * - free allocated spawn data array
+	 * - free the device list
+	 */
+	if (list)
+		mlx5_free(list);
+	MLX5_ASSERT(orig_devx_bdf_devs);
+	mlx5_glue->free_device_list(orig_devx_bdf_devs);
+	return ret;
+}
+
 /**
  * Set the reg_mr and dereg_mr call backs
  *
-- 
2.16.1.windows.4


  parent reply	other threads:[~2020-12-28 12:39 UTC|newest]

Thread overview: 78+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-12-17 17:30 [dpdk-dev] [PATCH 00/35] mlx5 Windows support - part #6 Tal Shnaiderman
2020-12-17 17:30 ` [dpdk-dev] [PATCH 01/35] common/mlx5/windows: add missing DV and IBV definitions Tal Shnaiderman
2020-12-28 12:32   ` [dpdk-dev] [PATCH v2 00/35] mlx5 Windows support - part #6 Tal Shnaiderman
2020-12-28 12:32     ` [dpdk-dev] [PATCH v2 01/35] common/mlx5/windows: add missing DV and IBV definitions Tal Shnaiderman
2020-12-28 12:32     ` [dpdk-dev] [PATCH v2 02/35] mlx5/windows: add mlx5 macros for fs name and path Tal Shnaiderman
2020-12-28 12:32     ` [dpdk-dev] [PATCH v2 03/35] common/mlx5: add rte compatibility header file Tal Shnaiderman
2020-12-28 12:32     ` [dpdk-dev] [PATCH v2 04/35] common/mlx5/windows: add DevX UAR getters Tal Shnaiderman
2021-01-08 12:08       ` Ferruh Yigit
2021-01-08 16:08         ` Ophir Munk
2020-12-28 12:32     ` [dpdk-dev] [PATCH v2 05/35] common/mlx5/windows: wrap event channel APIs with OS calls Tal Shnaiderman
2020-12-28 12:32     ` [dpdk-dev] [PATCH v2 06/35] net/mlx5/windows: add memory region callbacks Tal Shnaiderman
2020-12-28 12:32     ` [dpdk-dev] [PATCH v2 07/35] net/mlx5/windows: add stubs for MP requests Tal Shnaiderman
2020-12-28 12:32     ` [dpdk-dev] [PATCH v2 08/35] net/mlx5/windows: support get mac Tal Shnaiderman
2020-12-28 12:32     ` [dpdk-dev] [PATCH v2 09/35] net/mlx5/windows: add ethdev stub operations Tal Shnaiderman
2020-12-28 12:32     ` [dpdk-dev] [PATCH v2 10/35] net/mlx5/windows: support link update Tal Shnaiderman
2020-12-28 12:32     ` [dpdk-dev] [PATCH v2 11/35] net/mlx5/windows: support read clock Tal Shnaiderman
2020-12-28 12:32     ` [dpdk-dev] [PATCH v2 12/35] net/mlx5/windows: support get mtu Tal Shnaiderman
2020-12-28 12:32     ` [dpdk-dev] [PATCH v2 13/35] net/mlx5/windows: support get interface name Tal Shnaiderman
2020-12-28 12:32     ` [dpdk-dev] [PATCH v2 14/35] net/mlx5/windows: support device removed query Tal Shnaiderman
2020-12-28 12:32     ` [dpdk-dev] [PATCH v2 15/35] net/mlx5/windws: add VLAN stubs Tal Shnaiderman
2020-12-28 12:32     ` [dpdk-dev] [PATCH v2 16/35] net/mlx5/windows: support get pdn Tal Shnaiderman
2020-12-28 12:32     ` [dpdk-dev] [PATCH v2 17/35] net/mlx5/windows: support open device Tal Shnaiderman
2020-12-28 12:32     ` Tal Shnaiderman [this message]
2020-12-28 12:32     ` [dpdk-dev] [PATCH v2 19/35] net/mlx5/windws: spawn eth devices Tal Shnaiderman
2020-12-28 12:32     ` [dpdk-dev] [PATCH v2 20/35] net/mlx5/windows: support VF PCI address Tal Shnaiderman
2020-12-28 12:32     ` [dpdk-dev] [PATCH v2 21/35] net/mlx5/linux: wrap adjust flow priority with OS calls Tal Shnaiderman
2020-12-28 12:32     ` [dpdk-dev] [PATCH v2 22/35] net/mlx5/linux: add OS default miss flow action Tal Shnaiderman
2020-12-28 12:32     ` [dpdk-dev] [PATCH v2 23/35] net/mlx5/linux: fix add OS dest_devx_tir action Tal Shnaiderman
2020-12-28 12:32     ` [dpdk-dev] [PATCH v2 24/35] net/mlx5/windows: introduce flow support Tal Shnaiderman
2021-01-08 14:38       ` Ferruh Yigit
2021-01-08 14:55         ` Thomas Monjalon
2021-01-08 16:15           ` Ophir Munk
2020-12-28 12:32     ` [dpdk-dev] [PATCH v2 25/35] net/mlx5/windows: create flow matcher object Tal Shnaiderman
2020-12-28 12:32     ` [dpdk-dev] [PATCH v2 26/35] net/mlx5/windows: create flow action dest TIR object Tal Shnaiderman
2020-12-28 12:32     ` [dpdk-dev] [PATCH v2 27/35] net/mlx5/windows: create flow rule Tal Shnaiderman
2020-12-28 12:32     ` [dpdk-dev] [PATCH v2 28/35] net/mlx5: use HAVE_INFINIBAND_VERBS_H in shared code Tal Shnaiderman
2020-12-28 12:32     ` [dpdk-dev] [PATCH v2 29/35] net/mlx5: fix separating eth_dev_ops per OS Tal Shnaiderman
2020-12-28 12:32     ` [dpdk-dev] [PATCH v2 30/35] net/mlx5: define MLX5_NAMESIZE for device interface size Tal Shnaiderman
2020-12-28 12:32     ` [dpdk-dev] [PATCH v2 31/35] net/mlx5: use OS independent code in ASO feature Tal Shnaiderman
2020-12-28 12:32     ` [dpdk-dev] [PATCH v2 32/35] mlx5: use init function for ctx list mutex Tal Shnaiderman
2020-12-28 12:33     ` [dpdk-dev] [PATCH v2 33/35] mlx5: use OS function for dr_sync_domain Tal Shnaiderman
2020-12-28 12:33     ` [dpdk-dev] [PATCH v2 34/35] mlx5: do not fail on IPv6 broadcast flow creation failure Tal Shnaiderman
2020-12-28 12:33     ` [dpdk-dev] [PATCH v2 35/35] net/mlx5: fix warnings on comparison sign mismatch Tal Shnaiderman
2021-01-03  2:05     ` [dpdk-dev] [PATCH v2 00/35] mlx5 Windows support - part #6 Raslan Darawsheh
2020-12-17 17:30 ` [dpdk-dev] [PATCH 02/35] mlx5/windows: add mlx5 macros for fs name and path Tal Shnaiderman
2020-12-17 17:30 ` [dpdk-dev] [PATCH 03/35] common/mlx5: add rte compatibility header file Tal Shnaiderman
2020-12-17 17:30 ` [dpdk-dev] [PATCH 04/35] common/mlx5/windows: add DevX UAR getters Tal Shnaiderman
2020-12-17 17:30 ` [dpdk-dev] [PATCH 05/35] common/mlx5/windows: wrap event channel APIs with OS calls Tal Shnaiderman
2020-12-17 17:30 ` [dpdk-dev] [PATCH 06/35] net/mlx5/windows: add memory region callbacks Tal Shnaiderman
2020-12-17 17:30 ` [dpdk-dev] [PATCH 07/35] net/mlx5/windows: add stubs for MP requests Tal Shnaiderman
2020-12-17 17:30 ` [dpdk-dev] [PATCH 08/35] net/mlx5/windows: support get mac Tal Shnaiderman
2020-12-17 17:30 ` [dpdk-dev] [PATCH 09/35] net/mlx5/windows: add ethdev stub operations Tal Shnaiderman
2020-12-17 17:30 ` [dpdk-dev] [PATCH 10/35] net/mlx5/windows: support link update Tal Shnaiderman
2020-12-17 17:30 ` [dpdk-dev] [PATCH 11/35] net/mlx5/windows: support read clock Tal Shnaiderman
2020-12-17 17:30 ` [dpdk-dev] [PATCH 12/35] net/mlx5/windows: support get mtu Tal Shnaiderman
2020-12-17 17:30 ` [dpdk-dev] [PATCH 13/35] net/mlx5/windows: support get interface name Tal Shnaiderman
2020-12-17 17:30 ` [dpdk-dev] [PATCH 14/35] net/mlx5/windows: support device removed query Tal Shnaiderman
2020-12-17 17:30 ` [dpdk-dev] [PATCH 15/35] net/mlx5/windws: add VLAN stubs Tal Shnaiderman
2020-12-17 17:30 ` [dpdk-dev] [PATCH 16/35] net/mlx5/windows: support get pdn Tal Shnaiderman
2020-12-17 17:30 ` [dpdk-dev] [PATCH 17/35] net/mlx5/windows: support open device Tal Shnaiderman
2020-12-17 17:30 ` [dpdk-dev] [PATCH 18/35] net/mlx5/windows: initial probing implementation Tal Shnaiderman
2020-12-17 17:30 ` [dpdk-dev] [PATCH 19/35] net/mlx5/windws: spawn eth devices Tal Shnaiderman
2020-12-17 17:30 ` [dpdk-dev] [PATCH 20/35] net/mlx5/windows: support VF PCI address Tal Shnaiderman
2020-12-17 17:30 ` [dpdk-dev] [PATCH 21/35] net/mlx5/linux: wrap adjust flow priority with OS calls Tal Shnaiderman
2020-12-17 17:30 ` [dpdk-dev] [PATCH 22/35] net/mlx5/linux: add OS default miss flow action Tal Shnaiderman
2020-12-17 17:30 ` [dpdk-dev] [PATCH 23/35] net/mlx5/linux: fix add OS dest_devx_tir action Tal Shnaiderman
2020-12-17 17:30 ` [dpdk-dev] [PATCH 24/35] net/mlx5/windows: introduce flow support Tal Shnaiderman
2020-12-17 17:30 ` [dpdk-dev] [PATCH 25/35] net/mlx5/windows: create flow matcher object Tal Shnaiderman
2020-12-17 17:30 ` [dpdk-dev] [PATCH 26/35] net/mlx5/windows: create flow action dest TIR object Tal Shnaiderman
2020-12-17 17:30 ` [dpdk-dev] [PATCH 27/35] net/mlx5/windows: create flow rule Tal Shnaiderman
2020-12-17 17:30 ` [dpdk-dev] [PATCH 28/35] net/mlx5: use HAVE_INFINIBAND_VERBS_H in shared code Tal Shnaiderman
2020-12-17 17:30 ` [dpdk-dev] [PATCH 29/35] net/mlx5: fix separating eth_dev_ops per OS Tal Shnaiderman
2020-12-17 17:30 ` [dpdk-dev] [PATCH 30/35] net/mlx5: define MLX5_NAMESIZE for device interface size Tal Shnaiderman
2020-12-17 17:30 ` [dpdk-dev] [PATCH 31/35] net/mlx5: use OS independent code in ASO feature Tal Shnaiderman
2020-12-17 17:30 ` [dpdk-dev] [PATCH 32/35] mlx5: use init function for ctx list mutex Tal Shnaiderman
2020-12-17 17:30 ` [dpdk-dev] [PATCH 33/35] mlx5: use OS function for dr_sync_domain Tal Shnaiderman
2020-12-17 17:30 ` [dpdk-dev] [PATCH 34/35] mlx5: do not fail on IPv6 broadcast flow creation failure Tal Shnaiderman
2020-12-17 17:30 ` [dpdk-dev] [PATCH 35/35] net/mlx5: fix warnings on comparison sign mismatch Tal Shnaiderman

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20201228123302.3608-19-talshn@nvidia.com \
    --to=talshn@nvidia.com \
    --cc=dev@dpdk.org \
    --cc=matan@nvidia.com \
    --cc=ophirmu@nvidia.com \
    --cc=rasland@nvidia.com \
    --cc=thomas@monjalon.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).