[dpdk-dev] [PATCH v3 0/4] Memory corruption due to HW rings allocation

[dpdk-dev] [PATCH v3 0/4] Memory corruption due to HW rings allocation

[Renata Saiakhova]

igb and ixgbe and some other drivers allocate HW rings using rte_eth_dma_zone_reserve(),
which checks first if the memzone exists for a given name, consisting of port
id, queue_id, rx/tx direction, but not for the size, alignment, and socket_id.
If the memzone with a given name exists it is returned, otherwise it is
allocated.
Disconnecting dpdk port from one type of interface (igb) and connecting it
to another type of interface (ixgbe) for the same port id, potentially creates
memory overlap and corruption, because it may require memzone of bigger size.
That's what is happening from switching from igb to ixgbe having the same port
id.

v2->v3: Remove #undef ETH_DMA_MZONE_NAME and minor changes in code standard
v1->v2: Minor changes on code standard and additional fixes in i40e em and ice drivers

Renata Saiakhova (4):
  librte_ethdev: Introduce a function to release HW rings
  drivers/net: Fix in igb and ixgbe HW rings memory
  drivers/net: Fix in i40e HW rings memory overlap
  drivers/net: Fix in em and ice HW rings memory overlap

 drivers/net/e1000/em_rxtx.c              |  2 ++
 drivers/net/e1000/igb_rxtx.c             |  2 ++
 drivers/net/i40e/i40e_rxtx.c             |  2 ++
 drivers/net/ice/ice_rxtx.c               |  2 ++
 drivers/net/ixgbe/ixgbe_rxtx.c           |  2 ++
 lib/librte_ethdev/rte_ethdev.c           | 28 ++++++++++++++++++++++--
 lib/librte_ethdev/rte_ethdev_driver.h    | 20 +++++++++++++++++
 lib/librte_ethdev/rte_ethdev_version.map |  1 +
 8 files changed, 57 insertions(+), 2 deletions(-)

-- 
2.17.2

[dpdk-dev] [PATCH v3 1/4] librte_ethdev: Introduce a function to release HW rings

[Renata Saiakhova]

Free previously allocated memzone for HW rings

Signed-off-by: Renata Saiakhova <Renata.Saiakhova@ekinops.com>
---
 lib/librte_ethdev/rte_ethdev.c           | 28 ++++++++++++++++++++++--
 lib/librte_ethdev/rte_ethdev_driver.h    | 20 +++++++++++++++++
 lib/librte_ethdev/rte_ethdev_version.map |  1 +
 3 files changed, 47 insertions(+), 2 deletions(-)

diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
index 72aed59a5..55f047e22 100644
--- a/lib/librte_ethdev/rte_ethdev.c
+++ b/lib/librte_ethdev/rte_ethdev.c
@@ -4181,6 +4181,10 @@ rte_eth_dev_rx_intr_ctl_q_get_fd(uint16_t port_id, uint16_t queue_id)
 	return fd;
 }
 
+#define ETH_DMA_MZONE_NAME(_name, _port_id, _queue_id, _ring_name) \
+	snprintf(_name, sizeof(_name), "eth_p%d_q%d_%s", \
+			_port_id, _queue_id, _ring_name)
+
 const struct rte_memzone *
 rte_eth_dma_zone_reserve(const struct rte_eth_dev *dev, const char *ring_name,
 			 uint16_t queue_id, size_t size, unsigned align,
@@ -4190,8 +4194,7 @@ rte_eth_dma_zone_reserve(const struct rte_eth_dev *dev, const char *ring_name,
 	const struct rte_memzone *mz;
 	int rc;
 
-	rc = snprintf(z_name, sizeof(z_name), "eth_p%d_q%d_%s",
-		      dev->data->port_id, queue_id, ring_name);
+	rc = ETH_DMA_MZONE_NAME(z_name, dev->data->port_id, queue_id, ring_name);
 	if (rc >= RTE_MEMZONE_NAMESIZE) {
 		RTE_ETHDEV_LOG(ERR, "ring name too long\n");
 		rte_errno = ENAMETOOLONG;
@@ -4206,6 +4209,27 @@ rte_eth_dma_zone_reserve(const struct rte_eth_dev *dev, const char *ring_name,
 			RTE_MEMZONE_IOVA_CONTIG, align);
 }
 
+int
+rte_eth_dma_zone_free(const struct rte_eth_dev *dev, const char *ring_name,
+		uint16_t queue_id)
+{
+	char z_name[RTE_MEMZONE_NAMESIZE];
+	const struct rte_memzone *mz;
+	int rc = 0;
+
+	rc = ETH_DMA_MZONE_NAME(z_name, dev->data->port_id, queue_id, ring_name);
+	if (rc >= RTE_MEMZONE_NAMESIZE) {
+		RTE_ETHDEV_LOG(ERR, "ring name too long\n");
+		return -ENAMETOOLONG;
+	}
+
+	mz = rte_memzone_lookup(z_name);
+	if (mz)
+		rc = rte_memzone_free(mz);
+
+	return rc;
+}
+
 int
 rte_eth_dev_create(struct rte_device *device, const char *name,
 	size_t priv_data_size,
diff --git a/lib/librte_ethdev/rte_ethdev_driver.h b/lib/librte_ethdev/rte_ethdev_driver.h
index 99d4cd6cd..462e765d1 100644
--- a/lib/librte_ethdev/rte_ethdev_driver.h
+++ b/lib/librte_ethdev/rte_ethdev_driver.h
@@ -180,6 +180,26 @@ rte_eth_dma_zone_reserve(const struct rte_eth_dev *eth_dev, const char *name,
 			 uint16_t queue_id, size_t size,
 			 unsigned align, int socket_id);
 
+/**
+ * @warning
+ * @b EXPERIMENTAL: this API may change without prior notice.
+ *
+ * Free previously allocated memzone for HW rings.
+ *
+ * @param eth_dev
+ *   The *eth_dev* pointer is the address of the *rte_eth_dev* structure
+ * @param name
+ *   The name of the memory zone
+ * @param queue_id
+ *   The index of the queue to add to name
+ * @return
+ *   Negative errno value on error, 0 on success.
+ */
+__rte_experimental
+int
+rte_eth_dma_zone_free(const struct rte_eth_dev *dev, const char *ring_name,
+		 uint16_t queue_id);
+
 /**
  * @internal
  * Atomically set the link status for the specific device.
diff --git a/lib/librte_ethdev/rte_ethdev_version.map b/lib/librte_ethdev/rte_ethdev_version.map
index 715505604..139a81302 100644
--- a/lib/librte_ethdev/rte_ethdev_version.map
+++ b/lib/librte_ethdev/rte_ethdev_version.map
@@ -241,4 +241,5 @@ EXPERIMENTAL {
 	__rte_ethdev_trace_rx_burst;
 	__rte_ethdev_trace_tx_burst;
 	rte_flow_get_aged_flows;
+	rte_eth_dma_zone_free;
 };
-- 
2.17.2

[dpdk-dev] [PATCH v3 2/4] drivers/net: Fix in igb and ixgbe HW rings memory

[Renata Saiakhova]

Delete memzones for HW rings in igb and ixgbe while freeing queues

Signed-off-by: Renata Saiakhova <Renata.Saiakhova@ekinops.com>
---
 drivers/net/e1000/igb_rxtx.c   | 2 ++
 drivers/net/ixgbe/ixgbe_rxtx.c | 2 ++
 2 files changed, 4 insertions(+)

diff --git a/drivers/net/e1000/igb_rxtx.c b/drivers/net/e1000/igb_rxtx.c
index 684fa4ad8..fe80c0f0d 100644
--- a/drivers/net/e1000/igb_rxtx.c
+++ b/drivers/net/e1000/igb_rxtx.c
@@ -1884,12 +1884,14 @@ igb_dev_free_queues(struct rte_eth_dev *dev)
 	for (i = 0; i < dev->data->nb_rx_queues; i++) {
 		eth_igb_rx_queue_release(dev->data->rx_queues[i]);
 		dev->data->rx_queues[i] = NULL;
+		rte_eth_dma_zone_free(dev, "rx_ring", i);
 	}
 	dev->data->nb_rx_queues = 0;
 
 	for (i = 0; i < dev->data->nb_tx_queues; i++) {
 		eth_igb_tx_queue_release(dev->data->tx_queues[i]);
 		dev->data->tx_queues[i] = NULL;
+		rte_eth_dma_zone_free(dev, "tx_ring", i);
 	}
 	dev->data->nb_tx_queues = 0;
 }
diff --git a/drivers/net/ixgbe/ixgbe_rxtx.c b/drivers/net/ixgbe/ixgbe_rxtx.c
index 2e20e18c7..977ecf513 100644
--- a/drivers/net/ixgbe/ixgbe_rxtx.c
+++ b/drivers/net/ixgbe/ixgbe_rxtx.c
@@ -3368,12 +3368,14 @@ ixgbe_dev_free_queues(struct rte_eth_dev *dev)
 	for (i = 0; i < dev->data->nb_rx_queues; i++) {
 		ixgbe_dev_rx_queue_release(dev->data->rx_queues[i]);
 		dev->data->rx_queues[i] = NULL;
+		rte_eth_dma_zone_free(dev, "rx_ring", i);
 	}
 	dev->data->nb_rx_queues = 0;
 
 	for (i = 0; i < dev->data->nb_tx_queues; i++) {
 		ixgbe_dev_tx_queue_release(dev->data->tx_queues[i]);
 		dev->data->tx_queues[i] = NULL;
+		rte_eth_dma_zone_free(dev, "tx_ring", i);
 	}
 	dev->data->nb_tx_queues = 0;
 }
-- 
2.17.2

[dpdk-dev] [PATCH v3 3/4] drivers/net: Fix in i40e HW rings memory overlap

[Renata Saiakhova]

Delete memzones for HW rings in i40e while freeing queues

Signed-off-by: Renata Saiakhova <Renata.Saiakhova@ekinops.com>
---
 drivers/net/i40e/i40e_rxtx.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/drivers/net/i40e/i40e_rxtx.c b/drivers/net/i40e/i40e_rxtx.c
index 5e7c86ed8..99cec9b99 100644
--- a/drivers/net/i40e/i40e_rxtx.c
+++ b/drivers/net/i40e/i40e_rxtx.c
@@ -2900,6 +2900,7 @@ i40e_dev_free_queues(struct rte_eth_dev *dev)
 			continue;
 		i40e_dev_rx_queue_release(dev->data->rx_queues[i]);
 		dev->data->rx_queues[i] = NULL;
+		rte_eth_dma_zone_free(dev, "rx_ring", i);
 	}
 
 	for (i = 0; i < dev->data->nb_tx_queues; i++) {
@@ -2907,6 +2908,7 @@ i40e_dev_free_queues(struct rte_eth_dev *dev)
 			continue;
 		i40e_dev_tx_queue_release(dev->data->tx_queues[i]);
 		dev->data->tx_queues[i] = NULL;
+		rte_eth_dma_zone_free(dev, "tx_ring", i);
 	}
 }
 
-- 
2.17.2

[dpdk-dev] [PATCH v3 4/4] drivers/net: Fix in em and ice HW rings memory overlap

[Renata Saiakhova]

Delete memzones for HW rings in em and ice while freeing queues

Signed-off-by: Renata Saiakhova <Renata.Saiakhova@ekinops.com>
---
 drivers/net/e1000/em_rxtx.c | 2 ++
 drivers/net/ice/ice_rxtx.c  | 2 ++
 2 files changed, 4 insertions(+)

diff --git a/drivers/net/e1000/em_rxtx.c b/drivers/net/e1000/em_rxtx.c
index 49c53712a..67a271e8c 100644
--- a/drivers/net/e1000/em_rxtx.c
+++ b/drivers/net/e1000/em_rxtx.c
@@ -1611,12 +1611,14 @@ em_dev_free_queues(struct rte_eth_dev *dev)
 	for (i = 0; i < dev->data->nb_rx_queues; i++) {
 		eth_em_rx_queue_release(dev->data->rx_queues[i]);
 		dev->data->rx_queues[i] = NULL;
+		rte_eth_dma_zone_free(dev, "rx_ring", i);
 	}
 	dev->data->nb_rx_queues = 0;
 
 	for (i = 0; i < dev->data->nb_tx_queues; i++) {
 		eth_em_tx_queue_release(dev->data->tx_queues[i]);
 		dev->data->tx_queues[i] = NULL;
+		rte_eth_dma_zone_free(dev, "tx_ring", i);
 	}
 	dev->data->nb_tx_queues = 0;
 }
diff --git a/drivers/net/ice/ice_rxtx.c b/drivers/net/ice/ice_rxtx.c
index 1c9f31efd..7c2dad9d6 100644
--- a/drivers/net/ice/ice_rxtx.c
+++ b/drivers/net/ice/ice_rxtx.c
@@ -1905,6 +1905,7 @@ ice_free_queues(struct rte_eth_dev *dev)
 			continue;
 		ice_rx_queue_release(dev->data->rx_queues[i]);
 		dev->data->rx_queues[i] = NULL;
+		rte_eth_dma_zone_free(dev, "rx_ring", i);
 	}
 	dev->data->nb_rx_queues = 0;
 
@@ -1913,6 +1914,7 @@ ice_free_queues(struct rte_eth_dev *dev)
 			continue;
 		ice_tx_queue_release(dev->data->tx_queues[i]);
 		dev->data->tx_queues[i] = NULL;
+		rte_eth_dma_zone_free(dev, "tx_ring", i);
 	}
 	dev->data->nb_tx_queues = 0;
 }
-- 
2.17.2

Re: [dpdk-dev] [PATCH v3 0/4] Memory corruption due to HW rings allocation

[Ferruh Yigit]

On 5/13/2020 2:14 PM, Renata Saiakhova wrote:
> igb and ixgbe and some other drivers allocate HW rings using rte_eth_dma_zone_reserve(),
> which checks first if the memzone exists for a given name, consisting of port
> id, queue_id, rx/tx direction, but not for the size, alignment, and socket_id.
> If the memzone with a given name exists it is returned, otherwise it is
> allocated.
> Disconnecting dpdk port from one type of interface (igb) and connecting it
> to another type of interface (ixgbe) for the same port id, potentially creates
> memory overlap and corruption, because it may require memzone of bigger size.
> That's what is happening from switching from igb to ixgbe having the same port
> id.
> 
> v2->v3: Remove #undef ETH_DMA_MZONE_NAME and minor changes in code standard
> v1->v2: Minor changes on code standard and additional fixes in i40e em and ice drivers
> 
> Renata Saiakhova (4):
>   librte_ethdev: Introduce a function to release HW rings
>   drivers/net: Fix in igb and ixgbe HW rings memory
>   drivers/net: Fix in i40e HW rings memory overlap
>   drivers/net: Fix in em and ice HW rings memory overlap

I think all driver patches can be squashed into single patch, overall they are
implementing same logic.

But as mentioned before, there are multiple other drivers allocating HW rings
with exact same name. At least I can see:
iavf
nfp
fm10k
axgbe

Or how can we know if a new PMD won't cause exact same behavior? What to you
think adding pmd name as prefix to queue memzone name for all PMDs? This can
help new PMDs using existing code as sample.

I don't know if it has been discussed before, but wouldn't update the
'rte_eth_dma_zone_reserve()' to check the size & alignment instead of just name
fix the issue for all drivers without needing to update them?

Re: [dpdk-dev] [PATCH v3 1/4] librte_ethdev: Introduce a function to release HW rings

[Burakov, Anatoly]

On 13-May-20 2:14 PM, Renata Saiakhova wrote:
> Free previously allocated memzone for HW rings
> 
> Signed-off-by: Renata Saiakhova <Renata.Saiakhova@ekinops.com>
> ---

Unless significant changes to the patch was made, you can keep the ack 
from the previous version. In the future, please add it to the commit 
message.

Acked-by: Anatoly Burakov <anatoly.burakov@intel.com>

-- 
Thanks,
Anatoly

Re: [dpdk-dev] [PATCH v3 1/4] librte_ethdev: Introduce a function to release HW rings

[Jeff Guo]

hi, renata

On 5/13/2020 9:14 PM, Renata Saiakhova wrote:
> Free previously allocated memzone for HW rings
>
> Signed-off-by: Renata Saiakhova <Renata.Saiakhova@ekinops.com>
> ---
>   lib/librte_ethdev/rte_ethdev.c           | 28 ++++++++++++++++++++++--
>   lib/librte_ethdev/rte_ethdev_driver.h    | 20 +++++++++++++++++
>   lib/librte_ethdev/rte_ethdev_version.map |  1 +
>   3 files changed, 47 insertions(+), 2 deletions(-)
>
> diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
> index 72aed59a5..55f047e22 100644
> --- a/lib/librte_ethdev/rte_ethdev.c
> +++ b/lib/librte_ethdev/rte_ethdev.c
> @@ -4181,6 +4181,10 @@ rte_eth_dev_rx_intr_ctl_q_get_fd(uint16_t port_id, uint16_t queue_id)
>   	return fd;
>   }
>   
> +#define ETH_DMA_MZONE_NAME(_name, _port_id, _queue_id, _ring_name) \
> +	snprintf(_name, sizeof(_name), "eth_p%d_q%d_%s", \
> +			_port_id, _queue_id, _ring_name)
> +
>   const struct rte_memzone *
>   rte_eth_dma_zone_reserve(const struct rte_eth_dev *dev, const char *ring_name,
>   			 uint16_t queue_id, size_t size, unsigned align,
> @@ -4190,8 +4194,7 @@ rte_eth_dma_zone_reserve(const struct rte_eth_dev *dev, const char *ring_name,
>   	const struct rte_memzone *mz;
>   	int rc;
>   
> -	rc = snprintf(z_name, sizeof(z_name), "eth_p%d_q%d_%s",
> -		      dev->data->port_id, queue_id, ring_name);
> +	rc = ETH_DMA_MZONE_NAME(z_name, dev->data->port_id, queue_id, ring_name);
>   	if (rc >= RTE_MEMZONE_NAMESIZE) {
>   		RTE_ETHDEV_LOG(ERR, "ring name too long\n");
>   		rte_errno = ENAMETOOLONG;
> @@ -4206,6 +4209,27 @@ rte_eth_dma_zone_reserve(const struct rte_eth_dev *dev, const char *ring_name,
>   			RTE_MEMZONE_IOVA_CONTIG, align);
>   }
>   
> +int
> +rte_eth_dma_zone_free(const struct rte_eth_dev *dev, const char *ring_name,
> +		uint16_t queue_id)
> +{
> +	char z_name[RTE_MEMZONE_NAMESIZE];
> +	const struct rte_memzone *mz;
> +	int rc = 0;
> +
> +	rc = ETH_DMA_MZONE_NAME(z_name, dev->data->port_id, queue_id, ring_name);
> +	if (rc >= RTE_MEMZONE_NAMESIZE) {
> +		RTE_ETHDEV_LOG(ERR, "ring name too long\n");
> +		return -ENAMETOOLONG;
> +	}
> +
> +	mz = rte_memzone_lookup(z_name);
> +	if (mz)
> +		rc = rte_memzone_free(mz);
> +
> +	return rc;


If rc <  RTE_MEMZONE_NAMESIZE && !mz, what value should be return base 
on the parameter description in .h?

Although it would not affect the other patch set, but could you please 
clarify it?


> +}
> +
>   int
>   rte_eth_dev_create(struct rte_device *device, const char *name,
>   	size_t priv_data_size,
> diff --git a/lib/librte_ethdev/rte_ethdev_driver.h b/lib/librte_ethdev/rte_ethdev_driver.h
> index 99d4cd6cd..462e765d1 100644
> --- a/lib/librte_ethdev/rte_ethdev_driver.h
> +++ b/lib/librte_ethdev/rte_ethdev_driver.h
> @@ -180,6 +180,26 @@ rte_eth_dma_zone_reserve(const struct rte_eth_dev *eth_dev, const char *name,
>   			 uint16_t queue_id, size_t size,
>   			 unsigned align, int socket_id);
>   
> +/**
> + * @warning
> + * @b EXPERIMENTAL: this API may change without prior notice.
> + *
> + * Free previously allocated memzone for HW rings.
> + *
> + * @param eth_dev
> + *   The *eth_dev* pointer is the address of the *rte_eth_dev* structure
> + * @param name
> + *   The name of the memory zone
> + * @param queue_id
> + *   The index of the queue to add to name
> + * @return
> + *   Negative errno value on error, 0 on success.
> + */
> +__rte_experimental
> +int
> +rte_eth_dma_zone_free(const struct rte_eth_dev *dev, const char *ring_name,
> +		 uint16_t queue_id);
> +
>   /**
>    * @internal
>    * Atomically set the link status for the specific device.
> diff --git a/lib/librte_ethdev/rte_ethdev_version.map b/lib/librte_ethdev/rte_ethdev_version.map
> index 715505604..139a81302 100644
> --- a/lib/librte_ethdev/rte_ethdev_version.map
> +++ b/lib/librte_ethdev/rte_ethdev_version.map
> @@ -241,4 +241,5 @@ EXPERIMENTAL {
>   	__rte_ethdev_trace_rx_burst;
>   	__rte_ethdev_trace_tx_burst;
>   	rte_flow_get_aged_flows;
> +	rte_eth_dma_zone_free;
>   };

Re: [dpdk-dev] [PATCH v3 0/4] Memory corruption due to HW rings allocation

[Renata Saiakhova]

Hi Ferruh,

thanks for comments,

are the rte_eth_dma_zone_reserve() calls always used to allocate HW rings? It is not totally clear to me. That was partly the reason I don't do the fix for every driver which uses this API. I made fixes in the drivers which uses the same pattern to allocate / release queues, for other drivers I was not sure but anyway I couldn't spend more time for further investigations. In the company I work for we use dpdk for our project and maintain it in separate tree, and the vulnerability with HW rings is a real issue for igb and ixgbe drivers and needs to be fixed. Therefore I would like this patch to be accepted in order to not maintain the fix ourselves. But unfortunately I don't have resources (e.g. time) to fix the issue for all the drivers, because, as I mentioned, they are not following the same pattern to release their queues. So my proposal is that I fix it in this patch in a number of drivers (including igb, ixgbe and i40e) and others can take over and improve other drivers, if they see the same issue. This is also a reason why the drivers' changes are not in one commit for all the drivers.

For the proposal adding pmd name as prefix to queue memzone name or update the 'rte_eth_dma_zone_reserve()' to check the size & alignment instead of just a name, I don't know, as an external person, how sensitive dpdk project to change an internal API and existing code (the call should be changed in all the drivers). But anyway, I think the real problem is more an absence of memzone pointer, and in long term it should be solved in this way, rather than search by name.

Kind regards,
Renata
________________________________
From: Ferruh Yigit <ferruh.yigit@intel.com>
Sent: Wednesday, May 13, 2020 5:22 PM
To: Renata Saiakhova <renata.saiakhova@ekinops.com>; dev@dpdk.org <dev@dpdk.org>
Cc: Anatoly Burakov <anatoly.burakov@intel.com>; Thomas Monjalon <thomas@monjalon.net>; Neil Horman <nhorman@tuxdriver.com>
Subject: Re: [dpdk-dev] [PATCH v3 0/4] Memory corruption due to HW rings allocation

On 5/13/2020 2:14 PM, Renata Saiakhova wrote:
> igb and ixgbe and some other drivers allocate HW rings using rte_eth_dma_zone_reserve(),
> which checks first if the memzone exists for a given name, consisting of port
> id, queue_id, rx/tx direction, but not for the size, alignment, and socket_id.
> If the memzone with a given name exists it is returned, otherwise it is
> allocated.
> Disconnecting dpdk port from one type of interface (igb) and connecting it
> to another type of interface (ixgbe) for the same port id, potentially creates
> memory overlap and corruption, because it may require memzone of bigger size.
> That's what is happening from switching from igb to ixgbe having the same port
> id.
>
> v2->v3: Remove #undef ETH_DMA_MZONE_NAME and minor changes in code standard
> v1->v2: Minor changes on code standard and additional fixes in i40e em and ice drivers
>
> Renata Saiakhova (4):
>   librte_ethdev: Introduce a function to release HW rings
>   drivers/net: Fix in igb and ixgbe HW rings memory
>   drivers/net: Fix in i40e HW rings memory overlap
>   drivers/net: Fix in em and ice HW rings memory overlap

I think all driver patches can be squashed into single patch, overall they are
implementing same logic.

But as mentioned before, there are multiple other drivers allocating HW rings
with exact same name. At least I can see:
iavf
nfp
fm10k
axgbe

Or how can we know if a new PMD won't cause exact same behavior? What to you
think adding pmd name as prefix to queue memzone name for all PMDs? This can
help new PMDs using existing code as sample.

I don't know if it has been discussed before, but wouldn't update the
'rte_eth_dma_zone_reserve()' to check the size & alignment instead of just name
fix the issue for all drivers without needing to update them?

Re: [dpdk-dev] [PATCH v3 3/4] drivers/net: Fix in i40e HW rings memory overlap

[Zhao1, Wei]

Hi, Renata Saiakhova

   I think this patch is very important, It seems all kind of NIC has memory leak problem that used for store
Tx or rx descriptor. If that is true, memory point by rxq-> rx_ring/ txq-> tx_ring will never be freed even if dev_close? 
Is my understanding right or wrong?

If that is true, it seems you should also add in functioni40e_fdir_teardown(), because
Tx_ring allocated in i40e_fdir_setup_tx_resources() also need freed, and memzones need to be delete.
Is that so?

Thanks.



> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Renata Saiakhova
> Sent: Wednesday, May 13, 2020 9:14 PM
> To: dev@dpdk.org
> Cc: Renata Saiakhova <Renata.Saiakhova@ekinops.com>
> Subject: [dpdk-dev] [PATCH v3 3/4] drivers/net: Fix in i40e HW rings memory
> overlap
> 
> Delete memzones for HW rings in i40e while freeing queues
> 
> Signed-off-by: Renata Saiakhova <Renata.Saiakhova@ekinops.com>
> ---
>  drivers/net/i40e/i40e_rxtx.c | 2 ++
>  1 file changed, 2 insertions(+)
> 
> diff --git a/drivers/net/i40e/i40e_rxtx.c b/drivers/net/i40e/i40e_rxtx.c index
> 5e7c86ed8..99cec9b99 100644
> --- a/drivers/net/i40e/i40e_rxtx.c
> +++ b/drivers/net/i40e/i40e_rxtx.c
> @@ -2900,6 +2900,7 @@ i40e_dev_free_queues(struct rte_eth_dev *dev)
>  			continue;
>  		i40e_dev_rx_queue_release(dev->data->rx_queues[i]);
>  		dev->data->rx_queues[i] = NULL;
> +		rte_eth_dma_zone_free(dev, "rx_ring", i);
>  	}
> 
>  	for (i = 0; i < dev->data->nb_tx_queues; i++) { @@ -2907,6 +2908,7 @@
> i40e_dev_free_queues(struct rte_eth_dev *dev)
>  			continue;
>  		i40e_dev_tx_queue_release(dev->data->tx_queues[i]);
>  		dev->data->tx_queues[i] = NULL;
> +		rte_eth_dma_zone_free(dev, "tx_ring", i);
>  	}
>  }
> 
> --
> 2.17.2

Re: [dpdk-dev] [PATCH v3 0/4] Memory corruption due to HW rings allocation

[Zhao1, Wei]

I am sure i40e FDIR filter also allocate ring for filter programing, it also has the same issue. Ice NIC the same.
Although we can avoid this issue by change ring name to "igb_tx_ring" and so on, but the issue of memory for ring not freed after close is still exsit.
Support this comment. If no one take over all the work for all the PMD, we had better accept this patch.


> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Renata Saiakhova
> Sent: Monday, May 18, 2020 5:48 PM
> To: Yigit, Ferruh <ferruh.yigit@intel.com>; dev@dpdk.org
> Cc: Burakov, Anatoly <anatoly.burakov@intel.com>; Thomas Monjalon
> <thomas@monjalon.net>; Neil Horman <nhorman@tuxdriver.com>
> Subject: Re: [dpdk-dev] [PATCH v3 0/4] Memory corruption due to HW rings
> allocation
> 
> Hi Ferruh,
> 
> thanks for comments,
> 
> are the rte_eth_dma_zone_reserve() calls always used to allocate HW rings? It
> is not totally clear to me. That was partly the reason I don't do the fix for every
> driver which uses this API. I made fixes in the drivers which uses the same
> pattern to allocate / release queues, for other drivers I was not sure but
> anyway I couldn't spend more time for further investigations. In the company I
> work for we use dpdk for our project and maintain it in separate tree, and the
> vulnerability with HW rings is a real issue for igb and ixgbe drivers and needs to
> be fixed. Therefore I would like this patch to be accepted in order to not
> maintain the fix ourselves. But unfortunately I don't have resources (e.g. time)
> to fix the issue for all the drivers, because, as I mentioned, they are not
> following the same pattern to release their queues. So my proposal is that I fix
> it in this patch in a number of drivers (including igb, ixgbe and i40e) and others
> can take over and improve other drivers, if they see the same issue. This is also
> a reason why the drivers' changes are not in one commit for all the drivers.
> 
> For the proposal adding pmd name as prefix to queue memzone name or
> update the 'rte_eth_dma_zone_reserve()' to check the size & alignment
> instead of just a name, I don't know, as an external person, how sensitive dpdk
> project to change an internal API and existing code (the call should be changed
> in all the drivers). But anyway, I think the real problem is more an absence of
> memzone pointer, and in long term it should be solved in this way, rather than
> search by name.
> 
> Kind regards,
> Renata
> ________________________________
> From: Ferruh Yigit <ferruh.yigit@intel.com>
> Sent: Wednesday, May 13, 2020 5:22 PM
> To: Renata Saiakhova <renata.saiakhova@ekinops.com>; dev@dpdk.org
> <dev@dpdk.org>
> Cc: Anatoly Burakov <anatoly.burakov@intel.com>; Thomas Monjalon
> <thomas@monjalon.net>; Neil Horman <nhorman@tuxdriver.com>
> Subject: Re: [dpdk-dev] [PATCH v3 0/4] Memory corruption due to HW rings
> allocation
> 
> On 5/13/2020 2:14 PM, Renata Saiakhova wrote:
> > igb and ixgbe and some other drivers allocate HW rings using
> > rte_eth_dma_zone_reserve(), which checks first if the memzone exists
> > for a given name, consisting of port id, queue_id, rx/tx direction, but not for
> the size, alignment, and socket_id.
> > If the memzone with a given name exists it is returned, otherwise it
> > is allocated.
> > Disconnecting dpdk port from one type of interface (igb) and
> > connecting it to another type of interface (ixgbe) for the same port
> > id, potentially creates memory overlap and corruption, because it may
> require memzone of bigger size.
> > That's what is happening from switching from igb to ixgbe having the
> > same port id.
> >
> > v2->v3: Remove #undef ETH_DMA_MZONE_NAME and minor changes in
> code
> > v2->standard
> > v1->v2: Minor changes on code standard and additional fixes in i40e em
> > v1->and ice drivers
> >
> > Renata Saiakhova (4):
> >   librte_ethdev: Introduce a function to release HW rings
> >   drivers/net: Fix in igb and ixgbe HW rings memory
> >   drivers/net: Fix in i40e HW rings memory overlap
> >   drivers/net: Fix in em and ice HW rings memory overlap
> 
> I think all driver patches can be squashed into single patch, overall they are
> implementing same logic.
> 
> But as mentioned before, there are multiple other drivers allocating HW rings
> with exact same name. At least I can see:
> iavf
> nfp
> fm10k
> axgbe
> 
> Or how can we know if a new PMD won't cause exact same behavior? What to
> you think adding pmd name as prefix to queue memzone name for all PMDs?
> This can help new PMDs using existing code as sample.
> 
> I don't know if it has been discussed before, but wouldn't update the
> 'rte_eth_dma_zone_reserve()' to check the size & alignment instead of just
> name fix the issue for all drivers without needing to update them?

Re: [dpdk-dev] [PATCH v3 0/4] Memory corruption due to HW rings allocation

[Ferruh Yigit]

On 5/18/2020 10:48 AM, Renata Saiakhova wrote:
> Hi Ferruh,
> 
> thanks for comments,
> 
> are the rte_eth_dma_zone_reserve() calls always used to allocate HW rings? It is
> not totally clear to me. That was partly the reason I don't do the fix for every
> driver which uses this API. I made fixes in the drivers which uses the same
> pattern to allocate / release queues, for other drivers I was not sure but
> anyway I couldn't spend more time for further investigations. In the company I
> work for we use dpdk for our project and maintain it in separate tree, and the
> vulnerability with HW rings is a real issue for igb and ixgbe drivers and needs
> to be fixed. Therefore I would like this patch to be accepted in order to not
> maintain the fix ourselves. But unfortunately I don't have resources (e.g. time)
> to fix the issue for all the drivers, because, as I mentioned, they are not
> following the same pattern to release their queues. So my proposal is that I fix
> it in this patch in a number of drivers (including igb, ixgbe and i40e) and
> others can take over and improve other drivers, if they see the same issue. This
> is also a reason why the drivers' changes are not in one commit for all the drivers.
> 
> For the proposal adding pmd name as prefix to queue memzone name or update
> the 'rte_eth_dma_zone_reserve()' to check the size & alignment instead of just a
> name, I don't know, as an external person, how sensitive dpdk project to change
> an internal API and existing code (the call should be changed in all the
> drivers). But anyway, I think the real problem is more an absence of memzone
> pointer, and in long term it should be solved in this way, rather than search by
> name.

Hi Renata,

'rte_eth_dma_zone_reserve()' returning existing memzone just based on 'name'
without checking the size, alignment, socket_id seems wrong to me, let me check
it, it shouldn't be hard to update.

But even 'rte_eth_dma_zone_reserve()' checks won't be enough (what to do if
check fails and not returns a memzone?), the proper solution is PMD free the
memzone as they removed, as done in this patch.

Also I suggested prefixing the memzone name as workaround but instead we can go
with this patchset with the implemented PMDs and other PMD owners can implement
the free when they need instead of workaround.

There is a comment from Wie to add more frees to i40e, and there is another from
Jeff on 'rte_eth_dma_zone_free()' return type, can you please check them? So we
can proceed with this patch for the release.

Thanks,
ferruh

> 
> Kind regards,
> Renata
> --------------------------------------------------------------------------------
> *From:* Ferruh Yigit <ferruh.yigit@intel.com>
> *Sent:* Wednesday, May 13, 2020 5:22 PM
> *To:* Renata Saiakhova <renata.saiakhova@ekinops.com>; dev@dpdk.org <dev@dpdk.org>
> *Cc:* Anatoly Burakov <anatoly.burakov@intel.com>; Thomas Monjalon
> <thomas@monjalon.net>; Neil Horman <nhorman@tuxdriver.com>
> *Subject:* Re: [dpdk-dev] [PATCH v3 0/4] Memory corruption due to HW rings
> allocation
>  
> On 5/13/2020 2:14 PM, Renata Saiakhova wrote:
>> igb and ixgbe and some other drivers allocate HW rings using rte_eth_dma_zone_reserve(),
>> which checks first if the memzone exists for a given name, consisting of port
>> id, queue_id, rx/tx direction, but not for the size, alignment, and socket_id.
>> If the memzone with a given name exists it is returned, otherwise it is
>> allocated.
>> Disconnecting dpdk port from one type of interface (igb) and connecting it
>> to another type of interface (ixgbe) for the same port id, potentially creates
>> memory overlap and corruption, because it may require memzone of bigger size.
>> That's what is happening from switching from igb to ixgbe having the same port
>> id.
>> 
>> v2->v3: Remove #undef ETH_DMA_MZONE_NAME and minor changes in code standard
>> v1->v2: Minor changes on code standard and additional fixes in i40e em and ice drivers
>> 
>> Renata Saiakhova (4):
>>   librte_ethdev: Introduce a function to release HW rings
>>   drivers/net: Fix in igb and ixgbe HW rings memory
>>   drivers/net: Fix in i40e HW rings memory overlap
>>   drivers/net: Fix in em and ice HW rings memory overlap
> 
> I think all driver patches can be squashed into single patch, overall they are
> implementing same logic.
> 
> But as mentioned before, there are multiple other drivers allocating HW rings
> with exact same name. At least I can see:
> iavf
> nfp
> fm10k
> axgbe
> 
> Or how can we know if a new PMD won't cause exact same behavior? What to you
> think adding pmd name as prefix to queue memzone name for all PMDs? This can
> help new PMDs using existing code as sample.
> 
> I don't know if it has been discussed before, but wouldn't update the
> 'rte_eth_dma_zone_reserve()' to check the size & alignment instead of just name
> fix the issue for all drivers without needing to update them?

Re: [dpdk-dev] [PATCH v3 0/4] Memory corruption due to HW rings allocation

[Ferruh Yigit]

On 6/3/2020 2:36 AM, Zhao1, Wei wrote:
> I am sure i40e FDIR filter also allocate ring for filter programing, it also has the same issue. Ice NIC the same.
> Although we can avoid this issue by change ring name to "igb_tx_ring" and so on, but the issue of memory for ring not freed after close is still exsit.
> Support this comment. If no one take over all the work for all the PMD, we had better accept this patch.

Agreed.

> 
> 
>> -----Original Message-----
>> From: dev <dev-bounces@dpdk.org> On Behalf Of Renata Saiakhova
>> Sent: Monday, May 18, 2020 5:48 PM
>> To: Yigit, Ferruh <ferruh.yigit@intel.com>; dev@dpdk.org
>> Cc: Burakov, Anatoly <anatoly.burakov@intel.com>; Thomas Monjalon
>> <thomas@monjalon.net>; Neil Horman <nhorman@tuxdriver.com>
>> Subject: Re: [dpdk-dev] [PATCH v3 0/4] Memory corruption due to HW rings
>> allocation
>>
>> Hi Ferruh,
>>
>> thanks for comments,
>>
>> are the rte_eth_dma_zone_reserve() calls always used to allocate HW rings? It
>> is not totally clear to me. That was partly the reason I don't do the fix for every
>> driver which uses this API. I made fixes in the drivers which uses the same
>> pattern to allocate / release queues, for other drivers I was not sure but
>> anyway I couldn't spend more time for further investigations. In the company I
>> work for we use dpdk for our project and maintain it in separate tree, and the
>> vulnerability with HW rings is a real issue for igb and ixgbe drivers and needs to
>> be fixed. Therefore I would like this patch to be accepted in order to not
>> maintain the fix ourselves. But unfortunately I don't have resources (e.g. time)
>> to fix the issue for all the drivers, because, as I mentioned, they are not
>> following the same pattern to release their queues. So my proposal is that I fix
>> it in this patch in a number of drivers (including igb, ixgbe and i40e) and others
>> can take over and improve other drivers, if they see the same issue. This is also
>> a reason why the drivers' changes are not in one commit for all the drivers.
>>
>> For the proposal adding pmd name as prefix to queue memzone name or
>> update the 'rte_eth_dma_zone_reserve()' to check the size & alignment
>> instead of just a name, I don't know, as an external person, how sensitive dpdk
>> project to change an internal API and existing code (the call should be changed
>> in all the drivers). But anyway, I think the real problem is more an absence of
>> memzone pointer, and in long term it should be solved in this way, rather than
>> search by name.
>>
>> Kind regards,
>> Renata
>> ________________________________
>> From: Ferruh Yigit <ferruh.yigit@intel.com>
>> Sent: Wednesday, May 13, 2020 5:22 PM
>> To: Renata Saiakhova <renata.saiakhova@ekinops.com>; dev@dpdk.org
>> <dev@dpdk.org>
>> Cc: Anatoly Burakov <anatoly.burakov@intel.com>; Thomas Monjalon
>> <thomas@monjalon.net>; Neil Horman <nhorman@tuxdriver.com>
>> Subject: Re: [dpdk-dev] [PATCH v3 0/4] Memory corruption due to HW rings
>> allocation
>>
>> On 5/13/2020 2:14 PM, Renata Saiakhova wrote:
>>> igb and ixgbe and some other drivers allocate HW rings using
>>> rte_eth_dma_zone_reserve(), which checks first if the memzone exists
>>> for a given name, consisting of port id, queue_id, rx/tx direction, but not for
>> the size, alignment, and socket_id.
>>> If the memzone with a given name exists it is returned, otherwise it
>>> is allocated.
>>> Disconnecting dpdk port from one type of interface (igb) and
>>> connecting it to another type of interface (ixgbe) for the same port
>>> id, potentially creates memory overlap and corruption, because it may
>> require memzone of bigger size.
>>> That's what is happening from switching from igb to ixgbe having the
>>> same port id.
>>>
>>> v2->v3: Remove #undef ETH_DMA_MZONE_NAME and minor changes in
>> code
>>> v2->standard
>>> v1->v2: Minor changes on code standard and additional fixes in i40e em
>>> v1->and ice drivers
>>>
>>> Renata Saiakhova (4):
>>>   librte_ethdev: Introduce a function to release HW rings
>>>   drivers/net: Fix in igb and ixgbe HW rings memory
>>>   drivers/net: Fix in i40e HW rings memory overlap
>>>   drivers/net: Fix in em and ice HW rings memory overlap
>>
>> I think all driver patches can be squashed into single patch, overall they are
>> implementing same logic.
>>
>> But as mentioned before, there are multiple other drivers allocating HW rings
>> with exact same name. At least I can see:
>> iavf
>> nfp
>> fm10k
>> axgbe
>>
>> Or how can we know if a new PMD won't cause exact same behavior? What to
>> you think adding pmd name as prefix to queue memzone name for all PMDs?
>> This can help new PMDs using existing code as sample.
>>
>> I don't know if it has been discussed before, but wouldn't update the
>> 'rte_eth_dma_zone_reserve()' to check the size & alignment instead of just
>> name fix the issue for all drivers without needing to update them?

Re: [dpdk-dev] [PATCH v3 3/4] drivers/net: Fix in i40e HW rings memory overlap

[Ferruh Yigit]

On 6/1/2020 8:58 AM, Zhao1, Wei wrote:
> Hi, Renata Saiakhova
> 
>    I think this patch is very important, It seems all kind of NIC has memory leak problem that used for store
> Tx or rx descriptor. If that is true, memory point by rxq-> rx_ring/ txq-> tx_ring will never be freed even if dev_close? 
> Is my understanding right or wrong?

That is right as far as I understand, and yes we should free them in dev_close.
I think DPDK relies on that the memory will be freed on the process exit and
sometimes behaves lazy on the memory free as being this case.

> 
> If that is true, it seems you should also add in functioni40e_fdir_teardown(), because
> Tx_ring allocated in i40e_fdir_setup_tx_resources() also need freed, and memzones need to be delete.
> Is that so?
> 
> Thanks.
> 
> 
> 
>> -----Original Message-----
>> From: dev <dev-bounces@dpdk.org> On Behalf Of Renata Saiakhova
>> Sent: Wednesday, May 13, 2020 9:14 PM
>> To: dev@dpdk.org
>> Cc: Renata Saiakhova <Renata.Saiakhova@ekinops.com>
>> Subject: [dpdk-dev] [PATCH v3 3/4] drivers/net: Fix in i40e HW rings memory
>> overlap
>>
>> Delete memzones for HW rings in i40e while freeing queues
>>
>> Signed-off-by: Renata Saiakhova <Renata.Saiakhova@ekinops.com>
>> ---
>>  drivers/net/i40e/i40e_rxtx.c | 2 ++
>>  1 file changed, 2 insertions(+)
>>
>> diff --git a/drivers/net/i40e/i40e_rxtx.c b/drivers/net/i40e/i40e_rxtx.c index
>> 5e7c86ed8..99cec9b99 100644
>> --- a/drivers/net/i40e/i40e_rxtx.c
>> +++ b/drivers/net/i40e/i40e_rxtx.c
>> @@ -2900,6 +2900,7 @@ i40e_dev_free_queues(struct rte_eth_dev *dev)
>>  			continue;
>>  		i40e_dev_rx_queue_release(dev->data->rx_queues[i]);
>>  		dev->data->rx_queues[i] = NULL;
>> +		rte_eth_dma_zone_free(dev, "rx_ring", i);
>>  	}
>>
>>  	for (i = 0; i < dev->data->nb_tx_queues; i++) { @@ -2907,6 +2908,7 @@
>> i40e_dev_free_queues(struct rte_eth_dev *dev)
>>  			continue;
>>  		i40e_dev_tx_queue_release(dev->data->tx_queues[i]);
>>  		dev->data->tx_queues[i] = NULL;
>> +		rte_eth_dma_zone_free(dev, "tx_ring", i);
>>  	}
>>  }
>>
>> --
>> 2.17.2
> 

Re: [dpdk-dev] [PATCH v3 1/4] librte_ethdev: Introduce a function to release HW rings

[Ferruh Yigit]

On 5/15/2020 9:04 AM, Jeff Guo wrote:
> hi, renata
> 
> On 5/13/2020 9:14 PM, Renata Saiakhova wrote:
>> Free previously allocated memzone for HW rings
>>
>> Signed-off-by: Renata Saiakhova <Renata.Saiakhova@ekinops.com>
>> ---
>>   lib/librte_ethdev/rte_ethdev.c           | 28 ++++++++++++++++++++++--
>>   lib/librte_ethdev/rte_ethdev_driver.h    | 20 +++++++++++++++++
>>   lib/librte_ethdev/rte_ethdev_version.map |  1 +
>>   3 files changed, 47 insertions(+), 2 deletions(-)
>>
>> diff --git a/lib/librte_ethdev/rte_ethdev.c b/lib/librte_ethdev/rte_ethdev.c
>> index 72aed59a5..55f047e22 100644
>> --- a/lib/librte_ethdev/rte_ethdev.c
>> +++ b/lib/librte_ethdev/rte_ethdev.c
>> @@ -4181,6 +4181,10 @@ rte_eth_dev_rx_intr_ctl_q_get_fd(uint16_t port_id, uint16_t queue_id)
>>   	return fd;
>>   }
>>   
>> +#define ETH_DMA_MZONE_NAME(_name, _port_id, _queue_id, _ring_name) \
>> +	snprintf(_name, sizeof(_name), "eth_p%d_q%d_%s", \
>> +			_port_id, _queue_id, _ring_name)
>> +
>>   const struct rte_memzone *
>>   rte_eth_dma_zone_reserve(const struct rte_eth_dev *dev, const char *ring_name,
>>   			 uint16_t queue_id, size_t size, unsigned align,
>> @@ -4190,8 +4194,7 @@ rte_eth_dma_zone_reserve(const struct rte_eth_dev *dev, const char *ring_name,
>>   	const struct rte_memzone *mz;
>>   	int rc;
>>   
>> -	rc = snprintf(z_name, sizeof(z_name), "eth_p%d_q%d_%s",
>> -		      dev->data->port_id, queue_id, ring_name);
>> +	rc = ETH_DMA_MZONE_NAME(z_name, dev->data->port_id, queue_id, ring_name);
>>   	if (rc >= RTE_MEMZONE_NAMESIZE) {
>>   		RTE_ETHDEV_LOG(ERR, "ring name too long\n");
>>   		rte_errno = ENAMETOOLONG;
>> @@ -4206,6 +4209,27 @@ rte_eth_dma_zone_reserve(const struct rte_eth_dev *dev, const char *ring_name,
>>   			RTE_MEMZONE_IOVA_CONTIG, align);
>>   }
>>   
>> +int
>> +rte_eth_dma_zone_free(const struct rte_eth_dev *dev, const char *ring_name,
>> +		uint16_t queue_id)
>> +{
>> +	char z_name[RTE_MEMZONE_NAMESIZE];
>> +	const struct rte_memzone *mz;
>> +	int rc = 0;
>> +
>> +	rc = ETH_DMA_MZONE_NAME(z_name, dev->data->port_id, queue_id, ring_name);
>> +	if (rc >= RTE_MEMZONE_NAMESIZE) {
>> +		RTE_ETHDEV_LOG(ERR, "ring name too long\n");
>> +		return -ENAMETOOLONG;
>> +	}
>> +
>> +	mz = rte_memzone_lookup(z_name);
>> +	if (mz)
>> +		rc = rte_memzone_free(mz);
>> +
>> +	return rc;
> 
> 
> If rc <  RTE_MEMZONE_NAMESIZE && !mz, what value should be return base 
> on the parameter description in .h?

+1,

When 'mz' not found return value will be wrong according API documentation.

> 
> Although it would not affect the other patch set, but could you please 
> clarify it?
> 
> 
>> +}
>> +
>>   int
>>   rte_eth_dev_create(struct rte_device *device, const char *name,
>>   	size_t priv_data_size,
>> diff --git a/lib/librte_ethdev/rte_ethdev_driver.h b/lib/librte_ethdev/rte_ethdev_driver.h
>> index 99d4cd6cd..462e765d1 100644
>> --- a/lib/librte_ethdev/rte_ethdev_driver.h
>> +++ b/lib/librte_ethdev/rte_ethdev_driver.h
>> @@ -180,6 +180,26 @@ rte_eth_dma_zone_reserve(const struct rte_eth_dev *eth_dev, const char *name,
>>   			 uint16_t queue_id, size_t size,
>>   			 unsigned align, int socket_id);
>>   
>> +/**
>> + * @warning
>> + * @b EXPERIMENTAL: this API may change without prior notice.
>> + *
>> + * Free previously allocated memzone for HW rings.
>> + *
>> + * @param eth_dev
>> + *   The *eth_dev* pointer is the address of the *rte_eth_dev* structure
>> + * @param name
>> + *   The name of the memory zone
>> + * @param queue_id
>> + *   The index of the queue to add to name
>> + * @return
>> + *   Negative errno value on error, 0 on success.
>> + */
>> +__rte_experimental
>> +int
>> +rte_eth_dma_zone_free(const struct rte_eth_dev *dev, const char *ring_name,
>> +		 uint16_t queue_id);
>> +
>>   /**
>>    * @internal
>>    * Atomically set the link status for the specific device.
>> diff --git a/lib/librte_ethdev/rte_ethdev_version.map b/lib/librte_ethdev/rte_ethdev_version.map
>> index 715505604..139a81302 100644
>> --- a/lib/librte_ethdev/rte_ethdev_version.map
>> +++ b/lib/librte_ethdev/rte_ethdev_version.map
>> @@ -241,4 +241,5 @@ EXPERIMENTAL {
>>   	__rte_ethdev_trace_rx_burst;
>>   	__rte_ethdev_trace_tx_burst;
>>   	rte_flow_get_aged_flows;
>> +	rte_eth_dma_zone_free;
>>   };

Re: [dpdk-dev] [PATCH v3 3/4] drivers/net: Fix in i40e HW rings memory overlap

[Zhao1, Wei]

> -----Original Message-----
> From: Yigit, Ferruh <ferruh.yigit@intel.com>
> Sent: Saturday, June 20, 2020 12:56 AM
> To: Zhao1, Wei <wei.zhao1@intel.com>; Renata Saiakhova
> <Renata.Saiakhova@ekinops.com>; dev@dpdk.org
> Subject: Re: [dpdk-dev] [PATCH v3 3/4] drivers/net: Fix in i40e HW rings
> memory overlap
> 
> On 6/1/2020 8:58 AM, Zhao1, Wei wrote:
> > Hi, Renata Saiakhova
> >
> >    I think this patch is very important, It seems all kind of NIC has
> > memory leak problem that used for store Tx or rx descriptor. If that is true,
> memory point by rxq-> rx_ring/ txq-> tx_ring will never be freed even if
> dev_close?
> > Is my understanding right or wrong?
> 
> That is right as far as I understand, and yes we should free them in dev_close.
> I think DPDK relies on that the memory will be freed on the process exit and
> sometimes behaves lazy on the memory free as being this case.
> 

Thanks! Hope for accept for this patch.

> >
> > If that is true, it seems you should also add in
> > functioni40e_fdir_teardown(), because Tx_ring allocated in
> i40e_fdir_setup_tx_resources() also need freed, and memzones need to be
> delete.
> > Is that so?
> >
> > Thanks.
> >
> >
> >
> >> -----Original Message-----
> >> From: dev <dev-bounces@dpdk.org> On Behalf Of Renata Saiakhova
> >> Sent: Wednesday, May 13, 2020 9:14 PM
> >> To: dev@dpdk.org
> >> Cc: Renata Saiakhova <Renata.Saiakhova@ekinops.com>
> >> Subject: [dpdk-dev] [PATCH v3 3/4] drivers/net: Fix in i40e HW rings
> >> memory overlap
> >>
> >> Delete memzones for HW rings in i40e while freeing queues
> >>
> >> Signed-off-by: Renata Saiakhova <Renata.Saiakhova@ekinops.com>
> >> ---
> >>  drivers/net/i40e/i40e_rxtx.c | 2 ++
> >>  1 file changed, 2 insertions(+)
> >>
> >> diff --git a/drivers/net/i40e/i40e_rxtx.c
> >> b/drivers/net/i40e/i40e_rxtx.c index
> >> 5e7c86ed8..99cec9b99 100644
> >> --- a/drivers/net/i40e/i40e_rxtx.c
> >> +++ b/drivers/net/i40e/i40e_rxtx.c
> >> @@ -2900,6 +2900,7 @@ i40e_dev_free_queues(struct rte_eth_dev *dev)
> >>  			continue;
> >>  		i40e_dev_rx_queue_release(dev->data->rx_queues[i]);
> >>  		dev->data->rx_queues[i] = NULL;
> >> +		rte_eth_dma_zone_free(dev, "rx_ring", i);
> >>  	}
> >>
> >>  	for (i = 0; i < dev->data->nb_tx_queues; i++) { @@ -2907,6 +2908,7
> >> @@ i40e_dev_free_queues(struct rte_eth_dev *dev)
> >>  			continue;
> >>  		i40e_dev_tx_queue_release(dev->data->tx_queues[i]);
> >>  		dev->data->tx_queues[i] = NULL;
> >> +		rte_eth_dma_zone_free(dev, "tx_ring", i);
> >>  	}
> >>  }
> >>
> >> --
> >> 2.17.2
> >

Re: [dpdk-dev] [PATCH v3 0/4] Memory corruption due to HW rings allocation

[Zhao1, Wei]

Reviewed-by: Wei Zhao <wei.zhao1@intel.com>


> -----Original Message-----
> From: dev <dev-bounces@dpdk.org> On Behalf Of Renata Saiakhova
> Sent: Wednesday, May 13, 2020 9:14 PM
> To: dev@dpdk.org
> Cc: Renata Saiakhova <Renata.Saiakhova@ekinops.com>
> Subject: [dpdk-dev] [PATCH v3 0/4] Memory corruption due to HW rings
> allocation
> 
> igb and ixgbe and some other drivers allocate HW rings using
> rte_eth_dma_zone_reserve(), which checks first if the memzone exists for a
> given name, consisting of port id, queue_id, rx/tx direction, but not for the size,
> alignment, and socket_id.
> If the memzone with a given name exists it is returned, otherwise it is
> allocated.
> Disconnecting dpdk port from one type of interface (igb) and connecting it to
> another type of interface (ixgbe) for the same port id, potentially creates
> memory overlap and corruption, because it may require memzone of bigger
> size.
> That's what is happening from switching from igb to ixgbe having the same port
> id.
> 
> v2->v3: Remove #undef ETH_DMA_MZONE_NAME and minor changes in code
> v2->standard
> v1->v2: Minor changes on code standard and additional fixes in i40e em
> v1->and ice drivers
> 
> Renata Saiakhova (4):
>   librte_ethdev: Introduce a function to release HW rings
>   drivers/net: Fix in igb and ixgbe HW rings memory
>   drivers/net: Fix in i40e HW rings memory overlap
>   drivers/net: Fix in em and ice HW rings memory overlap
> 
>  drivers/net/e1000/em_rxtx.c              |  2 ++
>  drivers/net/e1000/igb_rxtx.c             |  2 ++
>  drivers/net/i40e/i40e_rxtx.c             |  2 ++
>  drivers/net/ice/ice_rxtx.c               |  2 ++
>  drivers/net/ixgbe/ixgbe_rxtx.c           |  2 ++
>  lib/librte_ethdev/rte_ethdev.c           | 28 ++++++++++++++++++++++--
>  lib/librte_ethdev/rte_ethdev_driver.h    | 20 +++++++++++++++++
>  lib/librte_ethdev/rte_ethdev_version.map |  1 +
>  8 files changed, 57 insertions(+), 2 deletions(-)
> 
> --
> 2.17.2

Re: [dpdk-dev] [PATCH v3 0/4] Memory corruption due to HW rings allocation

[Renata Saiakhova]

Hi Ferruh,

I've just send a new corrected version of the patchset, which takes into 
account the comments of Wie and Jeff, hope it is OK now.

Kind regards and thank you,

Renata


On 06/19/2020 06:54 PM, Ferruh Yigit wrote:
> On 5/18/2020 10:48 AM, Renata Saiakhova wrote:
>> Hi Ferruh,
>>
>> thanks for comments,
>>
>> are the rte_eth_dma_zone_reserve() calls always used to allocate HW rings? It is
>> not totally clear to me. That was partly the reason I don't do the fix for every
>> driver which uses this API. I made fixes in the drivers which uses the same
>> pattern to allocate / release queues, for other drivers I was not sure but
>> anyway I couldn't spend more time for further investigations. In the company I
>> work for we use dpdk for our project and maintain it in separate tree, and the
>> vulnerability with HW rings is a real issue for igb and ixgbe drivers and needs
>> to be fixed. Therefore I would like this patch to be accepted in order to not
>> maintain the fix ourselves. But unfortunately I don't have resources (e.g. time)
>> to fix the issue for all the drivers, because, as I mentioned, they are not
>> following the same pattern to release their queues. So my proposal is that I fix
>> it in this patch in a number of drivers (including igb, ixgbe and i40e) and
>> others can take over and improve other drivers, if they see the same issue. This
>> is also a reason why the drivers' changes are not in one commit for all the drivers.
>>
>> For the proposal adding pmd name as prefix to queue memzone name or update
>> the 'rte_eth_dma_zone_reserve()' to check the size & alignment instead of just a
>> name, I don't know, as an external person, how sensitive dpdk project to change
>> an internal API and existing code (the call should be changed in all the
>> drivers). But anyway, I think the real problem is more an absence of memzone
>> pointer, and in long term it should be solved in this way, rather than search by
>> name.
> Hi Renata,
>
> 'rte_eth_dma_zone_reserve()' returning existing memzone just based on 'name'
> without checking the size, alignment, socket_id seems wrong to me, let me check
> it, it shouldn't be hard to update.
>
> But even 'rte_eth_dma_zone_reserve()' checks won't be enough (what to do if
> check fails and not returns a memzone?), the proper solution is PMD free the
> memzone as they removed, as done in this patch.
>
> Also I suggested prefixing the memzone name as workaround but instead we can go
> with this patchset with the implemented PMDs and other PMD owners can implement
> the free when they need instead of workaround.
>
> There is a comment from Wie to add more frees to i40e, and there is another from
> Jeff on 'rte_eth_dma_zone_free()' return type, can you please check them? So we
> can proceed with this patch for the release.
>
> Thanks,
> ferruh
>
>> Kind regards,
>> Renata
>> --------------------------------------------------------------------------------
>> *From:* Ferruh Yigit <ferruh.yigit@intel.com>
>> *Sent:* Wednesday, May 13, 2020 5:22 PM
>> *To:* Renata Saiakhova <renata.saiakhova@ekinops.com>; dev@dpdk.org <dev@dpdk.org>
>> *Cc:* Anatoly Burakov <anatoly.burakov@intel.com>; Thomas Monjalon
>> <thomas@monjalon.net>; Neil Horman <nhorman@tuxdriver.com>
>> *Subject:* Re: [dpdk-dev] [PATCH v3 0/4] Memory corruption due to HW rings
>> allocation
>>   
>> On 5/13/2020 2:14 PM, Renata Saiakhova wrote:
>>> igb and ixgbe and some other drivers allocate HW rings using rte_eth_dma_zone_reserve(),
>>> which checks first if the memzone exists for a given name, consisting of port
>>> id, queue_id, rx/tx direction, but not for the size, alignment, and socket_id.
>>> If the memzone with a given name exists it is returned, otherwise it is
>>> allocated.
>>> Disconnecting dpdk port from one type of interface (igb) and connecting it
>>> to another type of interface (ixgbe) for the same port id, potentially creates
>>> memory overlap and corruption, because it may require memzone of bigger size.
>>> That's what is happening from switching from igb to ixgbe having the same port
>>> id.
>>>
>>> v2->v3: Remove #undef ETH_DMA_MZONE_NAME and minor changes in code standard
>>> v1->v2: Minor changes on code standard and additional fixes in i40e em and ice drivers
>>>
>>> Renata Saiakhova (4):
>>>     librte_ethdev: Introduce a function to release HW rings
>>>     drivers/net: Fix in igb and ixgbe HW rings memory
>>>     drivers/net: Fix in i40e HW rings memory overlap
>>>     drivers/net: Fix in em and ice HW rings memory overlap
>> I think all driver patches can be squashed into single patch, overall they are
>> implementing same logic.
>>
>> But as mentioned before, there are multiple other drivers allocating HW rings
>> with exact same name. At least I can see:
>> iavf
>> nfp
>> fm10k
>> axgbe
>>
>> Or how can we know if a new PMD won't cause exact same behavior? What to you
>> think adding pmd name as prefix to queue memzone name for all PMDs? This can
>> help new PMDs using existing code as sample.
>>
>> I don't know if it has been discussed before, but wouldn't update the
>> 'rte_eth_dma_zone_reserve()' to check the size & alignment instead of just name
>> fix the issue for all drivers without needing to update them?