[dpdk-dev] [PATCH] fm10k: fix switch manager high CPU usage

[dpdk-dev] [PATCH] fm10k: fix switch manager high CPU usage

[Shaopeng He]

fm10k switch core uses source MAC + VID + SGLORT to do
look up in MAC table. If no match, an exception interrupt
will be sent to the switch manager, and cause high CPU
usage.
This patch fixes this issue. A default SGLORT is assigned
to each TX queue. This default value works for non-VMDq mode
and current VMDq example. For advanced VMDq usage, e.g.
different source MAC address for different TX queue, FTAG
forwarding function could be used to change this default
SGLORT value.

Signed-off-by: Shaopeng He <shaopeng.he@intel.com>
---
 drivers/net/fm10k/fm10k_ethdev.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/net/fm10k/fm10k_ethdev.c b/drivers/net/fm10k/fm10k_ethdev.c
index e4aed94..f6eb05d 100644
--- a/drivers/net/fm10k/fm10k_ethdev.c
+++ b/drivers/net/fm10k/fm10k_ethdev.c
@@ -675,6 +675,9 @@ fm10k_dev_tx_init(struct rte_eth_dev *dev)
 		FM10K_WRITE_REG(hw, FM10K_TDBAH(i),
 				base_addr >> (CHAR_BIT * sizeof(uint32_t)));
 		FM10K_WRITE_REG(hw, FM10K_TDLEN(i), size);
+
+		/* assign default SGLORT for each TX queue */
+		FM10K_WRITE_REG(hw, FM10K_TX_SGLORT(i), hw->mac.dglort_map);
 	}
 
 	/* set up vector or scalar TX function as appropriate */
-- 
1.9.3

Re: [dpdk-dev] [PATCH] fm10k: fix switch manager high CPU usage

[Chen, Jing D]

Hi,


Best Regards,
Mark


> -----Original Message-----
> From: He, Shaopeng
> Sent: Thursday, January 28, 2016 1:47 PM
> To: dev@dpdk.org
> Cc: Chen, Jing D; Wang, Xiao W; He, Shaopeng
> Subject: [PATCH] fm10k: fix switch manager high CPU usage
> 
> fm10k switch core uses source MAC + VID + SGLORT to do
> look up in MAC table. If no match, an exception interrupt
> will be sent to the switch manager, and cause high CPU
> usage.

Above paragraph didn't describe the bug clearly. Can you add more
Words on it?

> This patch fixes this issue. A default SGLORT is assigned
> to each TX queue. This default value works for non-VMDq mode
> and current VMDq example. For advanced VMDq usage, e.g.
> different source MAC address for different TX queue, FTAG
> forwarding function could be used to change this default
> SGLORT value.
> 
> Signed-off-by: Shaopeng He <shaopeng.he@intel.com>
> ---
>  drivers/net/fm10k/fm10k_ethdev.c | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/drivers/net/fm10k/fm10k_ethdev.c
> b/drivers/net/fm10k/fm10k_ethdev.c
> index e4aed94..f6eb05d 100644
> --- a/drivers/net/fm10k/fm10k_ethdev.c
> +++ b/drivers/net/fm10k/fm10k_ethdev.c
> @@ -675,6 +675,9 @@ fm10k_dev_tx_init(struct rte_eth_dev *dev)
>  		FM10K_WRITE_REG(hw, FM10K_TDBAH(i),
>  				base_addr >> (CHAR_BIT * sizeof(uint32_t)));
>  		FM10K_WRITE_REG(hw, FM10K_TDLEN(i), size);
> +
> +		/* assign default SGLORT for each TX queue */
> +		FM10K_WRITE_REG(hw, FM10K_TX_SGLORT(i), hw-
> >mac.dglort_map);
>  	}
> 
>  	/* set up vector or scalar TX function as appropriate */
> --
> 1.9.3

Re: [dpdk-dev] [PATCH] fm10k: fix switch manager high CPU usage

[He, Shaopeng]

Hi Mark,

> -----Original Message-----
> From: Chen, Jing D
> Sent: Wednesday, February 03, 2016 3:23 PM
> To: He, Shaopeng; dev@dpdk.org
> Cc: Wang, Xiao W
> Subject: RE: [PATCH] fm10k: fix switch manager high CPU usage
> 
> Hi,
> 
> 
> Best Regards,
> Mark
> 
> 
> > -----Original Message-----
> > From: He, Shaopeng
> > Sent: Thursday, January 28, 2016 1:47 PM
> > To: dev@dpdk.org
> > Cc: Chen, Jing D; Wang, Xiao W; He, Shaopeng
> > Subject: [PATCH] fm10k: fix switch manager high CPU usage
> >
> > fm10k switch core uses source MAC + VID + SGLORT to do look up in MAC
> > table. If no match, an exception interrupt will be sent to the switch
> > manager, and cause high CPU usage.
> 
> Above paragraph didn't describe the bug clearly. Can you add more Words on
> it?

Thanks for the comments, I will send a v2.

> 
> > This patch fixes this issue. A default SGLORT is assigned to each TX
> > queue. This default value works for non-VMDq mode and current VMDq
> > example. For advanced VMDq usage, e.g.
> > different source MAC address for different TX queue, FTAG forwarding
> > function could be used to change this default SGLORT value.
> >
> > Signed-off-by: Shaopeng He <shaopeng.he@intel.com>
> > ---
> >  drivers/net/fm10k/fm10k_ethdev.c | 3 +++
> >  1 file changed, 3 insertions(+)
> >
> > diff --git a/drivers/net/fm10k/fm10k_ethdev.c
> > b/drivers/net/fm10k/fm10k_ethdev.c
> > index e4aed94..f6eb05d 100644
> > --- a/drivers/net/fm10k/fm10k_ethdev.c
> > +++ b/drivers/net/fm10k/fm10k_ethdev.c
> > @@ -675,6 +675,9 @@ fm10k_dev_tx_init(struct rte_eth_dev *dev)
> >  		FM10K_WRITE_REG(hw, FM10K_TDBAH(i),
> >  				base_addr >> (CHAR_BIT * sizeof(uint32_t)));
> >  		FM10K_WRITE_REG(hw, FM10K_TDLEN(i), size);
> > +
> > +		/* assign default SGLORT for each TX queue */
> > +		FM10K_WRITE_REG(hw, FM10K_TX_SGLORT(i), hw-
> > >mac.dglort_map);
> >  	}
> >
> >  	/* set up vector or scalar TX function as appropriate */
> > --
> > 1.9.3

[dpdk-dev] [PATCH v2] fm10k: fix switch manager high CPU usage

[Shaopeng He]

fm10k switch core uses source MAC + VID + SGLORT to do
look up in MAC table. If no match, an exception interrupt
will be sent to the switch manager. Too much of this kind
of exception interrupts cause switch manager side high CPU
usage.
To reproduce this issue, one DPDK testpmd runs on a server
with one fm10k NIC, mac forwards test traffic from one of
fm10k ports to another port. The CPU usage for the switch
manager will go up to about 20% for test traffic rate at
10G bps, comparing to near 0% for no test traffic.
This patch fixes this issue. A default SGLORT is assigned
to each TX queue. This default value works for non-VMDq mode
and current VMDq example. For advanced VMDq usage, e.g.
different source MAC address for different TX queue, FTAG
forwarding function could be used to change this default
SGLORT value.

Signed-off-by: Shaopeng He <shaopeng.he@intel.com>
---
 drivers/net/fm10k/fm10k_ethdev.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/net/fm10k/fm10k_ethdev.c b/drivers/net/fm10k/fm10k_ethdev.c
index e4aed94..f6eb05d 100644
--- a/drivers/net/fm10k/fm10k_ethdev.c
+++ b/drivers/net/fm10k/fm10k_ethdev.c
@@ -675,6 +675,9 @@ fm10k_dev_tx_init(struct rte_eth_dev *dev)
 		FM10K_WRITE_REG(hw, FM10K_TDBAH(i),
 				base_addr >> (CHAR_BIT * sizeof(uint32_t)));
 		FM10K_WRITE_REG(hw, FM10K_TDLEN(i), size);
+
+		/* assign default SGLORT for each TX queue */
+		FM10K_WRITE_REG(hw, FM10K_TX_SGLORT(i), hw->mac.dglort_map);
 	}
 
 	/* set up vector or scalar TX function as appropriate */
-- 
1.9.3

[dpdk-dev] [PATCH v3] fm10k: fix switch manager high CPU usage

[Shaopeng He]

fm10k switch core uses source MAC + VID + SGLORT to do
look up in MAC table. If no match, an exception interrupt
will be sent to the switch manager. Too much of this kind
of exception interrupts cause switch manager side high CPU
usage.
To reproduce this issue, one DPDK testpmd runs on a server
with one fm10k NIC, mac forwards test traffic from one of
fm10k ports to another port. The CPU usage for the switch
manager will go up to about 20% for test traffic rate at
10G bps, comparing to near 0% for no test traffic.
This patch fixes this issue. A default SGLORT is assigned
to each TX queue. This default value works for non-VMDq mode
and current VMDq example. For advanced VMDq usage, e.g.
different source MAC address for different TX queue, FTAG
forwarding function could be used to change this default
SGLORT value.

Fixes: 9ae6068c ("fm10k: add dev start/stop")

Signed-off-by: Shaopeng He <shaopeng.he@intel.com>
---
v3: - add fixes line
v2: - reword the commit log message
---
 drivers/net/fm10k/fm10k_ethdev.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/net/fm10k/fm10k_ethdev.c b/drivers/net/fm10k/fm10k_ethdev.c
index e4aed94..f6eb05d 100644
--- a/drivers/net/fm10k/fm10k_ethdev.c
+++ b/drivers/net/fm10k/fm10k_ethdev.c
@@ -675,6 +675,9 @@ fm10k_dev_tx_init(struct rte_eth_dev *dev)
 		FM10K_WRITE_REG(hw, FM10K_TDBAH(i),
 				base_addr >> (CHAR_BIT * sizeof(uint32_t)));
 		FM10K_WRITE_REG(hw, FM10K_TDLEN(i), size);
+
+		/* assign default SGLORT for each TX queue */
+		FM10K_WRITE_REG(hw, FM10K_TX_SGLORT(i), hw->mac.dglort_map);
 	}
 
 	/* set up vector or scalar TX function as appropriate */
-- 
1.9.3

Re: [dpdk-dev] [PATCH v2] fm10k: fix switch manager high CPU usage

[Chen, Jing D]

Hi,

Best Regards,
Mark


> -----Original Message-----
> From: He, Shaopeng
> Sent: Thursday, February 04, 2016 8:45 PM
> To: dev@dpdk.org
> Cc: Chen, Jing D; Wang, Xiao W; He, Shaopeng
> Subject: [PATCH v2] fm10k: fix switch manager high CPU usage
> 
> fm10k switch core uses source MAC + VID + SGLORT to do
> look up in MAC table. If no match, an exception interrupt
> will be sent to the switch manager. Too much of this kind
> of exception interrupts cause switch manager side high CPU
> usage.
> To reproduce this issue, one DPDK testpmd runs on a server
> with one fm10k NIC, mac forwards test traffic from one of
> fm10k ports to another port. The CPU usage for the switch
> manager will go up to about 20% for test traffic rate at
> 10G bps, comparing to near 0% for no test traffic.
> This patch fixes this issue. A default SGLORT is assigned
> to each TX queue. This default value works for non-VMDq mode
> and current VMDq example. For advanced VMDq usage, e.g.
> different source MAC address for different TX queue, FTAG
> forwarding function could be used to change this default
> SGLORT value.
> 
> Signed-off-by: Shaopeng He <shaopeng.he@intel.com>
Acked-by: Jing Chen <jing.d.chen@intel.com>

Re: [dpdk-dev] [PATCH v3] fm10k: fix switch manager high CPU usage

[Chen, Jing D]

Hi,

Best Regards,
Mark


> -----Original Message-----
> From: He, Shaopeng
> Sent: Friday, February 05, 2016 10:46 AM
> To: dev@dpdk.org
> Cc: Chen, Jing D; Wang, Xiao W; He, Shaopeng
> Subject: [PATCH v3] fm10k: fix switch manager high CPU usage
> 
> fm10k switch core uses source MAC + VID + SGLORT to do
> look up in MAC table. If no match, an exception interrupt
> will be sent to the switch manager. Too much of this kind
> of exception interrupts cause switch manager side high CPU
> usage.
> To reproduce this issue, one DPDK testpmd runs on a server
> with one fm10k NIC, mac forwards test traffic from one of
> fm10k ports to another port. The CPU usage for the switch
> manager will go up to about 20% for test traffic rate at
> 10G bps, comparing to near 0% for no test traffic.
> This patch fixes this issue. A default SGLORT is assigned
> to each TX queue. This default value works for non-VMDq mode
> and current VMDq example. For advanced VMDq usage, e.g.
> different source MAC address for different TX queue, FTAG
> forwarding function could be used to change this default
> SGLORT value.
> 
> Fixes: 9ae6068c ("fm10k: add dev start/stop")
> 
> Signed-off-by: Shaopeng He <shaopeng.he@intel.com>
Acked-by : Jing Chen <jing.d.chen@intel.com>

Re: [dpdk-dev] [PATCH v3] fm10k: fix switch manager high CPU usage

[Bruce Richardson]

On Tue, Feb 16, 2016 at 08:34:21AM +0000, Chen, Jing D wrote:
> Hi,
> 
> Best Regards,
> Mark
> 
> 
> > -----Original Message-----
> > From: He, Shaopeng
> > Sent: Friday, February 05, 2016 10:46 AM
> > To: dev@dpdk.org
> > Cc: Chen, Jing D; Wang, Xiao W; He, Shaopeng
> > Subject: [PATCH v3] fm10k: fix switch manager high CPU usage
> > 
> > fm10k switch core uses source MAC + VID + SGLORT to do
> > look up in MAC table. If no match, an exception interrupt
> > will be sent to the switch manager. Too much of this kind
> > of exception interrupts cause switch manager side high CPU
> > usage.
> > To reproduce this issue, one DPDK testpmd runs on a server
> > with one fm10k NIC, mac forwards test traffic from one of
> > fm10k ports to another port. The CPU usage for the switch
> > manager will go up to about 20% for test traffic rate at
> > 10G bps, comparing to near 0% for no test traffic.
> > This patch fixes this issue. A default SGLORT is assigned
> > to each TX queue. This default value works for non-VMDq mode
> > and current VMDq example. For advanced VMDq usage, e.g.
> > different source MAC address for different TX queue, FTAG
> > forwarding function could be used to change this default
> > SGLORT value.
> > 
> > Fixes: 9ae6068c ("fm10k: add dev start/stop")
> > 
> > Signed-off-by: Shaopeng He <shaopeng.he@intel.com>
> Acked-by : Jing Chen <jing.d.chen@intel.com>

Applied to dpdk-next-net/rel_16_04

Thanks,
/Bruce