[dpdk-dev] [PATCH] fm10k: conditionally disable RSS during device initialization

[dpdk-dev] [PATCH] fm10k: conditionally disable RSS during device initialization

[Michael Frasca]

If the provided configuration does not call for RSS, then RSS is
explicitly disabled. Without this change, the device continues to
operate under the previous RSS configuration.

Fixes: 57033cdf8fdc ("fm10k: add PF RSS")

Signed-off-by: Michael Frasca <michael.frasca@oracle.com>
---
 drivers/net/fm10k/fm10k_ethdev.c | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git a/drivers/net/fm10k/fm10k_ethdev.c b/drivers/net/fm10k/fm10k_ethdev.c
index 4b07a8b..4d5365a 100644
--- a/drivers/net/fm10k/fm10k_ethdev.c
+++ b/drivers/net/fm10k/fm10k_ethdev.c
@@ -531,8 +531,10 @@ fm10k_dev_rss_configure(struct rte_eth_dev *dev)
 
 	if (dev->data->nb_rx_queues == 1 ||
 	    dev_conf->rxmode.mq_mode != ETH_MQ_RX_RSS ||
-	    dev_conf->rx_adv_conf.rss_conf.rss_hf == 0)
+	    dev_conf->rx_adv_conf.rss_conf.rss_hf == 0) {
+		FM10K_WRITE_REG(hw, FM10K_MRQC(0), 0);
 		return;
+	}
 
 	/* random key is rss_intel_key (default) or user provided (rss_key) */
 	if (dev_conf->rx_adv_conf.rss_conf.rss_key == NULL)
-- 
2.5.0

Re: [dpdk-dev] [PATCH] fm10k: conditionally disable RSS during device initialization

[Chen, Jing D]

Hi,

> -----Original Message-----
> From: Michael Frasca [mailto:michael.frasca@oracle.com]
> Sent: Wednesday, March 23, 2016 12:58 AM
> To: Chen, Jing D
> Cc: dev@dpdk.org; Michael Frasca
> Subject: [PATCH] fm10k: conditionally disable RSS during device initialization
> 
> If the provided configuration does not call for RSS, then RSS is
> explicitly disabled. Without this change, the device continues to
> operate under the previous RSS configuration.
> 
> Fixes: 57033cdf8fdc ("fm10k: add PF RSS")
> 
> Signed-off-by: Michael Frasca <michael.frasca@oracle.com>
Acked-by : Jing Chen <jing.d.chen@intel.com>

Re: [dpdk-dev] [PATCH] fm10k: conditionally disable RSS during device initialization

[Michael Frasca]

Hi Jing,

I ran into this issue while trying to run experiments with different RSS
configurations (no RSS being one cases). It is not clear to me that setting this
register to zero is the best way to disable RSS.

After digging further, I have a theory that I'm having this issues because I've
only attached my DPDK application to SR-IOV ports. In
fm10k_dev_dglort_map_configure(), I see that 'RSS Length' is set for the DGLORT
decoder. However, it appears that this is only invoked for physical functions.

Could this be my problem? Is it required that I bind to the physical function
if I want to properly manipulate RSS? 

Thanks,
Mike



> On Mar 22, 2016, at 11:14 PM, Chen, Jing D <jing.d.chen@intel.com> wrote:
> 
> Hi,
> 
>> -----Original Message-----
>> From: Michael Frasca [mailto:michael.frasca@oracle.com]
>> Sent: Wednesday, March 23, 2016 12:58 AM
>> To: Chen, Jing D
>> Cc: dev@dpdk.org; Michael Frasca
>> Subject: [PATCH] fm10k: conditionally disable RSS during device initialization
>> 
>> If the provided configuration does not call for RSS, then RSS is
>> explicitly disabled. Without this change, the device continues to
>> operate under the previous RSS configuration.
>> 
>> Fixes: 57033cdf8fdc ("fm10k: add PF RSS")
>> 
>> Signed-off-by: Michael Frasca <michael.frasca@oracle.com>
> Acked-by : Jing Chen <jing.d.chen@intel.com>
> 

Re: [dpdk-dev] [PATCH] fm10k: conditionally disable RSS during device initialization

[Chen, Jing D]

Hi, Frasca,



> -----Original Message-----
> From: Michael Frasca [mailto:michael.frasca@oracle.com]
> Sent: Wednesday, March 23, 2016 9:43 PM
> To: Chen, Jing D
> Cc: dev@dpdk.org
> Subject: Re: [PATCH] fm10k: conditionally disable RSS during device
> initialization
> 
> Hi Jing,
> 
> I ran into this issue while trying to run experiments with different RSS
> configurations (no RSS being one cases). It is not clear to me that setting this
> register to zero is the best way to disable RSS.
> 
> After digging further, I have a theory that I'm having this issues because I've
> only attached my DPDK application to SR-IOV ports. In
> fm10k_dev_dglort_map_configure(), I see that 'RSS Length' is set for the
> DGLORT
> decoder. However, it appears that this is only invoked for physical functions.
> 
> Could this be my problem? Is it required that I bind to the physical function
> if I want to properly manipulate RSS?
> 
> Thanks,
> Mike
> 
I don't know exactly what problem you ran into. I think we needn't worry about 
those DGLORT setting when using VF device.

I've followed steps to use SRIOV device with RSS enabled and disabled, both
are worked well from my side after applying your patch. Below is my setup.

1. PF with Linux driver "fm10k-next_0.19.3".
2. DPDK with latest code from master branch, apply your patch.
3. Use 1 VF device created by kernel driver.
4. use l3fwd with " ./examples/l3fwd/build/l3fwd -c fc -n 4 -- -p 0x1 --config="(0,0,2),(0,1,2),(0,2,3),(0,3,3)""
    with RSS enabled. After sending packets, I can see all 4 queues received packets.
5. use l3fwd with " ./examples/l3fwd/build/l3fwd -c fc -n 4 -- -p 0x1 --config="(0,0,2)""
    with RSS disabled. After sending packets, I can see queue 0 received packets.

Can you explain what actual problem is?
We can talk offline.

Re: [dpdk-dev] [PATCH] fm10k: conditionally disable RSS during device initialization

[Michael Frasca]

Jing,

Thanks for your assistance. The experiment that you have built should allow you
to observe the bug. In [5], I would expect that queue 0 receives roughly 1/4 of
the packets that you sending, assuming the input packets have varied IP
addresses. Can you measure what % of packets are actually being received in this
single queue setup (after first running a 4-queue setup)?

When trying to running with only one RX queue, the fm10k retains the same RSS
hash function and redirection table that was configured from a previous run. As
a result, some packets are still being directed to other receive queues. I have
confirmed this by polling the queue specific stats, which I retrieved via
rte_eth_xstats_get().

Looking at fm10k_dev_rss_configure(), one should see that there is no
modification of fm10k registers when nb_rx_queues == 1. As far as I can tell,
this is the reason that only a certain partition of packets are being receive in
a single queue setup (after first running a multi-queue configuration).

I am unable to access my development environment today, but if you need, I can
later craft a patch to l3fwd that shows the measurement of packets received at
each queue.

Thanks,
Mike


> On Mar 24, 2016, at 2:40 AM, Chen, Jing D <jing.d.chen@intel.com> wrote:
> 
> Hi, Frasca,
> 
> 
> 
>> -----Original Message-----
>> From: Michael Frasca [mailto:michael.frasca@oracle.com <mailto:michael.frasca@oracle.com>]
>> Sent: Wednesday, March 23, 2016 9:43 PM
>> To: Chen, Jing D
>> Cc: dev@dpdk.org <mailto:dev@dpdk.org>
>> Subject: Re: [PATCH] fm10k: conditionally disable RSS during device
>> initialization
>> 
>> Hi Jing,
>> 
>> I ran into this issue while trying to run experiments with different RSS
>> configurations (no RSS being one cases). It is not clear to me that setting this
>> register to zero is the best way to disable RSS.
>> 
>> After digging further, I have a theory that I'm having this issues because I've
>> only attached my DPDK application to SR-IOV ports. In
>> fm10k_dev_dglort_map_configure(), I see that 'RSS Length' is set for the
>> DGLORT
>> decoder. However, it appears that this is only invoked for physical functions.
>> 
>> Could this be my problem? Is it required that I bind to the physical function
>> if I want to properly manipulate RSS?
>> 
>> Thanks,
>> Mike
>> 
> I don't know exactly what problem you ran into. I think we needn't worry about 
> those DGLORT setting when using VF device.
> 
> I've followed steps to use SRIOV device with RSS enabled and disabled, both
> are worked well from my side after applying your patch. Below is my setup.
> 
> 1. PF with Linux driver "fm10k-next_0.19.3".
> 2. DPDK with latest code from master branch, apply your patch.
> 3. Use 1 VF device created by kernel driver.
> 4. use l3fwd with " ./examples/l3fwd/build/l3fwd -c fc -n 4 -- -p 0x1 --config="(0,0,2),(0,1,2),(0,2,3),(0,3,3)""
>    with RSS enabled. After sending packets, I can see all 4 queues received packets.
> 5. use l3fwd with " ./examples/l3fwd/build/l3fwd -c fc -n 4 -- -p 0x1 --config="(0,0,2)""
>    with RSS disabled. After sending packets, I can see queue 0 received packets.
> 
> Can you explain what actual problem is?
> We can talk offline.

Re: [dpdk-dev] [PATCH] fm10k: conditionally disable RSS during device initialization

[Thomas Monjalon]

Please, anyone to confirm that the patch is valid and must be applied?
This discussion shows some doubts.


2016-03-24 13:35, Michael Frasca:
> Jing,
> 
> Thanks for your assistance. The experiment that you have built should allow you
> to observe the bug. In [5], I would expect that queue 0 receives roughly 1/4 of
> the packets that you sending, assuming the input packets have varied IP
> addresses. Can you measure what % of packets are actually being received in this
> single queue setup (after first running a 4-queue setup)?
> 
> When trying to running with only one RX queue, the fm10k retains the same RSS
> hash function and redirection table that was configured from a previous run. As
> a result, some packets are still being directed to other receive queues. I have
> confirmed this by polling the queue specific stats, which I retrieved via
> rte_eth_xstats_get().
> 
> Looking at fm10k_dev_rss_configure(), one should see that there is no
> modification of fm10k registers when nb_rx_queues == 1. As far as I can tell,
> this is the reason that only a certain partition of packets are being receive in
> a single queue setup (after first running a multi-queue configuration).
> 
> I am unable to access my development environment today, but if you need, I can
> later craft a patch to l3fwd that shows the measurement of packets received at
> each queue.
> 
> Thanks,
> Mike
> 
> 
> > On Mar 24, 2016, at 2:40 AM, Chen, Jing D <jing.d.chen@intel.com> wrote:
> > 
> > Hi, Frasca,
> > 
> > 
> > 
> >> -----Original Message-----
> >> From: Michael Frasca [mailto:michael.frasca@oracle.com <mailto:michael.frasca@oracle.com>]
> >> Sent: Wednesday, March 23, 2016 9:43 PM
> >> To: Chen, Jing D
> >> Cc: dev@dpdk.org <mailto:dev@dpdk.org>
> >> Subject: Re: [PATCH] fm10k: conditionally disable RSS during device
> >> initialization
> >> 
> >> Hi Jing,
> >> 
> >> I ran into this issue while trying to run experiments with different RSS
> >> configurations (no RSS being one cases). It is not clear to me that setting this
> >> register to zero is the best way to disable RSS.
> >> 
> >> After digging further, I have a theory that I'm having this issues because I've
> >> only attached my DPDK application to SR-IOV ports. In
> >> fm10k_dev_dglort_map_configure(), I see that 'RSS Length' is set for the
> >> DGLORT
> >> decoder. However, it appears that this is only invoked for physical functions.
> >> 
> >> Could this be my problem? Is it required that I bind to the physical function
> >> if I want to properly manipulate RSS?
> >> 
> >> Thanks,
> >> Mike
> >> 
> > I don't know exactly what problem you ran into. I think we needn't worry about 
> > those DGLORT setting when using VF device.
> > 
> > I've followed steps to use SRIOV device with RSS enabled and disabled, both
> > are worked well from my side after applying your patch. Below is my setup.
> > 
> > 1. PF with Linux driver "fm10k-next_0.19.3".
> > 2. DPDK with latest code from master branch, apply your patch.
> > 3. Use 1 VF device created by kernel driver.
> > 4. use l3fwd with " ./examples/l3fwd/build/l3fwd -c fc -n 4 -- -p 0x1 --config="(0,0,2),(0,1,2),(0,2,3),(0,3,3)""
> >    with RSS enabled. After sending packets, I can see all 4 queues received packets.
> > 5. use l3fwd with " ./examples/l3fwd/build/l3fwd -c fc -n 4 -- -p 0x1 --config="(0,0,2)""
> >    with RSS disabled. After sending packets, I can see queue 0 received packets.
> > 
> > Can you explain what actual problem is?
> > We can talk offline.
> 

Re: [dpdk-dev] [PATCH] fm10k: conditionally disable RSS during device initialization

[Chen, Jing D]

Thomas,

We've agreed offline that the patch works without side effect.
Please kindly apply if possible.

> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> Sent: Thursday, March 31, 2016 9:57 PM
> To: dev@dpdk.org
> Cc: Michael Frasca <michael.frasca@oracle.com>; Chen, Jing D
> <jing.d.chen@intel.com>
> Subject: Re: [dpdk-dev] [PATCH] fm10k: conditionally disable RSS during
> device initialization
> 
> Please, anyone to confirm that the patch is valid and must be applied?
> This discussion shows some doubts.
> 
> 
> 2016-03-24 13:35, Michael Frasca:
> > Jing,
> >
> > Thanks for your assistance. The experiment that you have built should
> > allow you to observe the bug. In [5], I would expect that queue 0
> > receives roughly 1/4 of the packets that you sending, assuming the
> > input packets have varied IP addresses. Can you measure what % of
> > packets are actually being received in this single queue setup (after first
> running a 4-queue setup)?
> >
> > When trying to running with only one RX queue, the fm10k retains the
> > same RSS hash function and redirection table that was configured from
> > a previous run. As a result, some packets are still being directed to
> > other receive queues. I have confirmed this by polling the queue
> > specific stats, which I retrieved via rte_eth_xstats_get().
> >
> > Looking at fm10k_dev_rss_configure(), one should see that there is no
> > modification of fm10k registers when nb_rx_queues == 1. As far as I
> > can tell, this is the reason that only a certain partition of packets
> > are being receive in a single queue setup (after first running a multi-queue
> configuration).
> >
> > I am unable to access my development environment today, but if you
> > need, I can later craft a patch to l3fwd that shows the measurement of
> > packets received at each queue.
> >
> > Thanks,
> > Mike
> >
> >
> > > On Mar 24, 2016, at 2:40 AM, Chen, Jing D <jing.d.chen@intel.com> wrote:
> > >
> > > Hi, Frasca,
> > >
> > >
> > >
> > >> -----Original Message-----
> > >> From: Michael Frasca [mailto:michael.frasca@oracle.com
> > >> <mailto:michael.frasca@oracle.com>]
> > >> Sent: Wednesday, March 23, 2016 9:43 PM
> > >> To: Chen, Jing D
> > >> Cc: dev@dpdk.org <mailto:dev@dpdk.org>
> > >> Subject: Re: [PATCH] fm10k: conditionally disable RSS during device
> > >> initialization
> > >>
> > >> Hi Jing,
> > >>
> > >> I ran into this issue while trying to run experiments with
> > >> different RSS configurations (no RSS being one cases). It is not
> > >> clear to me that setting this register to zero is the best way to disable
> RSS.
> > >>
> > >> After digging further, I have a theory that I'm having this issues
> > >> because I've only attached my DPDK application to SR-IOV ports. In
> > >> fm10k_dev_dglort_map_configure(), I see that 'RSS Length' is set
> > >> for the DGLORT decoder. However, it appears that this is only
> > >> invoked for physical functions.
> > >>
> > >> Could this be my problem? Is it required that I bind to the
> > >> physical function if I want to properly manipulate RSS?
> > >>
> > >> Thanks,
> > >> Mike
> > >>
> > > I don't know exactly what problem you ran into. I think we needn't
> > > worry about those DGLORT setting when using VF device.
> > >
> > > I've followed steps to use SRIOV device with RSS enabled and
> > > disabled, both are worked well from my side after applying your patch.
> Below is my setup.
> > >
> > > 1. PF with Linux driver "fm10k-next_0.19.3".
> > > 2. DPDK with latest code from master branch, apply your patch.
> > > 3. Use 1 VF device created by kernel driver.
> > > 4. use l3fwd with " ./examples/l3fwd/build/l3fwd -c fc -n 4 -- -p 0x1 --
> config="(0,0,2),(0,1,2),(0,2,3),(0,3,3)""
> > >    with RSS enabled. After sending packets, I can see all 4 queues received
> packets.
> > > 5. use l3fwd with " ./examples/l3fwd/build/l3fwd -c fc -n 4 -- -p 0x1 --
> config="(0,0,2)""
> > >    with RSS disabled. After sending packets, I can see queue 0 received
> packets.
> > >
> > > Can you explain what actual problem is?
> > > We can talk offline.
> >
> 

Re: [dpdk-dev] [PATCH] fm10k: conditionally disable RSS during device initialization

[Thomas Monjalon]

> > If the provided configuration does not call for RSS, then RSS is
> > explicitly disabled. Without this change, the device continues to
> > operate under the previous RSS configuration.
> > 
> > Fixes: 57033cdf8fdc ("fm10k: add PF RSS")
> > 
> > Signed-off-by: Michael Frasca <michael.frasca@oracle.com>
> Acked-by : Jing Chen <jing.d.chen@intel.com>

Applied, thanks