[dpdk-users] Same 5 tuple UDP packets have been delivered to different queues.

[dpdk-users] Same 5 tuple UDP packets have been delivered to different queues.

[Jun Gan]

Hi All,

I just added multiple queues and enabled RSS in our application, I set
"rss_hf" to "ETH_RSS_IP | ETH_RSS_UDP", but I can see that same 5 tuple
packets, will be delivered to different queues. Do I miss something?

Here is the code:

eth_conf.rxmode.mq_mode = ETH_MQ_RX_RSS;
eth_conf.rx_adv_conf.rss_conf.rss_hf = ETH_RSS_IP | ETH_RSS_UDP
eth_conf..rx_adv_conf.rss_conf.rss_hf &= dev_info.flow_type_rss_offloads;


I actually have two queues, and two different threads(lcore) busy
polling(rx_burst) on these two queues, I also met other weird issues like:

dpdk-stable-18.11.2/drivers/net/mlx5/mlx5_rxtx_vec_sse.h:743: rxq_burst_v:
Assertion `rxq->rq_pi == rxq->cq_ci' failed.

I don't understand what does this means, I think rx_burst on the same nic
different queue should be thread-safe, isn't it?

I'm using dpdk 18.11.2, OS is centos 7.6 and NIC is MLX CX-5.

I really appreciate it if anyone can help!!

Thanks,
Jun Gan


-- 
Jun Gan

Re: [dpdk-users] Same 5 tuple UDP packets have been delivered to different queues.

[Asaf Penso]

Hello Jun Gan,

Thanks for reaching out and posting this query.
I'm adding here our support team for further assistance.

Regards,
Asaf Penso

> -----Original Message-----
> From: users <users-bounces@dpdk.org> On Behalf Of Jun Gan
> Sent: Monday, October 7, 2019 6:13 AM
> To: users@dpdk.org
> Subject: [dpdk-users] Same 5 tuple UDP packets have been delivered to
> different queues.
> 
> Hi All,
> 
> I just added multiple queues and enabled RSS in our application, I set
> "rss_hf" to "ETH_RSS_IP | ETH_RSS_UDP", but I can see that same 5 tuple
> packets, will be delivered to different queues. Do I miss something?
> 
> Here is the code:
> 
> eth_conf.rxmode.mq_mode = ETH_MQ_RX_RSS;
> eth_conf.rx_adv_conf.rss_conf.rss_hf = ETH_RSS_IP | ETH_RSS_UDP
> eth_conf..rx_adv_conf.rss_conf.rss_hf &=
> dev_info.flow_type_rss_offloads;
> 
> 
> I actually have two queues, and two different threads(lcore) busy
> polling(rx_burst) on these two queues, I also met other weird issues like:
> 
> dpdk-stable-18.11.2/drivers/net/mlx5/mlx5_rxtx_vec_sse.h:743:
> rxq_burst_v:
> Assertion `rxq->rq_pi == rxq->cq_ci' failed.
> 
> I don't understand what does this means, I think rx_burst on the same nic
> different queue should be thread-safe, isn't it?
> 
> I'm using dpdk 18.11.2, OS is centos 7.6 and NIC is MLX CX-5.
> 
> I really appreciate it if anyone can help!!
> 
> Thanks,
> Jun Gan
> 
> 
> --
> Jun Gan

Re: [dpdk-users] Same 5 tuple UDP packets have been delivered to different queues.

[Jun Gan]

Hi Asaf,

Thank you so much for your help!

I tend to think there is something missing in my code, do you have any code
example how to enable RSS?

I did following setup:

eth_conf.rxmode.mq_mode = ETH_MQ_RX_RSS;
eth_conf.rx_adv_conf.rss_conf.rss_hf = ETH_RSS_IP | ETH_RSS_UDP
eth_conf..rx_adv_conf.rss_conf.rss_hf &= dev_info.flow_type_rss_offloads;

I also tried to add a flow rule, and I found mlx5 doesn't support
"rte_eth_dev_rss_hash_conf_get"?

Here is the action, all the UDP packets applied this action:

  struct rte_flow_action_rss action_rss = {
    .types = ETH_RSS_IP | ETH_RSS_UDP,
    .key_len = 40,
    .queue_num = 4,
    .key = rss_key,
    .queue = queue,
  };


And I use spdk reactor to launch multiple threads polling different queue.


Thanks,
Jun Gan




On Tue, Oct 8, 2019 at 4:52 AM Asaf Penso <asafp@mellanox.com> wrote:

> Hello Jun Gan,
>
> Thanks for reaching out and posting this query.
> I'm adding here our support team for further assistance.
>
> Regards,
> Asaf Penso
>
> > -----Original Message-----
> > From: users <users-bounces@dpdk.org> On Behalf Of Jun Gan
> > Sent: Monday, October 7, 2019 6:13 AM
> > To: users@dpdk.org
> > Subject: [dpdk-users] Same 5 tuple UDP packets have been delivered to
> > different queues.
> >
> > Hi All,
> >
> > I just added multiple queues and enabled RSS in our application, I set
> > "rss_hf" to "ETH_RSS_IP | ETH_RSS_UDP", but I can see that same 5 tuple
> > packets, will be delivered to different queues. Do I miss something?
> >
> > Here is the code:
> >
> > eth_conf.rxmode.mq_mode = ETH_MQ_RX_RSS;
> > eth_conf.rx_adv_conf.rss_conf.rss_hf = ETH_RSS_IP | ETH_RSS_UDP
> > eth_conf..rx_adv_conf.rss_conf.rss_hf &=
> > dev_info.flow_type_rss_offloads;
> >
> >
> > I actually have two queues, and two different threads(lcore) busy
> > polling(rx_burst) on these two queues, I also met other weird issues
> like:
> >
> > dpdk-stable-18.11.2/drivers/net/mlx5/mlx5_rxtx_vec_sse.h:743:
> > rxq_burst_v:
> > Assertion `rxq->rq_pi == rxq->cq_ci' failed.
> >
> > I don't understand what does this means, I think rx_burst on the same nic
> > different queue should be thread-safe, isn't it?
> >
> > I'm using dpdk 18.11.2, OS is centos 7.6 and NIC is MLX CX-5.
> >
> > I really appreciate it if anyone can help!!
> >
> > Thanks,
> > Jun Gan
> >
> >
> > --
> > Jun Gan
>


-- 
Jun Gan

Re: [dpdk-users] Same 5 tuple UDP packets have been delivered to different queues.

[Stephen Hemminger]

On Tue, 8 Oct 2019 18:56:31 -0700
Jun Gan <geminijun@gmail.com> wrote:

> Hi Asaf,
> 
> Thank you so much for your help!
> 
> I tend to think there is something missing in my code, do you have any code
> example how to enable RSS?
> 
> I did following setup:
> 
> eth_conf.rxmode.mq_mode = ETH_MQ_RX_RSS;
> eth_conf.rx_adv_conf.rss_conf.rss_hf = ETH_RSS_IP | ETH_RSS_UDP
> eth_conf..rx_adv_conf.rss_conf.rss_hf &= dev_info.flow_type_rss_offloads;
> 
> I also tried to add a flow rule, and I found mlx5 doesn't support
> "rte_eth_dev_rss_hash_conf_get"?
> 
> Here is the action, all the UDP packets applied this action:
> 
>   struct rte_flow_action_rss action_rss = {
>     .types = ETH_RSS_IP | ETH_RSS_UDP,
>     .key_len = 40,
>     .queue_num = 4,
>     .key = rss_key,
>     .queue = queue,
>   };
> 
> 
> And I use spdk reactor to launch multiple threads polling different queue.
> 
> 
> Thanks,
> Jun Gan
> 
> 
> 
> 
> On Tue, Oct 8, 2019 at 4:52 AM Asaf Penso <asafp@mellanox.com> wrote:
> 
> > Hello Jun Gan,
> >
> > Thanks for reaching out and posting this query.
> > I'm adding here our support team for further assistance.
> >
> > Regards,
> > Asaf Penso
> >  
> > > -----Original Message-----
> > > From: users <users-bounces@dpdk.org> On Behalf Of Jun Gan
> > > Sent: Monday, October 7, 2019 6:13 AM
> > > To: users@dpdk.org
> > > Subject: [dpdk-users] Same 5 tuple UDP packets have been delivered to
> > > different queues.
> > >
> > > Hi All,
> > >
> > > I just added multiple queues and enabled RSS in our application, I set
> > > "rss_hf" to "ETH_RSS_IP | ETH_RSS_UDP", but I can see that same 5 tuple
> > > packets, will be delivered to different queues. Do I miss something?
> > >
> > > Here is the code:
> > >
> > > eth_conf.rxmode.mq_mode = ETH_MQ_RX_RSS;
> > > eth_conf.rx_adv_conf.rss_conf.rss_hf = ETH_RSS_IP | ETH_RSS_UDP
> > > eth_conf..rx_adv_conf.rss_conf.rss_hf &=
> > > dev_info.flow_type_rss_offloads;
> > >
> > >
> > > I actually have two queues, and two different threads(lcore) busy
> > > polling(rx_burst) on these two queues, I also met other weird issues  
> > like:  
> > >
> > > dpdk-stable-18.11.2/drivers/net/mlx5/mlx5_rxtx_vec_sse.h:743:
> > > rxq_burst_v:
> > > Assertion `rxq->rq_pi == rxq->cq_ci' failed.
> > >
> > > I don't understand what does this means, I think rx_burst on the same nic
> > > different queue should be thread-safe, isn't it?
> > >
> > > I'm using dpdk 18.11.2, OS is centos 7.6 and NIC is MLX CX-5.
> > >
> > > I really appreciate it if anyone can help!!

I think you might have confused base RSS and rte_flow RSS.
The base RSS is about spreading incoming across all receive queues.
This is the part in rx_add_conf.

The flow RSS is about spreading matching flows (based on pattern) across a
restricted subset of queues.

My recommendation is use one or the others. Don't use base RSS and rte_flow
at the same time (it doesn't make sense).

Re: [dpdk-users] Same 5 tuple UDP packets have been delivered to different queues.

[Jun Gan]

Hi Stephen,

Thanks for the suggestion, I actually tried both, but no difference, the
problem for me is that same src&dsp ip&port udp packets, will be delivered
to different queues.

That breaks my assumption that the packet from some session will be
processed by the same thread on the server.

so besides rx_adv_conf, is there any other things I have to configure?

Thanks,
Jun Gan




On Tue, Oct 8, 2019 at 8:12 PM Stephen Hemminger <stephen@networkplumber.org>
wrote:

> On Tue, 8 Oct 2019 18:56:31 -0700
> Jun Gan <geminijun@gmail.com> wrote:
>
> > Hi Asaf,
> >
> > Thank you so much for your help!
> >
> > I tend to think there is something missing in my code, do you have any
> code
> > example how to enable RSS?
> >
> > I did following setup:
> >
> > eth_conf.rxmode.mq_mode = ETH_MQ_RX_RSS;
> > eth_conf.rx_adv_conf.rss_conf.rss_hf = ETH_RSS_IP | ETH_RSS_UDP
> > eth_conf..rx_adv_conf.rss_conf.rss_hf &= dev_info.flow_type_rss_offloads;
> >
> > I also tried to add a flow rule, and I found mlx5 doesn't support
> > "rte_eth_dev_rss_hash_conf_get"?
> >
> > Here is the action, all the UDP packets applied this action:
> >
> >   struct rte_flow_action_rss action_rss = {
> >     .types = ETH_RSS_IP | ETH_RSS_UDP,
> >     .key_len = 40,
> >     .queue_num = 4,
> >     .key = rss_key,
> >     .queue = queue,
> >   };
> >
> >
> > And I use spdk reactor to launch multiple threads polling different
> queue.
> >
> >
> > Thanks,
> > Jun Gan
> >
> >
> >
> >
> > On Tue, Oct 8, 2019 at 4:52 AM Asaf Penso <asafp@mellanox.com> wrote:
> >
> > > Hello Jun Gan,
> > >
> > > Thanks for reaching out and posting this query.
> > > I'm adding here our support team for further assistance.
> > >
> > > Regards,
> > > Asaf Penso
> > >
> > > > -----Original Message-----
> > > > From: users <users-bounces@dpdk.org> On Behalf Of Jun Gan
> > > > Sent: Monday, October 7, 2019 6:13 AM
> > > > To: users@dpdk.org
> > > > Subject: [dpdk-users] Same 5 tuple UDP packets have been delivered to
> > > > different queues.
> > > >
> > > > Hi All,
> > > >
> > > > I just added multiple queues and enabled RSS in our application, I
> set
> > > > "rss_hf" to "ETH_RSS_IP | ETH_RSS_UDP", but I can see that same 5
> tuple
> > > > packets, will be delivered to different queues. Do I miss something?
> > > >
> > > > Here is the code:
> > > >
> > > > eth_conf.rxmode.mq_mode = ETH_MQ_RX_RSS;
> > > > eth_conf.rx_adv_conf.rss_conf.rss_hf = ETH_RSS_IP | ETH_RSS_UDP
> > > > eth_conf..rx_adv_conf.rss_conf.rss_hf &=
> > > > dev_info.flow_type_rss_offloads;
> > > >
> > > >
> > > > I actually have two queues, and two different threads(lcore) busy
> > > > polling(rx_burst) on these two queues, I also met other weird
> issues
> > > like:
> > > >
> > > > dpdk-stable-18.11.2/drivers/net/mlx5/mlx5_rxtx_vec_sse.h:743:
> > > > rxq_burst_v:
> > > > Assertion `rxq->rq_pi == rxq->cq_ci' failed.
> > > >
> > > > I don't understand what does this means, I think rx_burst on the
> same nic
> > > > different queue should be thread-safe, isn't it?
> > > >
> > > > I'm using dpdk 18.11.2, OS is centos 7.6 and NIC is MLX CX-5.
> > > >
> > > > I really appreciate it if anyone can help!!
>
> I think you might have confused base RSS and rte_flow RSS.
> The base RSS is about spreading incoming across all receive queues.
> This is the part in rx_add_conf.
>
> The flow RSS is about spreading matching flows (based on pattern) across a
> restricted subset of queues.
>
> My recommendation is use one or the others. Don't use base RSS and rte_flow
> at the same time (it doesn't make sense).
>


-- 
Jun Gan

Re: [dpdk-users] Same 5 tuple UDP packets have been delivered to different queues.

[Stephen Hemminger]

On Tue, 8 Oct 2019 21:29:46 -0700
Jun Gan <geminijun@gmail.com> wrote:

> Hi Stephen,
> 
> Thanks for the suggestion, I actually tried both, but no difference, the
> problem for me is that same src&dsp ip&port udp packets, will be delivered
> to different queues.
> 
> That breaks my assumption that the packet from some session will be
> processed by the same thread on the server.
> 
> so besides rx_adv_conf, is there any other things I have to configure?

What is value of rss_hash in the received packet, is it the same?
Also what environment HW, NIC, etc?

Re: [dpdk-users] Same 5 tuple UDP packets have been delivered to different queues.

[Jun Gan]

Hi Stephen,

Thanks for the help, it turns out that it's because of a coding error.

I have fixed that and it's working well now.


On Wed, Oct 9, 2019 at 6:09 PM Stephen Hemminger <stephen@networkplumber.org>
wrote:

> On Tue, 8 Oct 2019 21:29:46 -0700
> Jun Gan <geminijun@gmail.com> wrote:
>
> > Hi Stephen,
> >
> > Thanks for the suggestion, I actually tried both, but no difference, the
> > problem for me is that same src&dsp ip&port udp packets, will be
> delivered
> > to different queues.
> >
> > That breaks my assumption that the packet from some session will be
> > processed by the same thread on the server.
> >
> > so besides rx_adv_conf, is there any other things I have to configure?
>
> What is value of rss_hash in the received packet, is it the same?
> Also what environment HW, NIC, etc?
>


-- 
Jun Gan