DPDK usage discussions
 help / color / Atom feed
* [dpdk-users] Same 5 tuple UDP packets have been delivered to different queues.
@ 2019-10-07  3:12 Jun Gan
  2019-10-08 11:52 ` Asaf Penso
  0 siblings, 1 reply; 7+ messages in thread
From: Jun Gan @ 2019-10-07  3:12 UTC (permalink / raw)
  To: users

Hi All,

I just added multiple queues and enabled RSS in our application, I set
"rss_hf" to "ETH_RSS_IP | ETH_RSS_UDP", but I can see that same 5 tuple
packets, will be delivered to different queues. Do I miss something?

Here is the code:

eth_conf.rxmode.mq_mode = ETH_MQ_RX_RSS;
eth_conf.rx_adv_conf.rss_conf.rss_hf = ETH_RSS_IP | ETH_RSS_UDP
eth_conf..rx_adv_conf.rss_conf.rss_hf &= dev_info.flow_type_rss_offloads;


I actually have two queues, and two different threads(lcore) busy
polling(rx_burst) on these two queues, I also met other weird issues like:

dpdk-stable-18.11.2/drivers/net/mlx5/mlx5_rxtx_vec_sse.h:743: rxq_burst_v:
Assertion `rxq->rq_pi == rxq->cq_ci' failed.

I don't understand what does this means, I think rx_burst on the same nic
different queue should be thread-safe, isn't it?

I'm using dpdk 18.11.2, OS is centos 7.6 and NIC is MLX CX-5.

I really appreciate it if anyone can help!!

Thanks,
Jun Gan


-- 
Jun Gan

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [dpdk-users] Same 5 tuple UDP packets have been delivered to different queues.
  2019-10-07  3:12 [dpdk-users] Same 5 tuple UDP packets have been delivered to different queues Jun Gan
@ 2019-10-08 11:52 ` Asaf Penso
  2019-10-09  1:56   ` Jun Gan
  0 siblings, 1 reply; 7+ messages in thread
From: Asaf Penso @ 2019-10-08 11:52 UTC (permalink / raw)
  To: Jun Gan, users; +Cc: Technical Support

Hello Jun Gan,

Thanks for reaching out and posting this query.
I'm adding here our support team for further assistance.

Regards,
Asaf Penso

> -----Original Message-----
> From: users <users-bounces@dpdk.org> On Behalf Of Jun Gan
> Sent: Monday, October 7, 2019 6:13 AM
> To: users@dpdk.org
> Subject: [dpdk-users] Same 5 tuple UDP packets have been delivered to
> different queues.
> 
> Hi All,
> 
> I just added multiple queues and enabled RSS in our application, I set
> "rss_hf" to "ETH_RSS_IP | ETH_RSS_UDP", but I can see that same 5 tuple
> packets, will be delivered to different queues. Do I miss something?
> 
> Here is the code:
> 
> eth_conf.rxmode.mq_mode = ETH_MQ_RX_RSS;
> eth_conf.rx_adv_conf.rss_conf.rss_hf = ETH_RSS_IP | ETH_RSS_UDP
> eth_conf..rx_adv_conf.rss_conf.rss_hf &=
> dev_info.flow_type_rss_offloads;
> 
> 
> I actually have two queues, and two different threads(lcore) busy
> polling(rx_burst) on these two queues, I also met other weird issues like:
> 
> dpdk-stable-18.11.2/drivers/net/mlx5/mlx5_rxtx_vec_sse.h:743:
> rxq_burst_v:
> Assertion `rxq->rq_pi == rxq->cq_ci' failed.
> 
> I don't understand what does this means, I think rx_burst on the same nic
> different queue should be thread-safe, isn't it?
> 
> I'm using dpdk 18.11.2, OS is centos 7.6 and NIC is MLX CX-5.
> 
> I really appreciate it if anyone can help!!
> 
> Thanks,
> Jun Gan
> 
> 
> --
> Jun Gan

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [dpdk-users] Same 5 tuple UDP packets have been delivered to different queues.
  2019-10-08 11:52 ` Asaf Penso
@ 2019-10-09  1:56   ` Jun Gan
  2019-10-09  3:12     ` Stephen Hemminger
  0 siblings, 1 reply; 7+ messages in thread
From: Jun Gan @ 2019-10-09  1:56 UTC (permalink / raw)
  To: Asaf Penso; +Cc: users, Technical Support

Hi Asaf,

Thank you so much for your help!

I tend to think there is something missing in my code, do you have any code
example how to enable RSS?

I did following setup:

eth_conf.rxmode.mq_mode = ETH_MQ_RX_RSS;
eth_conf.rx_adv_conf.rss_conf.rss_hf = ETH_RSS_IP | ETH_RSS_UDP
eth_conf..rx_adv_conf.rss_conf.rss_hf &= dev_info.flow_type_rss_offloads;

I also tried to add a flow rule, and I found mlx5 doesn't support
"rte_eth_dev_rss_hash_conf_get"?

Here is the action, all the UDP packets applied this action:

  struct rte_flow_action_rss action_rss = {
    .types = ETH_RSS_IP | ETH_RSS_UDP,
    .key_len = 40,
    .queue_num = 4,
    .key = rss_key,
    .queue = queue,
  };


And I use spdk reactor to launch multiple threads polling different queue.


Thanks,
Jun Gan




On Tue, Oct 8, 2019 at 4:52 AM Asaf Penso <asafp@mellanox.com> wrote:

> Hello Jun Gan,
>
> Thanks for reaching out and posting this query.
> I'm adding here our support team for further assistance.
>
> Regards,
> Asaf Penso
>
> > -----Original Message-----
> > From: users <users-bounces@dpdk.org> On Behalf Of Jun Gan
> > Sent: Monday, October 7, 2019 6:13 AM
> > To: users@dpdk.org
> > Subject: [dpdk-users] Same 5 tuple UDP packets have been delivered to
> > different queues.
> >
> > Hi All,
> >
> > I just added multiple queues and enabled RSS in our application, I set
> > "rss_hf" to "ETH_RSS_IP | ETH_RSS_UDP", but I can see that same 5 tuple
> > packets, will be delivered to different queues. Do I miss something?
> >
> > Here is the code:
> >
> > eth_conf.rxmode.mq_mode = ETH_MQ_RX_RSS;
> > eth_conf.rx_adv_conf.rss_conf.rss_hf = ETH_RSS_IP | ETH_RSS_UDP
> > eth_conf..rx_adv_conf.rss_conf.rss_hf &=
> > dev_info.flow_type_rss_offloads;
> >
> >
> > I actually have two queues, and two different threads(lcore) busy
> > polling(rx_burst) on these two queues, I also met other weird issues
> like:
> >
> > dpdk-stable-18.11.2/drivers/net/mlx5/mlx5_rxtx_vec_sse.h:743:
> > rxq_burst_v:
> > Assertion `rxq->rq_pi == rxq->cq_ci' failed.
> >
> > I don't understand what does this means, I think rx_burst on the same nic
> > different queue should be thread-safe, isn't it?
> >
> > I'm using dpdk 18.11.2, OS is centos 7.6 and NIC is MLX CX-5.
> >
> > I really appreciate it if anyone can help!!
> >
> > Thanks,
> > Jun Gan
> >
> >
> > --
> > Jun Gan
>


-- 
Jun Gan

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [dpdk-users] Same 5 tuple UDP packets have been delivered to different queues.
  2019-10-09  1:56   ` Jun Gan
@ 2019-10-09  3:12     ` Stephen Hemminger
  2019-10-09  4:29       ` Jun Gan
  0 siblings, 1 reply; 7+ messages in thread
From: Stephen Hemminger @ 2019-10-09  3:12 UTC (permalink / raw)
  To: Jun Gan; +Cc: Asaf Penso, users, Technical Support

On Tue, 8 Oct 2019 18:56:31 -0700
Jun Gan <geminijun@gmail.com> wrote:

> Hi Asaf,
> 
> Thank you so much for your help!
> 
> I tend to think there is something missing in my code, do you have any code
> example how to enable RSS?
> 
> I did following setup:
> 
> eth_conf.rxmode.mq_mode = ETH_MQ_RX_RSS;
> eth_conf.rx_adv_conf.rss_conf.rss_hf = ETH_RSS_IP | ETH_RSS_UDP
> eth_conf..rx_adv_conf.rss_conf.rss_hf &= dev_info.flow_type_rss_offloads;
> 
> I also tried to add a flow rule, and I found mlx5 doesn't support
> "rte_eth_dev_rss_hash_conf_get"?
> 
> Here is the action, all the UDP packets applied this action:
> 
>   struct rte_flow_action_rss action_rss = {
>     .types = ETH_RSS_IP | ETH_RSS_UDP,
>     .key_len = 40,
>     .queue_num = 4,
>     .key = rss_key,
>     .queue = queue,
>   };
> 
> 
> And I use spdk reactor to launch multiple threads polling different queue.
> 
> 
> Thanks,
> Jun Gan
> 
> 
> 
> 
> On Tue, Oct 8, 2019 at 4:52 AM Asaf Penso <asafp@mellanox.com> wrote:
> 
> > Hello Jun Gan,
> >
> > Thanks for reaching out and posting this query.
> > I'm adding here our support team for further assistance.
> >
> > Regards,
> > Asaf Penso
> >  
> > > -----Original Message-----
> > > From: users <users-bounces@dpdk.org> On Behalf Of Jun Gan
> > > Sent: Monday, October 7, 2019 6:13 AM
> > > To: users@dpdk.org
> > > Subject: [dpdk-users] Same 5 tuple UDP packets have been delivered to
> > > different queues.
> > >
> > > Hi All,
> > >
> > > I just added multiple queues and enabled RSS in our application, I set
> > > "rss_hf" to "ETH_RSS_IP | ETH_RSS_UDP", but I can see that same 5 tuple
> > > packets, will be delivered to different queues. Do I miss something?
> > >
> > > Here is the code:
> > >
> > > eth_conf.rxmode.mq_mode = ETH_MQ_RX_RSS;
> > > eth_conf.rx_adv_conf.rss_conf.rss_hf = ETH_RSS_IP | ETH_RSS_UDP
> > > eth_conf..rx_adv_conf.rss_conf.rss_hf &=
> > > dev_info.flow_type_rss_offloads;
> > >
> > >
> > > I actually have two queues, and two different threads(lcore) busy
> > > polling(rx_burst) on these two queues, I also met other weird issues  
> > like:  
> > >
> > > dpdk-stable-18.11.2/drivers/net/mlx5/mlx5_rxtx_vec_sse.h:743:
> > > rxq_burst_v:
> > > Assertion `rxq->rq_pi == rxq->cq_ci' failed.
> > >
> > > I don't understand what does this means, I think rx_burst on the same nic
> > > different queue should be thread-safe, isn't it?
> > >
> > > I'm using dpdk 18.11.2, OS is centos 7.6 and NIC is MLX CX-5.
> > >
> > > I really appreciate it if anyone can help!!

I think you might have confused base RSS and rte_flow RSS.
The base RSS is about spreading incoming across all receive queues.
This is the part in rx_add_conf.

The flow RSS is about spreading matching flows (based on pattern) across a
restricted subset of queues.

My recommendation is use one or the others. Don't use base RSS and rte_flow
at the same time (it doesn't make sense).

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [dpdk-users] Same 5 tuple UDP packets have been delivered to different queues.
  2019-10-09  3:12     ` Stephen Hemminger
@ 2019-10-09  4:29       ` Jun Gan
  2019-10-10  1:08         ` Stephen Hemminger
  0 siblings, 1 reply; 7+ messages in thread
From: Jun Gan @ 2019-10-09  4:29 UTC (permalink / raw)
  To: Stephen Hemminger; +Cc: Asaf Penso, users, Technical Support

Hi Stephen,

Thanks for the suggestion, I actually tried both, but no difference, the
problem for me is that same src&dsp ip&port udp packets, will be delivered
to different queues.

That breaks my assumption that the packet from some session will be
processed by the same thread on the server.

so besides rx_adv_conf, is there any other things I have to configure?

Thanks,
Jun Gan




On Tue, Oct 8, 2019 at 8:12 PM Stephen Hemminger <stephen@networkplumber.org>
wrote:

> On Tue, 8 Oct 2019 18:56:31 -0700
> Jun Gan <geminijun@gmail.com> wrote:
>
> > Hi Asaf,
> >
> > Thank you so much for your help!
> >
> > I tend to think there is something missing in my code, do you have any
> code
> > example how to enable RSS?
> >
> > I did following setup:
> >
> > eth_conf.rxmode.mq_mode = ETH_MQ_RX_RSS;
> > eth_conf.rx_adv_conf.rss_conf.rss_hf = ETH_RSS_IP | ETH_RSS_UDP
> > eth_conf..rx_adv_conf.rss_conf.rss_hf &= dev_info.flow_type_rss_offloads;
> >
> > I also tried to add a flow rule, and I found mlx5 doesn't support
> > "rte_eth_dev_rss_hash_conf_get"?
> >
> > Here is the action, all the UDP packets applied this action:
> >
> >   struct rte_flow_action_rss action_rss = {
> >     .types = ETH_RSS_IP | ETH_RSS_UDP,
> >     .key_len = 40,
> >     .queue_num = 4,
> >     .key = rss_key,
> >     .queue = queue,
> >   };
> >
> >
> > And I use spdk reactor to launch multiple threads polling different
> queue.
> >
> >
> > Thanks,
> > Jun Gan
> >
> >
> >
> >
> > On Tue, Oct 8, 2019 at 4:52 AM Asaf Penso <asafp@mellanox.com> wrote:
> >
> > > Hello Jun Gan,
> > >
> > > Thanks for reaching out and posting this query.
> > > I'm adding here our support team for further assistance.
> > >
> > > Regards,
> > > Asaf Penso
> > >
> > > > -----Original Message-----
> > > > From: users <users-bounces@dpdk.org> On Behalf Of Jun Gan
> > > > Sent: Monday, October 7, 2019 6:13 AM
> > > > To: users@dpdk.org
> > > > Subject: [dpdk-users] Same 5 tuple UDP packets have been delivered to
> > > > different queues.
> > > >
> > > > Hi All,
> > > >
> > > > I just added multiple queues and enabled RSS in our application, I
> set
> > > > "rss_hf" to "ETH_RSS_IP | ETH_RSS_UDP", but I can see that same 5
> tuple
> > > > packets, will be delivered to different queues. Do I miss something?
> > > >
> > > > Here is the code:
> > > >
> > > > eth_conf.rxmode.mq_mode = ETH_MQ_RX_RSS;
> > > > eth_conf.rx_adv_conf.rss_conf.rss_hf = ETH_RSS_IP | ETH_RSS_UDP
> > > > eth_conf..rx_adv_conf.rss_conf.rss_hf &=
> > > > dev_info.flow_type_rss_offloads;
> > > >
> > > >
> > > > I actually have two queues, and two different threads(lcore) busy
> > > > polling(rx_burst) on these two queues, I also met other weird
> issues
> > > like:
> > > >
> > > > dpdk-stable-18.11.2/drivers/net/mlx5/mlx5_rxtx_vec_sse.h:743:
> > > > rxq_burst_v:
> > > > Assertion `rxq->rq_pi == rxq->cq_ci' failed.
> > > >
> > > > I don't understand what does this means, I think rx_burst on the
> same nic
> > > > different queue should be thread-safe, isn't it?
> > > >
> > > > I'm using dpdk 18.11.2, OS is centos 7.6 and NIC is MLX CX-5.
> > > >
> > > > I really appreciate it if anyone can help!!
>
> I think you might have confused base RSS and rte_flow RSS.
> The base RSS is about spreading incoming across all receive queues.
> This is the part in rx_add_conf.
>
> The flow RSS is about spreading matching flows (based on pattern) across a
> restricted subset of queues.
>
> My recommendation is use one or the others. Don't use base RSS and rte_flow
> at the same time (it doesn't make sense).
>


-- 
Jun Gan

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [dpdk-users] Same 5 tuple UDP packets have been delivered to different queues.
  2019-10-09  4:29       ` Jun Gan
@ 2019-10-10  1:08         ` Stephen Hemminger
  2019-10-10  4:04           ` Jun Gan
  0 siblings, 1 reply; 7+ messages in thread
From: Stephen Hemminger @ 2019-10-10  1:08 UTC (permalink / raw)
  To: Jun Gan; +Cc: Asaf Penso, users, Technical Support

On Tue, 8 Oct 2019 21:29:46 -0700
Jun Gan <geminijun@gmail.com> wrote:

> Hi Stephen,
> 
> Thanks for the suggestion, I actually tried both, but no difference, the
> problem for me is that same src&dsp ip&port udp packets, will be delivered
> to different queues.
> 
> That breaks my assumption that the packet from some session will be
> processed by the same thread on the server.
> 
> so besides rx_adv_conf, is there any other things I have to configure?

What is value of rss_hash in the received packet, is it the same?
Also what environment HW, NIC, etc?

^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: [dpdk-users] Same 5 tuple UDP packets have been delivered to different queues.
  2019-10-10  1:08         ` Stephen Hemminger
@ 2019-10-10  4:04           ` Jun Gan
  0 siblings, 0 replies; 7+ messages in thread
From: Jun Gan @ 2019-10-10  4:04 UTC (permalink / raw)
  To: Stephen Hemminger; +Cc: Asaf Penso, users, Technical Support

Hi Stephen,

Thanks for the help, it turns out that it's because of a coding error.

I have fixed that and it's working well now.


On Wed, Oct 9, 2019 at 6:09 PM Stephen Hemminger <stephen@networkplumber.org>
wrote:

> On Tue, 8 Oct 2019 21:29:46 -0700
> Jun Gan <geminijun@gmail.com> wrote:
>
> > Hi Stephen,
> >
> > Thanks for the suggestion, I actually tried both, but no difference, the
> > problem for me is that same src&dsp ip&port udp packets, will be
> delivered
> > to different queues.
> >
> > That breaks my assumption that the packet from some session will be
> > processed by the same thread on the server.
> >
> > so besides rx_adv_conf, is there any other things I have to configure?
>
> What is value of rss_hash in the received packet, is it the same?
> Also what environment HW, NIC, etc?
>


-- 
Jun Gan

^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, back to index

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-10-07  3:12 [dpdk-users] Same 5 tuple UDP packets have been delivered to different queues Jun Gan
2019-10-08 11:52 ` Asaf Penso
2019-10-09  1:56   ` Jun Gan
2019-10-09  3:12     ` Stephen Hemminger
2019-10-09  4:29       ` Jun Gan
2019-10-10  1:08         ` Stephen Hemminger
2019-10-10  4:04           ` Jun Gan

DPDK usage discussions

Archives are clonable:
	git clone --mirror http://inbox.dpdk.org/users/0 users/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 users users/ http://inbox.dpdk.org/users \
		users@dpdk.org
	public-inbox-index users


Newsgroup available over NNTP:
	nntp://inbox.dpdk.org/inbox.dpdk.users


AGPL code for this site: git clone https://public-inbox.org/ public-inbox