Re: [dpdk-dev] [PATCH v4 0/6] ethdev: introduce shared Rx queue

[Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>]

Hi Xueming,

On 9/30/21 5:55 PM, Xueming Li wrote:
> In current DPDK framework, all RX queues is pre-loaded with mbufs for
> incoming packets. When number of representors scale out in a switch
> domain, the memory consumption became significant. Further more,
> polling all ports leads to high cache miss, high latency and low
> throughputs.
> 
> This patch introduces shared RX queue. PF and representors with same
> configuration in same switch domain could share RX queue set by
> specifying shared Rx queue offloading flag and sharing group.
> 
> All ports that Shared Rx queue actually shares One Rx queue and only
> pre-load mbufs to one Rx queue, memory is saved.
> 
> Polling any queue using same shared RX queue receives packets from all
> member ports. Source port is identified by mbuf->port.
> 
> Multiple groups is supported by group ID. Port queue number in a shared
> group should be identical. Queue index is 1:1 mapped in shared group.
> An example of polling two share groups:
>   core	group	queue
>   0	0	0
>   1	0	1
>   2	0	2
>   3	0	3
>   4	1	0
>   5	1	1
>   6	1	2
>   7	1	3
> 
> Shared RX queue must be polled on single thread or core. If both PF0 and
> representor0 joined same share group, can't poll pf0rxq0 on core1 and
> rep0rxq0 on core2. Actually, polling one port within share group is
> sufficient since polling any port in group will return packets for any
> port in group.

I apologize that I jump in into the review process that late.

Frankly speaking I doubt that it is the best design to solve
the problem. Yes, I confirm that the problem exists, but I
think there is better and simpler way to solve it.

The problem of the suggested solution is that it puts all
the headache about consistency to application and PMDs
without any help from ethdev layer to guarantee the
consistency. As the result I believe it will be either
missing/lost consistency checks or huge duplication in
each PMD which supports the feature. Shared RxQs must be
equally configured including number of queues, offloads
(taking device level Rx offloads into account), RSS
settings etc. So, applications must care about it and
PMDs (or ethdev layer) must check it.

The advantage of the solution is that any device may
create group and subsequent devices join. Absence of
primary device is nice. But do we really need it?
Will the design work if some representors are configured
to use shared RxQ, but some do not? Theoretically it
is possible, but could require extra non-trivial code
on fast path.

Also looking at the first two patch I don't understand
how application will find out which devices may share
RxQs. E.g. if we have two difference NICs which support
sharing, we can try to setup only one group 0, but
finally will have two devices (not one) which must be
polled.

1. We need extra flag in dev_info->dev_capa
   RTE_ETH_DEV_CAPA_RX_SHARE to advertise that
   the device supports Rx sharing.

2. I think we need "rx_domain" in device info
   (which should be treated in boundaries of the
   switch_domain) if and only if
   RTE_ETH_DEV_CAPA_RX_SHARE is advertised.
   Otherwise rx_domain value does not make sense.

(1) and (2) will allow application to find out which
devices can share Rx.

3. Primary device (representors backing device) should
   advertise shared RxQ offload. Enabling of the offload
   tells the device to provide packets to all device in
   the Rx domain with mbuf->port filled in appropriately.
   Also it allows app to identify primary device in the
   Rx domain. When application enables the offload, it
   must ensure that it does not treat used port_id as an
   input port_id, but always check mbuf->port for each
   packet.

4. A new Rx mode should be introduced for secondary
   devices. It should not allow to configure RSS, specify
   any Rx offloads etc. ethdev must ensure it.
   It is an open question right now if it should require
   to provide primary port_id. In theory representors
   have it. However, may be it is nice for consistency
   to ensure that application knows that it does.
   If shared Rx mode is specified for device, application
   does not need to setup RxQs and attempts to do it
   should be discarded in ethdev.
   For consistency it is better to ensure that number of
   queues match.
   It is an interesting question what should happen if
   primary device is reconfigured and shared Rx is
   disabled on reconfiguration.

5. If so, in theory implementation of the Rx burst
   in the secondary could simply call Rx burst on
   primary device.

Andrew.