IPv4 flows per queue on top of RSS for ixgbe and i40e drivers

IPv4 flows per queue on top of RSS for ixgbe and i40e drivers

[Pavel Vazharov]

Hi there,

We have a proxy application on top of DPDK where we use a symmetric
RSS key to receive the packets for a given connection in both
directions on a single queue. In addition to that, we've few IPv4
addresses where we need to receive their traffic on particular queues.
We use the rte flows functionality with RTE_FLOW_ACTION_TYPE_QUEUE to
achieve that and it seemed to work with the DPDK ixgbe driver.
However, today we tried the same application on top of DPDK i40e
driver and this setup doesn't seem to work there. It prints the
following errors:
`i40e_flow_add_del_fdir_filter(): Conflict with existing flow director rules!`
It seems that the i40e driver doesn't allow adding flow rules on top
of the already set RSS.

Can somebody suggest a way to achieve what we need with i40e: to use a
symmetric RSS key for most of the traffic but to redirect the traffic
for a few specific IPv4 addresses to particular queues?

Thanks,
Pavel.

Re: IPv4 flows per queue on top of RSS for ixgbe and i40e drivers

[Stephen Hemminger]

On Wed, 5 Feb 2025 18:21:33 +0200
Pavel Vazharov <freakpv@gmail.com> wrote:

> Hi there,
> 
> We have a proxy application on top of DPDK where we use a symmetric
> RSS key to receive the packets for a given connection in both
> directions on a single queue. In addition to that, we've few IPv4
> addresses where we need to receive their traffic on particular queues.
> We use the rte flows functionality with RTE_FLOW_ACTION_TYPE_QUEUE to
> achieve that and it seemed to work with the DPDK ixgbe driver.
> However, today we tried the same application on top of DPDK i40e
> driver and this setup doesn't seem to work there. It prints the
> following errors:
> `i40e_flow_add_del_fdir_filter(): Conflict with existing flow director rules!`
> It seems that the i40e driver doesn't allow adding flow rules on top
> of the already set RSS.
> 
> Can somebody suggest a way to achieve what we need with i40e: to use a
> symmetric RSS key for most of the traffic but to redirect the traffic
> for a few specific IPv4 addresses to particular queues?
> 
> Thanks,
> Pavel.

If you are mixing RSS and rte_flow the results are not well defined.
Many drivers treat all active queues (including those used by rte_flow)
as candidates for RSS.

If you want to mix, the the safe way is:
	- don't enable RSS in the device config (rx_mode)
	- define an rte_flow rule with RTE_FLOW_ACTION_TYPE_RSS
          with a match all
	- define a rte_flow rule with RTE_FLOW_ACTION_TYPE_QUEUE
	  that matches the IP
and set rule priorities so that specific IP rule matches before
the match all.

You still maybe at risk of hardware quirks and driver incompatibilities
which is part of the problem with rte_flow.

Re: IPv4 flows per queue on top of RSS for ixgbe and i40e drivers

[Pavel Vazharov]

On Wed, Feb 5, 2025 at 6:50 PM Stephen Hemminger
<stephen@networkplumber.org> wrote:
>
> On Wed, 5 Feb 2025 18:21:33 +0200
> Pavel Vazharov <freakpv@gmail.com> wrote:
>
> > Hi there,
> >
> > We have a proxy application on top of DPDK where we use a symmetric
> > RSS key to receive the packets for a given connection in both
> > directions on a single queue. In addition to that, we've few IPv4
> > addresses where we need to receive their traffic on particular queues.
> > We use the rte flows functionality with RTE_FLOW_ACTION_TYPE_QUEUE to
> > achieve that and it seemed to work with the DPDK ixgbe driver.
> > However, today we tried the same application on top of DPDK i40e
> > driver and this setup doesn't seem to work there. It prints the
> > following errors:
> > `i40e_flow_add_del_fdir_filter(): Conflict with existing flow director rules!`
> > It seems that the i40e driver doesn't allow adding flow rules on top
> > of the already set RSS.
> >
> > Can somebody suggest a way to achieve what we need with i40e: to use a
> > symmetric RSS key for most of the traffic but to redirect the traffic
> > for a few specific IPv4 addresses to particular queues?
> >
> > Thanks,
> > Pavel.
>
> If you are mixing RSS and rte_flow the results are not well defined.
> Many drivers treat all active queues (including those used by rte_flow)
> as candidates for RSS.
>
> If you want to mix, the the safe way is:
>         - don't enable RSS in the device config (rx_mode)
>         - define an rte_flow rule with RTE_FLOW_ACTION_TYPE_RSS
>           with a match all
>         - define a rte_flow rule with RTE_FLOW_ACTION_TYPE_QUEUE
>           that matches the IP
> and set rule priorities so that specific IP rule matches before
> the match all.
Thank you for this idea. I'll give it a try.

> You still maybe at risk of hardware quirks and driver incompatibilities
> which is part of the problem with rte_flow.

Re: IPv4 flows per queue on top of RSS for ixgbe and i40e drivers

[Pavel Vazharov]

On Wed, Feb 5, 2025 at 6:53 PM Pavel Vazharov <freakpv@gmail.com> wrote:
>
> On Wed, Feb 5, 2025 at 6:50 PM Stephen Hemminger
> <stephen@networkplumber.org> wrote:
> >
> > On Wed, 5 Feb 2025 18:21:33 +0200
> > Pavel Vazharov <freakpv@gmail.com> wrote:
> >
> > > Hi there,
> > >
> > > We have a proxy application on top of DPDK where we use a symmetric
> > > RSS key to receive the packets for a given connection in both
> > > directions on a single queue. In addition to that, we've few IPv4
> > > addresses where we need to receive their traffic on particular queues.
> > > We use the rte flows functionality with RTE_FLOW_ACTION_TYPE_QUEUE to
> > > achieve that and it seemed to work with the DPDK ixgbe driver.
> > > However, today we tried the same application on top of DPDK i40e
> > > driver and this setup doesn't seem to work there. It prints the
> > > following errors:
> > > `i40e_flow_add_del_fdir_filter(): Conflict with existing flow director rules!`
> > > It seems that the i40e driver doesn't allow adding flow rules on top
> > > of the already set RSS.
> > >
> > > Can somebody suggest a way to achieve what we need with i40e: to use a
> > > symmetric RSS key for most of the traffic but to redirect the traffic
> > > for a few specific IPv4 addresses to particular queues?
> > >
> > > Thanks,
> > > Pavel.
> >
> > If you are mixing RSS and rte_flow the results are not well defined.
> > Many drivers treat all active queues (including those used by rte_flow)
> > as candidates for RSS.
> >
> > If you want to mix, the the safe way is:
> >         - don't enable RSS in the device config (rx_mode)
> >         - define an rte_flow rule with RTE_FLOW_ACTION_TYPE_RSS
> >           with a match all
> >         - define a rte_flow rule with RTE_FLOW_ACTION_TYPE_QUEUE
> >           that matches the IP
> > and set rule priorities so that specific IP rule matches before
> > the match all.
> Thank you for this idea. I'll give it a try.
The proposed setup worked for NICs with e1000 and ixgb drivers but
with i40e driver it returns error "Not support priority.".
I poke through the code of the DPDK i40e driver and the
`i40e_flow_parse_attr` is called for every type of flow - rss, fdir,
etc.
And inside this function, there is this piece of code:
    /* Not supported */
    if (attr->priority) {
        rte_flow_error_set(error, EINVAL,
                   RTE_FLOW_ERROR_TYPE_ATTR_PRIORITY,
                   attr, "Not support priority.");
        return -rte_errno;
    }
So, it seems to me that the driver don't support any flows with priority.
>
> > You still maybe at risk of hardware quirks and driver incompatibilities
> > which is part of the problem with rte_flow.

Re: IPv4 flows per queue on top of RSS for ixgbe and i40e drivers

[Pavel Vazharov]

On Tue, Feb 11, 2025 at 9:45 AM Pavel Vazharov <freakpv@gmail.com> wrote:
>
> On Wed, Feb 5, 2025 at 6:53 PM Pavel Vazharov <freakpv@gmail.com> wrote:
> >
> > On Wed, Feb 5, 2025 at 6:50 PM Stephen Hemminger
> > <stephen@networkplumber.org> wrote:
> > >
> > > On Wed, 5 Feb 2025 18:21:33 +0200
> > > Pavel Vazharov <freakpv@gmail.com> wrote:
> > >
> > > > Hi there,
> > > >
> > > > We have a proxy application on top of DPDK where we use a symmetric
> > > > RSS key to receive the packets for a given connection in both
> > > > directions on a single queue. In addition to that, we've few IPv4
> > > > addresses where we need to receive their traffic on particular queues.
> > > > We use the rte flows functionality with RTE_FLOW_ACTION_TYPE_QUEUE to
> > > > achieve that and it seemed to work with the DPDK ixgbe driver.
> > > > However, today we tried the same application on top of DPDK i40e
> > > > driver and this setup doesn't seem to work there. It prints the
> > > > following errors:
> > > > `i40e_flow_add_del_fdir_filter(): Conflict with existing flow director rules!`
> > > > It seems that the i40e driver doesn't allow adding flow rules on top
> > > > of the already set RSS.
> > > >
> > > > Can somebody suggest a way to achieve what we need with i40e: to use a
> > > > symmetric RSS key for most of the traffic but to redirect the traffic
> > > > for a few specific IPv4 addresses to particular queues?
> > > >
> > > > Thanks,
> > > > Pavel.
> > >
> > > If you are mixing RSS and rte_flow the results are not well defined.
> > > Many drivers treat all active queues (including those used by rte_flow)
> > > as candidates for RSS.
> > >
> > > If you want to mix, the the safe way is:
> > >         - don't enable RSS in the device config (rx_mode)
> > >         - define an rte_flow rule with RTE_FLOW_ACTION_TYPE_RSS
> > >           with a match all
> > >         - define a rte_flow rule with RTE_FLOW_ACTION_TYPE_QUEUE
> > >           that matches the IP
> > > and set rule priorities so that specific IP rule matches before
> > > the match all.
> > Thank you for this idea. I'll give it a try.
> The proposed setup worked for NICs with e1000 and ixgb drivers but
> with i40e driver it returns error "Not support priority.".
> I poke through the code of the DPDK i40e driver and the
> `i40e_flow_parse_attr` is called for every type of flow - rss, fdir,
> etc.
> And inside this function, there is this piece of code:
>     /* Not supported */
>     if (attr->priority) {
>         rte_flow_error_set(error, EINVAL,
>                    RTE_FLOW_ERROR_TYPE_ATTR_PRIORITY,
>                    attr, "Not support priority.");
>         return -rte_errno;
>     }
> So, it seems to me that the driver don't support any flows with priority.
> >
> > > You still maybe at risk of hardware quirks and driver incompatibilities
> > > which is part of the problem with rte_flow.

Just for the record I found the core of my issue with the i40e driver.
The reason for the message "Conflict with existing flow director
rules!" was that I created the flows before starting the port.
The latter worked for e1000 and ixgb drivers but failed for i40e
driver due to the following reason:
- The fdir rules for the created flows are put internally in a
variable `fdir_list`
- When the port is started the driver calls `i40e_fdir_filter_restore`
which iterates over the `fdir_list` and tries to add again already
added flows
using `i40e_flow_add_del_fdir_filter` where
`i40e_sw_fdir_filter_insert` fails and the above error is logged.

So it seems that it's better if the `fdir_list` is empty when the port
is started and thus I moved the creation of the rte flows after the
start of the port and the issue is gone.

Thanks for the help and apologies for the possible confusion and time wasting.