Performance of CX7 with 'eth' pattern versus 'eth/ipv4' in hairpin

Performance of CX7 with 'eth' pattern versus 'eth/ipv4' in hairpin

[Tony Hart]

I'm using a CX7 and testing hairpin queues.  The test traffic is
entirely IPv4+UDP with distributed SIP,DIP pairs and received packets
are u-turned via hairpin in the CX7 (single 400G interface).

I see different performance when I use a pattern of 'eth' versus
'eth/ipv4' in the hairpin flow entry.  From testing it seems that
specifying just 'eth' is sufficient to invoke RSS and 'eth/ipv4'
should be equivalent since the traffic is all ipv4, but I'm getting
~104Gbps for the 'eth' pattern and  ~124Gbps for 'eth/ipv4' pattern.

Any thoughts on why there is such a performance difference here?

thanks
tony

This is the 'eth' pattern testpmd commands
flow create 0 ingress group 0 pattern end actions jump group 1 / end
flow create 0 ingress group 1 pattern eth / end actions count / rss
queues 6 7 8 9 end / end

The testpmd commands for 'eth/ipv4'
flow create 0 ingress group 0 pattern end actions jump group 1 / end
flow create 0 ingress group 1 pattern eth / ipv4 / end actions count /
rss queues 6 7 8 9 end / end


This is the testpmd command line...
dpdk-testpmd -l8-14 -a81:00.0,dv_flow_en=1 -- -i --nb-cores 6 --rxq 6
--txq 6 --port-topology loop --forward-mode=rxonly --hairpinq 4
--hairpin-mode 0x10

Versions
mlnx-ofa_kernel-24.04-OFED.24.04.0.6.6.1.rhel9u4.x86_64
kmod-mlnx-ofa_kernel-24.04-OFED.24.04.0.6.6.1.rhel9u4.x86_64
mlnx-ofa_kernel-devel-24.04-OFED.24.04.0.6.6.1.rhel9u4.x86_64
ofed-scripts-24.04-OFED.24.04.0.6.6.x86_64

DPDK: v24.03

RE: Performance of CX7 with 'eth' pattern versus 'eth/ipv4' in hairpin

[Bing Zhao]

Hi Tony,

Could you also try to test with:
1. QUEUE action instead of RSS and check 1 queue performance.
2. when trying to test IPv4 only case, try the following 3 commands with this order -
	flow create 0 ingress group 0 pattern end actions jump group 1 / end
	flow create 0 ingress group 1 pattern priority 1 eth / ipv6 / end actions count / rss queues 6 7 8 9 end / end
	flow create 0 ingress group 1 pattern priority 1 eth / ipv4 / end actions count / rss queues 6 7 8 9 end / end

BR. Bing

> -----Original Message-----
> From: Tony Hart <tony.hart@domainhart.com>
> Sent: Wednesday, June 26, 2024 7:39 PM
> To: users@dpdk.org
> Subject: Performance of CX7 with 'eth' pattern versus 'eth/ipv4' in hairpin
> 
> External email: Use caution opening links or attachments
> 
> 
> I'm using a CX7 and testing hairpin queues.  The test traffic is entirely
> IPv4+UDP with distributed SIP,DIP pairs and received packets are u-turned via
> hairpin in the CX7 (single 400G interface).
> 
> I see different performance when I use a pattern of 'eth' versus 'eth/ipv4' in
> the hairpin flow entry.  From testing it seems that specifying just 'eth' is
> sufficient to invoke RSS and 'eth/ipv4'
> should be equivalent since the traffic is all ipv4, but I'm getting ~104Gbps for
> the 'eth' pattern and  ~124Gbps for 'eth/ipv4' pattern.
> 
> Any thoughts on why there is such a performance difference here?
> 
> thanks
> tony
> 
> This is the 'eth' pattern testpmd commands flow create 0 ingress group 0
> pattern end actions jump group 1 / end flow create 0 ingress group 1 pattern
> eth / end actions count / rss queues 6 7 8 9 end / end
> 
> The testpmd commands for 'eth/ipv4'
> flow create 0 ingress group 0 pattern end actions jump group 1 / end flow
> create 0 ingress group 1 pattern eth / ipv4 / end actions count / rss queues 6 7
> 8 9 end / end
> 
> 
> This is the testpmd command line...
> dpdk-testpmd -l8-14 -a81:00.0,dv_flow_en=1 -- -i --nb-cores 6 --rxq 6 --txq 6
> --port-topology loop --forward-mode=rxonly --hairpinq 4 --hairpin-mode
> 0x10
> 
> Versions
> mlnx-ofa_kernel-24.04-OFED.24.04.0.6.6.1.rhel9u4.x86_64
> kmod-mlnx-ofa_kernel-24.04-OFED.24.04.0.6.6.1.rhel9u4.x86_64
> mlnx-ofa_kernel-devel-24.04-OFED.24.04.0.6.6.1.rhel9u4.x86_64
> ofed-scripts-24.04-OFED.24.04.0.6.6.x86_64
> 
> DPDK: v24.03

Re: Performance of CX7 with 'eth' pattern versus 'eth/ipv4' in hairpin

[Tony Hart]

Hi Bing,
Thanks for the quick reply.  The results are...

With a single hairpin queue I get approx the same rate for both
patterns, ~54Gbps.  I assume this is less than the RSS rates due to
fewer queues?
flow create 0 ingress group 1 pattern eth / end actions count / queue
index 6 / end
flow create 0 ingress group 1 pattern eth / ipv4 / end actions count /
queue index 6 / end

With the split ipv6/ipv4 I'm getting ~124Gbps

flow create 0 ingress group 1 priority 1 pattern eth / ipv6 / end
actions count / rss queues 6 7 8 9 end / end
flow create 0 ingress group 1 priority 1 pattern eth / ipv4 / end
actions count / rss queues 6 7 8 9 end / end

testpmd> flow list 0
ID Group Prio Attr Rule
0 0 0 i-- => JUMP
1 1 1 i-- ETH IPV6 => COUNT RSS
2 1 1 i-- ETH IPV4 => COUNT RSS

On Wed, Jun 26, 2024 at 8:10 AM Bing Zhao <bingz@nvidia.com> wrote:
>
> Hi Tony,
>
> Could you also try to test with:
> 1. QUEUE action instead of RSS and check 1 queue performance.
> 2. when trying to test IPv4 only case, try the following 3 commands with this order -
>         flow create 0 ingress group 0 pattern end actions jump group 1 / end
>         flow create 0 ingress group 1 pattern priority 1 eth / ipv6 / end actions count / rss queues 6 7 8 9 end / end
>         flow create 0 ingress group 1 pattern priority 1 eth / ipv4 / end actions count / rss queues 6 7 8 9 end / end
>
> BR. Bing
>
> > -----Original Message-----
> > From: Tony Hart <tony.hart@domainhart.com>
> > Sent: Wednesday, June 26, 2024 7:39 PM
> > To: users@dpdk.org
> > Subject: Performance of CX7 with 'eth' pattern versus 'eth/ipv4' in hairpin
> >
> > External email: Use caution opening links or attachments
> >
> >
> > I'm using a CX7 and testing hairpin queues.  The test traffic is entirely
> > IPv4+UDP with distributed SIP,DIP pairs and received packets are u-turned via
> > hairpin in the CX7 (single 400G interface).
> >
> > I see different performance when I use a pattern of 'eth' versus 'eth/ipv4' in
> > the hairpin flow entry.  From testing it seems that specifying just 'eth' is
> > sufficient to invoke RSS and 'eth/ipv4'
> > should be equivalent since the traffic is all ipv4, but I'm getting ~104Gbps for
> > the 'eth' pattern and  ~124Gbps for 'eth/ipv4' pattern.
> >
> > Any thoughts on why there is such a performance difference here?
> >
> > thanks
> > tony
> >
> > This is the 'eth' pattern testpmd commands flow create 0 ingress group 0
> > pattern end actions jump group 1 / end flow create 0 ingress group 1 pattern
> > eth / end actions count / rss queues 6 7 8 9 end / end
> >
> > The testpmd commands for 'eth/ipv4'
> > flow create 0 ingress group 0 pattern end actions jump group 1 / end flow
> > create 0 ingress group 1 pattern eth / ipv4 / end actions count / rss queues 6 7
> > 8 9 end / end
> >
> >
> > This is the testpmd command line...
> > dpdk-testpmd -l8-14 -a81:00.0,dv_flow_en=1 -- -i --nb-cores 6 --rxq 6 --txq 6
> > --port-topology loop --forward-mode=rxonly --hairpinq 4 --hairpin-mode
> > 0x10
> >
> > Versions
> > mlnx-ofa_kernel-24.04-OFED.24.04.0.6.6.1.rhel9u4.x86_64
> > kmod-mlnx-ofa_kernel-24.04-OFED.24.04.0.6.6.1.rhel9u4.x86_64
> > mlnx-ofa_kernel-devel-24.04-OFED.24.04.0.6.6.1.rhel9u4.x86_64
> > ofed-scripts-24.04-OFED.24.04.0.6.6.x86_64
> >
> > DPDK: v24.03



-- 
tony

Re: Performance of CX7 with 'eth' pattern versus 'eth/ipv4' in hairpin

[Tony Hart]

[-- Attachment #1: Type: text/plain, Size: 4214 bytes --]

FYI if I add a 'eth' only line (e.g. to match ARP packets) the performance
remains the same (which is good).

flow create 0 ingress group 1 priority 1 pattern eth / ipv6 / end
actions count / rss queues 6 7 8 9 end / end
flow create 0 ingress group 1 priority 1 pattern eth / ipv4 / end
actions count / rss queues 6 7 8 9 end / end
flow create 0 ingress group 1 priority 2 pattern eth / end
actions count / rss queues 6 7 8 9 end / end

Any thoughts why the single 'pattern eth / end' performance is worse?


On Wed, Jun 26, 2024 at 9:24 AM Tony Hart <tony.hart@domainhart.com> wrote:

> Hi Bing,
> Thanks for the quick reply.  The results are...
>
> With a single hairpin queue I get approx the same rate for both
> patterns, ~54Gbps.  I assume this is less than the RSS rates due to
> fewer queues?
> flow create 0 ingress group 1 pattern eth / end actions count / queue
> index 6 / end
> flow create 0 ingress group 1 pattern eth / ipv4 / end actions count /
> queue index 6 / end
>
> With the split ipv6/ipv4 I'm getting ~124Gbps
>
> flow create 0 ingress group 1 priority 1 pattern eth / ipv6 / end
> actions count / rss queues 6 7 8 9 end / end
> flow create 0 ingress group 1 priority 1 pattern eth / ipv4 / end
> actions count / rss queues 6 7 8 9 end / end
>
> testpmd> flow list 0
> ID Group Prio Attr Rule
> 0 0 0 i-- => JUMP
> 1 1 1 i-- ETH IPV6 => COUNT RSS
> 2 1 1 i-- ETH IPV4 => COUNT RSS
>
> On Wed, Jun 26, 2024 at 8:10 AM Bing Zhao <bingz@nvidia.com> wrote:
> >
> > Hi Tony,
> >
> > Could you also try to test with:
> > 1. QUEUE action instead of RSS and check 1 queue performance.
> > 2. when trying to test IPv4 only case, try the following 3 commands with
> this order -
> >         flow create 0 ingress group 0 pattern end actions jump group 1 /
> end
> >         flow create 0 ingress group 1 pattern priority 1 eth / ipv6 /
> end actions count / rss queues 6 7 8 9 end / end
> >         flow create 0 ingress group 1 pattern priority 1 eth / ipv4 /
> end actions count / rss queues 6 7 8 9 end / end
> >
> > BR. Bing
> >
> > > -----Original Message-----
> > > From: Tony Hart <tony.hart@domainhart.com>
> > > Sent: Wednesday, June 26, 2024 7:39 PM
> > > To: users@dpdk.org
> > > Subject: Performance of CX7 with 'eth' pattern versus 'eth/ipv4' in
> hairpin
> > >
> > > External email: Use caution opening links or attachments
> > >
> > >
> > > I'm using a CX7 and testing hairpin queues.  The test traffic is
> entirely
> > > IPv4+UDP with distributed SIP,DIP pairs and received packets are
> u-turned via
> > > hairpin in the CX7 (single 400G interface).
> > >
> > > I see different performance when I use a pattern of 'eth' versus
> 'eth/ipv4' in
> > > the hairpin flow entry.  From testing it seems that specifying just
> 'eth' is
> > > sufficient to invoke RSS and 'eth/ipv4'
> > > should be equivalent since the traffic is all ipv4, but I'm getting
> ~104Gbps for
> > > the 'eth' pattern and  ~124Gbps for 'eth/ipv4' pattern.
> > >
> > > Any thoughts on why there is such a performance difference here?
> > >
> > > thanks
> > > tony
> > >
> > > This is the 'eth' pattern testpmd commands flow create 0 ingress group
> 0
> > > pattern end actions jump group 1 / end flow create 0 ingress group 1
> pattern
> > > eth / end actions count / rss queues 6 7 8 9 end / end
> > >
> > > The testpmd commands for 'eth/ipv4'
> > > flow create 0 ingress group 0 pattern end actions jump group 1 / end
> flow
> > > create 0 ingress group 1 pattern eth / ipv4 / end actions count / rss
> queues 6 7
> > > 8 9 end / end
> > >
> > >
> > > This is the testpmd command line...
> > > dpdk-testpmd -l8-14 -a81:00.0,dv_flow_en=1 -- -i --nb-cores 6 --rxq 6
> --txq 6
> > > --port-topology loop --forward-mode=rxonly --hairpinq 4 --hairpin-mode
> > > 0x10
> > >
> > > Versions
> > > mlnx-ofa_kernel-24.04-OFED.24.04.0.6.6.1.rhel9u4.x86_64
> > > kmod-mlnx-ofa_kernel-24.04-OFED.24.04.0.6.6.1.rhel9u4.x86_64
> > > mlnx-ofa_kernel-devel-24.04-OFED.24.04.0.6.6.1.rhel9u4.x86_64
> > > ofed-scripts-24.04-OFED.24.04.0.6.6.x86_64
> > >
> > > DPDK: v24.03
>
>
>
> --
> tony
>


-- 
tony

[-- Attachment #2: Type: text/html, Size: 5969 bytes --]