RE: 21.11.4 patches review and test

["Jiang, YuX" <yux.jiang@intel.com>]

> -----Original Message-----
> From: Kevin Traynor <ktraynor@redhat.com>
> Sent: Monday, May 8, 2023 11:24 PM
> To: Xu, HailinX <hailinx.xu@intel.com>; stable@dpdk.org
> Cc: Stokes, Ian <ian.stokes@intel.com>; Mcnamara, John
> <john.mcnamara@intel.com>; Luca Boccassi <bluca@debian.org>; Xu, Qian Q
> <qian.q.xu@intel.com>; Thomas Monjalon <thomas@monjalon.net>; Peng,
> Yuan <yuan.peng@intel.com>; Chen, Zhaoyan <zhaoyan.chen@intel.com>;
> dev@dpdk.org
> Subject: Re: 21.11.4 patches review and test
> 
> On 05/05/2023 02:42, Xu, HailinX wrote:
> >> -----Original Message-----
> >> From: Kevin Traynor <ktraynor@redhat.com>
> >> Sent: Thursday, May 4, 2023 6:11 PM
> >> To: Xu, HailinX <hailinx.xu@intel.com>; stable@dpdk.org
> >> Cc: Stokes, Ian <ian.stokes@intel.com>; Mcnamara, John
> >> <john.mcnamara@intel.com>; Luca Boccassi <bluca@debian.org>; Xu,
> Qian
> >> Q <qian.q.xu@intel.com>; Thomas Monjalon <thomas@monjalon.net>;
> Peng,
> >> Yuan <yuan.peng@intel.com>; Chen, Zhaoyan
> <zhaoyan.chen@intel.com>;
> >> dev@dpdk.org
> >> Subject: Re: 21.11.4 patches review and test
> >>
> >> On 04/05/2023 03:13, Xu, HailinX wrote:
> >>>> -----Original Message-----
> >>>> From: Kevin Traynor <ktraynor@redhat.com>
> >>>> Sent: Tuesday, May 2, 2023 5:35 PM
> >>>> To: Xu, HailinX <hailinx.xu@intel.com>; stable@dpdk.org
> >>>> Cc: Stokes, Ian <ian.stokes@intel.com>; Mcnamara, John
> >>>> <john.mcnamara@intel.com>; Luca Boccassi <bluca@debian.org>; Xu,
> >>>> Qian Q <qian.q.xu@intel.com>; Thomas Monjalon
> >>>> <thomas@monjalon.net>;
> >> Peng,
> >>>> Yuan <yuan.peng@intel.com>; Chen, Zhaoyan
> <zhaoyan.chen@intel.com>;
> >>>> dev@dpdk.org
> >>>> Subject: Re: 21.11.4 patches review and test
> >>>>
> >>>> On 20/04/2023 11:32, Kevin Traynor wrote:
> >>>>> On 20/04/2023 03:40, Xu, HailinX wrote:
> >>>>>>> -----Original Message-----
> >>>>>>> From: Xu, HailinX <hailinx.xu@intel.com>
> >>>>>>> Sent: Thursday, April 13, 2023 2:13 PM
> >>>>>>> To: Kevin Traynor <ktraynor@redhat.com>; stable@dpdk.org
> >>>>>>> Cc: dev@dpdk.org; Abhishek Marathe
> >>>> <Abhishek.Marathe@microsoft.com>;
> >>>>>>> Ali Alnubani <alialnu@nvidia.com>; Walker, Benjamin
> >>>>>>> <benjamin.walker@intel.com>; David Christensen
> >>>>>>> <drc@linux.vnet.ibm.com>; Hemant Agrawal
> >>>> <hemant.agrawal@nxp.com>;
> >>>>>>> Stokes, Ian <ian.stokes@intel.com>; Jerin Jacob
> >>>>>>> <jerinj@marvell.com>; Mcnamara, John
> <john.mcnamara@intel.com>;
> >>>>>>> Ju-Hyoung Lee <juhlee@microsoft.com>; Luca Boccassi
> >>>>>>> <bluca@debian.org>; Pei Zhang <pezhang@redhat.com>; Xu, Qian
> Q
> >>>>>>> <qian.q.xu@intel.com>; Raslan Darawsheh <rasland@nvidia.com>;
> >>>> Thomas
> >>>>>>> Monjalon <thomas@monjalon.net>; yanghliu@redhat.com; Peng,
> Yuan
> >>>>>>> <yuan.peng@intel.com>; Chen, Zhaoyan
> <zhaoyan.chen@intel.com>
> >>>>>>> Subject: RE: 21.11.4 patches review and test
> >>>>>>>
> >>>>>>>> -----Original Message-----
> >>>>>>>> From: Kevin Traynor <ktraynor@redhat.com>
> >>>>>>>> Sent: Thursday, April 6, 2023 7:38 PM
> >>>>>>>> To: stable@dpdk.org
> >>>>>>>> Cc: dev@dpdk.org; Abhishek Marathe
> >>>>>>>> <Abhishek.Marathe@microsoft.com>; Ali Alnubani
> >>>>>>>> <alialnu@nvidia.com>; Walker, Benjamin
> >>>>>>>> <benjamin.walker@intel.com>; David Christensen
> >>>>>>>> <drc@linux.vnet.ibm.com>; Hemant Agrawal
> >>>>>>>> <hemant.agrawal@nxp.com>; Stokes, Ian
> <ian.stokes@intel.com>;
> >>>>>>>> Jerin Jacob <jerinj@marvell.com>; Mcnamara, John
> >>>>>>>> <john.mcnamara@intel.com>; Ju-Hyoung Lee
> >> <juhlee@microsoft.com>;
> >>>>>>>> Kevin Traynor <ktraynor@redhat.com>; Luca Boccassi
> >>>>>>>> <bluca@debian.org>; Pei Zhang <pezhang@redhat.com>; Xu,
> Qian Q
> >>>>>>>> <qian.q.xu@intel.com>; Raslan Darawsheh <rasland@nvidia.com>;
> >>>>>>> Thomas
> >>>>>>>> Monjalon <thomas@monjalon.net>; yanghliu@redhat.com; Peng,
> Yuan
> >>>>>>>> <yuan.peng@intel.com>; Chen, Zhaoyan
> <zhaoyan.chen@intel.com>
> >>>>>>>> Subject: 21.11.4 patches review and test
> >>>>>>>>
> >>>>>>>> Hi all,
> >>>>>>>>
> >>>>>>>> Here is a list of patches targeted for stable release 21.11.4.
> >>>>>>>>
> >>>>>>>> The planned date for the final release is 25th April.
> >>>>>>>>
> >>>>>>>> Please help with testing and validation of your use cases and
> >>>>>>>> report any issues/results with reply-all to this mail. For the
> >>>>>>>> final release the fixes and reported validations will be added
> >>>>>>>> to the
> >>>> release notes.
> >>>>>>>>
> >>>>>>>> A release candidate tarball can be found at:
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> https://dpdk.org/browse/dpdk-stable/tag/?id=v21.11.4-rc1
> >>>>>>>>
> >>>>>>>> These patches are located at branch 21.11 of dpdk-stable repo:
> >>>>>>>>         https://dpdk.org/browse/dpdk-stable/
> >>>>>>>>
> >>>>>>>> Thanks.
> >>>>>>>>
> >>>>>>>> Kevin
> >>>>>>>
> >>>>>>> HI All,
> >>>>>>>
> >>>>>>> Update the test status for Intel part. Till now dpdk21.11.4-rc1
> >>>>>>> validation test rate is 85%. No critical issue is found.
> >>>>>>> 2 new bugs are found, 1 new issue is under confirming by Intel Dev.
> >>>>>>> New bugs:   --20.11.8-rc1 also has these two issues
> >>>>>>>       1.
> >>>>
> pvp_qemu_multi_paths_port_restart:perf_pvp_qemu_vector_rx_mac:
> >>>>>>> performance drop about 23.5% when send small packets
> >>>>>>> 	https://bugs.dpdk.org/show_bug.cgi?id=1212    -- no fix yet
> >>>>>>>       2. some of the virtio tests are failing:    -- Intel dev is under
> >>>> investigating
> >>>>>>> # Basic Intel(R) NIC testing
> >>>>>>> * Build & CFLAG compile: cover the build test combination with
> >>>>>>> latest GCC/Clang version and the popular OS revision such as
> >>>>>>>       Ubuntu20.04, Ubuntu22.04, Fedora35, Fedora37, RHEL8.6,
> >>>>>>> RHEL8.4, FreeBSD13.1, SUSE15, CentOS7.9, etc.
> >>>>>>> - All test done. No new dpdk issue is found.
> >>>>>>> * PF(i40e, ixgbe): test scenarios including
> >>>>>>> RTE_FLOW/TSO/Jumboframe/checksum offload/VLAN/VXLAN, etc.
> >>>>>>> - All test done. No new dpdk issue is found.
> >>>>>>> * VF(i40e, ixgbe): test scenarios including
> >>>>>>> VF-RTE_FLOW/TSO/Jumboframe/checksum offload/VLAN/VXLAN,
> etc.
> >>>>>>> - All test done. No new dpdk issue is found.
> >>>>>>> * PF/VF(ice): test scenarios including Switch features/Package
> >>>>>>> Management/Flow Director/Advanced Tx/Advanced
> >> RSS/ACL/DCF/Flexible
> >>>>>>> Descriptor, etc.
> >>>>>>> - All test done. No new dpdk issue is found.
> >>>>>>> * Intel NIC single core/NIC performance: test scenarios
> >>>>>>> including PF/VF single core performance test, etc.
> >>>>>>> - All test done. No new dpdk issue is found.
> >>>>>>> * IPsec: test scenarios including ipsec/ipsec-gw/ipsec library
> >>>>>>> basic test - QAT&SW/FIB library, etc.
> >>>>>>> - On going.
> >>>>>>>
> >>>>>>> # Basic cryptodev and virtio testing
> >>>>>>> * Virtio: both function and performance test are covered. Such
> >>>>>>> as PVP/Virtio_loopback/virtio-user loopback/virtio-net VM2VM
> >>>>>>> perf testing/VMAWARE ESXI 8.0, etc.
> >>>>>>> - All test done. found bug1.
> >>>>>>> * Cryptodev:
> >>>>>>>       *Function test: test scenarios including Cryptodev API
> >>>>>>> testing/CompressDev ISA-L/QAT/ZLIB PMD Testing/FIPS, etc.
> >>>>>>>         - Execution rate is 90%. found bug2.
> >>>>>>>       *Performance test: test scenarios including Thoughput
> >>>>>>> Performance/Cryptodev Latency, etc.
> >>>>>>>         - All test done. No new dpdk issue is found.
> >>>>>>>
> >>>>>>> Regards,
> >>>>>>> Xu, Hailin
> >>>>>> Update the test status for Intel part. completed dpdk21.11.4-rc1
> >>>>>> all
> >>>> validation. No critical issue is found.
> >>>>>
> >>>>> Hi. Thanks for testing.
> >>>>>
> >>>>>> 2 new bugs are found, 1 new issue is under confirming by Intel Dev.
> >>>>>> New bugs: --20.11.8-rc1 also has these two issues
> >>>>>>       1.
> >>>>
> pvp_qemu_multi_paths_port_restart:perf_pvp_qemu_vector_rx_mac:
> >>>> performance drop about 23.5% when send small packets
> >>>>>> 	https://bugs.dpdk.org/show_bug.cgi?id=1212  --not fix yet,
> Only
> >>>>>> the specified platform exists
> >>>>>
> >>>>> Do you know which patch caaused the regression? I'm not fully
> >>>>> clear from the Bz for 20.11. The backported patch ID'd as root
> >>>>> cause [0] in
> >>>>> 20.11 is in the previous releases of 20.11 (and 21.11).
> >>>>>
> >>>>> Trying to understand because then it would have shown in testing
> >>>>> for previous releases. Or is this a new test introduced for latest
> >>>>> LTS releases? and if so, what is the baseline performance based on?
> >>>>>
> >>>>> [0]
> >>>>> commit 1c9a7fba5c90e0422b517404499ed106f647bcff
> >>>>> Author: Mattias Rönnblom <mattias.ronnblom@ericsson.com>
> >>>>> Date:   Mon Jul 11 14:11:32 2022 +0200
> >>>>>
> >>>>>         net: accept unaligned data in checksum routines
> >>>>>
> >>>>>>       2. some of the virtio tests are failing: -- Intel dev is
> >>>>>> under investigating
> >>>>>
> >>>>> ok, thank you.
> >>>>>
> >>>>
> >>>> Hi, it was mentioned in a separate mail that the performance drop
> >>>> issue was not the same as 20.11.
> >>>>
> >>>> Is there any update on that and the failing virtio tests ? Is there
> >>>> a regression introduced since in 21.11.4 ?
> >>>
> >>> Found bad commit id:
> >>> commit c41493361c87e730459ead9311c68528eb0874aa
> >>> Author: Boleslav Stankevich <boleslav.stankevich@oktetlabs.ru>
> >>> Date:   Fri Mar 3 14:19:29 2023 +0300
> >>>       net/virtio: deduce IP length for TSO checksum
> >>>
> >>> We try this issue on 3 different platforms Performance drop only
> >>> found on Intel(R) Xeon(R) Platinum 8280M CPU @ 2.70GHz CPU.
> >>>
> >>
> >> Ok, thanks for reporting. This commit was also in 23.03 and 20.11.8
> >> so not sure if they were tested on this platform?
> >>
> > Yes, But 23.03 doesn't have such issue on the same platform
> >
> 
> ok, thanks.
> 
> >> This commit fixes a functional issue, so I don't think it should be
> >> removed unless it is critical issue.
> >>
> >> There's no update on other performance report. Release is already 1
> >> week overdue, is there critical issues that we should hold release for?
> >>
> > No other issues found
> >
> 
> I'm not fully clear on the status, so let's separate. There were two reported
> issues.
> 
> 1. Performance drop on specific Intel platform.
> - Is this being investigated by Intel dev?
> - Is it is a release blocking issue ?
> 
> 2. Virtio issue caused by
> 
> commit c41493361c87e730459ead9311c68528eb0874aa
> Author: Boleslav Stankevich <boleslav.stankevich@oktetlabs.ru>
> Date:   Fri Mar 3 14:19:29 2023 +0300
>         net/virtio: deduce IP length for TSO checksum
> 
> - Is this being investigated by Intel dev?
> - Is it is a release blocking issue ?
> 
> thanks,
> Kevin.
> 
Hi Kevin,

I need correct/clarify the description of the two issues:
1, Performance drop on specific Intel platform -> should be pvp_qemu_multi_paths_port_restart:perf_pvp_qemu_vector_rx_mac: Benchmark pvp performance drop about 23.5% when send small packets" 
Its bad commit id: 
commit c41493361c87e730459ead9311c68528eb0874aa
Author: Boleslav Stankevich <boleslav.stankevich@oktetlabs.ru>
Date:   Fri Mar 3 14:19:29 2023 +0300

    net/virtio: deduce IP length for TSO checksum

    [ upstream commit d069c80a5d8c0a05033932421851cdb7159de0df ]

    The length of TSO payload could not fit into 16 bits provided by the
    IPv4 total length and IPv6 payload length fields. Thus, deduce it
    from the length of the packet.

    Fixes: 696573046e9e ("net/virtio: support TSO")

    Signed-off-by: Boleslav Stankevich <boleslav.stankevich@oktetlabs.ru>
    Reviewed-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
    Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com> 

Intel validation owner has done lots of verification and compared these performance data on 8280&8380 platforms.
This performance big drop is only found on special platform (Intel(R) Xeon(R) Platinum 8280M CPU @ 2.70GHz CPU) when vhost-user(back-end) 
and virtio-pmd(front-end) are not on the same socket (front-end is on the socket1, back-end and nic are on the socket0).
But our test case doesn't call this bad commit id's related code, suspect whether it may be related to compiler optimization on special platform. 
We hope the bad commit id owner or other experts can support to analysis the root cause.  

2, Virtio issue -> should be virtio crypto test failure.
It is not a regression issue, old lts and main branch also reproduce by validation owner, and there's no clear bad commit id found. 
Intel dev and validation owner are investigating it, but we don't find out the root cause yet. 

Best regards,
Yu Jiang

> > Regards,
> > Xu, Hailin
> >
> >> thanks,
> >> Kevin.
> >>
> >>> Regards,
> >>> Xu, Hailin
> >>>
> >>>>
> >>>> thanks,
> >>>> Kevin,
> >>>>
> >>>>> Kevin.
> >>>>>
> >>>>>> # Basic Intel(R) NIC testing
> >>>>>> * Build & CFLAG compile: cover the build test combination with
> >>>>>> latest
> >>>> GCC/Clang version and the popular OS revision such as
> >>>>>>       Ubuntu20.04, Ubuntu22.04, Fedora35, Fedora37, RHEL8.6,
> >>>>>> RHEL8.4,
> >>>> FreeBSD13.1, SUSE15, CentOS7.9, etc.
> >>>>>> - All test done. No new dpdk issue is found.
> >>>>>> * PF(i40e, ixgbe): test scenarios including
> >>>> RTE_FLOW/TSO/Jumboframe/checksum offload/VLAN/VXLAN, etc.
> >>>>>> - All test done. No new dpdk issue is found.
> >>>>>> * VF(i40e, ixgbe): test scenarios including
> >>>> VF-RTE_FLOW/TSO/Jumboframe/checksum offload/VLAN/VXLAN, etc.
> >>>>>> - All test done. No new dpdk issue is found.
> >>>>>> * PF/VF(ice): test scenarios including Switch features/Package
> >>>> Management/Flow Director/Advanced Tx/Advanced
> RSS/ACL/DCF/Flexible
> >>>> Descriptor, etc.
> >>>>>> - All test done. No new dpdk issue is found.
> >>>>>> * Intel NIC single core/NIC performance: test scenarios including
> >>>>>> PF/VF
> >>>> single core performance test, etc.
> >>>>>> - All test done. No new dpdk issue is found.
> >>>>>> * IPsec: test scenarios including ipsec/ipsec-gw/ipsec library
> >>>>>> basic test -
> >>>> QAT&SW/FIB library, etc.
> >>>>>> - All test done. No new dpdk issue is found.
> >>>>>>
> >>>>>> # Basic cryptodev and virtio testing
> >>>>>> * Virtio: both function and performance test are covered. Such as
> >>>> PVP/Virtio_loopback/virtio-user loopback/virtio-net VM2VM perf
> >>>> testing/VMAWARE ESXI 8.0, etc.
> >>>>>> - All test done. found bug1.
> >>>>>> * Cryptodev:
> >>>>>>       *Function test: test scenarios including Cryptodev API
> >>>> testing/CompressDev ISA-L/QAT/ZLIB PMD Testing/FIPS, etc.
> >>>>>>         - All test done. found bug2.
> >>>>>>       *Performance test: test scenarios including Thoughput
> >>>> Performance/Cryptodev Latency, etc.
> >>>>>>         - All test done. No new dpdk issue is found.
> >>>>>>
> >>>>>> Regards,
> >>>>>> Xu, Hailin
> >>>>>>
> >>>>>
> >>>
> >