Re: 21.11.4 patches review and test

[Kevin Traynor <ktraynor@redhat.com>]

On 11/05/2023 08:33, Jiang, YuX wrote:
>> -----Original Message-----
>> From: Kevin Traynor <ktraynor@redhat.com>
>> Sent: Monday, May 8, 2023 11:24 PM
>> To: Xu, HailinX <hailinx.xu@intel.com>; stable@dpdk.org
>> Cc: Stokes, Ian <ian.stokes@intel.com>; Mcnamara, John
>> <john.mcnamara@intel.com>; Luca Boccassi <bluca@debian.org>; Xu, Qian Q
>> <qian.q.xu@intel.com>; Thomas Monjalon <thomas@monjalon.net>; Peng,
>> Yuan <yuan.peng@intel.com>; Chen, Zhaoyan <zhaoyan.chen@intel.com>;
>> dev@dpdk.org
>> Subject: Re: 21.11.4 patches review and test
>>
>> On 05/05/2023 02:42, Xu, HailinX wrote:
>>>> -----Original Message-----
>>>> From: Kevin Traynor <ktraynor@redhat.com>
>>>> Sent: Thursday, May 4, 2023 6:11 PM
>>>> To: Xu, HailinX <hailinx.xu@intel.com>; stable@dpdk.org
>>>> Cc: Stokes, Ian <ian.stokes@intel.com>; Mcnamara, John
>>>> <john.mcnamara@intel.com>; Luca Boccassi <bluca@debian.org>; Xu,
>> Qian
>>>> Q <qian.q.xu@intel.com>; Thomas Monjalon <thomas@monjalon.net>;
>> Peng,
>>>> Yuan <yuan.peng@intel.com>; Chen, Zhaoyan
>> <zhaoyan.chen@intel.com>;
>>>> dev@dpdk.org
>>>> Subject: Re: 21.11.4 patches review and test
>>>>
>>>> On 04/05/2023 03:13, Xu, HailinX wrote:
>>>>>> -----Original Message-----
>>>>>> From: Kevin Traynor <ktraynor@redhat.com>
>>>>>> Sent: Tuesday, May 2, 2023 5:35 PM
>>>>>> To: Xu, HailinX <hailinx.xu@intel.com>; stable@dpdk.org
>>>>>> Cc: Stokes, Ian <ian.stokes@intel.com>; Mcnamara, John
>>>>>> <john.mcnamara@intel.com>; Luca Boccassi <bluca@debian.org>; Xu,
>>>>>> Qian Q <qian.q.xu@intel.com>; Thomas Monjalon
>>>>>> <thomas@monjalon.net>;
>>>> Peng,
>>>>>> Yuan <yuan.peng@intel.com>; Chen, Zhaoyan
>> <zhaoyan.chen@intel.com>;
>>>>>> dev@dpdk.org
>>>>>> Subject: Re: 21.11.4 patches review and test
>>>>>>
>>>>>> On 20/04/2023 11:32, Kevin Traynor wrote:
>>>>>>> On 20/04/2023 03:40, Xu, HailinX wrote:
>>>>>>>>> -----Original Message-----
>>>>>>>>> From: Xu, HailinX <hailinx.xu@intel.com>
>>>>>>>>> Sent: Thursday, April 13, 2023 2:13 PM
>>>>>>>>> To: Kevin Traynor <ktraynor@redhat.com>; stable@dpdk.org
>>>>>>>>> Cc: dev@dpdk.org; Abhishek Marathe
>>>>>> <Abhishek.Marathe@microsoft.com>;
>>>>>>>>> Ali Alnubani <alialnu@nvidia.com>; Walker, Benjamin
>>>>>>>>> <benjamin.walker@intel.com>; David Christensen
>>>>>>>>> <drc@linux.vnet.ibm.com>; Hemant Agrawal
>>>>>> <hemant.agrawal@nxp.com>;
>>>>>>>>> Stokes, Ian <ian.stokes@intel.com>; Jerin Jacob
>>>>>>>>> <jerinj@marvell.com>; Mcnamara, John
>> <john.mcnamara@intel.com>;
>>>>>>>>> Ju-Hyoung Lee <juhlee@microsoft.com>; Luca Boccassi
>>>>>>>>> <bluca@debian.org>; Pei Zhang <pezhang@redhat.com>; Xu, Qian
>> Q
>>>>>>>>> <qian.q.xu@intel.com>; Raslan Darawsheh <rasland@nvidia.com>;
>>>>>> Thomas
>>>>>>>>> Monjalon <thomas@monjalon.net>; yanghliu@redhat.com; Peng,
>> Yuan
>>>>>>>>> <yuan.peng@intel.com>; Chen, Zhaoyan
>> <zhaoyan.chen@intel.com>
>>>>>>>>> Subject: RE: 21.11.4 patches review and test
>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: Kevin Traynor <ktraynor@redhat.com>
>>>>>>>>>> Sent: Thursday, April 6, 2023 7:38 PM
>>>>>>>>>> To: stable@dpdk.org
>>>>>>>>>> Cc: dev@dpdk.org; Abhishek Marathe
>>>>>>>>>> <Abhishek.Marathe@microsoft.com>; Ali Alnubani
>>>>>>>>>> <alialnu@nvidia.com>; Walker, Benjamin
>>>>>>>>>> <benjamin.walker@intel.com>; David Christensen
>>>>>>>>>> <drc@linux.vnet.ibm.com>; Hemant Agrawal
>>>>>>>>>> <hemant.agrawal@nxp.com>; Stokes, Ian
>> <ian.stokes@intel.com>;
>>>>>>>>>> Jerin Jacob <jerinj@marvell.com>; Mcnamara, John
>>>>>>>>>> <john.mcnamara@intel.com>; Ju-Hyoung Lee
>>>> <juhlee@microsoft.com>;
>>>>>>>>>> Kevin Traynor <ktraynor@redhat.com>; Luca Boccassi
>>>>>>>>>> <bluca@debian.org>; Pei Zhang <pezhang@redhat.com>; Xu,
>> Qian Q
>>>>>>>>>> <qian.q.xu@intel.com>; Raslan Darawsheh <rasland@nvidia.com>;
>>>>>>>>> Thomas
>>>>>>>>>> Monjalon <thomas@monjalon.net>; yanghliu@redhat.com; Peng,
>> Yuan
>>>>>>>>>> <yuan.peng@intel.com>; Chen, Zhaoyan
>> <zhaoyan.chen@intel.com>
>>>>>>>>>> Subject: 21.11.4 patches review and test
>>>>>>>>>>
>>>>>>>>>> Hi all,
>>>>>>>>>>
>>>>>>>>>> Here is a list of patches targeted for stable release 21.11.4.
>>>>>>>>>>
>>>>>>>>>> The planned date for the final release is 25th April.
>>>>>>>>>>
>>>>>>>>>> Please help with testing and validation of your use cases and
>>>>>>>>>> report any issues/results with reply-all to this mail. For the
>>>>>>>>>> final release the fixes and reported validations will be added
>>>>>>>>>> to the
>>>>>> release notes.
>>>>>>>>>>
>>>>>>>>>> A release candidate tarball can be found at:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> https://dpdk.org/browse/dpdk-stable/tag/?id=v21.11.4-rc1
>>>>>>>>>>
>>>>>>>>>> These patches are located at branch 21.11 of dpdk-stable repo:
>>>>>>>>>>          https://dpdk.org/browse/dpdk-stable/
>>>>>>>>>>
>>>>>>>>>> Thanks.
>>>>>>>>>>
>>>>>>>>>> Kevin
>>>>>>>>>
>>>>>>>>> HI All,
>>>>>>>>>
>>>>>>>>> Update the test status for Intel part. Till now dpdk21.11.4-rc1
>>>>>>>>> validation test rate is 85%. No critical issue is found.
>>>>>>>>> 2 new bugs are found, 1 new issue is under confirming by Intel Dev.
>>>>>>>>> New bugs:   --20.11.8-rc1 also has these two issues
>>>>>>>>>        1.
>>>>>>
>> pvp_qemu_multi_paths_port_restart:perf_pvp_qemu_vector_rx_mac:
>>>>>>>>> performance drop about 23.5% when send small packets
>>>>>>>>> 	https://bugs.dpdk.org/show_bug.cgi?id=1212    -- no fix yet
>>>>>>>>>        2. some of the virtio tests are failing:    -- Intel dev is under
>>>>>> investigating
>>>>>>>>> # Basic Intel(R) NIC testing
>>>>>>>>> * Build & CFLAG compile: cover the build test combination with
>>>>>>>>> latest GCC/Clang version and the popular OS revision such as
>>>>>>>>>        Ubuntu20.04, Ubuntu22.04, Fedora35, Fedora37, RHEL8.6,
>>>>>>>>> RHEL8.4, FreeBSD13.1, SUSE15, CentOS7.9, etc.
>>>>>>>>> - All test done. No new dpdk issue is found.
>>>>>>>>> * PF(i40e, ixgbe): test scenarios including
>>>>>>>>> RTE_FLOW/TSO/Jumboframe/checksum offload/VLAN/VXLAN, etc.
>>>>>>>>> - All test done. No new dpdk issue is found.
>>>>>>>>> * VF(i40e, ixgbe): test scenarios including
>>>>>>>>> VF-RTE_FLOW/TSO/Jumboframe/checksum offload/VLAN/VXLAN,
>> etc.
>>>>>>>>> - All test done. No new dpdk issue is found.
>>>>>>>>> * PF/VF(ice): test scenarios including Switch features/Package
>>>>>>>>> Management/Flow Director/Advanced Tx/Advanced
>>>> RSS/ACL/DCF/Flexible
>>>>>>>>> Descriptor, etc.
>>>>>>>>> - All test done. No new dpdk issue is found.
>>>>>>>>> * Intel NIC single core/NIC performance: test scenarios
>>>>>>>>> including PF/VF single core performance test, etc.
>>>>>>>>> - All test done. No new dpdk issue is found.
>>>>>>>>> * IPsec: test scenarios including ipsec/ipsec-gw/ipsec library
>>>>>>>>> basic test - QAT&SW/FIB library, etc.
>>>>>>>>> - On going.
>>>>>>>>>
>>>>>>>>> # Basic cryptodev and virtio testing
>>>>>>>>> * Virtio: both function and performance test are covered. Such
>>>>>>>>> as PVP/Virtio_loopback/virtio-user loopback/virtio-net VM2VM
>>>>>>>>> perf testing/VMAWARE ESXI 8.0, etc.
>>>>>>>>> - All test done. found bug1.
>>>>>>>>> * Cryptodev:
>>>>>>>>>        *Function test: test scenarios including Cryptodev API
>>>>>>>>> testing/CompressDev ISA-L/QAT/ZLIB PMD Testing/FIPS, etc.
>>>>>>>>>          - Execution rate is 90%. found bug2.
>>>>>>>>>        *Performance test: test scenarios including Thoughput
>>>>>>>>> Performance/Cryptodev Latency, etc.
>>>>>>>>>          - All test done. No new dpdk issue is found.
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Xu, Hailin
>>>>>>>> Update the test status for Intel part. completed dpdk21.11.4-rc1
>>>>>>>> all
>>>>>> validation. No critical issue is found.
>>>>>>>
>>>>>>> Hi. Thanks for testing.
>>>>>>>
>>>>>>>> 2 new bugs are found, 1 new issue is under confirming by Intel Dev.
>>>>>>>> New bugs: --20.11.8-rc1 also has these two issues
>>>>>>>>        1.
>>>>>>
>> pvp_qemu_multi_paths_port_restart:perf_pvp_qemu_vector_rx_mac:
>>>>>> performance drop about 23.5% when send small packets
>>>>>>>> 	https://bugs.dpdk.org/show_bug.cgi?id=1212  --not fix yet,
>> Only
>>>>>>>> the specified platform exists
>>>>>>>
>>>>>>> Do you know which patch caaused the regression? I'm not fully
>>>>>>> clear from the Bz for 20.11. The backported patch ID'd as root
>>>>>>> cause [0] in
>>>>>>> 20.11 is in the previous releases of 20.11 (and 21.11).
>>>>>>>
>>>>>>> Trying to understand because then it would have shown in testing
>>>>>>> for previous releases. Or is this a new test introduced for latest
>>>>>>> LTS releases? and if so, what is the baseline performance based on?
>>>>>>>
>>>>>>> [0]
>>>>>>> commit 1c9a7fba5c90e0422b517404499ed106f647bcff
>>>>>>> Author: Mattias Rönnblom <mattias.ronnblom@ericsson.com>
>>>>>>> Date:   Mon Jul 11 14:11:32 2022 +0200
>>>>>>>
>>>>>>>          net: accept unaligned data in checksum routines
>>>>>>>
>>>>>>>>        2. some of the virtio tests are failing: -- Intel dev is
>>>>>>>> under investigating
>>>>>>>
>>>>>>> ok, thank you.
>>>>>>>
>>>>>>
>>>>>> Hi, it was mentioned in a separate mail that the performance drop
>>>>>> issue was not the same as 20.11.
>>>>>>
>>>>>> Is there any update on that and the failing virtio tests ? Is there
>>>>>> a regression introduced since in 21.11.4 ?
>>>>>
>>>>> Found bad commit id:
>>>>> commit c41493361c87e730459ead9311c68528eb0874aa
>>>>> Author: Boleslav Stankevich <boleslav.stankevich@oktetlabs.ru>
>>>>> Date:   Fri Mar 3 14:19:29 2023 +0300
>>>>>        net/virtio: deduce IP length for TSO checksum
>>>>>
>>>>> We try this issue on 3 different platforms Performance drop only
>>>>> found on Intel(R) Xeon(R) Platinum 8280M CPU @ 2.70GHz CPU.
>>>>>
>>>>
>>>> Ok, thanks for reporting. This commit was also in 23.03 and 20.11.8
>>>> so not sure if they were tested on this platform?
>>>>
>>> Yes, But 23.03 doesn't have such issue on the same platform
>>>
>>
>> ok, thanks.
>>
>>>> This commit fixes a functional issue, so I don't think it should be
>>>> removed unless it is critical issue.
>>>>
>>>> There's no update on other performance report. Release is already 1
>>>> week overdue, is there critical issues that we should hold release for?
>>>>
>>> No other issues found
>>>
>>
>> I'm not fully clear on the status, so let's separate. There were two reported
>> issues.
>>
>> 1. Performance drop on specific Intel platform.
>> - Is this being investigated by Intel dev?
>> - Is it is a release blocking issue ?
>>
>> 2. Virtio issue caused by
>>
>> commit c41493361c87e730459ead9311c68528eb0874aa
>> Author: Boleslav Stankevich <boleslav.stankevich@oktetlabs.ru>
>> Date:   Fri Mar 3 14:19:29 2023 +0300
>>          net/virtio: deduce IP length for TSO checksum
>>
>> - Is this being investigated by Intel dev?
>> - Is it is a release blocking issue ?
>>
>> thanks,
>> Kevin.
>>
> Hi Kevin,
> 

Hi Yu Jang,

Thanks for the information.

> I need correct/clarify the description of the two issues:
> 1, Performance drop on specific Intel platform -> should be pvp_qemu_multi_paths_port_restart:perf_pvp_qemu_vector_rx_mac: Benchmark pvp performance drop about 23.5% when send small packets"
> Its bad commit id:
> commit c41493361c87e730459ead9311c68528eb0874aa
> Author: Boleslav Stankevich <boleslav.stankevich@oktetlabs.ru>
> Date:   Fri Mar 3 14:19:29 2023 +0300
> 
>      net/virtio: deduce IP length for TSO checksum
> 
>      [ upstream commit d069c80a5d8c0a05033932421851cdb7159de0df ]
> 
>      The length of TSO payload could not fit into 16 bits provided by the
>      IPv4 total length and IPv6 payload length fields. Thus, deduce it
>      from the length of the packet.
> 
>      Fixes: 696573046e9e ("net/virtio: support TSO")
> 
>      Signed-off-by: Boleslav Stankevich <boleslav.stankevich@oktetlabs.ru>
>      Reviewed-by: Andrew Rybchenko <andrew.rybchenko@oktetlabs.ru>
>      Reviewed-by: Maxime Coquelin <maxime.coquelin@redhat.com>
> 
> Intel validation owner has done lots of verification and compared these performance data on 8280&8380 platforms.
> This performance big drop is only found on special platform (Intel(R) Xeon(R) Platinum 8280M CPU @ 2.70GHz CPU) when vhost-user(back-end)
> and virtio-pmd(front-end) are not on the same socket (front-end is on the socket1, back-end and nic are on the socket0).
> But our test case doesn't call this bad commit id's related code, suspect whether it may be related to compiler optimization on special platform.
> We hope the bad commit id owner or other experts can support to analysis the root cause.
> 

ok, for this one, it is a performance drop on a single test on a 
specific Intel platform and inter-socket related. It does not appear to 
be related to the commit being ID'd. To revert the commit would 
re-introduce a functional issue.

> 2, Virtio issue -> should be virtio crypto test failure.
> It is not a regression issue, old lts and main branch also reproduce by validation owner, and there's no clear bad commit id found.
> Intel dev and validation owner are investigating it, but we don't find out the root cause yet.
> 

ok, this is not a regression in 21.11.4.

On that basis, I will go ahead with the 21.11.4 release tomorrow. If 
there's any objection please let me know by end of day today.

thanks,
Kevin.

> Best regards,
> Yu Jiang
> 
>>> Regards,
>>> Xu, Hailin
>>>
>>>> thanks,
>>>> Kevin.
>>>>
>>>>> Regards,
>>>>> Xu, Hailin
>>>>>
>>>>>>
>>>>>> thanks,
>>>>>> Kevin,
>>>>>>
>>>>>>> Kevin.
>>>>>>>
>>>>>>>> # Basic Intel(R) NIC testing
>>>>>>>> * Build & CFLAG compile: cover the build test combination with
>>>>>>>> latest
>>>>>> GCC/Clang version and the popular OS revision such as
>>>>>>>>        Ubuntu20.04, Ubuntu22.04, Fedora35, Fedora37, RHEL8.6,
>>>>>>>> RHEL8.4,
>>>>>> FreeBSD13.1, SUSE15, CentOS7.9, etc.
>>>>>>>> - All test done. No new dpdk issue is found.
>>>>>>>> * PF(i40e, ixgbe): test scenarios including
>>>>>> RTE_FLOW/TSO/Jumboframe/checksum offload/VLAN/VXLAN, etc.
>>>>>>>> - All test done. No new dpdk issue is found.
>>>>>>>> * VF(i40e, ixgbe): test scenarios including
>>>>>> VF-RTE_FLOW/TSO/Jumboframe/checksum offload/VLAN/VXLAN, etc.
>>>>>>>> - All test done. No new dpdk issue is found.
>>>>>>>> * PF/VF(ice): test scenarios including Switch features/Package
>>>>>> Management/Flow Director/Advanced Tx/Advanced
>> RSS/ACL/DCF/Flexible
>>>>>> Descriptor, etc.
>>>>>>>> - All test done. No new dpdk issue is found.
>>>>>>>> * Intel NIC single core/NIC performance: test scenarios including
>>>>>>>> PF/VF
>>>>>> single core performance test, etc.
>>>>>>>> - All test done. No new dpdk issue is found.
>>>>>>>> * IPsec: test scenarios including ipsec/ipsec-gw/ipsec library
>>>>>>>> basic test -
>>>>>> QAT&SW/FIB library, etc.
>>>>>>>> - All test done. No new dpdk issue is found.
>>>>>>>>
>>>>>>>> # Basic cryptodev and virtio testing
>>>>>>>> * Virtio: both function and performance test are covered. Such as
>>>>>> PVP/Virtio_loopback/virtio-user loopback/virtio-net VM2VM perf
>>>>>> testing/VMAWARE ESXI 8.0, etc.
>>>>>>>> - All test done. found bug1.
>>>>>>>> * Cryptodev:
>>>>>>>>        *Function test: test scenarios including Cryptodev API
>>>>>> testing/CompressDev ISA-L/QAT/ZLIB PMD Testing/FIPS, etc.
>>>>>>>>          - All test done. found bug2.
>>>>>>>>        *Performance test: test scenarios including Thoughput
>>>>>> Performance/Cryptodev Latency, etc.
>>>>>>>>          - All test done. No new dpdk issue is found.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Xu, Hailin
>>>>>>>>
>>>>>>>
>>>>>
>>>
>