Re: 21.11.4 patches review and test

[Kevin Traynor <ktraynor@redhat.com>]

On 05/05/2023 02:42, Xu, HailinX wrote:
>> -----Original Message-----
>> From: Kevin Traynor <ktraynor@redhat.com>
>> Sent: Thursday, May 4, 2023 6:11 PM
>> To: Xu, HailinX <hailinx.xu@intel.com>; stable@dpdk.org
>> Cc: Stokes, Ian <ian.stokes@intel.com>; Mcnamara, John
>> <john.mcnamara@intel.com>; Luca Boccassi <bluca@debian.org>; Xu, Qian Q
>> <qian.q.xu@intel.com>; Thomas Monjalon <thomas@monjalon.net>; Peng,
>> Yuan <yuan.peng@intel.com>; Chen, Zhaoyan <zhaoyan.chen@intel.com>;
>> dev@dpdk.org
>> Subject: Re: 21.11.4 patches review and test
>>
>> On 04/05/2023 03:13, Xu, HailinX wrote:
>>>> -----Original Message-----
>>>> From: Kevin Traynor <ktraynor@redhat.com>
>>>> Sent: Tuesday, May 2, 2023 5:35 PM
>>>> To: Xu, HailinX <hailinx.xu@intel.com>; stable@dpdk.org
>>>> Cc: Stokes, Ian <ian.stokes@intel.com>; Mcnamara, John
>>>> <john.mcnamara@intel.com>; Luca Boccassi <bluca@debian.org>; Xu, Qian
>>>> Q <qian.q.xu@intel.com>; Thomas Monjalon <thomas@monjalon.net>;
>> Peng,
>>>> Yuan <yuan.peng@intel.com>; Chen, Zhaoyan <zhaoyan.chen@intel.com>;
>>>> dev@dpdk.org
>>>> Subject: Re: 21.11.4 patches review and test
>>>>
>>>> On 20/04/2023 11:32, Kevin Traynor wrote:
>>>>> On 20/04/2023 03:40, Xu, HailinX wrote:
>>>>>>> -----Original Message-----
>>>>>>> From: Xu, HailinX <hailinx.xu@intel.com>
>>>>>>> Sent: Thursday, April 13, 2023 2:13 PM
>>>>>>> To: Kevin Traynor <ktraynor@redhat.com>; stable@dpdk.org
>>>>>>> Cc: dev@dpdk.org; Abhishek Marathe
>>>> <Abhishek.Marathe@microsoft.com>;
>>>>>>> Ali Alnubani <alialnu@nvidia.com>; Walker, Benjamin
>>>>>>> <benjamin.walker@intel.com>; David Christensen
>>>>>>> <drc@linux.vnet.ibm.com>; Hemant Agrawal
>>>> <hemant.agrawal@nxp.com>;
>>>>>>> Stokes, Ian <ian.stokes@intel.com>; Jerin Jacob
>>>>>>> <jerinj@marvell.com>; Mcnamara, John <john.mcnamara@intel.com>;
>>>>>>> Ju-Hyoung Lee <juhlee@microsoft.com>; Luca Boccassi
>>>>>>> <bluca@debian.org>; Pei Zhang <pezhang@redhat.com>; Xu, Qian Q
>>>>>>> <qian.q.xu@intel.com>; Raslan Darawsheh <rasland@nvidia.com>;
>>>> Thomas
>>>>>>> Monjalon <thomas@monjalon.net>; yanghliu@redhat.com; Peng, Yuan
>>>>>>> <yuan.peng@intel.com>; Chen, Zhaoyan <zhaoyan.chen@intel.com>
>>>>>>> Subject: RE: 21.11.4 patches review and test
>>>>>>>
>>>>>>>> -----Original Message-----
>>>>>>>> From: Kevin Traynor <ktraynor@redhat.com>
>>>>>>>> Sent: Thursday, April 6, 2023 7:38 PM
>>>>>>>> To: stable@dpdk.org
>>>>>>>> Cc: dev@dpdk.org; Abhishek Marathe
>>>>>>>> <Abhishek.Marathe@microsoft.com>; Ali Alnubani
>>>>>>>> <alialnu@nvidia.com>; Walker, Benjamin
>>>>>>>> <benjamin.walker@intel.com>; David Christensen
>>>>>>>> <drc@linux.vnet.ibm.com>; Hemant Agrawal
>>>>>>>> <hemant.agrawal@nxp.com>; Stokes, Ian <ian.stokes@intel.com>;
>>>>>>>> Jerin Jacob <jerinj@marvell.com>; Mcnamara, John
>>>>>>>> <john.mcnamara@intel.com>; Ju-Hyoung Lee
>> <juhlee@microsoft.com>;
>>>>>>>> Kevin Traynor <ktraynor@redhat.com>; Luca Boccassi
>>>>>>>> <bluca@debian.org>; Pei Zhang <pezhang@redhat.com>; Xu, Qian Q
>>>>>>>> <qian.q.xu@intel.com>; Raslan Darawsheh <rasland@nvidia.com>;
>>>>>>> Thomas
>>>>>>>> Monjalon <thomas@monjalon.net>; yanghliu@redhat.com; Peng, Yuan
>>>>>>>> <yuan.peng@intel.com>; Chen, Zhaoyan <zhaoyan.chen@intel.com>
>>>>>>>> Subject: 21.11.4 patches review and test
>>>>>>>>
>>>>>>>> Hi all,
>>>>>>>>
>>>>>>>> Here is a list of patches targeted for stable release 21.11.4.
>>>>>>>>
>>>>>>>> The planned date for the final release is 25th April.
>>>>>>>>
>>>>>>>> Please help with testing and validation of your use cases and
>>>>>>>> report any issues/results with reply-all to this mail. For the
>>>>>>>> final release the fixes and reported validations will be added to
>>>>>>>> the
>>>> release notes.
>>>>>>>>
>>>>>>>> A release candidate tarball can be found at:
>>>>>>>>
>>>>>>>>         https://dpdk.org/browse/dpdk-stable/tag/?id=v21.11.4-rc1
>>>>>>>>
>>>>>>>> These patches are located at branch 21.11 of dpdk-stable repo:
>>>>>>>>         https://dpdk.org/browse/dpdk-stable/
>>>>>>>>
>>>>>>>> Thanks.
>>>>>>>>
>>>>>>>> Kevin
>>>>>>>
>>>>>>> HI All,
>>>>>>>
>>>>>>> Update the test status for Intel part. Till now dpdk21.11.4-rc1
>>>>>>> validation test rate is 85%. No critical issue is found.
>>>>>>> 2 new bugs are found, 1 new issue is under confirming by Intel Dev.
>>>>>>> New bugs:   --20.11.8-rc1 also has these two issues
>>>>>>>       1.
>>>> pvp_qemu_multi_paths_port_restart:perf_pvp_qemu_vector_rx_mac:
>>>>>>> performance drop about 23.5% when send small packets
>>>>>>> 	https://bugs.dpdk.org/show_bug.cgi?id=1212    -- no fix yet
>>>>>>>       2. some of the virtio tests are failing:    -- Intel dev is under
>>>> investigating
>>>>>>> # Basic Intel(R) NIC testing
>>>>>>> * Build & CFLAG compile: cover the build test combination with
>>>>>>> latest GCC/Clang version and the popular OS revision such as
>>>>>>>       Ubuntu20.04, Ubuntu22.04, Fedora35, Fedora37, RHEL8.6,
>>>>>>> RHEL8.4, FreeBSD13.1, SUSE15, CentOS7.9, etc.
>>>>>>> - All test done. No new dpdk issue is found.
>>>>>>> * PF(i40e, ixgbe): test scenarios including
>>>>>>> RTE_FLOW/TSO/Jumboframe/checksum offload/VLAN/VXLAN, etc.
>>>>>>> - All test done. No new dpdk issue is found.
>>>>>>> * VF(i40e, ixgbe): test scenarios including
>>>>>>> VF-RTE_FLOW/TSO/Jumboframe/checksum offload/VLAN/VXLAN, etc.
>>>>>>> - All test done. No new dpdk issue is found.
>>>>>>> * PF/VF(ice): test scenarios including Switch features/Package
>>>>>>> Management/Flow Director/Advanced Tx/Advanced
>> RSS/ACL/DCF/Flexible
>>>>>>> Descriptor, etc.
>>>>>>> - All test done. No new dpdk issue is found.
>>>>>>> * Intel NIC single core/NIC performance: test scenarios including
>>>>>>> PF/VF single core performance test, etc.
>>>>>>> - All test done. No new dpdk issue is found.
>>>>>>> * IPsec: test scenarios including ipsec/ipsec-gw/ipsec library
>>>>>>> basic test - QAT&SW/FIB library, etc.
>>>>>>> - On going.
>>>>>>>
>>>>>>> # Basic cryptodev and virtio testing
>>>>>>> * Virtio: both function and performance test are covered. Such as
>>>>>>> PVP/Virtio_loopback/virtio-user loopback/virtio-net VM2VM perf
>>>>>>> testing/VMAWARE ESXI 8.0, etc.
>>>>>>> - All test done. found bug1.
>>>>>>> * Cryptodev:
>>>>>>>       *Function test: test scenarios including Cryptodev API
>>>>>>> testing/CompressDev ISA-L/QAT/ZLIB PMD Testing/FIPS, etc.
>>>>>>>         - Execution rate is 90%. found bug2.
>>>>>>>       *Performance test: test scenarios including Thoughput
>>>>>>> Performance/Cryptodev Latency, etc.
>>>>>>>         - All test done. No new dpdk issue is found.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Xu, Hailin
>>>>>> Update the test status for Intel part. completed dpdk21.11.4-rc1
>>>>>> all
>>>> validation. No critical issue is found.
>>>>>
>>>>> Hi. Thanks for testing.
>>>>>
>>>>>> 2 new bugs are found, 1 new issue is under confirming by Intel Dev.
>>>>>> New bugs: --20.11.8-rc1 also has these two issues
>>>>>>       1.
>>>> pvp_qemu_multi_paths_port_restart:perf_pvp_qemu_vector_rx_mac:
>>>> performance drop about 23.5% when send small packets
>>>>>> 	https://bugs.dpdk.org/show_bug.cgi?id=1212  --not fix yet, Only
>>>>>> the specified platform exists
>>>>>
>>>>> Do you know which patch caaused the regression? I'm not fully clear
>>>>> from the Bz for 20.11. The backported patch ID'd as root cause [0]
>>>>> in
>>>>> 20.11 is in the previous releases of 20.11 (and 21.11).
>>>>>
>>>>> Trying to understand because then it would have shown in testing for
>>>>> previous releases. Or is this a new test introduced for latest LTS
>>>>> releases? and if so, what is the baseline performance based on?
>>>>>
>>>>> [0]
>>>>> commit 1c9a7fba5c90e0422b517404499ed106f647bcff
>>>>> Author: Mattias Rönnblom <mattias.ronnblom@ericsson.com>
>>>>> Date:   Mon Jul 11 14:11:32 2022 +0200
>>>>>
>>>>>         net: accept unaligned data in checksum routines
>>>>>
>>>>>>       2. some of the virtio tests are failing: -- Intel dev is under
>>>>>> investigating
>>>>>
>>>>> ok, thank you.
>>>>>
>>>>
>>>> Hi, it was mentioned in a separate mail that the performance drop
>>>> issue was not the same as 20.11.
>>>>
>>>> Is there any update on that and the failing virtio tests ? Is there a
>>>> regression introduced since in 21.11.4 ?
>>>
>>> Found bad commit id:
>>> commit c41493361c87e730459ead9311c68528eb0874aa
>>> Author: Boleslav Stankevich <boleslav.stankevich@oktetlabs.ru>
>>> Date:   Fri Mar 3 14:19:29 2023 +0300
>>>       net/virtio: deduce IP length for TSO checksum
>>>
>>> We try this issue on 3 different platforms Performance drop only found
>>> on Intel(R) Xeon(R) Platinum 8280M CPU @ 2.70GHz CPU.
>>>
>>
>> Ok, thanks for reporting. This commit was also in 23.03 and 20.11.8 so not sure
>> if they were tested on this platform?
>>
> Yes, But 23.03 doesn't have such issue on the same platform
> 

ok, thanks.

>> This commit fixes a functional issue, so I don't think it should be removed unless
>> it is critical issue.
>>
>> There's no update on other performance report. Release is already 1 week
>> overdue, is there critical issues that we should hold release for?
>>
> No other issues found
> 

I'm not fully clear on the status, so let's separate. There were two 
reported issues.

1. Performance drop on specific Intel platform.
- Is this being investigated by Intel dev?
- Is it is a release blocking issue ?

2. Virtio issue caused by

commit c41493361c87e730459ead9311c68528eb0874aa
Author: Boleslav Stankevich <boleslav.stankevich@oktetlabs.ru>
Date:   Fri Mar 3 14:19:29 2023 +0300
        net/virtio: deduce IP length for TSO checksum

- Is this being investigated by Intel dev?
- Is it is a release blocking issue ?

thanks,
Kevin.

> Regards,
> Xu, Hailin
> 
>> thanks,
>> Kevin.
>>
>>> Regards,
>>> Xu, Hailin
>>>
>>>>
>>>> thanks,
>>>> Kevin,
>>>>
>>>>> Kevin.
>>>>>
>>>>>> # Basic Intel(R) NIC testing
>>>>>> * Build & CFLAG compile: cover the build test combination with
>>>>>> latest
>>>> GCC/Clang version and the popular OS revision such as
>>>>>>       Ubuntu20.04, Ubuntu22.04, Fedora35, Fedora37, RHEL8.6,
>>>>>> RHEL8.4,
>>>> FreeBSD13.1, SUSE15, CentOS7.9, etc.
>>>>>> - All test done. No new dpdk issue is found.
>>>>>> * PF(i40e, ixgbe): test scenarios including
>>>> RTE_FLOW/TSO/Jumboframe/checksum offload/VLAN/VXLAN, etc.
>>>>>> - All test done. No new dpdk issue is found.
>>>>>> * VF(i40e, ixgbe): test scenarios including
>>>> VF-RTE_FLOW/TSO/Jumboframe/checksum offload/VLAN/VXLAN, etc.
>>>>>> - All test done. No new dpdk issue is found.
>>>>>> * PF/VF(ice): test scenarios including Switch features/Package
>>>> Management/Flow Director/Advanced Tx/Advanced RSS/ACL/DCF/Flexible
>>>> Descriptor, etc.
>>>>>> - All test done. No new dpdk issue is found.
>>>>>> * Intel NIC single core/NIC performance: test scenarios including
>>>>>> PF/VF
>>>> single core performance test, etc.
>>>>>> - All test done. No new dpdk issue is found.
>>>>>> * IPsec: test scenarios including ipsec/ipsec-gw/ipsec library
>>>>>> basic test -
>>>> QAT&SW/FIB library, etc.
>>>>>> - All test done. No new dpdk issue is found.
>>>>>>
>>>>>> # Basic cryptodev and virtio testing
>>>>>> * Virtio: both function and performance test are covered. Such as
>>>> PVP/Virtio_loopback/virtio-user loopback/virtio-net VM2VM perf
>>>> testing/VMAWARE ESXI 8.0, etc.
>>>>>> - All test done. found bug1.
>>>>>> * Cryptodev:
>>>>>>       *Function test: test scenarios including Cryptodev API
>>>> testing/CompressDev ISA-L/QAT/ZLIB PMD Testing/FIPS, etc.
>>>>>>         - All test done. found bug2.
>>>>>>       *Performance test: test scenarios including Thoughput
>>>> Performance/Cryptodev Latency, etc.
>>>>>>         - All test done. No new dpdk issue is found.
>>>>>>
>>>>>> Regards,
>>>>>> Xu, Hailin
>>>>>>
>>>>>
>>>
>