From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 5A8A5A04DD for ; Thu, 22 Oct 2020 15:12:17 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 06A68A9C0; Thu, 22 Oct 2020 15:12:16 +0200 (CEST) Received: from sonic303-2.consmr.mail.bf2.yahoo.com (sonic303-2.consmr.mail.bf2.yahoo.com [74.6.131.41]) by dpdk.org (Postfix) with ESMTP id 10CC6A95E for ; Thu, 22 Oct 2020 15:12:13 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1603372332; bh=u/lR37CWhxDqwpyH2a2Vacu0wEKm4uyJbn10WgB5UqQ=; h=Date:From:To:Subject:References:From:Subject; b=qUOj7Q3G4vKMT0LZ3u/e/ueH+ltttppvmpcOndrcxLyLtfzTYbc1aR1ctjJ3UPguMDXVbj+KfnB7G3f75/I1Tn5/nIPROj/3r8Z3FzKYF6TWEwpnA4n08YxXVJ8VWYkJQuzLrJPJexu+5ID81bcXOnzgxCb/Gt8R/DX2fIVjhzouj4Kbtsh+j5MjV2IDwJpYxh3rLD6jPiELkdF41pjV69bRFei8PbcoeBkL2tYpZxzTw8vbwApS88o0/ZzhKnyfz5jVNClrDHJc1cTh1RG/AxGkOdXJ8MQeMCjNrJVqi5iiT/LUsbwiZnYAUth4M+N90fdj+6k5WYvRAuZvmFc8Hg== X-SONIC-DKIM-SIGN: v=1; a=rsa-sha256; c=relaxed/relaxed; d=yahoo.com; s=s2048; t=1603372332; bh=sssY0bVMkMc6+m7jqPRIMnxsNHgC7HI7UcMcz9Q/Oap=; h=Date:From:To:Subject; b=hIHu1yDq4bsAKfwzklN1MzIyWHOlWJNmKMqOigKFFPDGcV07qPeCjpujNpPq8OPNHJJdftPPiQ1u9ogSDqTwxEKjLwdqUgiEqnAd+ipVlqeUI8hDkeboYh+bnq7+gyqIpls0LOjRyw709NRKKM+i6on7OmIW6EFjgictpb5Pl62K5M8jj3DjLZ0rXc/myt1FTPFYtwS8prPcHn+fq/uI/MM5nBGaTnMuWPsJbOt6AEuTpWoM3W0y1vSvpbTWeZrzZcF6FUwUnEcf7XBfja+3J4lx5Yi8+2VAyAUTUN2wrUOEwmTX4mgLtZmHKjnXGXiqwtgPIcBVR0uDgR3POq8jIw== X-YMail-OSG: fia6ojYVM1kdlNayxjABKDFa6JaXY_wah1YGgT14AnSw6j6E8RRHNZjOOdIeHzi bjZ.zLFYgnnRGsjlcMnKsuGpBik9ExpDXJ7ja2ub3JU8XPp6QGQenmefg4iXhhn0ZhA7jgvs.rP9 yTx4ZY7kFqaw.xKPCGP8S57PQhoUXCrztP5zsZeL6X8eyKk06p_VcEk8YmgkzRoh.jhN_pZeYRwZ d1WCVefTbqXICTBffTcbm1AV76.1O6ZlD07fF5nH5Bz7wxOyXtM5Pmk.Ls1EbZE6OZZO53dl51kL 1D28IUd5BWNP080gJwEDJJJhGpHkJPje5gRiui_D.Jonsxra2rJizSNlOuYToei5ij.hZzdRyuLw faQmKUw3tOB9SEiLqf.Rr8b8FBRgBj.BteyZdhIPkjYlYg8E788W1Ez2FEBfYgaj8FNeFKWw3ngL cCsbdHKXYrRAW2wICYVP4jqdZ1wVVjBIBU3f6N5eAZ_T8krcFeSDpdsk0.Fdcc7o5OItT89iA6l. J3O6shmQOqxQQiiDlYKLzD_Gmer3j9tHfClBMqQdVh.ccg.HXJC4B0suENcv3wG6yGyRVnKxkrEA A0S36.e7Y.0W5LJF5gyUlvrWYS0RRYlzf0v_Gk773GZgWtdiqhWHvGxh1L6Q7t7no1yL3ENyo8Ie 7OlBE1mOa8cVALtMiWeca0V967VfyX4IpuRX_Hxz.dhGlqNzCkbXxNHHUdoUej.fKl8UoBbSmILr irEt7gGGk.pXp8qKpipMDAKu23DnPb24dTcSMlaSvTXBCHv.23Galcq4fU1RXci7H6A_MB5G26C4 2vlMiVYB.1KfF3kaeKgb77Hu7obe5JxGUA5HdtGIaSe0GNllM0AzzGDQtWxzqeL9ARyFakdYlymX 5U_0R7KvZ6IEpQl43EfZq6GY9y25rC5rOjdWDwJcKBLV1EEF6IV1DXhhhOjWxjN37HGmOIPVFoi. 11dqhdofZ_2.uhiv4ja9kGU5t2qMPQxwTsHiKVnKnpLYdBjo8aWWsmQgdWUVQWC97GMph94QST7s ZFngQa1vkA5fHZmBezFzITRlfUOoP22ZO_a8tVHbULjMNSnD41UKpK_XvnnfYa.FTIVnzddo6D5H 9Bo1PqX2ohJrUDDTxu30G4Ry9NIZlwfj0KO57ypcAkO23_1N7LnokVyf38ekghYAltBqM8my0SNa pwISPbUKHfskRShvjE_YHBZTNKp3N0UVCc..xXrX61fEkHdIXWGkcqVtiGHfMgURIAFG93QNLRBp Cvc4fz.T6bnYG1TJgIrLFlEqS04bddDeqYxA8oMcbWqtgAoCS8YO6coBRtb06c.bX.or9Pe06eXm ViqClVsSpsYmYSsiDBylcTW7Cl11UQT1w_YsljoRy6ZxxqnMYk3UE1yy1hP5L2QPdq5.5lq8JYnS wRQEPjV.f7gf74lVJKYmW3rl54A0ZofnyjnWpbZsRkyID57xF9c9_fDLsIxPJHQ2pkhJXfSs43_s .FV7KINlrwWEGRxU- Received: from sonic.gate.mail.ne1.yahoo.com by sonic303.consmr.mail.bf2.yahoo.com with HTTP; Thu, 22 Oct 2020 13:12:12 +0000 Date: Thu, 22 Oct 2020 13:12:11 +0000 (UTC) From: Mohamed Mohamed To: "users@dpdk.org" Message-ID: <789541096.1689721.1603372331229@mail.yahoo.com> MIME-Version: 1.0 References: <789541096.1689721.1603372331229.ref@mail.yahoo.com> X-Mailer: WebService/1.1.16868 YMailNorrin Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.121 Safari/537.36 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-users] Iv40evf performance degradation when upgrade DPDK to 19-11 X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" Hi All: =C2=A0we are in the process of upgrading DPDK to 19-11 while testing i40evf= driver with multiple queues we are seeing performance drop significantly= =C2=A0the driver log seems mostly clean=C2=A010-21-2020=C2=A0 15:53:26.4749= 06 i40evf_handle_aq_msg(): Failed to read msg from AdminQ,ret: -5710-21-202= 0=C2=A0 15:53:26.635553 i40evf_check_api_version(): Peer is Linux PF host10= -21-2020=C2=A0 15:53:26.645551 EAL: PCI device 0000:00:0d.0 on NUMA socket = -110-21-2020=C2=A0 15:53:26.645554 EAL:=C2=A0 =C2=A0Invalid NUMA socket, de= fault to 010-21-2020=C2=A0 15:53:26.645556 EAL:=C2=A0 =C2=A0probe driver: 8= 086:154c net_i40e_vf10-21-2020=C2=A0 15:53:26.695593 i40evf_handle_aq_msg()= : Failed to read msg from AdminQ,ret: -5710-21-2020=C2=A0 15:53:26.858821 i= 40evf_check_api_version(): Peer is Linux PF host10-21-2020=C2=A0 15:53:26.8= 68819 EAL: PCI device 0000:00:0e.0 on NUMA socket -110-21-2020=C2=A0 15:53:= 26.868821 EAL:=C2=A0 =C2=A0Invalid NUMA socket, default to 010-21-2020=C2= =A0 15:53:26.868823 EAL:=C2=A0 =C2=A0probe driver: 8086:154c net_i40e_vf10-= 21-2020=C2=A0 15:53:26.918849 i40evf_handle_aq_msg(): Failed to read msg fr= om AdminQ,ret: -5710-21-2020=C2=A0 15:53:27.81824 i40evf_check_api_version(= ): Peer is Linux PF host10-21-2020=C2=A0 15:53:27.124821 set_rxtx_funcs(): = virtio: using standard Tx path on port 010-21-2020=C2=A0 15:53:27.124853 se= t_rxtx_funcs(): virtio: using mergeable buffer Rx path on port 010-21-2020= =C2=A0 15:53:27.125810 set_rxtx_funcs(): virtio: using standard Tx path on = port 110-21-2020=C2=A0 15:53:27.125822 set_rxtx_funcs(): virtio: using merg= eable buffer Rx path on port 110-21-2020=C2=A0 15:53:27.126606 set_rxtx_fun= cs(): virtio: using standard Tx path on port 210-21-2020=C2=A0 15:53:27.126= 615 set_rxtx_funcs(): virtio: using mergeable buffer Rx path on port 210-21= -2020=C2=A0 15:53:27.141849 i40evf_handle_aq_msg(): Failed to read msg from= AdminQ,ret: -57 from DPDK stats I see=C2=A0=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0= =C2=A0rx_dropped_packets : 7423883 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0rx_missed_errors : 7= 505403 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 in one run I have seen more errors in the log but that happened for me once= not consistent=C2=A010-20-2020=C2=A0 13:49:59.217581 i40evf_stop_queues():= Fail to stop queue 010-20-2020=C2=A0 13:49:59.217582 i40evf_execute_vf_cmd= (): fail to send cmd 910-20-2020=C2=A0 13:49:59.217584 i40evf_switch_queue(= ): fail to switch RX 1 off10-20-2020=C2=A0 13:49:59.217585 i40evf_dev_rx_qu= eue_stop(): Failed to switch RX queue 1 off10-20-2020=C2=A0 13:49:59.217587= i40evf_stop_queues(): Fail to stop queue 110-20-2020=C2=A0 13:49:59.217588= i40evf_execute_vf_cmd(): fail to send cmd 910-20-2020=C2=A0 13:49:59.21759= 0 i40evf_switch_queue(): fail to switch RX 2 off10-20-2020=C2=A0 13:49:59.2= 17591 i40evf_dev_rx_queue_stop(): Failed to switch RX queue 2 off I can't build any theory based on the above one time instance Can you pls provide some guidance how we can debug this further and how ? Thanks,Mohamed