From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 12AB4A04B7; Wed, 16 Sep 2020 16:20:11 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 0755A1C2AC; Wed, 16 Sep 2020 16:20:10 +0200 (CEST) Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [63.128.21.124]) by dpdk.org (Postfix) with ESMTP id D6F651C2A7 for ; Wed, 16 Sep 2020 16:20:07 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1600266007; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=Eq2wApvr6LJkWWNtmYGqYanTtM7NgB6ziWz6f+bW9II=; b=XMhcCTMkK7HhUVNSuX6z8V+G9jm2m/urB625xT5z+mskEM2unKaKpeud/TNDWU5PkRAE/o 9m4sg1b0tjKCirqel7hhbfbI50a2xOn5MrZ9EmRP1+Pcz9LR/zJmL5inX0PWpnc0gZD6k2 JFbk7epE6TjmXaVXlOcm+yl83dp55bM= Received: from mimecast-mx01.redhat.com (mimecast-mx01.redhat.com [209.132.183.4]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-388-P3nB-A9HNH23-iRduJdnMw-1; Wed, 16 Sep 2020 10:19:57 -0400 X-MC-Unique: P3nB-A9HNH23-iRduJdnMw-1 Received: from smtp.corp.redhat.com (int-mx04.intmail.prod.int.phx2.redhat.com [10.5.11.14]) (using TLSv1.2 with cipher AECDH-AES256-SHA (256/256 bits)) (No client certificate requested) by mimecast-mx01.redhat.com (Postfix) with ESMTPS id 5A69118B9F4F; Wed, 16 Sep 2020 14:18:22 +0000 (UTC) Received: from [10.33.36.243] (unknown [10.33.36.243]) by smtp.corp.redhat.com (Postfix) with ESMTP id 7DD1C5DE8B; Wed, 16 Sep 2020 14:18:20 +0000 (UTC) To: "Dey, Souvik" , "dev@dpdk.org" Cc: Stephen Hemminger , "longli@microsoft.com" , "Yigit, Ferruh" , "sthemmin@microsoft.com" References: From: Kevin Traynor Autocrypt: addr=ktraynor@redhat.com; keydata= mQINBF2J2awBEADUEPNhgNI+nJNgiTAUcw4YIgVXEoHlsNPyyzG1BEXkWXALy0Y3fNTiw6+r ltWDkF9jzL9kfkecgQ67itGfk1OaBXgSGKuw1PUpxAwX2Bi76LAR6M5OsyGM9TSVVQwARalz hMwRBIZPzPc7or6Pw7jAOJ8SQGJ1Zlp1YJCjrvpe87V1tH/LY8Wnxn/EuoseFmWILAQZAtYS tGjcrAgYn3SPMLR1B0BP5bTBY06vWQjiufH8drenfDnMJAzuBdG1mqjnTqCjULZ3Hunv4xqZ aMnkvL/K5Tj1c12Oe4930EE53LrXIBUltRg5mBudSWHnC7twjH0082HH9f963Z/2UI63SFIT iUvRvAzJYytgy7XnWLQ0+goZBADKYfolOuC0H8VgCaux8u8KFF28Dy+N6TV2KI58jTlyg1Zu l7QwykZpnOkJFiy37Gfbu3YEOzO72cP/S7/A+zvuqkxi63jyEkd+FY99vLt/HN2MUZwRmKDw UPbLkmrs8WU01/POVsqDcfvz7vu2St8hqqTiSIdQGS2zyTKB2/DvPSM3jws3udkIYSuhn+X4 QBiV6lkVZ7DSE6a065gnAauAql+b32Eymy+xnG5jCt1tR+0Cp2VZYCR9OU2gmomUKBDoX/He pSgED01CqYPNjN+TddirwmQX7ep4DtXc8FWvv2g/pq9WZFQk2QARAQABtCNLZXZpbiBUcmF5 bm9yIDxrdHJheW5vckByZWRoYXQuY29tPokCTgQTAQgAOBYhBAoiOaH51tHF7VYtEI9CINER a+yJBQJdidmsAhsDBQsJCAcCBhUKCQgLAgQWAgMBAh4BAheAAAoJEI9CINERa+yJoxIP/3VF 2TIgW4ckxhRFCvFu/606bnvCPie88ake4uWVWMAWwcMc4fKEltRWRCpkSVOwgqoMHnyHxK5r kOKzx2CLJMX5TgTMfKzPuaBDHngHLUzl2DStpBzrod0cVg5TShdmmfjY61uxRJKz+DlSkwgJ riADdVF5PPosQXTkKSGf2ombpTGpx/pue9ocjnr3x4SDpRLlnooM6Jf/3Y3Ib4jX6HPEyWuY b+owIIk9y2nRRGPQ6jbqAhsrXd9V+77UL0QuGWloMuKMZFbNg8hbu7X5aFijAbfxj4YUgojS ba7gfGZQan8h32A9KGQWrmsCBc3j2GqEPsX0r05X7cn7WL6IOPgQJ5EiQ7PlazQYVLrvZg9B n0GKK0k6895mLG0ZZ5v/qajOPF52etSmvFD1WUPb4OqaHqGA9ZtMpaKFRt7Y6rpXqKNU1xzW F5KjbTPtTb9WF3An8dciVv+AYUI7totkZYkWvQtgss8lfaX3NKUvXLVxqK0z3dQyr7rF/tYz PneTKypSksjCgaEBLSrsRmM5zKfe7tSNF/fDntfIq/029Jtcw29TcWEP57peNu6TtejewQD9 sTI+oqiXvW2D5l7LNUDYG8eMJp2oT7I0ZSBRvwcbmjH0DtN/bXCCFfCvk8Yic68F3tV1ctix wQARVKDBhT30uCxycRWojCYqTgNJJS71uQINBF2J2awBEADP57PR2IpSYBeNSrsAjeIcsahE N4SQP2C4s50S8QEWAUhqMRI7WNv5cfeef0nDvcl1IUA6oz5SokbcsbMa+mRgaNF4N5KikWTO LPYxq2YVJoXwJ+tKmNzyOLFUIfFJ4NBJZple5dTfWzD00Dbb19Mri1hy1mWMqNTPGBee1+hw Qcp6n3mmGECvajs8G5A7NyXbwL8ihN7HX9D01ucD62b4G03yKe2g/hvKgcdUVmhCldJlF27I 2fSR9tDxH9pZqRODY4rjbFZEey/vWKXqjE+DQ8AtMSEaDfFe5D+i4Aw6erWQ3Wr+DwZt1/7G dIAElGA/q90T1ENVwJX9y7fsQssawKYYdDqURHCl5JuDXI+VXUypExipUUT5SPycMmbLsx0D iKEqPPDQWKxkIDVKqj2+EhamSuJznZUwBLJKn0h4zrIWiXWUy07lRwtVuhaDXhF3GfW+5W/x wAg7Qg3w00ASsb/XTHBIhMnenKDfS7ihtQA8SacwX8ySdxb+15XPyiplM979qBQ0mhnilulm MIJzEf/JxoYR5huuj4f1PFqqrsP06Dl+YGB7dQZp3IKggS5c3/TAynARRg9N89UsDXNtp7X0 tgIPFF5k6fnHE0J5O64GYHeTqN/1aE6dAEOV9WrGzQAJxU9ipikb8jKAWXzLewRIKGmoPcRZ WdB0NmIjmQARAQABiQI2BBgBCAAgFiEECiI5ofnW0cXtVi0Qj0Ig0RFr7IkFAl2J2awCGwwA CgkQj0Ig0RFr7IkkORAAl/NbX93WK5MEoRw7/DaPTo/Lo6Pj1XMeSqGyACigHK/452UDvlEH NjNJMzYYrNIjMtEmN9VVCfjT38CSca7mpGQVwchc0mC7QSPAETLCS+UacVf/Kwxz5FfkEUUw UT7A+uyVOIgW3d9ldlRzkHA2czonSSgTQU+i2g6DM4ha+BuQb4byAXH6HQHt/Zh1J64z0ohH v6iGsCzCY/sMWF8+LEGSnzMGRCLiiwSF0vJBHbzWK68fANaF4gBV0Z/+6tQRFN7YMhj/INmk qgvHj1ZzHFNtirjMGPRxoZs51YoLQM/aBPxKrnmXThx1ufH+0L6sGmFTugiDt0XSEkC5reH7 a+VhQ1VTFFQrClA8NmDSPzFeuhru4ryaaDHO+uEB16cNHxHrQtlP/2hts2JM5lwkZRWJ5A57 h8eDEIK5be47T85NVHfuTaboNRmgg1HygVejhGUtt69u/0MVRg/roUTa0FyEbNsvz4qAecyW yWzMcVrcGJDQLC9JLKEpoyUF6gdTKaiDL2Vao4+XRIA3Y57b6MO35a3HuzAv7+i5Z0mnDEJO XxXqTOmKYpMIGexzM/PtuA0712sT1abG9tAJ17ao/B7cqMW5IkKkalemFbWfI2unns4Papvo tk9igVqyp6EJDU98z5TJioCVojwK2laDaoIjTJk9YYv3iwCsqPd5feU= Message-ID: <82871b97-8145-d3e4-a87f-28aa61cc8e61@redhat.com> Date: Wed, 16 Sep 2020 15:18:19 +0100 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:68.0) Gecko/20100101 Thunderbird/68.11.0 MIME-Version: 1.0 In-Reply-To: X-Scanned-By: MIMEDefang 2.79 on 10.5.11.14 Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=ktraynor@redhat.com X-Mimecast-Spam-Score: 0.002 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset=windows-1252 Content-Transfer-Encoding: 8bit Content-Language: en-US Subject: Re: [dpdk-dev] Issue with net/netvsc pmd in 18.11.9 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" On 16/09/2020 13:00, Dey, Souvik wrote: > Hi All, > I updated from dpdk 18.11.6 to 18.11.9 and my netvsc pmd stopped working as expected. Firstly, I saw a crash in tx packets as soon as the dpdk app comes up. In doing some googling found a bug that was fixed after 18.11.9 https://patches.dpdk.org/patch/75001/ . Ported this fix and was able to get rid of the seg fault, but now stuck at another issue. When we are transmitting packets of size within HN_TXCOPY_THRESHOLD we Hi, The patch mentioned is merged to the 18.11 branch in dpdk-stable repo and part of 18.11.10-rc1, so expected to be in 18.11.10. ETA 28th September. In case it's already fixed, you can try the 18.11.10-rc1 release. These netvsc backports were added since 18.11.9. The corresponding dpdk main commit id's are in the full log. $ git log --oneline v18.11.9..HEAD . 363bdf0f49 net/netvsc: fix chimney index 5b3a327709 net/netvsc: fix crash during Tx 77cf88c0c6 net/netvsc: fix underflow when Rx external mbuf 59257aaaa1 net/netvsc: do not spin forever waiting for reply The list of fixes that were tagged for stable but not backported is here: http://inbox.dpdk.org/stable/20200901100115.72365-1-ktraynor@redhat.com/ are all good but any larger packets/fragmented packets are getting dropped after some time. As soon as we start to receive the transmit completion event NVS_TYPE_RNDIS_ACK as failed, packets with size greater than HN_TXCOPY_THRESHOLD starts to drop and never recovers. Packets less than HN_TXCOPY_THRESHOLD works properly there after though. Any idea why this is happening and is there is some fix which is already there which is done after 18.11.9 release ? Maybe you can bisect the 18.11 branch to find where it stops working, there is only ~10 netvsc commits between 18.11.6 and 18.11.9. The release notes [1] or git history will tell you which commits are in which releases. Bear in mind that some commits were rebased for 18.11, so if you find the offending commit, you might want to check that it was correctly backported too. Kevin. [1] http://doc.dpdk.org/guides-18.11/rel_notes/release_18_11.html > We are at the critical part of our release, so any help in this regard will be highly appreciated. Thanks in advance for all the help. > > PS: I also tried to put the net/netvsc: return the correct chimney index fix in but nothing changed. > > -- > Regards, > Souvik > > Log Snippet: > I can see the below errors coming after some transmitting. > hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0 > hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0 > hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0 > hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0 > hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0 > hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0 > hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0 > hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0 > hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0 > hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0 > hn_flush_txagg() tx: port 2:0 tx 2048 size 302 > hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514 > hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514 > hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514 > hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514 > hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514 > hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514 > hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514 > hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514 > hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514 > hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514 > hn_nvs_send_completed() tx: port 2:0 complete tx 2048 packets 1 bytes 242 > > hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0 > hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0 > hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0 > hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0 > hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0 > hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0 > hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0 > hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0 > hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0 > hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0 > hn_flush_txagg() tx: port 2:0 tx 2624 size 302 > hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2 > hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2 > hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2 > hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2 > hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514 > hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514 > hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514 > hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514 > hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514 > hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 packets 1 bytes 1514 > hn_nvs_send_completed() tx: port 2:0 complete tx 2624 packets 1 bytes 242 > > hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0 > hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0 > hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0 > hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0 > hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0 > hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0 > hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0 > hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0 > hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 3 size 0 > hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0 > hn_flush_txagg() tx: port 2:0 tx 2688 size 302 > hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2 > hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2 > hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2 > hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2 > hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2 > hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2 > hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2 > hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2 > hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2 > hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2 > hn_nvs_send_completed() tx: port 2:0 complete tx 2688 packets 1 bytes 242 > hn_flush_txagg() tx: port 2:0 tx 3264 size 571 > hn_nvs_send_completed() tx: port 2:0 complete tx 3264 packets 1 bytes 511 > hn_rxpkt() rx: port 2:0 RX id 3 size 511 type 0x11 ol_flags 0x2 > hn_flush_txagg() tx: port 2:0 tx 3328 size 571 > hn_nvs_send_completed() tx: port 2:0 complete tx 3328 packets 1 bytes 511 > hn_rxpkt() rx: port 2:0 RX id 3 size 512 type 0x11 ol_flags 0x2 > hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0 > hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2 > hn_rxpkt() rx: port 2:0 RX id 3 size 512 type 0x11 ol_flags 0x2 > hn_xmit_sg() tx: port 2:0 tx 4294967295 segs 2 size 0 > hn_nvs_send_completed() tx: port 2:0 complete tx 4294967295 failed status 2 > hn_flush_txagg() tx: port 2:0 tx 3392 size 120 > hn_nvs_send_completed() tx: port 2:0 complete tx 3392 packets 1 bytes 60 > hn_rxpkt() rx: port 2:0 RX id 3 size 42 type 0x1 ol_flags 0 >