From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-io0-f182.google.com (mail-io0-f182.google.com [209.85.223.182]) by dpdk.org (Postfix) with ESMTP id 8C022379B for ; Tue, 31 May 2016 20:50:09 +0200 (CEST) Received: by mail-io0-f182.google.com with SMTP id t40so143069861ioi.0 for ; Tue, 31 May 2016 11:50:09 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc; bh=Bdqq4Jx77bgWLnNeAy3XwgjzMg4NAl323SZ2VmtxgMw=; b=G/MDbAD7lnjeO6zuDUVzXcQgCgVUBgDQ8S3q75jp5UdUWNiVh4ZQsYsvmT/0rSiXK3 4oxZiHB9Dm3k7ya9+zCalx5/Fxon+lscPgEQAXegY0UuyQ8sfh+WMNs/ufL9/IZ6jFSj 7+wQ/pPE1818VKXQJodilw5xTsCo32WQyg/876SntINj2u1rt8tAedw/eAGUZ0L3L+CW SNmZswwETDwnQvgwpGzZGMIhyaJMqapJqPHQLtJUUO/arK5XPIvgW/mXdFzoB2HPL0EC /EnuHOHJV+wWlJb03F1dn6WYdQgKgtBzl+EpgQ2F4wvQSecedJZNCSkXq0LfqVQmvNr8 MC7A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:cc; bh=Bdqq4Jx77bgWLnNeAy3XwgjzMg4NAl323SZ2VmtxgMw=; b=h7UNbsbj/0WkGrXb6/sSglyYYTH3zbprZV8c8FqSfaCVQEw7MWb8qv9Oq2nNwngIPO Zb2Qk+vomMnQ98Lwe1SHF0CjYgaqSt4BTrkur6svMLwvcPNfFnbOsIvlBCpp+jXbqyM9 x1EAFjyGgic+pe/VDA2e4RFc9HzmM3TkuE/5svmBvf8Hw5xfbRGyv5eMEUeyO8iSKYH1 IjKDK+rpRbMNraM3osjGWeV4Kf0PFTZvDWGlYiJak+YguXVMDIvBPct5QhzUzDZRqXyj UrgdDTGWGDqdpZ4ZZa0fhWhvSMmcKBcFCiryrNmyZkbvz8aR1YeJSdBCqfuHcxD5Ahgs jdcQ== X-Gm-Message-State: ALyK8tLpn/hUao1wm23u2nkdLz6n3M7q2PYHKtGrBJRsmRshy6pO2PcCH0aJmcg1AphTO6C7032G7bN11x7NkA== MIME-Version: 1.0 X-Received: by 10.107.59.18 with SMTP id i18mr18988ioa.161.1464720608963; Tue, 31 May 2016 11:50:08 -0700 (PDT) Received: by 10.107.142.65 with HTTP; Tue, 31 May 2016 11:50:08 -0700 (PDT) In-Reply-To: References: Date: Tue, 31 May 2016 11:50:08 -0700 Message-ID: From: SwamZ To: Alexander Duyck Cc: dev@dpdk.org Content-Type: multipart/mixed; boundary=94eb2c05ece297f9b5053427d7cd X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] Rx_missed_errors drops with larger packet size X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 31 May 2016 18:50:10 -0000 --94eb2c05ece297f9b5053427d7cd Content-Type: text/plain; charset=UTF-8 Hi Alex et al., I am seeing another case of rx_missed_error. In this case, rx_missed_errors are seen only when I send multiple flows. Rx_missed_errors are NOT seen if I send single flow. Following are the test details: Packet type: Unicast/Mulitcast Packet size : 512Bytes Number of flows: 1000 Rate : 24K PPS (98.3Mb) bidirectional traffic. CPU Speed : 2.6 GHz NIC : 82599ES 10-Gigabit IO Virtualization: SR-IOV Packet drops are seen for every 20-30 seconds once. Around 2K packets are getting dropped in each instance. Please let me know if anybody has seen this kind of issue in past and any pointers to debug this is really useful. This is Gen2.0 x8 PCI. Attached the lspci -vvv output for reference. Thanks for the help, Swamy SRK On Fri, May 20, 2016 at 2:47 PM, SwamZ wrote: > Hi Alex and Bruce, > Thanks for the reply. > > Alex, > Just saw your email. You are right, this issue was happening for > multicast packets only. On 82599 NIC, multicast and broadcast packet > replication (IXGBE_VT_CTL_REPLEN) was enabled even for single VF case, > hence NIC was dropping the packets as rx_missed_errors. After disabling the > replication, we are not seeing any drops. > > Regards, > Swamy SRK > > On Fri, May 20, 2016 at 9:45 AM, Alexander Duyck < > alexander.duyck@gmail.com> wrote: > >> On Fri, May 20, 2016 at 2:09 AM, SwamZ wrote: >> > Hi, >> > >> > >> > While doing performance testing with larger packet size (like 4000 >> bytes), >> > we are seeing rx_missed_errors on the interface. This issue is not seen >> > with packet size less than 2000. There were questions asked in this >> forum >> > on rx_missed_error, but there was not any conclusion. Please let me know >> > what could be the reason for these drops. >> > >> > >> > I tried the following without any luck: >> > >> > 1) Tried different burst size like 16, 32 an 64 >> > >> > 2) Tried different number of the rx descriptors like 512, 1024, 2048 and >> > 4096. >> >> I've seen issues like this occur on the 82599 using the Linux kernel >> diver in the past, but it was usually related to CPU C states. I'm >> assuming that shouldn't apply since you are using the poll mode driver >> right so there are no interrupts and the CPUs are busy polling. >> Another thing to check is that you have sufficient PCIe bandwidth. >> Take a look at an lspci -vvv for the device and make certain it is >> connected at x8 Gen2. >> >> > Setup and testing details: >> > >> > CPU Speed : 2.6 GHz >> > >> > NIC : 82599ES 10-Gigabit >> > >> > IO Virtualization: SR-IOV >> > >> > IO and packet processing cores are in the same NUMA. >> >> If you are running SR-IOV then the SRRCTL_DROP_EN bit should be set in >> all the SRRCTL registers if I am not mistaken. As such if you are >> triggering Rx missed errors it may not be related to the driver >> directly but could point to some sort of PCIe but utilization issue. >> How many VFs are you using and is any of this traffic multicast or >> broadcast? Multicast or broadcast traffic can potentially saturate >> the PCIe bus if there is enough of it on the wire as each packet is >> normally replicated for each active VF. So 1 packet on the wire (4K) >> can become 8 on the PCIe bus (32K) if you have 7 active VFs. >> >> > Packet size : 4000 bytes >> > >> > Traffic rate: 80 % line rate. If the traffic rate is < ~80% then drops >> are >> > not seen. >> > >> > Application: This is a sample application developed using L3fwd example >> app. >> > >> > DPDK version: 1.7 >> > Thanks, >> > Swamy >> > > > > -- > Swamz > -- Swamz --94eb2c05ece297f9b5053427d7cd Content-Type: text/plain; charset=US-ASCII; name="lspci_vvv.txt" Content-Disposition: attachment; filename="lspci_vvv.txt" Content-Transfer-Encoding: base64 X-Attachment-Id: f_iovssrw10 ODg6MDAuMCBFdGhlcm5ldCBjb250cm9sbGVyOiBJbnRlbCBDb3Jwb3JhdGlvbiA4MjU5OUVTIDEw LUdpZ2FiaXQgU0ZJL1NGUCsKTmV0d29yayBDb25uZWN0aW9uIChyZXYgMDEpCiAgICBTdWJzeXN0 ZW06IEhld2xldHQtUGFja2FyZCBDb21wYW55IEV0aGVybmV0IDEwR2IgMi1wb3J0IDU2MFNGUCsg QWRhcHRlcgogICAgUGh5c2ljYWwgU2xvdDogNAogICAgQ29udHJvbDogSS9PKyBNZW0rIEJ1c01h c3RlcisgU3BlY0N5Y2xlLSBNZW1XSU5WLSBWR0FTbm9vcC0gUGFyRXJyKwpTdGVwcGluZy0gU0VS UisgRmFzdEIyQi0gRGlzSU5UeCsKICAgIFN0YXR1czogQ2FwKyA2Nk1Iei0gVURGLSBGYXN0QjJC LSBQYXJFcnItIERFVlNFTD1mYXN0ID5UQWJvcnQtIDxUQWJvcnQtCjxNQWJvcnQtID5TRVJSLSA8 UEVSUi0gSU5UeC0KICAgIExhdGVuY3k6IDAsIENhY2hlIExpbmUgU2l6ZTogNjQgYnl0ZXMKICAg IEludGVycnVwdDogcGluIEIgcm91dGVkIHRvIElSUSAxNwogICAgUmVnaW9uIDA6IE1lbW9yeSBh dCBjODEwMDAwMCAoMzItYml0LCBub24tcHJlZmV0Y2hhYmxlKSBbc2l6ZT0xTV0KICAgIFJlZ2lv biAyOiBJL08gcG9ydHMgYXQgYTAyMCBbc2l6ZT0zMl0KICAgIFJlZ2lvbiAzOiBNZW1vcnkgYXQg YzgyMDQwMDAgKDMyLWJpdCwgbm9uLXByZWZldGNoYWJsZSkgW3NpemU9MTZLXQogICAgW3ZpcnR1 YWxdIEV4cGFuc2lvbiBST00gYXQgYzgyODAwMDAgW2Rpc2FibGVkXSBbc2l6ZT01MTJLXQogICAg Q2FwYWJpbGl0aWVzOiBbNDBdIFBvd2VyIE1hbmFnZW1lbnQgdmVyc2lvbiAzCiAgICAgICAgRmxh Z3M6IFBNRUNsay0gRFNJKyBEMS0gRDItIEF1eEN1cnJlbnQ9MG1BClBNRShEMCssRDEtLEQyLSxE M2hvdCssRDNjb2xkKykKICAgICAgICBTdGF0dXM6IEQwIE5vU29mdFJzdC0gUE1FLUVuYWJsZS0g RFNlbD0wIERTY2FsZT0xIFBNRS0KICAgIENhcGFiaWxpdGllczogWzUwXSBNU0k6IEVuYWJsZS0g Q291bnQ9MS8xIE1hc2thYmxlKyA2NGJpdCsKICAgICAgICBBZGRyZXNzOiAwMDAwMDAwMDAwMDAw MDAwICBEYXRhOiAwMDAwCiAgICAgICAgTWFza2luZzogMDAwMDAwMDAgIFBlbmRpbmc6IDAwMDAw MDAwCiAgICBDYXBhYmlsaXRpZXM6IFs3MF0gTVNJLVg6IEVuYWJsZSsgQ291bnQ9NjQgTWFza2Vk LQogICAgICAgIFZlY3RvciB0YWJsZTogQkFSPTMgb2Zmc2V0PTAwMDAwMDAwCiAgICAgICAgUEJB OiBCQVI9MyBvZmZzZXQ9MDAwMDIwMDAKICAgIENhcGFiaWxpdGllczogW2EwXSBFeHByZXNzICh2 MikgRW5kcG9pbnQsIE1TSSAwMAogICAgICAgIERldkNhcDogTWF4UGF5bG9hZCA1MTIgYnl0ZXMs IFBoYW50RnVuYyAwLCBMYXRlbmN5IEwwcyA8NTEybnMsIEwxIDw2NHVzCiAgICAgICAgICAgIEV4 dFRhZy0gQXR0bkJ0bi0gQXR0bkluZC0gUHdySW5kLSBSQkUrIEZMUmVzZXQrCiAgICAgICAgRGV2 Q3RsOiBSZXBvcnQgZXJyb3JzOiBDb3JyZWN0YWJsZSsgTm9uLUZhdGFsKyBGYXRhbCsgVW5zdXBw b3J0ZWQrCiAgICAgICAgICAgIFJseGRPcmQrIEV4dFRhZy0gUGhhbnRGdW5jLSBBdXhQd3ItIE5v U25vb3ArIEZMUmVzZXQtCiAgICAgICAgICAgIE1heFBheWxvYWQgMjU2IGJ5dGVzLCBNYXhSZWFk UmVxIDQwOTYgYnl0ZXMKICAgICAgICBEZXZTdGE6IENvcnJFcnIrIFVuY29yckVyci0gRmF0YWxF cnItIFVuc3VwcFJlcSsgQXV4UHdyKyBUcmFuc1BlbmQtCiAgICAgICAgTG5rQ2FwOiBQb3J0ICMw LCBTcGVlZCA1R1QvcywgV2lkdGggeDgsIEFTUE0gTDBzLCBFeGl0IExhdGVuY3kgTDBzCnVubGlt aXRlZCwgTDEgPDh1cwogICAgICAgICAgICBDbG9ja1BNLSBTdXJwcmlzZS0gTExBY3RSZXAtIEJ3 Tm90LQogICAgICAgIExua0N0bDogQVNQTSBEaXNhYmxlZDsgUkNCIDY0IGJ5dGVzIERpc2FibGVk LSBDb21tQ2xrKwogICAgICAgICAgICBFeHRTeW5jaC0gQ2xvY2tQTS0gQXV0V2lkRGlzLSBCV0lu dC0gQXV0QldJbnQtCiAgICAgICAgTG5rU3RhOiBTcGVlZCA1R1QvcywgV2lkdGggeDgsIFRyRXJy LSBUcmFpbi0gU2xvdENsaysgRExBY3RpdmUtIEJXTWdtdC0KQUJXTWdtdC0KICAgICAgICBEZXZD YXAyOiBDb21wbGV0aW9uIFRpbWVvdXQ6IFJhbmdlIEFCQ0QsIFRpbWVvdXREaXMrLCBMVFItLCBP QkZGIE5vdApTdXBwb3J0ZWQKICAgICAgICBEZXZDdGwyOiBDb21wbGV0aW9uIFRpbWVvdXQ6IDUw dXMgdG8gNTBtcywgVGltZW91dERpcy0sIExUUi0sIE9CRkYKRGlzYWJsZWQKICAgICAgICBMbmtD dGwyOiBUYXJnZXQgTGluayBTcGVlZDogNUdUL3MsIEVudGVyQ29tcGxpYW5jZS0gU3BlZWREaXMt CiAgICAgICAgICAgICBUcmFuc21pdCBNYXJnaW46IE5vcm1hbCBPcGVyYXRpbmcgUmFuZ2UsIEVu dGVyTW9kaWZpZWRDb21wbGlhbmNlLQpDb21wbGlhbmNlU09TLQogICAgICAgICAgICAgQ29tcGxp YW5jZSBEZS1lbXBoYXNpczogLTZkQgogICAgICAgIExua1N0YTI6IEN1cnJlbnQgRGUtZW1waGFz aXMgTGV2ZWw6IC02ZEIsIEVxdWFsaXphdGlvbkNvbXBsZXRlLSwKRXF1YWxpemF0aW9uUGhhc2Ux LQogICAgICAgICAgICAgRXF1YWxpemF0aW9uUGhhc2UyLSwgRXF1YWxpemF0aW9uUGhhc2UzLSwg TGlua0VxdWFsaXphdGlvblJlcXVlc3QtCiAgICBDYXBhYmlsaXRpZXM6IFtlMF0gVml0YWwgUHJv ZHVjdCBEYXRhCiAgICAgICAgUHJvZHVjdCBOYW1lOiBIUCBFdGhlcm5ldCAxMEdiIDItcG9ydCA1 NjBTRlArIEFkYXB0ZXIKICAgICAgICBSZWFkLW9ubHkgZmllbGRzOgogICAgICAgICAgICBbUE5d IFBhcnQgbnVtYmVyOiA2NjUyNDctMDAxCiAgICAgICAgICAgIFtFQ10gRW5naW5lZXJpbmcgY2hh bmdlczogRC01NTE0CiAgICAgICAgICAgIFtTTl0gU2VyaWFsIG51bWJlcjogTVlJNTI2MDhRWQog ICAgICAgICAgICBbVjBdIFZlbmRvciBzcGVjaWZpYzogMTFXLzhXIFBDSWVHMng4IDJwIDEwR2Ig U0ZQKyBJbnRlbCA4MjU5OQogICAgICAgICAgICBbVjJdIFZlbmRvciBzcGVjaWZpYzogNTUyNgog ICAgICAgICAgICBbVjRdIFZlbmRvciBzcGVjaWZpYzogMDAxMTBBNjc4NkQ4CiAgICAgICAgICAg IFtWNV0gVmVuZG9yIHNwZWNpZmljOiAwRAogICAgICAgICAgICBbUlZdIFJlc2VydmVkOiBjaGVj a3N1bSBnb29kLCAwIGJ5dGUocykgcmVzZXJ2ZWQKICAgICAgICBSZWFkL3dyaXRlIGZpZWxkczoK ICAgICAgICAgICAgW1YxXSBWZW5kb3Igc3BlY2lmaWM6IDQuNS4xOQogICAgICAgICAgICBbVjNd IFZlbmRvciBzcGVjaWZpYzogMy4wLjI0CiAgICAgICAgICAgIFtWNl0gVmVuZG9yIHNwZWNpZmlj OiAyLjMuMjAKICAgICAgICAgICAgW1lBXSBBc3NldCB0YWc6IE4vQQogICAgICAgICAgICBbWUJd IFN5c3RlbSBzcGVjaWZpYzogeHh4eHh4eHh4eHh4eHh4eAogICAgICAgICAgICBbWUNdIFN5c3Rl bSBzcGVjaWZpYzogeHh4eHh4eHh4eHh4eHh4eAogICAgICAgIEVuZAogICAgQ2FwYWJpbGl0aWVz OiBbMTAwIHYxXSBBZHZhbmNlZCBFcnJvciBSZXBvcnRpbmcKICAgICAgICBVRVN0YTogIERMUC0g U0RFUy0gVExQLSBGQ1AtIENtcGx0VE8tIENtcGx0QWJydC0gVW54Q21wbHQtIFJ4T0YtCk1hbGZU TFAtIEVDUkMtIFVuc3VwUmVxLSBBQ1NWaW9sLQogICAgICAgIFVFTXNrOiAgRExQLSBTREVTLSBU TFAtIEZDUC0gQ21wbHRUTy0gQ21wbHRBYnJ0LSBVbnhDbXBsdC0gUnhPRi0KTWFsZlRMUC0gRUNS Qy0gVW5zdXBSZXEtIEFDU1Zpb2wtCiAgICAgICAgVUVTdnJ0OiBETFArIFNERVMtIFRMUC0gRkNQ KyBDbXBsdFRPLSBDbXBsdEFicnQtIFVueENtcGx0LSBSeE9GKwpNYWxmVExQKyBFQ1JDLSBVbnN1 cFJlcS0gQUNTVmlvbC0KICAgICAgICBDRVN0YTogIFJ4RXJyLSBCYWRUTFAtIEJhZERMTFAtIFJv bGxvdmVyLSBUaW1lb3V0LSBOb25GYXRhbEVycisKICAgICAgICBDRU1zazogIFJ4RXJyLSBCYWRU TFAtIEJhZERMTFAtIFJvbGxvdmVyLSBUaW1lb3V0LSBOb25GYXRhbEVycisKICAgICAgICBBRVJD YXA6IEZpcnN0IEVycm9yIFBvaW50ZXI6IDAwLCBHZW5DYXArIENHZW5Fbi0gQ2hrQ2FwKyBDaGtF bi0KICAgIENhcGFiaWxpdGllczogWzE0MCB2MV0gRGV2aWNlIFNlcmlhbCBOdW1iZXIgMDAtMTEt MGEtZmYtZmYtNjctODYtZDgKICAgIENhcGFiaWxpdGllczogWzE1MCB2MV0gQWx0ZXJuYXRpdmUg Um91dGluZy1JRCBJbnRlcnByZXRhdGlvbiAoQVJJKQogICAgICAgIEFSSUNhcDogTUZWQy0gQUNT LSwgTmV4dCBGdW5jdGlvbjogMQogICAgICAgIEFSSUN0bDogTUZWQy0gQUNTLSwgRnVuY3Rpb24g R3JvdXA6IDAKICAgIENhcGFiaWxpdGllczogWzE2MCB2MV0gU2luZ2xlIFJvb3QgSS9PIFZpcnR1 YWxpemF0aW9uIChTUi1JT1YpCiAgICAgICAgSU9WQ2FwOiBNaWdyYXRpb24tLCBJbnRlcnJ1cHQg TWVzc2FnZSBOdW1iZXI6IDAwMAogICAgICAgIElPVkN0bDogRW5hYmxlKyBNaWdyYXRpb24tIElu dGVycnVwdC0gTVNFKyBBUklIaWVyYXJjaHkrCiAgICAgICAgSU9WU3RhOiBNaWdyYXRpb24tCiAg ICAgICAgSW5pdGlhbCBWRnM6IDY0LCBUb3RhbCBWRnM6IDY0LCBOdW1iZXIgb2YgVkZzOiA0LCBG dW5jdGlvbiBEZXBlbmRlbmN5Ckxpbms6IDAwCiAgICAgICAgVkYgb2Zmc2V0OiAxMjgsIHN0cmlk ZTogMiwgRGV2aWNlIElEOiAxMGVkCiAgICAgICAgU3VwcG9ydGVkIFBhZ2UgU2l6ZTogMDAwMDA1 NTMsIFN5c3RlbSBQYWdlIFNpemU6IDAwMDAwMDAxCiAgICAgICAgUmVnaW9uIDA6IE1lbW9yeSBh dCAwMDAwMDNiZmZmZTAwMDAwICg2NC1iaXQsIHByZWZldGNoYWJsZSkKICAgICAgICBSZWdpb24g MzogTWVtb3J5IGF0IDAwMDAwM2JmZmZkMDAwMDAgKDY0LWJpdCwgcHJlZmV0Y2hhYmxlKQogICAg ICAgIFZGIE1pZ3JhdGlvbjogb2Zmc2V0OiAwMDAwMDAwMCwgQklSOiAwCiAgICBLZXJuZWwgZHJp dmVyIGluIHVzZTogaXhnYmUK --94eb2c05ece297f9b5053427d7cd--