From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id A866AA046B for ; Fri, 23 Aug 2019 13:49:47 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 0D1C71BFBD; Fri, 23 Aug 2019 13:49:44 +0200 (CEST) Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id 7ECF81BF5C; Fri, 23 Aug 2019 13:49:40 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga001.jf.intel.com ([10.7.209.18]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 23 Aug 2019 04:49:39 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.64,421,1559545200"; d="scan'208";a="263174639" Received: from fyigit-mobl.ger.corp.intel.com (HELO [10.237.221.4]) ([10.237.221.4]) by orsmga001.jf.intel.com with ESMTP; 23 Aug 2019 04:49:37 -0700 To: =?UTF-8?Q?Miroslav_Kov=c3=a1=c4=8d?= , "dev@dpdk.org" , "users@dpdk.org" Cc: =?UTF-8?Q?Juraj_Linke=c5=a1?= , =?UTF-8?Q?Miroslav_Miklu=c5=a1?= , Beilei Xing , Qi Zhang References: <51bf2af189834b1d91717a3e8e648885@pantheon.tech> From: Ferruh Yigit Openpgp: preference=signencrypt Autocrypt: addr=ferruh.yigit@intel.com; prefer-encrypt=mutual; keydata= mQINBFXZCFABEADCujshBOAaqPZpwShdkzkyGpJ15lmxiSr3jVMqOtQS/sB3FYLT0/d3+bvy qbL9YnlbPyRvZfnP3pXiKwkRoR1RJwEo2BOf6hxdzTmLRtGtwWzI9MwrUPj6n/ldiD58VAGQ +iR1I/z9UBUN/ZMksElA2D7Jgg7vZ78iKwNnd+vLBD6I61kVrZ45Vjo3r+pPOByUBXOUlxp9 GWEKKIrJ4eogqkVNSixN16VYK7xR+5OUkBYUO+sE6etSxCr7BahMPKxH+XPlZZjKrxciaWQb +dElz3Ab4Opl+ZT/bK2huX+W+NJBEBVzjTkhjSTjcyRdxvS1gwWRuXqAml/sh+KQjPV1PPHF YK5LcqLkle+OKTCa82OvUb7cr+ALxATIZXQkgmn+zFT8UzSS3aiBBohg3BtbTIWy51jNlYdy ezUZ4UxKSsFuUTPt+JjHQBvF7WKbmNGS3fCid5Iag4tWOfZoqiCNzxApkVugltxoc6rG2TyX CmI2rP0mQ0GOsGXA3+3c1MCdQFzdIn/5tLBZyKy4F54UFo35eOX8/g7OaE+xrgY/4bZjpxC1 1pd66AAtKb3aNXpHvIfkVV6NYloo52H+FUE5ZDPNCGD0/btFGPWmWRmkPybzColTy7fmPaGz cBcEEqHK4T0aY4UJmE7Ylvg255Kz7s6wGZe6IR3N0cKNv++O7QARAQABtCVGZXJydWggWWln aXQgPGZlcnJ1aC55aWdpdEBpbnRlbC5jb20+iQJUBBMBCgA+AhsDAh4BAheABQkI71rKFiEE 0jZTh0IuwoTjmYHH+TPrQ98TYR8FAlznMMQFCwkIBwMFFQoJCAsFFgIDAQAACgkQ+TPrQ98T YR/B9Q//a57esjq996nfZVm7AsUl7zbvhN+Ojity25ib2gcSVVsAN2j6lcQS4hf6/OVvRj3q CgebJ4o2gXR6X12UzWBJL7NE8Xpc70MvUIe0r11ykurQ9n9jUaWMjxdSqBPF93hU+Z/MZe5M 1rW5O2VJLuTJzkDw3EYUCbHOwPjeaS8Qqj3RI0LYbGthbHBIp9CsjkgsJSjTT5GQ8AQWkE7I z+hvPx6f1rllfjxFyi4DI3jLhAI+j1Nm+l+ESyoX59HrLTHAvq4RPkLpTnGBj9gOnJ+5sVEr GE0fcffsNcuMSkpqSEoJCPAHmChoLgezskhhsy0BiU3xlSIj1Dx2XMDerUXFOK3ftlbYNRte HQy4EKubfZRB8H5Rvcpksom3fRBDcJT8zw+PTH14htRApU9f8I/RamQ7Ujks7KuaB7JX5QaG gMjfPzHGYX9PfF6KIchaFmAWLytIP1t0ht8LpJkjtvUCSQZ2VxpCXwKyUzPDIF3co3tp90o7 X07uiC5ymX0K0+Owqs6zeslLY6DMxNdt8ye+h1TVkSZ5g4dCs4C/aiEF230+luL1CnejOv/K /s1iSbXQzJNM7be3FlRUz4FdwsfKiJJF7xYALSBnSvEB04R7I2P2V9Zpudkq6DRT6HZjBeJ1 pBF2J655cdoenPBIeimjnnh4K7YZBzwOLJf2c6u76fe5Ag0EV9ZMvgEQAKc0Db17xNqtSwEv mfp4tkddwW9XA0tWWKtY4KUdd/jijYqc3fDD54ESYpV8QWj0xK4YM0dLxnDU2IYxjEshSB1T qAatVWz9WtBYvzalsyTqMKP3w34FciuL7orXP4AibPtrHuIXWQOBECcVZTTOdZYGAzaYzxiA ONzF9eTiwIqe9/oaOjTwTLnOarHt16QApTYQSnxDUQljeNvKYt1lZE/gAUUxNLWsYyTT+22/ vU0GDUahsJxs1+f1yEr+OGrFiEAmqrzpF0lCS3f/3HVTU6rS9cK3glVUeaTF4+1SK5ZNO35p iVQCwphmxa+dwTG/DvvHYCtgOZorTJ+OHfvCnSVjsM4kcXGjJPy3JZmUtyL9UxEbYlrffGPQ I3gLXIGD5AN5XdAXFCjjaID/KR1c9RHd7Oaw0Pdcq9UtMLgM1vdX8RlDuMGPrj5sQrRVbgYH fVU/TQCk1C9KhzOwg4Ap2T3tE1umY/DqrXQgsgH71PXFucVjOyHMYXXugLT8YQ0gcBPHy9mZ qw5mgOI5lCl6d4uCcUT0l/OEtPG/rA1lxz8ctdFBVOQOxCvwRG2QCgcJ/UTn5vlivul+cThi 6ERPvjqjblLncQtRg8izj2qgmwQkvfj+h7Ex88bI8iWtu5+I3K3LmNz/UxHBSWEmUnkg4fJl Rr7oItHsZ0ia6wWQ8lQnABEBAAGJAjwEGAEKACYCGwwWIQTSNlOHQi7ChOOZgcf5M+tD3xNh HwUCXOcvZgUJBvIWKAAKCRD5M+tD3xNhHxhBD/9toXMIaPIVFd9w1nKsRDM1GE6gZe4jie8q MJpeHB9O+936fSXA0W2X0het60wJQQ45O8TpTcxpc9nGzcE4MTaLAI3E8TjIXAO0cPqUNLyp g0DXezmTw5BU+SKZ51+jSKOtFmzJCHOJZQaMeCHD+G3CrdUHQVQBb5AeuH3KFv9ltgDcWsc8 YO70o3+tGHwcEnyXLdrI0q05wV7ncnLdkgVo+VUN4092bNMPwYly1TZWcU3Jw5gczOUEfTY7 sgo6E/sGX3B+FzgIs5t4yi1XOweCAQ/mPnb6uFeNENEFyGKyMG1HtjwBqnftbiFO3qitEIUY xWGQH23oKscv7i9lT0gg2D+ktzZhVWwHJVY/2vWSB9aCSWChcH2BT+lWrkwSpoPhy+almM84 Qz2wF72/d4ce4L27pSrS+vOXtXHLGOOGcAn8yr9TV0kM4aR+NbGBRXGKhG6w4lY54uNd9IBa ARIPUhij5JSygxZCBaJKo+X64AHGkk5bXq+f0anwAMNuJXbYC/lz4DEdKmPgQGShOWNs1Y1a N3cI87Hun/RBVwQ0a3Tr1g6OWJ6xK8cYbMcoR8NZ7L9ALMeJeuUDQR39+fEeHg/6sQN0P0mv 0sL+//BAJphCzDk8ztbrFw+JaPtgzZpRSM6JhxnY+YMAsatJRXA0WSpYP5zzl7yu/GZJIgsv VQ== Message-ID: Date: Fri, 23 Aug 2019 12:49:36 +0100 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.8.0 MIME-Version: 1.0 In-Reply-To: <51bf2af189834b1d91717a3e8e648885@pantheon.tech> Content-Type: text/plain; charset=utf-8 Content-Language: en-US Content-Transfer-Encoding: 8bit Subject: Re: [dpdk-users] [dpdk-dev] Intel XXV710 SR-IOV packet loss X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" On 8/20/2019 4:36 PM, Miroslav Kováč wrote: > Hello, > > > We are trying a setup with intel 25 GB card XXV710 and sr-iov. We need sr-iov to sort packets based on vlan in between the VFs. We are using trex on one machine to generate packets and multiple VPPs (each in docker container, using one VF) on another one. Trex machine contains the exact same hardware. > > > Each VF contains one vlan with spoof checking off and trust on and specific MAC address. For example -> > > > vf 0 MAC ba:dc:0f:fe:ed:00, vlan 1537, spoof checking off, link-state auto, trust on > > > > We are generating packets with VF destination MACs with the corresponding VLAN. When sending packets to 3 VFs trex shows 35 million tx-packets and Dpdk stats on the trex machine show that 35 million were in fact sent out: > > > ##### DPDK Statistics port0 ##### > { > "tx_good_bytes": 2142835740, > "tx_good_packets": 35713929, > "tx_size_64_packets": 35713929, > "tx_unicast_packets": 35713929 > } > > > rate= '96%'; pktSize= 64; frameLoss%=51.31%; bytesReceived/s= 1112966528.00; totalReceived= 17390102; totalSent= 35713929; frameLoss= 18323827; bytesReceived= 1112966528; targetDuration=1.0 > > > However VPP shows only 33 million rx-packets: > > VirtualFunctionEthernet17/a/0 2 up 9000/0/0/0 > rx packets 5718196 > rx bytes 343091760 > rx-miss 5572089 > > VirtualFunctionEthernet17/a/1 2 up 9000/0/0/0 > rx packets 5831396 > rx bytes 349883760 > rx-miss 5459089 > > VirtualFunctionEthernet17/a/2 2 up 9000/0/0/0 > rx packets 5840512 > rx bytes 350430720 > rx-miss 5449466 > > Sum of rx packets and rx-miss is 33,870,748. About 2 million is missing. > > > > Even when I check VFs stats I see only 33 million to come (out of which 9.9 million are rx-missed): > > > root@protonet:/home/protonet# for f in $(ls /sys/class/net/enp23s0f1/device/sriov/*/stats/rx_packets); do echo "$f: $(cat $f)"; done | grep -v ' 0$' > > /sys/class/net/enp23s0f1/device/sriov/0/stats/rx_packets: 11290290 > /sys/class/net/enp23s0f1/device/sriov/1/stats/rx_packets: 11290485 > /sys/class/net/enp23s0f1/device/sriov/2/stats/rx_packets: 11289978 > > > > When increasing the number of VFs the number of rx-packets on VPP is actually decreasing. Up to 6 or 7 VFs I still receive somewhere around 28-33 million packets, but when I use 8 VFs all the sudden it drops to 16 million packets (no rx-miss any more). The same goes with trunk mode: > > > VirtualFunctionEthernet17/a/0 2 up 9000/0/0/0 > rx packets 1959110 > rx bytes 117546600 > > > VirtualFunctionEthernet17/a/1 2 up 9000/0/0/0 > rx packets 1959181 > rx bytes 117550860 > > VirtualFunctionEthernet17/a/2 2 up 9000/0/0/0 > rx packets 1956242 > rx bytes 117374520 > . > . > . > Approximately the same amount of packets for each VPP instance which is 2 million packets * 8 = 16 million packets out of 35 million sent. Almost 20 million are gone > > > > We are using vfio-pci driver. > > > The strange thing is that when I use only PF, no sr-iov VFs are on and I try the same vpp setup I can see all 35 million packets to come across. > > > This leads us to believe that there could be something wrong the sr-iov on XXV710 but we don't know how to debug this any further - The packets seem to be lost somewhere in the NIC when using sr-iov and we don't know of any dpdk or linux tool that could help us with locating the lost packets. > > > Regards, > > Miroslav Kovac > +i40e maintainers.