From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <huawei.xie@intel.com> Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by dpdk.org (Postfix) with ESMTP id CB18956B7 for <dev@dpdk.org>; Wed, 16 Sep 2015 03:37:27 +0200 (CEST) Received: from fmsmga002.fm.intel.com ([10.253.24.26]) by fmsmga101.fm.intel.com with ESMTP; 15 Sep 2015 18:37:26 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.17,537,1437462000"; d="scan'208";a="806031555" Received: from pgsmsx107.gar.corp.intel.com ([10.221.44.105]) by fmsmga002.fm.intel.com with ESMTP; 15 Sep 2015 18:37:25 -0700 Received: from shsmsx102.ccr.corp.intel.com (10.239.4.154) by PGSMSX107.gar.corp.intel.com (10.221.44.105) with Microsoft SMTP Server (TLS) id 14.3.224.2; Wed, 16 Sep 2015 09:37:24 +0800 Received: from shsmsx101.ccr.corp.intel.com ([169.254.1.75]) by shsmsx102.ccr.corp.intel.com ([169.254.2.179]) with mapi id 14.03.0248.002; Wed, 16 Sep 2015 09:37:22 +0800 From: "Xie, Huawei" <huawei.xie@intel.com> To: Kyle Larose <eomereadig@gmail.com>, Thomas Monjalon <thomas.monjalon@6wind.com> Thread-Topic: [dpdk-dev] vhost-net stops sending to virito pmd -- already fixed? Thread-Index: AdDwIDscVVsIEQ6iVkioYbNKSNqzXQ== Date: Wed, 16 Sep 2015 01:37:21 +0000 Message-ID: <C37D651A908B024F974696C65296B57B4057019E@SHSMSX101.ccr.corp.intel.com> References: <CAMFWN9k9euifKwLv0E6Z_F1r-nXdo9RdidDw6rnqa63miRc2Vg@mail.gmail.com> <2603066.XvJpW2pAu2@xps13> <CAMFWN9=wBusSSwr+7dvZNrzORfb06HUcvhgfGyaM0OwWEX76Zg@mail.gmail.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.239.127.40] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "dev@dpdk.org" <dev@dpdk.org> Subject: Re: [dpdk-dev] vhost-net stops sending to virito pmd -- already fixed? X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK <dev.dpdk.org> List-Unsubscribe: <http://dpdk.org/ml/options/dev>, <mailto:dev-request@dpdk.org?subject=unsubscribe> List-Archive: <http://dpdk.org/ml/archives/dev/> List-Post: <mailto:dev@dpdk.org> List-Help: <mailto:dev-request@dpdk.org?subject=help> List-Subscribe: <http://dpdk.org/ml/listinfo/dev>, <mailto:dev-request@dpdk.org?subject=subscribe> X-List-Received-Date: Wed, 16 Sep 2015 01:37:28 -0000 On 9/16/2015 5:05 AM, Kyle Larose wrote:=0A= > On Sun, Sep 13, 2015 at 5:43 PM, Thomas Monjalon=0A= > <thomas.monjalon@6wind.com> wrote:=0A= >> Hi,=0A= >>=0A= >> 2015-09-11 12:32, Kyle Larose:=0A= >>> Looking through the version tree for virtio_rxtx.c, I saw the following= =0A= >>> commit:=0A= >>>=0A= >>> http://dpdk.org/browse/dpdk/commit/lib/librte_pmd_virtio?id=3D8c09c20fb= 4cde76e53d87bd50acf2b441ecf6eb8=0A= >>>=0A= >>> Does anybody know offhand if the issue fixed by that commit could be th= e=0A= >>> root cause of what I am seeing?=0A= >> I won't have the definitive answer but I would like to use your question= =0A= >> to highlight a common issue in git messages:=0A= >>=0A= >> PLEASE, authors of fixes, explain the bug you are fixing and how it can= =0A= >> be reproduced. Good commit messages are REALLY read and useful.=0A= >>=0A= >> Thanks=0A= >>=0A= > I've figured out what happened. It has nothing to do with the fix I=0A= > pasted above. Instead, the issue has to do with running low on mbufs.=0A= >=0A= > Here's the general logic:=0A= >=0A= > 1. If packets are not queued, return=0A= > 2. Fetch each queued packet, as an mbuf, into the provided array. This=0A= > may involve some merging/etc=0A= > 3. Try to fill the virtio receive ring with new mbufs=0A= > 3.a. If we fail to allocate an mbuf, break out of the refill loop=0A= > 4. Update the receive ring information and kick the host=0A= >=0A= > This is obviously a simplification, but the key point is 3.a. If we=0A= > hit this logic when the virtio receive ring is completely used up, we=0A= > essentially lock up. The host will have no buffers with which to queue=0A= > packets, so the next time we poll, we will hit case 1. However, since=0A= > we hit case 1, we will not allocate mbufs to the virtio receive ring,=0A= > regardless of how many are now free. Rinse and repeat; we are stuck=0A= > until the pmd is restarted or the link is restarted.=0A= >=0A= > This is very easy to reproduce when the mbuf pool is fairly small, and=0A= > packets are being passed to worker threads/processes which may=0A= > increase the length of the pipeline.=0A= Sorry for the trouble, and thanks a lot for your investigation. It quite=0A= makes sense. We would check the code and come back to you.=0A= I remember we fixed a similar problem before. Would check if there is=0A= other dead lock issue.=0A= >=0A= > I took a quick look at the ixgbe driver, and it looks like it checks=0A= > if it needs to allocate mbufs to the ring before trying to pull=0A= > packets off the nic. Should we not be doing something similar for=0A= > virtio? Rather than breaking out early if no packets are queued, we=0A= > should first make sure there are resources with which to queue=0A= > packets!=0A= Yes, this is a implementation bug.=0A= >=0A= > One solution here is to increase the mbuf pool to a size where such=0A= > exhaustion is impossible, but that doesn't seem like a graceful=0A= > solution. For example, it may be desirable to drop packets rather than=0A= > have a large memory pool, and becoming stuck under such a situation is=0A= > not good. Further, it isn't easy to know the exact size required. You=0A= > may end up wasting a bunch of resources allocating far more than=0A= > necessary, or you may unknowingly under allocate, only to find out=0A= > once your application has been deployed into production, and it's=0A= > dropping everything on the floor.=0A= Kyle:=0A= Could you tell us how did you produce this issue, very small pool size=0A= or you are using pipeline model?=0A= >=0A= > Does anyone have thoughts on this? I took a look at virtio_rxtx and=0A= > head and I didn't see anything resembling my suggestion.=0A= >=0A= > Comments would be appreciated. Thanks,=0A= >=0A= > Kyle=0A= >=0A= =0A=