From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id 21DB03195 for ; Fri, 18 Sep 2015 07:58:35 +0200 (CEST) Received: from orsmga002.jf.intel.com ([10.7.209.21]) by fmsmga103.fm.intel.com with ESMTP; 17 Sep 2015 22:58:34 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.17,550,1437462000"; d="scan'208";a="807154253" Received: from pgsmsx105.gar.corp.intel.com ([10.221.44.96]) by orsmga002.jf.intel.com with ESMTP; 17 Sep 2015 22:58:34 -0700 Received: from shsmsx102.ccr.corp.intel.com (10.239.4.154) by PGSMSX105.gar.corp.intel.com (10.221.44.96) with Microsoft SMTP Server (TLS) id 14.3.224.2; Fri, 18 Sep 2015 13:58:32 +0800 Received: from shsmsx101.ccr.corp.intel.com ([169.254.1.75]) by shsmsx102.ccr.corp.intel.com ([169.254.2.179]) with mapi id 14.03.0248.002; Fri, 18 Sep 2015 13:58:31 +0800 From: "Xie, Huawei" To: Kyle Larose Thread-Topic: [dpdk-dev] vhost-net stops sending to virito pmd -- already fixed? Thread-Index: AdDwIDscVVsIEQ6iVkioYbNKSNqzXQ== Date: Fri, 18 Sep 2015 05:58:30 +0000 Message-ID: References: <2603066.XvJpW2pAu2@xps13> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.239.127.40] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "dev@dpdk.org" Subject: Re: [dpdk-dev] vhost-net stops sending to virito pmd -- already fixed? X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 18 Sep 2015 05:58:36 -0000 On 9/17/2015 1:25 AM, Kyle Larose wrote:=0A= > Hi Huawei,=0A= >=0A= >> Kyle:=0A= >> Could you tell us how did you produce this issue, very small pool size= =0A= >> or you are using pipeline model?=0A= > If I understand correctly, by pipeline model you mean a model whereby=0A= > multiple threads handle a given packet, with some sort IPC (e.g. dpdk=0A= > rings) between them? If so, yes: we are using such a model. And I=0A= > suspect that this model is where we run into issues: the length of the=0A= > pipeline, combined with the queuing between stages, can lead to us=0A= > exhausting the mbufs, particularly when a stage's load causes queuing.=0A= Yes, exactly.=0A= >=0A= > When I initially ran into this issue, I had a fairly large mbuf pool=0A= > (32K entries), with 3 stages in the pipeline: rx, worker, tx. There=0A= > were two worker threads, with a total of 6 rings. I was sending some=0A= > fairly bursty traffic, at a high packet rate (it was bursting up to=0A= > around 1Mpkt/s). There was a low chance that this actually caused the=0A= > problem. However, when I decreased the mbuf pool to 1000 entries, it=0A= > *always* happened.=0A= >=0A= > In summary: the pipeline model is important here, and a small pool=0A= > size definitely exacerbates the problem.=0A= >=0A= > I was able to reproduce the problem using the load_balancer sample=0A= > application, though it required some modification to get it to run=0A= > with virtio. I'm not sure if this is because I'm using DPDK 1.8, or=0A= > something else. Either way, I made the number of mbufs configurable=0A= > via an environment variable, and was able to show that decreasing it=0A= > from the default of 32K to 1K would cause the problem to always happen=0A= > when using the same traffic as with my application. Applying the below=0A= > patch fixed the problem.=0A= >=0A= > The following patch seems to fix the problem for me, though I'm not=0A= > sure it's the optimal solution. It does so by removing the early exit=0A= > which prevents us from allocating mbufs. After we skip over the packet=0A= > processing loop since there are no packets, the mbuf allocation loop=0A= > runs. Note that the patch is on dpdk 1.8.=0A= Yes, it will fix your problem. We could try to do the refill each time=0A= we enter the loop no matter there is avail packets or not.=0A= =0A= > diff --git a/lib/librte_pmd_virtio/virtio_rxtx.c=0A= > b/lib/librte_pmd_virtio/virtio_rxtx.c=0A= > index c013f97..7cadf52 100644=0A= > --- a/lib/librte_pmd_virtio/virtio_rxtx.c=0A= > +++ b/lib/librte_pmd_virtio/virtio_rxtx.c=0A= > @@ -463,9 +463,6 @@ virtio_recv_pkts(void *rx_queue, struct rte_mbuf=0A= > **rx_pkts, uint16_t nb_pkts)=0A= > if (likely(num > DESC_PER_CACHELINE))=0A= > num =3D num - ((rxvq->vq_used_cons_idx + num) %=0A= > DESC_PER_CACHELINE);=0A= >=0A= > - if (num =3D=3D 0)=0A= > - return 0;=0A= > -=0A= > num =3D virtqueue_dequeue_burst_rx(rxvq, rcv_pkts, len, num);=0A= > PMD_RX_LOG(DEBUG, "used:%d dequeue:%d", nb_used, num);=0A= > for (i =3D 0; i < num ; i++) {=0A= > @@ -549,9 +546,6 @@ virtio_recv_mergeable_pkts(void *rx_queue,=0A= >=0A= > rmb();=0A= >=0A= > - if (nb_used =3D=3D 0)=0A= > - return 0;=0A= > -=0A= > PMD_RX_LOG(DEBUG, "used:%d\n", nb_used);=0A= >=0A= > while (i < nb_used) {=0A= >=0A= > Thanks,=0A= >=0A= > Kyle=0A= >=0A= =0A=