From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mail-oi0-f51.google.com (mail-oi0-f51.google.com [209.85.218.51]) by dpdk.org (Postfix) with ESMTP id 879202A61 for ; Wed, 16 Sep 2015 19:24:49 +0200 (CEST) Received: by oixx17 with SMTP id x17so131288895oix.0 for ; Wed, 16 Sep 2015 10:24:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=q20N5/4aAY36eWuR0KrF3kjJEKf66ES8YvEK9yPU3DA=; b=gZ+zWwVWOUPFbGzEowCqKbKFa7o5/7I3GkLZqk/5T8mV/iNS2bK42alM4iFz9UHQAe JKe/H825TzQ8EWEV7m7/0wlx8TjRoIXObgm61SCCddqpNJiOQmui7GZ+u4alhBI91KVI 6MA7tHr5HfceUAd8LLJLqgxc2S9jCgPeZr5iNq1cP2+NCEnzwjsyuPYo5d8DN3yDFfZP TaX13fhyqZzxvFc3CgnrLhrSFsU6JnVTuImJW1RT8bpsELoPHrh31VQXpyzpLkfRpE84 wz7/+ZRAsmK5K9dHpeOiBZEirB++P7vEDe7IUeV/rwMy6dq+2W+O/YOrmu3vavb0a0Sb 0J0w== MIME-Version: 1.0 X-Received: by 10.202.51.10 with SMTP id z10mr23570047oiz.89.1442424288891; Wed, 16 Sep 2015 10:24:48 -0700 (PDT) Received: by 10.76.83.164 with HTTP; Wed, 16 Sep 2015 10:24:48 -0700 (PDT) In-Reply-To: References: <2603066.XvJpW2pAu2@xps13> Date: Wed, 16 Sep 2015 13:24:48 -0400 Message-ID: From: Kyle Larose To: "Xie, Huawei" Content-Type: text/plain; charset=UTF-8 Cc: "dev@dpdk.org" Subject: Re: [dpdk-dev] vhost-net stops sending to virito pmd -- already fixed? X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 16 Sep 2015 17:24:50 -0000 Hi Huawei, > Kyle: > Could you tell us how did you produce this issue, very small pool size > or you are using pipeline model? If I understand correctly, by pipeline model you mean a model whereby multiple threads handle a given packet, with some sort IPC (e.g. dpdk rings) between them? If so, yes: we are using such a model. And I suspect that this model is where we run into issues: the length of the pipeline, combined with the queuing between stages, can lead to us exhausting the mbufs, particularly when a stage's load causes queuing. When I initially ran into this issue, I had a fairly large mbuf pool (32K entries), with 3 stages in the pipeline: rx, worker, tx. There were two worker threads, with a total of 6 rings. I was sending some fairly bursty traffic, at a high packet rate (it was bursting up to around 1Mpkt/s). There was a low chance that this actually caused the problem. However, when I decreased the mbuf pool to 1000 entries, it *always* happened. In summary: the pipeline model is important here, and a small pool size definitely exacerbates the problem. I was able to reproduce the problem using the load_balancer sample application, though it required some modification to get it to run with virtio. I'm not sure if this is because I'm using DPDK 1.8, or something else. Either way, I made the number of mbufs configurable via an environment variable, and was able to show that decreasing it from the default of 32K to 1K would cause the problem to always happen when using the same traffic as with my application. Applying the below patch fixed the problem. The following patch seems to fix the problem for me, though I'm not sure it's the optimal solution. It does so by removing the early exit which prevents us from allocating mbufs. After we skip over the packet processing loop since there are no packets, the mbuf allocation loop runs. Note that the patch is on dpdk 1.8. diff --git a/lib/librte_pmd_virtio/virtio_rxtx.c b/lib/librte_pmd_virtio/virtio_rxtx.c index c013f97..7cadf52 100644 --- a/lib/librte_pmd_virtio/virtio_rxtx.c +++ b/lib/librte_pmd_virtio/virtio_rxtx.c @@ -463,9 +463,6 @@ virtio_recv_pkts(void *rx_queue, struct rte_mbuf **rx_pkts, uint16_t nb_pkts) if (likely(num > DESC_PER_CACHELINE)) num = num - ((rxvq->vq_used_cons_idx + num) % DESC_PER_CACHELINE); - if (num == 0) - return 0; - num = virtqueue_dequeue_burst_rx(rxvq, rcv_pkts, len, num); PMD_RX_LOG(DEBUG, "used:%d dequeue:%d", nb_used, num); for (i = 0; i < num ; i++) { @@ -549,9 +546,6 @@ virtio_recv_mergeable_pkts(void *rx_queue, rmb(); - if (nb_used == 0) - return 0; - PMD_RX_LOG(DEBUG, "used:%d\n", nb_used); while (i < nb_used) { Thanks, Kyle