From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 1A22AA046B; Thu, 9 Jan 2020 16:55:47 +0100 (CET) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id EC5A51DFE6; Thu, 9 Jan 2020 16:55:46 +0100 (CET) Received: from us-smtp-delivery-1.mimecast.com (us-smtp-1.mimecast.com [207.211.31.81]) by dpdk.org (Postfix) with ESMTP id 1CB351DFE3 for ; Thu, 9 Jan 2020 16:55:45 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1578585344; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=HndLka/IbIAJApVG56kAgVOP/WqBM7gV1/diKUaU7zs=; b=PXkZhvia/Ce5IcF9d6RyAfajE5FfP8YZhak3N5hF2/zEGT9gctyUEotZZR8O7gcXHPZKMe MOD7ahaj1kgB9AEge2Jnc0xrVJrF+CQbY0x1rmj/zWpSkm4raqgps6XKXAVPSANr/SHbOG aajsmNKjNRyNOT4nPOklFmfAm4EUi4A= Received: from mail-wm1-f72.google.com (mail-wm1-f72.google.com [209.85.128.72]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-73-GuBKiZbPPCqgNufZBnacxg-1; Thu, 09 Jan 2020 10:55:40 -0500 Received: by mail-wm1-f72.google.com with SMTP id n17so608651wmk.1 for ; Thu, 09 Jan 2020 07:55:40 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:message-id:subject:from:to:cc:date:in-reply-to :references:mime-version:content-transfer-encoding; bh=HndLka/IbIAJApVG56kAgVOP/WqBM7gV1/diKUaU7zs=; b=fzhSan655cnmb8apW5LVZQTbZ9I1uGtO8d6E3U41HEHH6DZ4LVz1KSFWQbW8OczfJy kbNaTjgwcXXgGqnWKmiOpl908qMrQSk/dXc6uCZW5NzDkCSLaTCygyZxCkNlmuEFxvbl Rl34wq+hgJFp6vfb3Q/9JFmIWv/qxpW/pyWtASvmoIh/NBYwcYeWU5pBJtlrknq5yQJw rQ3/GbEzswMAMaRYjLr1hBPJkjSOI8Yv9nLnp+bYlJCg7ZbPRNF2QAjdabUUM7dhBo39 +TUax2blBBWBcay0IVemUPykHqR7e3mv+ggPaq8ebcL51VNgKBXVoFwsN2h6Wb5QcHUK SwYw== X-Gm-Message-State: APjAAAUHgPCJZ4zbfwBCSIb5DkzZuggwvHx0OzFU0LnHHqSoqXej9c1B Wnbl8f8PV9jJ0aZlgK7mdVYq+8ZvKe683W/wDVaxRyEMSQkO5qG94uD4taJZvkweJojsQIBcswX RLKE= X-Received: by 2002:a7b:ce8b:: with SMTP id q11mr5692188wmj.100.1578585339376; Thu, 09 Jan 2020 07:55:39 -0800 (PST) X-Google-Smtp-Source: APXvYqzmce8sN6zVIvtv450mhIAlag6xyDGhr10s5TaA+hLzjWJ/WYWuXFMKz23h98VxAapLQgPNiA== X-Received: by 2002:a7b:ce8b:: with SMTP id q11mr5692174wmj.100.1578585339141; Thu, 09 Jan 2020 07:55:39 -0800 (PST) Received: from eperezma.remote.csb (113.142.221.87.dynamic.jazztel.es. [87.221.142.113]) by smtp.gmail.com with ESMTPSA id y139sm3535420wmd.24.2020.01.09.07.55.38 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 09 Jan 2020 07:55:38 -0800 (PST) Message-ID: <6d0d5484704aa5cc79f20ac94ac6103e430f712f.camel@redhat.com> From: eperezma@redhat.com To: bugzilla@dpdk.org, dev@dpdk.org, Maxime Coquelin Cc: Jason Wang , "Michael S. Tsirkin" , Adrian Moreno Zapata Date: Thu, 09 Jan 2020 16:55:37 +0100 In-Reply-To: References: X-Mailer: Evolution 3.28.5 (3.28.5-6.el8) Mime-Version: 1.0 X-MC-Unique: GuBKiZbPPCqgNufZBnacxg-1 X-Mimecast-Spam-Score: 0 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [Bug 383] dpdk virtio_user lack of notifications make vhost_net+napi stops tx buffers X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" Proposal for patch - Requesting For Comments. Just running the shadow copy-flush-call unconditionally in vhost_flush_dequeue_packed solve the issue, and it gives the best latency I can get in the tests (8101.959 trans/sec if I run netperf TCP_RR, 1331.735 trans/sec if I run TCP_STREAM at the same time). Apart from that, testpmd is able to tx about 820Kpps to the guest. However, to still do a little bit of batching I replace the condition for the one attached here. Although it implies a read barrier, I am able to achieve a little more of throughput (about 890Kpps), reducing to 8048.919 the numbers of transactions/sec in TCP_RR test (1372.327 if it runs in parallel with TCP_STREAM). I also tried to move the vhost_flush_dequeue_shadow_packed and host_flush_dequeue_shadow_packed after the do_data_copy_dequeue in virtio_dev_tx_packed, more or less the same way virtio_dev_rx_packed do, but I repeatedly find less throughput in this case, even if I add the !next_desc_is_avail(vq) test. Not sure why, since both ways should be very similar. About 836Kpps are achieved this way, and TCP_RR is able to do 8120.154 trans/sec by itself and 1363.341 trans/sec if it runs with another TCP_STREAM test in parallel. So, is there room for improvement, either in the patches or in the tests? Is one of the solutions preferred over another? All tests were run with: * producer in a different processor than consumer (host testpmd and VM never run in the same core) * 256 descriptors queues in guest's testpmd and tx vq Thanks! PS: Sorry for the from mail address change, DPDK bugzilla doesn't send me the confirmation mail to this account. diff --git a/lib/librte_vhost/virtio_net.c b/lib/librte_vhost/virtio_net.c index 21c311732..f7137149c 100644 --- a/lib/librte_vhost/virtio_net.c +++ b/lib/librte_vhost/virtio_net.c @@ -382,6 +382,20 @@ vhost_shadow_enqueue_single_packed(struct virtio_net *dev, } } +static __rte_always_inline bool +next_desc_is_avail(const struct vhost_virtqueue *vq) +{ + bool wrap_counter = vq->avail_wrap_counter; + uint16_t next_used_idx = vq->last_used_idx + 1; + + if (next_used_idx >= vq->size) { + next_used_idx -= vq->size; + wrap_counter ^= 1; + } + + return desc_is_avail(&vq->desc_packed[next_used_idx], wrap_counter); +} + static __rte_always_inline void vhost_flush_dequeue_packed(struct virtio_net *dev, struct vhost_virtqueue *vq) @@ -394,7 +408,8 @@ vhost_flush_dequeue_packed(struct virtio_net *dev, if (shadow_count <= 0) shadow_count += vq->size; - if ((uint32_t)shadow_count >= (vq->size - MAX_PKT_BURST)) { + if ((uint32_t)shadow_count >= (vq->size - MAX_PKT_BURST) + || !next_desc_is_avail(vq)) { do_data_copy_dequeue(vq); vhost_flush_dequeue_shadow_packed(dev, vq); vhost_vring_call_packed(dev, vq); On Thu, 2020-01-09 at 15:47 +0000, bugzilla@dpdk.org wrote: > https://bugs.dpdk.org/show_bug.cgi?id=383 > > Bug ID: 383 > Summary: dpdk virtio_user lack of notifications make > vhost_net+napi stops tx buffers > Product: DPDK > Version: unspecified > Hardware: All > OS: Linux > Status: UNCONFIRMED > Severity: normal > Priority: Normal > Component: vhost/virtio > Assignee: dev@dpdk.org > Reporter: eupm90@gmail.com > Target Milestone: --- > > Using the current testpmd vhost_user as: > > ./app/testpmd -l 6,7,8 --vdev='net_vhost1,iface=/tmp/vhost-user1' > --vdev='net_vhost2,iface=/tmp/vhost-user2' -- -a -i --rxq=1 --txq=1 > --txd=1024 > --forward-mode=rxonly > > And starting qemu using packed=on on the interface: > > -netdev vhost-user,chardev=charnet1,id=hostnet1 -device > virtio-net-pci,rx_queue_size=256,...,packed=on > > And start to tx in the guest using: > > ./dpdk/build/app/testpmd -l 1,2 --vdev=eth_af_packet0,iface=eth0 -- \ > --forward-mode=txonly --txq=1 --txd=256 --auto-start --txpkts > 1500 \ > --stats-period 1 > > After first burst of packets (512 or a little more), sendto() will > start to > return EBUSY. kernel NAPI is refusing to send more packets to > virtio_net device > until it free old skbs. > > However, virtio_net driver is unable to free old buffers since host > does not return them in `vhost_flush_dequeue_packed` until shadow > queue is full > except for MAX_PKT_BURST (32) packets. > > Sometimes we are lucky and reach this point, or packets are small > enough to > fill the queue and flush, but if the packets and the virtqueue are > big enough, > we will not be able to tx anymore. >