From: Maxime Coquelin <maxime.coquelin@redhat.com>
To: Yuanhan Liu <yliu@fridaylinux.org>
Cc: dev@dpdk.org, jfreiman@redhat.com, tiwei.bie@intel.com,
mst@redhat.com, vkaplans@redhat.com, jasowang@redhat.com
Subject: Re: [dpdk-dev] [PATCH 21/21] vhost: iotlb: reduce iotlb read lock usage
Date: Mon, 11 Sep 2017 09:34:30 +0200 [thread overview]
Message-ID: <a82793b6-a339-76b1-5b12-5cbbb811665c@redhat.com> (raw)
In-Reply-To: <20170911041821.GI9736@yliu-home>
Hi Yuanhan,
On 09/11/2017 06:18 AM, Yuanhan Liu wrote:
> On Thu, Aug 31, 2017 at 11:50:23AM +0200, Maxime Coquelin wrote:
>> Prior to this patch, iotlb cache's read/write lock was
>> read-locked at every guest IOVA to app VA translation,
>> i.e. at least once per packet with indirect off and twice
>> with indirect on.
>>
>> The problem is that rte_rwlock_read_lock() makes use of atomic
>> operation, which is costly.
>>
>> This patch introduces iotlb lock helpers, so that a full burst
>> can be protected with taking the lock once, which reduces the
>> number of atomic operations by up to 64 with indirect
>> descriptors.
>
> You were assuming there is no single miss during a burst. If a miss
> happens, it requries 2 locks: one for _pending_miss and another one
> for _pending_insert. From this point of view, it's actually more
> expensive.
It's actually more expensive only when a miss happens. And in that case,
the cost of taking the lock is negligible compared to the miss itself.
> However, I won't call it's a bad assumption (for the case of virtio
> PMD). And if you take this assumption, why not just deleting the
> pending list and moving the lock outside the _iotlb_find function()
> like what you did in this patch?
Because we need the pending list. When the is no matching entry in the
IOTLB cache, we have to send a miss request through the slave channel.
On miss request reception, Qemu performs the translation, and in case of
success, sends it through the main channel using an update request.
While all this is done, the backend could wait for it, blocking
processing on the PMD thread. But it would be really inefficient in case
other queues are being processed on the same lcore. Moreover, if the
iova is invalid, no update requst is sent, so the lcore would be blocked
forever.
To overcome this, what is done is that in case of miss, it exits the
burst and try again later, letting a chance for other virtqueues to be
processed while the update arrives.
And here comes the pending list. On the next try, the update may have
not arrived yet, so we need the check whether a miss has already been
sent for the same address & perm. Else, we would flood Qemu with miss
requests for the same address.
> I don't really see the point of introducing the pending list.
Hope the above clarifies.
I will see if I can improve the pending list protection, but honestly,
its cost is negligible.
Cheers,
Maxime
> --yliu
>
next prev parent reply other threads:[~2017-09-11 7:34 UTC|newest]
Thread overview: 59+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-08-31 9:50 [dpdk-dev] [PATCH 00/21] Vhost-user: Implement device IOTLB support Maxime Coquelin
2017-08-31 9:50 ` [dpdk-dev] [PATCH 01/21] Revert "vhost: workaround MQ fails to startup" Maxime Coquelin
2017-09-07 11:54 ` Yuanhan Liu
2017-09-07 12:59 ` Maxime Coquelin
2017-09-24 10:41 ` Maxime Coquelin
2017-08-31 9:50 ` [dpdk-dev] [PATCH 02/21] vhost: make error handling consistent in rx path Maxime Coquelin
2017-08-31 9:50 ` [dpdk-dev] [PATCH 03/21] vhost: protect virtio_net device struct Maxime Coquelin
2017-09-05 4:45 ` Tiwei Bie
2017-09-05 9:24 ` Maxime Coquelin
2017-09-05 10:07 ` Tiwei Bie
2017-09-05 11:00 ` Maxime Coquelin
2017-09-06 1:15 ` Tiwei Bie
2017-09-06 2:59 ` Stephen Hemminger
2017-09-06 7:50 ` Maxime Coquelin
2017-09-06 7:15 ` Maxime Coquelin
2017-09-06 7:30 ` Tiwei Bie
2017-09-06 20:02 ` Maxime Coquelin
2017-09-07 5:08 ` Tiwei Bie
2017-09-07 13:44 ` Yuanhan Liu
2017-09-07 14:01 ` Maxime Coquelin
2017-08-31 9:50 ` [dpdk-dev] [PATCH 04/21] vhost: prepare send_vhost_message() to slave requests Maxime Coquelin
2017-08-31 9:50 ` [dpdk-dev] [PATCH 05/21] vhost: add support to slave requests channel Maxime Coquelin
2017-09-05 4:19 ` Tiwei Bie
2017-09-05 8:18 ` Maxime Coquelin
2017-08-31 9:50 ` [dpdk-dev] [PATCH 06/21] vhost: declare missing IOMMU-related definitions for old kernels Maxime Coquelin
2017-08-31 9:50 ` [dpdk-dev] [PATCH 07/21] vhost: add iotlb helper functions Maxime Coquelin
2017-09-05 6:02 ` Tiwei Bie
2017-09-05 15:16 ` Maxime Coquelin
2017-09-08 8:08 ` Yuanhan Liu
2017-09-08 8:24 ` Maxime Coquelin
2017-09-08 8:36 ` Yuanhan Liu
2017-09-08 8:50 ` Maxime Coquelin
2017-09-08 9:21 ` Yuanhan Liu
2017-09-08 9:28 ` Maxime Coquelin
2017-08-31 9:50 ` [dpdk-dev] [PATCH 08/21] vhost: iotlb: add pending miss request list and helpers Maxime Coquelin
2017-09-05 7:11 ` Tiwei Bie
2017-09-05 15:18 ` Maxime Coquelin
2017-08-31 9:50 ` [dpdk-dev] [PATCH 09/21] vhost-user: add support to IOTLB miss slave requests Maxime Coquelin
2017-08-31 9:50 ` [dpdk-dev] [PATCH 10/21] vhost: initialize vrings IOTLB caches Maxime Coquelin
2017-09-04 13:57 ` Remy Horton
2017-09-04 15:45 ` Maxime Coquelin
2017-08-31 9:50 ` [dpdk-dev] [PATCH 11/21] vhost-user: handle IOTLB update and invalidate requests Maxime Coquelin
2017-08-31 9:50 ` [dpdk-dev] [PATCH 12/21] vhost: introduce guest IOVA to backend VA helper Maxime Coquelin
2017-09-05 4:14 ` Tiwei Bie
2017-09-05 7:05 ` Maxime Coquelin
2017-08-31 9:50 ` [dpdk-dev] [PATCH 13/21] vhost: use the guest IOVA to host " Maxime Coquelin
2017-08-31 9:50 ` [dpdk-dev] [PATCH 14/21] vhost: enable rings at the right time Maxime Coquelin
2017-08-31 9:50 ` [dpdk-dev] [PATCH 15/21] vhost: don't dereference invalid dev pointer after its reallocation Maxime Coquelin
2017-09-04 13:58 ` Remy Horton
2017-08-31 9:50 ` [dpdk-dev] [PATCH 16/21] vhost: postpone rings addresses translation Maxime Coquelin
2017-08-31 9:50 ` [dpdk-dev] [PATCH 17/21] vhost-user: translate ring addresses when IOMMU enabled Maxime Coquelin
2017-08-31 9:50 ` [dpdk-dev] [PATCH 18/21] vhost-user: iommu: postpone device creation until ring are mapped Maxime Coquelin
2017-08-31 9:50 ` [dpdk-dev] [PATCH 19/21] vhost: iommu: Invalidate vring in case of matching IOTLB invalidate Maxime Coquelin
2017-08-31 9:50 ` [dpdk-dev] [PATCH 20/21] vhost: enable IOMMU support Maxime Coquelin
2017-08-31 9:50 ` [dpdk-dev] [PATCH 21/21] vhost: iotlb: reduce iotlb read lock usage Maxime Coquelin
2017-09-11 4:18 ` Yuanhan Liu
2017-09-11 7:34 ` Maxime Coquelin [this message]
2017-09-11 9:39 ` Yuanhan Liu
2017-09-04 13:58 ` [dpdk-dev] [PATCH 00/21] Vhost-user: Implement device IOTLB support Remy Horton
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=a82793b6-a339-76b1-5b12-5cbbb811665c@redhat.com \
--to=maxime.coquelin@redhat.com \
--cc=dev@dpdk.org \
--cc=jasowang@redhat.com \
--cc=jfreiman@redhat.com \
--cc=mst@redhat.com \
--cc=tiwei.bie@intel.com \
--cc=vkaplans@redhat.com \
--cc=yliu@fridaylinux.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).