From: "Michael S. Tsirkin" <mst@redhat.com>
To: Maxime Coquelin <maxime.coquelin@redhat.com>
Cc: "Yao, Lei A" <lei.a.yao@intel.com>, "dev@dpdk.org" <dev@dpdk.org>,
"yliu@fridaylinux.org" <yliu@fridaylinux.org>,
"Horton, Remy" <remy.horton@intel.com>,
"Bie, Tiwei" <tiwei.bie@intel.com>,
"jfreiman@redhat.com" <jfreiman@redhat.com>,
"vkaplans@redhat.com" <vkaplans@redhat.com>,
"jasowang@redhat.com" <jasowang@redhat.com>
Subject: Re: [dpdk-dev] [PATCH v3 17/19] vhost-user: iommu: postpone device creation until ring are mapped
Date: Fri, 3 Nov 2017 17:15:15 +0200 [thread overview]
Message-ID: <20171103171230-mutt-send-email-mst@kernel.org> (raw)
In-Reply-To: <c915792b-34f7-5c22-334b-cde17b09d47a@redhat.com>
On Fri, Nov 03, 2017 at 09:25:58AM +0100, Maxime Coquelin wrote:
>
>
> On 11/02/2017 05:02 PM, Maxime Coquelin wrote:
> >
> >
> > On 11/02/2017 09:21 AM, Maxime Coquelin wrote:
> > > Hi Lei,
> > >
> > > On 11/02/2017 08:21 AM, Yao, Lei A wrote:
> > > >
> > > ...
> > > > Hi, Maxime > I met one issue with your patch set during the v17.11 test.
> > >
> > > Is it with v17.11-rc2 or -rc1?
> > >
> > > > The test scenario is following,
> > > > 1. Bind one NIC, use test-pmd set vhost-user with 2 queue
> > > > usertools/dpdk-devbind.py --bind=igb_uio 0000:05:00.0
> > > > ./x86_64-native-linuxapp-gcc/app/testpmd -c 0xe -n 4
> > > > --socket-mem 1024,1024 \
> > > > --vdev 'net_vhost0,iface=vhost-net,queues=2' - -i --rxq=2
> > > > --txq=2 --nb-cores=2 --rss-ip
> > > > 2. Launch qemu with virtio device which has 2 queue
> > > > 3. In VM, launch testpmd with virtio-pmd using only 1 queue.
> > > > x86_64-native-linuxapp-gcc/app/testpmd -c 0x07 -n 3 - -i
> > > > --txqflags=0xf01 \
> > > > --rxq=1 --txq=1 --rss-ip --nb-cores=1
> > > >
> > > > First,
> > > > commit 09927b5249694bad1c094d3068124673722e6b8f
> > > > vhost: translate ring addresses when IOMMU enabled
> > > > The patch causes no traffic in PVP test. but link status is
> > > > still up in vhost-user.
> > > >
> > > > Second,
> > > > eefac9536a901a1f0bb52aa3b6fec8f375f09190
> > > > vhost: postpone device creation until rings are mapped
> > > > The patch causes link status "down" in vhost-user.
> >
> > I reproduced this one, and understand why link status remains down.
> > My series did fixed a potential issue Michael raised, that the vring
> > addresses should only interpreted once the ring is enabled.
> > When VHOST_USER_F_PROTOCOL_FEATURES is negotiated, the rings addrs are
> > translated when ring is enabled via VHOST_USER_SET_VRING_ENABLE.
> > When not negotiated, the ring is considered started enabled, so
> > translation is done at VHOST_USER_SET_VRING_KICK time.
> >
> > In your case, protocol features are negotiated, so the ring addresses
> > are translated at enable time. The problem is that the code considers
> > the device is ready once addresses for all the rings are translated.
> > But since only the first pair of rings is used, it never happens, and
> > the link remains down.
> >
> > One of the reason this check is done is to avoid starting the PMD
> > threads before the addresses are translated in case of NUMA
> > reallocation, as virtqueues and virtio-net device structs can be
> > reallocated on a different node.
> >
> > I think the right fix would be to only perform NUMA reallocation for
> > vring 0, as today we would end-up reallocating virtio-net struct
> > mulitple time if VQs are on different NUMA nodes.
> >
> > Doing that, we could then consider the device is ready if vring 0 is
> > enabled and its ring addresses are translated, and if other vrings have
> > been kicked.
> >
> > I'll post a patch shortly implementing this idea.
>
> The proposed solution doesn't work, because disabled queues get accessed at
> device start time:
>
> int
> rte_vhost_enable_guest_notification(int vid, uint16_t queue_id, int enable)
> {
> ..
> dev->virtqueue[queue_id]->used->flags = VRING_USED_F_NO_NOTIFY;
> return 0;
> }
>
> The above function being called in Vhost PMD for every queues, enabled
> or not. While we could fix the PMD, it could break other applications
> using the Vhost lib API directly, so we cannot translate at enable
> time reliably.
>
> I think we may be a bit less conservative, and postpone addresses
> translation at kick time, whatever VHOST_USER_F_PROTOCOL_FEATURES is
> negotiated or not.
>
> Regards,
> Maxime
>
> > Thanks,
> > Maxime
I agree, enabling has nothing to do with it.
The spec is quite explicit:
Client must only process each ring when it is started.
and
Client must start ring upon receiving a kick (that is, detecting that file
descriptor is readable) on the descriptor specified by
VHOST_USER_SET_VRING_KICK, and stop ring upon receiving
VHOST_USER_GET_VRING_BASE.
--
MST
next prev parent reply other threads:[~2017-11-03 15:15 UTC|newest]
Thread overview: 48+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-10-05 8:36 [dpdk-dev] [PATCH v3 00/19] Vhost-user: Implement device IOTLB support Maxime Coquelin
2017-10-05 8:36 ` [dpdk-dev] [PATCH v3 01/19] Revert "vhost: workaround MQ fails to startup" Maxime Coquelin
2017-11-01 17:11 ` Kavanagh, Mark B
2017-11-02 9:40 ` Maxime Coquelin
2017-11-03 13:05 ` Yuanhan Liu
2017-11-03 14:28 ` Maxime Coquelin
2017-11-06 12:00 ` Yuanhan Liu
2017-11-06 12:07 ` Maxime Coquelin
2017-11-06 12:24 ` Yuanhan Liu
2017-11-06 12:50 ` Maxime Coquelin
2017-11-06 13:36 ` Yuanhan Liu
2017-11-03 15:34 ` Thomas Monjalon
2017-11-03 16:31 ` Kavanagh, Mark B
2017-10-05 8:36 ` [dpdk-dev] [PATCH v3 02/19] vhost: make error handling consistent in rx path Maxime Coquelin
2017-10-05 8:36 ` [dpdk-dev] [PATCH v3 03/19] vhost: prepare send_vhost_message() to slave requests Maxime Coquelin
2017-10-05 8:36 ` [dpdk-dev] [PATCH v3 04/19] vhost: add support to slave requests channel Maxime Coquelin
2017-10-05 8:36 ` [dpdk-dev] [PATCH v3 05/19] vhost: declare missing IOMMU-related definitions for old kernels Maxime Coquelin
2017-10-05 8:36 ` [dpdk-dev] [PATCH v3 06/19] vhost: add iotlb helper functions Maxime Coquelin
2017-10-05 8:36 ` [dpdk-dev] [PATCH v3 07/19] vhost: iotlb: add pending miss request list and helpers Maxime Coquelin
2017-10-05 8:36 ` [dpdk-dev] [PATCH v3 08/19] vhost-user: add support to IOTLB miss slave requests Maxime Coquelin
2017-10-05 8:36 ` [dpdk-dev] [PATCH v3 09/19] vhost: initialize vrings IOTLB caches Maxime Coquelin
2017-10-05 8:36 ` [dpdk-dev] [PATCH v3 10/19] vhost-user: handle IOTLB update and invalidate requests Maxime Coquelin
2017-10-05 8:36 ` [dpdk-dev] [PATCH v3 11/19] vhost: introduce guest IOVA to backend VA helper Maxime Coquelin
2017-10-05 8:36 ` [dpdk-dev] [PATCH v3 12/19] vhost: use the guest IOVA to host " Maxime Coquelin
2017-10-05 8:36 ` [dpdk-dev] [PATCH v3 13/19] vhost: enable rings at the right time Maxime Coquelin
2017-10-05 8:36 ` [dpdk-dev] [PATCH v3 14/19] vhost: don't dereference invalid dev pointer after its reallocation Maxime Coquelin
2017-10-05 8:36 ` [dpdk-dev] [PATCH v3 15/19] vhost: postpone rings addresses translation Maxime Coquelin
2017-10-13 1:47 ` Yao, Lei A
2017-10-13 7:32 ` Maxime Coquelin
2017-10-13 7:55 ` Yao, Lei A
2017-10-13 7:56 ` Maxime Coquelin
2017-10-16 5:59 ` Yao, Lei A
2017-10-16 6:23 ` Yao, Lei A
2017-10-16 9:47 ` Maxime Coquelin
2017-10-16 10:47 ` Maxime Coquelin
2017-10-17 1:24 ` Yao, Lei A
2017-10-17 8:06 ` Maxime Coquelin
2017-10-05 8:36 ` [dpdk-dev] [PATCH v3 16/19] vhost-user: translate ring addresses when IOMMU enabled Maxime Coquelin
2017-10-05 8:36 ` [dpdk-dev] [PATCH v3 17/19] vhost-user: iommu: postpone device creation until ring are mapped Maxime Coquelin
2017-11-02 7:21 ` Yao, Lei A
2017-11-02 8:21 ` Maxime Coquelin
2017-11-02 16:02 ` Maxime Coquelin
2017-11-03 8:25 ` Maxime Coquelin
2017-11-03 15:15 ` Michael S. Tsirkin [this message]
2017-11-03 15:54 ` Maxime Coquelin
2017-10-05 8:36 ` [dpdk-dev] [PATCH v3 18/19] vhost: iommu: Invalidate vring in case of matching IOTLB invalidate Maxime Coquelin
2017-10-05 8:36 ` [dpdk-dev] [PATCH v3 19/19] vhost: enable IOMMU support Maxime Coquelin
2017-10-06 6:24 ` [dpdk-dev] [PATCH v3 00/19] Vhost-user: Implement device IOTLB support Yuanhan Liu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20171103171230-mutt-send-email-mst@kernel.org \
--to=mst@redhat.com \
--cc=dev@dpdk.org \
--cc=jasowang@redhat.com \
--cc=jfreiman@redhat.com \
--cc=lei.a.yao@intel.com \
--cc=maxime.coquelin@redhat.com \
--cc=remy.horton@intel.com \
--cc=tiwei.bie@intel.com \
--cc=vkaplans@redhat.com \
--cc=yliu@fridaylinux.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).