DPDK patches and discussions
 help / color / mirror / Atom feed
From: "Zhang, Roy Fan" <roy.fan.zhang@intel.com>
To: Maxime Coquelin <maxime.coquelin@redhat.com>,
	"Xia, Chenbo" <chenbo.xia@intel.com>,
	"Liu, Changpeng" <changpeng.liu@intel.com>,
	"dev@dpdk.org" <dev@dpdk.org>
Cc: "matan@mellanox.com" <matan@mellanox.com>,
	"Zawadzki, Tomasz" <tomasz.zawadzki@intel.com>,
	"Yigit, Ferruh" <ferruh.yigit@intel.com>
Subject: Re: [dpdk-dev] [PATCH] vhost: return ready when at least 1 vring is configured
Date: Thu, 1 Oct 2020 08:42:49 +0000	[thread overview]
Message-ID: <BL0PR11MB30435C08C3A383644104B8ADB8300@BL0PR11MB3043.namprd11.prod.outlook.com> (raw)
In-Reply-To: <b44b8b34-1a8f-b200-ec39-c3ede54c8a74@redhat.com>

Hi Maxime,

Yes I totally missed. Sorry about that.
Will try this way. Should work. Thanks a lot.

Regards,
Fan

> -----Original Message-----
> From: Maxime Coquelin <maxime.coquelin@redhat.com>
> Sent: Thursday, October 1, 2020 9:26 AM
> To: Zhang, Roy Fan <roy.fan.zhang@intel.com>; Xia, Chenbo
> <chenbo.xia@intel.com>; Liu, Changpeng <changpeng.liu@intel.com>;
> dev@dpdk.org
> Cc: matan@mellanox.com; Zawadzki, Tomasz <tomasz.zawadzki@intel.com>;
> Yigit, Ferruh <ferruh.yigit@intel.com>
> Subject: Re: [dpdk-dev] [PATCH] vhost: return ready when at least 1 vring is
> configured
> 
> Hi Fan,
> 
> On 10/1/20 10:07 AM, Zhang, Roy Fan wrote:
> > Hi Maxime,
> >
> >> -----Original Message-----
> >> From: Maxime Coquelin <maxime.coquelin@redhat.com>
> >> Sent: Thursday, October 1, 2020 8:55 AM
> >> To: Zhang, Roy Fan <roy.fan.zhang@intel.com>; Xia, Chenbo
> >> <chenbo.xia@intel.com>; Liu, Changpeng <changpeng.liu@intel.com>;
> >> dev@dpdk.org
> >> Cc: matan@mellanox.com; Zawadzki, Tomasz
> <tomasz.zawadzki@intel.com>;
> >> Yigit, Ferruh <ferruh.yigit@intel.com>
> >> Subject: Re: [dpdk-dev] [PATCH] vhost: return ready when at least 1 vring
> is
> >> configured
> >>
> >>
> >>
> >> On 9/30/20 6:37 PM, Zhang, Roy Fan wrote:
> >>> Hi Chenbo and Maxime,
> >>>
> >>> Thanks for replying the email.
> >>>
> >>>
> >>>> -----Original Message-----
> >>>> From: Maxime Coquelin <maxime.coquelin@redhat.com>
> >>>> Sent: Wednesday, September 30, 2020 4:37 PM
> >>>> To: Xia, Chenbo <chenbo.xia@intel.com>; Zhang, Roy Fan
> >>>> <roy.fan.zhang@intel.com>; Liu, Changpeng
> <changpeng.liu@intel.com>;
> >>>> dev@dpdk.org
> >>>> Cc: matan@mellanox.com; Zawadzki, Tomasz
> >> <tomasz.zawadzki@intel.com>;
> >>>> Yigit, Ferruh <ferruh.yigit@intel.com>
> >>>> Subject: Re: [dpdk-dev] [PATCH] vhost: return ready when at least 1
> vring
> >> is
> >>>> configured
> >>>>
> >>>> Hi,
> >>>>
> >>>> On 9/30/20 4:48 AM, Xia, Chenbo wrote:
> >>>>> Hi Fan & Maxime,
> >>>>>
> >>>>> I am thinking that should we move set_features outside of
> new_device
> >>>> callback
> >>>>> for crypto device? I see that net and blk devices both set features
> >> between
> >>>> register
> >>>>> and start, and personally I think this makes sense that device features
> >> are
> >>>> set
> >>>>> before device start and ready. How do you think 😊?
> >>>
> >>> The reason it is set inside rte_vhost_crypto_create() is logically speaking
> >>> the user shouldn't expect to have to set the feature flags even after the
> >> create
> >>> function is called - and what I know in the application the only way to
> know
> >> the
> >>> vid for the first time is when new_device() is invoked. So if there is away
> to
> >> know
> >>> the vid before new_device() is invoked I am happy to change the sample
> >> app.
> >>
> >> I think the Vhost-crypto API should be fixed.
> >> The good news is that it is still experimental, so we can fix it without
> >> worries (BTW, except the DPDK example, are there other application
> using
> >> the Vhost-crypto API?).
> >>
> >> The .new_device() callback is called when the Virtio device is ready,
> >> meaning when the backend can start processing the virtqueues. So
> feature
> >> negotiation should have taken place before that.
> >>
> >> I'm surprised it worked before, because doesn't the features negotiation
> >> takes place before the memory table are set? If so, how can the first
> >> virtqueue can be tested as ready if the vring address is not set?
> >>
> >> One other issue here, which is triggering the issue is that given how
> >> the registration is done, VIRTIO_DEV_BUILTIN_VIRTIO_NET flag is set for
> >> Vhost-crypto, which shouldn't happen. Even before last release rework,
> >> you should have faced issues when more than 2 queues where in used:
> >>
> >
> > Vhost-crypto was not working since 20.05. Changpeng's patch which set the
> > Number of queues to one made it working again so we waited it merged.
> 
> But the patch introducing the regression was introduced in v20.08, I'm
> confused.
> 
> > However the patch was rejected by you.
> 
> Indeed, I rejected the patch because it would break net backends.
> > I suppose there is another way - adding a new API called
> > "rte_vhost_crypto_set_feature(const char *socket)" so we don't have to
> > rely on rte_vhost_crypto_create() to set the feature flags
> >
> > what do you think?
> 
> The set_features thing is just another problem. The main problem is that
> VIRTIO_DEV_BUILTIN_VIRTIO_NET gets set for crypto backend, which does
> not make sense.
> 
> I proposed a fix below to be able to differentiate between builtin net
> and crypto backends below, but I think you missed it. Please check
> below.
> 
> > Regards,
> > Fan
> >
> >> static int
> >> vhost_user_set_features(struct virtio_net **pdev, struct VhostUserMsg
> >> *msg,
> >> 			int main_fd __rte_unused)
> >> {
> >> ...
> >> 	if ((dev->flags & VIRTIO_DEV_BUILTIN_VIRTIO_NET) &&
> >> 	    !(dev->features & (1ULL << VIRTIO_NET_F_MQ))) {
> >> 		/*
> >> 		 * Remove all but first queue pair if MQ hasn't been
> >> 		 * negotiated. This is safe because the device is not
> >> 		 * running at this stage.
> >> 		 */
> >> 		while (dev->nr_vring > 2) {
> >> 			struct vhost_virtqueue *vq;
> >>
> >> 			vq = dev->virtqueue[--dev->nr_vring];
> >> 			if (!vq)
> >> 				continue;
> >>
> >> 			dev->virtqueue[dev->nr_vring] = NULL;
> >> 			cleanup_vq(vq, 1);
> >> 			cleanup_vq_inflight(dev, vq);
> >> 			free_vq(dev, vq);
> >> 		}
> >> 	}
> >>
> >> As VIRTIO_NET_F_MQ is never negotiated with crypto devices, it means
> you
> >> can not have more than two queues.
> >>
> >>>>
> >>>> Indeed, we cannot consider the device to be ready (and so call
> >>>> .new_device callback) if features haven't been negotiated.
> >>>>
> >>>> I agree, rte_vhost_driver_set_features() has to be called before
> >>>> .new_device(), and that's the reason why it takes socket's path and not
> >>>> vid as input.
> >>>
> >>> Different than vhost_blk, vhost_crypto lies in the library and needs to be
> >>> able to be treated evenly as virtio_net. Without the new_device() calling
> >>> rte_vhost_crypto_create() the feature flag cannot be set. Without
> setting
> >>> the feature flag the device is always treated as virtio_net device, hence
> it
> >>> cannot pass the virtio_is_ready() check as the number of queues virtio
> >>> crypt uses is only 1 instead of 2 (required by virtio_net).
> >>
> >> OK, so we are aligned, we need to find a proper solution. I think you
> >> need a specific driver start function that does not set
> >> VIRTIO_DEV_BUILTIN_VIRTIO_NET.
> >>
> >> First we can change that VIRTIO_DEV_BUILTIN_VIRTIO_NET flag  by a new
> >> field in the device without breaking the API:
> >>
> >> enum virtio_backend_type {
> >> 	VIRTIO_DEV_UNKNOWN = 0, /* Likely external */
> >>  	VIRTIO_DEV_BUILTIN_NET,
> >> 	VIRTIO_DEV_BUILTIN_CRYPTO,
> >> };
> >>
> >> struct virtio_net {
> >> ...
> >> 	enum virtio_backend_type type;
> >> };
> >>
> >>
> >> Then, introduce a new API to start crypto backend that would be called
> >> instead of rte_vhost_driver_start():
> >>
> >> int
> >> rte_vhost_crypto_driver_start(const char *path)
> >> {
> >>
> >> 	return vhost_driver_start(path, VIRTIO_DEV_BUILTIN_CRYPTO);
> >> }
> >>
> >>
> >> int
> >> rte_vhost_driver_start(const char *path)
> >> {
> >>
> >> 	return vhost_driver_start(path, VIRTIO_DEV_BUILTIN_NET);
> >> }
> >>
> >> And then propagate the info down to vhost_new_device().
> >>
> >> Does that make sense?
> 
> Note that it does not fix the set_feature thing, which would also need
> to be fixed. But it should revert the behaviour for crypto backends back
> to pre-v20.08, as Changpeng patch did.
> 
> >> Note that issue has been reported during v20.11 cycle was it was
> >> introduced in v20.08. It means it has not been tested. Does Intel QE has
> >> some Vhost crypto tests?
> 
> 
> >> Thanks,
> >> Maxime


      reply	other threads:[~2020-10-01  8:43 UTC|newest]

Thread overview: 17+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-09-01  7:07 Changpeng Liu
2020-09-18  9:53 ` Maxime Coquelin
2020-09-21  5:03   ` Liu, Changpeng
2020-09-21 10:19     ` Maxime Coquelin
2020-09-22  7:22       ` Liu, Changpeng
2020-09-23  8:05         ` Maxime Coquelin
2020-09-23  8:14           ` Liu, Changpeng
2020-09-29 13:54           ` Zhang, Roy Fan
2020-09-29 14:05             ` Maxime Coquelin
2020-09-29 18:15               ` Zhang, Roy Fan
2020-09-30  2:48                 ` Xia, Chenbo
2020-09-30 15:36                   ` Maxime Coquelin
2020-09-30 16:37                     ` Zhang, Roy Fan
2020-10-01  7:55                       ` Maxime Coquelin
2020-10-01  8:07                         ` Zhang, Roy Fan
2020-10-01  8:26                           ` Maxime Coquelin
2020-10-01  8:42                             ` Zhang, Roy Fan [this message]

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=BL0PR11MB30435C08C3A383644104B8ADB8300@BL0PR11MB3043.namprd11.prod.outlook.com \
    --to=roy.fan.zhang@intel.com \
    --cc=changpeng.liu@intel.com \
    --cc=chenbo.xia@intel.com \
    --cc=dev@dpdk.org \
    --cc=ferruh.yigit@intel.com \
    --cc=matan@mellanox.com \
    --cc=maxime.coquelin@redhat.com \
    --cc=tomasz.zawadzki@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).