Re: [dpdk-dev] [PATCH v2] vhost: add virtio configuration space messages

["Liu, Changpeng" <changpeng.liu@intel.com>]

> -----Original Message-----
> From: Maxime Coquelin [mailto:maxime.coquelin@redhat.com]
> Sent: Wednesday, March 28, 2018 6:11 PM
> To: Liu, Changpeng <changpeng.liu@intel.com>; Kulasek, TomaszX
> <tomaszx.kulasek@intel.com>; yliu@fridaylinux.org
> Cc: Verkamp, Daniel <daniel.verkamp@intel.com>; Harris, James R
> <james.r.harris@intel.com>; Wodkowski, PawelX
> <pawelx.wodkowski@intel.com>; dev@dpdk.org; Tan, Jianfeng
> <jianfeng.tan@intel.com>
> Subject: Re: [dpdk-dev] [PATCH v2] vhost: add virtio configuration space
> messages
> 
> 
> 
> On 03/28/2018 12:03 PM, Liu, Changpeng wrote:
> >
> >
> >> -----Original Message-----
> >> From: Maxime Coquelin [mailto:maxime.coquelin@redhat.com]
> >> Sent: Wednesday, March 28, 2018 5:58 PM
> >> To: Liu, Changpeng <changpeng.liu@intel.com>; Kulasek, TomaszX
> >> <tomaszx.kulasek@intel.com>; yliu@fridaylinux.org
> >> Cc: Verkamp, Daniel <daniel.verkamp@intel.com>; Harris, James R
> >> <james.r.harris@intel.com>; Wodkowski, PawelX
> >> <pawelx.wodkowski@intel.com>; dev@dpdk.org; Tan, Jianfeng
> >> <jianfeng.tan@intel.com>
> >> Subject: Re: [dpdk-dev] [PATCH v2] vhost: add virtio configuration space
> >> messages
> >>
> >>
> >>
> >> On 03/28/2018 11:50 AM, Liu, Changpeng wrote:
> >>>
> >>>
> >>>> -----Original Message-----
> >>>> From: Maxime Coquelin [mailto:maxime.coquelin@redhat.com]
> >>>> Sent: Wednesday, March 28, 2018 5:12 PM
> >>>> To: Kulasek, TomaszX <tomaszx.kulasek@intel.com>; yliu@fridaylinux.org
> >>>> Cc: Verkamp, Daniel <daniel.verkamp@intel.com>; Harris, James R
> >>>> <james.r.harris@intel.com>; Wodkowski, PawelX
> >>>> <pawelx.wodkowski@intel.com>; dev@dpdk.org; Liu, Changpeng
> >>>> <changpeng.liu@intel.com>; Tan, Jianfeng <jianfeng.tan@intel.com>
> >>>> Subject: Re: [dpdk-dev] [PATCH v2] vhost: add virtio configuration space
> >>>> messages
> >>>>
> >>>>
> >>>>
> >>>> On 03/27/2018 05:35 PM, Tomasz Kulasek wrote:
> >>>>> This patch adds new vhost user messages GET_CONFIG and SET_CONFIG
> >> used
> >>>>> for get/set virtio device's configuration space.
> >>>>>
> >>>>> Signed-off-by: Changpeng Liu <changpeng.liu@intel.com>
> >>>>> Signed-off-by: Tomasz Kulasek <tomaszx.kulasek@intel.com>
> >>>>> ---
> >>>>> Changes in v2:
> >>>>>     - code cleanup
> >>>>>
> >>>>>     lib/librte_vhost/rte_vhost.h  |  4 ++++
> >>>>>     lib/librte_vhost/vhost_user.c | 22 ++++++++++++++++++++++
> >>>>>     lib/librte_vhost/vhost_user.h | 16 ++++++++++++++++
> >>>>>     3 files changed, 42 insertions(+)
> >>>>>
> >>>>> diff --git a/lib/librte_vhost/rte_vhost.h b/lib/librte_vhost/rte_vhost.h
> >>>>> index d332069..fe30518 100644
> >>>>> --- a/lib/librte_vhost/rte_vhost.h
> >>>>> +++ b/lib/librte_vhost/rte_vhost.h
> >>>>> @@ -84,6 +84,10 @@ struct vhost_device_ops {
> >>>>>     	int (*new_connection)(int vid);
> >>>>>     	void (*destroy_connection)(int vid);
> >>>>>
> >>>>> +	int (*get_config)(int vid, uint8_t *config, uint32_t config_len);
> >>>>> +	int (*set_config)(int vid, uint8_t *config, uint32_t offset,
> >>>>> +			uint32_t len, uint32_t flags);
> >>>>> +
> >>>>>     	void *reserved[2]; /**< Reserved for future extension */
> >>>>
> >>>> You are breaking the ABI, as you grow the size of the ops struct.
> >>>>
> >>>> Also, I'm wondering if we shouldn't have a different ops for external
> >>>> backends. Here these ops are more intended to the application, we could
> >>>> have a specific ops struct for external backends IMHO.
> >>>>
> >>>>>     };
> >>>>>
> >>>>> diff --git a/lib/librte_vhost/vhost_user.c b/lib/librte_vhost/vhost_user.c
> >>>>> index 90ed211..0ed6a5a 100644
> >>>>> --- a/lib/librte_vhost/vhost_user.c
> >>>>> +++ b/lib/librte_vhost/vhost_user.c
> >>>>> @@ -50,6 +50,8 @@ static const char
> >> *vhost_message_str[VHOST_USER_MAX]
> >>>> = {
> >>>>>     	[VHOST_USER_NET_SET_MTU]  = "VHOST_USER_NET_SET_MTU",
> >>>>>     	[VHOST_USER_SET_SLAVE_REQ_FD]  =
> >>>> "VHOST_USER_SET_SLAVE_REQ_FD",
> >>>>>     	[VHOST_USER_IOTLB_MSG]  = "VHOST_USER_IOTLB_MSG",
> >>>>> +	[VHOST_USER_GET_CONFIG] = "VHOST_USER_GET_CONFIG",
> >>>>> +	[VHOST_USER_SET_CONFIG] = "VHOST_USER_SET_CONFIG",
> >>>>>     };
> >>>>>
> >>>>>     static uint64_t
> >>>>> @@ -1355,6 +1357,7 @@ vhost_user_msg_handler(int vid, int fd)
> >>>>>     	 * would cause a dead lock.
> >>>>>     	 */
> >>>>>     	switch (msg.request.master) {
> >>>>> +	case VHOST_USER_SET_CONFIG:
> >>>>
> >>>> It seems VHOST_USER_GET_CONFIG is missing here.
> >>>>
> >>>>>     	case VHOST_USER_SET_FEATURES:
> >>>>>     	case VHOST_USER_SET_PROTOCOL_FEATURES:
> >>>>>     	case VHOST_USER_SET_OWNER:
> >>>>> @@ -1380,6 +1383,25 @@ vhost_user_msg_handler(int vid, int fd)
> >>>>>     	}
> >>>>>
> >>>>>     	switch (msg.request.master) {
> >>>>> +	case VHOST_USER_GET_CONFIG:
> >>>>> +		if (dev->notify_ops->get_config(dev->vid,
> >>>> Please check ->get_config is set before calling it.
> >>>>
> >>>>> +				msg.payload.config.region,
> >>>>> +				msg.payload.config.size) != 0) {
> >>>>> +			msg.size = sizeof(uint64_t);
> >>>>> +		}
> >>>>> +		send_vhost_reply(fd, &msg);
> >>>>> +		break;
> >>>>> +	case VHOST_USER_SET_CONFIG:
> >>>>> +		if ((dev->notify_ops->set_config(dev->vid,
> >>>> Ditto.
> >>>>
> >>>>> +				msg.payload.config.region,
> >>>>> +				msg.payload.config.offset,
> >>>>> +				msg.payload.config.size,
> >>>>> +				msg.payload.config.flags)) != 0) {
> >>>>> +			ret = 1;
> >>>>> +		} else {
> >>>>> +			ret = 0;
> >>>>> +		}
> >>>>
> >>>> ret = dev->notify_ops->set_config instead?
> >>>>> +		break;
> >>>>>     	case VHOST_USER_GET_FEATURES:
> >>>>>     		msg.payload.u64 = vhost_user_get_features(dev);
> >>>>>     		msg.size = sizeof(msg.payload.u64);
> >>>>> diff --git a/lib/librte_vhost/vhost_user.h b/lib/librte_vhost/vhost_user.h
> >>>>> index d4bd604..25cc026 100644
> >>>>> --- a/lib/librte_vhost/vhost_user.h
> >>>>> +++ b/lib/librte_vhost/vhost_user.h
> >>>>> @@ -14,6 +14,11 @@
> >>>>>
> >>>>>     #define VHOST_MEMORY_MAX_NREGIONS 8
> >>>>>
> >>>>> +/*
> >>>>> + * Maximum size of virtio device config space
> >>>>> + */
> >>>>> +#define VHOST_USER_MAX_CONFIG_SIZE 256
> >>>>> +
> >>>>>     #define VHOST_USER_PROTOCOL_F_MQ	0
> >>>>>     #define VHOST_USER_PROTOCOL_F_LOG_SHMFD	1
> >>>>>     #define VHOST_USER_PROTOCOL_F_RARP	2
> >>>>
> >>>> Shouldn't there be a protocol feature associated to these new messages?
> >>>> Else how QEMU knows the backend supports it or not?
> >>>>
> >>>> I looked at QEMU code and indeed no protocol feature associated, that's
> >>>> strange...
> >>> Nice to have, for now not all the QEMU host driver need to get this
> >> configuration space from slave backend
> >>> when getting start. This message can be used for migration of vhost-user
> >> devices.
> >>
> >> So if QEMU sends this message but the DPDK version does not support it
> >> yet, vhost_user_msg_handler() will return an error ("vhost read
> >> incorrect message") and the socket will be closed.
> >>
> >> How do we overcome this? I think we really need a spec update ASAP,
> >> before QEMU v2.12 is out (-rc1 already).
> >>
> >> Do you have time to take care of this?
> > For now there are no other users except us care about this message, :), it's no
> hurry.
> > I can take this after QEMU 2.12 release adding a new protocol feature bit.
> 
> Are you sure?
> If I understand the code correctly, as the guest writes in config regs
> of a virtio-blk device, .set_config callback will be called.
Exactly.
> 
> If you have a vhost-user backend, it will receive the SET_CONFIG
> request, no?
For now this only enabled for QEMU vhost-user-blk driver, QEMU virtio-blk driver didn't have such issue.
> 
> Cheers,
> Maxime
> 
> >>
> >> Thanks,
> >> Maxime