Re: [dpdk-dev] [PATCH v3 01/19] Revert "vhost: workaround MQ fails to startup"

[Maxime Coquelin <maxime.coquelin@redhat.com>]

Hi Mark,

On 11/01/2017 06:11 PM, Kavanagh, Mark B wrote:
> Hi Maxime,
> 
> First off, apologies for the lateness of this reply - I realize that this patch has already been upstreamed.

No worries, great to see DPDK integration being tested before v17.11 is
released. Is the v17.11 upgrade patch available somewhere?

> Unfortunately, during OvS-DPDK regression testing for DPDK v17.11-rc2 just today, a regression involving vHost multiq was detected, and pinpointed to this patch.
> 
> Version info for the components involved during the aforementioned testing is as follows:
> DPDK:	v17.11-rc2
> OvS:	af2e40c ("sparse: eliminate "duplicate initialization") + DPDK v17.11 upgrade patch
> QEMU:	v2.7.0
> 
> The regression may be reproduced as follows:
> - Set up OvS-DPDK as normal, and add one vHostUser client port:
> 	$OVS_DIR/utilities/ovs-vsctl add-port br0 vhost-user0 -- set Interface vhost-user0 type=dpdkvhostuserclient
>   
> - Start QEMU, specifying 2 ports for the guest interface"
> 	$QEMU_DIR/x86_64-softmmu/qemu-system-x86_64 \
>      -cpu host -enable-kvm -m 4096M \
>      -object memory-backend-file,id=mem,size=4096M,mem-path=/dev/hugepages,share=on \
>      -numa node,memdev=mem -mem-prealloc \
>      -drive file=$VM_IMAGE \
>      -chardev socket,id=char0,path=/tmp/sock0,server \
>      -netdev type=vhost-user,id=mynet1,chardev=char0,vhostforce,queues=2 \
>      -device virtio-net-pci,mac=00:00:00:00:00:01,netdev=mynet1,mrg_rxbuf=off,mq=on,vectors=6 \
>      -nographic"
>   
> - The guest subsequently starts as normal, but then hangs completely, rendering it completely unusable.
>    The last lines of ovs-vswitchd.log read as follows:
> 	|00051|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_VRING_ENABLE
> 	|00052|dpdk|INFO|VHOST_CONFIG: set queue enable: 1 to qp idx: 1
> 	|00053|dpdk|INFO|VHOST_CONFIG: read message VHOST_USER_SET_FEATURES
> 
> Moving from QEMU v2.7.0 to v2.10.0 resolves the issue. However, herein lies the issue: QEMU v2.10.0 was only released in August of this year; anecdotally, we know that many OvS-DPDK customers use older versions of QEMU (typically, v2.7.0), and are likely un[able|willing] to move. With this patch, a hard dependency on QEMU v2.10 is created for users who want to use the vHU multiq feature in DPDK v17.11 (and subsequently, the upcoming OvS v2.9.0), which IMO will likely be unacceptable for many.

Do you mean that upstream Qemu v2.7.0 is used in production?
I would expect the customers to use a distro Qemu which should contain
relevant fixes, or follow upstream's stable branches.

FYI, Qemu v2.9.1 contains a backport of the fix.

> One potential solution to this problem is to introduce a compile-time option that would allow the user to [dis|en]able the VHOST_USER_PROTOCOL_F_REPLY_ACK feature - is that something that would be acceptable to you Maxime?

Yes, that's one option, but:
1. VHOST_USER_PROTOCOL_F_REPLY_ACK enabled should be the default
2. VHOST_USER_PROTOCOL_F_REPLY_ACK disabled will be less extensively
tested.

Yuanhan, what do you think?

Regards,
Maxime

> Thanks in advance,
> Mark
> 
> 
>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Maxime Coquelin
>> Sent: Thursday, October 5, 2017 9:36 AM
>> To: dev@dpdk.org; Horton, Remy <remy.horton@intel.com>; Bie, Tiwei
>> <tiwei.bie@intel.com>; yliu@fridaylinux.org
>> Cc: mst@redhat.com; jfreiman@redhat.com; vkaplans@redhat.com;
>> jasowang@redhat.com; Maxime Coquelin <maxime.coquelin@redhat.com>
>> Subject: [dpdk-dev] [PATCH v3 01/19] Revert "vhost: workaround MQ fails to
>> startup"
>>
>> This reverts commit 04d81227960b5c1cf2f11f492100979ead20c526.
>>
>> As agreed when this workaround was introduced, it can be reverted
>> as Qemu v2.10 that fixes the issue is now out.
>>
>> The reply-ack feature is required for vhost-user IOMMU support.
>>
>> Signed-off-by: Maxime Coquelin <maxime.coquelin@redhat.com>
>> ---
>> lib/librte_vhost/vhost_user.h | 6 +-----
>> 1 file changed, 1 insertion(+), 5 deletions(-)
>>
>> diff --git a/lib/librte_vhost/vhost_user.h b/lib/librte_vhost/vhost_user.h
>> index 35ebd7190..2ba22dbb0 100644
>> --- a/lib/librte_vhost/vhost_user.h
>> +++ b/lib/librte_vhost/vhost_user.h
>> @@ -49,14 +49,10 @@
>> #define VHOST_USER_PROTOCOL_F_REPLY_ACK	3
>> #define VHOST_USER_PROTOCOL_F_NET_MTU 4
>>
>> -/*
>> - * disable REPLY_ACK feature to workaround the buggy QEMU implementation.
>> - * Proved buggy QEMU includes v2.7 - v2.9.
>> - */
>> #define VHOST_USER_PROTOCOL_FEATURES	((1ULL << VHOST_USER_PROTOCOL_F_MQ) | \
>> 					 (1ULL << VHOST_USER_PROTOCOL_F_LOG_SHMFD) |\
>> 					 (1ULL << VHOST_USER_PROTOCOL_F_RARP) | \
>> -					 (0ULL << VHOST_USER_PROTOCOL_F_REPLY_ACK) | \
>> +					 (1ULL << VHOST_USER_PROTOCOL_F_REPLY_ACK) | \
>> 					 (1ULL << VHOST_USER_PROTOCOL_F_NET_MTU))
>>
>> typedef enum VhostUserRequest {
>> --
>> 2.13.6
>