Re: [dpdk-dev] [RFC 0/4] SocketPair Broker support for vhost and virtio-user.

[Maxime Coquelin <maxime.coquelin@redhat.com>]

On 3/24/21 10:39 PM, Ilya Maximets wrote:
> On 3/24/21 9:56 PM, Maxime Coquelin wrote:
>> Hi Ilya,
>>
>> On 3/19/21 5:45 PM, Ilya Maximets wrote:
>>> On 3/19/21 5:11 PM, Ilya Maximets wrote:
>>>> On 3/19/21 3:39 PM, Stefan Hajnoczi wrote:
>>>>> Hi Ilya,
>>>>> By the way, it's not clear to me why dpdkvhostuser is deprecated. If OVS
>>>>> is restarted then existing vhost-user connections drop with an error but
>>>>> QEMU could attempt to reconnect to the UNIX domain socket which the new
>>>>> OVS instance will set up.
>>>>>
>>>>> Why is it impossible to reconnect when OVS owns the listen socket?
>>>>
>>>> Well, AFAIK, qemu reconnects client connections only:
>>>>
>>>>     ``reconnect`` sets the timeout for reconnecting on non-server
>>>>     sockets when the remote end goes away. qemu will delay this many
>>>>     seconds and then attempt to reconnect. Zero disables reconnecting,
>>>>     and is the default.
>>>>
>>>> I'm not sure about exact reason.  It was historically this way.
>>>> For me it doesn't make much sense.  I mean, your right that it's
>>>> just a socket, so it should not matter who listens and who connects.
>>>> If reconnection is possible in one direction, it should be possible
>>>> in the opposite direction too.
>>>
>>> Sorry, my thought slipped. :)  Yes, QEMU supports re-connection
>>> for client sockets.  So, in theory, dpdkvhostuser ports should work
>>> after re-connection.  And that would be nice.  I don't remember
>>> right now why this doesn't work...  Maybe vhost-user parts in QEMU
>>> doesn't handle this case.  Need to dig some more into that and refresh
>>> my memory.  It was so long ago...
>>>
>>> Maxime, do you remember?
>>
>> Sorry for the delay. I didn't remember, so I wanted to have a try.
>>
>> I can confirm reconnect works with QEMU as client and with Vhost PMD as
>> server with:
>>
>>
>>     <interface type='vhostuser'>
>>       <mac address='56:48:4f:53:54:01'/>
>>       <source type='unix' path='/tmp/vhost-user1' mode='client'>
>>         <reconnect enabled='yes' timeout='1'/>
>>       </source>
>>       <model type='virtio'/>
>>       <driver name='vhost' rx_queue_size='256'/>
>>       <address type='pci' domain='0x0000' bus='0x07' slot='0x00'
>> function='0x0'/>
>>     </interface>
> 
> Cool, thanks for checking. :)
> If it works with vhost PMD, it should probably work with OVS too.
> 
> There is still a couple of problems:
> 
> 1. OpenStack Nova doesn't support configuration of a 'reconnect'
>    in libvirt xml (it only adds the 'source'):
>    https://github.com/openstack/nova/blob/master/nova/virt/libvirt/config.py#L1834
> 
> 2. 'reconnect' configuration supported only starting from libvirt 4.1.0.
>    It's released in 2018, but still some systems are using older versions.
>    e.g. Ubuntu 18.04 which will be supported until 2023 uses libvirt 4.0.0.

Is it really a problem? We can keep using OVS as client for OSP, and use
OVS as server for containers.

>>
>>>
>>>>
>>>> dpdkvhostuser was deprecated just to scare users and force them to
>>>> migrate to dpdkvhostuserclient and avoid constant bug reports like:
>>>>
>>>>   "OVS service restarted and network is lost now".
>>>>
>>>> BTW, virtio-user ports in DPDK doesn't support re-connection in client
>>>> mode too.
>>>
>>> This is still true, though.  virtio-user in client mode doesn't reconnect.
>>
>> That could be added, and it is maybe not as important for containers as
>> it is for VM to support it, given the ephemeral nature of containers?
> 
> Well, restart of OVS should not require restarting of all the containers
> on the host even though they are "stateless".
> 
> BTW, some infrastructure changes that I made in this series might be
> reused to implement client-side reconnection for virtio-user.

Great, I'll look at the series when we'll work on adding reconnect for
clients.

Thanks,
Maxime

>>
>> Regards,
>> Maxime
>>
>>>
>>>>
>>>> BTW2, with SocketPair Broker it might be cheaper to implement server
>>>> reconnection in QEMU because all connections in these case are client
>>>> connections, i.e. both ends will connect() to a broker.
>>>>
>>>> Bets regards, Ilya Maximets.
>>>>
>>>
>