Re: [dpdk-dev] [RFC 0/4] SocketPair Broker support for vhost and virtio-user.

[Ilya Maximets <i.maximets@ovn.org>]

On 3/24/21 10:51 PM, Maxime Coquelin wrote:
> 
> 
> On 3/24/21 10:39 PM, Ilya Maximets wrote:
>> On 3/24/21 9:56 PM, Maxime Coquelin wrote:
>>> Hi Ilya,
>>>
>>> On 3/19/21 5:45 PM, Ilya Maximets wrote:
>>>> On 3/19/21 5:11 PM, Ilya Maximets wrote:
>>>>> On 3/19/21 3:39 PM, Stefan Hajnoczi wrote:
>>>>>> Hi Ilya,
>>>>>> By the way, it's not clear to me why dpdkvhostuser is deprecated. If OVS
>>>>>> is restarted then existing vhost-user connections drop with an error but
>>>>>> QEMU could attempt to reconnect to the UNIX domain socket which the new
>>>>>> OVS instance will set up.
>>>>>>
>>>>>> Why is it impossible to reconnect when OVS owns the listen socket?
>>>>>
>>>>> Well, AFAIK, qemu reconnects client connections only:
>>>>>
>>>>>     ``reconnect`` sets the timeout for reconnecting on non-server
>>>>>     sockets when the remote end goes away. qemu will delay this many
>>>>>     seconds and then attempt to reconnect. Zero disables reconnecting,
>>>>>     and is the default.
>>>>>
>>>>> I'm not sure about exact reason.  It was historically this way.
>>>>> For me it doesn't make much sense.  I mean, your right that it's
>>>>> just a socket, so it should not matter who listens and who connects.
>>>>> If reconnection is possible in one direction, it should be possible
>>>>> in the opposite direction too.
>>>>
>>>> Sorry, my thought slipped. :)  Yes, QEMU supports re-connection
>>>> for client sockets.  So, in theory, dpdkvhostuser ports should work
>>>> after re-connection.  And that would be nice.  I don't remember
>>>> right now why this doesn't work...  Maybe vhost-user parts in QEMU
>>>> doesn't handle this case.  Need to dig some more into that and refresh
>>>> my memory.  It was so long ago...
>>>>
>>>> Maxime, do you remember?
>>>
>>> Sorry for the delay. I didn't remember, so I wanted to have a try.
>>>
>>> I can confirm reconnect works with QEMU as client and with Vhost PMD as
>>> server with:
>>>
>>>
>>>     <interface type='vhostuser'>
>>>       <mac address='56:48:4f:53:54:01'/>
>>>       <source type='unix' path='/tmp/vhost-user1' mode='client'>
>>>         <reconnect enabled='yes' timeout='1'/>
>>>       </source>
>>>       <model type='virtio'/>
>>>       <driver name='vhost' rx_queue_size='256'/>
>>>       <address type='pci' domain='0x0000' bus='0x07' slot='0x00'
>>> function='0x0'/>
>>>     </interface>
>>
>> Cool, thanks for checking. :)
>> If it works with vhost PMD, it should probably work with OVS too.
>>
>> There is still a couple of problems:
>>
>> 1. OpenStack Nova doesn't support configuration of a 'reconnect'
>>    in libvirt xml (it only adds the 'source'):
>>    https://github.com/openstack/nova/blob/master/nova/virt/libvirt/config.py#L1834
>>
>> 2. 'reconnect' configuration supported only starting from libvirt 4.1.0.
>>    It's released in 2018, but still some systems are using older versions.
>>    e.g. Ubuntu 18.04 which will be supported until 2023 uses libvirt 4.0.0.
> 
> Is it really a problem? We can keep using OVS as client for OSP,

Sure, no problem here.  I think, we scared enough users so they
mostly switched to this configuration and it's a good thing.

> and use OVS as server for containers.

There is a problem here that OVS emits a deprecation warning each
time you're trying to use dpdkvhostuser ports.  There are no problems
functionally, though, except that deprecation also implies that
we're consciously not adding new features to these ports.

Another thing is that current state of the art of k8s CNI requires
mounting of the whole /var/run/openvswitch or managing directories
by hands.  So, IMO, Broker idea is still valid is some form in terms
of hassle reduction for users.

> 
>>>
>>>>
>>>>>
>>>>> dpdkvhostuser was deprecated just to scare users and force them to
>>>>> migrate to dpdkvhostuserclient and avoid constant bug reports like:
>>>>>
>>>>>   "OVS service restarted and network is lost now".
>>>>>
>>>>> BTW, virtio-user ports in DPDK doesn't support re-connection in client
>>>>> mode too.
>>>>
>>>> This is still true, though.  virtio-user in client mode doesn't reconnect.
>>>
>>> That could be added, and it is maybe not as important for containers as
>>> it is for VM to support it, given the ephemeral nature of containers?
>>
>> Well, restart of OVS should not require restarting of all the containers
>> on the host even though they are "stateless".
>>
>> BTW, some infrastructure changes that I made in this series might be
>> reused to implement client-side reconnection for virtio-user.
> 
> Great, I'll look at the series when we'll work on adding reconnect for
> clients.

Please, take a look at the patch #1 in this set sooner.
It's a bug fix. :)

> 
> Thanks,
> Maxime
> 
>>>
>>> Regards,
>>> Maxime
>>>
>>>>
>>>>>
>>>>> BTW2, with SocketPair Broker it might be cheaper to implement server
>>>>> reconnection in QEMU because all connections in these case are client
>>>>> connections, i.e. both ends will connect() to a broker.
>>>>>
>>>>> Bets regards, Ilya Maximets.
>>>>>
>>>>
>>
>