Re: [dpdk-dev] [PATCH 03/21] vhost: protect virtio_net device struct

[Tiwei Bie <tiwei.bie@intel.com>]

On Thu, Aug 31, 2017 at 11:50:05AM +0200, Maxime Coquelin wrote:
> virtio_net device might be accessed while being reallocated
> in case of NUMA awareness. This case might be theoretical,
> but it will be needed anyway to protect vrings pages against
> invalidation.
> 
> The virtio_net devs are now protected with a readers/writers
> lock, so that before reallocating the device, it is ensured
> that it is not being referenced by the processing threads.
> 
[...]
>  
> +struct virtio_net *
> +get_device(int vid)
> +{
> +	struct virtio_net *dev;
> +
> +	rte_rwlock_read_lock(&vhost_devices[vid].lock);
> +
> +	dev = __get_device(vid);
> +	if (unlikely(!dev))
> +		rte_rwlock_read_unlock(&vhost_devices[vid].lock);
> +
> +	return dev;
> +}
> +
> +void
> +put_device(int vid)
> +{
> +	rte_rwlock_read_unlock(&vhost_devices[vid].lock);
> +}
> +

This patch introduced a per-device rwlock which needs to be acquired
unconditionally in the data path. So for each vhost device, the IO
threads of different queues will need to acquire/release this lock
during each enqueue and dequeue operation, which will cause cache
contention when multiple queues are enabled and handled by different
cores. With this patch alone, I saw ~7% performance drop when enabling
6 queues to do 64bytes iofwd loopback test. Is there any way to avoid
introducing this lock to the data path?

Best regards,
Tiwei Bie