[dpdk-dev] cannot use virtio-user pmd in IOVA=PA mode anymore

[dpdk-dev] cannot use virtio-user pmd in IOVA=PA mode anymore

[Olivier Matz]

Hello,

I recently tested a use-case with our application using the main
branch of dpdk.org.

- the application runs inside a standard x86 VM (no IOMMU)
- there are emulated physical NICs inside the VM
- we use virtio-user pmds connected to tap interfaces through the
  vhost-net backend for exception path

This use-case works with the the stable 20.11 branch, but not with the
current main dpdk.org branch. The virtio-user driver refuses to start in
IOVA=PA mode:

  vdev_probe_all_drivers(): virtio_user0 requires VA IOVA mode but current mode is PA, not initializing
  EAL: Driver cannot attach the device (virtio_user0)

This is likely due to these commits:

  8d935fff5546 ("bus/vdev: add driver IOVA VA mode requirement")
  17043a2909bb ("net/virtio: force IOVA as VA mode for virtio-user")

We didn't see this problem before because we are only testing dpdk.org
main branch on physical machines that have an IOMMU.

I'm not sure to understand the reasons for which the ability to run a
virtio-user pmd in IOVA=PA mode was removed.

The commitlog of 17043a2909bb says:

  At least Vhost-user backend of Virtio-user PMD requires
  IOVA as VA mode. Until now, it was implemented as a hack
  by forcing to use mbuf's buf_addr field instead of buf_iova.

I don't get why vhost-user backend would require IOVA=VA. Note that we
also have a use-case where a virtio-user pmd is connected to our
pmd-vhost in another application, which was working with 20.11.

If there is a constraint with vhost-user backend, what about vhost-net
backend?

Would it make sense to re-allow this feature by somehow reverting
17043a2909bb with some additional cleanup?

I am aware that a solution can be to configure qemu to enable a vIOMMU,
but it is not my preferred solution yet, as it would impact our users
that do not do this currently.

Thanks!
Olivier

Re: [dpdk-dev] cannot use virtio-user pmd in IOVA=PA mode anymore

[Maxime Coquelin]

Hi Olivier,

On 9/24/21 10:21, Olivier Matz wrote:
> Hello,
> 
> I recently tested a use-case with our application using the main
> branch of dpdk.org.
> 
> - the application runs inside a standard x86 VM (no IOMMU)
> - there are emulated physical NICs inside the VM
> - we use virtio-user pmds connected to tap interfaces through the
>    vhost-net backend for exception path
> 
> This use-case works with the the stable 20.11 branch, but not with the
> current main dpdk.org branch. The virtio-user driver refuses to start in
> IOVA=PA mode:
> 
>    vdev_probe_all_drivers(): virtio_user0 requires VA IOVA mode but current mode is PA, not initializing
>    EAL: Driver cannot attach the device (virtio_user0)
> 
> This is likely due to these commits:
> 
>    8d935fff5546 ("bus/vdev: add driver IOVA VA mode requirement")
>    17043a2909bb ("net/virtio: force IOVA as VA mode for virtio-user")
> 
> We didn't see this problem before because we are only testing dpdk.org
> main branch on physical machines that have an IOMMU.
> 
> I'm not sure to understand the reasons for which the ability to run a
> virtio-user pmd in IOVA=PA mode was removed.
> 
> The commitlog of 17043a2909bb says:
> 
>    At least Vhost-user backend of Virtio-user PMD requires
>    IOVA as VA mode. Until now, it was implemented as a hack
>    by forcing to use mbuf's buf_addr field instead of buf_iova.
> 
> I don't get why vhost-user backend would require IOVA=VA. Note that we
> also have a use-case where a virtio-user pmd is connected to our
> pmd-vhost in another application, which was working with 20.11.
> 
> If there is a constraint with vhost-user backend, what about vhost-net
> backend?
> 
> Would it make sense to re-allow this feature by somehow reverting
> 17043a2909bb with some additional cleanup?

I think so.

> I am aware that a solution can be to configure qemu to enable a vIOMMU,
> but it is not my preferred solution yet, as it would impact our users
> that do not do this currently.

The initial purpose of this change was to simplify the code, I did not 
thought about this adverse side-effect.

While moving to VFIO with IOMMU enabled would be a wise decision :), I
will work on reverting that patch to restore the old behavior as that
was not something that was agreed when the patch was submitted.

Thanks for reporting the issue,
Maxime

> Thanks!
> Olivier
>