Re: [dpdk-dev] [dpdk-techboard] A new bus for mediated devices

Re: [dpdk-dev] [dpdk-techboard] A new bus for mediated devices

[Alejandro Lucero]

Hi Steve,

On Tue, Jan 15, 2019 at 2:19 PM Liang, Cunming <cunming.liang@intel.com>
wrote:

> Hi Alejandro,
>
>
>
> Good to know we have common interest in DPDK native mdev support.
>
>
>
> We’re working on something which mdev based PMD driver is part of. It was
> going to collect others’ interest & feedback on DPDK summit before we start
> upstream effort.
>
>
Which DPDK summit do you refer to? the last one is Santa Jose in December?


>
>
> There was a few considerations.
>
> -          VT-d Spec 3.0 is publish, but no platform available to support
> even PCIe device might have the ability
>
> -          Except Intel, not sure other network IHVs is going to design
> their device by the new spec.
>
> -          w/o available platform, it only supports singleton mdev
> instance per parent device
>
> -          even in singleton mdev support, it requires IOMMU aware
> mediate device which is WIP in kernel
>
>
>

Yes, I know this is new stuff and it will not be usable as I have
previously commented by now, but I think this is going to be really
important in the near future. It adds a lot of flexibility for creating
ad-hoc net devices to be used by VMs.

In our initial case, we just need one mdev per parent device, and the IOMMU
mapping would be managed by the parent device after the proper ioctl call
from user space (NFP PMD for mediated device).


> For these reason, we hold on the upstream effort on DPDK side.
>
>
>

I understand. However, I think this should be discussed asap and to figure
out which is what is needed. When implementing the mdev bus for DPDK
myself, I found the mdev interface is so flexible (or maybe undefined), it
is not clear how it should be done.


> I’m actually quite interest in your use case, what’s the benefit you’re
> looking forward for kernel vfio mdev. If you don’t mind, could you share
> with us?
>
>
>

We need to use the PF and VFs in user space, this is DPDK, and the VF
creation is not possible when PF is bound to the VFIO driver (vfio-pci). Mu
idea is just to create a mediated device for allowing this, with the kernel
driver helping with mmaping the right BAR areas. After that, the PMD will
work almost as current NFP PMD, although certain things like link up/down
or getting extended stats will be through the kernel netdev.


> Our initial minimum goals to DPDK native mdev support,
>
> -          scan/probe/… kernel mdev bus sysfs
>
> -          keep consistent vfio uapi in DPDK
>
> -          reuse/unmodified any existing PMD previous built for pci bus
>
>
>

This last point seems quite complicated if not impossible, at least in our
case.


> We had patch set base on DPDK 18.05 and haven’t rebased yet to main
> stream, which includes
>
> -          intro new rte_mdev_bus for kernel mdev bus
>
> -          intro new rte_mdev_driver for ‘vfio-pci’ mdev type
>
> (allows to register other bus driver according to mdev type --
> ‘device_api’)
>
> -          whitelist & blacklist uuid support
>
> -          a pci vfio change to map resource according to general sysfs
>
>
>

Good. I have almost a mdev bus driver implemented and a specific NFP PMD
for a NFP mediated device. But I have been working for the shake of probing
this as an option for our purposes. Of course, my idea was to work on a
full mdev support for DPDK so that was the reason of my email to the
techboard.

Knowing you have been working on this longer than me, and likely having a
more complete implementation, I will not try to duplicate work here, and I
hope I can contribute to the final implementation once I see your design.

Thanks!


>
>
>
> Thanks,
>
> Steve
>
>
>
>
>
> *From:* techboard [mailto:techboard-bounces@dpdk.org
> <techboard-bounces@dpdk.org>] *On Behalf Of *Alejandro Lucero
> *Sent:* Monday, January 14, 2019 7:29 PM
> *To:* techboard@dpdk.org
> *Subject:* [dpdk-techboard] A new bus for mediated devices
>
>
>
> Hi all,
>
>
>
> I think there is none working on supporting mediated devices within DPDK.
> I am working on this for solving a requirement we have in Netronome but
> apart from that, I think it is something we are going to need in DPDK
> sooner or later.
>
>
>
> Because it is a really new interface and the way a mediated device can be
> created is really flexible, the proper way to support it should be broadly
> discussed. My plan is to send a RFC where the mdev bus is implemented along
> with a new Netronome's PMD supporting Netronome's mediated devices created
> by Netronome's kernel driver. Having an example of a mdev device will help.
>
>
>
> The reason for this email is twofold:
>
>
>
> 1) To be sure there is no other person working on supporting mdev inside
> the DPDK community, just for avoiding duplicate work. I found some
> presentations describing this interface in userspace  but I have found no
> patch related nor RFC regarding DPDK.
>
>
>
> 2) To inform the techboard about my intentions and to introduce the mdev
> interface for those not aware of it yet.
>
>
>
> If you consider it would be good to discuss this in next techboard
> meeting, it will be a pleasure to attend.
>
>
>
> Thanks
>
>
>
> (*)
> https://archive.fosdem.org/2018/schedule/event/netmdev/attachments/slides/2176/export/events/attachments/netmdev/slides/2176/net_mdev___fosdem_2018.pdf
>

Re: [dpdk-dev] [dpdk-techboard] A new bus for mediated devices

[Liang, Cunming]

Hi Alejandro,

From: Alejandro Lucero [mailto:alejandro.lucero@netronome.com]
Sent: Wednesday, January 16, 2019 1:59 AM
To: Liang, Cunming <cunming.liang@intel.com>; dev <dev@dpdk.org>
Cc: Richardson, Bruce <bruce.richardson@intel.com>; Lu, Xiuchun <xiuchun.lu@intel.com>
Subject: Re: [dpdk-techboard] A new bus for mediated devices

Hi Steve,

On Tue, Jan 15, 2019 at 2:19 PM Liang, Cunming <cunming.liang@intel.com<mailto:cunming.liang@intel.com>> wrote:
Hi Alejandro,

Good to know we have common interest in DPDK native mdev support.

We’re working on something which mdev based PMD driver is part of. It was going to collect others’ interest & feedback on DPDK summit before we start upstream effort.

Which DPDK summit do you refer to? the last one is Santa Jose in December?
[LC] Yes, it is. You can find it from the link https://schd.ws/hosted_files/dpdksummitnorthamerica2018/7b/DPDK_Summit18_MDEV_Fine-Grained-Slicing_Steve_John.pdf


There was a few considerations.

-          VT-d Spec 3.0 is publish, but no platform available to support even PCIe device might have the ability

-          Except Intel, not sure other network IHVs is going to design their device by the new spec.

-          w/o available platform, it only supports singleton mdev instance per parent device

-          even in singleton mdev support, it requires IOMMU aware mediate device which is WIP in kernel


Yes, I know this is new stuff and it will not be usable as I have previously commented by now, but I think this is going to be really important in the near future. It adds a lot of flexibility for creating ad-hoc net devices to be used by VMs.
[LC] Fully agree.

In our initial case, we just need one mdev per parent device, and the IOMMU mapping would be managed by the parent device after the proper ioctl call from user space (NFP PMD for mediated device).
[LC] I see, so essentially it’s singleton mdev instance base on IOMMU & SR-IOV platform. It requires mdev being capable to use parent device’ IOMMU domain, which does WIP. There’s no extra platform need by this usage, it’s good.

For these reason, we hold on the upstream effort on DPDK side.


I understand. However, I think this should be discussed asap and to figure out which is what is needed. When implementing the mdev bus for DPDK myself, I found the mdev interface is so flexible (or maybe undefined), it is not clear how it should be done.

I’m actually quite interest in your use case, what’s the benefit you’re looking forward for kernel vfio mdev. If you don’t mind, could you share with us?


We need to use the PF and VFs in user space, this is DPDK, and the VF creation is not possible when PF is bound to the VFIO driver (vfio-pci). Mu idea is just to create a mediated device for allowing this, with the kernel driver helping with mmaping the right BAR areas. After that, the PMD will work almost as current NFP PMD, although certain things like link up/down or getting extended stats will be through the kernel netdev.
[LC] Yeah, it separates device control and packet I/O, which is the most straight forward benefit introduced by mdev. It’s definitely a good usage as you mentioned.

Our initial minimum goals to DPDK native mdev support,

-          scan/probe/… kernel mdev bus sysfs

-          keep consistent vfio uapi in DPDK

-          reuse/unmodified any existing PMD previous built for pci bus


This last point seems quite complicated if not impossible, at least in our case.
[LC] That’s for case having exact the same device function but only being different on the granularity (e.g. number of queue-pairs). It sucks to have a duplicated PMD just for mdev bus.
It’s not your case, which is good to build from scratch a lightweight PMD and preserve the device control by kernel.

We had patch set base on DPDK 18.05 and haven’t rebased yet to main stream, which includes

-          intro new rte_mdev_bus for kernel mdev bus

-          intro new rte_mdev_driver for ‘vfio-pci’ mdev type

(allows to register other bus driver according to mdev type -- ‘device_api’)

-          whitelist & blacklist uuid support

-          a pci vfio change to map resource according to general sysfs


Good. I have almost a mdev bus driver implemented and a specific NFP PMD for a NFP mediated device. But I have been working for the shake of probing this as an option for our purposes. Of course, my idea was to work on a full mdev support for DPDK so that was the reason of my email to the techboard.
[LC] The goal is fully aligned. Mdev is much easier to manage the device lifecycle, is able to support different bus layout (e.g. pci, platform, ccw) and etc. We’d like DPDK mdev enabling preserve most of the benefits.


Knowing you have been working on this longer than me, and likely having a more complete implementation, I will not try to duplicate work here, and I hope I can contribute to the final implementation once I see your design.
[LC] That’s great. We’ll initialize a RFC, your input from different view would be really helpful, looking forward to the collaboration.

Thanks!


Thanks!



Thanks,
Steve


From: techboard [mailto:techboard-bounces@dpdk.org] On Behalf Of Alejandro Lucero
Sent: Monday, January 14, 2019 7:29 PM
To: techboard@dpdk.org<mailto:techboard@dpdk.org>
Subject: [dpdk-techboard] A new bus for mediated devices

Hi all,

I think there is none working on supporting mediated devices within DPDK. I am working on this for solving a requirement we have in Netronome but apart from that, I think it is something we are going to need in DPDK sooner or later.

Because it is a really new interface and the way a mediated device can be created is really flexible, the proper way to support it should be broadly discussed. My plan is to send a RFC where the mdev bus is implemented along with a new Netronome's PMD supporting Netronome's mediated devices created by Netronome's kernel driver. Having an example of a mdev device will help.

The reason for this email is twofold:

1) To be sure there is no other person working on supporting mdev inside the DPDK community, just for avoiding duplicate work. I found some presentations describing this interface in userspace  but I have found no patch related nor RFC regarding DPDK.

2) To inform the techboard about my intentions and to introduce the mdev interface for those not aware of it yet.

If you consider it would be good to discuss this in next techboard meeting, it will be a pleasure to attend.

Thanks

(*) https://archive.fosdem.org/2018/schedule/event/netmdev/attachments/slides/2176/export/events/attachments/netmdev/slides/2176/net_mdev___fosdem_2018.pdf

Re: [dpdk-dev] [dpdk-techboard] A new bus for mediated devices

[Liang, Cunming]

Resend in plain text.

-----Original Message-----
 From: Liang, Cunming
 Sent: Wednesday, January 16, 2019 6:49 PM
 To: Alejandro Lucero <alejandro.lucero@netronome.com>; dev <dev@dpdk.org>
 Cc: Richardson, Bruce <bruce.richardson@intel.com>; Lu, Xiuchun
 <xiuchun.lu@intel.com>
 Subject: RE: [dpdk-techboard] A new bus for mediated devices
 
Hi Alejandro,

> 
> From: Alejandro Lucero [mailto:alejandro.lucero@netronome.com]
> Sent: Wednesday, January 16, 2019 1:59 AM
> To: Liang, Cunming <cunming.liang@intel.com>; dev <dev@dpdk.org>
> Cc: Richardson, Bruce <bruce.richardson@intel.com>; Lu, Xiuchun
> <xiuchun.lu@intel.com>
> Subject: Re: [dpdk-techboard] A new bus for mediated devices
> 
> Hi Steve,
> 
>> On Tue, Jan 15, 2019 at 2:19 PM Liang, Cunming <cunming.liang@intel.com> wrote:
>> Hi Alejandro,
>> 
>> Good to know we have common interest in DPDK native mdev support.
>> 
>> We’re working on something which mdev based PMD driver is part of. It was going
>> to collect others’ interest & feedback on DPDK summit before we start upstream
>> effort.
> 
> Which DPDK summit do you refer to? the last one is Santa Jose in December?
[LC] Yes, it is. You can find it from the link 
https://schd.ws/hosted_files/dpdksummitnorthamerica2018/7b/DPDK_Summit18_MDEV_Fine-Grained-Slicing_Steve_John.pdf
 
> 
>> There was a few considerations.
>> -          VT-d Spec 3.0 is publish, but no platform available to support even PCIe device
>> might have the ability
>> -          Except Intel, not sure other network IHVs is going to design their device by the
>> new spec.
>> -          w/o available platform, it only supports singleton mdev instance per parent
>> device
>> -          even in singleton mdev support, it requires IOMMU aware mediate device
>> which is WIP in kernel
> >
> Yes, I know this is new stuff and it will not be usable as I have previously commented
> by now, but I think this is going to be really important in the near future. It adds a lot
> of flexibility for creating ad-hoc net devices to be used by VMs.
[LC] Fully agree.
 
> In our initial case, we just need one mdev per parent device, and the IOMMU
> mapping would be managed by the parent device after the proper ioctl call from
> user space (NFP PMD for mediated device).
[LC] I see, so essentially it’s singleton mdev instance base on IOMMU & SR-IOV
platform. It requires mdev being capable to use parent device’ IOMMU domain,
which does WIP. There’s no extra platform need by this usage, it’s good.

>> 
>> For these reason, we hold on the upstream effort on DPDK side.
>> 
> I understand. However, I think this should be discussed asap and to figure out which
> is what is needed. When implementing the mdev bus for DPDK myself, I found the
> mdev interface is so flexible (or maybe undefined), it is not clear how it should be
> done.
> 
>> I’m actually quite interest in your use case, what’s the benefit you’re looking
>> forward for kernel vfio mdev. If you don’t mind, could you share with us?
> 
> We need to use the PF and VFs in user space, this is DPDK, and the VF creation is not
> possible when PF is bound to the VFIO driver (vfio-pci). Mu idea is just to create a
> mediated device for allowing this, with the kernel driver helping with mmaping the
> right BAR areas. After that, the PMD will work almost as current NFP PMD, although
> certain things like link up/down or getting extended stats will be through the kernel
> netdev.
[LC] Yeah, it separates device control and packet I/O, which is the most straight
forward benefit introduced by mdev. It’s definitely a good usage as you mentioned.

>> 
>> Our initial minimum goals to DPDK native mdev support,
>> -          scan/probe/… kernel mdev bus sysfs
>> -          keep consistent vfio uapi in DPDK
>> -          reuse/unmodified any existing PMD previous built for pci bus
>> 
> This last point seems quite complicated if not impossible, at least in our case.
[LC] That’s for case having exact the same device function but only being different on
the granularity (e.g. number of queue-pairs). It sucks to have a duplicated PMD just
for mdev bus.
It’s not your case, which is good to build from scratch a lightweight PMD and
preserve the device control by kernel.

>> 
>> We had patch set base on DPDK 18.05 and haven’t rebased yet to main stream,
>> which includes
>> -          intro new rte_mdev_bus for kernel mdev bus
>> -          intro new rte_mdev_driver for ‘vfio-pci’ mdev type
>> (allows to register other bus driver according to mdev type -- ‘device_api’)
>> -          whitelist & blacklist uuid support
>> -          a pci vfio change to map resource according to general sysfs
>> 
>> 
> Good. I have almost a mdev bus driver implemented and a specific NFP PMD for a
> NFP mediated device. But I have been working for the shake of probing this as an
> option for our purposes. Of course, my idea was to work on a full mdev support for
> DPDK so that was the reason of my email to the techboard.
[LC] The goal is fully aligned. Mdev is much easier to manage the device lifecycle, is
able to support different bus layout (e.g. pci, platform, ccw) and etc. We’d like DPDK
mdev enabling preserve most of the benefits.
 
> 
> Knowing you have been working on this longer than me, and likely having a more
> complete implementation, I will not try to duplicate work here, and I hope I can
> contribute to the final implementation once I see your design.
[LC] That’s great. We’ll initialize a RFC, your input from different view would be
really helpful, looking forward to the collaboration.

Thanks!
> 
> 
> Thanks!
> 
> 
>> 
>> Thanks,
>> Steve
>>> 
>>> 
>>> From: techboard [mailto:techboard-bounces@dpdk.org] On Behalf Of Alejandro
>>> Lucero
>>> Sent: Monday, January 14, 2019 7:29 PM
>>> To: techboard@dpdk.org
>>> Subject: [dpdk-techboard] A new bus for mediated devices
>>> 
>>> Hi all,
>>> 
>>> I think there is none working on supporting mediated devices within DPDK. I am
>>> working on this for solving a requirement we have in Netronome but apart from
>>> that, I think it is something we are going to need in DPDK sooner or later.
>>> 
>>> Because it is a really new interface and the way a mediated device can be created is
>>> really flexible, the proper way to support it should be broadly discussed. My plan is
>>> to send a RFC where the mdev bus is implemented along with a new Netronome's
>>> PMD supporting Netronome's mediated devices created by Netronome's kernel
>>> driver. Having an example of a mdev device will help.
>>> 
>>> The reason for this email is twofold:
>>> 
>>> 1) To be sure there is no other person working on supporting mdev inside the DPDK
>>> community, just for avoiding duplicate work. I found some presentations describing
>>> this interface in userspace  but I have found no patch related nor RFC regarding
>>> DPDK.
>>> 
>>> 2) To inform the techboard about my intentions and to introduce the mdev interface
>>> for those not aware of it yet.
>>> 
>>> If you consider it would be good to discuss this in next techboard meeting, it will be a
>>> pleasure to attend.
>>> 
>>> Thanks
>>> 
>>> (*) https://archive.fosdem.org/2018/schedule/event/netmdev/attachments/slides/
>>> 2176/export/events/attachments/netmdev/slides/2176/net_mdev___fosdem_201
>>> 8.pdf