Re: [dpdk-dev] vhost compliant virtio based networking interface in container

Re: [dpdk-dev] vhost compliant virtio based networking interface in container

[Xie, Huawei]

Added dev@dpdk.org

On 8/20/2015 6:04 PM, Xie, Huawei wrote:
> Yanping:
> I read your mail, seems what we did are quite similar. Here i wrote a
> quick mail to describe our design. Let me know if it is the same thing.
>
> Problem Statement:
> We don't have a high performance networking interface in container for
> NFV. Current veth pair based interface couldn't be easily accelerated.
>
> The key components involved:
>     1.    DPDK based virtio PMD driver in container.
>     2.    device simulation framework in container.
>     3.    dpdk(or kernel) vhost running in host.
>
> How virtio is created?
> A:  There is no "real" virtio-pci device in container environment.
> 1). Host maintains pools of memories, and shares memory to container.
> This could be accomplished through host share a huge page file to container.
> 2). Containers creates virtio rings based on the shared memory.
> 3). Container creates mbuf memory pools on the shared memory.
> 4) Container send the memory and vring information to vhost through
> vhost message. This could be done either through ioctl call or vhost
> user message.
>
> How vhost message is sent?
> A: There are two alternative ways to do this.
> 1) The customized virtio PMD is responsible for all the vring creation,
> and vhost message sending.
> 2) We could do this through a lightweight device simulation framework.
>     The device simulation creates simple PCI bus. On the PCI bus,
> virtio-net PCI devices are created. The device simulations provides
> IOAPI for MMIO/IO access.
>    2.1  virtio PMD configures the pseudo virtio device as how it does in
> KVM guest enviroment.
>    2.2  Rather than using io instruction, virtio PMD uses IOAPI for IO
> operation on the virtio-net PCI device.
>    2.3  The device simulation is responsible for device state machine
> simulation.
>    2.4   The device simulation is responsbile for talking to vhost.
>      With this approach, we could minimize the virtio PMD modifications.
> The virtio PMD is like configuring a real virtio-net PCI device.
>
> Memory mapping?
> A: QEMU could access the whole guest memory in KVM enviroment. We need
> to fill the gap.
> container maps the shared memory to container's virtual address space
> and host maps it to host's virtual address space. There is a fixed
> offset mapping.
> Container creates shared vring based on the memory. Container also
> creates mbuf memory pool based on the shared memroy.
> In VHOST_SET_MEMORY_TABLE message, we send the memory mapping
> information for the shared memory. As we require mbuf pool created on
> the shared memory, and buffers are allcoated from the mbuf pools, dpdk
> vhost could translate the GPA in vring desc to host virtual.
>
>
> GPA or CVA in vring desc?
> To ease the memory translation, rather than using GPA, here we use
> CVA(container virtual address). This the tricky thing here.
> 1) virtio PMD writes vring's VFN rather than PFN to PFN register through
> IOAPI.
> 2) device simulation framework will use VFN as PFN.
> 3) device simulation sends SET_VRING_ADDR with CVA.
> 4) virtio PMD fills vring desc with CVA of the mbuf data pointer rather
> than GPA.
> So when host sees the CVA, it could translates it to HVA(host virtual
> address).
>
> Worth to note:
> The virtio interface in container follows the vhost message format, and
> is compliant with dpdk vhost implmentation, i.e, no dpdk vhost
> modification is needed.
> vHost isn't aware whether the incoming virtio comes from KVM guest or
> container.
>
> The pretty much covers the high level design. There are quite some low
> level issues. For example, 32bit PFN is enough for KVM guest, since we
> use 64bit VFN(virtual page frame number),  trick is done here through a
> special IOAPI.
>
> /huawei
>
>  
>
>
>
>
>
>

Re: [dpdk-dev] vhost compliant virtio based networking interface in container

[Tetsuya Mukawa]

Hi Xie and Yanping,


May I ask you some questions?
It seems we are also developing an almost same one.

On 2015/08/20 19:14, Xie, Huawei wrote:
> Added dev@dpdk.org
>
> On 8/20/2015 6:04 PM, Xie, Huawei wrote:
>> Yanping:
>> I read your mail, seems what we did are quite similar. Here i wrote a
>> quick mail to describe our design. Let me know if it is the same thing.
>>
>> Problem Statement:
>> We don't have a high performance networking interface in container for
>> NFV. Current veth pair based interface couldn't be easily accelerated.
>>
>> The key components involved:
>>     1.    DPDK based virtio PMD driver in container.
>>     2.    device simulation framework in container.
>>     3.    dpdk(or kernel) vhost running in host.
>>
>> How virtio is created?
>> A:  There is no "real" virtio-pci device in container environment.
>> 1). Host maintains pools of memories, and shares memory to container.
>> This could be accomplished through host share a huge page file to container.
>> 2). Containers creates virtio rings based on the shared memory.
>> 3). Container creates mbuf memory pools on the shared memory.
>> 4) Container send the memory and vring information to vhost through
>> vhost message. This could be done either through ioctl call or vhost
>> user message.
>>
>> How vhost message is sent?
>> A: There are two alternative ways to do this.
>> 1) The customized virtio PMD is responsible for all the vring creation,
>> and vhost message sending.

Above is our approach so far.
It seems Yanping also takes this kind of approach.
We are using vhost-user functionality instead of using the vhost-net
kernel module.
Probably this is the difference between Yanping and us.

BTW, we are going to submit a vhost PMD for DPDK-2.2.
This PMD is implemented on librte_vhost.
It allows DPDK application to handle a vhost-user(cuse) backend as a
normal NIC port.
This PMD should work with both Xie and Yanping approach.
(In the case of Yanping approach, we may need vhost-cuse)

>> 2) We could do this through a lightweight device simulation framework.
>>     The device simulation creates simple PCI bus. On the PCI bus,
>> virtio-net PCI devices are created. The device simulations provides
>> IOAPI for MMIO/IO access.

Does it mean you implemented a kernel module?
If so, do you still need vhost-cuse functionality to handle vhost
messages n userspace?

>>    2.1  virtio PMD configures the pseudo virtio device as how it does in
>> KVM guest enviroment.
>>    2.2  Rather than using io instruction, virtio PMD uses IOAPI for IO
>> operation on the virtio-net PCI device.
>>    2.3  The device simulation is responsible for device state machine
>> simulation.
>>    2.4   The device simulation is responsbile for talking to vhost.
>>      With this approach, we could minimize the virtio PMD modifications.
>> The virtio PMD is like configuring a real virtio-net PCI device.
>>
>> Memory mapping?
>> A: QEMU could access the whole guest memory in KVM enviroment. We need
>> to fill the gap.
>> container maps the shared memory to container's virtual address space
>> and host maps it to host's virtual address space. There is a fixed
>> offset mapping.
>> Container creates shared vring based on the memory. Container also
>> creates mbuf memory pool based on the shared memroy.
>> In VHOST_SET_MEMORY_TABLE message, we send the memory mapping
>> information for the shared memory. As we require mbuf pool created on
>> the shared memory, and buffers are allcoated from the mbuf pools, dpdk
>> vhost could translate the GPA in vring desc to host virtual.
>>
>>
>> GPA or CVA in vring desc?
>> To ease the memory translation, rather than using GPA, here we use
>> CVA(container virtual address). This the tricky thing here.
>> 1) virtio PMD writes vring's VFN rather than PFN to PFN register through
>> IOAPI.
>> 2) device simulation framework will use VFN as PFN.
>> 3) device simulation sends SET_VRING_ADDR with CVA.
>> 4) virtio PMD fills vring desc with CVA of the mbuf data pointer rather
>> than GPA.
>> So when host sees the CVA, it could translates it to HVA(host virtual
>> address).
>>
>> Worth to note:
>> The virtio interface in container follows the vhost message format, and
>> is compliant with dpdk vhost implmentation, i.e, no dpdk vhost
>> modification is needed.
>> vHost isn't aware whether the incoming virtio comes from KVM guest or
>> container.
>>
>> The pretty much covers the high level design. There are quite some low
>> level issues. For example, 32bit PFN is enough for KVM guest, since we
>> use 64bit VFN(virtual page frame number),  trick is done here through a
>> special IOAPI.

In addition above, we might consider "namespace" kernel functionality.
Technically, it would not be a big problem, but related with security.
So it would be nice to take account.

Regards,
Tetsuya

>> /huawei
>>
>>  
>>
>>
>>
>>
>>
>>

Re: [dpdk-dev] vhost compliant virtio based networking interface in container

[Xie, Huawei]

On 8/25/2015 10:59 AM, Tetsuya Mukawa wrote:
> Hi Xie and Yanping,
>
>
> May I ask you some questions?
> It seems we are also developing an almost same one.

Good to know that we are tackling the same problem and have the similar
idea.
What is your status now? We had the POC running, and compliant with
dpdkvhost.
Interrupt like notification isn't supported.

>
> On 2015/08/20 19:14, Xie, Huawei wrote:
>> Added dev@dpdk.org
>>
>> On 8/20/2015 6:04 PM, Xie, Huawei wrote:
>>> Yanping:
>>> I read your mail, seems what we did are quite similar. Here i wrote a
>>> quick mail to describe our design. Let me know if it is the same thing.
>>>
>>> Problem Statement:
>>> We don't have a high performance networking interface in container for
>>> NFV. Current veth pair based interface couldn't be easily accelerated.
>>>
>>> The key components involved:
>>>     1.    DPDK based virtio PMD driver in container.
>>>     2.    device simulation framework in container.
>>>     3.    dpdk(or kernel) vhost running in host.
>>>
>>> How virtio is created?
>>> A:  There is no "real" virtio-pci device in container environment.
>>> 1). Host maintains pools of memories, and shares memory to container.
>>> This could be accomplished through host share a huge page file to container.
>>> 2). Containers creates virtio rings based on the shared memory.
>>> 3). Container creates mbuf memory pools on the shared memory.
>>> 4) Container send the memory and vring information to vhost through
>>> vhost message. This could be done either through ioctl call or vhost
>>> user message.
>>>
>>> How vhost message is sent?
>>> A: There are two alternative ways to do this.
>>> 1) The customized virtio PMD is responsible for all the vring creation,
>>> and vhost message sending.
> Above is our approach so far.
> It seems Yanping also takes this kind of approach.
> We are using vhost-user functionality instead of using the vhost-net
> kernel module.
> Probably this is the difference between Yanping and us.

In my current implementation, the device simulation layer talks to "user
space" vhost through cuse interface. It could also be done through vhost
user socket. This isn't the key point.
Here vhost-user is kind of confusing, maybe user space vhost is more
accurate, either cuse or unix domain socket. :).

As for yanping, they are now connecting to vhost-net kernel module, but
they are also trying to connect to "user space" vhost.  Correct me if wrong.
Yes, there is some difference between these two. Vhost-net kernel module
could directly access other process's memory, while using
vhost-user(cuse/user), we need do the memory mapping.
>
> BTW, we are going to submit a vhost PMD for DPDK-2.2.
> This PMD is implemented on librte_vhost.
> It allows DPDK application to handle a vhost-user(cuse) backend as a
> normal NIC port.
> This PMD should work with both Xie and Yanping approach.
> (In the case of Yanping approach, we may need vhost-cuse)
>
>>> 2) We could do this through a lightweight device simulation framework.
>>>     The device simulation creates simple PCI bus. On the PCI bus,
>>> virtio-net PCI devices are created. The device simulations provides
>>> IOAPI for MMIO/IO access.
> Does it mean you implemented a kernel module?
> If so, do you still need vhost-cuse functionality to handle vhost
> messages n userspace?

The device simulation is  a library running in user space in container. 
It is linked with DPDK app. It creates pseudo buses and virtio-net PCI
devices.
The virtio-container-PMD configures the virtio-net pseudo devices
through IOAPI provided by the device simulation rather than IO
instructions as in KVM.
Why we use device simulation?
We could create other virtio devices in container, and provide an common
way to talk to vhost-xx module.

>>>    2.1  virtio PMD configures the pseudo virtio device as how it does in
>>> KVM guest enviroment.
>>>    2.2  Rather than using io instruction, virtio PMD uses IOAPI for IO
>>> operation on the virtio-net PCI device.
>>>    2.3  The device simulation is responsible for device state machine
>>> simulation.
>>>    2.4   The device simulation is responsbile for talking to vhost.
>>>      With this approach, we could minimize the virtio PMD modifications.
>>> The virtio PMD is like configuring a real virtio-net PCI device.
>>>
>>> Memory mapping?
>>> A: QEMU could access the whole guest memory in KVM enviroment. We need
>>> to fill the gap.
>>> container maps the shared memory to container's virtual address space
>>> and host maps it to host's virtual address space. There is a fixed
>>> offset mapping.
>>> Container creates shared vring based on the memory. Container also
>>> creates mbuf memory pool based on the shared memroy.
>>> In VHOST_SET_MEMORY_TABLE message, we send the memory mapping
>>> information for the shared memory. As we require mbuf pool created on
>>> the shared memory, and buffers are allcoated from the mbuf pools, dpdk
>>> vhost could translate the GPA in vring desc to host virtual.
>>>
>>>
>>> GPA or CVA in vring desc?
>>> To ease the memory translation, rather than using GPA, here we use
>>> CVA(container virtual address). This the tricky thing here.
>>> 1) virtio PMD writes vring's VFN rather than PFN to PFN register through
>>> IOAPI.
>>> 2) device simulation framework will use VFN as PFN.
>>> 3) device simulation sends SET_VRING_ADDR with CVA.
>>> 4) virtio PMD fills vring desc with CVA of the mbuf data pointer rather
>>> than GPA.
>>> So when host sees the CVA, it could translates it to HVA(host virtual
>>> address).
>>>
>>> Worth to note:
>>> The virtio interface in container follows the vhost message format, and
>>> is compliant with dpdk vhost implmentation, i.e, no dpdk vhost
>>> modification is needed.
>>> vHost isn't aware whether the incoming virtio comes from KVM guest or
>>> container.
>>>
>>> The pretty much covers the high level design. There are quite some low
>>> level issues. For example, 32bit PFN is enough for KVM guest, since we
>>> use 64bit VFN(virtual page frame number),  trick is done here through a
>>> special IOAPI.
> In addition above, we might consider "namespace" kernel functionality.
> Technically, it would not be a big problem, but related with security.
> So it would be nice to take account.

There is no namespace concept here because we don't generate kernel
netdev devices. It might be usefull if we could extend our work to
support kernel netdev interface and assign to container's namespace.

>
> Regards,
> Tetsuya
>
>>> /huawei
>>>
>>>  
>>>
>>>
>>>
>>>
>>>
>>>
>

Re: [dpdk-dev] vhost compliant virtio based networking interface in container

[Tetsuya Mukawa]

On 2015/08/25 18:56, Xie, Huawei wrote:
> On 8/25/2015 10:59 AM, Tetsuya Mukawa wrote:
>> Hi Xie and Yanping,
>>
>>
>> May I ask you some questions?
>> It seems we are also developing an almost same one.
> Good to know that we are tackling the same problem and have the similar
> idea.
> What is your status now? We had the POC running, and compliant with
> dpdkvhost.
> Interrupt like notification isn't supported.

We implemented vhost PMD first, so we just start implementing it.

>
>> On 2015/08/20 19:14, Xie, Huawei wrote:
>>> Added dev@dpdk.org
>>>
>>> On 8/20/2015 6:04 PM, Xie, Huawei wrote:
>>>> Yanping:
>>>> I read your mail, seems what we did are quite similar. Here i wrote a
>>>> quick mail to describe our design. Let me know if it is the same thing.
>>>>
>>>> Problem Statement:
>>>> We don't have a high performance networking interface in container for
>>>> NFV. Current veth pair based interface couldn't be easily accelerated.
>>>>
>>>> The key components involved:
>>>>     1.    DPDK based virtio PMD driver in container.
>>>>     2.    device simulation framework in container.
>>>>     3.    dpdk(or kernel) vhost running in host.
>>>>
>>>> How virtio is created?
>>>> A:  There is no "real" virtio-pci device in container environment.
>>>> 1). Host maintains pools of memories, and shares memory to container.
>>>> This could be accomplished through host share a huge page file to container.
>>>> 2). Containers creates virtio rings based on the shared memory.
>>>> 3). Container creates mbuf memory pools on the shared memory.
>>>> 4) Container send the memory and vring information to vhost through
>>>> vhost message. This could be done either through ioctl call or vhost
>>>> user message.
>>>>
>>>> How vhost message is sent?
>>>> A: There are two alternative ways to do this.
>>>> 1) The customized virtio PMD is responsible for all the vring creation,
>>>> and vhost message sending.
>> Above is our approach so far.
>> It seems Yanping also takes this kind of approach.
>> We are using vhost-user functionality instead of using the vhost-net
>> kernel module.
>> Probably this is the difference between Yanping and us.
> In my current implementation, the device simulation layer talks to "user
> space" vhost through cuse interface. It could also be done through vhost
> user socket. This isn't the key point.
> Here vhost-user is kind of confusing, maybe user space vhost is more
> accurate, either cuse or unix domain socket. :).
>
> As for yanping, they are now connecting to vhost-net kernel module, but
> they are also trying to connect to "user space" vhost.  Correct me if wrong.
> Yes, there is some difference between these two. Vhost-net kernel module
> could directly access other process's memory, while using
> vhost-user(cuse/user), we need do the memory mapping.
>> BTW, we are going to submit a vhost PMD for DPDK-2.2.
>> This PMD is implemented on librte_vhost.
>> It allows DPDK application to handle a vhost-user(cuse) backend as a
>> normal NIC port.
>> This PMD should work with both Xie and Yanping approach.
>> (In the case of Yanping approach, we may need vhost-cuse)
>>
>>>> 2) We could do this through a lightweight device simulation framework.
>>>>     The device simulation creates simple PCI bus. On the PCI bus,
>>>> virtio-net PCI devices are created. The device simulations provides
>>>> IOAPI for MMIO/IO access.
>> Does it mean you implemented a kernel module?
>> If so, do you still need vhost-cuse functionality to handle vhost
>> messages n userspace?
> The device simulation is  a library running in user space in container. 
> It is linked with DPDK app. It creates pseudo buses and virtio-net PCI
> devices.
> The virtio-container-PMD configures the virtio-net pseudo devices
> through IOAPI provided by the device simulation rather than IO
> instructions as in KVM.
> Why we use device simulation?
> We could create other virtio devices in container, and provide an common
> way to talk to vhost-xx module.

Thanks for explanation.
At first reading, I thought the difference between approach1 and
approach2 is whether we need to implement a new kernel module, or not.
But I understand how you implemented.

Please let me explain our design more.
We might use a kind of similar approach to handle a pseudo virtio-net
device in DPDK.
(Anyway, we haven't finished implementing yet, this overview might have
some technical problems)

Step1. Separate virtio-net and vhost-user socket related code from QEMU,
then implement it as a separated program.
The program also has below features.
 - Create a directory that contains almost same files like
/sys/bus/pci/device/<pci address>/*
   (To scan these file located on outside sysfs, we need to fix EAL)
 - This dummy device is driven by dummy-virtio-net-driver. This name is
specified by '<pci addr>/driver' file.
 - Create a shared file that represents pci configuration space, then
mmap it, also specify the path in '<pci addr>/resource_path'

The program will be GPL, but it will be like a bridge on the shared
memory between virtio-net PMD and DPDK vhost backend.
Actually, It will work under virtio-net PMD, but we don't need to link it.
So I guess we don't have GPL license issue.

Step2. Fix pci scan code of EAL to scan dummy devices.
 - To scan above files, extend pci_scan() of EAL.

Step3. Add a new kdrv type to EAL.
 - To handle the 'dummy-virtio-net-driver', add a new kdrv type to EAL.

Step4. Implement pci_dummy_virtio_net_map/unmap().
 - It will have almost same functionality like pci_uio_map(), but for
dummy virtio-net device.
 - The dummy device will be mmaped using a path specified in '<pci
addr>/resource_path'.

Step5. Add a new compile option for virtio-net device to replace IO
functions.
 - The IO functions of virtio-net PMD will be replaced by read() and
write() to access to the shared memory.
 - Add notification mechanism to IO functions. This will be used when
write() to the shared memory is done.
 (Not sure exactly, but probably we need it)

Does it make sense?
I guess Step1&2 is different from your approach, but the rest might be
similar.

Actually, we just need sysfs entries for a virtio-net dummy device, but
so far, I don't have a fine way to register them from user space without
loading a kernel module.
This is because I need to change pci_scan() also.

It seems you have implemented a virtio-net pseudo device as BSD license.
If so, this kind of PMD would be nice to use it.
In the case that it takes much time to implement some lost
functionalities like interrupt mode, using QEMU code might be an one of
options.

Anyway, we just need a fine virtual NIC between containers and host.
So we don't hold to our approach and implementation.

Thanks,
Tetsuya

>>>>    2.1  virtio PMD configures the pseudo virtio device as how it does in
>>>> KVM guest enviroment.
>>>>    2.2  Rather than using io instruction, virtio PMD uses IOAPI for IO
>>>> operation on the virtio-net PCI device.
>>>>    2.3  The device simulation is responsible for device state machine
>>>> simulation.
>>>>    2.4   The device simulation is responsbile for talking to vhost.
>>>>      With this approach, we could minimize the virtio PMD modifications.
>>>> The virtio PMD is like configuring a real virtio-net PCI device.
>>>>
>>>> Memory mapping?
>>>> A: QEMU could access the whole guest memory in KVM enviroment. We need
>>>> to fill the gap.
>>>> container maps the shared memory to container's virtual address space
>>>> and host maps it to host's virtual address space. There is a fixed
>>>> offset mapping.
>>>> Container creates shared vring based on the memory. Container also
>>>> creates mbuf memory pool based on the shared memroy.
>>>> In VHOST_SET_MEMORY_TABLE message, we send the memory mapping
>>>> information for the shared memory. As we require mbuf pool created on
>>>> the shared memory, and buffers are allcoated from the mbuf pools, dpdk
>>>> vhost could translate the GPA in vring desc to host virtual.
>>>>
>>>>
>>>> GPA or CVA in vring desc?
>>>> To ease the memory translation, rather than using GPA, here we use
>>>> CVA(container virtual address). This the tricky thing here.
>>>> 1) virtio PMD writes vring's VFN rather than PFN to PFN register through
>>>> IOAPI.
>>>> 2) device simulation framework will use VFN as PFN.
>>>> 3) device simulation sends SET_VRING_ADDR with CVA.
>>>> 4) virtio PMD fills vring desc with CVA of the mbuf data pointer rather
>>>> than GPA.
>>>> So when host sees the CVA, it could translates it to HVA(host virtual
>>>> address).
>>>>
>>>> Worth to note:
>>>> The virtio interface in container follows the vhost message format, and
>>>> is compliant with dpdk vhost implmentation, i.e, no dpdk vhost
>>>> modification is needed.
>>>> vHost isn't aware whether the incoming virtio comes from KVM guest or
>>>> container.
>>>>
>>>> The pretty much covers the high level design. There are quite some low
>>>> level issues. For example, 32bit PFN is enough for KVM guest, since we
>>>> use 64bit VFN(virtual page frame number),  trick is done here through a
>>>> special IOAPI.
>> In addition above, we might consider "namespace" kernel functionality.
>> Technically, it would not be a big problem, but related with security.
>> So it would be nice to take account.
> There is no namespace concept here because we don't generate kernel
> netdev devices. It might be usefull if we could extend our work to
> support kernel netdev interface and assign to container's namespace.
>> Regards,
>> Tetsuya
>>
>>>> /huawei
>>>>
>>>>  
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>

Re: [dpdk-dev] vhost compliant virtio based networking interface in container

[Xie, Huawei]

On 8/26/2015 5:23 PM, Tetsuya Mukawa wrote:
> On 2015/08/25 18:56, Xie, Huawei wrote:
>> On 8/25/2015 10:59 AM, Tetsuya Mukawa wrote:
>>> Hi Xie and Yanping,
>>>
>>>
>>> May I ask you some questions?
>>> It seems we are also developing an almost same one.
>> Good to know that we are tackling the same problem and have the similar
>> idea.
>> What is your status now? We had the POC running, and compliant with
>> dpdkvhost.
>> Interrupt like notification isn't supported.
> We implemented vhost PMD first, so we just start implementing it.
>
>>> On 2015/08/20 19:14, Xie, Huawei wrote:
>>>> Added dev@dpdk.org
>>>>
>>>> On 8/20/2015 6:04 PM, Xie, Huawei wrote:
>>>>> Yanping:
>>>>> I read your mail, seems what we did are quite similar. Here i wrote a
>>>>> quick mail to describe our design. Let me know if it is the same thing.
>>>>>
>>>>> Problem Statement:
>>>>> We don't have a high performance networking interface in container for
>>>>> NFV. Current veth pair based interface couldn't be easily accelerated.
>>>>>
>>>>> The key components involved:
>>>>>     1.    DPDK based virtio PMD driver in container.
>>>>>     2.    device simulation framework in container.
>>>>>     3.    dpdk(or kernel) vhost running in host.
>>>>>
>>>>> How virtio is created?
>>>>> A:  There is no "real" virtio-pci device in container environment.
>>>>> 1). Host maintains pools of memories, and shares memory to container.
>>>>> This could be accomplished through host share a huge page file to container.
>>>>> 2). Containers creates virtio rings based on the shared memory.
>>>>> 3). Container creates mbuf memory pools on the shared memory.
>>>>> 4) Container send the memory and vring information to vhost through
>>>>> vhost message. This could be done either through ioctl call or vhost
>>>>> user message.
>>>>>
>>>>> How vhost message is sent?
>>>>> A: There are two alternative ways to do this.
>>>>> 1) The customized virtio PMD is responsible for all the vring creation,
>>>>> and vhost message sending.
>>> Above is our approach so far.
>>> It seems Yanping also takes this kind of approach.
>>> We are using vhost-user functionality instead of using the vhost-net
>>> kernel module.
>>> Probably this is the difference between Yanping and us.
>> In my current implementation, the device simulation layer talks to "user
>> space" vhost through cuse interface. It could also be done through vhost
>> user socket. This isn't the key point.
>> Here vhost-user is kind of confusing, maybe user space vhost is more
>> accurate, either cuse or unix domain socket. :).
>>
>> As for yanping, they are now connecting to vhost-net kernel module, but
>> they are also trying to connect to "user space" vhost.  Correct me if wrong.
>> Yes, there is some difference between these two. Vhost-net kernel module
>> could directly access other process's memory, while using
>> vhost-user(cuse/user), we need do the memory mapping.
>>> BTW, we are going to submit a vhost PMD for DPDK-2.2.
>>> This PMD is implemented on librte_vhost.
>>> It allows DPDK application to handle a vhost-user(cuse) backend as a
>>> normal NIC port.
>>> This PMD should work with both Xie and Yanping approach.
>>> (In the case of Yanping approach, we may need vhost-cuse)
>>>
>>>>> 2) We could do this through a lightweight device simulation framework.
>>>>>     The device simulation creates simple PCI bus. On the PCI bus,
>>>>> virtio-net PCI devices are created. The device simulations provides
>>>>> IOAPI for MMIO/IO access.
>>> Does it mean you implemented a kernel module?
>>> If so, do you still need vhost-cuse functionality to handle vhost
>>> messages n userspace?
>> The device simulation is  a library running in user space in container. 
>> It is linked with DPDK app. It creates pseudo buses and virtio-net PCI
>> devices.
>> The virtio-container-PMD configures the virtio-net pseudo devices
>> through IOAPI provided by the device simulation rather than IO
>> instructions as in KVM.
>> Why we use device simulation?
>> We could create other virtio devices in container, and provide an common
>> way to talk to vhost-xx module.
> Thanks for explanation.
> At first reading, I thought the difference between approach1 and
> approach2 is whether we need to implement a new kernel module, or not.
> But I understand how you implemented.
>
> Please let me explain our design more.
> We might use a kind of similar approach to handle a pseudo virtio-net
> device in DPDK.
> (Anyway, we haven't finished implementing yet, this overview might have
> some technical problems)
>
> Step1. Separate virtio-net and vhost-user socket related code from QEMU,
> then implement it as a separated program.
> The program also has below features.
>  - Create a directory that contains almost same files like
> /sys/bus/pci/device/<pci address>/*
>    (To scan these file located on outside sysfs, we need to fix EAL)
>  - This dummy device is driven by dummy-virtio-net-driver. This name is
> specified by '<pci addr>/driver' file.
>  - Create a shared file that represents pci configuration space, then
> mmap it, also specify the path in '<pci addr>/resource_path'
>
> The program will be GPL, but it will be like a bridge on the shared
> memory between virtio-net PMD and DPDK vhost backend.
> Actually, It will work under virtio-net PMD, but we don't need to link it.
> So I guess we don't have GPL license issue.
>
> Step2. Fix pci scan code of EAL to scan dummy devices.
>  - To scan above files, extend pci_scan() of EAL.
>
> Step3. Add a new kdrv type to EAL.
>  - To handle the 'dummy-virtio-net-driver', add a new kdrv type to EAL.
>
> Step4. Implement pci_dummy_virtio_net_map/unmap().
>  - It will have almost same functionality like pci_uio_map(), but for
> dummy virtio-net device.
>  - The dummy device will be mmaped using a path specified in '<pci
> addr>/resource_path'.
>
> Step5. Add a new compile option for virtio-net device to replace IO
> functions.
>  - The IO functions of virtio-net PMD will be replaced by read() and
> write() to access to the shared memory.
>  - Add notification mechanism to IO functions. This will be used when
> write() to the shared memory is done.
>  (Not sure exactly, but probably we need it)
>
> Does it make sense?
> I guess Step1&2 is different from your approach, but the rest might be
> similar.
>
> Actually, we just need sysfs entries for a virtio-net dummy device, but
> so far, I don't have a fine way to register them from user space without
> loading a kernel module.
Tetsuya:
I don't quite get the details. Who will create those sysfs entries? A
kernel module right?
The virtio-net is configured through read/write to sharing
memory(between host and guest), right?
Where is shared vring created and shared memory created, on shared huge
page between host and guest?
Who will talk to dpdkvhost?
> This is because I need to change pci_scan() also.
>
> It seems you have implemented a virtio-net pseudo device as BSD license.
> If so, this kind of PMD would be nice to use it.
Currently it is based on native linux kvm tool.
> In the case that it takes much time to implement some lost
> functionalities like interrupt mode, using QEMU code might be an one of
> options.
For interrupt mode, i plan to use eventfd for sleep/wake, have not tried
yet.
> Anyway, we just need a fine virtual NIC between containers and host.
> So we don't hold to our approach and implementation.

Do you have comments to my implementation?
We could publish the version without the device framework first for
reference.
>
> Thanks,
> Tetsuya
>
>>>>>    2.1  virtio PMD configures the pseudo virtio device as how it does in
>>>>> KVM guest enviroment.
>>>>>    2.2  Rather than using io instruction, virtio PMD uses IOAPI for IO
>>>>> operation on the virtio-net PCI device.
>>>>>    2.3  The device simulation is responsible for device state machine
>>>>> simulation.
>>>>>    2.4   The device simulation is responsbile for talking to vhost.
>>>>>      With this approach, we could minimize the virtio PMD modifications.
>>>>> The virtio PMD is like configuring a real virtio-net PCI device.
>>>>>
>>>>> Memory mapping?
>>>>> A: QEMU could access the whole guest memory in KVM enviroment. We need
>>>>> to fill the gap.
>>>>> container maps the shared memory to container's virtual address space
>>>>> and host maps it to host's virtual address space. There is a fixed
>>>>> offset mapping.
>>>>> Container creates shared vring based on the memory. Container also
>>>>> creates mbuf memory pool based on the shared memroy.
>>>>> In VHOST_SET_MEMORY_TABLE message, we send the memory mapping
>>>>> information for the shared memory. As we require mbuf pool created on
>>>>> the shared memory, and buffers are allcoated from the mbuf pools, dpdk
>>>>> vhost could translate the GPA in vring desc to host virtual.
>>>>>
>>>>>
>>>>> GPA or CVA in vring desc?
>>>>> To ease the memory translation, rather than using GPA, here we use
>>>>> CVA(container virtual address). This the tricky thing here.
>>>>> 1) virtio PMD writes vring's VFN rather than PFN to PFN register through
>>>>> IOAPI.
>>>>> 2) device simulation framework will use VFN as PFN.
>>>>> 3) device simulation sends SET_VRING_ADDR with CVA.
>>>>> 4) virtio PMD fills vring desc with CVA of the mbuf data pointer rather
>>>>> than GPA.
>>>>> So when host sees the CVA, it could translates it to HVA(host virtual
>>>>> address).
>>>>>
>>>>> Worth to note:
>>>>> The virtio interface in container follows the vhost message format, and
>>>>> is compliant with dpdk vhost implmentation, i.e, no dpdk vhost
>>>>> modification is needed.
>>>>> vHost isn't aware whether the incoming virtio comes from KVM guest or
>>>>> container.
>>>>>
>>>>> The pretty much covers the high level design. There are quite some low
>>>>> level issues. For example, 32bit PFN is enough for KVM guest, since we
>>>>> use 64bit VFN(virtual page frame number),  trick is done here through a
>>>>> special IOAPI.
>>> In addition above, we might consider "namespace" kernel functionality.
>>> Technically, it would not be a big problem, but related with security.
>>> So it would be nice to take account.
>> There is no namespace concept here because we don't generate kernel
>> netdev devices. It might be usefull if we could extend our work to
>> support kernel netdev interface and assign to container's namespace.
Yes, it would be great if we could extend this to support both kernel
networking and user space networking.
No progress so far.

>>> Regards,
>>> Tetsuya
>>>
>>>>> /huawei
>>>>>
>>>>>  
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>

Re: [dpdk-dev] vhost compliant virtio based networking interface in container

[Tetsuya Mukawa]

On 2015/09/07 14:54, Xie, Huawei wrote:
> On 8/26/2015 5:23 PM, Tetsuya Mukawa wrote:
>> On 2015/08/25 18:56, Xie, Huawei wrote:
>>> On 8/25/2015 10:59 AM, Tetsuya Mukawa wrote:
>>>> Hi Xie and Yanping,
>>>>
>>>>
>>>> May I ask you some questions?
>>>> It seems we are also developing an almost same one.
>>> Good to know that we are tackling the same problem and have the similar
>>> idea.
>>> What is your status now? We had the POC running, and compliant with
>>> dpdkvhost.
>>> Interrupt like notification isn't supported.
>> We implemented vhost PMD first, so we just start implementing it.
>>
>>>> On 2015/08/20 19:14, Xie, Huawei wrote:
>>>>> Added dev@dpdk.org
>>>>>
>>>>> On 8/20/2015 6:04 PM, Xie, Huawei wrote:
>>>>>> Yanping:
>>>>>> I read your mail, seems what we did are quite similar. Here i wrote a
>>>>>> quick mail to describe our design. Let me know if it is the same thing.
>>>>>>
>>>>>> Problem Statement:
>>>>>> We don't have a high performance networking interface in container for
>>>>>> NFV. Current veth pair based interface couldn't be easily accelerated.
>>>>>>
>>>>>> The key components involved:
>>>>>>     1.    DPDK based virtio PMD driver in container.
>>>>>>     2.    device simulation framework in container.
>>>>>>     3.    dpdk(or kernel) vhost running in host.
>>>>>>
>>>>>> How virtio is created?
>>>>>> A:  There is no "real" virtio-pci device in container environment.
>>>>>> 1). Host maintains pools of memories, and shares memory to container.
>>>>>> This could be accomplished through host share a huge page file to container.
>>>>>> 2). Containers creates virtio rings based on the shared memory.
>>>>>> 3). Container creates mbuf memory pools on the shared memory.
>>>>>> 4) Container send the memory and vring information to vhost through
>>>>>> vhost message. This could be done either through ioctl call or vhost
>>>>>> user message.
>>>>>>
>>>>>> How vhost message is sent?
>>>>>> A: There are two alternative ways to do this.
>>>>>> 1) The customized virtio PMD is responsible for all the vring creation,
>>>>>> and vhost message sending.
>>>> Above is our approach so far.
>>>> It seems Yanping also takes this kind of approach.
>>>> We are using vhost-user functionality instead of using the vhost-net
>>>> kernel module.
>>>> Probably this is the difference between Yanping and us.
>>> In my current implementation, the device simulation layer talks to "user
>>> space" vhost through cuse interface. It could also be done through vhost
>>> user socket. This isn't the key point.
>>> Here vhost-user is kind of confusing, maybe user space vhost is more
>>> accurate, either cuse or unix domain socket. :).
>>>
>>> As for yanping, they are now connecting to vhost-net kernel module, but
>>> they are also trying to connect to "user space" vhost.  Correct me if wrong.
>>> Yes, there is some difference between these two. Vhost-net kernel module
>>> could directly access other process's memory, while using
>>> vhost-user(cuse/user), we need do the memory mapping.
>>>> BTW, we are going to submit a vhost PMD for DPDK-2.2.
>>>> This PMD is implemented on librte_vhost.
>>>> It allows DPDK application to handle a vhost-user(cuse) backend as a
>>>> normal NIC port.
>>>> This PMD should work with both Xie and Yanping approach.
>>>> (In the case of Yanping approach, we may need vhost-cuse)
>>>>
>>>>>> 2) We could do this through a lightweight device simulation framework.
>>>>>>     The device simulation creates simple PCI bus. On the PCI bus,
>>>>>> virtio-net PCI devices are created. The device simulations provides
>>>>>> IOAPI for MMIO/IO access.
>>>> Does it mean you implemented a kernel module?
>>>> If so, do you still need vhost-cuse functionality to handle vhost
>>>> messages n userspace?
>>> The device simulation is  a library running in user space in container. 
>>> It is linked with DPDK app. It creates pseudo buses and virtio-net PCI
>>> devices.
>>> The virtio-container-PMD configures the virtio-net pseudo devices
>>> through IOAPI provided by the device simulation rather than IO
>>> instructions as in KVM.
>>> Why we use device simulation?
>>> We could create other virtio devices in container, and provide an common
>>> way to talk to vhost-xx module.
>> Thanks for explanation.
>> At first reading, I thought the difference between approach1 and
>> approach2 is whether we need to implement a new kernel module, or not.
>> But I understand how you implemented.
>>
>> Please let me explain our design more.
>> We might use a kind of similar approach to handle a pseudo virtio-net
>> device in DPDK.
>> (Anyway, we haven't finished implementing yet, this overview might have
>> some technical problems)
>>
>> Step1. Separate virtio-net and vhost-user socket related code from QEMU,
>> then implement it as a separated program.
>> The program also has below features.
>>  - Create a directory that contains almost same files like
>> /sys/bus/pci/device/<pci address>/*
>>    (To scan these file located on outside sysfs, we need to fix EAL)
>>  - This dummy device is driven by dummy-virtio-net-driver. This name is
>> specified by '<pci addr>/driver' file.
>>  - Create a shared file that represents pci configuration space, then
>> mmap it, also specify the path in '<pci addr>/resource_path'
>>
>> The program will be GPL, but it will be like a bridge on the shared
>> memory between virtio-net PMD and DPDK vhost backend.
>> Actually, It will work under virtio-net PMD, but we don't need to link it.
>> So I guess we don't have GPL license issue.
>>
>> Step2. Fix pci scan code of EAL to scan dummy devices.
>>  - To scan above files, extend pci_scan() of EAL.
>>
>> Step3. Add a new kdrv type to EAL.
>>  - To handle the 'dummy-virtio-net-driver', add a new kdrv type to EAL.
>>
>> Step4. Implement pci_dummy_virtio_net_map/unmap().
>>  - It will have almost same functionality like pci_uio_map(), but for
>> dummy virtio-net device.
>>  - The dummy device will be mmaped using a path specified in '<pci
>> addr>/resource_path'.
>>
>> Step5. Add a new compile option for virtio-net device to replace IO
>> functions.
>>  - The IO functions of virtio-net PMD will be replaced by read() and
>> write() to access to the shared memory.
>>  - Add notification mechanism to IO functions. This will be used when
>> write() to the shared memory is done.
>>  (Not sure exactly, but probably we need it)
>>
>> Does it make sense?
>> I guess Step1&2 is different from your approach, but the rest might be
>> similar.
>>
>> Actually, we just need sysfs entries for a virtio-net dummy device, but
>> so far, I don't have a fine way to register them from user space without
>> loading a kernel module.
> Tetsuya:
> I don't quite get the details. Who will create those sysfs entries? A
> kernel module right?

Hi Xie,

I don't create sysfs entries. Just create a directory that contains
files looks like sysfs entries.
And initialize EAL with not only sysfs but also the above directory.

In quoted last sentence, I wanted to say we just needed files looks like
sysfs entries.
But I don't know a good way to create files under sysfs without loading
kernel module.
This is because I try to create the additional directory.

> The virtio-net is configured through read/write to sharing
> memory(between host and guest), right?

Yes, I agree.

> Where is shared vring created and shared memory created, on shared huge
> page between host and guest?

The vritqueues(vrings) are on guest hugepage.

Let me explain.
Guest container should have read/write access to a part of hugepage
directory on host.
(For example, /mnt/huge/conainer1/ is shared between host and guest.)
Also host and guest needs to communicate through a unix domain socket.
(For example, host and guest can communicate with using
"/tmp/container1/sock")

If we can do like above, a virtio-net PMD on guest can creates
virtqueues(vrings) on it's hugepage, and writes these information to a
pseudo virtio-net device that is a process created in guest container.
Then the pseudo virtio-net device sends it to vhost-user backend(host
DPDK application) through a unix domain socket.

So with my plan, there are 3 processes.
DPDK applications on host and guest, also a process that works like
virtio-net device.

> Who will talk to dpdkvhost?

If we need to talk to a cuse device or the vhost-net kernel module, an
above pseudo virtio-net device could talk to.
(But, so far, my target is only vhost-user.)

>> This is because I need to change pci_scan() also.
>>
>> It seems you have implemented a virtio-net pseudo device as BSD license.
>> If so, this kind of PMD would be nice to use it.
> Currently it is based on native linux kvm tool.

Great, I hadn't noticed this option.

>> In the case that it takes much time to implement some lost
>> functionalities like interrupt mode, using QEMU code might be an one of
>> options.
> For interrupt mode, i plan to use eventfd for sleep/wake, have not tried
> yet.
>> Anyway, we just need a fine virtual NIC between containers and host.
>> So we don't hold to our approach and implementation.
> Do you have comments to my implementation?
> We could publish the version without the device framework first for
> reference.

No I don't have. Could you please share it?
I am looking forward to seeing it.

Tetsuya

Re: [dpdk-dev] vhost compliant virtio based networking interface in container

[Xie, Huawei]

On 9/8/2015 12:45 PM, Tetsuya Mukawa wrote:
> On 2015/09/07 14:54, Xie, Huawei wrote:
>> On 8/26/2015 5:23 PM, Tetsuya Mukawa wrote:
>>> On 2015/08/25 18:56, Xie, Huawei wrote:
>>>> On 8/25/2015 10:59 AM, Tetsuya Mukawa wrote:
>>>>> Hi Xie and Yanping,
>>>>>
>>>>>
>>>>> May I ask you some questions?
>>>>> It seems we are also developing an almost same one.
>>>> Good to know that we are tackling the same problem and have the similar
>>>> idea.
>>>> What is your status now? We had the POC running, and compliant with
>>>> dpdkvhost.
>>>> Interrupt like notification isn't supported.
>>> We implemented vhost PMD first, so we just start implementing it.
>>>
>>>>> On 2015/08/20 19:14, Xie, Huawei wrote:
>>>>>> Added dev@dpdk.org
>>>>>>
>>>>>> On 8/20/2015 6:04 PM, Xie, Huawei wrote:
>>>>>>> Yanping:
>>>>>>> I read your mail, seems what we did are quite similar. Here i wrote a
>>>>>>> quick mail to describe our design. Let me know if it is the same thing.
>>>>>>>
>>>>>>> Problem Statement:
>>>>>>> We don't have a high performance networking interface in container for
>>>>>>> NFV. Current veth pair based interface couldn't be easily accelerated.
>>>>>>>
>>>>>>> The key components involved:
>>>>>>>     1.    DPDK based virtio PMD driver in container.
>>>>>>>     2.    device simulation framework in container.
>>>>>>>     3.    dpdk(or kernel) vhost running in host.
>>>>>>>
>>>>>>> How virtio is created?
>>>>>>> A:  There is no "real" virtio-pci device in container environment.
>>>>>>> 1). Host maintains pools of memories, and shares memory to container.
>>>>>>> This could be accomplished through host share a huge page file to container.
>>>>>>> 2). Containers creates virtio rings based on the shared memory.
>>>>>>> 3). Container creates mbuf memory pools on the shared memory.
>>>>>>> 4) Container send the memory and vring information to vhost through
>>>>>>> vhost message. This could be done either through ioctl call or vhost
>>>>>>> user message.
>>>>>>>
>>>>>>> How vhost message is sent?
>>>>>>> A: There are two alternative ways to do this.
>>>>>>> 1) The customized virtio PMD is responsible for all the vring creation,
>>>>>>> and vhost message sending.
>>>>> Above is our approach so far.
>>>>> It seems Yanping also takes this kind of approach.
>>>>> We are using vhost-user functionality instead of using the vhost-net
>>>>> kernel module.
>>>>> Probably this is the difference between Yanping and us.
>>>> In my current implementation, the device simulation layer talks to "user
>>>> space" vhost through cuse interface. It could also be done through vhost
>>>> user socket. This isn't the key point.
>>>> Here vhost-user is kind of confusing, maybe user space vhost is more
>>>> accurate, either cuse or unix domain socket. :).
>>>>
>>>> As for yanping, they are now connecting to vhost-net kernel module, but
>>>> they are also trying to connect to "user space" vhost.  Correct me if wrong.
>>>> Yes, there is some difference between these two. Vhost-net kernel module
>>>> could directly access other process's memory, while using
>>>> vhost-user(cuse/user), we need do the memory mapping.
>>>>> BTW, we are going to submit a vhost PMD for DPDK-2.2.
>>>>> This PMD is implemented on librte_vhost.
>>>>> It allows DPDK application to handle a vhost-user(cuse) backend as a
>>>>> normal NIC port.
>>>>> This PMD should work with both Xie and Yanping approach.
>>>>> (In the case of Yanping approach, we may need vhost-cuse)
>>>>>
>>>>>>> 2) We could do this through a lightweight device simulation framework.
>>>>>>>     The device simulation creates simple PCI bus. On the PCI bus,
>>>>>>> virtio-net PCI devices are created. The device simulations provides
>>>>>>> IOAPI for MMIO/IO access.
>>>>> Does it mean you implemented a kernel module?
>>>>> If so, do you still need vhost-cuse functionality to handle vhost
>>>>> messages n userspace?
>>>> The device simulation is  a library running in user space in container. 
>>>> It is linked with DPDK app. It creates pseudo buses and virtio-net PCI
>>>> devices.
>>>> The virtio-container-PMD configures the virtio-net pseudo devices
>>>> through IOAPI provided by the device simulation rather than IO
>>>> instructions as in KVM.
>>>> Why we use device simulation?
>>>> We could create other virtio devices in container, and provide an common
>>>> way to talk to vhost-xx module.
>>> Thanks for explanation.
>>> At first reading, I thought the difference between approach1 and
>>> approach2 is whether we need to implement a new kernel module, or not.
>>> But I understand how you implemented.
>>>
>>> Please let me explain our design more.
>>> We might use a kind of similar approach to handle a pseudo virtio-net
>>> device in DPDK.
>>> (Anyway, we haven't finished implementing yet, this overview might have
>>> some technical problems)
>>>
>>> Step1. Separate virtio-net and vhost-user socket related code from QEMU,
>>> then implement it as a separated program.
>>> The program also has below features.
>>>  - Create a directory that contains almost same files like
>>> /sys/bus/pci/device/<pci address>/*
>>>    (To scan these file located on outside sysfs, we need to fix EAL)
>>>  - This dummy device is driven by dummy-virtio-net-driver. This name is
>>> specified by '<pci addr>/driver' file.
>>>  - Create a shared file that represents pci configuration space, then
>>> mmap it, also specify the path in '<pci addr>/resource_path'
>>>
>>> The program will be GPL, but it will be like a bridge on the shared
>>> memory between virtio-net PMD and DPDK vhost backend.
>>> Actually, It will work under virtio-net PMD, but we don't need to link it.
>>> So I guess we don't have GPL license issue.
>>>
>>> Step2. Fix pci scan code of EAL to scan dummy devices.
>>>  - To scan above files, extend pci_scan() of EAL.
>>>
>>> Step3. Add a new kdrv type to EAL.
>>>  - To handle the 'dummy-virtio-net-driver', add a new kdrv type to EAL.
>>>
>>> Step4. Implement pci_dummy_virtio_net_map/unmap().
>>>  - It will have almost same functionality like pci_uio_map(), but for
>>> dummy virtio-net device.
>>>  - The dummy device will be mmaped using a path specified in '<pci
>>> addr>/resource_path'.
>>>
>>> Step5. Add a new compile option for virtio-net device to replace IO
>>> functions.
>>>  - The IO functions of virtio-net PMD will be replaced by read() and
>>> write() to access to the shared memory.
>>>  - Add notification mechanism to IO functions. This will be used when
>>> write() to the shared memory is done.
>>>  (Not sure exactly, but probably we need it)
>>>
>>> Does it make sense?
>>> I guess Step1&2 is different from your approach, but the rest might be
>>> similar.
>>>
>>> Actually, we just need sysfs entries for a virtio-net dummy device, but
>>> so far, I don't have a fine way to register them from user space without
>>> loading a kernel module.
>> Tetsuya:
>> I don't quite get the details. Who will create those sysfs entries? A
>> kernel module right?
> Hi Xie,
>
> I don't create sysfs entries. Just create a directory that contains
> files looks like sysfs entries.
> And initialize EAL with not only sysfs but also the above directory.
>
> In quoted last sentence, I wanted to say we just needed files looks like
> sysfs entries.
> But I don't know a good way to create files under sysfs without loading
> kernel module.
> This is because I try to create the additional directory.
>
>> The virtio-net is configured through read/write to sharing
>> memory(between host and guest), right?
> Yes, I agree.
>
>> Where is shared vring created and shared memory created, on shared huge
>> page between host and guest?
> The vritqueues(vrings) are on guest hugepage.
>
> Let me explain.
> Guest container should have read/write access to a part of hugepage
> directory on host.
> (For example, /mnt/huge/conainer1/ is shared between host and guest.)
> Also host and guest needs to communicate through a unix domain socket.
> (For example, host and guest can communicate with using
> "/tmp/container1/sock")
>
> If we can do like above, a virtio-net PMD on guest can creates
> virtqueues(vrings) on it's hugepage, and writes these information to a
> pseudo virtio-net device that is a process created in guest container.
> Then the pseudo virtio-net device sends it to vhost-user backend(host
> DPDK application) through a unix domain socket.
>
> So with my plan, there are 3 processes.
> DPDK applications on host and guest, also a process that works like
> virtio-net device.
>
>> Who will talk to dpdkvhost?
> If we need to talk to a cuse device or the vhost-net kernel module, an
> above pseudo virtio-net device could talk to.
> (But, so far, my target is only vhost-user.)
>
>>> This is because I need to change pci_scan() also.
>>>
>>> It seems you have implemented a virtio-net pseudo device as BSD license.
>>> If so, this kind of PMD would be nice to use it.
>> Currently it is based on native linux kvm tool.
> Great, I hadn't noticed this option.
>
>>> In the case that it takes much time to implement some lost
>>> functionalities like interrupt mode, using QEMU code might be an one of
>>> options.
>> For interrupt mode, i plan to use eventfd for sleep/wake, have not tried
>> yet.
>>> Anyway, we just need a fine virtual NIC between containers and host.
>>> So we don't hold to our approach and implementation.
>> Do you have comments to my implementation?
>> We could publish the version without the device framework first for
>> reference.
> No I don't have. Could you please share it?
> I am looking forward to seeing it.
OK,  we are removing the device framework. Hope to publish it  in one
month's time.

>
> Tetsuya
>