[dpdk-dev] Can xenvirt pmd work in xen guest (aka DomU) without xen-vhost in Dom0 ?

[dpdk-dev] Can xenvirt pmd work in xen guest (aka DomU) without xen-vhost in Dom0 ?

[Bill Bonaparte]

Hi, everybody:
     I am trying to run dpdk in xen guest, after trying dpdk17.05 without
success, I tried dpdk16.04 again.  and it's sucessful to run testpmd this
time. but it can't receive any packets.
     the command I used is :
./testpmd -l 1,2 -n 4 --vdev="eth_xenvirt0,mac=a6:38:57:e8:d5:be"
--vdev="eth_xenvirt1,mac=32:f1:42:55:94:74" -- --total-num-mbufs=2048

I read the xen guide again, I found the guide says:
DPDK provides a para-virtualization packet switching solution, based on the
Xen hypervisor’s Grant Table, Note 1, which provides simple and fast packet
switching capability between guest domains and host domain based on MAC
address or VLAN tag. This solution is comprised of two components; a Poll
Mode Driver (PMD) as the front end in the guest domain and a switching back
end in the host domain.

does it mean that dpdk running in DomU must cooperate with xen-vhost in
Dom0 ?
I really need your help.
I appreciate any reply from you .

Re: [dpdk-dev] Can xenvirt pmd work in xen guest (aka DomU) without xen-vhost in Dom0 ?

[Tan, Jianfeng]

Hi Bill


On 9/30/2017 4:47 PM, Bill Bonaparte wrote:
> Hi, everybody:
>       I am trying to run dpdk in xen guest, after trying dpdk17.05 without
> success, I tried dpdk16.04 again.  and it's sucessful to run testpmd this
> time. but it can't receive any packets.
>       the command I used is :
> ./testpmd -l 1,2 -n 4 --vdev="eth_xenvirt0,mac=a6:38:57:e8:d5:be"
> --vdev="eth_xenvirt1,mac=32:f1:42:55:94:74" -- --total-num-mbufs=2048
>
> I read the xen guide again, I found the guide says:
> DPDK provides a para-virtualization packet switching solution, based on the
> Xen hypervisor’s Grant Table, Note 1, which provides simple and fast packet
> switching capability between guest domains and host domain based on MAC
> address or VLAN tag. This solution is comprised of two components; a Poll
> Mode Driver (PMD) as the front end in the guest domain and a switching back
> end in the host domain.
>
> does it mean that dpdk running in DomU must cooperate with xen-vhost in
> Dom0 ?

Yes, we need to run vhost_xen at dom0 so that xenvirt at domU can work. 
But it has been long lack of test, not sure of it still works.

And FYI, we are going to remove xen support (dom0, xenvirt, vhost_xen) 
in the coming release.

Thanks,
Jianfeng

> I really need your help.
> I appreciate any reply from you .

Re: [dpdk-dev] Can xenvirt pmd work in xen guest (aka DomU) without xen-vhost in Dom0 ?

[Bill Bonaparte]

Hi Jianfeng,
 Thank you for replying, I appreciate so much for this.
  we are trying to run our dpdk application on AWS cloud which use xen
platform.
  for this case, what should I do to support AWS cloud ?
  Is there any way to do this ?

On Sat, Sep 30, 2017 at 4:53 PM, Tan, Jianfeng <jianfeng.tan@intel.com>
wrote:

> Hi Bill
>
>
> On 9/30/2017 4:47 PM, Bill Bonaparte wrote:
>
>> Hi, everybody:
>>       I am trying to run dpdk in xen guest, after trying dpdk17.05 without
>> success, I tried dpdk16.04 again.  and it's sucessful to run testpmd this
>> time. but it can't receive any packets.
>>       the command I used is :
>> ./testpmd -l 1,2 -n 4 --vdev="eth_xenvirt0,mac=a6:38:57:e8:d5:be"
>> --vdev="eth_xenvirt1,mac=32:f1:42:55:94:74" -- --total-num-mbufs=2048
>>
>> I read the xen guide again, I found the guide says:
>> DPDK provides a para-virtualization packet switching solution, based on
>> the
>> Xen hypervisor’s Grant Table, Note 1, which provides simple and fast
>> packet
>> switching capability between guest domains and host domain based on MAC
>> address or VLAN tag. This solution is comprised of two components; a Poll
>> Mode Driver (PMD) as the front end in the guest domain and a switching
>> back
>> end in the host domain.
>>
>> does it mean that dpdk running in DomU must cooperate with xen-vhost in
>> Dom0 ?
>>
>
> Yes, we need to run vhost_xen at dom0 so that xenvirt at domU can work.
> But it has been long lack of test, not sure of it still works.
>
> And FYI, we are going to remove xen support (dom0, xenvirt, vhost_xen) in
> the coming release.
>
> Thanks,
> Jianfeng
>
>
> I really need your help.
>> I appreciate any reply from you .
>>
>
>

Re: [dpdk-dev] Can xenvirt pmd work in xen guest (aka DomU) without xen-vhost in Dom0 ?

[Tan, Jianfeng]

On 9/30/2017 5:25 PM, Bill Bonaparte wrote:
> Hi Jianfeng,
>  Thank you for replying, I appreciate so much for this.
>   we are trying to run our dpdk application on AWS cloud which use xen 
> platform.
>   for this case, what should I do to support AWS cloud ?
>   Is there any way to do this ?

Sorry, I don't have knowledge to answer this question as I don't know 
what kind of net devices in AWS cloud.

Thanks,
Jianfeng

Re: [dpdk-dev] Can xenvirt pmd work in xen guest (aka DomU) without xen-vhost in Dom0 ?

[Bill Bonaparte]

Thanks Jianfeng for taking time to reply.

please allow me to briefly explain why I want to run dpdk on xen.
our system is based on dpdk, which means we use dpdk as packet
receive/transmit engine,
and with integrated dpdk virtio/vmxnet3 driver, our system can run on
KVM/VMware platform .
this year, we have plan to run our system on AWS cloud, but I found that AWS
uses xen as its virtualization platform, and the bus-info of nic is vif-x
(x could be 0,1,2...),
the driver used in kernel is vif. this should be para-virtualized nic used
on xen.

I don't know which dpdk drvier can manage this pv nic. then I see xenvirt,
I think this driver can
did this job, like virtio can manage virtio nic which is used on kvm.
unfortunately, after some study work, I run testpmd successfully on xen,
but no packets received.

with the informain got from you, I know It's need to run vhost_xen at dom0
so that xenvirt at domU can work.
but for my case, I have no change to run vhost_xen at dom0, because I only
can operate my own domU.

for this case, If I want to run system which is based on dpdk at domU, what
should I do?
appreciate any idea or suggestion from you.

On Mon, Oct 2, 2017 at 7:50 AM, Tan, Jianfeng <jianfeng.tan@intel.com>
wrote:

>
>
> On 9/30/2017 5:25 PM, Bill Bonaparte wrote:
>
>> Hi Jianfeng,
>>  Thank you for replying, I appreciate so much for this.
>>   we are trying to run our dpdk application on AWS cloud which use xen
>> platform.
>>   for this case, what should I do to support AWS cloud ?
>>   Is there any way to do this ?
>>
>
> Sorry, I don't have knowledge to answer this question as I don't know what
> kind of net devices in AWS cloud.
>
> Thanks,
> Jianfeng
>
>

Re: [dpdk-dev] Can xenvirt pmd work in xen guest (aka DomU) without xen-vhost in Dom0 ?

[Bill Bonaparte]

Thanks Jianfeng for taking time to reply.

please allow me to briefly explain why I want to run dpdk on xen.
our system is based on dpdk, which means we use dpdk as packet
receive/transmit engine,
and with integrated dpdk virtio/vmxnet3 driver, our system can run on
KVM/VMware platform .
this year, we have plan to run our system on AWS cloud, but I found that AWS
uses xen as its virtualization platform, and the bus-info of nic is vif-x
(x could be 0,1,2...),
the driver used in kernel is vif. this should be para-virtualized nic used
on xen.

I don't know which dpdk drvier can manage this pv nic. then I see xenvirt,
I think this driver can
did this job, like virtio can manage virtio nic which is used on kvm.
unfortunately, after some study work, I run testpmd successfully on xen,
but no packets received.

with the informain got from you, I know It's need to run vhost_xen at dom0
so that xenvirt at domU can work.
but for my case, I have no change to run vhost_xen at dom0, because I only
can operate my own domU.

for this case, If I want to run system which is based on dpdk at domU, what
should I do?
appreciate any idea or suggestion from you.

On Mon, Oct 2, 2017 at 7:50 AM, Tan, Jianfeng <jianfeng.tan@intel.com>
wrote:

>
>
> On 9/30/2017 5:25 PM, Bill Bonaparte wrote:
>
>> Hi Jianfeng,
>>  Thank you for replying, I appreciate so much for this.
>>   we are trying to run our dpdk application on AWS cloud which use xen
>> platform.
>>   for this case, what should I do to support AWS cloud ?
>>   Is there any way to do this ?
>>
>
> Sorry, I don't have knowledge to answer this question as I don't know what
> kind of net devices in AWS cloud.
>
> Thanks,
> Jianfeng
>
>

Re: [dpdk-dev] Can xenvirt pmd work in xen guest (aka DomU) without xen-vhost in Dom0 ?

[Tan, Jianfeng]

Hi,


On 10/8/2017 12:54 PM, Bill Bonaparte wrote:
> Thanks Jianfeng for taking time to reply.
>
> please allow me to briefly explain why I want to run dpdk on xen.
> our system is based on dpdk, which means we use dpdk as packet 
> receive/transmit engine,
> and with integrated dpdk virtio/vmxnet3 driver, our system can run on 
> KVM/VMware platform .
> this year, we have plan to run our system on AWS cloud, but I found 
> that AWS
> uses xen as its virtualization platform, and the bus-info of nic is 
> vif-x (x could be 0,1,2...),
> the driver used in kernel is vif. this should be para-virtualized nic 
> used on xen.

My guess is exactly as you describe. In AWS, we lack of a PMD for xen 
netfront (vif) nic. And even we got such a PMD, we still need a PMD for 
xen netback. Both are missing.

>
> I don't know which dpdk drvier can manage this pv nic. then I see 
> xenvirt, I think this driver can
> did this job, like virtio can manage virtio nic which is used on kvm.
> unfortunately, after some study work, I run testpmd successfully on 
> xen, but no packets received.
>
> with the informain got from you, I know It's need to run vhost_xen at 
> dom0 so that xenvirt at domU can work.
> but for my case, I have no change to run vhost_xen at dom0, because I 
> only can operate my own domU.
>
> for this case, If I want to run system which is based on dpdk at domU, 
> what should I do?
> appreciate any idea or suggestion from you.

What kind of performance are you seeking? Only accelerating the frontend 
by a new PMD, i.e. netfront, we can bypass the VM kernel (). But without 
accelerating the backend, it only brings limited improvement.

Not sure if anyone from Amazon or Oracle can comment?

Thanks,
Jianfeng

Re: [dpdk-dev] Can xenvirt pmd work in xen guest (aka DomU) without xen-vhost in Dom0 ?

[Bill Bonaparte]

On Mon, Oct 9, 2017 at 12:13 AM, Tan, Jianfeng <jianfeng.tan@intel.com>
wrote:

> Hi,
>
> On 10/8/2017 12:54 PM, Bill Bonaparte wrote:
>
> Thanks Jianfeng for taking time to reply.
>
> please allow me to briefly explain why I want to run dpdk on xen.
> our system is based on dpdk, which means we use dpdk as packet
> receive/transmit engine,
> and with integrated dpdk virtio/vmxnet3 driver, our system can run on
> KVM/VMware platform .
> this year, we have plan to run our system on AWS cloud, but I found that
> AWS
> uses xen as its virtualization platform, and the bus-info of nic is vif-x
> (x could be 0,1,2...),
> the driver used in kernel is vif. this should be para-virtualized nic used
> on xen.
>
>
> My guess is exactly as you describe. In AWS, we lack of a PMD for xen
> netfront (vif) nic. And even we got such a PMD, we still need a PMD for xen
> netback. Both are missing.
>
>
> I don't know which dpdk drvier can manage this pv nic. then I see
> xenvirt, I think this driver can
> did this job, like virtio can manage virtio nic which is used on kvm.
> unfortunately, after some study work, I run testpmd successfully on xen,
> but no packets received.
>
> with the informain got from you, I know It's need to run vhost_xen at dom0
> so that xenvirt at domU can work.
> but for my case, I have no change to run vhost_xen at dom0, because I only
> can operate my own domU.
>
> for this case, If I want to run system which is based on dpdk at domU,
> what should I do?
> appreciate any idea or suggestion from you.
>
>
> What kind of performance are you seeking? Only accelerating the frontend
>> by a new PMD, i.e. netfront, we can bypass the VM kernel (). But without
>> accelerating the backend, it only brings limited improvement.
>
> for me, the most important thing at the moment, is to run our system on
xen guest (aka DomU), with the full functions. the performance could be a
secondary consideration.
most of time, for our guests,  full functions, with the acceptable
performance,
is  fine.
do we have plan to support xen netfront pmd in dpdk ?
in my humble opinion,  limited performance is better than "not work" .

>
> Not sure if anyone from Amazon or Oracle can comment?
>
> Thanks,
> Jianfeng
>

Re: [dpdk-dev] Can xenvirt pmd work in xen guest (aka DomU) without xen-vhost in Dom0 ?

[Tan, Jianfeng]

> for me, the most important thing at the moment, is to run our system 
> on xen guest (aka DomU), with the full functions. the performance 
> could be a secondary consideration.
> most of time, for our guests,  full functions, with the 
> acceptable performance, is  fine.
> do we have plan to support xen netfront pmd in dpdk ?
> in my humble opinion,  limited performance is better than "not work" .

If you only care about the functionality, you can try AF_PACKET or pcap PMD.

Thanks,
Jianfeng

Re: [dpdk-dev] Can xenvirt pmd work in xen guest (aka DomU) without xen-vhost in Dom0 ?

[Stephen Hemminger]

On Mon, 9 Oct 2017 00:13:47 +0800
"Tan, Jianfeng" <jianfeng.tan@intel.com> wrote:

> Hi,
> 
> 
> On 10/8/2017 12:54 PM, Bill Bonaparte wrote:
> > Thanks Jianfeng for taking time to reply.
> >
> > please allow me to briefly explain why I want to run dpdk on xen.
> > our system is based on dpdk, which means we use dpdk as packet 
> > receive/transmit engine,
> > and with integrated dpdk virtio/vmxnet3 driver, our system can run on 
> > KVM/VMware platform .
> > this year, we have plan to run our system on AWS cloud, but I found 
> > that AWS
> > uses xen as its virtualization platform, and the bus-info of nic is 
> > vif-x (x could be 0,1,2...),
> > the driver used in kernel is vif. this should be para-virtualized nic 
> > used on xen.  
> 
> My guess is exactly as you describe. In AWS, we lack of a PMD for xen 
> netfront (vif) nic. And even we got such a PMD, we still need a PMD for 
> xen netback. Both are missing.
> 
> >
> > I don't know which dpdk drvier can manage this pv nic. then I see 
> > xenvirt, I think this driver can
> > did this job, like virtio can manage virtio nic which is used on kvm.
> > unfortunately, after some study work, I run testpmd successfully on 
> > xen, but no packets received.
> >
> > with the informain got from you, I know It's need to run vhost_xen at 
> > dom0 so that xenvirt at domU can work.
> > but for my case, I have no change to run vhost_xen at dom0, because I 
> > only can operate my own domU.
> >
> > for this case, If I want to run system which is based on dpdk at domU, 
> > what should I do?
> > appreciate any idea or suggestion from you.  
> 
> What kind of performance are you seeking? Only accelerating the frontend 
> by a new PMD, i.e. netfront, we can bypass the VM kernel (). But without 
> accelerating the backend, it only brings limited improvement.

Brocade wrote a xen net-front driver. It was never integrated upstream
and had many issues (such as pretending to be PCI and it wasn't).

Re: [dpdk-dev] Can xenvirt pmd work in xen guest (aka DomU) without xen-vhost in Dom0 ?

[Konrad Rzeszutek Wilk]

On Mon, Oct 09, 2017 at 12:13:47AM +0800, Tan, Jianfeng wrote:
> Hi,
> 
> 
> On 10/8/2017 12:54 PM, Bill Bonaparte wrote:
> > Thanks Jianfeng for taking time to reply.
> > 
> > please allow me to briefly explain why I want to run dpdk on xen.
> > our system is based on dpdk, which means we use dpdk as packet
> > receive/transmit engine,
> > and with integrated dpdk virtio/vmxnet3 driver, our system can run on
> > KVM/VMware platform .
> > this year, we have plan to run our system on AWS cloud, but I found that
> > AWS
> > uses xen as its virtualization platform, and the bus-info of nic is
> > vif-x (x could be 0,1,2...),
> > the driver used in kernel is vif. this should be para-virtualized nic
> > used on xen.
> 
> My guess is exactly as you describe. In AWS, we lack of a PMD for xen
> netfront (vif) nic. And even we got such a PMD, we still need a PMD for xen
> netback. Both are missing.
> 
> > 
> > I don't know which dpdk drvier can manage this pv nic. then I see
> > xenvirt, I think this driver can
> > did this job, like virtio can manage virtio nic which is used on kvm.
> > unfortunately, after some study work, I run testpmd successfully on xen,
> > but no packets received.
> > 
> > with the informain got from you, I know It's need to run vhost_xen at
> > dom0 so that xenvirt at domU can work.
> > but for my case, I have no change to run vhost_xen at dom0, because I
> > only can operate my own domU.
> > 
> > for this case, If I want to run system which is based on dpdk at domU,
> > what should I do?
> > appreciate any idea or suggestion from you.
> 
> What kind of performance are you seeking? Only accelerating the frontend by
> a new PMD, i.e. netfront, we can bypass the VM kernel (). But without
> accelerating the backend, it only brings limited improvement.
> 
> Not sure if anyone from Amazon or Oracle can comment?

Joao has been working on updates in the Xen PV network driver such
that there is a higher acceleration (aka recycle grants). See in xen.git
tree:
commit 30655e2bf0ce8ee50e61728e2558e0363f4af1c9
Author: Joao Martins <joao.m.martins@oracle.com>
Date:   Tue Oct 3 18:46:08 2017 +0100

    public/io/netif.h: add gref mapping control messages



This combined with the drivers that Broadcom wrote (I think that is
the name of the company that posted them a year or so ago), should
provide awesome performance.

> 
> Thanks,
> Jianfeng