[dpdk-dev] Trouble with Tx on some i40 ports in KVM VM

[dpdk-dev] Trouble with Tx on some i40 ports in KVM VM

[Andrew Theurer]

Hello.  I am having some trouble with getting various dpdk applications to work when using in a KVM VM.  This happens with the 2nd port on a dual port adapter.  What I have configured:  

Haswell-ep host with KVM, 2 x i40e phys functions bound to vfio-pci, 2 functions assigned to KVM guest.  In the guest, the i40 functions bound to uio_pci_generic, running testpmd with these options: --nb-cores=2 --nb-ports=2 --portmask=3 --interactive --auto-start.  

On another host directly connected to these adapters, I run pktgen on the 2 i40e ports.  If I Tx on port1, I get packets back on port0.  If I Tx on port0, no packets come back on port0.  For some reason Tx simply does not happen on that second port.

I have tried this test where there is no KVM, just using host, and it works as expected.  I have also tried this test with ixgbe adapter in a KVM VM (I have both in same system), and it works as expected.  I have also tried just using a bridge in the VM with i40e ports, with i40e module in use, and that also works as expected.  So, I don't think this has anything to do with the adapters themselves, or cables, etc.  Something is not working when using the KVM VM, I just don't know what it is.

Here is xstats from testpmd:
###### NIC extended statistics for port 1
rx_packets: 134615456
tx_packets: 0
rx_bytes: 8615389184
tx_bytes: 0
tx_errors: 0
rx_missed_errors: 0
rx_crc_errors: 0
rx_bad_length_errors: 0
rx_errors: 0
alloc_rx_buff_failed: 0
fdir_match: 0
fdir_miss: 0
tx_flow_control_xon: 0
rx_flow_control_xon: 0
tx_flow_control_xoff: 0
rx_flow_control_xoff: 0
rx_queue_0_rx_packets: 0
rx_queue_0_rx_bytes: 0
tx_queue_0_tx_packets: 0
tx_queue_0_tx_bytes: 0
tx_queue_0_tx_errors: 0

I can't find any evidence of errors anywhere.  It's just not doing Tx at all on that port.  I have also tried using ptkgen in the VM, to manually send packets on that port, and again, no packets sent, but no errors either.

Has anyone else tried 2 i40e ports in a KVM VM?  Any ideas what could be going on here?

Thanks,

-Andrew

Re: [dpdk-dev] Trouble with Tx on some i40 ports in KVM VM

[Zhou, Danny]

On your system, what is your iommu and intel_iommu kernel parameters? 

Could you try to use uio_pci_generic for both PF on host and VF in guest after configuring "iommu=pt" on your system?

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Andrew Theurer
> Sent: Wednesday, June 10, 2015 5:48 AM
> To: dev@dpdk.org
> Subject: [dpdk-dev] Trouble with Tx on some i40 ports in KVM VM
> 
> Hello.  I am having some trouble with getting various dpdk applications to work when using in a KVM VM.  This happens
> with the 2nd port on a dual port adapter.  What I have configured:
> 
> Haswell-ep host with KVM, 2 x i40e phys functions bound to vfio-pci, 2 functions assigned to KVM guest.  In the guest, the
> i40 functions bound to uio_pci_generic, running testpmd with these options: --nb-cores=2 --nb-ports=2 --portmask=3
> --interactive --auto-start.
> 
> On another host directly connected to these adapters, I run pktgen on the 2 i40e ports.  If I Tx on port1, I get packets back
> on port0.  If I Tx on port0, no packets come back on port0.  For some reason Tx simply does not happen on that second
> port.
> 
> I have tried this test where there is no KVM, just using host, and it works as expected.  I have also tried this test with ixgbe
> adapter in a KVM VM (I have both in same system), and it works as expected.  I have also tried just using a bridge in the VM
> with i40e ports, with i40e module in use, and that also works as expected.  So, I don't think this has anything to do with the
> adapters themselves, or cables, etc.  Something is not working when using the KVM VM, I just don't know what it is.
> 
> Here is xstats from testpmd:
> ###### NIC extended statistics for port 1
> rx_packets: 134615456
> tx_packets: 0
> rx_bytes: 8615389184
> tx_bytes: 0
> tx_errors: 0
> rx_missed_errors: 0
> rx_crc_errors: 0
> rx_bad_length_errors: 0
> rx_errors: 0
> alloc_rx_buff_failed: 0
> fdir_match: 0
> fdir_miss: 0
> tx_flow_control_xon: 0
> rx_flow_control_xon: 0
> tx_flow_control_xoff: 0
> rx_flow_control_xoff: 0
> rx_queue_0_rx_packets: 0
> rx_queue_0_rx_bytes: 0
> tx_queue_0_tx_packets: 0
> tx_queue_0_tx_bytes: 0
> tx_queue_0_tx_errors: 0
> 
> I can't find any evidence of errors anywhere.  It's just not doing Tx at all on that port.  I have also tried using ptkgen in the
> VM, to manually send packets on that port, and again, no packets sent, but no errors either.
> 
> Has anyone else tried 2 i40e ports in a KVM VM?  Any ideas what could be going on here?
> 
> Thanks,
> 
> -Andrew

Re: [dpdk-dev] Trouble with Tx on some i40 ports in KVM VM

[Andrew Theurer]

----- Original Message -----
> From: "Danny Zhou" <danny.zhou@intel.com>
> To: "Andrew Theurer" <atheurer@redhat.com>
> Cc: dev@dpdk.org
> Sent: Tuesday, June 9, 2015 6:38:49 PM
> Subject: RE: Trouble with Tx on some i40 ports in KVM VM
> 
> On your system, what is your iommu and intel_iommu kernel parameters?
> 
> Could you try to use uio_pci_generic for both PF on host and VF in guest
> after configuring "iommu=pt" on your system?

I use iommu=pt intel_iommu=on

To clarify, I am not using a VF in the VM.  I am only using PF's, assigning them to the VM.  The host needs vfio module for kvm to assign these to the VM.
I don't see how uio_pci_generic can be used in the host if the PF is being passed to the VM.

Thanks,

-Andrew
> 
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Andrew Theurer
> > Sent: Wednesday, June 10, 2015 5:48 AM
> > To: dev@dpdk.org
> > Subject: [dpdk-dev] Trouble with Tx on some i40 ports in KVM VM
> > 
> > Hello.  I am having some trouble with getting various dpdk applications to
> > work when using in a KVM VM.  This happens
> > with the 2nd port on a dual port adapter.  What I have configured:
> > 
> > Haswell-ep host with KVM, 2 x i40e phys functions bound to vfio-pci, 2
> > functions assigned to KVM guest.  In the guest, the
> > i40 functions bound to uio_pci_generic, running testpmd with these options:
> > --nb-cores=2 --nb-ports=2 --portmask=3
> > --interactive --auto-start.
> > 
> > On another host directly connected to these adapters, I run pktgen on the 2
> > i40e ports.  If I Tx on port1, I get packets back
> > on port0.  If I Tx on port0, no packets come back on port0.  For some
> > reason Tx simply does not happen on that second
> > port.
> > 
> > I have tried this test where there is no KVM, just using host, and it works
> > as expected.  I have also tried this test with ixgbe
> > adapter in a KVM VM (I have both in same system), and it works as expected.
> > I have also tried just using a bridge in the VM
> > with i40e ports, with i40e module in use, and that also works as expected.
> > So, I don't think this has anything to do with the
> > adapters themselves, or cables, etc.  Something is not working when using
> > the KVM VM, I just don't know what it is.
> > 
> > Here is xstats from testpmd:
> > ###### NIC extended statistics for port 1
> > rx_packets: 134615456
> > tx_packets: 0
> > rx_bytes: 8615389184
> > tx_bytes: 0
> > tx_errors: 0
> > rx_missed_errors: 0
> > rx_crc_errors: 0
> > rx_bad_length_errors: 0
> > rx_errors: 0
> > alloc_rx_buff_failed: 0
> > fdir_match: 0
> > fdir_miss: 0
> > tx_flow_control_xon: 0
> > rx_flow_control_xon: 0
> > tx_flow_control_xoff: 0
> > rx_flow_control_xoff: 0
> > rx_queue_0_rx_packets: 0
> > rx_queue_0_rx_bytes: 0
> > tx_queue_0_tx_packets: 0
> > tx_queue_0_tx_bytes: 0
> > tx_queue_0_tx_errors: 0
> > 
> > I can't find any evidence of errors anywhere.  It's just not doing Tx at
> > all on that port.  I have also tried using ptkgen in the
> > VM, to manually send packets on that port, and again, no packets sent, but
> > no errors either.
> > 
> > Has anyone else tried 2 i40e ports in a KVM VM?  Any ideas what could be
> > going on here?
> > 
> > Thanks,
> > 
> > -Andrew
> 

Re: [dpdk-dev] Trouble with Tx on some i40 ports in KVM VM

[Zhang, Helin]

Hi Andrew

So you are using PF pass-through?
What version of DPDK were you using? You need to use try the latest DPDK on
dpdk.org master branch, as there is a bug fix recently which might be helpful
for using PF pass-through.
Please try the latest and see if the issue is still there or not!

Regards,
Helin

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Andrew Theurer
> Sent: Wednesday, June 10, 2015 8:47 AM
> To: Zhou, Danny
> Cc: dev@dpdk.org
> Subject: Re: [dpdk-dev] Trouble with Tx on some i40 ports in KVM VM
> 
> 
> 
> ----- Original Message -----
> > From: "Danny Zhou" <danny.zhou@intel.com>
> > To: "Andrew Theurer" <atheurer@redhat.com>
> > Cc: dev@dpdk.org
> > Sent: Tuesday, June 9, 2015 6:38:49 PM
> > Subject: RE: Trouble with Tx on some i40 ports in KVM VM
> >
> > On your system, what is your iommu and intel_iommu kernel parameters?
> >
> > Could you try to use uio_pci_generic for both PF on host and VF in
> > guest after configuring "iommu=pt" on your system?
> 
> I use iommu=pt intel_iommu=on
> 
> To clarify, I am not using a VF in the VM.  I am only using PF's, assigning them to
> the VM.  The host needs vfio module for kvm to assign these to the VM.
> I don't see how uio_pci_generic can be used in the host if the PF is being passed
> to the VM.
> 
> Thanks,
> 
> -Andrew
> >
> > > -----Original Message-----
> > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Andrew Theurer
> > > Sent: Wednesday, June 10, 2015 5:48 AM
> > > To: dev@dpdk.org
> > > Subject: [dpdk-dev] Trouble with Tx on some i40 ports in KVM VM
> > >
> > > Hello.  I am having some trouble with getting various dpdk
> > > applications to work when using in a KVM VM.  This happens with the
> > > 2nd port on a dual port adapter.  What I have configured:
> > >
> > > Haswell-ep host with KVM, 2 x i40e phys functions bound to vfio-pci,
> > > 2 functions assigned to KVM guest.  In the guest, the
> > > i40 functions bound to uio_pci_generic, running testpmd with these options:
> > > --nb-cores=2 --nb-ports=2 --portmask=3 --interactive --auto-start.
> > >
> > > On another host directly connected to these adapters, I run pktgen
> > > on the 2 i40e ports.  If I Tx on port1, I get packets back on port0.
> > > If I Tx on port0, no packets come back on port0.  For some reason Tx
> > > simply does not happen on that second port.
> > >
> > > I have tried this test where there is no KVM, just using host, and
> > > it works as expected.  I have also tried this test with ixgbe
> > > adapter in a KVM VM (I have both in same system), and it works as expected.
> > > I have also tried just using a bridge in the VM with i40e ports,
> > > with i40e module in use, and that also works as expected.
> > > So, I don't think this has anything to do with the adapters
> > > themselves, or cables, etc.  Something is not working when using the
> > > KVM VM, I just don't know what it is.
> > >
> > > Here is xstats from testpmd:
> > > ###### NIC extended statistics for port 1
> > > rx_packets: 134615456
> > > tx_packets: 0
> > > rx_bytes: 8615389184
> > > tx_bytes: 0
> > > tx_errors: 0
> > > rx_missed_errors: 0
> > > rx_crc_errors: 0
> > > rx_bad_length_errors: 0
> > > rx_errors: 0
> > > alloc_rx_buff_failed: 0
> > > fdir_match: 0
> > > fdir_miss: 0
> > > tx_flow_control_xon: 0
> > > rx_flow_control_xon: 0
> > > tx_flow_control_xoff: 0
> > > rx_flow_control_xoff: 0
> > > rx_queue_0_rx_packets: 0
> > > rx_queue_0_rx_bytes: 0
> > > tx_queue_0_tx_packets: 0
> > > tx_queue_0_tx_bytes: 0
> > > tx_queue_0_tx_errors: 0
> > >
> > > I can't find any evidence of errors anywhere.  It's just not doing
> > > Tx at all on that port.  I have also tried using ptkgen in the VM,
> > > to manually send packets on that port, and again, no packets sent,
> > > but no errors either.
> > >
> > > Has anyone else tried 2 i40e ports in a KVM VM?  Any ideas what
> > > could be going on here?
> > >
> > > Thanks,
> > >
> > > -Andrew
> >

Re: [dpdk-dev] Trouble with Tx on some i40 ports in KVM VM

[Andrew Theurer]

----- Original Message -----
> From: "Helin Zhang" <helin.zhang@intel.com>
> To: "Andrew Theurer" <atheurer@redhat.com>, "Danny Zhou" <danny.zhou@intel.com>
> Cc: dev@dpdk.org
> Sent: Wednesday, June 10, 2015 8:59:20 PM
> Subject: RE: Trouble with Tx on some i40 ports in KVM VM
> 
> Hi Andrew
> 
> So you are using PF pass-through?
> What version of DPDK were you using? You need to use try the latest DPDK on
> dpdk.org master branch, as there is a bug fix recently which might be helpful
> for using PF pass-through.
> Please try the latest and see if the issue is still there or not!

I have tried git master branch, and it works!  Thanks for your help.

-Andrew
> 
> Regards,
> Helin
> 
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Andrew Theurer
> > Sent: Wednesday, June 10, 2015 8:47 AM
> > To: Zhou, Danny
> > Cc: dev@dpdk.org
> > Subject: Re: [dpdk-dev] Trouble with Tx on some i40 ports in KVM VM
> > 
> > 
> > 
> > ----- Original Message -----
> > > From: "Danny Zhou" <danny.zhou@intel.com>
> > > To: "Andrew Theurer" <atheurer@redhat.com>
> > > Cc: dev@dpdk.org
> > > Sent: Tuesday, June 9, 2015 6:38:49 PM
> > > Subject: RE: Trouble with Tx on some i40 ports in KVM VM
> > >
> > > On your system, what is your iommu and intel_iommu kernel parameters?
> > >
> > > Could you try to use uio_pci_generic for both PF on host and VF in
> > > guest after configuring "iommu=pt" on your system?
> > 
> > I use iommu=pt intel_iommu=on
> > 
> > To clarify, I am not using a VF in the VM.  I am only using PF's, assigning
> > them to
> > the VM.  The host needs vfio module for kvm to assign these to the VM.
> > I don't see how uio_pci_generic can be used in the host if the PF is being
> > passed
> > to the VM.
> > 
> > Thanks,
> > 
> > -Andrew
> > >
> > > > -----Original Message-----
> > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Andrew Theurer
> > > > Sent: Wednesday, June 10, 2015 5:48 AM
> > > > To: dev@dpdk.org
> > > > Subject: [dpdk-dev] Trouble with Tx on some i40 ports in KVM VM
> > > >
> > > > Hello.  I am having some trouble with getting various dpdk
> > > > applications to work when using in a KVM VM.  This happens with the
> > > > 2nd port on a dual port adapter.  What I have configured:
> > > >
> > > > Haswell-ep host with KVM, 2 x i40e phys functions bound to vfio-pci,
> > > > 2 functions assigned to KVM guest.  In the guest, the
> > > > i40 functions bound to uio_pci_generic, running testpmd with these
> > > > options:
> > > > --nb-cores=2 --nb-ports=2 --portmask=3 --interactive --auto-start.
> > > >
> > > > On another host directly connected to these adapters, I run pktgen
> > > > on the 2 i40e ports.  If I Tx on port1, I get packets back on port0.
> > > > If I Tx on port0, no packets come back on port0.  For some reason Tx
> > > > simply does not happen on that second port.
> > > >
> > > > I have tried this test where there is no KVM, just using host, and
> > > > it works as expected.  I have also tried this test with ixgbe
> > > > adapter in a KVM VM (I have both in same system), and it works as
> > > > expected.
> > > > I have also tried just using a bridge in the VM with i40e ports,
> > > > with i40e module in use, and that also works as expected.
> > > > So, I don't think this has anything to do with the adapters
> > > > themselves, or cables, etc.  Something is not working when using the
> > > > KVM VM, I just don't know what it is.
> > > >
> > > > Here is xstats from testpmd:
> > > > ###### NIC extended statistics for port 1
> > > > rx_packets: 134615456
> > > > tx_packets: 0
> > > > rx_bytes: 8615389184
> > > > tx_bytes: 0
> > > > tx_errors: 0
> > > > rx_missed_errors: 0
> > > > rx_crc_errors: 0
> > > > rx_bad_length_errors: 0
> > > > rx_errors: 0
> > > > alloc_rx_buff_failed: 0
> > > > fdir_match: 0
> > > > fdir_miss: 0
> > > > tx_flow_control_xon: 0
> > > > rx_flow_control_xon: 0
> > > > tx_flow_control_xoff: 0
> > > > rx_flow_control_xoff: 0
> > > > rx_queue_0_rx_packets: 0
> > > > rx_queue_0_rx_bytes: 0
> > > > tx_queue_0_tx_packets: 0
> > > > tx_queue_0_tx_bytes: 0
> > > > tx_queue_0_tx_errors: 0
> > > >
> > > > I can't find any evidence of errors anywhere.  It's just not doing
> > > > Tx at all on that port.  I have also tried using ptkgen in the VM,
> > > > to manually send packets on that port, and again, no packets sent,
> > > > but no errors either.
> > > >
> > > > Has anyone else tried 2 i40e ports in a KVM VM?  Any ideas what
> > > > could be going on here?
> > > >
> > > > Thanks,
> > > >
> > > > -Andrew
> > >
> 

Re: [dpdk-dev] Trouble with Tx on some i40 ports in KVM VM

[Zhang, Helin]

Good to know that!

- Helin

> -----Original Message-----
> From: Andrew Theurer [mailto:atheurer@redhat.com]
> Sent: Thursday, June 11, 2015 10:26 PM
> To: Zhang, Helin
> Cc: Zhou, Danny; dev@dpdk.org
> Subject: Re: Trouble with Tx on some i40 ports in KVM VM
> 
> 
> 
> ----- Original Message -----
> > From: "Helin Zhang" <helin.zhang@intel.com>
> > To: "Andrew Theurer" <atheurer@redhat.com>, "Danny Zhou"
> > <danny.zhou@intel.com>
> > Cc: dev@dpdk.org
> > Sent: Wednesday, June 10, 2015 8:59:20 PM
> > Subject: RE: Trouble with Tx on some i40 ports in KVM VM
> >
> > Hi Andrew
> >
> > So you are using PF pass-through?
> > What version of DPDK were you using? You need to use try the latest
> > DPDK on dpdk.org master branch, as there is a bug fix recently which
> > might be helpful for using PF pass-through.
> > Please try the latest and see if the issue is still there or not!
> 
> I have tried git master branch, and it works!  Thanks for your help.
> 
> -Andrew
> >
> > Regards,
> > Helin
> >
> > > -----Original Message-----
> > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Andrew Theurer
> > > Sent: Wednesday, June 10, 2015 8:47 AM
> > > To: Zhou, Danny
> > > Cc: dev@dpdk.org
> > > Subject: Re: [dpdk-dev] Trouble with Tx on some i40 ports in KVM VM
> > >
> > >
> > >
> > > ----- Original Message -----
> > > > From: "Danny Zhou" <danny.zhou@intel.com>
> > > > To: "Andrew Theurer" <atheurer@redhat.com>
> > > > Cc: dev@dpdk.org
> > > > Sent: Tuesday, June 9, 2015 6:38:49 PM
> > > > Subject: RE: Trouble with Tx on some i40 ports in KVM VM
> > > >
> > > > On your system, what is your iommu and intel_iommu kernel parameters?
> > > >
> > > > Could you try to use uio_pci_generic for both PF on host and VF in
> > > > guest after configuring "iommu=pt" on your system?
> > >
> > > I use iommu=pt intel_iommu=on
> > >
> > > To clarify, I am not using a VF in the VM.  I am only using PF's,
> > > assigning them to the VM.  The host needs vfio module for kvm to
> > > assign these to the VM.
> > > I don't see how uio_pci_generic can be used in the host if the PF is
> > > being passed to the VM.
> > >
> > > Thanks,
> > >
> > > -Andrew
> > > >
> > > > > -----Original Message-----
> > > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Andrew
> > > > > Theurer
> > > > > Sent: Wednesday, June 10, 2015 5:48 AM
> > > > > To: dev@dpdk.org
> > > > > Subject: [dpdk-dev] Trouble with Tx on some i40 ports in KVM VM
> > > > >
> > > > > Hello.  I am having some trouble with getting various dpdk
> > > > > applications to work when using in a KVM VM.  This happens with
> > > > > the 2nd port on a dual port adapter.  What I have configured:
> > > > >
> > > > > Haswell-ep host with KVM, 2 x i40e phys functions bound to
> > > > > vfio-pci,
> > > > > 2 functions assigned to KVM guest.  In the guest, the
> > > > > i40 functions bound to uio_pci_generic, running testpmd with
> > > > > these
> > > > > options:
> > > > > --nb-cores=2 --nb-ports=2 --portmask=3 --interactive --auto-start.
> > > > >
> > > > > On another host directly connected to these adapters, I run
> > > > > pktgen on the 2 i40e ports.  If I Tx on port1, I get packets back on port0.
> > > > > If I Tx on port0, no packets come back on port0.  For some
> > > > > reason Tx simply does not happen on that second port.
> > > > >
> > > > > I have tried this test where there is no KVM, just using host,
> > > > > and it works as expected.  I have also tried this test with
> > > > > ixgbe adapter in a KVM VM (I have both in same system), and it
> > > > > works as expected.
> > > > > I have also tried just using a bridge in the VM with i40e ports,
> > > > > with i40e module in use, and that also works as expected.
> > > > > So, I don't think this has anything to do with the adapters
> > > > > themselves, or cables, etc.  Something is not working when using
> > > > > the KVM VM, I just don't know what it is.
> > > > >
> > > > > Here is xstats from testpmd:
> > > > > ###### NIC extended statistics for port 1
> > > > > rx_packets: 134615456
> > > > > tx_packets: 0
> > > > > rx_bytes: 8615389184
> > > > > tx_bytes: 0
> > > > > tx_errors: 0
> > > > > rx_missed_errors: 0
> > > > > rx_crc_errors: 0
> > > > > rx_bad_length_errors: 0
> > > > > rx_errors: 0
> > > > > alloc_rx_buff_failed: 0
> > > > > fdir_match: 0
> > > > > fdir_miss: 0
> > > > > tx_flow_control_xon: 0
> > > > > rx_flow_control_xon: 0
> > > > > tx_flow_control_xoff: 0
> > > > > rx_flow_control_xoff: 0
> > > > > rx_queue_0_rx_packets: 0
> > > > > rx_queue_0_rx_bytes: 0
> > > > > tx_queue_0_tx_packets: 0
> > > > > tx_queue_0_tx_bytes: 0
> > > > > tx_queue_0_tx_errors: 0
> > > > >
> > > > > I can't find any evidence of errors anywhere.  It's just not
> > > > > doing Tx at all on that port.  I have also tried using ptkgen in
> > > > > the VM, to manually send packets on that port, and again, no
> > > > > packets sent, but no errors either.
> > > > >
> > > > > Has anyone else tried 2 i40e ports in a KVM VM?  Any ideas what
> > > > > could be going on here?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > -Andrew
> > > >
> >