[dpdk-users] pkts not transmitting with DPDK 2.2.0

[dpdk-users] pkts not transmitting with DPDK 2.2.0

[Toby DiPasquale]

Hi all,

I'm having an issue getting packets to actually transmit out of the
NIC with DPDK 2.2.0. I've built a simple UDP echo server here:
https://github.com/codeslinger/udpecho

Packets are received just fine, and they appear to say they are
transmitted, as well, but they never actually leave the NIC. Here is
some sample output with the details of how I'm running this:
https://gist.github.com/codeslinger/d2e59b00bdc1208f4369

I'm kinda stumped. I had a local person with DPDK experience look at
this code and he was similarly confused as to why it wasn't working.
Anyone have any ideas on this one? Thanks in advance!

-- 
Toby DiPasquale

Re: [dpdk-users] pkts not transmitting with DPDK 2.2.0

[Jesper Wramberg]

Hey Toby,

It seems you are using a VF yes ? I did some testing of SR-IOV a while ago
on an Intel NIC. I remember having to fiddle with it to get it working, so
here are a new notes/questions you might find useful (or maybe not :-)).

Are there any errors in the stats on the physical function (ethtool -S
<linux PF interface>) ?
Since you are receiving packets the MAC is probably correct. But anyway:
Did you check the MAC addresses of the VF using "ip link show <linux PF
interface>" - do they match what DPDK uses ?
Is spoof checking enabled on the PF ?

My tests were with an XL710 only using Linux (no DPDK). To be able to TX
from the VF after changing stuff with "ip link set..." I think I may have
had to reset the VF or PF using "ip link set .. down/up".

I know I'm not exactly providing any answers here - and my memory isn't
helping - but maybe something can point you in the right direction.

Regards,
Jesper


2016-03-26 20:24 GMT+01:00 Toby DiPasquale <toby@cbcg.net>:

> Hi all,
>
> I'm having an issue getting packets to actually transmit out of the
> NIC with DPDK 2.2.0. I've built a simple UDP echo server here:
> https://github.com/codeslinger/udpecho
>
> Packets are received just fine, and they appear to say they are
> transmitted, as well, but they never actually leave the NIC. Here is
> some sample output with the details of how I'm running this:
> https://gist.github.com/codeslinger/d2e59b00bdc1208f4369
>
> I'm kinda stumped. I had a local person with DPDK experience look at
> this code and he was similarly confused as to why it wasn't working.
> Anyone have any ideas on this one? Thanks in advance!
>
> --
> Toby DiPasquale
>

Re: [dpdk-users] pkts not transmitting with DPDK 2.2.0

[Toby DiPasquale]

Hi Jesper,

Neither ethtool nor ip return for this NIC:

  # ethtool -S eth1
  Cannot get stats strings information: No such device
  # ip link show eth1
  Device "eth1" does not exist.

The NIC has been disconnected from the OS via dpdk_nic_bind.py. The
MAC addresses do match, also. If the NIC is not visible to the OS, and
I'm not turning spoof checking on in the DPDK app, would the setting
while it is attached to the kernel make a difference?


On Sun, Mar 27, 2016 at 4:57 AM, Jesper Wramberg
<jesper.wramberg@gmail.com> wrote:
> Hey Toby,
>
> It seems you are using a VF yes ? I did some testing of SR-IOV a while ago
> on an Intel NIC. I remember having to fiddle with it to get it working, so
> here are a new notes/questions you might find useful (or maybe not :-)).
>
> Are there any errors in the stats on the physical function (ethtool -S
> <linux PF interface>) ?
> Since you are receiving packets the MAC is probably correct. But anyway:
> Did you check the MAC addresses of the VF using "ip link show <linux PF
> interface>" - do they match what DPDK uses ?
> Is spoof checking enabled on the PF ?
>
> My tests were with an XL710 only using Linux (no DPDK). To be able to TX
> from the VF after changing stuff with "ip link set..." I think I may have
> had to reset the VF or PF using "ip link set .. down/up".
>
> I know I'm not exactly providing any answers here - and my memory isn't
> helping - but maybe something can point you in the right direction.
>
> Regards,
> Jesper
>
>
> 2016-03-26 20:24 GMT+01:00 Toby DiPasquale <toby@cbcg.net>:
>>
>> Hi all,
>>
>> I'm having an issue getting packets to actually transmit out of the
>> NIC with DPDK 2.2.0. I've built a simple UDP echo server here:
>> https://github.com/codeslinger/udpecho
>>
>> Packets are received just fine, and they appear to say they are
>> transmitted, as well, but they never actually leave the NIC. Here is
>> some sample output with the details of how I'm running this:
>> https://gist.github.com/codeslinger/d2e59b00bdc1208f4369
>>
>> I'm kinda stumped. I had a local person with DPDK experience look at
>> this code and he was similarly confused as to why it wasn't working.
>> Anyone have any ideas on this one? Thanks in advance!
>>
>> --
>> Toby DiPasquale
>
>



-- 
Toby DiPasquale

Re: [dpdk-users] pkts not transmitting with DPDK 2.2.0

[Jesper Wramberg]

Hi,

I may have been a bit unclear in my previous mail. Are you using EC2 or
some kind of virtual server without access to the hypervisor ? The NIC you
are using seems to be a virtual PCI function (SR-IOV). I was assuming you
had access to the physical function (i.e. the actual NIC interface and not
the virtualized/limited one).

If you don't have access to any information or controls on the hypervisor I
guess we can only check xstats for errors through DPDK. But since the PMD
does not report any errors in your application my best guess is that the
physical function (the physical NIC) receives your packets from the
virtualized interface but discards them for some reason.

>From my (limited) experience, my guess is that your packets are dropped due
to MAC spoofcheck or some kind of IP anti-spoofing. I don't know anything
about EC2 but how do you obtain an IP for your network interface(s) in
Linux ? Can you use static IPs in Linux ? Also, what is the destination IP
of the packets you are receiving ? From my quick glance at your source it
seems you use the dest. IP as source IP when TXing. Since you set the port
in promisc. mode you could be using others IPs (although I don't think
promisc. does much in this particular case since you are using a virtual
function).

Maybe a simpler application could be used to isolate the error. You could
always try sending a DHCP discover or something that doesn't require an IP
(and use the source MAC reported by the rte_ethdev_macaddr_get() function.)

Hope it helps

2016-03-27 20:17 GMT+02:00 Toby DiPasquale <toby@cbcg.net>:

> Hi Jesper,
>
> Neither ethtool nor ip return for this NIC:
>
>   # ethtool -S eth1
>   Cannot get stats strings information: No such device
>   # ip link show eth1
>   Device "eth1" does not exist.
>
> The NIC has been disconnected from the OS via dpdk_nic_bind.py. The
> MAC addresses do match, also. If the NIC is not visible to the OS, and
> I'm not turning spoof checking on in the DPDK app, would the setting
> while it is attached to the kernel make a difference?
>
>
> On Sun, Mar 27, 2016 at 4:57 AM, Jesper Wramberg
> <jesper.wramberg@gmail.com> wrote:
> > Hey Toby,
> >
> > It seems you are using a VF yes ? I did some testing of SR-IOV a while
> ago
> > on an Intel NIC. I remember having to fiddle with it to get it working,
> so
> > here are a new notes/questions you might find useful (or maybe not :-)).
> >
> > Are there any errors in the stats on the physical function (ethtool -S
> > <linux PF interface>) ?
> > Since you are receiving packets the MAC is probably correct. But anyway:
> > Did you check the MAC addresses of the VF using "ip link show <linux PF
> > interface>" - do they match what DPDK uses ?
> > Is spoof checking enabled on the PF ?
> >
> > My tests were with an XL710 only using Linux (no DPDK). To be able to TX
> > from the VF after changing stuff with "ip link set..." I think I may have
> > had to reset the VF or PF using "ip link set .. down/up".
> >
> > I know I'm not exactly providing any answers here - and my memory isn't
> > helping - but maybe something can point you in the right direction.
> >
> > Regards,
> > Jesper
> >
> >
> > 2016-03-26 20:24 GMT+01:00 Toby DiPasquale <toby@cbcg.net>:
> >>
> >> Hi all,
> >>
> >> I'm having an issue getting packets to actually transmit out of the
> >> NIC with DPDK 2.2.0. I've built a simple UDP echo server here:
> >> https://github.com/codeslinger/udpecho
> >>
> >> Packets are received just fine, and they appear to say they are
> >> transmitted, as well, but they never actually leave the NIC. Here is
> >> some sample output with the details of how I'm running this:
> >> https://gist.github.com/codeslinger/d2e59b00bdc1208f4369
> >>
> >> I'm kinda stumped. I had a local person with DPDK experience look at
> >> this code and he was similarly confused as to why it wasn't working.
> >> Anyone have any ideas on this one? Thanks in advance!
> >>
> >> --
> >> Toby DiPasquale
> >
> >
>
>
>
> --
> Toby DiPasquale
>

Re: [dpdk-users] pkts not transmitting with DPDK 2.2.0

[John Boyle]

Look at the output of "dmesg", specifically for messages printed by ixgbe.
I've encountered two problems of the sort "DPDK thinks packets were
successfully sent, but the packets never left the NIC", and both would
manifest as some errors in dmesg.  One particularly common failure mode was
due to the spoof checking Jesper mentions, where the NIC will drop packets
where the source MAC doesn't match the VF's MAC.  In that case, "dmesg"
will print lines like this:

[92990.018296] ixgbe 0000:01:00.1 p5p2: 2 Spoofed packets detected

-- John Boyle
*Science is what we understand well enough to explain to a computer. Art is
everything else we do.* --Knuth

On Sun, Mar 27, 2016 at 1:19 PM, Jesper Wramberg <jesper.wramberg@gmail.com>
wrote:

> Hi,
>
> I may have been a bit unclear in my previous mail. Are you using EC2 or
> some kind of virtual server without access to the hypervisor ? The NIC you
> are using seems to be a virtual PCI function (SR-IOV). I was assuming you
> had access to the physical function (i.e. the actual NIC interface and not
> the virtualized/limited one).
>
> If you don't have access to any information or controls on the hypervisor I
> guess we can only check xstats for errors through DPDK. But since the PMD
> does not report any errors in your application my best guess is that the
> physical function (the physical NIC) receives your packets from the
> virtualized interface but discards them for some reason.
>
> From my (limited) experience, my guess is that your packets are dropped due
> to MAC spoofcheck or some kind of IP anti-spoofing. I don't know anything
> about EC2 but how do you obtain an IP for your network interface(s) in
> Linux ? Can you use static IPs in Linux ? Also, what is the destination IP
> of the packets you are receiving ? From my quick glance at your source it
> seems you use the dest. IP as source IP when TXing. Since you set the port
> in promisc. mode you could be using others IPs (although I don't think
> promisc. does much in this particular case since you are using a virtual
> function).
>
> Maybe a simpler application could be used to isolate the error. You could
> always try sending a DHCP discover or something that doesn't require an IP
> (and use the source MAC reported by the rte_ethdev_macaddr_get() function.)
>
> Hope it helps
>
> 2016-03-27 20:17 GMT+02:00 Toby DiPasquale <toby@cbcg.net>:
>
> > Hi Jesper,
> >
> > Neither ethtool nor ip return for this NIC:
> >
> >   # ethtool -S eth1
> >   Cannot get stats strings information: No such device
> >   # ip link show eth1
> >   Device "eth1" does not exist.
> >
> > The NIC has been disconnected from the OS via dpdk_nic_bind.py. The
> > MAC addresses do match, also. If the NIC is not visible to the OS, and
> > I'm not turning spoof checking on in the DPDK app, would the setting
> > while it is attached to the kernel make a difference?
> >
> >
> > On Sun, Mar 27, 2016 at 4:57 AM, Jesper Wramberg
> > <jesper.wramberg@gmail.com> wrote:
> > > Hey Toby,
> > >
> > > It seems you are using a VF yes ? I did some testing of SR-IOV a while
> > ago
> > > on an Intel NIC. I remember having to fiddle with it to get it working,
> > so
> > > here are a new notes/questions you might find useful (or maybe not
> :-)).
> > >
> > > Are there any errors in the stats on the physical function (ethtool -S
> > > <linux PF interface>) ?
> > > Since you are receiving packets the MAC is probably correct. But
> anyway:
> > > Did you check the MAC addresses of the VF using "ip link show <linux PF
> > > interface>" - do they match what DPDK uses ?
> > > Is spoof checking enabled on the PF ?
> > >
> > > My tests were with an XL710 only using Linux (no DPDK). To be able to
> TX
> > > from the VF after changing stuff with "ip link set..." I think I may
> have
> > > had to reset the VF or PF using "ip link set .. down/up".
> > >
> > > I know I'm not exactly providing any answers here - and my memory isn't
> > > helping - but maybe something can point you in the right direction.
> > >
> > > Regards,
> > > Jesper
> > >
> > >
> > > 2016-03-26 20:24 GMT+01:00 Toby DiPasquale <toby@cbcg.net>:
> > >>
> > >> Hi all,
> > >>
> > >> I'm having an issue getting packets to actually transmit out of the
> > >> NIC with DPDK 2.2.0. I've built a simple UDP echo server here:
> > >> https://github.com/codeslinger/udpecho
> > >>
> > >> Packets are received just fine, and they appear to say they are
> > >> transmitted, as well, but they never actually leave the NIC. Here is
> > >> some sample output with the details of how I'm running this:
> > >> https://gist.github.com/codeslinger/d2e59b00bdc1208f4369
> > >>
> > >> I'm kinda stumped. I had a local person with DPDK experience look at
> > >> this code and he was similarly confused as to why it wasn't working.
> > >> Anyone have any ideas on this one? Thanks in advance!
> > >>
> > >> --
> > >> Toby DiPasquale
> > >
> > >
> >
> >
> >
> > --
> > Toby DiPasquale
> >
>