* [dpdk-users] pkts not transmitting with DPDK 2.2.0 @ 2016-03-26 19:24 Toby DiPasquale 2016-03-27 8:57 ` Jesper Wramberg 0 siblings, 1 reply; 5+ messages in thread From: Toby DiPasquale @ 2016-03-26 19:24 UTC (permalink / raw) To: users Hi all, I'm having an issue getting packets to actually transmit out of the NIC with DPDK 2.2.0. I've built a simple UDP echo server here: https://github.com/codeslinger/udpecho Packets are received just fine, and they appear to say they are transmitted, as well, but they never actually leave the NIC. Here is some sample output with the details of how I'm running this: https://gist.github.com/codeslinger/d2e59b00bdc1208f4369 I'm kinda stumped. I had a local person with DPDK experience look at this code and he was similarly confused as to why it wasn't working. Anyone have any ideas on this one? Thanks in advance! -- Toby DiPasquale ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [dpdk-users] pkts not transmitting with DPDK 2.2.0 2016-03-26 19:24 [dpdk-users] pkts not transmitting with DPDK 2.2.0 Toby DiPasquale @ 2016-03-27 8:57 ` Jesper Wramberg 2016-03-27 18:17 ` Toby DiPasquale 0 siblings, 1 reply; 5+ messages in thread From: Jesper Wramberg @ 2016-03-27 8:57 UTC (permalink / raw) To: Toby DiPasquale; +Cc: users Hey Toby, It seems you are using a VF yes ? I did some testing of SR-IOV a while ago on an Intel NIC. I remember having to fiddle with it to get it working, so here are a new notes/questions you might find useful (or maybe not :-)). Are there any errors in the stats on the physical function (ethtool -S <linux PF interface>) ? Since you are receiving packets the MAC is probably correct. But anyway: Did you check the MAC addresses of the VF using "ip link show <linux PF interface>" - do they match what DPDK uses ? Is spoof checking enabled on the PF ? My tests were with an XL710 only using Linux (no DPDK). To be able to TX from the VF after changing stuff with "ip link set..." I think I may have had to reset the VF or PF using "ip link set .. down/up". I know I'm not exactly providing any answers here - and my memory isn't helping - but maybe something can point you in the right direction. Regards, Jesper 2016-03-26 20:24 GMT+01:00 Toby DiPasquale <toby@cbcg.net>: > Hi all, > > I'm having an issue getting packets to actually transmit out of the > NIC with DPDK 2.2.0. I've built a simple UDP echo server here: > https://github.com/codeslinger/udpecho > > Packets are received just fine, and they appear to say they are > transmitted, as well, but they never actually leave the NIC. Here is > some sample output with the details of how I'm running this: > https://gist.github.com/codeslinger/d2e59b00bdc1208f4369 > > I'm kinda stumped. I had a local person with DPDK experience look at > this code and he was similarly confused as to why it wasn't working. > Anyone have any ideas on this one? Thanks in advance! > > -- > Toby DiPasquale > ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [dpdk-users] pkts not transmitting with DPDK 2.2.0 2016-03-27 8:57 ` Jesper Wramberg @ 2016-03-27 18:17 ` Toby DiPasquale 2016-03-27 20:19 ` Jesper Wramberg 0 siblings, 1 reply; 5+ messages in thread From: Toby DiPasquale @ 2016-03-27 18:17 UTC (permalink / raw) To: Jesper Wramberg; +Cc: users Hi Jesper, Neither ethtool nor ip return for this NIC: # ethtool -S eth1 Cannot get stats strings information: No such device # ip link show eth1 Device "eth1" does not exist. The NIC has been disconnected from the OS via dpdk_nic_bind.py. The MAC addresses do match, also. If the NIC is not visible to the OS, and I'm not turning spoof checking on in the DPDK app, would the setting while it is attached to the kernel make a difference? On Sun, Mar 27, 2016 at 4:57 AM, Jesper Wramberg <jesper.wramberg@gmail.com> wrote: > Hey Toby, > > It seems you are using a VF yes ? I did some testing of SR-IOV a while ago > on an Intel NIC. I remember having to fiddle with it to get it working, so > here are a new notes/questions you might find useful (or maybe not :-)). > > Are there any errors in the stats on the physical function (ethtool -S > <linux PF interface>) ? > Since you are receiving packets the MAC is probably correct. But anyway: > Did you check the MAC addresses of the VF using "ip link show <linux PF > interface>" - do they match what DPDK uses ? > Is spoof checking enabled on the PF ? > > My tests were with an XL710 only using Linux (no DPDK). To be able to TX > from the VF after changing stuff with "ip link set..." I think I may have > had to reset the VF or PF using "ip link set .. down/up". > > I know I'm not exactly providing any answers here - and my memory isn't > helping - but maybe something can point you in the right direction. > > Regards, > Jesper > > > 2016-03-26 20:24 GMT+01:00 Toby DiPasquale <toby@cbcg.net>: >> >> Hi all, >> >> I'm having an issue getting packets to actually transmit out of the >> NIC with DPDK 2.2.0. I've built a simple UDP echo server here: >> https://github.com/codeslinger/udpecho >> >> Packets are received just fine, and they appear to say they are >> transmitted, as well, but they never actually leave the NIC. Here is >> some sample output with the details of how I'm running this: >> https://gist.github.com/codeslinger/d2e59b00bdc1208f4369 >> >> I'm kinda stumped. I had a local person with DPDK experience look at >> this code and he was similarly confused as to why it wasn't working. >> Anyone have any ideas on this one? Thanks in advance! >> >> -- >> Toby DiPasquale > > -- Toby DiPasquale ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [dpdk-users] pkts not transmitting with DPDK 2.2.0 2016-03-27 18:17 ` Toby DiPasquale @ 2016-03-27 20:19 ` Jesper Wramberg 2016-03-28 1:02 ` John Boyle 0 siblings, 1 reply; 5+ messages in thread From: Jesper Wramberg @ 2016-03-27 20:19 UTC (permalink / raw) To: Toby DiPasquale; +Cc: users Hi, I may have been a bit unclear in my previous mail. Are you using EC2 or some kind of virtual server without access to the hypervisor ? The NIC you are using seems to be a virtual PCI function (SR-IOV). I was assuming you had access to the physical function (i.e. the actual NIC interface and not the virtualized/limited one). If you don't have access to any information or controls on the hypervisor I guess we can only check xstats for errors through DPDK. But since the PMD does not report any errors in your application my best guess is that the physical function (the physical NIC) receives your packets from the virtualized interface but discards them for some reason. >From my (limited) experience, my guess is that your packets are dropped due to MAC spoofcheck or some kind of IP anti-spoofing. I don't know anything about EC2 but how do you obtain an IP for your network interface(s) in Linux ? Can you use static IPs in Linux ? Also, what is the destination IP of the packets you are receiving ? From my quick glance at your source it seems you use the dest. IP as source IP when TXing. Since you set the port in promisc. mode you could be using others IPs (although I don't think promisc. does much in this particular case since you are using a virtual function). Maybe a simpler application could be used to isolate the error. You could always try sending a DHCP discover or something that doesn't require an IP (and use the source MAC reported by the rte_ethdev_macaddr_get() function.) Hope it helps 2016-03-27 20:17 GMT+02:00 Toby DiPasquale <toby@cbcg.net>: > Hi Jesper, > > Neither ethtool nor ip return for this NIC: > > # ethtool -S eth1 > Cannot get stats strings information: No such device > # ip link show eth1 > Device "eth1" does not exist. > > The NIC has been disconnected from the OS via dpdk_nic_bind.py. The > MAC addresses do match, also. If the NIC is not visible to the OS, and > I'm not turning spoof checking on in the DPDK app, would the setting > while it is attached to the kernel make a difference? > > > On Sun, Mar 27, 2016 at 4:57 AM, Jesper Wramberg > <jesper.wramberg@gmail.com> wrote: > > Hey Toby, > > > > It seems you are using a VF yes ? I did some testing of SR-IOV a while > ago > > on an Intel NIC. I remember having to fiddle with it to get it working, > so > > here are a new notes/questions you might find useful (or maybe not :-)). > > > > Are there any errors in the stats on the physical function (ethtool -S > > <linux PF interface>) ? > > Since you are receiving packets the MAC is probably correct. But anyway: > > Did you check the MAC addresses of the VF using "ip link show <linux PF > > interface>" - do they match what DPDK uses ? > > Is spoof checking enabled on the PF ? > > > > My tests were with an XL710 only using Linux (no DPDK). To be able to TX > > from the VF after changing stuff with "ip link set..." I think I may have > > had to reset the VF or PF using "ip link set .. down/up". > > > > I know I'm not exactly providing any answers here - and my memory isn't > > helping - but maybe something can point you in the right direction. > > > > Regards, > > Jesper > > > > > > 2016-03-26 20:24 GMT+01:00 Toby DiPasquale <toby@cbcg.net>: > >> > >> Hi all, > >> > >> I'm having an issue getting packets to actually transmit out of the > >> NIC with DPDK 2.2.0. I've built a simple UDP echo server here: > >> https://github.com/codeslinger/udpecho > >> > >> Packets are received just fine, and they appear to say they are > >> transmitted, as well, but they never actually leave the NIC. Here is > >> some sample output with the details of how I'm running this: > >> https://gist.github.com/codeslinger/d2e59b00bdc1208f4369 > >> > >> I'm kinda stumped. I had a local person with DPDK experience look at > >> this code and he was similarly confused as to why it wasn't working. > >> Anyone have any ideas on this one? Thanks in advance! > >> > >> -- > >> Toby DiPasquale > > > > > > > > -- > Toby DiPasquale > ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [dpdk-users] pkts not transmitting with DPDK 2.2.0 2016-03-27 20:19 ` Jesper Wramberg @ 2016-03-28 1:02 ` John Boyle 0 siblings, 0 replies; 5+ messages in thread From: John Boyle @ 2016-03-28 1:02 UTC (permalink / raw) To: Jesper Wramberg; +Cc: Toby DiPasquale, users Look at the output of "dmesg", specifically for messages printed by ixgbe. I've encountered two problems of the sort "DPDK thinks packets were successfully sent, but the packets never left the NIC", and both would manifest as some errors in dmesg. One particularly common failure mode was due to the spoof checking Jesper mentions, where the NIC will drop packets where the source MAC doesn't match the VF's MAC. In that case, "dmesg" will print lines like this: [92990.018296] ixgbe 0000:01:00.1 p5p2: 2 Spoofed packets detected -- John Boyle *Science is what we understand well enough to explain to a computer. Art is everything else we do.* --Knuth On Sun, Mar 27, 2016 at 1:19 PM, Jesper Wramberg <jesper.wramberg@gmail.com> wrote: > Hi, > > I may have been a bit unclear in my previous mail. Are you using EC2 or > some kind of virtual server without access to the hypervisor ? The NIC you > are using seems to be a virtual PCI function (SR-IOV). I was assuming you > had access to the physical function (i.e. the actual NIC interface and not > the virtualized/limited one). > > If you don't have access to any information or controls on the hypervisor I > guess we can only check xstats for errors through DPDK. But since the PMD > does not report any errors in your application my best guess is that the > physical function (the physical NIC) receives your packets from the > virtualized interface but discards them for some reason. > > From my (limited) experience, my guess is that your packets are dropped due > to MAC spoofcheck or some kind of IP anti-spoofing. I don't know anything > about EC2 but how do you obtain an IP for your network interface(s) in > Linux ? Can you use static IPs in Linux ? Also, what is the destination IP > of the packets you are receiving ? From my quick glance at your source it > seems you use the dest. IP as source IP when TXing. Since you set the port > in promisc. mode you could be using others IPs (although I don't think > promisc. does much in this particular case since you are using a virtual > function). > > Maybe a simpler application could be used to isolate the error. You could > always try sending a DHCP discover or something that doesn't require an IP > (and use the source MAC reported by the rte_ethdev_macaddr_get() function.) > > Hope it helps > > 2016-03-27 20:17 GMT+02:00 Toby DiPasquale <toby@cbcg.net>: > > > Hi Jesper, > > > > Neither ethtool nor ip return for this NIC: > > > > # ethtool -S eth1 > > Cannot get stats strings information: No such device > > # ip link show eth1 > > Device "eth1" does not exist. > > > > The NIC has been disconnected from the OS via dpdk_nic_bind.py. The > > MAC addresses do match, also. If the NIC is not visible to the OS, and > > I'm not turning spoof checking on in the DPDK app, would the setting > > while it is attached to the kernel make a difference? > > > > > > On Sun, Mar 27, 2016 at 4:57 AM, Jesper Wramberg > > <jesper.wramberg@gmail.com> wrote: > > > Hey Toby, > > > > > > It seems you are using a VF yes ? I did some testing of SR-IOV a while > > ago > > > on an Intel NIC. I remember having to fiddle with it to get it working, > > so > > > here are a new notes/questions you might find useful (or maybe not > :-)). > > > > > > Are there any errors in the stats on the physical function (ethtool -S > > > <linux PF interface>) ? > > > Since you are receiving packets the MAC is probably correct. But > anyway: > > > Did you check the MAC addresses of the VF using "ip link show <linux PF > > > interface>" - do they match what DPDK uses ? > > > Is spoof checking enabled on the PF ? > > > > > > My tests were with an XL710 only using Linux (no DPDK). To be able to > TX > > > from the VF after changing stuff with "ip link set..." I think I may > have > > > had to reset the VF or PF using "ip link set .. down/up". > > > > > > I know I'm not exactly providing any answers here - and my memory isn't > > > helping - but maybe something can point you in the right direction. > > > > > > Regards, > > > Jesper > > > > > > > > > 2016-03-26 20:24 GMT+01:00 Toby DiPasquale <toby@cbcg.net>: > > >> > > >> Hi all, > > >> > > >> I'm having an issue getting packets to actually transmit out of the > > >> NIC with DPDK 2.2.0. I've built a simple UDP echo server here: > > >> https://github.com/codeslinger/udpecho > > >> > > >> Packets are received just fine, and they appear to say they are > > >> transmitted, as well, but they never actually leave the NIC. Here is > > >> some sample output with the details of how I'm running this: > > >> https://gist.github.com/codeslinger/d2e59b00bdc1208f4369 > > >> > > >> I'm kinda stumped. I had a local person with DPDK experience look at > > >> this code and he was similarly confused as to why it wasn't working. > > >> Anyone have any ideas on this one? Thanks in advance! > > >> > > >> -- > > >> Toby DiPasquale > > > > > > > > > > > > > > -- > > Toby DiPasquale > > > ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2016-03-28 1:02 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2016-03-26 19:24 [dpdk-users] pkts not transmitting with DPDK 2.2.0 Toby DiPasquale 2016-03-27 8:57 ` Jesper Wramberg 2016-03-27 18:17 ` Toby DiPasquale 2016-03-27 20:19 ` Jesper Wramberg 2016-03-28 1:02 ` John Boyle
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).