Re: [dpdk-users] Solved : Unable to see incoming packets with example KNI application

Re: [dpdk-users] Solved : Unable to see incoming packets with example KNI application

[Pavey, Nicholas]

Hi All,

Thanks for the emails, and thanks for trying experiments in your environments.

I’ve got the KNI working now, thanks to a colleague who knows a lot more about networking than I do. I’ll do my best to describe what happened (although I may mangle some networking terminology)…

I realized that I’d programmed the DPDK virtual interfaces to have IP addresses on the same subnet as the eth2 control interface. I was aiming to make the machine be multi-homed, but instead put both interfaces in the same subnet.

We set up a new VLAN on the router, with a different subnet, so that the KNI interfaces could be on a different subnet from the management interface: I’ll call the new VLAN “new_vlan/24"

Once I fixed that problem, I was able to ping the router using the vEth0_0 interface, and get the response as you would expect. However, I still wasn’t able to ping all the machines I wanted to.

The real problem ended up being lack of a route in the DPDK machine’s routing table to steer data destined for a given subnet to the correct interface. This meant that traffic destined for locations other than the current subnet was exiting the machine
 via the default route, which was the copper ethernet management interface on eth2.

I think this explains why I was able to see incoming packets with ‘tcpdump’, but was not able to actually use them: The kernel’s routing was discarding the packets after ‘tcpdump’, but before they were made available to user space.

Our load generator has a large number of simulated clients on the “loadgen/12" CIDR. We have an emulated router at address “emulated_router" to prevent all the IP addresses of the clients from blowing out the arp table in the router. As far as our actual
 router is concerned, all the clients are behind this single emulated router.


We added the following routes to the DPDK machine’s routing table:

  ip –4 route add emulated_router/12 via gateway(new_vlan)
  ip –4 route add loadgen/12 via gateway(new_vlan)


Where ‘gateway’ generates the gateway router address for the VLAN. If the VLAN is 1.2.3.0/24, then ‘gateway(VLAN)’ would be address 1.2.3.1.

The machine was able to see the incoming packets from the load generator, so I was able to do a quick benchmark for the KNI path.

Thanks again for your help, and I look forward to working with you all,

Regards,


Nick




From: Ferruh Yigit <ferruh.yigit@intel.com>
Date: Wednesday, May 11, 2016 at 11:37 AM
To: "Pavey, Nicholas" <npavey@akamai.com>, SAKTHIVEL ANAND S <anand.sa88@gmail.com>, Andriy Berestovskyy <aber@semihalf.com>
Cc: "users@dpdk.org" <users@dpdk.org>
Subject: Re: [dpdk-users] Unable to see incoming packets with example KNI application


On 5/11/2016 3:31 PM, Pavey, Nicholas wrote:
>Hi Sakthivel, Andriy,
>Thanks for the email.
>I’m still having trouble, although things do seem to be working better than before.
>It seems that the MAC address suggestion and also the IPs on the same class B network aren’t the root cause.
>Current symptoms
>================
>I rebooted my machine, and took a note of the MAC addresses as the machine started up. It turns out that the KNI virtual interfaces actually did initialize with the correct MAC addresses.
>I also reconfigured the virtual interfaces to be using different network classes, so that should no longer be a problem.
>Something has improved because I’m able to see traffic on the KNI virtual interface with tcpdump, which I was not able to do previously.
>Unfortunately, even though tcpdump is able to see the traffic correctly, it seems that the networking stack isn’t working as I would expect.
>Outbound
>========
>Ping
>----
>I can see outbound ‘pings’ (with initial ARP requests) with ‘tcpdump', and I can see a response coming back. However, the ‘ping’ application reports 100% packet loss.
>The ARP traffic is definitely getting out, because I see the IP address registered in the router’s ARP cache. Likewise, I see a response to the original ‘ping’ packet, so the outbound direction seems to be working.
>Inbound
>=======
>The problems seem to be on the inbound side. As we saw above, the outbound side appears to be working reasonably, but I don’t appear to be able to capture inbound packets.
>TCP
>---
>For example, if I set up a simple ‘netcat’ listener (using TCP for transport) on the target server:
>   nc –l 172.25.48.200 9876
>And then attempt to connect to it from another machine, as follows:
>   nc 172.25.48.200 9876
>‘tcpdump’ on the target server will show me the incoming ‘syn’ and a ‘syn’ retransmission, but there are no outbound ‘ack’ packets.
>
>UDP
>---
>Similarly, inbound UDP traffic never appears to be routed to the user space application.
>I can counters incrementing on the virtual interface with ‘sar –n DEV 1 100’. ‘tcpdump’ also shows me the incoming data.
>However, if I look at the UDP stats with ‘sar –n UDP 1 100’, I’m not seeing any packets arriving, even with ‘no port’ or ‘idgmerr’, which I’d normally expect if there’s no listening application bound to the IP address.
>Next steps
>==========
>It almost seems as if the receive side of the network stack simply isn’t seeing the inbound data (regardless of whether it’s ICMP, TCP or UDP) and therefore isn’t sending responses.
>The thing I’m confused about here is how ‘tcpdump’ is able to see the traffic - after all, if it’s able to see the inbound traffic, then a good part of the RX side of the stack must be working. I’d have thought that if ‘tcpdump’ can see the traffic, then
> the rest of the stack should be working too.
>It makes me wondering whether perhaps I’m misunderstanding the purpose of the KNI system?
>My interpretation is that it’s supposed to route traffic from the DPDK into the regular Linux network stack, where it can be used as if it were regular traffic. Do I have that right?
>Do you have any ideas? 
>Thanks,
>Nick
>From:  SAKTHIVEL ANAND S <anand.sa88@gmail.com>
>Date:  Wednesday, May 11, 2016 at 3:30 AM
>To:  "Pavey, Nicholas" <npavey@akamai.com>
>Subject:  Re: [dpdk-users] Unable to see incoming packets with example KNI application
>Hi
>When you try to send echo packets(outbound), your PC will try to resolve ARP, which it could not complete properly .. due to random mac generation for KNI interface(your KNI interface having different MAC than actual interface).
>After starting KNI app write your hardware address on KNI by doing, "ifconfig <vETHname> hw ether <real HW address>" and try ping. Let me know the results.
>use  "tcpdump -n -i vEth** -e | grep <filter>" 
>Regards
>Sakthivel S OM

Hi Nick,

This may not help for your problem but can work as a reference:

I did test two things with latest code and both worked well:
1- Two NICs , as in your case, with packet generator each end.
Internally created an bridge and added KNI interfaces to bridge, I
observe received packets in generator.

2- Tested with netcat, KNI interface able to send/receive packets, both
for udp and tcp.

I checked for your case, not able to find any obvious error.

Regards,
ferruh

Re: [dpdk-users] Solved : Unable to see incoming packets with example KNI application

[Pavey, Nicholas]

Hi All,

Thanks for the emails, and thanks for trying experiments in your environments.

I’ve got the KNI working now, thanks to a colleague who knows a lot more about networking than I do. I’ll do my best to describe what happened (although I may mangle some networking terminology)…

I realized that I’d programmed the DPDK virtual interfaces to have IP addresses on the same subnet as the eth2 control interface. I was aiming to make the machine be multi-homed, but instead put both interfaces in the same subnet.

We set up a new VLAN on the router, with a different subnet, so that the KNI interfaces could be on a different subnet from the management interface: I’ll call the new VLAN “new_vlan/24"

Once I fixed that problem, I was able to ping the router using the vEth0_0 interface, and get the response as you would expect. However, I still wasn’t able to ping all the machines I wanted to.

The real problem ended up being lack of a route in the DPDK machine’s routing table to steer data destined for a given subnet to the correct interface. This meant that traffic destined for locations other than the current subnet was exiting the machine via the default route, which was the copper ethernet management interface on eth2.

I think this explains why I was able to see incoming packets with ‘tcpdump’, but was not able to actually use them: The kernel’s routing was discarding the packets after ‘tcpdump’, but before they were made available to user space.

Our load generator has a large number of simulated clients on the “loadgen/12" CIDR. We have an emulated router at address “emulated_router" to prevent all the IP addresses of the clients from blowing out the arp table in the router. As far as our actual router is concerned, all the clients are behind this single emulated router.

We added the following routes to the DPDK machine’s routing table:

  ip –4 route add emulated_router/12 via gateway(new_vlan)
  ip –4 route add loadgen/12 via gateway(new_vlan)

Where ‘gateway’ generates the gateway router address for the VLAN. If the VLAN is 1.2.3.0/24, then ‘gateway(VLAN)’ would be address 1.2.3.1.

The machine was able to see the incoming packets from the load generator, so I was able to do a quick benchmark for the KNI path.

Thanks again for your help, and I look forward to working with you all,

Regards,


Nick




From: Ferruh Yigit <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>
Date: Wednesday, May 11, 2016 at 11:37 AM
To: "Pavey, Nicholas" <npavey@akamai.com<mailto:npavey@akamai.com>>, SAKTHIVEL ANAND S <anand.sa88@gmail.com<mailto:anand.sa88@gmail.com>>, Andriy Berestovskyy <aber@semihalf.com<mailto:aber@semihalf.com>>
Cc: "users@dpdk.org<mailto:users@dpdk.org>" <users@dpdk.org<mailto:users@dpdk.org>>
Subject: Re: [dpdk-users] Unable to see incoming packets with example KNI application

On 5/11/2016 3:31 PM, Pavey, Nicholas wrote:
Hi Sakthivel, Andriy,
Thanks for the email.
I’m still having trouble, although things do seem to be working better than before.
It seems that the MAC address suggestion and also the IPs on the same class B network aren’t the root cause.
Current symptoms
================
I rebooted my machine, and took a note of the MAC addresses as the machine started up. It turns out that the KNI virtual interfaces actually did initialize with the correct MAC addresses.
I also reconfigured the virtual interfaces to be using different network classes, so that should no longer be a problem.
Something has improved because I’m able to see traffic on the KNI virtual interface with tcpdump, which I was not able to do previously.
Unfortunately, even though tcpdump is able to see the traffic correctly, it seems that the networking stack isn’t working as I would expect.
Outbound
========
Ping
----
I can see outbound ‘pings’ (with initial ARP requests) with ‘tcpdump', and I can see a response coming back. However, the ‘ping’ application reports 100% packet loss.
The ARP traffic is definitely getting out, because I see the IP address registered in the router’s ARP cache. Likewise, I see a response to the original ‘ping’ packet, so the outbound direction seems to be working.
Inbound
=======
The problems seem to be on the inbound side. As we saw above, the outbound side appears to be working reasonably, but I don’t appear to be able to capture inbound packets.
TCP
---
For example, if I set up a simple ‘netcat’ listener (using TCP for transport) on the target server:
   nc –l 172.25.48.200 9876
And then attempt to connect to it from another machine, as follows:
   nc 172.25.48.200 9876
‘tcpdump’ on the target server will show me the incoming ‘syn’ and a ‘syn’ retransmission, but there are no outbound ‘ack’ packets.
UDP
---
Similarly, inbound UDP traffic never appears to be routed to the user space application.
I can counters incrementing on the virtual interface with ‘sar –n DEV 1 100’. ‘tcpdump’ also shows me the incoming data.
However, if I look at the UDP stats with ‘sar –n UDP 1 100’, I’m not seeing any packets arriving, even with ‘no port’ or ‘idgmerr’, which I’d normally expect if there’s no listening application bound to the IP address.
Next steps
==========
It almost seems as if the receive side of the network stack simply isn’t seeing the inbound data (regardless of whether it’s ICMP, TCP or UDP) and therefore isn’t sending responses.
The thing I’m confused about here is how ‘tcpdump’ is able to see the traffic - after all, if it’s able to see the inbound traffic, then a good part of the RX side of the stack must be working. I’d have thought that if ‘tcpdump’ can see the traffic, then the rest of the stack should be working too.
It makes me wondering whether perhaps I’m misunderstanding the purpose of the KNI system?
My interpretation is that it’s supposed to route traffic from the DPDK into the regular Linux network stack, where it can be used as if it were regular traffic. Do I have that right?
Do you have any ideas?
Thanks,
Nick
From:  SAKTHIVEL ANAND S <anand.sa88@gmail.com<mailto:anand.sa88@gmail.com>>
Date:  Wednesday, May 11, 2016 at 3:30 AM
To:  "Pavey, Nicholas" <npavey@akamai.com<mailto:npavey@akamai.com>>
Subject:  Re: [dpdk-users] Unable to see incoming packets with example KNI application
Hi
When you try to send echo packets(outbound), your PC will try to resolve ARP, which it could not complete properly .. due to random mac generation for KNI interface(your KNI interface having different MAC than actual interface).
After starting KNI app write your hardware address on KNI by doing, "ifconfig <vETHname> hw ether <real HW address>" and try ping. Let me know the results.
use  "tcpdump -n -i vEth** -e | grep <filter>"
Regards
Sakthivel S OM

Hi Nick,

This may not help for your problem but can work as a reference:

I did test two things with latest code and both worked well:
1- Two NICs , as in your case, with packet generator each end.
Internally created an bridge and added KNI interfaces to bridge, I
observe received packets in generator.

2- Tested with netcat, KNI interface able to send/receive packets, both
for udp and tcp.

I checked for your case, not able to find any obvious error.

Regards,
ferruh