Re: [dpdk-users] Unable to see incoming packets with example KNI application

["Pavey, Nicholas" <npavey@akamai.com>]

Hi Shawn,

I don’t _think_ that’s the problem, but I’m not an expert in low level networking so I could be mistaken.

If I shut down the KNI application, then the vEth* interfaces shut down and the arp table entries related to those interfaces are cleared.

Here’s the arp table before I start the KNI app:

  arp
  Address                  HWtype  HWaddress           Flags Mask            Iface
  172.25.48.100            ether   a0:42:3f:25:0b:a7   C                     eth2
  172.25.48.1              ether   18:e7:28:96:83:01   C                     eth2

172.25.48.1 is the router’s IP. ‘eth2’ is my 1Gbps copper management interface.

I start the KNI app:

  ./build/kni -c 0xf0 -n 4 -- -P -p 0x3 --config="(0,4,6,8),(1,5,7,9)”

The arp table is unchanged from above.

Now I set up the IP addresses for the KNI virtual interfaces:

  ifconfig vEth0_0 172.25.48.200
  ifconfig vEth1_0 10.25.48.200


Arp is still unchanged.

Now, I ping the router using the ‘vEth0_0’ inteface, which will require an ARP to start with:

  ping -I vEth0_0 172.25.48.1

My DPDK machine’s MAC appears in the router’s arp table.

The arp table on the DPDK machine is now:

  Address                  HWtype  HWaddress           Flags Mask            Iface
  >>>> 172.25.48.1              ether   18:e7:28:96:83:01   C                     vEth0_0
  172.25.48.100            ether   a0:42:3f:25:0b:a7   C                     eth2
  172.25.48.1              ether   18:e7:28:96:83:01   C                     eth2


The 1st line is the new entry.

This looks to me like the ARP process is working correctly, and it seems that ‘tcpdump’ is also working as expected.

What do you think?


Thanks,


Nick




From:  Shawn Lewis <smlsr@tencara.com>
Date:  Wednesday, May 11, 2016 at 10:36 AM
To:  "Pavey, Nicholas" <npavey@akamai.com>
Subject:  Re: [dpdk-users] Unable to see incoming packets with example KNI application




> On May 11, 2016, at 10:31 AM, Pavey, Nicholas <npavey@akamai.com> wrote:
> 
> Hi Sakthivel, Andriy,
> 
> Thanks for the email.
> 
> I’m still having trouble, although things do seem to be working better than before.
> 
> It seems that the MAC address suggestion and also the IPs on the same class B network aren’t the root cause.
> 
> 
> Current symptoms
> ================
> 
> I rebooted my machine, and took a note of the MAC addresses as the machine started up. It turns out that the KNI virtual interfaces actually did initialize with the correct MAC addresses.
> 
> I also reconfigured the virtual interfaces to be using different network classes, so that should no longer be a problem.
> 
> Something has improved because I’m able to see traffic on the KNI virtual interface with tcpdump, which I was not able to do previously.
> 
> Unfortunately, even though tcpdump is able to see the traffic correctly, it seems that the networking stack isn’t working as I would expect.
> 
> 
> 
> Outbound
> ========
> 
> Ping
> ----
> 
> I can see outbound ‘pings’ (with initial ARP requests) with ‘tcpdump', and I can see a response coming back. However, the ‘ping’ application reports 100% packet loss.
> 
> The ARP traffic is definitely getting out, because I see the IP address registered in the router’s ARP cache. Likewise, I see a response to the original ‘ping’ packet, so the outbound direction seems to be working.
> 
> 
> Inbound
> =======
> 
> The problems seem to be on the inbound side. As we saw above, the outbound side appears to be working reasonably, but I don’t appear to be able to capture inbound packets.
> 
> TCP
> ---
> 
> For example, if I set up a simple ‘netcat’ listener (using TCP for transport) on the target server:
> 
>  nc –l 172.25.48.200 9876
> 
> And then attempt to connect to it from another machine, as follows:
> 
>  nc 172.25.48.200 9876
> 
> 
> ‘tcpdump’ on the target server will show me the incoming ‘syn’ and a ‘syn’ retransmission, but there are no outbound ‘ack’ packets. 
> 
> 
> UDP
> ---
> 
> Similarly, inbound UDP traffic never appears to be routed to the user space application.
> 
> I can counters incrementing on the virtual interface with ‘sar –n DEV 1 100’. ‘tcpdump’ also shows me the incoming data.
> 
> However, if I look at the UDP stats with ‘sar –n UDP 1 100’, I’m not seeing any packets arriving, even with ‘no port’ or ‘idgmerr’, which I’d normally expect if there’s no listening application bound to the IP address.
> 
> 
> Next steps
> ==========
> 
> It almost seems as if the receive side of the network stack simply isn’t seeing the inbound data (regardless of whether it’s ICMP, TCP or UDP) and therefore isn’t sending responses.
> 
> 
> The thing I’m confused about here is how ‘tcpdump’ is able to see the traffic - after all, if it’s able to see the inbound traffic, then a good part of the RX side of the stack must be working. I’d have thought that if ‘tcpdump’ can see the traffic, then the rest of the stack should be working too.
> 
> It makes me wondering whether perhaps I’m misunderstanding the purpose of the KNI system?
> 
> My interpretation is that it’s supposed to route traffic from the DPDK into the regular Linux network stack, where it can be used as if it were regular traffic. Do I have that right?
> 
> 
> 
> Do you have any ideas? 
> 
> Thanks,
> 
> 
> Nick
> 
> 
> From:  SAKTHIVEL ANAND S <anand.sa88@gmail.com>
> Date:  Wednesday, May 11, 2016 at 3:30 AM
> To:  "Pavey, Nicholas" <npavey@akamai.com>
> Subject:  Re: [dpdk-users] Unable to see incoming packets with example KNI application
> 
> 
> Hi
> 
> When you try to send echo packets(outbound), your PC will try to resolve ARP, which it could not complete properly .. due to random mac generation for KNI interface(your KNI interface having different MAC than actual interface).
> 
> 
> After starting KNI app write your hardware address on KNI by doing, "ifconfig <vETHname> hw ether <real HW address>" and try ping. Let me know the results.
> 
> 
> use  "tcpdump -n -i vEth** -e | grep <filter>" 
> 
> Regards
> 
> Sakthivel S OM
> 
> 

In many cases I have seen where the ether Mac header is 0s for arp packets. So when pushing the packets to the kernel you may have to repopulate the Mac header. This way the kernel can build its internal arp cache. 

Took me a week to find that !!

After doing that all worked fine