[dpdk-users] Possible problem populating the target directory during build process

[dpdk-users] Possible problem populating the target directory during build process

[Pavey, Nicholas]

Hi Ferruh,

I did eventually figure out the problem with the debug / optimization flags that we IRCed about.

It seems that the target directory (e.g. 'x86_64-native-linuxapp-gcc')  wasn’t being properly cleared down and repopulated. It seems the contents survived a ‘make clean’.

Because of this, the object files that the ip_reassembly example was linking against didn’t have all the debug information compiled in, which was confusing gdb pretty badly.

If I set a breakpoint against rte_mempool_create, gdb would break, but the ‘list’ command would show me code from the ‘main’ subroutine.

I eventually completely removed the target directory. It seems that it’s not populated when you do a regular ‘make T=<target>’. Instead, you need to do a ‘make install T=<target>’ to get the directory rebuilt.

Once I figured that out, I got a clean gdb debug session where it breaks as expected.

I haven’t fully debugged the problem yet, so the above description may have some inaccuracies. I’ll look into it more closely next week and give you the exact sequence of commands to reproduce the problem.

Thanks,


Nick


From: Ferruh Yigit <ferruh.yigit@intel.com<mailto:ferruh.yigit@intel.com>>
Date: Wednesday, May 11, 2016 at 11:37 AM
To: "Pavey, Nicholas" <npavey@akamai.com<mailto:npavey@akamai.com>>, SAKTHIVEL ANAND S <anand.sa88@gmail.com<mailto:anand.sa88@gmail.com>>, Andriy Berestovskyy <aber@semihalf.com<mailto:aber@semihalf.com>>
Cc: "users@dpdk.org<mailto:users@dpdk.org>" <users@dpdk.org<mailto:users@dpdk.org>>
Subject: Re: [dpdk-users] Unable to see incoming packets with example KNI application

On 5/11/2016 3:31 PM, Pavey, Nicholas wrote:
Hi Sakthivel, Andriy,
Thanks for the email.
I’m still having trouble, although things do seem to be working better than before.
It seems that the MAC address suggestion and also the IPs on the same class B network aren’t the root cause.
Current symptoms
================
I rebooted my machine, and took a note of the MAC addresses as the machine started up. It turns out that the KNI virtual interfaces actually did initialize with the correct MAC addresses.
I also reconfigured the virtual interfaces to be using different network classes, so that should no longer be a problem.
Something has improved because I’m able to see traffic on the KNI virtual interface with tcpdump, which I was not able to do previously.
Unfortunately, even though tcpdump is able to see the traffic correctly, it seems that the networking stack isn’t working as I would expect.
Outbound
========
Ping
----
I can see outbound ‘pings’ (with initial ARP requests) with ‘tcpdump', and I can see a response coming back. However, the ‘ping’ application reports 100% packet loss.
The ARP traffic is definitely getting out, because I see the IP address registered in the router’s ARP cache. Likewise, I see a response to the original ‘ping’ packet, so the outbound direction seems to be working.
Inbound
=======
The problems seem to be on the inbound side. As we saw above, the outbound side appears to be working reasonably, but I don’t appear to be able to capture inbound packets.
TCP
---
For example, if I set up a simple ‘netcat’ listener (using TCP for transport) on the target server:
   nc –l 172.25.48.200 9876
And then attempt to connect to it from another machine, as follows:
   nc 172.25.48.200 9876
‘tcpdump’ on the target server will show me the incoming ‘syn’ and a ‘syn’ retransmission, but there are no outbound ‘ack’ packets.
UDP
---
Similarly, inbound UDP traffic never appears to be routed to the user space application.
I can counters incrementing on the virtual interface with ‘sar –n DEV 1 100’. ‘tcpdump’ also shows me the incoming data.
However, if I look at the UDP stats with ‘sar –n UDP 1 100’, I’m not seeing any packets arriving, even with ‘no port’ or ‘idgmerr’, which I’d normally expect if there’s no listening application bound to the IP address.
Next steps
==========
It almost seems as if the receive side of the network stack simply isn’t seeing the inbound data (regardless of whether it’s ICMP, TCP or UDP) and therefore isn’t sending responses.
The thing I’m confused about here is how ‘tcpdump’ is able to see the traffic - after all, if it’s able to see the inbound traffic, then a good part of the RX side of the stack must be working. I’d have thought that if ‘tcpdump’ can see the traffic, then the rest of the stack should be working too.
It makes me wondering whether perhaps I’m misunderstanding the purpose of the KNI system?
My interpretation is that it’s supposed to route traffic from the DPDK into the regular Linux network stack, where it can be used as if it were regular traffic. Do I have that right?
Do you have any ideas?
Thanks,
Nick
From:  SAKTHIVEL ANAND S <anand.sa88@gmail.com<mailto:anand.sa88@gmail.com>>
Date:  Wednesday, May 11, 2016 at 3:30 AM
To:  "Pavey, Nicholas" <npavey@akamai.com<mailto:npavey@akamai.com>>
Subject:  Re: [dpdk-users] Unable to see incoming packets with example KNI application
Hi
When you try to send echo packets(outbound), your PC will try to resolve ARP, which it could not complete properly .. due to random mac generation for KNI interface(your KNI interface having different MAC than actual interface).
After starting KNI app write your hardware address on KNI by doing, "ifconfig <vETHname> hw ether <real HW address>" and try ping. Let me know the results.
use  "tcpdump -n -i vEth** -e | grep <filter>"
Regards
Sakthivel S OM

Hi Nick,

This may not help for your problem but can work as a reference:

I did test two things with latest code and both worked well:
1- Two NICs , as in your case, with packet generator each end.
Internally created an bridge and added KNI interfaces to bridge, I
observe received packets in generator.

2- Tested with netcat, KNI interface able to send/receive packets, both
for udp and tcp.

I checked for your case, not able to find any obvious error.

Regards,
ferruh