[dpdk-users] Unable to see incoming packets with example KNI application

[dpdk-users] Unable to see incoming packets with example KNI application

[Pavey, Nicholas]

Good afternoon,

I’m a new user of the DPDK library - this is my first post to this list.

I’ve been experimenting with the various example applications, including ‘skeleton’ and ‘kni’.

I have been able to get ‘skeleton’ to work correctly, and observed good forwarding performance from it. I have been unable to get ‘kni’ to work however.

The symptom appears to be that incoming packets are not received correctly and are not forwarded to the KNI stack. Likewise, it appears that outbound ICMP ‘ping’ packets sent via the traditional linux ‘ping’ command on the KNI interface are not sent (see observations, below).

I’ve searched the mail archives and have seen several people with what sounds like similar problems. I don’t believe I saw a problem resolution there, however.

Here’s a fair amount of detail about my environment. Please let me know if I’m missing something - I’ll happily provide extra details.

Does anyone have any suggestions about what might be going on here?

Regards,



Nick Pavey



Networking environment
======================

  * DPDK machine has dual 10Gbps NICs
    - Intel 82599EB 10Gbps NIC
  * DPDK machine is directly connected to an Ixia load generator
    - SFP+ Direct Attached Copper (DAC) cabling
    - Running BreakingPoint load generation software
  * The linux kernel is unaware of the 10Gbps interfaces, so the DPDK can attach correctly
  * The control interface is an Intel I350 1Gbps copper NIC
    - This has a statically assigned IP address
    - I control the machine through this NIC

DPDK versions
=============

I have tried the ‘kni’ application with two versions of the DPDK:

  * DPDK-16.04, unpacked from tarball
  * DPDK cloned from git. Head commit is db340cf2ef71af231af67be8e42fd603e4bab0ac
    - "i40e: fix VLAN stripping from inner header” by JingJing Wu, 5/4/2016

Machine details
===============

Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz

Dual socket, 8 core, hyperthreaded. 32 total execution contexts.
64GB RAM
NICS are dual Intel 82599EB 10Gbps

OS/compiler details
===================

The machine is running a modified version of the Kernel. IIRC, it’s derived from Ubuntu 12.04.

The kernel version is reported as 3.14.43. Unfortunately I’m unable to detail all the modifications, but it’s probably fair to assume the kernel is roughly equivalent to 3.13.43.

The compiler is GCC version 4.6.3.


KNI application startup
=======================

I have 384 (size chosen fairly arbitrarily) available huge pages, which seems to be enough to allow the ‘kni’ application to start up.

I have built the ‘rte_kni’ kernel module on the DPDK target machine. I load it with no arguments, and it loads without complaint.

I’m invoking the ‘kni’ application as follows:

  ./build/kni -c 0xf0 -n 4 -- -P -p 0x3 --config="(0,4,6,8),(1,5,7,9)"

So, the ‘kni’ application is in Promiscuous mode.

Once the application is started up, I set up two IP addresses for the KNI interfaces

  ifconfig vEth0_0 172.25.48.200
  ifconfig vEth1_0 172.25.48.201

Both seem to initialize correctly, and the ‘kni’ application notices that the NIC status has changed.


The ‘kni’ application seems to start up correctly. I’m not getting any errors, but equally, I’m not seeing traffic get routed into the Linux network stack as I’d expect.


Observations
============

I’ve done several tests:

  * Sent a low level of traffic from the Ixia (a few packets a second)
    - The Ixia is directly connected, so the data is not going missing in the networking infrastructure
    - The ‘skeleton’ application works as expected, so the Ixia, the cabling and a good part of the DPDK appears to be working
    - A ‘tcpdump’ on the relevant network interfaces shows no incoming traffic
    - Watching the incoming packet rate with ‘sar -n DEV 1 1000’ shows the kernel seeing no incoming traffic on the DPDK interfaces
    - The statistics from the ‘kni’ application show no inbound data
    - A packet capture on the Ixia interface shows outbound packets but no return packets.

  * Started the ‘kni’ application and the Ixia, then attempted to ping the Ixia
    - In this case, the statistics from the ‘kni’ application do show the TX counters incrementing
    - However, when I examine the Ixia’s packet capture, I see no signs of ‘ping’ packets.


Dmesg contents
==============

When I bring up the KNI interfaces, I get the following lines in the ‘dmesg’ log:

[71051.312668] KNI: /dev/kni opened
[71051.470926] KNI: Creating kni...
[71051.471280] KNI: tx_phys:      0x00000000375313c0, tx_q addr:      0xffff8800
375313c0
[71051.471281] KNI: rx_phys:      0x000000003752f340, rx_q addr:      0xffff8800
3752f340
[71051.471282] KNI: alloc_phys:   0x000000003752d2c0, alloc_q addr:   0xffff8800
3752d2c0
[71051.471282] KNI: free_phys:    0x000000003752b240, free_q addr:    0xffff8800
3752b240
[71051.471284] KNI: req_phys:     0x00000000375291c0, req_q addr:     0xffff8800
375291c0
[71051.471285] KNI: resp_phys:    0x0000000037527140, resp_q addr:    0xffff8800
37527140
[71051.471286] KNI: mbuf_phys:    0x000000083c27dec0, mbuf_kva:       0xffff8808
3c27dec0
[71051.471287] KNI: mbuf_va:      0x00007fe07ce7dec0
[71051.471287] KNI: mbuf_size:    2048
[71051.471306] KNI: pci_bus: 03:00:00 
[71051.506633] uio_pci_generic 0000:03:00.0: (PCI Express:5.0GT/s:Width x8) 
[71051.506981] a0:42:3f:29:b3:ae
[71051.507710] uio_pci_generic 0000:03:00.0 (unregistered net_device): MAC: 2, P
HY: 0, PBA No: FFFFFF-0FF
[71051.508340] uio_pci_generic 0000:03:00.0 (unregistered net_device): Enabled F
eatures: RxQ: 1 TxQ: 1 
[71051.510093] uio_pci_generic 0000:03:00.0 (unregistered net_device): Intel(R) 
10 Gigabit Network Connection

[71051.668942] KNI: Creating kni...
[71051.669291] KNI: tx_phys:      0x0000000037522f40, tx_q addr:      0xffff880037522f40
[71051.669292] KNI: rx_phys:      0x0000000037520ec0, rx_q addr:      0xffff880037520ec0
[71051.669294] KNI: alloc_phys:   0x000000003751ee40, alloc_q addr:   0xffff88003751ee40
[71051.669295] KNI: free_phys:    0x000000003751cdc0, free_q addr:    0xffff88003751cdc0
[71051.669296] KNI: req_phys:     0x000000003751ad40, req_q addr:     0xffff88003751ad40
[71051.669297] KNI: resp_phys:    0x0000000037518cc0, resp_q addr:    0xffff880037518cc0
[71051.669297] KNI: mbuf_phys:    0x000000083c27dec0, mbuf_kva:       0xffff88083c27dec0
[71051.669298] KNI: mbuf_va:      0x00007fe07ce7dec0
[71051.669299] KNI: mbuf_size:    2048
[71051.669306] KNI: pci_bus: 03:00:00 
[71051.669308] KNI: pci_bus: 03:00:01 
[71051.704749] uio_pci_generic 0000:03:00.1: (PCI Express:5.0GT/s:Width x8) 
[71051.705096] a0:42:3f:29:b3:af
[71051.705824] uio_pci_generic 0000:03:00.1 (unregistered net_device): MAC: 2, PHY: 0, PBA No: FFFFFF-0FF
[71051.706452] uio_pci_generic 0000:03:00.1 (unregistered net_device): Enabled Features: RxQ: 1 TxQ: 1 
[71051.708210] uio_pci_generic 0000:03:00.1 (unregistered net_device): Intel(R) 10 Gigabit Network Connection
[74051.258432] KNI: Successfully release kni named vEth0_0
[74054.324270] KNI: Successfully release kni named vEth1_0
[74054.379908] KNI: /dev/kni closed

Re: [dpdk-users] Unable to see incoming packets with example KNI application

[Andriy Berestovskyy]

Hi Nicholas,
Those two lines look suspicious:

ifconfig vEth0_0 172.25.48.200
ifconfig vEth1_0 172.25.48.201

Those lines configure two interfaces in the same class B network.
Could it be the reason for the issues?

Also, do you see the correct ARP entries on both sides?

Andriy

On Tue, May 10, 2016 at 7:58 PM, Pavey, Nicholas <npavey@akamai.com> wrote:
> Good afternoon,
>
> I’m a new user of the DPDK library - this is my first post to this list.
>
> I’ve been experimenting with the various example applications, including ‘skeleton’ and ‘kni’.
>
> I have been able to get ‘skeleton’ to work correctly, and observed good forwarding performance from it. I have been unable to get ‘kni’ to work however.
>
> The symptom appears to be that incoming packets are not received correctly and are not forwarded to the KNI stack. Likewise, it appears that outbound ICMP ‘ping’ packets sent via the traditional linux ‘ping’ command on the KNI interface are not sent (see observations, below).
>
> I’ve searched the mail archives and have seen several people with what sounds like similar problems. I don’t believe I saw a problem resolution there, however.
>
> Here’s a fair amount of detail about my environment. Please let me know if I’m missing something - I’ll happily provide extra details.
>
> Does anyone have any suggestions about what might be going on here?
>
> Regards,
>
>
>
> Nick Pavey
>
>
>
> Networking environment
> ======================
>
>   * DPDK machine has dual 10Gbps NICs
>     - Intel 82599EB 10Gbps NIC
>   * DPDK machine is directly connected to an Ixia load generator
>     - SFP+ Direct Attached Copper (DAC) cabling
>     - Running BreakingPoint load generation software
>   * The linux kernel is unaware of the 10Gbps interfaces, so the DPDK can attach correctly
>   * The control interface is an Intel I350 1Gbps copper NIC
>     - This has a statically assigned IP address
>     - I control the machine through this NIC
>
> DPDK versions
> =============
>
> I have tried the ‘kni’ application with two versions of the DPDK:
>
>   * DPDK-16.04, unpacked from tarball
>   * DPDK cloned from git. Head commit is db340cf2ef71af231af67be8e42fd603e4bab0ac
>     - "i40e: fix VLAN stripping from inner header” by JingJing Wu, 5/4/2016
>
> Machine details
> ===============
>
> Intel(R) Xeon(R) CPU E5-2640 v3 @ 2.60GHz
>
> Dual socket, 8 core, hyperthreaded. 32 total execution contexts.
> 64GB RAM
> NICS are dual Intel 82599EB 10Gbps
>
> OS/compiler details
> ===================
>
> The machine is running a modified version of the Kernel. IIRC, it’s derived from Ubuntu 12.04.
>
> The kernel version is reported as 3.14.43. Unfortunately I’m unable to detail all the modifications, but it’s probably fair to assume the kernel is roughly equivalent to 3.13.43.
>
> The compiler is GCC version 4.6.3.
>
>
> KNI application startup
> =======================
>
> I have 384 (size chosen fairly arbitrarily) available huge pages, which seems to be enough to allow the ‘kni’ application to start up.
>
> I have built the ‘rte_kni’ kernel module on the DPDK target machine. I load it with no arguments, and it loads without complaint.
>
> I’m invoking the ‘kni’ application as follows:
>
>   ./build/kni -c 0xf0 -n 4 -- -P -p 0x3 --config="(0,4,6,8),(1,5,7,9)"
>
> So, the ‘kni’ application is in Promiscuous mode.
>
> Once the application is started up, I set up two IP addresses for the KNI interfaces
>
>   ifconfig vEth0_0 172.25.48.200
>   ifconfig vEth1_0 172.25.48.201
>
> Both seem to initialize correctly, and the ‘kni’ application notices that the NIC status has changed.
>
>
> The ‘kni’ application seems to start up correctly. I’m not getting any errors, but equally, I’m not seeing traffic get routed into the Linux network stack as I’d expect.
>
>
> Observations
> ============
>
> I’ve done several tests:
>
>   * Sent a low level of traffic from the Ixia (a few packets a second)
>     - The Ixia is directly connected, so the data is not going missing in the networking infrastructure
>     - The ‘skeleton’ application works as expected, so the Ixia, the cabling and a good part of the DPDK appears to be working
>     - A ‘tcpdump’ on the relevant network interfaces shows no incoming traffic
>     - Watching the incoming packet rate with ‘sar -n DEV 1 1000’ shows the kernel seeing no incoming traffic on the DPDK interfaces
>     - The statistics from the ‘kni’ application show no inbound data
>     - A packet capture on the Ixia interface shows outbound packets but no return packets.
>
>   * Started the ‘kni’ application and the Ixia, then attempted to ping the Ixia
>     - In this case, the statistics from the ‘kni’ application do show the TX counters incrementing
>     - However, when I examine the Ixia’s packet capture, I see no signs of ‘ping’ packets.
>
>
> Dmesg contents
> ==============
>
> When I bring up the KNI interfaces, I get the following lines in the ‘dmesg’ log:
>
> [71051.312668] KNI: /dev/kni opened
> [71051.470926] KNI: Creating kni...
> [71051.471280] KNI: tx_phys:      0x00000000375313c0, tx_q addr:      0xffff8800
> 375313c0
> [71051.471281] KNI: rx_phys:      0x000000003752f340, rx_q addr:      0xffff8800
> 3752f340
> [71051.471282] KNI: alloc_phys:   0x000000003752d2c0, alloc_q addr:   0xffff8800
> 3752d2c0
> [71051.471282] KNI: free_phys:    0x000000003752b240, free_q addr:    0xffff8800
> 3752b240
> [71051.471284] KNI: req_phys:     0x00000000375291c0, req_q addr:     0xffff8800
> 375291c0
> [71051.471285] KNI: resp_phys:    0x0000000037527140, resp_q addr:    0xffff8800
> 37527140
> [71051.471286] KNI: mbuf_phys:    0x000000083c27dec0, mbuf_kva:       0xffff8808
> 3c27dec0
> [71051.471287] KNI: mbuf_va:      0x00007fe07ce7dec0
> [71051.471287] KNI: mbuf_size:    2048
> [71051.471306] KNI: pci_bus: 03:00:00
> [71051.506633] uio_pci_generic 0000:03:00.0: (PCI Express:5.0GT/s:Width x8)
> [71051.506981] a0:42:3f:29:b3:ae
> [71051.507710] uio_pci_generic 0000:03:00.0 (unregistered net_device): MAC: 2, P
> HY: 0, PBA No: FFFFFF-0FF
> [71051.508340] uio_pci_generic 0000:03:00.0 (unregistered net_device): Enabled F
> eatures: RxQ: 1 TxQ: 1
> [71051.510093] uio_pci_generic 0000:03:00.0 (unregistered net_device): Intel(R)
> 10 Gigabit Network Connection
>
> [71051.668942] KNI: Creating kni...
> [71051.669291] KNI: tx_phys:      0x0000000037522f40, tx_q addr:      0xffff880037522f40
> [71051.669292] KNI: rx_phys:      0x0000000037520ec0, rx_q addr:      0xffff880037520ec0
> [71051.669294] KNI: alloc_phys:   0x000000003751ee40, alloc_q addr:   0xffff88003751ee40
> [71051.669295] KNI: free_phys:    0x000000003751cdc0, free_q addr:    0xffff88003751cdc0
> [71051.669296] KNI: req_phys:     0x000000003751ad40, req_q addr:     0xffff88003751ad40
> [71051.669297] KNI: resp_phys:    0x0000000037518cc0, resp_q addr:    0xffff880037518cc0
> [71051.669297] KNI: mbuf_phys:    0x000000083c27dec0, mbuf_kva:       0xffff88083c27dec0
> [71051.669298] KNI: mbuf_va:      0x00007fe07ce7dec0
> [71051.669299] KNI: mbuf_size:    2048
> [71051.669306] KNI: pci_bus: 03:00:00
> [71051.669308] KNI: pci_bus: 03:00:01
> [71051.704749] uio_pci_generic 0000:03:00.1: (PCI Express:5.0GT/s:Width x8)
> [71051.705096] a0:42:3f:29:b3:af
> [71051.705824] uio_pci_generic 0000:03:00.1 (unregistered net_device): MAC: 2, PHY: 0, PBA No: FFFFFF-0FF
> [71051.706452] uio_pci_generic 0000:03:00.1 (unregistered net_device): Enabled Features: RxQ: 1 TxQ: 1
> [71051.708210] uio_pci_generic 0000:03:00.1 (unregistered net_device): Intel(R) 10 Gigabit Network Connection
> [74051.258432] KNI: Successfully release kni named vEth0_0
> [74054.324270] KNI: Successfully release kni named vEth1_0
> [74054.379908] KNI: /dev/kni closed
>
>



-- 
Andriy Berestovskyy

Re: [dpdk-users] Unable to see incoming packets with example KNI application

[Pavey, Nicholas]

Hi Sakthivel, Andriy,

Thanks for the email.

I’m still having trouble, although things do seem to be working better than before.

It seems that the MAC address suggestion and also the IPs on the same class B network aren’t the root cause.


Current symptoms
================

I rebooted my machine, and took a note of the MAC addresses as the machine started up. It turns out that the KNI virtual interfaces actually did initialize with the correct MAC addresses.

I also reconfigured the virtual interfaces to be using different network classes, so that should no longer be a problem.

Something has improved because I’m able to see traffic on the KNI virtual interface with tcpdump, which I was not able to do previously.

Unfortunately, even though tcpdump is able to see the traffic correctly, it seems that the networking stack isn’t working as I would expect.



Outbound
========

Ping
----

I can see outbound ‘pings’ (with initial ARP requests) with ‘tcpdump', and I can see a response coming back. However, the ‘ping’ application reports 100% packet loss.

The ARP traffic is definitely getting out, because I see the IP address registered in the router’s ARP cache. Likewise, I see a response to the original ‘ping’ packet, so the outbound direction seems to be working.


Inbound
=======

The problems seem to be on the inbound side. As we saw above, the outbound side appears to be working reasonably, but I don’t appear to be able to capture inbound packets.

TCP
---

For example, if I set up a simple ‘netcat’ listener (using TCP for transport) on the target server:

  nc –l 172.25.48.200 9876

And then attempt to connect to it from another machine, as follows:

  nc 172.25.48.200 9876


‘tcpdump’ on the target server will show me the incoming ‘syn’ and a ‘syn’ retransmission, but there are no outbound ‘ack’ packets. 


UDP
---

Similarly, inbound UDP traffic never appears to be routed to the user space application.

I can counters incrementing on the virtual interface with ‘sar –n DEV 1 100’. ‘tcpdump’ also shows me the incoming data.

However, if I look at the UDP stats with ‘sar –n UDP 1 100’, I’m not seeing any packets arriving, even with ‘no port’ or ‘idgmerr’, which I’d normally expect if there’s no listening application bound to the IP address.


Next steps
==========

It almost seems as if the receive side of the network stack simply isn’t seeing the inbound data (regardless of whether it’s ICMP, TCP or UDP) and therefore isn’t sending responses.


The thing I’m confused about here is how ‘tcpdump’ is able to see the traffic - after all, if it’s able to see the inbound traffic, then a good part of the RX side of the stack must be working. I’d have thought that if ‘tcpdump’ can see the traffic, then the rest of the stack should be working too.

It makes me wondering whether perhaps I’m misunderstanding the purpose of the KNI system?

My interpretation is that it’s supposed to route traffic from the DPDK into the regular Linux network stack, where it can be used as if it were regular traffic. Do I have that right?



Do you have any ideas? 

Thanks,


Nick


From:  SAKTHIVEL ANAND S <anand.sa88@gmail.com>
Date:  Wednesday, May 11, 2016 at 3:30 AM
To:  "Pavey, Nicholas" <npavey@akamai.com>
Subject:  Re: [dpdk-users] Unable to see incoming packets with example KNI application


Hi

When you try to send echo packets(outbound), your PC will try to resolve ARP, which it could not complete properly .. due to random mac generation for KNI interface(your KNI interface having different MAC than actual interface).


After starting KNI app write your hardware address on KNI by doing, "ifconfig <vETHname> hw ether <real HW address>" and try ping. Let me know the results.


use  "tcpdump -n -i vEth** -e | grep <filter>" 

Regards

Sakthivel S OM

Re: [dpdk-users] Unable to see incoming packets with example KNI application

[Pavey, Nicholas]

Hi Shawn,

I don’t _think_ that’s the problem, but I’m not an expert in low level networking so I could be mistaken.

If I shut down the KNI application, then the vEth* interfaces shut down and the arp table entries related to those interfaces are cleared.

Here’s the arp table before I start the KNI app:

  arp
  Address                  HWtype  HWaddress           Flags Mask            Iface
  172.25.48.100            ether   a0:42:3f:25:0b:a7   C                     eth2
  172.25.48.1              ether   18:e7:28:96:83:01   C                     eth2

172.25.48.1 is the router’s IP. ‘eth2’ is my 1Gbps copper management interface.

I start the KNI app:

  ./build/kni -c 0xf0 -n 4 -- -P -p 0x3 --config="(0,4,6,8),(1,5,7,9)”

The arp table is unchanged from above.

Now I set up the IP addresses for the KNI virtual interfaces:

  ifconfig vEth0_0 172.25.48.200
  ifconfig vEth1_0 10.25.48.200


Arp is still unchanged.

Now, I ping the router using the ‘vEth0_0’ inteface, which will require an ARP to start with:

  ping -I vEth0_0 172.25.48.1

My DPDK machine’s MAC appears in the router’s arp table.

The arp table on the DPDK machine is now:

  Address                  HWtype  HWaddress           Flags Mask            Iface
  >>>> 172.25.48.1              ether   18:e7:28:96:83:01   C                     vEth0_0
  172.25.48.100            ether   a0:42:3f:25:0b:a7   C                     eth2
  172.25.48.1              ether   18:e7:28:96:83:01   C                     eth2


The 1st line is the new entry.

This looks to me like the ARP process is working correctly, and it seems that ‘tcpdump’ is also working as expected.

What do you think?


Thanks,


Nick




From:  Shawn Lewis <smlsr@tencara.com>
Date:  Wednesday, May 11, 2016 at 10:36 AM
To:  "Pavey, Nicholas" <npavey@akamai.com>
Subject:  Re: [dpdk-users] Unable to see incoming packets with example KNI application




> On May 11, 2016, at 10:31 AM, Pavey, Nicholas <npavey@akamai.com> wrote:
> 
> Hi Sakthivel, Andriy,
> 
> Thanks for the email.
> 
> I’m still having trouble, although things do seem to be working better than before.
> 
> It seems that the MAC address suggestion and also the IPs on the same class B network aren’t the root cause.
> 
> 
> Current symptoms
> ================
> 
> I rebooted my machine, and took a note of the MAC addresses as the machine started up. It turns out that the KNI virtual interfaces actually did initialize with the correct MAC addresses.
> 
> I also reconfigured the virtual interfaces to be using different network classes, so that should no longer be a problem.
> 
> Something has improved because I’m able to see traffic on the KNI virtual interface with tcpdump, which I was not able to do previously.
> 
> Unfortunately, even though tcpdump is able to see the traffic correctly, it seems that the networking stack isn’t working as I would expect.
> 
> 
> 
> Outbound
> ========
> 
> Ping
> ----
> 
> I can see outbound ‘pings’ (with initial ARP requests) with ‘tcpdump', and I can see a response coming back. However, the ‘ping’ application reports 100% packet loss.
> 
> The ARP traffic is definitely getting out, because I see the IP address registered in the router’s ARP cache. Likewise, I see a response to the original ‘ping’ packet, so the outbound direction seems to be working.
> 
> 
> Inbound
> =======
> 
> The problems seem to be on the inbound side. As we saw above, the outbound side appears to be working reasonably, but I don’t appear to be able to capture inbound packets.
> 
> TCP
> ---
> 
> For example, if I set up a simple ‘netcat’ listener (using TCP for transport) on the target server:
> 
>  nc –l 172.25.48.200 9876
> 
> And then attempt to connect to it from another machine, as follows:
> 
>  nc 172.25.48.200 9876
> 
> 
> ‘tcpdump’ on the target server will show me the incoming ‘syn’ and a ‘syn’ retransmission, but there are no outbound ‘ack’ packets. 
> 
> 
> UDP
> ---
> 
> Similarly, inbound UDP traffic never appears to be routed to the user space application.
> 
> I can counters incrementing on the virtual interface with ‘sar –n DEV 1 100’. ‘tcpdump’ also shows me the incoming data.
> 
> However, if I look at the UDP stats with ‘sar –n UDP 1 100’, I’m not seeing any packets arriving, even with ‘no port’ or ‘idgmerr’, which I’d normally expect if there’s no listening application bound to the IP address.
> 
> 
> Next steps
> ==========
> 
> It almost seems as if the receive side of the network stack simply isn’t seeing the inbound data (regardless of whether it’s ICMP, TCP or UDP) and therefore isn’t sending responses.
> 
> 
> The thing I’m confused about here is how ‘tcpdump’ is able to see the traffic - after all, if it’s able to see the inbound traffic, then a good part of the RX side of the stack must be working. I’d have thought that if ‘tcpdump’ can see the traffic, then the rest of the stack should be working too.
> 
> It makes me wondering whether perhaps I’m misunderstanding the purpose of the KNI system?
> 
> My interpretation is that it’s supposed to route traffic from the DPDK into the regular Linux network stack, where it can be used as if it were regular traffic. Do I have that right?
> 
> 
> 
> Do you have any ideas? 
> 
> Thanks,
> 
> 
> Nick
> 
> 
> From:  SAKTHIVEL ANAND S <anand.sa88@gmail.com>
> Date:  Wednesday, May 11, 2016 at 3:30 AM
> To:  "Pavey, Nicholas" <npavey@akamai.com>
> Subject:  Re: [dpdk-users] Unable to see incoming packets with example KNI application
> 
> 
> Hi
> 
> When you try to send echo packets(outbound), your PC will try to resolve ARP, which it could not complete properly .. due to random mac generation for KNI interface(your KNI interface having different MAC than actual interface).
> 
> 
> After starting KNI app write your hardware address on KNI by doing, "ifconfig <vETHname> hw ether <real HW address>" and try ping. Let me know the results.
> 
> 
> use  "tcpdump -n -i vEth** -e | grep <filter>" 
> 
> Regards
> 
> Sakthivel S OM
> 
> 

In many cases I have seen where the ether Mac header is 0s for arp packets. So when pushing the packets to the kernel you may have to repopulate the Mac header. This way the kernel can build its internal arp cache. 

Took me a week to find that !!

After doing that all worked fine

Re: [dpdk-users] Unable to see incoming packets with example KNI application

[Andriy Berestovskyy]

I think you should check the ARP table on the other side.  The
destination MAC of the ICMP packets should be the same as your vEth0_0
MAC.

Could you please check the tcpdump -e -i vEth0_0 with the pings and
then ifconfig vEth0_0?

Andriy

On Wed, May 11, 2016 at 5:00 PM, Pavey, Nicholas <npavey@akamai.com> wrote:
> Hi Shawn,
>
> I don’t _think_ that’s the problem, but I’m not an expert in low level networking so I could be mistaken.
>
> If I shut down the KNI application, then the vEth* interfaces shut down and the arp table entries related to those interfaces are cleared.
>
> Here’s the arp table before I start the KNI app:
>
>   arp
>   Address                  HWtype  HWaddress           Flags Mask            Iface
>   172.25.48.100            ether   a0:42:3f:25:0b:a7   C                     eth2
>   172.25.48.1              ether   18:e7:28:96:83:01   C                     eth2
>
> 172.25.48.1 is the router’s IP. ‘eth2’ is my 1Gbps copper management interface.
>
> I start the KNI app:
>
>   ./build/kni -c 0xf0 -n 4 -- -P -p 0x3 --config="(0,4,6,8),(1,5,7,9)”
>
> The arp table is unchanged from above.
>
> Now I set up the IP addresses for the KNI virtual interfaces:
>
>   ifconfig vEth0_0 172.25.48.200
>   ifconfig vEth1_0 10.25.48.200
>
>
> Arp is still unchanged.
>
> Now, I ping the router using the ‘vEth0_0’ inteface, which will require an ARP to start with:
>
>   ping -I vEth0_0 172.25.48.1
>
> My DPDK machine’s MAC appears in the router’s arp table.
>
> The arp table on the DPDK machine is now:
>
>   Address                  HWtype  HWaddress           Flags Mask            Iface
>   >>>> 172.25.48.1              ether   18:e7:28:96:83:01   C                     vEth0_0
>   172.25.48.100            ether   a0:42:3f:25:0b:a7   C                     eth2
>   172.25.48.1              ether   18:e7:28:96:83:01   C                     eth2
>
>
> The 1st line is the new entry.
>
> This looks to me like the ARP process is working correctly, and it seems that ‘tcpdump’ is also working as expected.
>
> What do you think?
>
>
> Thanks,
>
>
> Nick
>
>
>
>
> From:  Shawn Lewis <smlsr@tencara.com>
> Date:  Wednesday, May 11, 2016 at 10:36 AM
> To:  "Pavey, Nicholas" <npavey@akamai.com>
> Subject:  Re: [dpdk-users] Unable to see incoming packets with example KNI application
>
>
>
>
>> On May 11, 2016, at 10:31 AM, Pavey, Nicholas <npavey@akamai.com> wrote:
>>
>> Hi Sakthivel, Andriy,
>>
>> Thanks for the email.
>>
>> I’m still having trouble, although things do seem to be working better than before.
>>
>> It seems that the MAC address suggestion and also the IPs on the same class B network aren’t the root cause.
>>
>>
>> Current symptoms
>> ================
>>
>> I rebooted my machine, and took a note of the MAC addresses as the machine started up. It turns out that the KNI virtual interfaces actually did initialize with the correct MAC addresses.
>>
>> I also reconfigured the virtual interfaces to be using different network classes, so that should no longer be a problem.
>>
>> Something has improved because I’m able to see traffic on the KNI virtual interface with tcpdump, which I was not able to do previously.
>>
>> Unfortunately, even though tcpdump is able to see the traffic correctly, it seems that the networking stack isn’t working as I would expect.
>>
>>
>>
>> Outbound
>> ========
>>
>> Ping
>> ----
>>
>> I can see outbound ‘pings’ (with initial ARP requests) with ‘tcpdump', and I can see a response coming back. However, the ‘ping’ application reports 100% packet loss.
>>
>> The ARP traffic is definitely getting out, because I see the IP address registered in the router’s ARP cache. Likewise, I see a response to the original ‘ping’ packet, so the outbound direction seems to be working.
>>
>>
>> Inbound
>> =======
>>
>> The problems seem to be on the inbound side. As we saw above, the outbound side appears to be working reasonably, but I don’t appear to be able to capture inbound packets.
>>
>> TCP
>> ---
>>
>> For example, if I set up a simple ‘netcat’ listener (using TCP for transport) on the target server:
>>
>>  nc –l 172.25.48.200 9876
>>
>> And then attempt to connect to it from another machine, as follows:
>>
>>  nc 172.25.48.200 9876
>>
>>
>> ‘tcpdump’ on the target server will show me the incoming ‘syn’ and a ‘syn’ retransmission, but there are no outbound ‘ack’ packets.
>>
>>
>> UDP
>> ---
>>
>> Similarly, inbound UDP traffic never appears to be routed to the user space application.
>>
>> I can counters incrementing on the virtual interface with ‘sar –n DEV 1 100’. ‘tcpdump’ also shows me the incoming data.
>>
>> However, if I look at the UDP stats with ‘sar –n UDP 1 100’, I’m not seeing any packets arriving, even with ‘no port’ or ‘idgmerr’, which I’d normally expect if there’s no listening application bound to the IP address.
>>
>>
>> Next steps
>> ==========
>>
>> It almost seems as if the receive side of the network stack simply isn’t seeing the inbound data (regardless of whether it’s ICMP, TCP or UDP) and therefore isn’t sending responses.
>>
>>
>> The thing I’m confused about here is how ‘tcpdump’ is able to see the traffic - after all, if it’s able to see the inbound traffic, then a good part of the RX side of the stack must be working. I’d have thought that if ‘tcpdump’ can see the traffic, then the rest of the stack should be working too.
>>
>> It makes me wondering whether perhaps I’m misunderstanding the purpose of the KNI system?
>>
>> My interpretation is that it’s supposed to route traffic from the DPDK into the regular Linux network stack, where it can be used as if it were regular traffic. Do I have that right?
>>
>>
>>
>> Do you have any ideas?
>>
>> Thanks,
>>
>>
>> Nick
>>
>>
>> From:  SAKTHIVEL ANAND S <anand.sa88@gmail.com>
>> Date:  Wednesday, May 11, 2016 at 3:30 AM
>> To:  "Pavey, Nicholas" <npavey@akamai.com>
>> Subject:  Re: [dpdk-users] Unable to see incoming packets with example KNI application
>>
>>
>> Hi
>>
>> When you try to send echo packets(outbound), your PC will try to resolve ARP, which it could not complete properly .. due to random mac generation for KNI interface(your KNI interface having different MAC than actual interface).
>>
>>
>> After starting KNI app write your hardware address on KNI by doing, "ifconfig <vETHname> hw ether <real HW address>" and try ping. Let me know the results.
>>
>>
>> use  "tcpdump -n -i vEth** -e | grep <filter>"
>>
>> Regards
>>
>> Sakthivel S OM
>>
>>
>
> In many cases I have seen where the ether Mac header is 0s for arp packets. So when pushing the packets to the kernel you may have to repopulate the Mac header. This way the kernel can build its internal arp cache.
>
> Took me a week to find that !!
>
> After doing that all worked fine
>
>
>



-- 
Andriy Berestovskyy

Re: [dpdk-users] Unable to see incoming packets with example KNI application

[Ferruh Yigit]

On 5/11/2016 3:31 PM, Pavey, Nicholas wrote:
> Hi Sakthivel, Andriy,
> 
> Thanks for the email.
> 
> I’m still having trouble, although things do seem to be working better than before.
> 
> It seems that the MAC address suggestion and also the IPs on the same class B network aren’t the root cause.
> 
> 
> Current symptoms
> ================
> 
> I rebooted my machine, and took a note of the MAC addresses as the machine started up. It turns out that the KNI virtual interfaces actually did initialize with the correct MAC addresses.
> 
> I also reconfigured the virtual interfaces to be using different network classes, so that should no longer be a problem.
> 
> Something has improved because I’m able to see traffic on the KNI virtual interface with tcpdump, which I was not able to do previously.
> 
> Unfortunately, even though tcpdump is able to see the traffic correctly, it seems that the networking stack isn’t working as I would expect.
> 
> 
> 
> Outbound
> ========
> 
> Ping
> ----
> 
> I can see outbound ‘pings’ (with initial ARP requests) with ‘tcpdump', and I can see a response coming back. However, the ‘ping’ application reports 100% packet loss.
> 
> The ARP traffic is definitely getting out, because I see the IP address registered in the router’s ARP cache. Likewise, I see a response to the original ‘ping’ packet, so the outbound direction seems to be working.
> 
> 
> Inbound
> =======
> 
> The problems seem to be on the inbound side. As we saw above, the outbound side appears to be working reasonably, but I don’t appear to be able to capture inbound packets.
> 
> TCP
> ---
> 
> For example, if I set up a simple ‘netcat’ listener (using TCP for transport) on the target server:
> 
>   nc –l 172.25.48.200 9876
> 
> And then attempt to connect to it from another machine, as follows:
> 
>   nc 172.25.48.200 9876
> 
> 
> ‘tcpdump’ on the target server will show me the incoming ‘syn’ and a ‘syn’ retransmission, but there are no outbound ‘ack’ packets. 
> 
> 
> UDP
> ---
> 
> Similarly, inbound UDP traffic never appears to be routed to the user space application.
> 
> I can counters incrementing on the virtual interface with ‘sar –n DEV 1 100’. ‘tcpdump’ also shows me the incoming data.
> 
> However, if I look at the UDP stats with ‘sar –n UDP 1 100’, I’m not seeing any packets arriving, even with ‘no port’ or ‘idgmerr’, which I’d normally expect if there’s no listening application bound to the IP address.
> 
> 
> Next steps
> ==========
> 
> It almost seems as if the receive side of the network stack simply isn’t seeing the inbound data (regardless of whether it’s ICMP, TCP or UDP) and therefore isn’t sending responses.
> 
> 
> The thing I’m confused about here is how ‘tcpdump’ is able to see the traffic - after all, if it’s able to see the inbound traffic, then a good part of the RX side of the stack must be working. I’d have thought that if ‘tcpdump’ can see the traffic, then the rest of the stack should be working too.
> 
> It makes me wondering whether perhaps I’m misunderstanding the purpose of the KNI system?
> 
> My interpretation is that it’s supposed to route traffic from the DPDK into the regular Linux network stack, where it can be used as if it were regular traffic. Do I have that right?
> 
> 
> 
> Do you have any ideas? 
> 
> Thanks,
> 
> 
> Nick
> 
> 
> From:  SAKTHIVEL ANAND S <anand.sa88@gmail.com>
> Date:  Wednesday, May 11, 2016 at 3:30 AM
> To:  "Pavey, Nicholas" <npavey@akamai.com>
> Subject:  Re: [dpdk-users] Unable to see incoming packets with example KNI application
> 
> 
> Hi
> 
> When you try to send echo packets(outbound), your PC will try to resolve ARP, which it could not complete properly .. due to random mac generation for KNI interface(your KNI interface having different MAC than actual interface).
> 
> 
> After starting KNI app write your hardware address on KNI by doing, "ifconfig <vETHname> hw ether <real HW address>" and try ping. Let me know the results.
> 
> 
> use  "tcpdump -n -i vEth** -e | grep <filter>" 
> 
> Regards
> 
> Sakthivel S OM
> 
> 

Hi Nick,

This may not help for your problem but can work as a reference:

I did test two things with latest code and both worked well:
1- Two NICs , as in your case, with packet generator each end.
Internally created an bridge and added KNI interfaces to bridge, I
observe received packets in generator.

2- Tested with netcat, KNI interface able to send/receive packets, both
for udp and tcp.

I checked for your case, not able to find any obvious error.

Regards,
ferruh