DPDK server packets with AF_PACKET driver are not received by the client connected to the same virtual bridge.

DPDK server packets with AF_PACKET driver are not received by the client connected to the same virtual bridge.

[Boris Ouretskey]

[-- Attachment #1: Type: text/plain, Size: 2481 bytes --]

Hi

I am trying to run a docker based system in one VM, with some servers using
the DPDK and facing a very strange issue.

The relevant part of the system is the echo server which is built with DPDK
library and runs in docker, connected to the virtual bridge (with AF_PACKET
driver), and echo client built with regular sockets (java application). See
the drawing below for visual details.

The strange thing is that echo replies sent from DPDK echo server docker
(Docker2), can be seen in tcpdump in the docker of echo client (Docker1)
however the application (java client) does not receive it on socket level.
When the echo server runs as a socket application everything works. Seams
like OS (linux) decide not to route the packet to the application for
some reason.

Some quick important notes.

- DPDK layer starts ok and no EAL errors are reported. Its echo replies as
mentioned before can clearly be seen when capturing traffic on eth0 in
client docker.
- It is not the issue of parsing the echo replies on application level. I
checked the strace logs and cannot see any traffic delivered to the
application
- I have compared the captures from working socket based echo server and
MACs seam to be ok, also the packets sent from DPDK server are formed well.
- No logs for martian packets when enabled.

I would like to focus the attention of dear DPDK experts on the following
question: is it possible that DPDK AF_PACKET driver intercepts both packets
destined to  the virtual nic eth0 of Docker1 AND eth0 of Docker2,
preventing it from being received by application?

DPDK Version 18

VIRTUAL BOX VM (Alma Linux)

    Docker1                                 Docker2
+---------------+                       +------------+
|               |    Request            |            |
|               |   -------------->     |  DPDK      |
| Socket Echo   |                       |  Echo Server
|   Client      |         Reply         |  Server    |
|               |   <-------------      |            |
--vdev=eth_af_packet0,iface=eth0,qdisc_bypass=1
|               |                       |            |
|          tcpdump -i eth0              +------+-----+
+-------+-------+                       eth0   |
        | eth0                                 |
        |                               No IP  |
 +------v--------------------------------------v-----+
 |             Virtual Docker Bridge                 |
 +---------------------------------------------------+


Thanks

[-- Attachment #2: Type: text/html, Size: 3356 bytes --]

Re: DPDK server packets with AF_PACKET driver are not received by the client connected to the same virtual bridge.

[J.J. Mars]

[-- Attachment #1: Type: text/plain, Size: 2980 bytes --]

If the nic on docker 1 of echo client can receive replies, the reason
should not be PMD I think.
Did you make sure every field in every layer is completely the same between
eth0 in docker 1 and eth0 in docker 2, and all of them are expected?

Boris Ouretskey <borisusun@gmail.com> 于2022年11月9日周三 03:51写道：

>
>
> Hi
>
> I am trying to run a docker based system in one VM, with some servers
> using the DPDK and facing a very strange issue.
>
> The relevant part of the system is the echo server which is built with
> DPDK library and runs in docker, connected to the virtual bridge (with
> AF_PACKET driver), and echo client built with regular sockets (java
> application). See the drawing below for visual details.
>
> The strange thing is that echo replies sent from DPDK echo server docker
> (Docker2), can be seen in tcpdump in the docker of echo client (Docker1)
> however the application (java client) does not receive it on socket level.
> When the echo server runs as a socket application everything works. Seams
> like OS (linux) decide not to route the packet to the application for
> some reason.
>
> Some quick important notes.
>
> - DPDK layer starts ok and no EAL errors are reported. Its echo replies as
> mentioned before can clearly be seen when capturing traffic on eth0 in
> client docker.
> - It is not the issue of parsing the echo replies on application level. I
> checked the strace logs and cannot see any traffic delivered to the
> application
> - I have compared the captures from working socket based echo server and
> MACs seam to be ok, also the packets sent from DPDK server are formed well.
> - No logs for martian packets when enabled.
>
> I would like to focus the attention of dear DPDK experts on the following
> question: is it possible that DPDK AF_PACKET driver intercepts both packets
> destined to  the virtual nic eth0 of Docker1 AND eth0 of Docker2,
> preventing it from being received by application?
>
> DPDK Version 18
>
> VIRTUAL BOX VM (Alma Linux)
>
>     Docker1                                 Docker2
> +---------------+                       +------------+
> |               |    Request            |            |
> |               |   -------------->     |  DPDK      |
> | Socket Echo   |                       |  Echo Server
> |   Client      |         Reply         |  Server    |
> |               |   <-------------      |            |
> --vdev=eth_af_packet0,iface=eth0,qdisc_bypass=1
> |               |                       |            |
> |          tcpdump -i eth0              +------+-----+
> +-------+-------+                       eth0   |
>         | eth0                                 |
>         |                               No IP  |
>  +------v--------------------------------------v-----+
>  |             Virtual Docker Bridge                 |
>  +---------------------------------------------------+
>
>
> Thanks
>

[-- Attachment #2: Type: text/html, Size: 3860 bytes --]

Re: DPDK server packets with AF_PACKET driver are not received by the client connected to the same virtual bridge.

[Boris Ouretskey]

[-- Attachment #1.1: Type: text/plain, Size: 5274 bytes --]

Hi

I have narrowed down the problem of sending packets via the
AF_PACKET/SOCK_RAW interface, which might be what DPDK does in coressonding
PMD drivers. The problem does not seem to be related to DPDK codebase
directly, but still can be interesting for the users who are trying to use
DPDK with AF_PACKET driver inside a dockerized environment.

To reiterate the issue: Any UDP message sent via AF_PACKET/SOCK_RAW socket
on one interface on virtual bridge, is not received by application
listening on the other interface attached to the same bridge, however
strangely enough it can be seen in tcpdump listening on the receiving
interface. The application I used based on the example here [
https://www.kernel.org/doc/html/latest/_sources/networking/packet_mmap.rst.txt]
but instead of receiving I added a loop for sending 5 pre-recorded L2
frames.

I am attaching the small project to easily reproduce the issue (assuming
the MAC addresses are preserved between machines by the docker because the
L2 packet is pre-recorded) .
------
The environment is two dockers (basic AlmaOS 8.5 having only network utils
installed) attached to the same bridge. The environment is set up with
docker-compose.

To reproduce the issue:

- make dockers and applications using Makefile and start the dockers.


*make docker-compose up*
- in other window set up server in first docker:


* docker exec -it client /bin/bash nc -l -4 -u 192.168.2.1 5997*
- in other window set up tcpdump to see the incoming packets on 5997 port:


* docker exec -it client /bin/bash tcpdump -vneA -i eth0 port 5997*
- in other window send  5 messages  via application then via netcat:



* docker exec -it server /bin/bash /share/a.out eth0 hash printf "ok1\r\n"
| nc -s 192.168.2.5 -p 5998 -u -4 192.168.2.1 5997*
Observe that only packets sent by nc are received. Though there's no
difference in L2 frame structure apart from identification and checksum
(wrong for both cases).

The question is how can I debug the kernel to understand what makes it drop
the packet and not to pass it to the application (netcat in this case).

On Wed, Nov 9, 2022 at 10:50 AM J.J. Mars <mars14850@gmail.com> wrote:

> If the nic on docker 1 of echo client can receive replies, the reason
> should not be PMD I think.
> Did you make sure every field in every layer is completely the same
> between eth0 in docker 1 and eth0 in docker 2, and all of them are expected?
>
> Boris Ouretskey <borisusun@gmail.com> 于2022年11月9日周三 03:51写道：
>
>>
>>
>> Hi
>>
>> I am trying to run a docker based system in one VM, with some servers
>> using the DPDK and facing a very strange issue.
>>
>> The relevant part of the system is the echo server which is built with
>> DPDK library and runs in docker, connected to the virtual bridge (with
>> AF_PACKET driver), and echo client built with regular sockets (java
>> application). See the drawing below for visual details.
>>
>> The strange thing is that echo replies sent from DPDK echo server docker
>> (Docker2), can be seen in tcpdump in the docker of echo client (Docker1)
>> however the application (java client) does not receive it on socket level.
>> When the echo server runs as a socket application everything works. Seams
>> like OS (linux) decide not to route the packet to the application for
>> some reason.
>>
>> Some quick important notes.
>>
>> - DPDK layer starts ok and no EAL errors are reported. Its echo replies
>> as mentioned before can clearly be seen when capturing traffic on eth0 in
>> client docker.
>> - It is not the issue of parsing the echo replies on application level. I
>> checked the strace logs and cannot see any traffic delivered to the
>> application
>> - I have compared the captures from working socket based echo server and
>> MACs seam to be ok, also the packets sent from DPDK server are formed well.
>> - No logs for martian packets when enabled.
>>
>> I would like to focus the attention of dear DPDK experts on the following
>> question: is it possible that DPDK AF_PACKET driver intercepts both packets
>> destined to  the virtual nic eth0 of Docker1 AND eth0 of Docker2,
>> preventing it from being received by application?
>>
>> DPDK Version 18
>>
>> VIRTUAL BOX VM (Alma Linux)
>>
>>     Docker1                                 Docker2
>> +---------------+                       +------------+
>> |               |    Request            |            |
>> |               |   -------------->     |  DPDK      |
>> | Socket Echo   |                       |  Echo Server
>> |   Client      |         Reply         |  Server    |
>> |               |   <-------------      |            |
>> --vdev=eth_af_packet0,iface=eth0,qdisc_bypass=1
>> |               |                       |            |
>> |          tcpdump -i eth0              +------+-----+
>> +-------+-------+                       eth0   |
>>         | eth0                                 |
>>         |                               No IP  |
>>  +------v--------------------------------------v-----+
>>  |             Virtual Docker Bridge                 |
>>  +---------------------------------------------------+
>>
>>
>> Thanks
>>
>

[-- Attachment #1.2: Type: text/html, Size: 6549 bytes --]

[-- Attachment #2: test.tar.gz --]
[-- Type: application/x-gzip, Size: 2398 bytes --]