Hi

I have narrowed down the problem of sending packets via the AF_PACKET/SOCK_RAW interface, which might be what DPDK does in coressonding PMD drivers. The problem does not seem to be related to DPDK codebase directly, but still can be interesting for the users who are trying to use DPDK with AF_PACKET driver inside a dockerized environment.

To reiterate the issue: Any UDP message sent via AF_PACKET/SOCK_RAW socket on one interface on virtual bridge, is not received by application listening on the other interface attached to the same bridge, however strangely enough it can be seen in tcpdump listening on the receiving interface. The application I used based on the example here [https://www.kernel.org/doc/html/latest/_sources/networking/packet_mmap.rst.txt] but instead of receiving I added a loop for sending 5 pre-recorded L2 frames.

I am attaching the small project to easily reproduce the issue (assuming the MAC addresses are preserved between machines by the docker because the L2 packet is pre-recorded) .
------
The environment is two dockers (basic AlmaOS 8.5 having only network utils installed) attached to the same bridge. The environment is set up with docker-compose.

To reproduce the issue:

- make dockers and applications using Makefile and start the dockers.
make
docker-compose up

- in other window set up server in first docker:
docker exec -it client /bin/bash
nc -l -4 -u 192.168.2.1 5997

- in other window set up tcpdump to see the incoming packets on 5997 port:
docker exec -it client /bin/bash
tcpdump -vneA -i eth0 port 5997

- in other window send  5 messages  via application then via netcat:
docker exec -it server /bin/bash
/share/a.out eth0 hash
printf "ok1\r\n" | nc -s 192.168.2.5 -p 5998 -u -4 192.168.2.1 5997

Observe that only packets sent by nc are received. Though there's no difference in L2 frame structure apart from identification and checksum (wrong for both cases).

The question is how can I debug the kernel to understand what makes it drop the packet and not to pass it to the application (netcat in this case). 

On Wed, Nov 9, 2022 at 10:50 AM J.J. Mars <mars14850@gmail.com> wrote:
If the nic on docker 1 of echo client can receive replies, the reason should not be PMD I think.
Did you make sure every field in every layer is completely the same between eth0 in docker 1 and eth0 in docker 2, and all of them are expected?

Boris Ouretskey <borisusun@gmail.com> 于2022年11月9日周三 03:51写道：


Hi

I am trying to run a docker based system in one VM, with some servers using the DPDK and facing a very strange issue.

The relevant part of the system is the echo server which is built with DPDK library and runs in docker, connected to the virtual bridge (with AF_PACKET driver), and echo client built with regular sockets (java application). See the drawing below for visual details.

The strange thing is that echo replies sent from DPDK echo server docker (Docker2), can be seen in tcpdump in the docker of echo client (Docker1) however the application (java client) does not receive it on socket level. When the echo server runs as a socket application everything works. Seams like OS (linux) decide not to route the packet to the application for some reason.

Some quick important notes.

- DPDK layer starts ok and no EAL errors are reported. Its echo replies as mentioned before can clearly be seen when capturing traffic on eth0 in client docker.
- It is not the issue of parsing the echo replies on application level. I checked the strace logs and cannot see any traffic delivered to the application
- I have compared the captures from working socket based echo server and MACs seam to be ok, also the packets sent from DPDK server are formed well.
- No logs for martian packets when enabled.

I would like to focus the attention of dear DPDK experts on the following question: is it possible that DPDK AF_PACKET driver intercepts both packets destined to  the virtual nic eth0 of Docker1 AND eth0 of Docker2, preventing it from being received by application?

DPDK Version 18

VIRTUAL BOX VM (Alma Linux)

    Docker1                                 Docker2
+---------------+                       +------------+
|               |    Request            |            |
|               |   -------------->     |  DPDK      |
| Socket Echo   |                       |  Echo Server
|   Client      |         Reply         |  Server    |
|               |   <-------------      |            | --vdev=eth_af_packet0,iface=eth0,qdisc_bypass=1
|               |                       |            |
|          tcpdump -i eth0              +------+-----+
+-------+-------+                       eth0   |
        | eth0                                 |
        |                               No IP  |
 +------v--------------------------------------v-----+
 |             Virtual Docker Bridge                 |
 +---------------------------------------------------+
 

Thanks