From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 3AE4B45E3D; Fri, 6 Dec 2024 00:01:47 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id DAC204027D; Fri, 6 Dec 2024 00:01:46 +0100 (CET) Received: from inbox.dpdk.org (inbox.dpdk.org [95.142.172.178]) by mails.dpdk.org (Postfix) with ESMTP id A555D40267 for ; Fri, 6 Dec 2024 00:01:44 +0100 (CET) Received: by inbox.dpdk.org (Postfix, from userid 33) id 8E7D145E3E; Fri, 6 Dec 2024 00:01:44 +0100 (CET) From: bugzilla@dpdk.org To: dev@dpdk.org Subject: [DPDK/ethdev Bug 1592] AF_PACKET PMD loops back packets on veth with tc Date: Thu, 05 Dec 2024 23:01:44 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: DPDK X-Bugzilla-Component: ethdev X-Bugzilla-Version: unspecified X-Bugzilla-Keywords: X-Bugzilla-Severity: normal X-Bugzilla-Who: thea.rossman@cs.stanford.edu X-Bugzilla-Status: UNCONFIRMED X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: dev@dpdk.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter target_milestone Message-ID: Content-Type: multipart/alternative; boundary=17334397040.B3418AAf.322596 Content-Transfer-Encoding: 7bit X-Bugzilla-URL: http://bugs.dpdk.org/ Auto-Submitted: auto-generated X-Auto-Response-Suppress: All MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org --17334397040.B3418AAf.322596 Date: Fri, 6 Dec 2024 00:01:44 +0100 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: http://bugs.dpdk.org/ Auto-Submitted: auto-generated X-Auto-Response-Suppress: All https://bugs.dpdk.org/show_bug.cgi?id=3D1592 Bug ID: 1592 Summary: AF_PACKET PMD loops back packets on veth with tc Product: DPDK Version: unspecified Hardware: All OS: All Status: UNCONFIRMED Severity: normal Priority: Normal Component: ethdev Assignee: dev@dpdk.org Reporter: thea.rossman@cs.stanford.edu Target Milestone: --- Found using AF_PACKET PMD on veth with Linux TC via Mininet. Ubuntu 24.04.1 LTS. High-level, I have a basic Mininet topology with `h1 <-> r1 <-> h2`. (Each = link represents a pair of veths.) `r1` is a transparent middlebox running the DP= DK `skeleton/basicfwd.c` example, which should forward packets through r1 (eth= 0 -> eth1 and vice verse). I ping h1 <-> h2, expecting `r1` to perform this bridging.=20 The DPDK basicfwd program was started with the vdev configuration: `--vdev=3Deth_af_packet0,iface=3Dr1-eth0,qdisc_bypass=3D0 --vdev=3Deth_af_packet1,iface=3Dr1-eth1,qdisc_bypass=3D0 &`. (I turn off qdisc_bypass because I'd like to use TC.)=20 When one of r1's veths have TC params (e.g., delay), this all works fine.=20 However, when both of r1's veths (r1-eth0 and r1-eth1) have TC params set u= p, traffic is looped forever. Here's an example output from h1:=20 ``` ... 64 bytes from 10.0.1.9: icmp_seq=3D1 ttl=3D64 time=3D161 ms (DUP!) 64 bytes from 10.0.1.9: icmp_seq=3D1 ttl=3D64 time=3D161 ms (DUP!) 64 bytes from 10.0.1.9: icmp_seq=3D1 ttl=3D64 time=3D161 ms (DUP!) 64 bytes from 10.0.1.9: icmp_seq=3D1 ttl=3D64 time=3D161 ms (DUP!) 64 bytes from 10.0.1.9: icmp_seq=3D1 ttl=3D64 time=3D161 ms (DUP!) 64 bytes from 10.0.1.9: icmp_seq=3D1 ttl=3D64 time=3D181 ms (DUP!) ... (continues perpetually until stopped) ``` Note that this issue only occurs if TC parameters (e.g., delay) are configu= red on both egress veths. If qdisc_bypass is disabled on both, but no actual TC parameters are set, the duplication does not happen.=20 I took packet captures (tcpdump), and I also printed packets that the application was actually seeing. It appears that when the middlebox writes a packet to a socket, it immediately reads the same packet out from the same socket.=20 I also tried making small modifications in the DPDK driver to investigate whether this could be a Linux or Mininet issue (e.g., packet mmap + TC?) vs= . a DPDK driver issue. I reduced the # of RX/TX queues to 1, and I also turned = of packet mmap (by removing the setsockopt calls in rte_eth_af_packet.c for `PACKET_RX_RING` and `PACKET_TX_RING` -- not sure if there's anything else I would need to do here?).=20=20 A basic python script for repro is below. This works fine when only one of = the `r1.cmd("tc qdisc add dev r1-ethX root netem delay 10ms")` commands is pres= ent. When both are added, the pings begin to loop.=20 ```python from mininet.net import Mininet from mininet.link import TCLink from mininet.cli import CLI net =3D Mininet(controller=3DNone, link=3DTCLink) # Add hosts in same subnet h1 =3D net.addHost('h1', ip=3D'10.0.1.10/24', mac=3D'00:00:00:00:00:01') h2 =3D net.addHost('h2', ip=3D'10.0.1.9/24', mac=3D'00:00:00:00:00:02') r1 =3D net.addHost('r1') net.addLink(r1, h1) net.addLink(r1, h2) net.build() # Configure TC on egress=20 # (could also be done by adding `delay` param to `addLink` above) h2.cmd("tc qdisc add dev h2-eth0 root netem delay 10ms") h1.cmd("tc qdisc add dev h1-eth0 root netem delay 10ms") r1.cmd("tc qdisc add dev r1-eth0 root netem delay 10ms") r1.cmd("tc qdisc add dev r1-eth1 root netem delay 10ms") r1.cmd('sudo /path/to/dpdk-24.07/examples/skeleton/build/basicfwd 0 1 --vdev=3Deth_af_packet0,iface=3Dr1-eth0,qdisc_bypass=3D0 --vdev=3Deth_af_packet1,iface=3Dr1-eth1,qdisc_bypass=3D0 &') CLI(net) net.stop() ``` I think there is a bug here, though I haven't been able to figure out for s= ure what might be going on or confirm 100% whether the issue is in the PMD. --=20 You are receiving this mail because: You are the assignee for the bug.= --17334397040.B3418AAf.322596 Date: Fri, 6 Dec 2024 00:01:44 +0100 MIME-Version: 1.0 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: http://bugs.dpdk.org/ Auto-Submitted: auto-generated X-Auto-Response-Suppress: All
Bug ID 1592
Summary AF_PACKET PMD loops back packets on veth with tc
Product DPDK
Version unspecified
Hardware All
OS All
Status UNCONFIRMED
Severity normal
Priority Normal
Component ethdev
Assignee dev@dpdk.org
Reporter thea.rossman@cs.stanford.edu
Target Milestone --- 

Found using AF_PACKET PMD on veth =
with Linux TC via Mininet. Ubuntu 24.04.1
LTS.

High-level, I have a basic Mininet topology with `h1 <-> r1 <->=
 h2`. (Each link
represents a pair of veths.) `r1` is a transparent middlebox running the DP=
DK
`skeleton/basicfwd.c` example, which should forward packets through r1 (eth=
0 ->
eth1 and vice verse). I ping h1 <-> h2, expecting `r1` to perform this
bridging.=20

The DPDK basicfwd program was started with the vdev configuration:
`--vdev=3Deth_af_packet0,iface=3Dr1-eth0,qdisc_bypass=3D0
--vdev=3Deth_af_packet1,iface=3Dr1-eth1,qdisc_bypass=3D0 &`.

(I turn off qdisc_bypass because I'd like to use TC.)=20

When one of r1's veths have TC params (e.g., delay), this all works fine.=20

However, when both of r1's veths (r1-eth0 and r1-eth1) have TC params set u=
p,
traffic is looped forever. Here's an example output from h1:=20

```
...
64 bytes from 10.0.1.9: icmp_seq=3D1 ttl=3D64 time=3D161 ms (DUP!)
64 bytes from 10.0.1.9: icmp_seq=3D1 ttl=3D64 time=3D161 ms (DUP!)
64 bytes from 10.0.1.9: icmp_seq=3D1 ttl=3D64 time=3D161 ms (DUP!)
64 bytes from 10.0.1.9: icmp_seq=3D1 ttl=3D64 time=3D161 ms (DUP!)
64 bytes from 10.0.1.9: icmp_seq=3D1 ttl=3D64 time=3D161 ms (DUP!)
64 bytes from 10.0.1.9: icmp_seq=3D1 ttl=3D64 time=3D181 ms (DUP!)
... (continues perpetually until stopped)
```

Note that this issue only occurs if TC parameters (e.g., delay) are configu=
red
on both egress veths. If qdisc_bypass is disabled on both, but no actual TC
parameters are set, the duplication does not happen.=20

I took packet captures (tcpdump), and I also printed packets that the
application was actually seeing. It appears that when the middlebox writes a
packet to a socket, it immediately reads the same packet out from the same
socket.=20

I also tried making small modifications in the DPDK driver to investigate
whether this could be a Linux or Mininet issue (e.g., packet mmap + TC?) vs=
. a
DPDK driver issue. I reduced the # of RX/TX queues to 1, and I also turned =
of
packet mmap (by removing the setsockopt calls in rte_eth_af_packet.c for
`PACKET_RX_RING` and `PACKET_TX_RING` -- not sure if there's anything else I
would need to do here?).=20=20

A basic python script for repro is below. This works fine when only one of =
the
`r1.cmd("tc qdisc add dev r1-ethX root netem delay 10ms")` comman=
ds is present.
When both are added, the pings begin to loop.=20

```python
from mininet.net import Mininet
from mininet.link import TCLink
from mininet.cli import CLI

net =3D Mininet(controller=3DNone, link=3DTCLink)
# Add hosts in same subnet
h1 =3D net.addHost('h1', ip=3D'10.0.1.10/24', mac=3D'00:00:00:00:00:01')
h2 =3D net.addHost('h2', ip=3D'10.0.1.9/24', mac=3D'00:00:00:00:00:02')
r1 =3D net.addHost('r1')
net.addLink(r1, h1)
net.addLink(r1, h2)
net.build()

# Configure TC on egress=20
# (could also be done by adding `delay` param to `addLink` above)
h2.cmd("tc qdisc add dev h2-eth0 root netem delay 10ms")
h1.cmd("tc qdisc add dev h1-eth0 root netem delay 10ms")
r1.cmd("tc qdisc add dev r1-eth0 root netem delay 10ms")
r1.cmd("tc qdisc add dev r1-eth1 root netem delay 10ms")

r1.cmd('sudo /path/to/dpdk-24.07/examples/skeleton/build/basicfwd 0 1
--vdev=3Deth_af_packet0,iface=3Dr1-eth0,qdisc_bypass=3D0
--vdev=3Deth_af_packet1,iface=3Dr1-eth1,qdisc_bypass=3D0 &')

CLI(net)
net.stop()
```

I think there is a bug here, though I haven't been able to figure out for s=
ure
what might be going on or confirm 100% whether the issue is in the PMD.

You are receiving this mail because:
  • You are the assignee for the bug.
=20=20=20=20=20=20=20=20=20=20
= --17334397040.B3418AAf.322596--