From: wenxu <wenxu@ucloud.cn>
To: dev@dpdk.org, asafp@nvidia.com
Subject: [dpdk-dev] multiple dpdk instances problem in mlx5
Date: Tue, 11 May 2021 14:49:57 +0800 (GMT+08:00) [thread overview]
Message-ID: <AKYA*AAqFESkpTyUsTY1xKrI.1.1620715797762.Hmail.wenxu@ucloud.cn> (raw)
Hi mlnx teams,
I test the upstream dpdk with the mutiple dpdk instance with dpdk-testpmd
I start two testpmd program and only receive the vxlan packets.
# ./dpdk-testpmd -c 0x1f -n 4 -m 4096 --file-prefix=one -w "0000:19:00.0" --huge-dir=/mnt/dpdk1 -- -i --flow-isolate-all --forward-mode=rxonly --rxq=4 --txq=4 --auto-start --nb-cores=4
testpmd>> flow create 0 ingress pattern eth / ipv4 / udp / vxlan / end actions rss level 2 types ip udp tcp end queues 0 1 2 3 end / end
# ./dpdk-testpmd -c 0x1f00 -n 4 -m 4096 --file-prefix=two -w "0000:19:00.0" --huge-dir=/mnt/pdk2 -- -i --flow-isolate-all --forward-mode=rxonly --rxq=4 --txq=4 --auto-start --nb-cores=4
testpmd>> flow create 0 ingress pattern eth / ipv4 / udp / vxlan / end actions rss level 2 types ip udp tcp end queues 0 1 2 3 end / end
# lspci | grep Ether
19:00.0 Ethernet controller: Mellanox Technologies MT27800 Family [ConnectX-5]
Fw version is 16.30.1004
# ethtool -i net2
driver: mlx5_core
version: 5.12.0-rc7+
firmware-version: 16.30.1004 (MT_0000000080)
expansion-rom-version:
bus-info: 0000:19:00.0
# ifconfig net2 172.168.153.50/24
On the remote server 172.168.153.73 interface net2 connect with the 172.168.153.50
# ifconfig net2 172.168.153.73/24
# ping 172.168.153.50
PING 172.168.153.50 (172.168.153.50) 56(84) bytes of data.
64 bytes from 172.168.153.50: icmp_seq=1 ttl=64 time=0.070 ms
64 bytes from 172.168.153.50: icmp_seq=2 ttl=64 time=0.026 ms
64 bytes from 172.168.153.50: icmp_seq=2 ttl=64 time=0.031 ms (DUP!)
64 bytes from 172.168.153.50: icmp_seq=3 ttl=64 time=0.022 ms
64 bytes from 172.168.153.50: icmp_seq=3 ttl=64 time=0.022 ms (DUP!)
64 bytes from 172.168.153.50: icmp_seq=4 ttl=64 time=0.020 ms
64 bytes from 172.168.153.50: icmp_seq=4 ttl=64 time=0.026 ms (DUP!)
64 bytes from 172.168.153.50: icmp_seq=5 ttl=64 time=0.023 ms
64 bytes from 172.168.153.50: icmp_seq=5 ttl=64 time=0.030 ms (DUP!)
64 bytes from 172.168.153.50: icmp_seq=6 ttl=64 time=0.020 ms
64 bytes from 172.168.153.50: icmp_seq=6 ttl=64 time=0.026 ms (DUP!)
On the server 172.168.153.73
# tcpdump -i net2 -nnn
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on net2, link-type EN10MB (Ethernet), capture size 262144 bytes
14:36:29.248516 IP 172.168.153.73 > 172.168.153.50: ICMP echo request, id 1987, seq 1, length 64
14:36:29.248558 IP 172.168.153.50 > 172.168.153.73: ICMP echo reply, id 1987, seq 1, length 64
14:36:30.254438 IP 172.168.153.73 > 172.168.153.50: ICMP echo request, id 1987, seq 2, length 64
14:36:30.254462 IP 172.168.153.50 > 172.168.153.73: ICMP echo reply, id 1987, seq 2, length 64
14:36:30.254468 IP 172.168.153.50 > 172.168.153.73: ICMP echo reply, id 1987, seq 2, length 64
14:36:31.278439 IP 172.168.153.73 > 172.168.153.50: ICMP echo request, id 1987, seq 3, length 64
14:36:31.278462 IP 172.168.153.50 > 172.168.153.73: ICMP echo reply, id 1987, seq 3, length 64
14:36:31.278467 IP 172.168.153.50 > 172.168.153.73: ICMP echo reply, id 1987, seq 3, length 64
14:36:32.302437 IP 172.168.153.73 > 172.168.153.50: ICMP echo request, id 1987, seq 4, length 64
14:36:32.302462 IP 172.168.153.50 > 172.168.153.73: ICMP echo reply, id 1987, seq 4, length 64
14:36:32.302467 IP 172.168.153.50 > 172.168.153.73: ICMP echo reply, id 1987, seq 4, length 64
14:36:33.326437 IP 172.168.153.73 > 172.168.153.50: ICMP echo request, id 1987, seq 5, length 64
14:36:33.326458 IP 172.168.153.50 > 172.168.153.73: ICMP echo reply, id 1987, seq 5, length 64
14:36:33.326463 IP 172.168.153.50 > 172.168.153.73: ICMP echo reply, id 1987, seq 5, length 64
Only send one request packet but receive two reply
on the server 172.168.153.50:
# tcpdump -i net2 -nnn
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on net2, link-type EN10MB (Ethernet), capture size 262144 bytes
14:38:04.991537 IP 172.168.153.73 > 172.168.153.50: ICMP echo request, id 1987, seq 1, length 64
14:38:04.991537 IP 172.168.153.73 > 172.168.153.50: ICMP echo request, id 1987, seq 1, length 64
14:38:04.991556 IP 172.168.153.50 > 172.168.153.73: ICMP echo reply, id 1987, seq 1, length 64
14:38:05.997464 IP 172.168.153.73 > 172.168.153.50: ICMP echo request, id 1987, seq 2, length 64
14:38:05.997464 IP 172.168.153.73 > 172.168.153.50: ICMP echo request, id 1987, seq 2, length 64
14:38:05.997471 IP 172.168.153.50 > 172.168.153.73: ICMP echo reply, id 1987, seq 2, length 64
14:38:05.997474 IP 172.168.153.50 > 172.168.153.73: ICMP echo reply, id 1987, seq 2, length 64
14:38:07.021475 IP 172.168.153.73 > 172.168.153.50: ICMP echo request, id 1987, seq 3, length 64
14:38:07.021475 IP 172.168.153.73 > 172.168.153.50: ICMP echo request, id 1987, seq 3, length 64
14:38:07.021482 IP 172.168.153.50 > 172.168.153.73: ICMP echo reply, id 1987, seq 3, length 64
14:38:07.021484 IP 172.168.153.50 > 172.168.153.73: ICMP echo reply, id 1987, seq 3, length 64
14:38:08.045482 IP 172.168.153.73 > 172.168.153.50: ICMP echo request, id 1987, seq 4, length 64
14:38:08.045482 IP 172.168.153.73 > 172.168.153.50: ICMP echo request, id 1987, seq 4, length 64
14:38:08.045492 IP 172.168.153.50 > 172.168.153.73: ICMP echo reply, id 1987, seq 4, length 64
14:38:08.045495 IP 172.168.153.50 > 172.168.153.73: ICMP echo reply, id 1987, seq 4, length 64
14:38:09.069491 IP 172.168.153.73 > 172.168.153.50: ICMP echo request, id 1987, seq 5, length 64
14:38:09.069491 IP 172.168.153.73 > 172.168.153.50: ICMP echo request, id 1987, seq 5, length 64
14:38:09.069498 IP 172.168.153.50 > 172.168.153.73: ICMP echo reply, id 1987, seq 5, length 64
14:38:09.069500 IP 172.168.153.50 > 172.168.153.73: ICMP echo reply, id 1987, seq 5, length 64
On the 172.168.153.50 It receives the two same request packet .
If we shutdown one testpmd program, It will goes well.
I think the hardware copy n（dpdk instance number） packets.
Then I test send with a vxlan packet.
On the server 172.168.153.73:
# scapy
# a=Ether(src='fa:ff:ff:ff:ff:ff',dst='1c:34:da:77:fb:d8')/IP(src='172.168.153.73', dst="172.168.153.50")/UDP(sport=46943,dport=4789)/VXLAN(flags=0x8,vni=1000)/Ether(src='52:54:00:00:00:00', dst='fa:ff:ff:ff:ff:ff')/IP(src='2.2.2.2', dst="1.1.1.1")/UDP(sport=22346, dport=5001)
# sendp（a, iface="net2"）
.
Sent 1 packets.
testpmd> show port stats 0
# tcpdump -i net2 -nnn
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on net2, link-type EN10MB (Ethernet), capture size 262144 bytes
14:42:23.532370 IP 172.168.153.73.46943 > 172.168.153.50.4789: VXLAN, flags [I] (0x08), vni 1000
IP 2.2.2.2.22346 > 1.1.1.1.5001: UDP, length 0
Only one vxlan pkt send in the server 172.168.153.73
On the dpdk server 172.168.153.50
At the beigin all the counter is 0
# testpmd> show port stats 0
######################## NIC statistics for port 0 ########################
RX-packets: 0 RX-missed: 0 RX-bytes: 0
RX-errors: 0
RX-nombuf: 0
TX-packets: 0 TX-errors: 0 TX-bytes: 0
Throughput (since last show)
Rx-pps: 0 Rx-bps: 0
Tx-pps: 0 Tx-bps: 0
############################################################################
After the remote server send one vxlan packet
The two dpdk testpmd shows receive two packets
testpmd> show port stats 0
######################## NIC statistics for port 0 ########################
RX-packets: 2 RX-missed: 0 RX-bytes: 168
RX-errors: 0
RX-nombuf: 0
TX-packets: 0 TX-errors: 0 TX-bytes: 0
Throughput (since last show)
Rx-pps: 0 Rx-bps: 200
Tx-pps: 0 Tx-bps: 0
############################################################################
I test with the upstream dpdk and rdma-core.
And also test with the rdma-core and dpdk in ofed-5.3.
both the same problem.
BR
wenxu
reply other threads:[~2021-05-11 6:50 UTC|newest]
Thread overview: [no followups] expand[flat|nested] mbox.gz Atom feed
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to='AKYA*AAqFESkpTyUsTY1xKrI.1.1620715797762.Hmail.wenxu@ucloud.cn' \
--to=wenxu@ucloud.cn \
--cc=asafp@nvidia.com \
--cc=dev@dpdk.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).