DPDK patches and discussions
 help / color / mirror / Atom feed
From: wangyunjian <wangyunjian@huawei.com>
To: Dmitry Kozlyuk <dkozlyuk@nvidia.com>,
	"dev@dpdk.org" <dev@dpdk.org>, "users@dpdk.org" <users@dpdk.org>,
	Matan Azrad <matan@nvidia.com>,
	"Slava Ovsiienko" <viacheslavo@nvidia.com>
Cc: Huangshaozhang <huangshaozhang@huawei.com>,
	dingxiaoxiong <dingxiaoxiong@huawei.com>
Subject: RE: [dpdk-dev] [dpdk-users] A question about Mellanox ConnectX-5 and ConnectX-4 Lx nic can't send packets?
Date: Tue, 11 Jan 2022 08:21:06 +0000	[thread overview]
Message-ID: <2507d6c0239547c8b3f30870578ce392@huawei.com> (raw)
In-Reply-To: <BN8PR12MB289903D60FD1FDA6442DDE7FB9519@BN8PR12MB2899.namprd12.prod.outlook.com>

> -----Original Message-----
> From: Dmitry Kozlyuk [mailto:dkozlyuk@nvidia.com]
> Sent: Tuesday, January 11, 2022 3:37 PM
> To: wangyunjian <wangyunjian@huawei.com>; dev@dpdk.org; users@dpdk.org;
> Matan Azrad <matan@nvidia.com>; Slava Ovsiienko <viacheslavo@nvidia.com>
> Cc: Huangshaozhang <huangshaozhang@huawei.com>; dingxiaoxiong
> <dingxiaoxiong@huawei.com>
> Subject: RE: [dpdk-dev] [dpdk-users] A question about Mellanox ConnectX-5 and
> ConnectX-4 Lx nic can't send packets?
> 
> Hello,
> 
> Thanks for attaching all the details.
> Can you please reproduce it with --log-level=pmd.common.mlx5:debug and
> send the logs?
> 
> > For example, if the environment is configured with 10GB hugepages but
> > each hugepage is physically discontinuous, this problem can be
> > reproduced.

# ./x86_64-native-linuxapp-gcc/app/dpdk-testpmd -c 0xFC0 --iova-mode pa --legacy-mem  -a af:00.0 -a af:00.1 --log-level=pmd.common.mlx5:debug -m 0,8192 -- -a -i --forward-mode=fwd --rxq=2 --txq=2   --total-num-mbufs=1000000
EAL: Detected CPU lcores: 80
EAL: Detected NUMA nodes: 2
EAL: Static memory layout is selected, amount of reserved memory can be adjusted with -m or --socket-mem
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:af:00.0 (socket 1)
mlx5_common: probe device "af:00.0".
mlx5_common: mr_ext_memseg_en is 1.
mlx5_common: mr_mempool_reg_en is 1.
mlx5_common: sys_mem_en is 0.
mlx5_common: Tx doorbell mapping parameter is -1.
mlx5_common: RTE_MEM is selected.
mlx5_common: Checking device "mlx5_3"..
mlx5_common: Checking device "mlx5_2"..
mlx5_common: Dev information matches for device "mlx5_2".
mlx5_common: DevX is supported.
mlx5_common: initialized B-tree 0x8fffe9dc1 with table 0x197fffd800
mlx5_common: Netlink socket send buffer: 212992
mlx5_common: Netlink socket recv buffer: 212992
mlx5_common: Netlink socket send buffer: 212992
mlx5_common: Netlink socket recv buffer: 212992
mlx5_common: Netlink socket send buffer: 212992
mlx5_common: Netlink socket recv buffer: 212992
mlx5_common: Netlink socket send buffer: 212992
mlx5_common: Netlink socket recv buffer: 212992
mlx5_common: Hash list mlx5_2_flow_table with size 0x400 was created.
mlx5_common: mlx5 list mlx5_2_port_id_action_list was created.
mlx5_common: mlx5 list mlx5_2_push_vlan_action_list was created.
mlx5_common: mlx5 list mlx5_2_sample_action_list was created.
mlx5_common: mlx5 list mlx5_2_dest_array_list was created.
mlx5_common: mlx5 list mlx5_2_flex_parsers_list was created.
mlx5_common: Can't allocate q counter set by DevX - error 22.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50d80 new: 1.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x8ffc504c0 new: 1.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x8ffc504c0 removed.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50d80 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50d80 new: 1.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x8ffc504c0 new: 1.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x8ffc504c0 removed.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50d80 removed.
mlx5_common: mlx5 list hrxq was created.
mlx5_common: mlx5 list NIC_ingress_65534_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50d80 new: 1.
mlx5_common: Hash list mlx5_2_hdr_modify with size 0x8000 was created.
mlx5_common: MLX5 list mlx5_2_hdr_modify c0 entry 0x8fd9d4548 new: 1.
mlx5_common: mlx5 list NIC_ingress_65533_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50e40 new: 1.
mlx5_common: mlx5 list NIC_ingress_65533_0_matcher_lis entry 0x8fd9d2e00 new: 1.
mlx5_common: mlx5 list NIC_ingress_65533_0_matcher_lis entry 0x8fd9d2e00 removed.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50e40 removed.
mlx5_common: mlx5 list mlx5_2_hdr_modify entry 0x8fd9d4548 removed.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50d80 removed.
mlx5_common: mlx5 list NIC_ingress_65534_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50d80 new: 1.
mlx5_common: MLX5 list mlx5_2_hdr_modify c0 entry 0x8fd9d4548 new: 1.
mlx5_common: mlx5 list NIC_ingress_65533_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50e40 new: 1.
mlx5_common: mlx5 list NIC_ingress_65533_0_matcher_lis entry 0x8fd9d2d40 new: 1.
mlx5_common: mlx5 list NIC_ingress_65533_0_matcher_lis entry 0x8fd9d2d40 removed.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50e40 removed.
mlx5_common: mlx5 list mlx5_2_hdr_modify entry 0x8fd9d4548 removed.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50d80 removed.
mlx5_common: mlx5 list NIC_ingress_65534_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50d80 new: 1.
mlx5_common: MLX5 list mlx5_2_hdr_modify c0 entry 0x8fd9d4548 new: 1.
mlx5_common: mlx5 list NIC_ingress_65533_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50e40 new: 1.
mlx5_common: mlx5 list NIC_ingress_65533_0_matcher_lis entry 0x8fd9d2c80 new: 1.
mlx5_common: mlx5 list NIC_ingress_65533_0_matcher_lis entry 0x8fd9d2c80 removed.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50e40 removed.
mlx5_common: mlx5 list mlx5_2_hdr_modify entry 0x8fd9d4548 removed.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50d80 removed.
mlx5_common: mlx5 list NIC_ingress_65534_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50d80 new: 1.
mlx5_common: MLX5 list mlx5_2_hdr_modify c0 entry 0x8fd9d4548 new: 1.
mlx5_common: mlx5 list NIC_ingress_65533_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50e40 new: 1.
mlx5_common: mlx5 list NIC_ingress_65533_0_matcher_lis entry 0x8fd9d2bc0 new: 1.
mlx5_common: mlx5 list NIC_ingress_65533_0_matcher_lis entry 0x8fd9d2bc0 removed.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50e40 removed.
mlx5_common: mlx5 list mlx5_2_hdr_modify entry 0x8fd9d4548 removed.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50d80 removed.
mlx5_common: mlx5 list NIC_ingress_65534_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50d80 new: 1.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50d80 removed.
mlx5_common: mlx5 list NIC_ingress_65534_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50d80 new: 1.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50d80 removed.
mlx5_common: mlx5 list NIC_egress_0_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50d80 new: 1.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50d80 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50d80 new: 1.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50d80 removed.
EAL: Probe PCI driver: mlx5_pci (15b3:1017) device: 0000:af:00.1 (socket 1)
mlx5_common: probe device "af:00.1".
mlx5_common: mr_ext_memseg_en is 1.
mlx5_common: mr_mempool_reg_en is 1.
mlx5_common: sys_mem_en is 0.
mlx5_common: Tx doorbell mapping parameter is -1.
mlx5_common: Checking device "mlx5_3"..
mlx5_common: Dev information matches for device "mlx5_3".
mlx5_common: DevX is supported.
mlx5_common: initialized B-tree 0x8ffc2a641 with table 0x197ff6c780
mlx5_common: Netlink socket send buffer: 212992
mlx5_common: Netlink socket recv buffer: 212992
mlx5_common: Netlink socket send buffer: 212992
mlx5_common: Netlink socket recv buffer: 212992
mlx5_common: Netlink socket send buffer: 212992
mlx5_common: Netlink socket recv buffer: 212992
mlx5_common: Netlink socket send buffer: 212992
mlx5_common: Netlink socket recv buffer: 212992
mlx5_common: Hash list mlx5_3_flow_table with size 0x400 was created.
mlx5_common: mlx5 list mlx5_3_port_id_action_list was created.
mlx5_common: mlx5 list mlx5_3_push_vlan_action_list was created.
mlx5_common: mlx5 list mlx5_3_sample_action_list was created.
mlx5_common: mlx5 list mlx5_3_dest_array_list was created.
mlx5_common: mlx5 list mlx5_3_flex_parsers_list was created.
mlx5_common: Can't allocate q counter set by DevX - error 22.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670d80 new: 1.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x8fd670580 new: 1.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x8fd670580 removed.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670d80 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670d80 new: 1.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x8fd670580 new: 1.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x8fd670580 removed.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670d80 removed.
mlx5_common: mlx5 list hrxq was created.
mlx5_common: mlx5 list NIC_ingress_65534_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670d80 new: 1.
mlx5_common: Hash list mlx5_3_hdr_modify with size 0x8000 was created.
mlx5_common: MLX5 list mlx5_3_hdr_modify c0 entry 0x8fb3f46c8 new: 1.
mlx5_common: mlx5 list NIC_ingress_65533_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670e40 new: 1.
mlx5_common: mlx5 list NIC_ingress_65533_0_matcher_lis entry 0x8fb3f3040 new: 1.
mlx5_common: mlx5 list NIC_ingress_65533_0_matcher_lis entry 0x8fb3f3040 removed.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670e40 removed.
mlx5_common: mlx5 list mlx5_3_hdr_modify entry 0x8fb3f46c8 removed.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670d80 removed.
mlx5_common: mlx5 list NIC_ingress_65534_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670d80 new: 1.
mlx5_common: MLX5 list mlx5_3_hdr_modify c0 entry 0x8fb3f46c8 new: 1.
mlx5_common: mlx5 list NIC_ingress_65533_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670e40 new: 1.
mlx5_common: mlx5 list NIC_ingress_65533_0_matcher_lis entry 0x8fb3f2f80 new: 1.
mlx5_common: mlx5 list NIC_ingress_65533_0_matcher_lis entry 0x8fb3f2f80 removed.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670e40 removed.
mlx5_common: mlx5 list mlx5_3_hdr_modify entry 0x8fb3f46c8 removed.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670d80 removed.
mlx5_common: mlx5 list NIC_ingress_65534_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670d80 new: 1.
mlx5_common: MLX5 list mlx5_3_hdr_modify c0 entry 0x8fb3f46c8 new: 1.
mlx5_common: mlx5 list NIC_ingress_65533_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670e40 new: 1.
mlx5_common: mlx5 list NIC_ingress_65533_0_matcher_lis entry 0x8fb3f2ec0 new: 1.
mlx5_common: mlx5 list NIC_ingress_65533_0_matcher_lis entry 0x8fb3f2ec0 removed.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670e40 removed.
mlx5_common: mlx5 list mlx5_3_hdr_modify entry 0x8fb3f46c8 removed.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670d80 removed.
mlx5_common: mlx5 list NIC_ingress_65534_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670d80 new: 1.
mlx5_common: MLX5 list mlx5_3_hdr_modify c0 entry 0x8fb3f46c8 new: 1.
mlx5_common: mlx5 list NIC_ingress_65533_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670e40 new: 1.
mlx5_common: mlx5 list NIC_ingress_65533_0_matcher_lis entry 0x8fb3f2e00 new: 1.
mlx5_common: mlx5 list NIC_ingress_65533_0_matcher_lis entry 0x8fb3f2e00 removed.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670e40 removed.
mlx5_common: mlx5 list mlx5_3_hdr_modify entry 0x8fb3f46c8 removed.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670d80 removed.
mlx5_common: mlx5 list NIC_ingress_65534_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670d80 new: 1.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670d80 removed.
mlx5_common: mlx5 list NIC_ingress_65534_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670d80 new: 1.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670d80 removed.
mlx5_common: mlx5 list NIC_egress_0_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670d80 new: 1.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670d80 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670d80 new: 1.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670d80 removed.
TELEMETRY: No legacy callbacks, legacy socket not created
Auto-start selected
Interactive-mode selected
Invalid fwd packet forwarding mode
testpmd: create a new mbuf pool <mb_pool_0>: n=1000000, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc
testpmd: create a new mbuf pool <mb_pool_1>: n=1000000, size=2176, socket=1
testpmd: preferred mempool ops selected: ring_mp_mc
Configuring Port 0 (socket 1)
mlx5_common: initialized B-tree 0x197fede628 with table 0x17dcb6b3c0
mlx5_common: initialized B-tree 0x197fee86e8 with table 0x17dcb69f40
mlx5_common: initialized B-tree 0x17dcb69074 with table 0x17dcb67b00
mlx5_common: initialized B-tree 0x17dcb66c34 with table 0x17dcb656c0
mlx5_common: mr_ctrl(0x17dcb68fc4): flushed, cur_gen=0
mlx5_common: Collecting chunks of regular mempool mb_pool_1
mlx5_common: Created a new MR 0x5920 in PD 0x36a25a0 for address range [0x17dcb6c000, 0x1800000000] (592003072 bytes) for mempool mb_pool_1
mlx5_common: Created a new MR 0x5b21 in PD 0x36a25a0 for address range [0x185cb6c000, 0x1880000000] (592003072 bytes) for mempool mb_pool_1
mlx5_common: Created a new MR 0x6822 in PD 0x36a25a0 for address range [0x18dcb6c000, 0x1900000000] (592003072 bytes) for mempool mb_pool_1
mlx5_common: Created a new MR 0x7023 in PD 0x36a25a0 for address range [0x195c0c8000, 0x197f55c000] (592003072 bytes) for mempool mb_pool_1
mlx5_common: inserted B-tree(0x17dcb69074)[1], [0x17dcb6c000, 0x1800000000) lkey=0x20590000
mlx5_common: inserted B-tree(0x17dcb69074)[2], [0x185cb6c000, 0x1880000000) lkey=0x215b0000
mlx5_common: inserted B-tree(0x17dcb69074)[3], [0x18dcb6c000, 0x1900000000) lkey=0x22680000
mlx5_common: inserted B-tree(0x17dcb69074)[4], [0x195c0c8000, 0x197f55c000) lkey=0x23700000
mlx5_common: mr_ctrl(0x17dcb66b84): flushed, cur_gen=0
mlx5_common: Mempool mb_pool_1 is already registered for PD 0x36a25a0
mlx5_common: inserted B-tree(0x17dcb66c34)[1], [0x17dcb6c000, 0x1800000000) lkey=0x20590000
mlx5_common: inserted B-tree(0x17dcb66c34)[2], [0x185cb6c000, 0x1880000000) lkey=0x215b0000
mlx5_common: inserted B-tree(0x17dcb66c34)[3], [0x18dcb6c000, 0x1900000000) lkey=0x22680000
mlx5_common: inserted B-tree(0x17dcb66c34)[4], [0x195c0c8000, 0x197f55c000) lkey=0x23700000
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50d80 new: 1.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caac840 new: 1.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 2.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caac340 new: 1.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 3.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caac0c0 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa9d00 new: 1.
mlx5_common: mlx5 list hrxq entry 0x75caa9df0 new: 1.
mlx5_common: mlx5 list hrxq entry 0x75caa9ee0 new: 1.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 4.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa9640 new: 1.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 5.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa9140 new: 1.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 6.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa8ec0 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa9d78 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa9e68 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa9f58 ref: 2.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 7.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caac5c0 ref: 2.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 8.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caac0c0 ref: 3.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 9.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caac0c0 ref: 4.
mlx5_common: mlx5 list hrxq entry 0x75caa9d78 ref: 3.
mlx5_common: mlx5 list hrxq entry 0x75caa9e68 ref: 3.
mlx5_common: mlx5 list hrxq entry 0x75caa9f58 ref: 3.
mlx5_common: Collecting chunks of regular mempool mb_pool_0
mlx5_common: Created a new MR 0x7b24 in PD 0x36a25a0 for address range [0x75cb6c000, 0x780000000] (592003072 bytes) for mempool mb_pool_0
mlx5_common: Created a new MR 0x1d425 in PD 0x36a25a0 for address range [0x7dcb6c000, 0x800000000] (592003072 bytes) for mempool mb_pool_0
mlx5_common: Created a new MR 0x1da26 in PD 0x36a25a0 for address range [0x85cb6c000, 0x880000000] (592003072 bytes) for mempool mb_pool_0
mlx5_common: Created a new MR 0x1e227 in PD 0x36a25a0 for address range [0x8d6cca000, 0x8fa15e000] (592003072 bytes) for mempool mb_pool_0
mlx5_common: Mempool mb_pool_1 is already registered for PD 0x36a25a0
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa9140 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa9640 removed.
mlx5_common: mlx5 list hrxq entry 0x75caa9d00 removed.
mlx5_common: mlx5 list hrxq entry 0x75caa9df0 removed.
mlx5_common: mlx5 list hrxq entry 0x75caa9ee0 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caac340 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caac840 removed.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50d80 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50d80 new: 1.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caabc00 new: 1.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 2.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caac680 new: 1.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 3.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caac400 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa9d00 new: 1.
mlx5_common: mlx5 list hrxq entry 0x75caa9df0 new: 1.
mlx5_common: mlx5 list hrxq entry 0x75caa9ee0 new: 1.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 4.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa9480 new: 1.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 5.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa8f80 new: 1.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 6.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa8a40 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa9d78 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa9e68 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa9f58 ref: 2.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 7.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caac900 ref: 2.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 8.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caac400 ref: 3.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 9.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caac400 ref: 4.
mlx5_common: mlx5 list hrxq entry 0x75caa9d78 ref: 3.
mlx5_common: mlx5 list hrxq entry 0x75caa9e68 ref: 3.
mlx5_common: mlx5 list hrxq entry 0x75caa9f58 ref: 3.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa8f80 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa9480 removed.
mlx5_common: mlx5 list hrxq entry 0x75caa9d00 removed.
mlx5_common: mlx5 list hrxq entry 0x75caa9df0 removed.
mlx5_common: mlx5 list hrxq entry 0x75caa9ee0 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caac680 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caabc00 removed.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50d80 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50d80 new: 1.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caabc00 new: 1.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 2.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caac680 new: 1.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 3.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caac400 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa9d00 new: 1.
mlx5_common: mlx5 list hrxq entry 0x75caa9df0 new: 1.
mlx5_common: mlx5 list hrxq entry 0x75caa9ee0 new: 1.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 4.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa9480 new: 1.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 5.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa8f80 new: 1.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 6.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa8a40 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa9d78 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa9e68 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa9f58 ref: 2.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 7.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caac900 ref: 2.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 8.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caac400 ref: 3.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 9.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caac400 ref: 4.
mlx5_common: mlx5 list hrxq entry 0x75caa9d78 ref: 3.
mlx5_common: mlx5 list hrxq entry 0x75caa9e68 ref: 3.
mlx5_common: mlx5 list hrxq entry 0x75caa9f58 ref: 3.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa8f80 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa9480 removed.
mlx5_common: mlx5 list hrxq entry 0x75caa9d00 removed.
mlx5_common: mlx5 list hrxq entry 0x75caa9df0 removed.
mlx5_common: mlx5 list hrxq entry 0x75caa9ee0 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caac680 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caabc00 removed.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50d80 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50d80 new: 1.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caabc00 new: 1.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 2.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caac680 new: 1.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 3.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caac400 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa9d00 new: 1.
mlx5_common: mlx5 list hrxq entry 0x75caa9df0 new: 1.
mlx5_common: mlx5 list hrxq entry 0x75caa9ee0 new: 1.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 4.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa9480 new: 1.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 5.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa8f80 new: 1.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 6.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa8a40 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa9d78 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa9e68 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa9f58 ref: 2.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 7.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caac900 ref: 2.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 8.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caac400 ref: 3.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 9.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caac400 ref: 4.
mlx5_common: mlx5 list hrxq entry 0x75caa9d78 ref: 3.
mlx5_common: mlx5 list hrxq entry 0x75caa9e68 ref: 3.
mlx5_common: mlx5 list hrxq entry 0x75caa9f58 ref: 3.
Port 0: 28:DE:E5:AB:95:33
Configuring Port 1 (socket 1)
mlx5_common: initialized B-tree 0x17dcb4c6e8 with table 0x17dcb49b80
mlx5_common: initialized B-tree 0x17dcb516e8 with table 0x17dcb48700
mlx5_common: initialized B-tree 0x17dcb47834 with table 0x17dcb462c0
mlx5_common: initialized B-tree 0x17dcb453f4 with table 0x17dcb43e80
mlx5_common: mr_ctrl(0x17dcb47784): flushed, cur_gen=0
mlx5_common: Collecting chunks of regular mempool mb_pool_1
mlx5_common: Created a new MR 0x8ee23 in PD 0x4864ab0 for address range [0x17dcb6c000, 0x1800000000] (592003072 bytes) for mempool mb_pool_1
mlx5_common: Created a new MR 0x8ff24 in PD 0x4864ab0 for address range [0x185cb6c000, 0x1880000000] (592003072 bytes) for mempool mb_pool_1
mlx5_common: Created a new MR 0x90a25 in PD 0x4864ab0 for address range [0x18dcb6c000, 0x1900000000] (592003072 bytes) for mempool mb_pool_1
mlx5_common: Created a new MR 0x91a26 in PD 0x4864ab0 for address range [0x195c0c8000, 0x197f55c000] (592003072 bytes) for mempool mb_pool_1
mlx5_common: inserted B-tree(0x17dcb47834)[1], [0x17dcb6c000, 0x1800000000) lkey=0x23ee0800
mlx5_common: inserted B-tree(0x17dcb47834)[2], [0x185cb6c000, 0x1880000000) lkey=0x24ff0800
mlx5_common: inserted B-tree(0x17dcb47834)[3], [0x18dcb6c000, 0x1900000000) lkey=0x250a0900
mlx5_common: inserted B-tree(0x17dcb47834)[4], [0x195c0c8000, 0x197f55c000) lkey=0x261a0900
mlx5_common: mr_ctrl(0x17dcb45344): flushed, cur_gen=0
mlx5_common: Mempool mb_pool_1 is already registered for PD 0x4864ab0
mlx5_common: inserted B-tree(0x17dcb453f4)[1], [0x17dcb6c000, 0x1800000000) lkey=0x23ee0800
mlx5_common: inserted B-tree(0x17dcb453f4)[2], [0x185cb6c000, 0x1880000000) lkey=0x24ff0800
mlx5_common: inserted B-tree(0x17dcb453f4)[3], [0x18dcb6c000, 0x1900000000) lkey=0x250a0900
mlx5_common: inserted B-tree(0x17dcb453f4)[4], [0x195c0c8000, 0x197f55c000) lkey=0x261a0900
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670d80 new: 1.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa2e40 new: 1.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 2.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa2940 new: 1.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 3.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa26c0 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa0300 new: 1.
mlx5_common: mlx5 list hrxq entry 0x75caa03f0 new: 1.
mlx5_common: mlx5 list hrxq entry 0x75caa04e0 new: 1.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 4.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9fc40 new: 1.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 5.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9f740 new: 1.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 6.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9f4c0 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa0378 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa0468 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa0558 ref: 2.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 7.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa2bc0 ref: 2.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 8.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa26c0 ref: 3.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 9.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa26c0 ref: 4.
mlx5_common: mlx5 list hrxq entry 0x75caa0378 ref: 3.
mlx5_common: mlx5 list hrxq entry 0x75caa0468 ref: 3.
mlx5_common: mlx5 list hrxq entry 0x75caa0558 ref: 3.
mlx5_common: Collecting chunks of regular mempool mb_pool_0
mlx5_common: Created a new MR 0x92827 in PD 0x4864ab0 for address range [0x75cb6c000, 0x780000000] (592003072 bytes) for mempool mb_pool_0
mlx5_common: Created a new MR 0x93528 in PD 0x4864ab0 for address range [0x7dcb6c000, 0x800000000] (592003072 bytes) for mempool mb_pool_0
mlx5_common: Created a new MR 0x94529 in PD 0x4864ab0 for address range [0x85cb6c000, 0x880000000] (592003072 bytes) for mempool mb_pool_0
mlx5_common: Created a new MR 0x9562a in PD 0x4864ab0 for address range [0x8d6cca000, 0x8fa15e000] (592003072 bytes) for mempool mb_pool_0
mlx5_common: Mempool mb_pool_1 is already registered for PD 0x4864ab0
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9f740 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9fc40 removed.
mlx5_common: mlx5 list hrxq entry 0x75caa0300 removed.
mlx5_common: mlx5 list hrxq entry 0x75caa03f0 removed.
mlx5_common: mlx5 list hrxq entry 0x75caa04e0 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa2940 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa2e40 removed.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670d80 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670d80 new: 1.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa2200 new: 1.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 2.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa2c80 new: 1.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 3.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa2a00 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa0300 new: 1.
mlx5_common: mlx5 list hrxq entry 0x75caa03f0 new: 1.
mlx5_common: mlx5 list hrxq entry 0x75caa04e0 new: 1.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 4.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9fa80 new: 1.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 5.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9f580 new: 1.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 6.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9f040 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa0378 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa0468 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa0558 ref: 2.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 7.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa2f00 ref: 2.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 8.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa2a00 ref: 3.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 9.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa2a00 ref: 4.
mlx5_common: mlx5 list hrxq entry 0x75caa0378 ref: 3.
mlx5_common: mlx5 list hrxq entry 0x75caa0468 ref: 3.
mlx5_common: mlx5 list hrxq entry 0x75caa0558 ref: 3.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9f580 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9fa80 removed.
mlx5_common: mlx5 list hrxq entry 0x75caa0300 removed.
mlx5_common: mlx5 list hrxq entry 0x75caa03f0 removed.
mlx5_common: mlx5 list hrxq entry 0x75caa04e0 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa2c80 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa2200 removed.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670d80 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670d80 new: 1.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa2200 new: 1.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 2.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa2c80 new: 1.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 3.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa2a00 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa0300 new: 1.
mlx5_common: mlx5 list hrxq entry 0x75caa03f0 new: 1.
mlx5_common: mlx5 list hrxq entry 0x75caa04e0 new: 1.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 4.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9fa80 new: 1.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 5.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9f580 new: 1.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 6.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9f040 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa0378 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa0468 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa0558 ref: 2.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 7.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa2f00 ref: 2.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 8.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa2a00 ref: 3.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 9.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa2a00 ref: 4.
mlx5_common: mlx5 list hrxq entry 0x75caa0378 ref: 3.
mlx5_common: mlx5 list hrxq entry 0x75caa0468 ref: 3.
mlx5_common: mlx5 list hrxq entry 0x75caa0558 ref: 3.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9f580 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9fa80 removed.
mlx5_common: mlx5 list hrxq entry 0x75caa0300 removed.
mlx5_common: mlx5 list hrxq entry 0x75caa03f0 removed.
mlx5_common: mlx5 list hrxq entry 0x75caa04e0 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa2c80 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa2200 removed.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670d80 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670d80 new: 1.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa2200 new: 1.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 2.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa2c80 new: 1.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 3.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa2a00 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa0300 new: 1.
mlx5_common: mlx5 list hrxq entry 0x75caa03f0 new: 1.
mlx5_common: mlx5 list hrxq entry 0x75caa04e0 new: 1.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 4.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9fa80 new: 1.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 5.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9f580 new: 1.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 6.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9f040 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa0378 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa0468 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa0558 ref: 2.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 7.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa2f00 ref: 2.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 8.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa2a00 ref: 3.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 9.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa2a00 ref: 4.
mlx5_common: mlx5 list hrxq entry 0x75caa0378 ref: 3.
mlx5_common: mlx5 list hrxq entry 0x75caa0468 ref: 3.
mlx5_common: mlx5 list hrxq entry 0x75caa0558 ref: 3.
Port 1: 28:DE:E5:AB:95:34
Checking link statuses...
Done
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa8f80 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa9480 removed.
mlx5_common: mlx5 list hrxq entry 0x75caa9d00 removed.
mlx5_common: mlx5 list hrxq entry 0x75caa9df0 removed.
mlx5_common: mlx5 list hrxq entry 0x75caa9ee0 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caac680 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caabc00 removed.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50d80 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50d80 new: 1.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caabc00 new: 1.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 2.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caac900 new: 1.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 3.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caac680 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa9d00 new: 1.
mlx5_common: mlx5 list hrxq entry 0x75caa9df0 new: 1.
mlx5_common: mlx5 list hrxq entry 0x75caa9ee0 new: 1.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 4.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caac0c0 new: 1.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 5.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa92c0 new: 1.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 6.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa9040 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa9d78 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa9e68 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa9f58 ref: 2.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 7.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9edc0 new: 1.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 8.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9e8c0 new: 1.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 9.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9e640 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa9d78 ref: 3.
mlx5_common: mlx5 list hrxq entry 0x75caa9e68 ref: 3.
mlx5_common: mlx5 list hrxq entry 0x75caa9f58 ref: 3.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 10.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa9540 ref: 2.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 11.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa9040 ref: 3.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50de0 ref: 12.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa9040 ref: 4.
mlx5_common: mlx5 list hrxq entry 0x75caa9d78 ref: 4.
mlx5_common: mlx5 list hrxq entry 0x75caa9e68 ref: 4.
mlx5_common: mlx5 list hrxq entry 0x75caa9f58 ref: 4.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9f580 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9fa80 removed.
mlx5_common: mlx5 list hrxq entry 0x75caa0300 removed.
mlx5_common: mlx5 list hrxq entry 0x75caa03f0 removed.
mlx5_common: mlx5 list hrxq entry 0x75caa04e0 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa2c80 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa2200 removed.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670d80 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list was created.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670d80 new: 1.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa2200 new: 1.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 2.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa2f00 new: 1.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 3.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa2c80 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa0300 new: 1.
mlx5_common: mlx5 list hrxq entry 0x75caa03f0 new: 1.
mlx5_common: mlx5 list hrxq entry 0x75caa04e0 new: 1.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 4.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa26c0 new: 1.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 5.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9f8c0 new: 1.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 6.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9f640 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa0378 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa0468 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa0558 ref: 2.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 7.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9e3c0 new: 1.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 8.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9dec0 new: 1.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 9.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9dc40 ref: 2.
mlx5_common: mlx5 list hrxq entry 0x75caa0378 ref: 3.
mlx5_common: mlx5 list hrxq entry 0x75caa0468 ref: 3.
mlx5_common: mlx5 list hrxq entry 0x75caa0558 ref: 3.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 10.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9fb40 ref: 2.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 11.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9f640 ref: 3.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670de0 ref: 12.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9f640 ref: 4.
mlx5_common: mlx5 list hrxq entry 0x75caa0378 ref: 4.
mlx5_common: mlx5 list hrxq entry 0x75caa0468 ref: 4.
mlx5_common: mlx5 list hrxq entry 0x75caa0558 ref: 4.
Start automatic packet forwarding
io packet forwarding - ports=2 - cores=1 - streams=4 - NUMA support enabled, MP allocation mode: native
Logical Core 7 (socket 0) forwards packets on 4 streams:
  RX P=0/Q=0 (socket 1) -> TX P=1/Q=0 (socket 1) peer=02:00:00:00:00:01
  RX P=1/Q=0 (socket 1) -> TX P=0/Q=0 (socket 1) peer=02:00:00:00:00:00
  RX P=0/Q=1 (socket 1) -> TX P=1/Q=1 (socket 1) peer=02:00:00:00:00:01
  RX P=1/Q=1 (socket 1) -> TX P=0/Q=1 (socket 1) peer=02:00:00:00:00:00

  io packet forwarding packets/burst=32
  nb forwarding cores=1 - nb forwarding ports=2
  port 0: RX queue number: 2 Tx queue number: 2
    Rx offloads=0x0 Tx offloads=0x10000
    RX queue: 0
      RX desc=256 - RX free threshold=64
      RX threshold registers: pthresh=0 hthresh=0  wthresh=0
      RX Offloads=0x0
    TX queue: 0
      TX desc=256 - TX free threshold=0
      TX threshold registers: pthresh=0 hthresh=0  wthresh=0
      TX offloads=0x10000 - TX RS bit threshold=0
  port 1: RX queue number: 2 Tx queue number: 2
    Rx offloads=0x0 Tx offloads=0x10000
    RX queue: 0
      RX desc=256 - RX free threshold=64
      RX threshold registers: pthresh=0 hthresh=0  wthresh=0
      RX Offloads=0x0
    TX queue: 0
      TX desc=256 - TX free threshold=0
      TX threshold registers: pthresh=0 hthresh=0  wthresh=0
      TX offloads=0x10000 - TX RS bit threshold=0
testpmd> mlx5_common: inserted B-tree(0x17dcb4c6e8)[1], [0x17dcb6c000, 0x1800000000) lkey=0x23ee0800
mlx5_common: inserted B-tree(0x197fede628)[1], [0x17dcb6c000, 0x1800000000) lkey=0x20590000
mlx5_common: abort insertion to B-tree(0x17dcb4c6e8): already exist at idx=1 [0x17dcb6c000, 0x1800000000) lkey=0x23ee0800
mlx5_common: abort insertion to B-tree(0x197fede628): already exist at idx=1 [0x17dcb6c000, 0x1800000000) lkey=0x20590000
mlx5_common: abort insertion to B-tree(0x17dcb4c6e8): already exist at idx=1 [0x17dcb6c000, 0x1800000000) lkey=0x23ee0800
mlx5_common: abort insertion to B-tree(0x197fede628): already exist at idx=1 [0x17dcb6c000, 0x1800000000) lkey=0x20590000
mlx5_common: abort insertion to B-tree(0x17dcb4c6e8): already exist at idx=1 [0x17dcb6c000, 0x1800000000) lkey=0x23ee0800
mlx5_common: abort insertion to B-tree(0x197fede628): already exist at idx=1 [0x17dcb6c000, 0x1800000000) lkey=0x20590000
mlx5_common: abort insertion to B-tree(0x17dcb4c6e8): already exist at idx=1 [0x17dcb6c000, 0x1800000000) lkey=0x23ee0800
mlx5_common: abort insertion to B-tree(0x197fede628): already exist at idx=1 [0x17dcb6c000, 0x1800000000) lkey=0x20590000
mlx5_common: abort insertion to B-tree(0x17dcb4c6e8): already exist at idx=1 [0x17dcb6c000, 0x1800000000) lkey=0x23ee0800
mlx5_common: abort insertion to B-tree(0x197fede628): already exist at idx=1 [0x17dcb6c000, 0x1800000000) lkey=0x20590000
mlx5_common: abort insertion to B-tree(0x17dcb4c6e8): already exist at idx=1 [0x17dcb6c000, 0x1800000000) lkey=0x23ee0800
mlx5_common: abort insertion to B-tree(0x197fede628): already exist at idx=1 [0x17dcb6c000, 0x1800000000) lkey=0x20590000
mlx5_common: abort insertion to B-tree(0x17dcb4c6e8): already exist at idx=1 [0x17dcb6c000, 0x1800000000) lkey=0x23ee0800
mlx5_common: abort insertion to B-tree(0x197fede628): already exist at idx=1 [0x17dcb6c000, 0x1800000000) lkey=0x20590000

testpmd> mlx5_common: abort insertion to B-tree(0x17dcb4c6e8): already exist at idx=1 [0x17dcb6c000, 0x1800000000) lkey=0x23ee0800
mlx5_common: abort insertion to B-tree(0x197fede628): already exist at idx=1 [0x17dcb6c000, 0x1800000000) lkey=0x20590000
mlx5_common: abort insertion to B-tree(0x17dcb4c6e8): already exist at idx=1 [0x17dcb6c000, 0x1800000000) lkey=0x23ee0800
mlx5_common: abort insertion to B-tree(0x197fede628): already exist at idx=1 [0x17dcb6c000, 0x1800000000) lkey=0x20590000
qumlx5_common: abort insertion to B-tree(0x17dcb4c6e8): already exist at idx=1 [0x17dcb6c000, 0x1800000000) lkey=0x23ee0800
mlx5_common: abort insertion to B-tree(0x197fede628): already exist at idx=1 [0x17dcb6c000, 0x1800000000) lkey=0x20590000
it
Telling cores to stop...
Waiting for lcores to finish...

  ------- Forward Stats for RX Port= 0/Queue= 0 -> TX Port= 1/Queue= 0 -------
  RX-packets: 11             TX-packets: 11             TX-dropped: 0             

  ------- Forward Stats for RX Port= 1/Queue= 0 -> TX Port= 0/Queue= 0 -------
  RX-packets: 11             TX-packets: 11             TX-dropped: 0             

  ---------------------- Forward statistics for port 0  ----------------------
  RX-packets: 11             RX-dropped: 0             RX-total: 11
  TX-packets: 11             TX-dropped: 0             TX-total: 11
  ----------------------------------------------------------------------------

  ---------------------- Forward statistics for port 1  ----------------------
  RX-packets: 11             RX-dropped: 0             RX-total: 11
  TX-packets: 11             TX-dropped: 0             TX-total: 11
  ----------------------------------------------------------------------------

  +++++++++++++++ Accumulated forward statistics for all ports+++++++++++++++
  RX-packets: 22             RX-dropped: 0             RX-total: 22
  TX-packets: 22             TX-dropped: 20            TX-total: 42
  ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Done.

Stopping port 0...
Stopping ports...
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caac900 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caabc00 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9e8c0 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9edc0 removed.
mlx5_common: mlx5 list hrxq entry 0x75caa9d00 removed.
mlx5_common: mlx5 list hrxq entry 0x75caa9df0 removed.
mlx5_common: mlx5 list hrxq entry 0x75caa9ee0 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa92c0 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caac0c0 removed.
mlx5_common: mlx5 list mlx5_2_flow_table entry 0x8ffc50d80 removed.
Done

Stopping port 1...
Stopping ports...
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa2f00 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa2200 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9dec0 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9e3c0 removed.
mlx5_common: mlx5 list hrxq entry 0x75caa0300 removed.
mlx5_common: mlx5 list hrxq entry 0x75caa03f0 removed.
mlx5_common: mlx5 list hrxq entry 0x75caa04e0 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75ca9f8c0 removed.
mlx5_common: mlx5 list NIC_ingress_0_0_matcher_list entry 0x75caa26c0 removed.
mlx5_common: mlx5 list mlx5_3_flow_table entry 0x8fd670d80 removed.
Done

Shutting down port 0...
Closing ports...
mlx5_common: freeing B-tree 0x17dcb69074 with table 0x17dcb67b00
mlx5_common: freeing B-tree 0x17dcb66c34 with table 0x17dcb656c0
mlx5_common: freeing B-tree 0x197fede628 with table 0x17dcb6b3c0
mlx5_common: freeing B-tree 0x197fee86e8 with table 0x17dcb69f40
Port 0 is closed
Done

Shutting down port 1...
Closing ports...
mlx5_common: freeing B-tree 0x17dcb47834 with table 0x17dcb462c0
mlx5_common: freeing B-tree 0x17dcb453f4 with table 0x17dcb43e80
mlx5_common: freeing B-tree 0x17dcb4c6e8 with table 0x17dcb49b80
mlx5_common: freeing B-tree 0x17dcb516e8 with table 0x17dcb48700
Port 1 is closed
Done

> 
> What the hugepage size?
> In general, net/mlx5 does not rely on physical addresses.
> (You probably mean that a range of hugepages is discontiguous, because
> **each** hugepage is contiguous by definition.)
> 
> > This problem is introduced by this patch:
> >
> https://git.dpdk.org/dpdk/commit/?id=fec28ca0e3a93143829f3b41a28a8da93
> 3f28499.
> 
> Did you find it with bisection?

Yes


  reply	other threads:[~2022-01-11  8:21 UTC|newest]

Thread overview: 7+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-01-11  6:08 wangyunjian
2022-01-11  7:36 ` Dmitry Kozlyuk
2022-01-11  8:21   ` wangyunjian [this message]
2022-01-11 11:42     ` Dmitry Kozlyuk
2022-01-11 12:29       ` wangyunjian
2022-01-12  4:21       ` wangyunjian
2022-01-11  8:45   ` wangyunjian

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=2507d6c0239547c8b3f30870578ce392@huawei.com \
    --to=wangyunjian@huawei.com \
    --cc=dev@dpdk.org \
    --cc=dingxiaoxiong@huawei.com \
    --cc=dkozlyuk@nvidia.com \
    --cc=huangshaozhang@huawei.com \
    --cc=matan@nvidia.com \
    --cc=users@dpdk.org \
    --cc=viacheslavo@nvidia.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).