From: "Das, Surajit" <Surajit.Das@commscope.com>
To: Raslan Darawsheh <rasland@nvidia.com>, users <users@dpdk.org>
Cc: "Dharwadkar, Sriram" <Sriram.Dharwadkar@commscope.com>
Subject: Re: [dpdk-users] Skeleton basicfwd sample app hits error [net_mlx5: DV flow is not supported] on Mellanox VF in VM
Date: Thu, 28 Jan 2021 11:06:40 +0000 [thread overview]
Message-ID: <SN6PR14MB22530E89D4137934D3F335E2E9BA9@SN6PR14MB2253.namprd14.prod.outlook.com> (raw)
In-Reply-To: <DM6PR12MB2748E6622FC19955C51DC1C7CFBA9@DM6PR12MB2748.namprd12.prod.outlook.com>
Hi Raslan,
If I try that option inside the VM where MLNX_OFED is not installed, I still hit an error, but not the same one.
On bare metal where I have MLNX_OFED installed, I do not encounter the error and the same binary runs fine.
In the VM, I checked for the presence of drivers, and found them to be present.
Driver Info VM:
[root@dpdk-dev-1 build]# lsmod | grep mlx
mlx5_ib 262895 0
ib_uverbs 102208 2 mlx5_ib,rdma_ucm
ib_core 255469 13 rdma_cm,ib_cm,iw_cm,rpcrdma,mlx5_ib,ib_srp,ib_iser,ib_srpt,ib_umad,ib_uverbs,rdma_ucm,ib_ipoib,ib_isert
mlx5_core 628569 1 mlx5_ib
mlxfw 18227 1 mlx5_core
devlink 60067 1 mlx5_core
ptp 19231 1 mlx5_core
[root@dpdk-dev-1 build]# modinfo mlx5_core | grep version
version: 5.0-0
rhelversion: 7.7
srcversion: 8657804ACAFA1FD34AFF1A7
vermagic: 3.10.0-1062.el7.x86_64 SMP mod_unload modversions
[root@dpdk-dev-1 build]# /root/dpdk-stable-19.11.6/usertools/dpdk-devbind.py --status | grep mlx
0000:00:0d.0 'MT27700 Family [ConnectX-4 Virtual Function] 1014' if=ens13 drv=mlx5_core unused=igb_uio,uio_pci_generic
0000:00:0e.0 'MT27700 Family [ConnectX-4 Virtual Function] 1014' if=ens14 drv=mlx5_core unused=igb_uio,uio_pci_generic
VM error output here:
[root@dpdk-dev-1 build]# ./basicfwd -l 1 -n 4 --pci-whitelist 0000:00:0d.0,dv_flow_en=0 --pci-whitelist 0000:00:0e.0,dv_flow_en=0 --log-level 8
EAL: Detected 8 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: PCI device 0000:00:0d.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 15b3:1014 net_mlx5
net_mlx5: mlx5.c:3422: mlx5_pci_probe(): probe of PCI device 0000:00:0d.0 aborted after encountering an error: Operation not supported
EAL: Requested device 0000:00:0d.0 cannot be used
EAL: PCI device 0000:00:0e.0 on NUMA socket -1
EAL: Invalid NUMA socket, default to 0
EAL: probe driver: 15b3:1014 net_mlx5
net_mlx5: mlx5.c:3422: mlx5_pci_probe(): probe of PCI device 0000:00:0e.0 aborted after encountering an error: Operation not supported
EAL: Requested device 0000:00:0e.0 cannot be used
EAL: Error - exiting with code: 1
Cause: Error: number of ports must be even
[root@dpdk-dev-1 build]#
If I try to the same binary on a bare metal where MLNX_OFED is installed, the binary runs fine.
Here:
[root@experimental build]# ./basicfwd -l 1 -n 4 --pci-whitelist 0000:65:01.0,dv_flow_en=0 --pci-whitelist 0000:65:01.1,dv_flow_en=0 --log-level 8
EAL: Detected 48 lcore(s)
EAL: Detected 1 NUMA nodes
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'VA'
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL: PCI device 0000:65:01.0 on NUMA socket 0
EAL: probe driver: 15b3:1014 net_mlx5
EAL: PCI device 0000:65:01.1 on NUMA socket 0
EAL: probe driver: 15b3:1014 net_mlx5
Port 0 MAC: 12 e4 97 76 bf af
Port 1 MAC: 56 f9 e2 3f c7 34
Drivers on bare metal:
[root@experimental ~]# lsmod | grep mlx
mlx5_ib 262797 0
ib_uverbs 102208 1 mlx5_ib
ib_core 255552 2 mlx5_ib,ib_uverbs
mlx5_core 638172 1 mlx5_ib
mlxfw 18227 1 mlx5_core
devlink 60067 1 mlx5_core
ptp 19231 2 i40e,mlx5_core
[root@experimental ~]# modinfo mlx5_core | grep version
version: 5.0-0
rhelversion: 7.8
srcversion: 347B715A7BFEF96508E57F2
vermagic: 3.10.0-1127.el7.x86_64 SMP mod_unload modversions
[root@experimental ~]# ./dpdk-devbind.py --status | grep "65:01.0\|65:01.1"
0000:65:01.0 'MT27700 Family [ConnectX-4 Virtual Function] 1014' if=enp101s1 drv=mlx5_core unused=vfio-pci
0000:65:01.1 'MT27700 Family [ConnectX-4 Virtual Function] 1014' if=enp101s1f1 drv=mlx5_core unused=vfio-pci
[root@experimental ~]#
Now, my question is, what is it that MLNX_OFED providing that I am missing?
Regards,
Surajit
From: Raslan Darawsheh <rasland@nvidia.com>
Sent: Thursday, January 28, 2021 1:56 PM
To: Das, Surajit <Surajit.Das@commscope.com>; users <users@dpdk.org>
Cc: Dharwadkar, Sriram <Sriram.Dharwadkar@commscope.com>
Subject: RE: Skeleton basicfwd sample app hits error [net_mlx5: DV flow is not supported] on Mellanox VF in VM
Hello Das, Can you kindly try to add the following devarg to your pci-whitelist ? --pci-whitelist 0000:00:0d.0,dv_flow_en=0 -- pci-whitelist 0000:00:0e.0,dv_flow_en=0 Kindest regards, Raslan Darawsheh
External (rasland@nvidia.com<mailto:rasland@nvidia.com>)
Report This Email<https://shared.outlook.inky.com/report?id=Y29tbXNjb3BlL3N1cmFqaXQuZGFzQGNvbW1zY29wZS5jb20vMzEzMzgyMzQ4NDk0MWU0YzhjOTEwN2UyMDUzYjFkZjkvMTYxMTgyMjM2My40OQ==#key=7cb49c853831773fdc4118bcf5249b2e> FAQ<https://www.inky.com/banner-faq/> Protection by INKY<https://www.inky.com>
Hello Das,
Can you kindly try to add the following devarg to your pci-whitelist ?
--pci-whitelist 0000:00:0d.0,dv_flow_en=0 -- pci-whitelist 0000:00:0e.0,dv_flow_en=0
Kindest regards,
Raslan Darawsheh
> -----Original Message-----
> From: users <users-bounces@dpdk.org<mailto:users-bounces@dpdk.org>> On Behalf Of Das, Surajit
> Sent: Wednesday, January 27, 2021 7:08 PM
> To: users <users@dpdk.org<mailto:users@dpdk.org>>
> Cc: Dharwadkar, Sriram <Sriram.Dharwadkar@commscope.com<mailto:Sriram.Dharwadkar@commscope.com>>
> Subject: [dpdk-users] Skeleton basicfwd sample app hits error [net_mlx5: DV
> flow is not supported] on Mellanox VF in VM
>
> Hi DPDK experts,
>
> I am hitting an error when running the sample basicfwd app on Mellanox VFs.
> The error is same with l2fwd, l2fwd-crypto and ipsec-secgw apps.
>
> I have pushed the Mellanox VFs in a VM by PCI-Passthrough.
> Pasting the command, output and the relevant version info below.
> If anyone else has faced this, please do share what could fix this.
>
>
> Command and Output:
> [root@dpdk-dev-1 build]# ./basicfwd -l 1 -n 4 --pci-whitelist 0000:00:0d.0 --
> pci-whitelist 0000:00:0e.0 --log-level 8
> EAL: Detected 8 lcore(s)
> EAL: Detected 1 NUMA nodes
> EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
> EAL: Selected IOVA mode 'PA'
> EAL: No available hugepages reported in hugepages-1048576kB
> EAL: Probing VFIO support...
> EAL: PCI device 0000:00:0d.0 on NUMA socket -1
> EAL: Invalid NUMA socket, default to 0
> EAL: probe driver: 15b3:1014 net_mlx5
> net_mlx5: DV flow is not supported
> net_mlx5: probe of PCI device 0000:00:0d.0 aborted after encountering an
> error: Operation not supported
> EAL: Requested device 0000:00:0d.0 cannot be used
> EAL: PCI device 0000:00:0e.0 on NUMA socket -1
> EAL: Invalid NUMA socket, default to 0
> EAL: probe driver: 15b3:1014 net_mlx5
> net_mlx5: DV flow is not supported
> net_mlx5: probe of PCI device 0000:00:0e.0 aborted after encountering an
> error: Operation not supported
> EAL: Requested device 0000:00:0e.0 cannot be used
> EAL: Error - exiting with code: 1
> Cause: Error: number of ports must be even
> [root@dpdk-dev-1 build]#
>
> Versions:
> 3.10.0-1062.el7.x86_64
> CentOS Linux release 7.7.1908 (Core)
> dpdk-stable-19.11.6
>
> NIC detail:
> 00:0c.0 Ethernet controller: Intel Corporation Ethernet Virtual Function 700
> Series (rev 09)
> Subsystem: Super Micro Computer Inc Device 0000
> Physical Slot: 12
> Control: I/O- Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop-
> ParErr- Stepping- SERR- FastB2B- DisINTx+
> Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort-
> <TAbort- <MAbort- >SERR- <PERR- INTx-
> Latency: 0
> Region 0: Memory at fe220000 (64-bit, prefetchable) [size=128K]
> Region 3: Memory at fe244000 (64-bit, prefetchable) [size=16K]
> Capabilities: [70] MSI-X: Enable+ Count=5 Masked-
> Vector table: BAR=3 offset=00000000
> PBA: BAR=3 offset=00002000
> Capabilities: [a0] Express (v2) Endpoint, MSI 00
> DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1
> <64us
> ExtTag+ AttnBtn- AttnInd- PwrInd- RBE+ FLReset+
> SlotPowerLimit 0.000W
> DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
> RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- FLReset-
> MaxPayload 128 bytes, MaxReadReq 128 bytes
> DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr-
> TransPend-
> LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency
> L0s <64ns, L1 <1us
> ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
> LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
> ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
> LnkSta: Speed unknown, Width x0, TrErr- Train- SlotClk- DLActive-
> BWMgmt- ABWMgmt-
> DevCap2: Completion Timeout: Range AB, TimeoutDis+, LTR-, OBFF
> Not Supported
> DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF
> Disabled
> LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-,
> EqualizationPhase1-
> EqualizationPhase2-, EqualizationPhase3-,
> LinkEqualizationRequest-
> Kernel driver in use: iavf
> Kernel modules: iavf
>
> Regards,
> Surajit
prev parent reply other threads:[~2021-01-28 11:06 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-01-27 17:07 Das, Surajit
2021-01-28 8:25 ` Raslan Darawsheh
2021-01-28 11:06 ` Das, Surajit [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=SN6PR14MB22530E89D4137934D3F335E2E9BA9@SN6PR14MB2253.namprd14.prod.outlook.com \
--to=surajit.das@commscope.com \
--cc=Sriram.Dharwadkar@commscope.com \
--cc=rasland@nvidia.com \
--cc=users@dpdk.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).