* [dpdk-users] mellanox connect-x 3 pro dpdk with openvswitch
@ 2018-06-11 9:15 Tobias Urdin
2018-06-11 9:51 ` Raslan Darawsheh
0 siblings, 1 reply; 14+ messages in thread
From: Tobias Urdin @ 2018-06-11 9:15 UTC (permalink / raw)
To: users
Hello,
I'm stuck trying to test DPDK with a Mellanox Connect-X 3 Pro card on Ubuntu 16.04
I've enabled iommu, fixed hugepages and has installed and are running the openvswitch-switch-dpdk alternative.
$lspci | grep -i mellanox
3b:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]
$lsmod | grep -i mlx
mlx4_en 110592 0
vxlan 49152 2 i40e,mlx4_en
mlx4_core 286720 1 mlx4_en
ptp 20480 3 igb,i40e,mlx4_en
$cat /etc/modprobe.d/mlx4_core.conf
options mlx4_core log_num_mgm_entry_size=-1
$ovs-vsctl add-port br-mlx dpdk0 -- set Interface dpdk0 type=dpdk
Bridge br-mlx
Port "dpdk0"
Interface "dpdk0"
type: dpdk
error: "'dpdk0' is missing 'options:dpdk-devargs'. The old 'dpdk<port_id>' names are not supported"
$ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 type=dpdk options:dpdk-devargs=0000:3b:00.0,1
ovs-vsctl: Error detected while setting up 'enp59s0d1': Error attaching device '0000:3b:00.0,1' to DPDK. See ovs-vswitchd log for details.
ovs-vsctl: The default log directory is "/var/log/openvswitch".
2018-06-11T09:10:38.696Z|00177|netdev_dpdk|WARN|Error attaching device '0000:3b:00.0,1' to DPDK
2018-06-11T09:10:38.696Z|00178|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument)
2018-06-11T09:12:50.994Z|00181|dpdk|ERR|EAL: Driver cannot attach the device (0000:3b:00.0)
2018-06-11T09:12:50.994Z|00182|dpdk|ERR|EAL: No port found for device (0000:3b:00.0)
2018-06-11T09:12:50.994Z|00183|netdev_dpdk|WARN|Error attaching device '0000:3b:00.0,1' to DPDK
2018-06-11T09:12:50.994Z|00184|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument)
$ovs-vsctl add-port br-mlx dpdk0 -- set Interface dpdk0 type=dpdk options:dpdk-devargs="class=eth,mac=ec:0d:9a:b0:13:21"
ovs-vsctl: Error detected while setting up 'dpdk0': Error attaching device 'class=eth,mac=ec:0d:9a:b0:13:21' to DPDK. See ovs-vswitchd log for details.
ovs-vsctl: The default log directory is "/var/log/openvswitch".
Port "dpdk0"
Interface "dpdk0"
type: dpdk
options: {dpdk-devargs="class=eth,mac=ec:0d:9a:b0:13:21"}
error: "Error attaching device 'class=eth,mac=ec:0d:9a:b0:13:21' to DPDK"
Anybody know what I might have missed? I have tried restarting the openvswitch services after adding the interfaces as well but no difference.
The logs doesn't say anything interesting from what I can see :(
Best regards
^ permalink raw reply [flat|nested] 14+ messages in thread* Re: [dpdk-users] mellanox connect-x 3 pro dpdk with openvswitch 2018-06-11 9:15 [dpdk-users] mellanox connect-x 3 pro dpdk with openvswitch Tobias Urdin @ 2018-06-11 9:51 ` Raslan Darawsheh 2018-06-11 9:59 ` Tobias Urdin 0 siblings, 1 reply; 14+ messages in thread From: Raslan Darawsheh @ 2018-06-11 9:51 UTC (permalink / raw) To: Tobias Urdin, users Hi Tobias, Can you try using the following ? ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 type=dpdk options:dpdk-devargs=mlx4_0 if it didn't work can you send the OVS-DPDK init log ? Kindest regards, Raslan Darawsheh -----Original Message----- From: users [mailto:users-bounces@dpdk.org] On Behalf Of Tobias Urdin Sent: Monday, June 11, 2018 12:15 PM To: users@dpdk.org Subject: [dpdk-users] mellanox connect-x 3 pro dpdk with openvswitch Hello, I'm stuck trying to test DPDK with a Mellanox Connect-X 3 Pro card on Ubuntu 16.04 I've enabled iommu, fixed hugepages and has installed and are running the openvswitch-switch-dpdk alternative. $lspci | grep -i mellanox 3b:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro] $lsmod | grep -i mlx mlx4_en 110592 0 vxlan 49152 2 i40e,mlx4_en mlx4_core 286720 1 mlx4_en ptp 20480 3 igb,i40e,mlx4_en $cat /etc/modprobe.d/mlx4_core.conf options mlx4_core log_num_mgm_entry_size=-1 $ovs-vsctl add-port br-mlx dpdk0 -- set Interface dpdk0 type=dpdk Bridge br-mlx Port "dpdk0" Interface "dpdk0" type: dpdk error: "'dpdk0' is missing 'options:dpdk-devargs'. The old 'dpdk<port_id>' names are not supported" $ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 type=dpdk options:dpdk-devargs=0000:3b:00.0,1 ovs-vsctl: Error detected while setting up 'enp59s0d1': Error attaching device '0000:3b:00.0,1' to DPDK. See ovs-vswitchd log for details. ovs-vsctl: The default log directory is "/var/log/openvswitch". 2018-06-11T09:10:38.696Z|00177|netdev_dpdk|WARN|Error attaching device '0000:3b:00.0,1' to DPDK 2018-06-11T09:10:38.696Z|00178|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T09:12:50.994Z|00181|dpdk|ERR|EAL: Driver cannot attach the device (0000:3b:00.0) 2018-06-11T09:12:50.994Z|00182|dpdk|ERR|EAL: No port found for device (0000:3b:00.0) 2018-06-11T09:12:50.994Z|00183|netdev_dpdk|WARN|Error attaching device '0000:3b:00.0,1' to DPDK 2018-06-11T09:12:50.994Z|00184|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) $ovs-vsctl add-port br-mlx dpdk0 -- set Interface dpdk0 type=dpdk options:dpdk-devargs="class=eth,mac=ec:0d:9a:b0:13:21" ovs-vsctl: Error detected while setting up 'dpdk0': Error attaching device 'class=eth,mac=ec:0d:9a:b0:13:21' to DPDK. See ovs-vswitchd log for details. ovs-vsctl: The default log directory is "/var/log/openvswitch". Port "dpdk0" Interface "dpdk0" type: dpdk options: {dpdk-devargs="class=eth,mac=ec:0d:9a:b0:13:21"} error: "Error attaching device 'class=eth,mac=ec:0d:9a:b0:13:21' to DPDK" Anybody know what I might have missed? I have tried restarting the openvswitch services after adding the interfaces as well but no difference. The logs doesn't say anything interesting from what I can see :( Best regards ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [dpdk-users] mellanox connect-x 3 pro dpdk with openvswitch 2018-06-11 9:51 ` Raslan Darawsheh @ 2018-06-11 9:59 ` Tobias Urdin 2018-06-11 10:57 ` Raslan Darawsheh 0 siblings, 1 reply; 14+ messages in thread From: Tobias Urdin @ 2018-06-11 9:59 UTC (permalink / raw) To: Raslan Darawsheh; +Cc: users Hello Raslan, Thank you for your reply. For reference the br-mlx bridge was created like this: $ovs-vsctl add-br br-mlx -- set bridge br-mlx datapath_type=netdev $ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 type=dpdk options:dpdk-devargs=mlx4_0 ovs-vsctl: Error detected while setting up 'enp59s0d1': Error attaching device 'mlx4_0' to DPDK. See ovs-vswitchd log for details. ovs-vsctl: The default log directory is "/var/log/openvswitch". In the ovs-vswitchd.log 2018-06-11T09:56:20.613Z|00185|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T09:56:20.613Z|00186|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T09:56:20.613Z|00187|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) Here is the complete log after restarting openvswitch-switch service: 2018-06-11T09:57:49.380Z|00134|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.9.0 2018-06-11T09:57:52.812Z|00135|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T09:57:52.812Z|00136|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T09:57:52.812Z|00137|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T09:57:52.825Z|00138|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T09:57:52.825Z|00139|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T09:57:52.825Z|00140|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T09:57:52.832Z|00141|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T09:57:52.832Z|00142|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T09:57:52.832Z|00143|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T09:57:52.843Z|00144|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T09:57:52.843Z|00145|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T09:57:52.843Z|00146|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T09:57:52.856Z|00147|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T09:57:52.856Z|00148|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T09:57:52.856Z|00149|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T09:57:52.862Z|00150|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T09:57:52.862Z|00151|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T09:57:52.862Z|00152|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T09:57:52.870Z|00153|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T09:57:52.870Z|00154|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T09:57:52.870Z|00155|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T09:57:52.889Z|00156|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T09:57:52.889Z|00157|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T09:57:52.889Z|00158|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T09:57:52.903Z|00159|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T09:57:52.903Z|00160|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T09:57:52.903Z|00161|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T09:57:52.908Z|00162|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T09:57:52.908Z|00163|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T09:57:52.908Z|00164|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T09:57:52.914Z|00165|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T09:57:52.914Z|00166|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T09:57:52.914Z|00167|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T09:57:52.923Z|00168|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T09:57:52.923Z|00169|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T09:57:52.923Z|00170|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T09:57:54.808Z|00171|memory|INFO|1406100 kB peak resident set size after 10.1 seconds 2018-06-11T09:57:54.808Z|00172|memory|INFO|handlers:118 ofconns:3 ports:8 revalidators:42 rules:36 udpif keys:1 2018-06-11T09:58:02.850Z|00173|connmgr|INFO|br-int<->tcp:127.0.0.1:6633: 5 flow_mods 10 s ago (5 adds) 2018-06-11T09:58:02.898Z|00174|connmgr|INFO|br-ex<->tcp:127.0.0.1:6633: 2 flow_mods 10 s ago (2 adds) 2018-06-11T09:58:02.930Z|00175|connmgr|INFO|br-tun<->tcp:127.0.0.1:6633: 12 flow_mods 10 s ago (12 adds) Was that what you mean by init log? Best regards On 06/11/2018 11:51 AM, Raslan Darawsheh wrote: > Hi Tobias, > > Can you try using the following ? > > ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 type=dpdk options:dpdk-devargs=mlx4_0 > > if it didn't work can you send the OVS-DPDK init log ? > > Kindest regards, > Raslan Darawsheh > > -----Original Message----- > From: users [mailto:users-bounces@dpdk.org] On Behalf Of Tobias Urdin > Sent: Monday, June 11, 2018 12:15 PM > To: users@dpdk.org > Subject: [dpdk-users] mellanox connect-x 3 pro dpdk with openvswitch > > Hello, > > I'm stuck trying to test DPDK with a Mellanox Connect-X 3 Pro card on Ubuntu 16.04 > > I've enabled iommu, fixed hugepages and has installed and are running the openvswitch-switch-dpdk alternative. > > $lspci | grep -i mellanox > 3b:00.0 Ethernet controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro] > > $lsmod | grep -i mlx > mlx4_en 110592 0 > vxlan 49152 2 i40e,mlx4_en > mlx4_core 286720 1 mlx4_en > ptp 20480 3 igb,i40e,mlx4_en > > $cat /etc/modprobe.d/mlx4_core.conf > options mlx4_core log_num_mgm_entry_size=-1 > > > $ovs-vsctl add-port br-mlx dpdk0 -- set Interface dpdk0 type=dpdk > > Bridge br-mlx > Port "dpdk0" > Interface "dpdk0" > type: dpdk > error: "'dpdk0' is missing 'options:dpdk-devargs'. The old 'dpdk<port_id>' names are not supported" > > > $ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 type=dpdk options:dpdk-devargs=0000:3b:00.0,1 > ovs-vsctl: Error detected while setting up 'enp59s0d1': Error attaching device '0000:3b:00.0,1' to DPDK. See ovs-vswitchd log for details. > ovs-vsctl: The default log directory is "/var/log/openvswitch". > > 2018-06-11T09:10:38.696Z|00177|netdev_dpdk|WARN|Error attaching device '0000:3b:00.0,1' to DPDK > 2018-06-11T09:10:38.696Z|00178|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) > > 2018-06-11T09:12:50.994Z|00181|dpdk|ERR|EAL: Driver cannot attach the device (0000:3b:00.0) > 2018-06-11T09:12:50.994Z|00182|dpdk|ERR|EAL: No port found for device (0000:3b:00.0) 2018-06-11T09:12:50.994Z|00183|netdev_dpdk|WARN|Error attaching device '0000:3b:00.0,1' to DPDK > 2018-06-11T09:12:50.994Z|00184|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) > > $ovs-vsctl add-port br-mlx dpdk0 -- set Interface dpdk0 type=dpdk options:dpdk-devargs="class=eth,mac=ec:0d:9a:b0:13:21" > ovs-vsctl: Error detected while setting up 'dpdk0': Error attaching device 'class=eth,mac=ec:0d:9a:b0:13:21' to DPDK. See ovs-vswitchd log for details. > ovs-vsctl: The default log directory is "/var/log/openvswitch". > > Port "dpdk0" > Interface "dpdk0" > type: dpdk > options: {dpdk-devargs="class=eth,mac=ec:0d:9a:b0:13:21"} > error: "Error attaching device 'class=eth,mac=ec:0d:9a:b0:13:21' to DPDK" > > > Anybody know what I might have missed? I have tried restarting the openvswitch services after adding the interfaces as well but no difference. > > The logs doesn't say anything interesting from what I can see :( > > Best regards > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [dpdk-users] mellanox connect-x 3 pro dpdk with openvswitch 2018-06-11 9:59 ` Tobias Urdin @ 2018-06-11 10:57 ` Raslan Darawsheh 2018-06-11 11:15 ` Tobias Urdin 0 siblings, 1 reply; 14+ messages in thread From: Raslan Darawsheh @ 2018-06-11 10:57 UTC (permalink / raw) To: Tobias Urdin; +Cc: users What I really want to see is the init process for OVS-DPDK that dumps the DPDK initialization. I think you will find it at the beginning of the log that you just sent. Kindest regards, Raslan Darawsheh -----Original Message----- From: Tobias Urdin [mailto:tobias.urdin@crystone.com] Sent: Monday, June 11, 2018 1:00 PM To: Raslan Darawsheh <rasland@mellanox.com> Cc: users@dpdk.org Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch Hello Raslan, Thank you for your reply. For reference the br-mlx bridge was created like this: $ovs-vsctl add-br br-mlx -- set bridge br-mlx datapath_type=netdev $ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 type=dpdk options:dpdk-devargs=mlx4_0 ovs-vsctl: Error detected while setting up 'enp59s0d1': Error attaching device 'mlx4_0' to DPDK. See ovs-vswitchd log for details. ovs-vsctl: The default log directory is "/var/log/openvswitch". In the ovs-vswitchd.log 2018-06-11T09:56:20.613Z|00185|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T09:56:20.613Z|00186|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T09:56:20.613Z|00187|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) Here is the complete log after restarting openvswitch-switch service: 2018-06-11T09:57:49.380Z|00134|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.9.0 2018-06-11T09:57:52.812Z|00135|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T09:57:52.812Z|00136|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T09:57:52.812Z|00137|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T09:57:52.825Z|00138|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T09:57:52.825Z|00139|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T09:57:52.825Z|00140|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T09:57:52.832Z|00141|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T09:57:52.832Z|00142|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T09:57:52.832Z|00143|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T09:57:52.843Z|00144|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T09:57:52.843Z|00145|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T09:57:52.843Z|00146|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T09:57:52.856Z|00147|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T09:57:52.856Z|00148|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T09:57:52.856Z|00149|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T09:57:52.862Z|00150|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T09:57:52.862Z|00151|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T09:57:52.862Z|00152|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T09:57:52.870Z|00153|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T09:57:52.870Z|00154|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T09:57:52.870Z|00155|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T09:57:52.889Z|00156|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T09:57:52.889Z|00157|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T09:57:52.889Z|00158|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T09:57:52.903Z|00159|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T09:57:52.903Z|00160|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T09:57:52.903Z|00161|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T09:57:52.908Z|00162|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T09:57:52.908Z|00163|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T09:57:52.908Z|00164|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T09:57:52.914Z|00165|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T09:57:52.914Z|00166|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T09:57:52.914Z|00167|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T09:57:52.923Z|00168|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T09:57:52.923Z|00169|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T09:57:52.923Z|00170|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T09:57:54.808Z|00171|memory|INFO|1406100 kB peak resident set size after 10.1 seconds 2018-06-11T09:57:54.808Z|00172|memory|INFO|handlers:118 ofconns:3 ports:8 revalidators:42 rules:36 udpif keys:1 2018-06-11T09:58:02.850Z|00173|connmgr|INFO|br-int<->tcp:127.0.0.1:6633: 5 flow_mods 10 s ago (5 adds) 2018-06-11T09:58:02.898Z|00174|connmgr|INFO|br-ex<->tcp:127.0.0.1:6633: 2 flow_mods 10 s ago (2 adds) 2018-06-11T09:58:02.930Z|00175|connmgr|INFO|br-tun<->tcp:127.0.0.1:6633: 12 flow_mods 10 s ago (12 adds) Was that what you mean by init log? Best regards On 06/11/2018 11:51 AM, Raslan Darawsheh wrote: > Hi Tobias, > > Can you try using the following ? > > ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 > type=dpdk options:dpdk-devargs=mlx4_0 > > if it didn't work can you send the OVS-DPDK init log ? > > Kindest regards, > Raslan Darawsheh > > -----Original Message----- > From: users [mailto:users-bounces@dpdk.org] On Behalf Of Tobias Urdin > Sent: Monday, June 11, 2018 12:15 PM > To: users@dpdk.org > Subject: [dpdk-users] mellanox connect-x 3 pro dpdk with openvswitch > > Hello, > > I'm stuck trying to test DPDK with a Mellanox Connect-X 3 Pro card on > Ubuntu 16.04 > > I've enabled iommu, fixed hugepages and has installed and are running the openvswitch-switch-dpdk alternative. > > $lspci | grep -i mellanox > 3b:00.0 Ethernet controller: Mellanox Technologies MT27520 Family > [ConnectX-3 Pro] > > $lsmod | grep -i mlx > mlx4_en 110592 0 > vxlan 49152 2 i40e,mlx4_en > mlx4_core 286720 1 mlx4_en > ptp 20480 3 igb,i40e,mlx4_en > > $cat /etc/modprobe.d/mlx4_core.conf > options mlx4_core log_num_mgm_entry_size=-1 > > > $ovs-vsctl add-port br-mlx dpdk0 -- set Interface dpdk0 type=dpdk > > Bridge br-mlx > Port "dpdk0" > Interface "dpdk0" > type: dpdk > error: "'dpdk0' is missing 'options:dpdk-devargs'. The old 'dpdk<port_id>' names are not supported" > > > $ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 > type=dpdk options:dpdk-devargs=0000:3b:00.0,1 > ovs-vsctl: Error detected while setting up 'enp59s0d1': Error attaching device '0000:3b:00.0,1' to DPDK. See ovs-vswitchd log for details. > ovs-vsctl: The default log directory is "/var/log/openvswitch". > > 2018-06-11T09:10:38.696Z|00177|netdev_dpdk|WARN|Error attaching device > '0000:3b:00.0,1' to DPDK > 2018-06-11T09:10:38.696Z|00178|netdev|WARN|enp59s0d1: could not set > configuration (Invalid argument) > > 2018-06-11T09:12:50.994Z|00181|dpdk|ERR|EAL: Driver cannot attach the > device (0000:3b:00.0) > 2018-06-11T09:12:50.994Z|00182|dpdk|ERR|EAL: No port found for device > (0000:3b:00.0) 2018-06-11T09:12:50.994Z|00183|netdev_dpdk|WARN|Error > attaching device '0000:3b:00.0,1' to DPDK > 2018-06-11T09:12:50.994Z|00184|netdev|WARN|enp59s0d1: could not set > configuration (Invalid argument) > > $ovs-vsctl add-port br-mlx dpdk0 -- set Interface dpdk0 type=dpdk options:dpdk-devargs="class=eth,mac=ec:0d:9a:b0:13:21" > ovs-vsctl: Error detected while setting up 'dpdk0': Error attaching device 'class=eth,mac=ec:0d:9a:b0:13:21' to DPDK. See ovs-vswitchd log for details. > ovs-vsctl: The default log directory is "/var/log/openvswitch". > > Port "dpdk0" > Interface "dpdk0" > type: dpdk > options: {dpdk-devargs="class=eth,mac=ec:0d:9a:b0:13:21"} > error: "Error attaching device 'class=eth,mac=ec:0d:9a:b0:13:21' to DPDK" > > > Anybody know what I might have missed? I have tried restarting the openvswitch services after adding the interfaces as well but no difference. > > The logs doesn't say anything interesting from what I can see :( > > Best regards > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [dpdk-users] mellanox connect-x 3 pro dpdk with openvswitch 2018-06-11 10:57 ` Raslan Darawsheh @ 2018-06-11 11:15 ` Tobias Urdin 2018-06-11 11:19 ` Raslan Darawsheh 0 siblings, 1 reply; 14+ messages in thread From: Tobias Urdin @ 2018-06-11 11:15 UTC (permalink / raw) To: Raslan Darawsheh; +Cc: users Ok, here is the complete log when restarting the openvswitch-switch service, see below. Also here is the kernel for information: $uname -r 4.4.0-127-generic 2018-06-11T11:13:42.729Z|00176|bridge|INFO|bridge br-mlx: deleted interface br-mlx on port 65534 2018-06-11T11:13:43.274Z|00177|bridge|INFO|bridge br-int: deleted interface br-int on port 65534 2018-06-11T11:13:43.275Z|00178|bridge|INFO|bridge br-int: deleted interface int-br-ex on port 1 2018-06-11T11:13:43.275Z|00179|bridge|INFO|bridge br-int: deleted interface patch-tun on port 2 2018-06-11T11:13:43.762Z|00180|bridge|INFO|bridge br-tun: deleted interface patch-int on port 1 2018-06-11T11:13:43.762Z|00181|bridge|INFO|bridge br-tun: deleted interface br-tun on port 65534 2018-06-11T11:13:44.231Z|00182|bridge|INFO|bridge br-ex: deleted interface br-ex on port 65534 2018-06-11T11:13:44.232Z|00183|bridge|INFO|bridge br-ex: deleted interface phy-br-ex on port 1 2018-06-11T11:13:45.245Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log 2018-06-11T11:13:45.252Z|00002|ovs_numa|INFO|Discovered 40 CPU cores on NUMA node 0 2018-06-11T11:13:45.252Z|00003|ovs_numa|INFO|Discovered 40 CPU cores on NUMA node 1 2018-06-11T11:13:45.252Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 80 CPU cores 2018-06-11T11:13:45.252Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting... 2018-06-11T11:13:45.252Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected 2018-06-11T11:13:45.256Z|00007|dpdk|INFO|Using DPDK 17.11.0 2018-06-11T11:13:45.256Z|00008|dpdk|INFO|DPDK Enabled - initializing... 2018-06-11T11:13:45.256Z|00009|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch 2018-06-11T11:13:45.256Z|00010|dpdk|INFO|IOMMU support for vhost-user-client disabled. 2018-06-11T11:13:45.256Z|00011|dpdk|INFO|EAL ARGS: ovs-vswitchd --socket-mem 1024,0 -c 0x00000001 2018-06-11T11:13:45.258Z|00012|dpdk|INFO|EAL: Detected 80 lcore(s) 2018-06-11T11:13:45.285Z|00013|dpdk|INFO|EAL: Probing VFIO support... 2018-06-11T11:13:45.285Z|00014|dpdk|INFO|EAL: VFIO support initialized 2018-06-11T11:13:48.214Z|00015|dpdk|INFO|EAL: PCI device 0000:01:00.0 on NUMA socket 0 2018-06-11T11:13:48.214Z|00016|dpdk|INFO|EAL: probe driver: 8086:1521 net_e1000_igb 2018-06-11T11:13:48.214Z|00017|dpdk|INFO|EAL: PCI device 0000:01:00.1 on NUMA socket 0 2018-06-11T11:13:48.214Z|00018|dpdk|INFO|EAL: probe driver: 8086:1521 net_e1000_igb 2018-06-11T11:13:48.214Z|00019|dpdk|INFO|EAL: PCI device 0000:18:00.0 on NUMA socket 0 2018-06-11T11:13:48.214Z|00020|dpdk|INFO|EAL: probe driver: 8086:1572 net_i40e 2018-06-11T11:13:48.214Z|00021|dpdk|INFO|EAL: PCI device 0000:18:00.1 on NUMA socket 0 2018-06-11T11:13:48.214Z|00022|dpdk|INFO|EAL: probe driver: 8086:1572 net_i40e 2018-06-11T11:13:48.214Z|00023|dpdk|INFO|DPDK pdump packet capture enabled 2018-06-11T11:13:48.224Z|00024|dpdk|INFO|DPDK Enabled - initialized 2018-06-11T11:13:48.235Z|00025|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports recirculation 2018-06-11T11:13:48.235Z|00026|ofproto_dpif|INFO|netdev@ovs-netdev: VLAN header stack length probed as 1 2018-06-11T11:13:48.235Z|00027|ofproto_dpif|INFO|netdev@ovs-netdev: MPLS label stack length probed as 3 2018-06-11T11:13:48.235Z|00028|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports truncate action 2018-06-11T11:13:48.235Z|00029|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports unique flow ids 2018-06-11T11:13:48.235Z|00030|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports clone action 2018-06-11T11:13:48.235Z|00031|ofproto_dpif|INFO|netdev@ovs-netdev: Max sample nesting level probed as 10 2018-06-11T11:13:48.235Z|00032|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports eventmask in conntrack action 2018-06-11T11:13:48.235Z|00033|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_clear action 2018-06-11T11:13:48.235Z|00034|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_state 2018-06-11T11:13:48.235Z|00035|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_zone 2018-06-11T11:13:48.235Z|00036|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_mark 2018-06-11T11:13:48.235Z|00037|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_label 2018-06-11T11:13:48.235Z|00038|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_state_nat 2018-06-11T11:13:48.235Z|00039|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_orig_tuple 2018-06-11T11:13:48.235Z|00040|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_orig_tuple6 2018-06-11T11:13:48.605Z|00041|ofproto_dpif|INFO|system@ovs-system: Datapath supports recirculation 2018-06-11T11:13:48.605Z|00042|ofproto_dpif|INFO|system@ovs-system: VLAN header stack length probed as 1 2018-06-11T11:13:48.605Z|00043|ofproto_dpif|INFO|system@ovs-system: MPLS label stack length probed as 1 2018-06-11T11:13:48.605Z|00044|ofproto_dpif|INFO|system@ovs-system: Datapath does not support truncate action 2018-06-11T11:13:48.605Z|00045|ofproto_dpif|INFO|system@ovs-system: Datapath supports unique flow ids 2018-06-11T11:13:48.605Z|00046|ofproto_dpif|INFO|system@ovs-system: Datapath does not support clone action 2018-06-11T11:13:48.605Z|00047|ofproto_dpif|INFO|system@ovs-system: Max sample nesting level probed as 3 2018-06-11T11:13:48.605Z|00048|ofproto_dpif|INFO|system@ovs-system: Datapath does not support eventmask in conntrack action 2018-06-11T11:13:48.605Z|00049|ofproto_dpif|INFO|system@ovs-system: Datapath does not support ct_clear action 2018-06-11T11:13:48.605Z|00050|ofproto_dpif|INFO|system@ovs-system: Datapath supports ct_state 2018-06-11T11:13:48.605Z|00051|ofproto_dpif|INFO|system@ovs-system: Datapath supports ct_zone 2018-06-11T11:13:48.605Z|00052|ofproto_dpif|INFO|system@ovs-system: Datapath supports ct_mark 2018-06-11T11:13:48.605Z|00053|ofproto_dpif|INFO|system@ovs-system: Datapath supports ct_label 2018-06-11T11:13:48.605Z|00054|ofproto_dpif|INFO|system@ovs-system: Datapath does not support ct_state_nat 2018-06-11T11:13:48.605Z|00055|ofproto_dpif|INFO|system@ovs-system: Datapath does not support ct_orig_tuple 2018-06-11T11:13:48.605Z|00056|ofproto_dpif|INFO|system@ovs-system: Datapath does not support ct_orig_tuple6 2018-06-11T11:13:48.981Z|00057|bridge|INFO|bridge br-mlx: added interface br-mlx on port 65534 2018-06-11T11:13:48.981Z|00058|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T11:13:48.981Z|00059|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T11:13:48.981Z|00060|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T11:13:48.982Z|00061|bridge|INFO|bridge br-int: added interface br-int on port 65534 2018-06-11T11:13:48.982Z|00062|bridge|INFO|bridge br-int: added interface int-br-ex on port 1 2018-06-11T11:13:48.982Z|00063|bridge|INFO|bridge br-int: added interface patch-tun on port 2 2018-06-11T11:13:48.982Z|00064|bridge|INFO|bridge br-tun: added interface patch-int on port 1 2018-06-11T11:13:48.982Z|00065|bridge|INFO|bridge br-tun: added interface br-tun on port 65534 2018-06-11T11:13:48.982Z|00066|bridge|INFO|bridge br-ex: added interface br-ex on port 65534 2018-06-11T11:13:48.982Z|00067|bridge|INFO|bridge br-ex: added interface phy-br-ex on port 1 2018-06-11T11:13:48.982Z|00068|bridge|INFO|bridge br-mlx: using datapath ID 000072528484a642 2018-06-11T11:13:48.982Z|00069|connmgr|INFO|br-mlx: added service controller "punix:/var/run/openvswitch/br-mlx.mgmt" 2018-06-11T11:13:49.221Z|00070|bridge|INFO|bridge br-int: using datapath ID 00006680927b6b40 2018-06-11T11:13:49.221Z|00071|connmgr|INFO|br-int: added service controller "punix:/var/run/openvswitch/br-int.mgmt" 2018-06-11T11:13:49.221Z|00072|connmgr|INFO|br-int: added primary controller "tcp:127.0.0.1:6633" 2018-06-11T11:13:49.221Z|00073|rconn|INFO|br-int<->tcp:127.0.0.1:6633: connecting... 2018-06-11T11:13:49.460Z|00074|bridge|INFO|bridge br-tun: using datapath ID 000066532e5f0443 2018-06-11T11:13:49.461Z|00075|connmgr|INFO|br-tun: added service controller "punix:/var/run/openvswitch/br-tun.mgmt" 2018-06-11T11:13:49.461Z|00076|connmgr|INFO|br-tun: added primary controller "tcp:127.0.0.1:6633" 2018-06-11T11:13:49.461Z|00077|rconn|INFO|br-tun<->tcp:127.0.0.1:6633: connecting... 2018-06-11T11:13:49.692Z|00078|bridge|INFO|bridge br-ex: using datapath ID 0000e2d1bdbe8e45 2018-06-11T11:13:49.692Z|00079|connmgr|INFO|br-ex: added service controller "punix:/var/run/openvswitch/br-ex.mgmt" 2018-06-11T11:13:49.692Z|00080|connmgr|INFO|br-ex: added primary controller "tcp:127.0.0.1:6633" 2018-06-11T11:13:49.692Z|00081|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: connecting... 2018-06-11T11:13:49.916Z|00082|timeval|WARN|Unreasonably long 4661ms poll interval (16ms user, 4472ms system) 2018-06-11T11:13:49.916Z|00083|timeval|WARN|faults: 3584706 minor, 0 major 2018-06-11T11:13:49.916Z|00084|timeval|WARN|disk: 0 reads, 16 writes 2018-06-11T11:13:49.916Z|00085|timeval|WARN|context switches: 147 voluntary, 25 involuntary 2018-06-11T11:13:49.916Z|00086|coverage|INFO|Event coverage, avg rate over last: 5 seconds, last minute, last hour, hash=2ba35161: 2018-06-11T11:13:49.916Z|00087|coverage|INFO|bridge_reconfigure 0.0/sec 0.000/sec 0.0000/sec total: 1 2018-06-11T11:13:49.916Z|00088|coverage|INFO|ofproto_flush 0.0/sec 0.000/sec 0.0000/sec total: 4 2018-06-11T11:13:49.916Z|00089|coverage|INFO|ofproto_update_port 0.0/sec 0.000/sec 0.0000/sec total: 8 2018-06-11T11:13:49.916Z|00090|coverage|INFO|rev_flow_table 0.0/sec 0.000/sec 0.0000/sec total: 1 2018-06-11T11:13:49.916Z|00091|coverage|INFO|cmap_expand 0.0/sec 0.000/sec 0.0000/sec total: 52 2018-06-11T11:13:49.916Z|00092|coverage|INFO|cmap_shrink 0.0/sec 0.000/sec 0.0000/sec total: 23 2018-06-11T11:13:49.916Z|00093|coverage|INFO|dpif_port_add 0.0/sec 0.000/sec 0.0000/sec total: 1 2018-06-11T11:13:49.916Z|00094|coverage|INFO|dpif_flow_flush 0.0/sec 0.000/sec 0.0000/sec total: 6 2018-06-11T11:13:49.916Z|00095|coverage|INFO|dpif_flow_get 0.0/sec 0.000/sec 0.0000/sec total: 29 2018-06-11T11:13:49.916Z|00096|coverage|INFO|dpif_flow_put 0.0/sec 0.000/sec 0.0000/sec total: 37 2018-06-11T11:13:49.916Z|00097|coverage|INFO|dpif_flow_del 0.0/sec 0.000/sec 0.0000/sec total: 29 2018-06-11T11:13:49.916Z|00098|coverage|INFO|dpif_execute 0.0/sec 0.000/sec 0.0000/sec total: 8 2018-06-11T11:13:49.916Z|00099|coverage|INFO|flow_extract 0.0/sec 0.000/sec 0.0000/sec total: 6 2018-06-11T11:13:49.916Z|00100|coverage|INFO|miniflow_malloc 0.0/sec 0.000/sec 0.0000/sec total: 34 2018-06-11T11:13:49.916Z|00101|coverage|INFO|hmap_expand 0.0/sec 0.000/sec 0.0000/sec total: 498 2018-06-11T11:13:49.916Z|00102|coverage|INFO|netdev_received 0.0/sec 0.000/sec 0.0000/sec total: 1 2018-06-11T11:13:49.916Z|00103|coverage|INFO|netdev_get_stats 0.0/sec 0.000/sec 0.0000/sec total: 8 2018-06-11T11:13:49.916Z|00104|coverage|INFO|poll_create_node 0.0/sec 0.000/sec 0.0000/sec total: 23 2018-06-11T11:13:49.916Z|00105|coverage|INFO|seq_change 0.0/sec 0.000/sec 0.0000/sec total: 2193 2018-06-11T11:13:49.916Z|00106|coverage|INFO|pstream_open 0.0/sec 0.000/sec 0.0000/sec total: 9 2018-06-11T11:13:49.916Z|00107|coverage|INFO|stream_open 0.0/sec 0.000/sec 0.0000/sec total: 4 2018-06-11T11:13:49.916Z|00108|coverage|INFO|util_xalloc 0.0/sec 0.000/sec 0.0000/sec total: 12803 2018-06-11T11:13:49.916Z|00109|coverage|INFO|vconn_open 0.0/sec 0.000/sec 0.0000/sec total: 3 2018-06-11T11:13:49.916Z|00110|coverage|INFO|netdev_set_policing 0.0/sec 0.000/sec 0.0000/sec total: 4 2018-06-11T11:13:49.916Z|00111|coverage|INFO|netdev_get_ifindex 0.0/sec 0.000/sec 0.0000/sec total: 4 2018-06-11T11:13:49.916Z|00112|coverage|INFO|netdev_get_hwaddr 0.0/sec 0.000/sec 0.0000/sec total: 9 2018-06-11T11:13:49.916Z|00113|coverage|INFO|netdev_get_ethtool 0.0/sec 0.000/sec 0.0000/sec total: 2 2018-06-11T11:13:49.916Z|00114|coverage|INFO|netlink_received 0.0/sec 0.000/sec 0.0000/sec total: 85 2018-06-11T11:13:49.916Z|00115|coverage|INFO|netlink_recv_jumbo 0.0/sec 0.000/sec 0.0000/sec total: 1 2018-06-11T11:13:49.916Z|00116|coverage|INFO|netlink_sent 0.0/sec 0.000/sec 0.0000/sec total: 102 2018-06-11T11:13:49.916Z|00117|coverage|INFO|74 events never hit 2018-06-11T11:13:49.916Z|00118|poll_loop|INFO|wakeup due to [POLLIN] on fd 433 (FIFO pipe:[1025715]) at ../lib/ovs-rcu.c:229 (97% CPU usage) 2018-06-11T11:13:49.917Z|00119|poll_loop|INFO|wakeup due to [POLLIN] on fd 433 (FIFO pipe:[1025715]) at ../lib/ovs-rcu.c:229 (97% CPU usage) 2018-06-11T11:13:49.917Z|00120|poll_loop|INFO|wakeup due to 0-ms timeout at ../lib/ovs-rcu.c:229 (97% CPU usage) 2018-06-11T11:13:49.917Z|00121|poll_loop|INFO|wakeup due to 0-ms timeout at ../lib/ovs-rcu.c:229 (97% CPU usage) 2018-06-11T11:13:49.920Z|00122|rconn|INFO|br-int<->tcp:127.0.0.1:6633: connected 2018-06-11T11:13:49.920Z|00123|rconn|INFO|br-tun<->tcp:127.0.0.1:6633: connected 2018-06-11T11:13:49.920Z|00124|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: connected 2018-06-11T11:13:49.926Z|00125|poll_loop|INFO|wakeup due to [POLLIN] on fd 10 (NETLINK_ROUTE<->NETLINK_ROUTE) at ../lib/netlink-socket.c:1331 (97% CPU usage) 2018-06-11T11:13:49.926Z|00126|poll_loop|INFO|wakeup due to [POLLIN] on fd 421 (character device /dev/net/tun) at ../lib/netdev-linux.c:1171 (97% CPU usage) 2018-06-11T11:13:49.926Z|00127|poll_loop|INFO|wakeup due to [POLLIN] on fd 423 (127.0.0.1:33712<->127.0.0.1:6633) at ../lib/stream-fd.c:157 (97% CPU usage) 2018-06-11T11:13:49.926Z|00128|poll_loop|INFO|wakeup due to [POLLIN] on fd 428 (127.0.0.1:33714<->127.0.0.1:6633) at ../lib/stream-fd.c:157 (97% CPU usage) 2018-06-11T11:13:49.926Z|00129|poll_loop|INFO|wakeup due to [POLLIN] on fd 11 (<->/var/run/openvswitch/db.sock) at ../lib/stream-fd.c:157 (97% CPU usage) 2018-06-11T11:13:49.926Z|00130|poll_loop|INFO|wakeup due to [POLLIN] on fd 429 (127.0.0.1:33716<->127.0.0.1:6633) at ../lib/stream-fd.c:157 (97% CPU usage) 2018-06-11T11:13:49.930Z|00131|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T11:13:49.930Z|00132|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T11:13:49.930Z|00133|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T11:13:49.931Z|00134|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.9.0 2018-06-11T11:13:53.662Z|00135|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T11:13:53.662Z|00136|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T11:13:53.662Z|00137|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T11:13:53.675Z|00138|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T11:13:53.675Z|00139|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T11:13:53.675Z|00140|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T11:13:53.682Z|00141|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T11:13:53.682Z|00142|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T11:13:53.682Z|00143|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T11:13:53.693Z|00144|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T11:13:53.693Z|00145|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T11:13:53.693Z|00146|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T11:13:53.705Z|00147|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T11:13:53.705Z|00148|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T11:13:53.705Z|00149|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T11:13:53.711Z|00150|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T11:13:53.711Z|00151|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T11:13:53.711Z|00152|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T11:13:53.717Z|00153|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T11:13:53.717Z|00154|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T11:13:53.717Z|00155|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T11:13:53.728Z|00156|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T11:13:53.728Z|00157|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T11:13:53.728Z|00158|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T11:13:53.743Z|00159|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T11:13:53.743Z|00160|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T11:13:53.743Z|00161|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T11:13:53.749Z|00162|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T11:13:53.749Z|00163|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T11:13:53.749Z|00164|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T11:13:53.756Z|00165|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T11:13:53.756Z|00166|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T11:13:53.756Z|00167|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T11:13:53.766Z|00168|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T11:13:53.766Z|00169|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T11:13:53.766Z|00170|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T11:13:55.270Z|00171|memory|INFO|1406100 kB peak resident set size after 10.0 seconds 2018-06-11T11:13:55.270Z|00172|memory|INFO|handlers:118 ofconns:3 ports:8 revalidators:42 rules:36 udpif keys:1 2018-06-11T11:14:03.699Z|00173|connmgr|INFO|br-int<->tcp:127.0.0.1:6633: 5 flow_mods 10 s ago (5 adds) 2018-06-11T11:14:03.737Z|00174|connmgr|INFO|br-ex<->tcp:127.0.0.1:6633: 2 flow_mods 10 s ago (2 adds) 2018-06-11T11:14:03.773Z|00175|connmgr|INFO|br-tun<->tcp:127.0.0.1:6633: 12 flow_mods 10 s ago (12 adds) Best regards On 06/11/2018 12:57 PM, Raslan Darawsheh wrote: > What I really want to see is the init process for OVS-DPDK that dumps the DPDK initialization. > I think you will find it at the beginning of the log that you just sent. > > Kindest regards, > Raslan Darawsheh > > -----Original Message----- > From: Tobias Urdin [mailto:tobias.urdin@crystone.com] > Sent: Monday, June 11, 2018 1:00 PM > To: Raslan Darawsheh <rasland@mellanox.com> > Cc: users@dpdk.org > Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch > > Hello Raslan, > Thank you for your reply. > > For reference the br-mlx bridge was created like this: > $ovs-vsctl add-br br-mlx -- set bridge br-mlx datapath_type=netdev > > $ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 type=dpdk options:dpdk-devargs=mlx4_0 > ovs-vsctl: Error detected while setting up 'enp59s0d1': Error attaching device 'mlx4_0' to DPDK. See ovs-vswitchd log for details. > ovs-vsctl: The default log directory is "/var/log/openvswitch". > > In the ovs-vswitchd.log > 2018-06-11T09:56:20.613Z|00185|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T09:56:20.613Z|00186|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK > 2018-06-11T09:56:20.613Z|00187|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) > > Here is the complete log after restarting openvswitch-switch service: > 2018-06-11T09:57:49.380Z|00134|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.9.0 > 2018-06-11T09:57:52.812Z|00135|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T09:57:52.812Z|00136|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK > 2018-06-11T09:57:52.812Z|00137|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) > 2018-06-11T09:57:52.825Z|00138|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T09:57:52.825Z|00139|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK > 2018-06-11T09:57:52.825Z|00140|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) > 2018-06-11T09:57:52.832Z|00141|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T09:57:52.832Z|00142|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK > 2018-06-11T09:57:52.832Z|00143|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) > 2018-06-11T09:57:52.843Z|00144|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T09:57:52.843Z|00145|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK > 2018-06-11T09:57:52.843Z|00146|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) > 2018-06-11T09:57:52.856Z|00147|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T09:57:52.856Z|00148|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK > 2018-06-11T09:57:52.856Z|00149|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) > 2018-06-11T09:57:52.862Z|00150|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T09:57:52.862Z|00151|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK > 2018-06-11T09:57:52.862Z|00152|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) > 2018-06-11T09:57:52.870Z|00153|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T09:57:52.870Z|00154|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK > 2018-06-11T09:57:52.870Z|00155|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) > 2018-06-11T09:57:52.889Z|00156|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T09:57:52.889Z|00157|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK > 2018-06-11T09:57:52.889Z|00158|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) > 2018-06-11T09:57:52.903Z|00159|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T09:57:52.903Z|00160|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK > 2018-06-11T09:57:52.903Z|00161|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) > 2018-06-11T09:57:52.908Z|00162|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T09:57:52.908Z|00163|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK > 2018-06-11T09:57:52.908Z|00164|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) > 2018-06-11T09:57:52.914Z|00165|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T09:57:52.914Z|00166|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK > 2018-06-11T09:57:52.914Z|00167|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) > 2018-06-11T09:57:52.923Z|00168|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T09:57:52.923Z|00169|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK > 2018-06-11T09:57:52.923Z|00170|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) > 2018-06-11T09:57:54.808Z|00171|memory|INFO|1406100 kB peak resident set size after 10.1 seconds > 2018-06-11T09:57:54.808Z|00172|memory|INFO|handlers:118 ofconns:3 > ports:8 revalidators:42 rules:36 udpif keys:1 > 2018-06-11T09:58:02.850Z|00173|connmgr|INFO|br-int<->tcp:127.0.0.1:6633: > 5 flow_mods 10 s ago (5 adds) > 2018-06-11T09:58:02.898Z|00174|connmgr|INFO|br-ex<->tcp:127.0.0.1:6633: > 2 flow_mods 10 s ago (2 adds) > 2018-06-11T09:58:02.930Z|00175|connmgr|INFO|br-tun<->tcp:127.0.0.1:6633: > 12 flow_mods 10 s ago (12 adds) > > Was that what you mean by init log? > Best regards > > On 06/11/2018 11:51 AM, Raslan Darawsheh wrote: >> Hi Tobias, >> >> Can you try using the following ? >> >> ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 >> type=dpdk options:dpdk-devargs=mlx4_0 >> >> if it didn't work can you send the OVS-DPDK init log ? >> >> Kindest regards, >> Raslan Darawsheh >> >> -----Original Message----- >> From: users [mailto:users-bounces@dpdk.org] On Behalf Of Tobias Urdin >> Sent: Monday, June 11, 2018 12:15 PM >> To: users@dpdk.org >> Subject: [dpdk-users] mellanox connect-x 3 pro dpdk with openvswitch >> >> Hello, >> >> I'm stuck trying to test DPDK with a Mellanox Connect-X 3 Pro card on >> Ubuntu 16.04 >> >> I've enabled iommu, fixed hugepages and has installed and are running the openvswitch-switch-dpdk alternative. >> >> $lspci | grep -i mellanox >> 3b:00.0 Ethernet controller: Mellanox Technologies MT27520 Family >> [ConnectX-3 Pro] >> >> $lsmod | grep -i mlx >> mlx4_en 110592 0 >> vxlan 49152 2 i40e,mlx4_en >> mlx4_core 286720 1 mlx4_en >> ptp 20480 3 igb,i40e,mlx4_en >> >> $cat /etc/modprobe.d/mlx4_core.conf >> options mlx4_core log_num_mgm_entry_size=-1 >> >> >> $ovs-vsctl add-port br-mlx dpdk0 -- set Interface dpdk0 type=dpdk >> >> Bridge br-mlx >> Port "dpdk0" >> Interface "dpdk0" >> type: dpdk >> error: "'dpdk0' is missing 'options:dpdk-devargs'. The old 'dpdk<port_id>' names are not supported" >> >> >> $ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 >> type=dpdk options:dpdk-devargs=0000:3b:00.0,1 >> ovs-vsctl: Error detected while setting up 'enp59s0d1': Error attaching device '0000:3b:00.0,1' to DPDK. See ovs-vswitchd log for details. >> ovs-vsctl: The default log directory is "/var/log/openvswitch". >> >> 2018-06-11T09:10:38.696Z|00177|netdev_dpdk|WARN|Error attaching device >> '0000:3b:00.0,1' to DPDK >> 2018-06-11T09:10:38.696Z|00178|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> >> 2018-06-11T09:12:50.994Z|00181|dpdk|ERR|EAL: Driver cannot attach the >> device (0000:3b:00.0) >> 2018-06-11T09:12:50.994Z|00182|dpdk|ERR|EAL: No port found for device >> (0000:3b:00.0) 2018-06-11T09:12:50.994Z|00183|netdev_dpdk|WARN|Error >> attaching device '0000:3b:00.0,1' to DPDK >> 2018-06-11T09:12:50.994Z|00184|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> >> $ovs-vsctl add-port br-mlx dpdk0 -- set Interface dpdk0 type=dpdk options:dpdk-devargs="class=eth,mac=ec:0d:9a:b0:13:21" >> ovs-vsctl: Error detected while setting up 'dpdk0': Error attaching device 'class=eth,mac=ec:0d:9a:b0:13:21' to DPDK. See ovs-vswitchd log for details. >> ovs-vsctl: The default log directory is "/var/log/openvswitch". >> >> Port "dpdk0" >> Interface "dpdk0" >> type: dpdk >> options: {dpdk-devargs="class=eth,mac=ec:0d:9a:b0:13:21"} >> error: "Error attaching device 'class=eth,mac=ec:0d:9a:b0:13:21' to DPDK" >> >> >> Anybody know what I might have missed? I have tried restarting the openvswitch services after adding the interfaces as well but no difference. >> >> The logs doesn't say anything interesting from what I can see :( >> >> Best regards >> > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [dpdk-users] mellanox connect-x 3 pro dpdk with openvswitch 2018-06-11 11:15 ` Tobias Urdin @ 2018-06-11 11:19 ` Raslan Darawsheh 2018-06-11 11:34 ` Tobias Urdin 0 siblings, 1 reply; 14+ messages in thread From: Raslan Darawsheh @ 2018-06-11 11:19 UTC (permalink / raw) To: Tobias Urdin; +Cc: users >From what I'm seeing it looks like it didn't prop the MLX4 device at all. Can you make sure you have it enabled in your DPDK package when you compiled DPDK ? Kindest regards, Raslan Darawsheh -----Original Message----- From: Tobias Urdin [mailto:tobias.urdin@crystone.com] Sent: Monday, June 11, 2018 2:16 PM To: Raslan Darawsheh <rasland@mellanox.com> Cc: users@dpdk.org Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch Ok, here is the complete log when restarting the openvswitch-switch service, see below. Also here is the kernel for information: $uname -r 4.4.0-127-generic 2018-06-11T11:13:42.729Z|00176|bridge|INFO|bridge br-mlx: deleted interface br-mlx on port 65534 2018-06-11T11:13:43.274Z|00177|bridge|INFO|bridge br-int: deleted interface br-int on port 65534 2018-06-11T11:13:43.275Z|00178|bridge|INFO|bridge br-int: deleted interface int-br-ex on port 1 2018-06-11T11:13:43.275Z|00179|bridge|INFO|bridge br-int: deleted interface patch-tun on port 2 2018-06-11T11:13:43.762Z|00180|bridge|INFO|bridge br-tun: deleted interface patch-int on port 1 2018-06-11T11:13:43.762Z|00181|bridge|INFO|bridge br-tun: deleted interface br-tun on port 65534 2018-06-11T11:13:44.231Z|00182|bridge|INFO|bridge br-ex: deleted interface br-ex on port 65534 2018-06-11T11:13:44.232Z|00183|bridge|INFO|bridge br-ex: deleted interface phy-br-ex on port 1 2018-06-11T11:13:45.245Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log 2018-06-11T11:13:45.252Z|00002|ovs_numa|INFO|Discovered 40 CPU cores on NUMA node 0 2018-06-11T11:13:45.252Z|00003|ovs_numa|INFO|Discovered 40 CPU cores on NUMA node 1 2018-06-11T11:13:45.252Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and 80 CPU cores 2018-06-11T11:13:45.252Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connecting... 2018-06-11T11:13:45.252Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: connected 2018-06-11T11:13:45.256Z|00007|dpdk|INFO|Using DPDK 17.11.0 2018-06-11T11:13:45.256Z|00008|dpdk|INFO|DPDK Enabled - initializing... 2018-06-11T11:13:45.256Z|00009|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch 2018-06-11T11:13:45.256Z|00010|dpdk|INFO|IOMMU support for vhost-user-client disabled. 2018-06-11T11:13:45.256Z|00011|dpdk|INFO|EAL ARGS: ovs-vswitchd --socket-mem 1024,0 -c 0x00000001 2018-06-11T11:13:45.258Z|00012|dpdk|INFO|EAL: Detected 80 lcore(s) 2018-06-11T11:13:45.285Z|00013|dpdk|INFO|EAL: Probing VFIO support... 2018-06-11T11:13:45.285Z|00014|dpdk|INFO|EAL: VFIO support initialized 2018-06-11T11:13:48.214Z|00015|dpdk|INFO|EAL: PCI device 0000:01:00.0 on NUMA socket 0 2018-06-11T11:13:48.214Z|00016|dpdk|INFO|EAL: probe driver: 8086:1521 net_e1000_igb 2018-06-11T11:13:48.214Z|00017|dpdk|INFO|EAL: PCI device 0000:01:00.1 on NUMA socket 0 2018-06-11T11:13:48.214Z|00018|dpdk|INFO|EAL: probe driver: 8086:1521 net_e1000_igb 2018-06-11T11:13:48.214Z|00019|dpdk|INFO|EAL: PCI device 0000:18:00.0 on NUMA socket 0 2018-06-11T11:13:48.214Z|00020|dpdk|INFO|EAL: probe driver: 8086:1572 net_i40e 2018-06-11T11:13:48.214Z|00021|dpdk|INFO|EAL: PCI device 0000:18:00.1 on NUMA socket 0 2018-06-11T11:13:48.214Z|00022|dpdk|INFO|EAL: probe driver: 8086:1572 net_i40e 2018-06-11T11:13:48.214Z|00023|dpdk|INFO|DPDK pdump packet capture enabled 2018-06-11T11:13:48.224Z|00024|dpdk|INFO|DPDK Enabled - initialized 2018-06-11T11:13:48.235Z|00025|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports recirculation 2018-06-11T11:13:48.235Z|00026|ofproto_dpif|INFO|netdev@ovs-netdev: VLAN header stack length probed as 1 2018-06-11T11:13:48.235Z|00027|ofproto_dpif|INFO|netdev@ovs-netdev: MPLS label stack length probed as 3 2018-06-11T11:13:48.235Z|00028|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports truncate action 2018-06-11T11:13:48.235Z|00029|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports unique flow ids 2018-06-11T11:13:48.235Z|00030|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports clone action 2018-06-11T11:13:48.235Z|00031|ofproto_dpif|INFO|netdev@ovs-netdev: Max sample nesting level probed as 10 2018-06-11T11:13:48.235Z|00032|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports eventmask in conntrack action 2018-06-11T11:13:48.235Z|00033|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_clear action 2018-06-11T11:13:48.235Z|00034|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_state 2018-06-11T11:13:48.235Z|00035|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_zone 2018-06-11T11:13:48.235Z|00036|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_mark 2018-06-11T11:13:48.235Z|00037|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_label 2018-06-11T11:13:48.235Z|00038|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_state_nat 2018-06-11T11:13:48.235Z|00039|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_orig_tuple 2018-06-11T11:13:48.235Z|00040|ofproto_dpif|INFO|netdev@ovs-netdev: Datapath supports ct_orig_tuple6 2018-06-11T11:13:48.605Z|00041|ofproto_dpif|INFO|system@ovs-system: Datapath supports recirculation 2018-06-11T11:13:48.605Z|00042|ofproto_dpif|INFO|system@ovs-system: VLAN header stack length probed as 1 2018-06-11T11:13:48.605Z|00043|ofproto_dpif|INFO|system@ovs-system: MPLS label stack length probed as 1 2018-06-11T11:13:48.605Z|00044|ofproto_dpif|INFO|system@ovs-system: Datapath does not support truncate action 2018-06-11T11:13:48.605Z|00045|ofproto_dpif|INFO|system@ovs-system: Datapath supports unique flow ids 2018-06-11T11:13:48.605Z|00046|ofproto_dpif|INFO|system@ovs-system: Datapath does not support clone action 2018-06-11T11:13:48.605Z|00047|ofproto_dpif|INFO|system@ovs-system: Max sample nesting level probed as 3 2018-06-11T11:13:48.605Z|00048|ofproto_dpif|INFO|system@ovs-system: Datapath does not support eventmask in conntrack action 2018-06-11T11:13:48.605Z|00049|ofproto_dpif|INFO|system@ovs-system: Datapath does not support ct_clear action 2018-06-11T11:13:48.605Z|00050|ofproto_dpif|INFO|system@ovs-system: Datapath supports ct_state 2018-06-11T11:13:48.605Z|00051|ofproto_dpif|INFO|system@ovs-system: Datapath supports ct_zone 2018-06-11T11:13:48.605Z|00052|ofproto_dpif|INFO|system@ovs-system: Datapath supports ct_mark 2018-06-11T11:13:48.605Z|00053|ofproto_dpif|INFO|system@ovs-system: Datapath supports ct_label 2018-06-11T11:13:48.605Z|00054|ofproto_dpif|INFO|system@ovs-system: Datapath does not support ct_state_nat 2018-06-11T11:13:48.605Z|00055|ofproto_dpif|INFO|system@ovs-system: Datapath does not support ct_orig_tuple 2018-06-11T11:13:48.605Z|00056|ofproto_dpif|INFO|system@ovs-system: Datapath does not support ct_orig_tuple6 2018-06-11T11:13:48.981Z|00057|bridge|INFO|bridge br-mlx: added interface br-mlx on port 65534 2018-06-11T11:13:48.981Z|00058|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T11:13:48.981Z|00059|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T11:13:48.981Z|00060|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T11:13:48.982Z|00061|bridge|INFO|bridge br-int: added interface br-int on port 65534 2018-06-11T11:13:48.982Z|00062|bridge|INFO|bridge br-int: added interface int-br-ex on port 1 2018-06-11T11:13:48.982Z|00063|bridge|INFO|bridge br-int: added interface patch-tun on port 2 2018-06-11T11:13:48.982Z|00064|bridge|INFO|bridge br-tun: added interface patch-int on port 1 2018-06-11T11:13:48.982Z|00065|bridge|INFO|bridge br-tun: added interface br-tun on port 65534 2018-06-11T11:13:48.982Z|00066|bridge|INFO|bridge br-ex: added interface br-ex on port 65534 2018-06-11T11:13:48.982Z|00067|bridge|INFO|bridge br-ex: added interface phy-br-ex on port 1 2018-06-11T11:13:48.982Z|00068|bridge|INFO|bridge br-mlx: using datapath ID 000072528484a642 2018-06-11T11:13:48.982Z|00069|connmgr|INFO|br-mlx: added service controller "punix:/var/run/openvswitch/br-mlx.mgmt" 2018-06-11T11:13:49.221Z|00070|bridge|INFO|bridge br-int: using datapath ID 00006680927b6b40 2018-06-11T11:13:49.221Z|00071|connmgr|INFO|br-int: added service controller "punix:/var/run/openvswitch/br-int.mgmt" 2018-06-11T11:13:49.221Z|00072|connmgr|INFO|br-int: added primary controller "tcp:127.0.0.1:6633" 2018-06-11T11:13:49.221Z|00073|rconn|INFO|br-int<->tcp:127.0.0.1:6633: connecting... 2018-06-11T11:13:49.460Z|00074|bridge|INFO|bridge br-tun: using datapath ID 000066532e5f0443 2018-06-11T11:13:49.461Z|00075|connmgr|INFO|br-tun: added service controller "punix:/var/run/openvswitch/br-tun.mgmt" 2018-06-11T11:13:49.461Z|00076|connmgr|INFO|br-tun: added primary controller "tcp:127.0.0.1:6633" 2018-06-11T11:13:49.461Z|00077|rconn|INFO|br-tun<->tcp:127.0.0.1:6633: connecting... 2018-06-11T11:13:49.692Z|00078|bridge|INFO|bridge br-ex: using datapath ID 0000e2d1bdbe8e45 2018-06-11T11:13:49.692Z|00079|connmgr|INFO|br-ex: added service controller "punix:/var/run/openvswitch/br-ex.mgmt" 2018-06-11T11:13:49.692Z|00080|connmgr|INFO|br-ex: added primary controller "tcp:127.0.0.1:6633" 2018-06-11T11:13:49.692Z|00081|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: connecting... 2018-06-11T11:13:49.916Z|00082|timeval|WARN|Unreasonably long 4661ms poll interval (16ms user, 4472ms system) 2018-06-11T11:13:49.916Z|00083|timeval|WARN|faults: 3584706 minor, 0 major 2018-06-11T11:13:49.916Z|00084|timeval|WARN|disk: 0 reads, 16 writes 2018-06-11T11:13:49.916Z|00085|timeval|WARN|context switches: 147 voluntary, 25 involuntary 2018-06-11T11:13:49.916Z|00086|coverage|INFO|Event coverage, avg rate over last: 5 seconds, last minute, last hour, hash=2ba35161: 2018-06-11T11:13:49.916Z|00087|coverage|INFO|bridge_reconfigure 0.0/sec 0.000/sec 0.0000/sec total: 1 2018-06-11T11:13:49.916Z|00088|coverage|INFO|ofproto_flush 0.0/sec 0.000/sec 0.0000/sec total: 4 2018-06-11T11:13:49.916Z|00089|coverage|INFO|ofproto_update_port 0.0/sec 0.000/sec 0.0000/sec total: 8 2018-06-11T11:13:49.916Z|00090|coverage|INFO|rev_flow_table 0.0/sec 0.000/sec 0.0000/sec total: 1 2018-06-11T11:13:49.916Z|00091|coverage|INFO|cmap_expand 0.0/sec 0.000/sec 0.0000/sec total: 52 2018-06-11T11:13:49.916Z|00092|coverage|INFO|cmap_shrink 0.0/sec 0.000/sec 0.0000/sec total: 23 2018-06-11T11:13:49.916Z|00093|coverage|INFO|dpif_port_add 0.0/sec 0.000/sec 0.0000/sec total: 1 2018-06-11T11:13:49.916Z|00094|coverage|INFO|dpif_flow_flush 0.0/sec 0.000/sec 0.0000/sec total: 6 2018-06-11T11:13:49.916Z|00095|coverage|INFO|dpif_flow_get 0.0/sec 0.000/sec 0.0000/sec total: 29 2018-06-11T11:13:49.916Z|00096|coverage|INFO|dpif_flow_put 0.0/sec 0.000/sec 0.0000/sec total: 37 2018-06-11T11:13:49.916Z|00097|coverage|INFO|dpif_flow_del 0.0/sec 0.000/sec 0.0000/sec total: 29 2018-06-11T11:13:49.916Z|00098|coverage|INFO|dpif_execute 0.0/sec 0.000/sec 0.0000/sec total: 8 2018-06-11T11:13:49.916Z|00099|coverage|INFO|flow_extract 0.0/sec 0.000/sec 0.0000/sec total: 6 2018-06-11T11:13:49.916Z|00100|coverage|INFO|miniflow_malloc 0.0/sec 0.000/sec 0.0000/sec total: 34 2018-06-11T11:13:49.916Z|00101|coverage|INFO|hmap_expand 0.0/sec 0.000/sec 0.0000/sec total: 498 2018-06-11T11:13:49.916Z|00102|coverage|INFO|netdev_received 0.0/sec 0.000/sec 0.0000/sec total: 1 2018-06-11T11:13:49.916Z|00103|coverage|INFO|netdev_get_stats 0.0/sec 0.000/sec 0.0000/sec total: 8 2018-06-11T11:13:49.916Z|00104|coverage|INFO|poll_create_node 0.0/sec 0.000/sec 0.0000/sec total: 23 2018-06-11T11:13:49.916Z|00105|coverage|INFO|seq_change 0.0/sec 0.000/sec 0.0000/sec total: 2193 2018-06-11T11:13:49.916Z|00106|coverage|INFO|pstream_open 0.0/sec 0.000/sec 0.0000/sec total: 9 2018-06-11T11:13:49.916Z|00107|coverage|INFO|stream_open 0.0/sec 0.000/sec 0.0000/sec total: 4 2018-06-11T11:13:49.916Z|00108|coverage|INFO|util_xalloc 0.0/sec 0.000/sec 0.0000/sec total: 12803 2018-06-11T11:13:49.916Z|00109|coverage|INFO|vconn_open 0.0/sec 0.000/sec 0.0000/sec total: 3 2018-06-11T11:13:49.916Z|00110|coverage|INFO|netdev_set_policing 0.0/sec 0.000/sec 0.0000/sec total: 4 2018-06-11T11:13:49.916Z|00111|coverage|INFO|netdev_get_ifindex 0.0/sec 0.000/sec 0.0000/sec total: 4 2018-06-11T11:13:49.916Z|00112|coverage|INFO|netdev_get_hwaddr 0.0/sec 0.000/sec 0.0000/sec total: 9 2018-06-11T11:13:49.916Z|00113|coverage|INFO|netdev_get_ethtool 0.0/sec 0.000/sec 0.0000/sec total: 2 2018-06-11T11:13:49.916Z|00114|coverage|INFO|netlink_received 0.0/sec 0.000/sec 0.0000/sec total: 85 2018-06-11T11:13:49.916Z|00115|coverage|INFO|netlink_recv_jumbo 0.0/sec 0.000/sec 0.0000/sec total: 1 2018-06-11T11:13:49.916Z|00116|coverage|INFO|netlink_sent 0.0/sec 0.000/sec 0.0000/sec total: 102 2018-06-11T11:13:49.916Z|00117|coverage|INFO|74 events never hit 2018-06-11T11:13:49.916Z|00118|poll_loop|INFO|wakeup due to [POLLIN] on fd 433 (FIFO pipe:[1025715]) at ../lib/ovs-rcu.c:229 (97% CPU usage) 2018-06-11T11:13:49.917Z|00119|poll_loop|INFO|wakeup due to [POLLIN] on fd 433 (FIFO pipe:[1025715]) at ../lib/ovs-rcu.c:229 (97% CPU usage) 2018-06-11T11:13:49.917Z|00120|poll_loop|INFO|wakeup due to 0-ms timeout at ../lib/ovs-rcu.c:229 (97% CPU usage) 2018-06-11T11:13:49.917Z|00121|poll_loop|INFO|wakeup due to 0-ms timeout at ../lib/ovs-rcu.c:229 (97% CPU usage) 2018-06-11T11:13:49.920Z|00122|rconn|INFO|br-int<->tcp:127.0.0.1:6633: connected 2018-06-11T11:13:49.920Z|00123|rconn|INFO|br-tun<->tcp:127.0.0.1:6633: connected 2018-06-11T11:13:49.920Z|00124|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: connected 2018-06-11T11:13:49.926Z|00125|poll_loop|INFO|wakeup due to [POLLIN] on fd 10 (NETLINK_ROUTE<->NETLINK_ROUTE) at ../lib/netlink-socket.c:1331 (97% CPU usage) 2018-06-11T11:13:49.926Z|00126|poll_loop|INFO|wakeup due to [POLLIN] on fd 421 (character device /dev/net/tun) at ../lib/netdev-linux.c:1171 (97% CPU usage) 2018-06-11T11:13:49.926Z|00127|poll_loop|INFO|wakeup due to [POLLIN] on fd 423 (127.0.0.1:33712<->127.0.0.1:6633) at ../lib/stream-fd.c:157 (97% CPU usage) 2018-06-11T11:13:49.926Z|00128|poll_loop|INFO|wakeup due to [POLLIN] on fd 428 (127.0.0.1:33714<->127.0.0.1:6633) at ../lib/stream-fd.c:157 (97% CPU usage) 2018-06-11T11:13:49.926Z|00129|poll_loop|INFO|wakeup due to [POLLIN] on fd 11 (<->/var/run/openvswitch/db.sock) at ../lib/stream-fd.c:157 (97% CPU usage) 2018-06-11T11:13:49.926Z|00130|poll_loop|INFO|wakeup due to [POLLIN] on fd 429 (127.0.0.1:33716<->127.0.0.1:6633) at ../lib/stream-fd.c:157 (97% CPU usage) 2018-06-11T11:13:49.930Z|00131|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T11:13:49.930Z|00132|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T11:13:49.930Z|00133|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T11:13:49.931Z|00134|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.9.0 2018-06-11T11:13:53.662Z|00135|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T11:13:53.662Z|00136|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T11:13:53.662Z|00137|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T11:13:53.675Z|00138|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T11:13:53.675Z|00139|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T11:13:53.675Z|00140|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T11:13:53.682Z|00141|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T11:13:53.682Z|00142|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T11:13:53.682Z|00143|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T11:13:53.693Z|00144|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T11:13:53.693Z|00145|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T11:13:53.693Z|00146|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T11:13:53.705Z|00147|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T11:13:53.705Z|00148|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T11:13:53.705Z|00149|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T11:13:53.711Z|00150|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T11:13:53.711Z|00151|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T11:13:53.711Z|00152|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T11:13:53.717Z|00153|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T11:13:53.717Z|00154|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T11:13:53.717Z|00155|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T11:13:53.728Z|00156|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T11:13:53.728Z|00157|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T11:13:53.728Z|00158|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T11:13:53.743Z|00159|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T11:13:53.743Z|00160|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T11:13:53.743Z|00161|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T11:13:53.749Z|00162|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T11:13:53.749Z|00163|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T11:13:53.749Z|00164|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T11:13:53.756Z|00165|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T11:13:53.756Z|00166|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T11:13:53.756Z|00167|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T11:13:53.766Z|00168|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' 2018-06-11T11:13:53.766Z|00169|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK 2018-06-11T11:13:53.766Z|00170|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T11:13:55.270Z|00171|memory|INFO|1406100 kB peak resident set size after 10.0 seconds 2018-06-11T11:13:55.270Z|00172|memory|INFO|handlers:118 ofconns:3 ports:8 revalidators:42 rules:36 udpif keys:1 2018-06-11T11:14:03.699Z|00173|connmgr|INFO|br-int<->tcp:127.0.0.1:6633: 5 flow_mods 10 s ago (5 adds) 2018-06-11T11:14:03.737Z|00174|connmgr|INFO|br-ex<->tcp:127.0.0.1:6633: 2 flow_mods 10 s ago (2 adds) 2018-06-11T11:14:03.773Z|00175|connmgr|INFO|br-tun<->tcp:127.0.0.1:6633: 12 flow_mods 10 s ago (12 adds) Best regards On 06/11/2018 12:57 PM, Raslan Darawsheh wrote: > What I really want to see is the init process for OVS-DPDK that dumps the DPDK initialization. > I think you will find it at the beginning of the log that you just sent. > > Kindest regards, > Raslan Darawsheh > > -----Original Message----- > From: Tobias Urdin [mailto:tobias.urdin@crystone.com] > Sent: Monday, June 11, 2018 1:00 PM > To: Raslan Darawsheh <rasland@mellanox.com> > Cc: users@dpdk.org > Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch > > Hello Raslan, > Thank you for your reply. > > For reference the br-mlx bridge was created like this: > $ovs-vsctl add-br br-mlx -- set bridge br-mlx datapath_type=netdev > > $ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 > type=dpdk options:dpdk-devargs=mlx4_0 > ovs-vsctl: Error detected while setting up 'enp59s0d1': Error attaching device 'mlx4_0' to DPDK. See ovs-vswitchd log for details. > ovs-vsctl: The default log directory is "/var/log/openvswitch". > > In the ovs-vswitchd.log > 2018-06-11T09:56:20.613Z|00185|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T09:56:20.613Z|00186|netdev_dpdk|WARN|Error attaching device > 'mlx4_0' to DPDK > 2018-06-11T09:56:20.613Z|00187|netdev|WARN|enp59s0d1: could not set > configuration (Invalid argument) > > Here is the complete log after restarting openvswitch-switch service: > 2018-06-11T09:57:49.380Z|00134|bridge|INFO|ovs-vswitchd (Open vSwitch) > 2.9.0 > 2018-06-11T09:57:52.812Z|00135|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T09:57:52.812Z|00136|netdev_dpdk|WARN|Error attaching device > 'mlx4_0' to DPDK > 2018-06-11T09:57:52.812Z|00137|netdev|WARN|enp59s0d1: could not set > configuration (Invalid argument) > 2018-06-11T09:57:52.825Z|00138|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T09:57:52.825Z|00139|netdev_dpdk|WARN|Error attaching device > 'mlx4_0' to DPDK > 2018-06-11T09:57:52.825Z|00140|netdev|WARN|enp59s0d1: could not set > configuration (Invalid argument) > 2018-06-11T09:57:52.832Z|00141|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T09:57:52.832Z|00142|netdev_dpdk|WARN|Error attaching device > 'mlx4_0' to DPDK > 2018-06-11T09:57:52.832Z|00143|netdev|WARN|enp59s0d1: could not set > configuration (Invalid argument) > 2018-06-11T09:57:52.843Z|00144|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T09:57:52.843Z|00145|netdev_dpdk|WARN|Error attaching device > 'mlx4_0' to DPDK > 2018-06-11T09:57:52.843Z|00146|netdev|WARN|enp59s0d1: could not set > configuration (Invalid argument) > 2018-06-11T09:57:52.856Z|00147|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T09:57:52.856Z|00148|netdev_dpdk|WARN|Error attaching device > 'mlx4_0' to DPDK > 2018-06-11T09:57:52.856Z|00149|netdev|WARN|enp59s0d1: could not set > configuration (Invalid argument) > 2018-06-11T09:57:52.862Z|00150|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T09:57:52.862Z|00151|netdev_dpdk|WARN|Error attaching device > 'mlx4_0' to DPDK > 2018-06-11T09:57:52.862Z|00152|netdev|WARN|enp59s0d1: could not set > configuration (Invalid argument) > 2018-06-11T09:57:52.870Z|00153|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T09:57:52.870Z|00154|netdev_dpdk|WARN|Error attaching device > 'mlx4_0' to DPDK > 2018-06-11T09:57:52.870Z|00155|netdev|WARN|enp59s0d1: could not set > configuration (Invalid argument) > 2018-06-11T09:57:52.889Z|00156|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T09:57:52.889Z|00157|netdev_dpdk|WARN|Error attaching device > 'mlx4_0' to DPDK > 2018-06-11T09:57:52.889Z|00158|netdev|WARN|enp59s0d1: could not set > configuration (Invalid argument) > 2018-06-11T09:57:52.903Z|00159|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T09:57:52.903Z|00160|netdev_dpdk|WARN|Error attaching device > 'mlx4_0' to DPDK > 2018-06-11T09:57:52.903Z|00161|netdev|WARN|enp59s0d1: could not set > configuration (Invalid argument) > 2018-06-11T09:57:52.908Z|00162|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T09:57:52.908Z|00163|netdev_dpdk|WARN|Error attaching device > 'mlx4_0' to DPDK > 2018-06-11T09:57:52.908Z|00164|netdev|WARN|enp59s0d1: could not set > configuration (Invalid argument) > 2018-06-11T09:57:52.914Z|00165|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T09:57:52.914Z|00166|netdev_dpdk|WARN|Error attaching device > 'mlx4_0' to DPDK > 2018-06-11T09:57:52.914Z|00167|netdev|WARN|enp59s0d1: could not set > configuration (Invalid argument) > 2018-06-11T09:57:52.923Z|00168|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T09:57:52.923Z|00169|netdev_dpdk|WARN|Error attaching device > 'mlx4_0' to DPDK > 2018-06-11T09:57:52.923Z|00170|netdev|WARN|enp59s0d1: could not set > configuration (Invalid argument) > 2018-06-11T09:57:54.808Z|00171|memory|INFO|1406100 kB peak resident > set size after 10.1 seconds > 2018-06-11T09:57:54.808Z|00172|memory|INFO|handlers:118 ofconns:3 > ports:8 revalidators:42 rules:36 udpif keys:1 > 2018-06-11T09:58:02.850Z|00173|connmgr|INFO|br-int<->tcp:127.0.0.1:6633: > 5 flow_mods 10 s ago (5 adds) > 2018-06-11T09:58:02.898Z|00174|connmgr|INFO|br-ex<->tcp:127.0.0.1:6633: > 2 flow_mods 10 s ago (2 adds) > 2018-06-11T09:58:02.930Z|00175|connmgr|INFO|br-tun<->tcp:127.0.0.1:6633: > 12 flow_mods 10 s ago (12 adds) > > Was that what you mean by init log? > Best regards > > On 06/11/2018 11:51 AM, Raslan Darawsheh wrote: >> Hi Tobias, >> >> Can you try using the following ? >> >> ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 >> type=dpdk options:dpdk-devargs=mlx4_0 >> >> if it didn't work can you send the OVS-DPDK init log ? >> >> Kindest regards, >> Raslan Darawsheh >> >> -----Original Message----- >> From: users [mailto:users-bounces@dpdk.org] On Behalf Of Tobias Urdin >> Sent: Monday, June 11, 2018 12:15 PM >> To: users@dpdk.org >> Subject: [dpdk-users] mellanox connect-x 3 pro dpdk with openvswitch >> >> Hello, >> >> I'm stuck trying to test DPDK with a Mellanox Connect-X 3 Pro card on >> Ubuntu 16.04 >> >> I've enabled iommu, fixed hugepages and has installed and are running the openvswitch-switch-dpdk alternative. >> >> $lspci | grep -i mellanox >> 3b:00.0 Ethernet controller: Mellanox Technologies MT27520 Family >> [ConnectX-3 Pro] >> >> $lsmod | grep -i mlx >> mlx4_en 110592 0 >> vxlan 49152 2 i40e,mlx4_en >> mlx4_core 286720 1 mlx4_en >> ptp 20480 3 igb,i40e,mlx4_en >> >> $cat /etc/modprobe.d/mlx4_core.conf >> options mlx4_core log_num_mgm_entry_size=-1 >> >> >> $ovs-vsctl add-port br-mlx dpdk0 -- set Interface dpdk0 type=dpdk >> >> Bridge br-mlx >> Port "dpdk0" >> Interface "dpdk0" >> type: dpdk >> error: "'dpdk0' is missing 'options:dpdk-devargs'. The old 'dpdk<port_id>' names are not supported" >> >> >> $ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 >> type=dpdk options:dpdk-devargs=0000:3b:00.0,1 >> ovs-vsctl: Error detected while setting up 'enp59s0d1': Error attaching device '0000:3b:00.0,1' to DPDK. See ovs-vswitchd log for details. >> ovs-vsctl: The default log directory is "/var/log/openvswitch". >> >> 2018-06-11T09:10:38.696Z|00177|netdev_dpdk|WARN|Error attaching >> device '0000:3b:00.0,1' to DPDK >> 2018-06-11T09:10:38.696Z|00178|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> >> 2018-06-11T09:12:50.994Z|00181|dpdk|ERR|EAL: Driver cannot attach the >> device (0000:3b:00.0) >> 2018-06-11T09:12:50.994Z|00182|dpdk|ERR|EAL: No port found for device >> (0000:3b:00.0) 2018-06-11T09:12:50.994Z|00183|netdev_dpdk|WARN|Error >> attaching device '0000:3b:00.0,1' to DPDK >> 2018-06-11T09:12:50.994Z|00184|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> >> $ovs-vsctl add-port br-mlx dpdk0 -- set Interface dpdk0 type=dpdk options:dpdk-devargs="class=eth,mac=ec:0d:9a:b0:13:21" >> ovs-vsctl: Error detected while setting up 'dpdk0': Error attaching device 'class=eth,mac=ec:0d:9a:b0:13:21' to DPDK. See ovs-vswitchd log for details. >> ovs-vsctl: The default log directory is "/var/log/openvswitch". >> >> Port "dpdk0" >> Interface "dpdk0" >> type: dpdk >> options: {dpdk-devargs="class=eth,mac=ec:0d:9a:b0:13:21"} >> error: "Error attaching device 'class=eth,mac=ec:0d:9a:b0:13:21' to DPDK" >> >> >> Anybody know what I might have missed? I have tried restarting the openvswitch services after adding the interfaces as well but no difference. >> >> The logs doesn't say anything interesting from what I can see :( >> >> Best regards >> > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [dpdk-users] mellanox connect-x 3 pro dpdk with openvswitch 2018-06-11 11:19 ` Raslan Darawsheh @ 2018-06-11 11:34 ` Tobias Urdin 2018-06-11 11:37 ` Raslan Darawsheh 0 siblings, 1 reply; 14+ messages in thread From: Tobias Urdin @ 2018-06-11 11:34 UTC (permalink / raw) To: Raslan Darawsheh; +Cc: users I haven't compiled any DPDK or Mellanox related modules/packages. What I have is ovs, ovs-dpdk, mlx4 module, I have also loaded the ib_uverbs driver. Is it the librte_pmd_mlx4 I have to compile manually? Do you know any updated documentation about this, there isn't much relevant and searching for documentation and the correct mlnx driver and code is a struggle. Best regards On 06/11/2018 01:20 PM, Raslan Darawsheh wrote: > From what I'm seeing it looks like it didn't prop the MLX4 device at all. > Can you make sure you have it enabled in your DPDK package when you compiled DPDK ? > > Kindest regards, > Raslan Darawsheh > > -----Original Message----- > From: Tobias Urdin [mailto:tobias.urdin@crystone.com] > Sent: Monday, June 11, 2018 2:16 PM > To: Raslan Darawsheh <rasland@mellanox.com> > Cc: users@dpdk.org > Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch > > Ok, here is the complete log when restarting the openvswitch-switch service, see below. > > Also here is the kernel for information: > $uname -r > 4.4.0-127-generic > > 2018-06-11T11:13:42.729Z|00176|bridge|INFO|bridge br-mlx: deleted interface br-mlx on port 65534 2018-06-11T11:13:43.274Z|00177|bridge|INFO|bridge br-int: deleted interface br-int on port 65534 2018-06-11T11:13:43.275Z|00178|bridge|INFO|bridge br-int: deleted interface int-br-ex on port 1 2018-06-11T11:13:43.275Z|00179|bridge|INFO|bridge br-int: deleted interface patch-tun on port 2 2018-06-11T11:13:43.762Z|00180|bridge|INFO|bridge br-tun: deleted interface patch-int on port 1 2018-06-11T11:13:43.762Z|00181|bridge|INFO|bridge br-tun: deleted interface br-tun on port 65534 2018-06-11T11:13:44.231Z|00182|bridge|INFO|bridge br-ex: deleted interface br-ex on port 65534 2018-06-11T11:13:44.232Z|00183|bridge|INFO|bridge br-ex: deleted interface phy-br-ex on port 1 2018-06-11T11:13:45.245Z|00001|vlog|INFO|opened log file /var/log/openvswitch/ovs-vswitchd.log > 2018-06-11T11:13:45.252Z|00002|ovs_numa|INFO|Discovered 40 CPU cores on NUMA node 0 2018-06-11T11:13:45.252Z|00003|ovs_numa|INFO|Discovered 40 CPU cores on NUMA node 1 2018-06-11T11:13:45.252Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes and > 80 CPU cores > 2018-06-11T11:13:45.252Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: > connecting... > 2018-06-11T11:13:45.252Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: > connected > 2018-06-11T11:13:45.256Z|00007|dpdk|INFO|Using DPDK 17.11.0 2018-06-11T11:13:45.256Z|00008|dpdk|INFO|DPDK Enabled - initializing... > 2018-06-11T11:13:45.256Z|00009|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch 2018-06-11T11:13:45.256Z|00010|dpdk|INFO|IOMMU support for vhost-user-client disabled. > 2018-06-11T11:13:45.256Z|00011|dpdk|INFO|EAL ARGS: ovs-vswitchd --socket-mem 1024,0 -c 0x00000001 > 2018-06-11T11:13:45.258Z|00012|dpdk|INFO|EAL: Detected 80 lcore(s) > 2018-06-11T11:13:45.285Z|00013|dpdk|INFO|EAL: Probing VFIO support... > 2018-06-11T11:13:45.285Z|00014|dpdk|INFO|EAL: VFIO support initialized > 2018-06-11T11:13:48.214Z|00015|dpdk|INFO|EAL: PCI device 0000:01:00.0 on NUMA socket 0 > 2018-06-11T11:13:48.214Z|00016|dpdk|INFO|EAL: probe driver: 8086:1521 > net_e1000_igb > 2018-06-11T11:13:48.214Z|00017|dpdk|INFO|EAL: PCI device 0000:01:00.1 on NUMA socket 0 > 2018-06-11T11:13:48.214Z|00018|dpdk|INFO|EAL: probe driver: 8086:1521 > net_e1000_igb > 2018-06-11T11:13:48.214Z|00019|dpdk|INFO|EAL: PCI device 0000:18:00.0 on NUMA socket 0 > 2018-06-11T11:13:48.214Z|00020|dpdk|INFO|EAL: probe driver: 8086:1572 > net_i40e > 2018-06-11T11:13:48.214Z|00021|dpdk|INFO|EAL: PCI device 0000:18:00.1 on NUMA socket 0 > 2018-06-11T11:13:48.214Z|00022|dpdk|INFO|EAL: probe driver: 8086:1572 > net_i40e > 2018-06-11T11:13:48.214Z|00023|dpdk|INFO|DPDK pdump packet capture enabled 2018-06-11T11:13:48.224Z|00024|dpdk|INFO|DPDK Enabled - initialized > 2018-06-11T11:13:48.235Z|00025|ofproto_dpif|INFO|netdev@ovs-netdev: > Datapath supports recirculation > 2018-06-11T11:13:48.235Z|00026|ofproto_dpif|INFO|netdev@ovs-netdev: VLAN header stack length probed as 1 > 2018-06-11T11:13:48.235Z|00027|ofproto_dpif|INFO|netdev@ovs-netdev: MPLS label stack length probed as 3 > 2018-06-11T11:13:48.235Z|00028|ofproto_dpif|INFO|netdev@ovs-netdev: > Datapath supports truncate action > 2018-06-11T11:13:48.235Z|00029|ofproto_dpif|INFO|netdev@ovs-netdev: > Datapath supports unique flow ids > 2018-06-11T11:13:48.235Z|00030|ofproto_dpif|INFO|netdev@ovs-netdev: > Datapath supports clone action > 2018-06-11T11:13:48.235Z|00031|ofproto_dpif|INFO|netdev@ovs-netdev: Max sample nesting level probed as 10 > 2018-06-11T11:13:48.235Z|00032|ofproto_dpif|INFO|netdev@ovs-netdev: > Datapath supports eventmask in conntrack action > 2018-06-11T11:13:48.235Z|00033|ofproto_dpif|INFO|netdev@ovs-netdev: > Datapath supports ct_clear action > 2018-06-11T11:13:48.235Z|00034|ofproto_dpif|INFO|netdev@ovs-netdev: > Datapath supports ct_state > 2018-06-11T11:13:48.235Z|00035|ofproto_dpif|INFO|netdev@ovs-netdev: > Datapath supports ct_zone > 2018-06-11T11:13:48.235Z|00036|ofproto_dpif|INFO|netdev@ovs-netdev: > Datapath supports ct_mark > 2018-06-11T11:13:48.235Z|00037|ofproto_dpif|INFO|netdev@ovs-netdev: > Datapath supports ct_label > 2018-06-11T11:13:48.235Z|00038|ofproto_dpif|INFO|netdev@ovs-netdev: > Datapath supports ct_state_nat > 2018-06-11T11:13:48.235Z|00039|ofproto_dpif|INFO|netdev@ovs-netdev: > Datapath supports ct_orig_tuple > 2018-06-11T11:13:48.235Z|00040|ofproto_dpif|INFO|netdev@ovs-netdev: > Datapath supports ct_orig_tuple6 > 2018-06-11T11:13:48.605Z|00041|ofproto_dpif|INFO|system@ovs-system: > Datapath supports recirculation > 2018-06-11T11:13:48.605Z|00042|ofproto_dpif|INFO|system@ovs-system: VLAN header stack length probed as 1 > 2018-06-11T11:13:48.605Z|00043|ofproto_dpif|INFO|system@ovs-system: MPLS label stack length probed as 1 > 2018-06-11T11:13:48.605Z|00044|ofproto_dpif|INFO|system@ovs-system: > Datapath does not support truncate action > 2018-06-11T11:13:48.605Z|00045|ofproto_dpif|INFO|system@ovs-system: > Datapath supports unique flow ids > 2018-06-11T11:13:48.605Z|00046|ofproto_dpif|INFO|system@ovs-system: > Datapath does not support clone action > 2018-06-11T11:13:48.605Z|00047|ofproto_dpif|INFO|system@ovs-system: Max sample nesting level probed as 3 > 2018-06-11T11:13:48.605Z|00048|ofproto_dpif|INFO|system@ovs-system: > Datapath does not support eventmask in conntrack action > 2018-06-11T11:13:48.605Z|00049|ofproto_dpif|INFO|system@ovs-system: > Datapath does not support ct_clear action > 2018-06-11T11:13:48.605Z|00050|ofproto_dpif|INFO|system@ovs-system: > Datapath supports ct_state > 2018-06-11T11:13:48.605Z|00051|ofproto_dpif|INFO|system@ovs-system: > Datapath supports ct_zone > 2018-06-11T11:13:48.605Z|00052|ofproto_dpif|INFO|system@ovs-system: > Datapath supports ct_mark > 2018-06-11T11:13:48.605Z|00053|ofproto_dpif|INFO|system@ovs-system: > Datapath supports ct_label > 2018-06-11T11:13:48.605Z|00054|ofproto_dpif|INFO|system@ovs-system: > Datapath does not support ct_state_nat > 2018-06-11T11:13:48.605Z|00055|ofproto_dpif|INFO|system@ovs-system: > Datapath does not support ct_orig_tuple > 2018-06-11T11:13:48.605Z|00056|ofproto_dpif|INFO|system@ovs-system: > Datapath does not support ct_orig_tuple6 2018-06-11T11:13:48.981Z|00057|bridge|INFO|bridge br-mlx: added interface br-mlx on port 65534 > 2018-06-11T11:13:48.981Z|00058|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T11:13:48.981Z|00059|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK > 2018-06-11T11:13:48.981Z|00060|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T11:13:48.982Z|00061|bridge|INFO|bridge br-int: added interface br-int on port 65534 2018-06-11T11:13:48.982Z|00062|bridge|INFO|bridge br-int: added interface int-br-ex on port 1 2018-06-11T11:13:48.982Z|00063|bridge|INFO|bridge br-int: added interface patch-tun on port 2 2018-06-11T11:13:48.982Z|00064|bridge|INFO|bridge br-tun: added interface patch-int on port 1 2018-06-11T11:13:48.982Z|00065|bridge|INFO|bridge br-tun: added interface br-tun on port 65534 2018-06-11T11:13:48.982Z|00066|bridge|INFO|bridge br-ex: added interface br-ex on port 65534 2018-06-11T11:13:48.982Z|00067|bridge|INFO|bridge br-ex: added interface phy-br-ex on port 1 2018-06-11T11:13:48.982Z|00068|bridge|INFO|bridge br-mlx: using datapath ID 000072528484a642 > 2018-06-11T11:13:48.982Z|00069|connmgr|INFO|br-mlx: added service controller "punix:/var/run/openvswitch/br-mlx.mgmt" > 2018-06-11T11:13:49.221Z|00070|bridge|INFO|bridge br-int: using datapath ID 00006680927b6b40 > 2018-06-11T11:13:49.221Z|00071|connmgr|INFO|br-int: added service controller "punix:/var/run/openvswitch/br-int.mgmt" > 2018-06-11T11:13:49.221Z|00072|connmgr|INFO|br-int: added primary controller "tcp:127.0.0.1:6633" > 2018-06-11T11:13:49.221Z|00073|rconn|INFO|br-int<->tcp:127.0.0.1:6633: > connecting... > 2018-06-11T11:13:49.460Z|00074|bridge|INFO|bridge br-tun: using datapath ID 000066532e5f0443 > 2018-06-11T11:13:49.461Z|00075|connmgr|INFO|br-tun: added service controller "punix:/var/run/openvswitch/br-tun.mgmt" > 2018-06-11T11:13:49.461Z|00076|connmgr|INFO|br-tun: added primary controller "tcp:127.0.0.1:6633" > 2018-06-11T11:13:49.461Z|00077|rconn|INFO|br-tun<->tcp:127.0.0.1:6633: > connecting... > 2018-06-11T11:13:49.692Z|00078|bridge|INFO|bridge br-ex: using datapath ID 0000e2d1bdbe8e45 > 2018-06-11T11:13:49.692Z|00079|connmgr|INFO|br-ex: added service controller "punix:/var/run/openvswitch/br-ex.mgmt" > 2018-06-11T11:13:49.692Z|00080|connmgr|INFO|br-ex: added primary controller "tcp:127.0.0.1:6633" > 2018-06-11T11:13:49.692Z|00081|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: > connecting... > 2018-06-11T11:13:49.916Z|00082|timeval|WARN|Unreasonably long 4661ms poll interval (16ms user, 4472ms system) > 2018-06-11T11:13:49.916Z|00083|timeval|WARN|faults: 3584706 minor, 0 major > 2018-06-11T11:13:49.916Z|00084|timeval|WARN|disk: 0 reads, 16 writes 2018-06-11T11:13:49.916Z|00085|timeval|WARN|context switches: 147 voluntary, 25 involuntary 2018-06-11T11:13:49.916Z|00086|coverage|INFO|Event coverage, avg rate over last: 5 seconds, last minute, last hour, hash=2ba35161: > 2018-06-11T11:13:49.916Z|00087|coverage|INFO|bridge_reconfigure > 0.0/sec 0.000/sec 0.0000/sec total: 1 > 2018-06-11T11:13:49.916Z|00088|coverage|INFO|ofproto_flush > 0.0/sec 0.000/sec 0.0000/sec total: 4 > 2018-06-11T11:13:49.916Z|00089|coverage|INFO|ofproto_update_port > 0.0/sec 0.000/sec 0.0000/sec total: 8 > 2018-06-11T11:13:49.916Z|00090|coverage|INFO|rev_flow_table > 0.0/sec 0.000/sec 0.0000/sec total: 1 > 2018-06-11T11:13:49.916Z|00091|coverage|INFO|cmap_expand > 0.0/sec 0.000/sec 0.0000/sec total: 52 > 2018-06-11T11:13:49.916Z|00092|coverage|INFO|cmap_shrink > 0.0/sec 0.000/sec 0.0000/sec total: 23 > 2018-06-11T11:13:49.916Z|00093|coverage|INFO|dpif_port_add > 0.0/sec 0.000/sec 0.0000/sec total: 1 > 2018-06-11T11:13:49.916Z|00094|coverage|INFO|dpif_flow_flush > 0.0/sec 0.000/sec 0.0000/sec total: 6 > 2018-06-11T11:13:49.916Z|00095|coverage|INFO|dpif_flow_get > 0.0/sec 0.000/sec 0.0000/sec total: 29 > 2018-06-11T11:13:49.916Z|00096|coverage|INFO|dpif_flow_put > 0.0/sec 0.000/sec 0.0000/sec total: 37 > 2018-06-11T11:13:49.916Z|00097|coverage|INFO|dpif_flow_del > 0.0/sec 0.000/sec 0.0000/sec total: 29 > 2018-06-11T11:13:49.916Z|00098|coverage|INFO|dpif_execute > 0.0/sec 0.000/sec 0.0000/sec total: 8 > 2018-06-11T11:13:49.916Z|00099|coverage|INFO|flow_extract > 0.0/sec 0.000/sec 0.0000/sec total: 6 > 2018-06-11T11:13:49.916Z|00100|coverage|INFO|miniflow_malloc > 0.0/sec 0.000/sec 0.0000/sec total: 34 > 2018-06-11T11:13:49.916Z|00101|coverage|INFO|hmap_expand > 0.0/sec 0.000/sec 0.0000/sec total: 498 > 2018-06-11T11:13:49.916Z|00102|coverage|INFO|netdev_received > 0.0/sec 0.000/sec 0.0000/sec total: 1 > 2018-06-11T11:13:49.916Z|00103|coverage|INFO|netdev_get_stats > 0.0/sec 0.000/sec 0.0000/sec total: 8 > 2018-06-11T11:13:49.916Z|00104|coverage|INFO|poll_create_node > 0.0/sec 0.000/sec 0.0000/sec total: 23 > 2018-06-11T11:13:49.916Z|00105|coverage|INFO|seq_change > 0.0/sec 0.000/sec 0.0000/sec total: 2193 > 2018-06-11T11:13:49.916Z|00106|coverage|INFO|pstream_open > 0.0/sec 0.000/sec 0.0000/sec total: 9 > 2018-06-11T11:13:49.916Z|00107|coverage|INFO|stream_open > 0.0/sec 0.000/sec 0.0000/sec total: 4 > 2018-06-11T11:13:49.916Z|00108|coverage|INFO|util_xalloc > 0.0/sec 0.000/sec 0.0000/sec total: 12803 > 2018-06-11T11:13:49.916Z|00109|coverage|INFO|vconn_open > 0.0/sec 0.000/sec 0.0000/sec total: 3 > 2018-06-11T11:13:49.916Z|00110|coverage|INFO|netdev_set_policing > 0.0/sec 0.000/sec 0.0000/sec total: 4 > 2018-06-11T11:13:49.916Z|00111|coverage|INFO|netdev_get_ifindex > 0.0/sec 0.000/sec 0.0000/sec total: 4 > 2018-06-11T11:13:49.916Z|00112|coverage|INFO|netdev_get_hwaddr > 0.0/sec 0.000/sec 0.0000/sec total: 9 > 2018-06-11T11:13:49.916Z|00113|coverage|INFO|netdev_get_ethtool > 0.0/sec 0.000/sec 0.0000/sec total: 2 > 2018-06-11T11:13:49.916Z|00114|coverage|INFO|netlink_received > 0.0/sec 0.000/sec 0.0000/sec total: 85 > 2018-06-11T11:13:49.916Z|00115|coverage|INFO|netlink_recv_jumbo > 0.0/sec 0.000/sec 0.0000/sec total: 1 > 2018-06-11T11:13:49.916Z|00116|coverage|INFO|netlink_sent > 0.0/sec 0.000/sec 0.0000/sec total: 102 > 2018-06-11T11:13:49.916Z|00117|coverage|INFO|74 events never hit 2018-06-11T11:13:49.916Z|00118|poll_loop|INFO|wakeup due to [POLLIN] on fd 433 (FIFO pipe:[1025715]) at ../lib/ovs-rcu.c:229 (97% CPU usage) 2018-06-11T11:13:49.917Z|00119|poll_loop|INFO|wakeup due to [POLLIN] on fd 433 (FIFO pipe:[1025715]) at ../lib/ovs-rcu.c:229 (97% CPU usage) 2018-06-11T11:13:49.917Z|00120|poll_loop|INFO|wakeup due to 0-ms timeout at ../lib/ovs-rcu.c:229 (97% CPU usage) 2018-06-11T11:13:49.917Z|00121|poll_loop|INFO|wakeup due to 0-ms timeout at ../lib/ovs-rcu.c:229 (97% CPU usage) > 2018-06-11T11:13:49.920Z|00122|rconn|INFO|br-int<->tcp:127.0.0.1:6633: > connected > 2018-06-11T11:13:49.920Z|00123|rconn|INFO|br-tun<->tcp:127.0.0.1:6633: > connected > 2018-06-11T11:13:49.920Z|00124|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: > connected > 2018-06-11T11:13:49.926Z|00125|poll_loop|INFO|wakeup due to [POLLIN] on fd 10 (NETLINK_ROUTE<->NETLINK_ROUTE) at ../lib/netlink-socket.c:1331 (97% CPU usage) 2018-06-11T11:13:49.926Z|00126|poll_loop|INFO|wakeup due to [POLLIN] on fd 421 (character device /dev/net/tun) at ../lib/netdev-linux.c:1171 (97% CPU usage) 2018-06-11T11:13:49.926Z|00127|poll_loop|INFO|wakeup due to [POLLIN] on fd 423 (127.0.0.1:33712<->127.0.0.1:6633) at ../lib/stream-fd.c:157 (97% CPU usage) 2018-06-11T11:13:49.926Z|00128|poll_loop|INFO|wakeup due to [POLLIN] on fd 428 (127.0.0.1:33714<->127.0.0.1:6633) at ../lib/stream-fd.c:157 (97% CPU usage) 2018-06-11T11:13:49.926Z|00129|poll_loop|INFO|wakeup due to [POLLIN] on fd 11 (<->/var/run/openvswitch/db.sock) at ../lib/stream-fd.c:157 (97% CPU usage) 2018-06-11T11:13:49.926Z|00130|poll_loop|INFO|wakeup due to [POLLIN] on fd 429 (127.0.0.1:33716<->127.0.0.1:6633) at ../lib/stream-fd.c:157 (97% CPU usage) > 2018-06-11T11:13:49.930Z|00131|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T11:13:49.930Z|00132|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK > 2018-06-11T11:13:49.930Z|00133|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) 2018-06-11T11:13:49.931Z|00134|bridge|INFO|ovs-vswitchd (Open vSwitch) 2.9.0 > 2018-06-11T11:13:53.662Z|00135|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T11:13:53.662Z|00136|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK > 2018-06-11T11:13:53.662Z|00137|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) > 2018-06-11T11:13:53.675Z|00138|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T11:13:53.675Z|00139|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK > 2018-06-11T11:13:53.675Z|00140|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) > 2018-06-11T11:13:53.682Z|00141|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T11:13:53.682Z|00142|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK > 2018-06-11T11:13:53.682Z|00143|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) > 2018-06-11T11:13:53.693Z|00144|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T11:13:53.693Z|00145|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK > 2018-06-11T11:13:53.693Z|00146|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) > 2018-06-11T11:13:53.705Z|00147|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T11:13:53.705Z|00148|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK > 2018-06-11T11:13:53.705Z|00149|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) > 2018-06-11T11:13:53.711Z|00150|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T11:13:53.711Z|00151|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK > 2018-06-11T11:13:53.711Z|00152|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) > 2018-06-11T11:13:53.717Z|00153|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T11:13:53.717Z|00154|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK > 2018-06-11T11:13:53.717Z|00155|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) > 2018-06-11T11:13:53.728Z|00156|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T11:13:53.728Z|00157|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK > 2018-06-11T11:13:53.728Z|00158|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) > 2018-06-11T11:13:53.743Z|00159|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T11:13:53.743Z|00160|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK > 2018-06-11T11:13:53.743Z|00161|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) > 2018-06-11T11:13:53.749Z|00162|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T11:13:53.749Z|00163|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK > 2018-06-11T11:13:53.749Z|00164|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) > 2018-06-11T11:13:53.756Z|00165|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T11:13:53.756Z|00166|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK > 2018-06-11T11:13:53.756Z|00167|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) > 2018-06-11T11:13:53.766Z|00168|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T11:13:53.766Z|00169|netdev_dpdk|WARN|Error attaching device 'mlx4_0' to DPDK > 2018-06-11T11:13:53.766Z|00170|netdev|WARN|enp59s0d1: could not set configuration (Invalid argument) > 2018-06-11T11:13:55.270Z|00171|memory|INFO|1406100 kB peak resident set size after 10.0 seconds > 2018-06-11T11:13:55.270Z|00172|memory|INFO|handlers:118 ofconns:3 > ports:8 revalidators:42 rules:36 udpif keys:1 > 2018-06-11T11:14:03.699Z|00173|connmgr|INFO|br-int<->tcp:127.0.0.1:6633: > 5 flow_mods 10 s ago (5 adds) > 2018-06-11T11:14:03.737Z|00174|connmgr|INFO|br-ex<->tcp:127.0.0.1:6633: > 2 flow_mods 10 s ago (2 adds) > 2018-06-11T11:14:03.773Z|00175|connmgr|INFO|br-tun<->tcp:127.0.0.1:6633: > 12 flow_mods 10 s ago (12 adds) > > Best regards > > On 06/11/2018 12:57 PM, Raslan Darawsheh wrote: >> What I really want to see is the init process for OVS-DPDK that dumps the DPDK initialization. >> I think you will find it at the beginning of the log that you just sent. >> >> Kindest regards, >> Raslan Darawsheh >> >> -----Original Message----- >> From: Tobias Urdin [mailto:tobias.urdin@crystone.com] >> Sent: Monday, June 11, 2018 1:00 PM >> To: Raslan Darawsheh <rasland@mellanox.com> >> Cc: users@dpdk.org >> Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch >> >> Hello Raslan, >> Thank you for your reply. >> >> For reference the br-mlx bridge was created like this: >> $ovs-vsctl add-br br-mlx -- set bridge br-mlx datapath_type=netdev >> >> $ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 >> type=dpdk options:dpdk-devargs=mlx4_0 >> ovs-vsctl: Error detected while setting up 'enp59s0d1': Error attaching device 'mlx4_0' to DPDK. See ovs-vswitchd log for details. >> ovs-vsctl: The default log directory is "/var/log/openvswitch". >> >> In the ovs-vswitchd.log >> 2018-06-11T09:56:20.613Z|00185|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T09:56:20.613Z|00186|netdev_dpdk|WARN|Error attaching device >> 'mlx4_0' to DPDK >> 2018-06-11T09:56:20.613Z|00187|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> >> Here is the complete log after restarting openvswitch-switch service: >> 2018-06-11T09:57:49.380Z|00134|bridge|INFO|ovs-vswitchd (Open vSwitch) >> 2.9.0 >> 2018-06-11T09:57:52.812Z|00135|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T09:57:52.812Z|00136|netdev_dpdk|WARN|Error attaching device >> 'mlx4_0' to DPDK >> 2018-06-11T09:57:52.812Z|00137|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T09:57:52.825Z|00138|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T09:57:52.825Z|00139|netdev_dpdk|WARN|Error attaching device >> 'mlx4_0' to DPDK >> 2018-06-11T09:57:52.825Z|00140|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T09:57:52.832Z|00141|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T09:57:52.832Z|00142|netdev_dpdk|WARN|Error attaching device >> 'mlx4_0' to DPDK >> 2018-06-11T09:57:52.832Z|00143|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T09:57:52.843Z|00144|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T09:57:52.843Z|00145|netdev_dpdk|WARN|Error attaching device >> 'mlx4_0' to DPDK >> 2018-06-11T09:57:52.843Z|00146|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T09:57:52.856Z|00147|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T09:57:52.856Z|00148|netdev_dpdk|WARN|Error attaching device >> 'mlx4_0' to DPDK >> 2018-06-11T09:57:52.856Z|00149|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T09:57:52.862Z|00150|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T09:57:52.862Z|00151|netdev_dpdk|WARN|Error attaching device >> 'mlx4_0' to DPDK >> 2018-06-11T09:57:52.862Z|00152|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T09:57:52.870Z|00153|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T09:57:52.870Z|00154|netdev_dpdk|WARN|Error attaching device >> 'mlx4_0' to DPDK >> 2018-06-11T09:57:52.870Z|00155|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T09:57:52.889Z|00156|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T09:57:52.889Z|00157|netdev_dpdk|WARN|Error attaching device >> 'mlx4_0' to DPDK >> 2018-06-11T09:57:52.889Z|00158|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T09:57:52.903Z|00159|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T09:57:52.903Z|00160|netdev_dpdk|WARN|Error attaching device >> 'mlx4_0' to DPDK >> 2018-06-11T09:57:52.903Z|00161|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T09:57:52.908Z|00162|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T09:57:52.908Z|00163|netdev_dpdk|WARN|Error attaching device >> 'mlx4_0' to DPDK >> 2018-06-11T09:57:52.908Z|00164|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T09:57:52.914Z|00165|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T09:57:52.914Z|00166|netdev_dpdk|WARN|Error attaching device >> 'mlx4_0' to DPDK >> 2018-06-11T09:57:52.914Z|00167|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T09:57:52.923Z|00168|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T09:57:52.923Z|00169|netdev_dpdk|WARN|Error attaching device >> 'mlx4_0' to DPDK >> 2018-06-11T09:57:52.923Z|00170|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T09:57:54.808Z|00171|memory|INFO|1406100 kB peak resident >> set size after 10.1 seconds >> 2018-06-11T09:57:54.808Z|00172|memory|INFO|handlers:118 ofconns:3 >> ports:8 revalidators:42 rules:36 udpif keys:1 >> 2018-06-11T09:58:02.850Z|00173|connmgr|INFO|br-int<->tcp:127.0.0.1:6633: >> 5 flow_mods 10 s ago (5 adds) >> 2018-06-11T09:58:02.898Z|00174|connmgr|INFO|br-ex<->tcp:127.0.0.1:6633: >> 2 flow_mods 10 s ago (2 adds) >> 2018-06-11T09:58:02.930Z|00175|connmgr|INFO|br-tun<->tcp:127.0.0.1:6633: >> 12 flow_mods 10 s ago (12 adds) >> >> Was that what you mean by init log? >> Best regards >> >> On 06/11/2018 11:51 AM, Raslan Darawsheh wrote: >>> Hi Tobias, >>> >>> Can you try using the following ? >>> >>> ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 >>> type=dpdk options:dpdk-devargs=mlx4_0 >>> >>> if it didn't work can you send the OVS-DPDK init log ? >>> >>> Kindest regards, >>> Raslan Darawsheh >>> >>> -----Original Message----- >>> From: users [mailto:users-bounces@dpdk.org] On Behalf Of Tobias Urdin >>> Sent: Monday, June 11, 2018 12:15 PM >>> To: users@dpdk.org >>> Subject: [dpdk-users] mellanox connect-x 3 pro dpdk with openvswitch >>> >>> Hello, >>> >>> I'm stuck trying to test DPDK with a Mellanox Connect-X 3 Pro card on >>> Ubuntu 16.04 >>> >>> I've enabled iommu, fixed hugepages and has installed and are running the openvswitch-switch-dpdk alternative. >>> >>> $lspci | grep -i mellanox >>> 3b:00.0 Ethernet controller: Mellanox Technologies MT27520 Family >>> [ConnectX-3 Pro] >>> >>> $lsmod | grep -i mlx >>> mlx4_en 110592 0 >>> vxlan 49152 2 i40e,mlx4_en >>> mlx4_core 286720 1 mlx4_en >>> ptp 20480 3 igb,i40e,mlx4_en >>> >>> $cat /etc/modprobe.d/mlx4_core.conf >>> options mlx4_core log_num_mgm_entry_size=-1 >>> >>> >>> $ovs-vsctl add-port br-mlx dpdk0 -- set Interface dpdk0 type=dpdk >>> >>> Bridge br-mlx >>> Port "dpdk0" >>> Interface "dpdk0" >>> type: dpdk >>> error: "'dpdk0' is missing 'options:dpdk-devargs'. The old 'dpdk<port_id>' names are not supported" >>> >>> >>> $ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 >>> type=dpdk options:dpdk-devargs=0000:3b:00.0,1 >>> ovs-vsctl: Error detected while setting up 'enp59s0d1': Error attaching device '0000:3b:00.0,1' to DPDK. See ovs-vswitchd log for details. >>> ovs-vsctl: The default log directory is "/var/log/openvswitch". >>> >>> 2018-06-11T09:10:38.696Z|00177|netdev_dpdk|WARN|Error attaching >>> device '0000:3b:00.0,1' to DPDK >>> 2018-06-11T09:10:38.696Z|00178|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> >>> 2018-06-11T09:12:50.994Z|00181|dpdk|ERR|EAL: Driver cannot attach the >>> device (0000:3b:00.0) >>> 2018-06-11T09:12:50.994Z|00182|dpdk|ERR|EAL: No port found for device >>> (0000:3b:00.0) 2018-06-11T09:12:50.994Z|00183|netdev_dpdk|WARN|Error >>> attaching device '0000:3b:00.0,1' to DPDK >>> 2018-06-11T09:12:50.994Z|00184|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> >>> $ovs-vsctl add-port br-mlx dpdk0 -- set Interface dpdk0 type=dpdk options:dpdk-devargs="class=eth,mac=ec:0d:9a:b0:13:21" >>> ovs-vsctl: Error detected while setting up 'dpdk0': Error attaching device 'class=eth,mac=ec:0d:9a:b0:13:21' to DPDK. See ovs-vswitchd log for details. >>> ovs-vsctl: The default log directory is "/var/log/openvswitch". >>> >>> Port "dpdk0" >>> Interface "dpdk0" >>> type: dpdk >>> options: {dpdk-devargs="class=eth,mac=ec:0d:9a:b0:13:21"} >>> error: "Error attaching device 'class=eth,mac=ec:0d:9a:b0:13:21' to DPDK" >>> >>> >>> Anybody know what I might have missed? I have tried restarting the openvswitch services after adding the interfaces as well but no difference. >>> >>> The logs doesn't say anything interesting from what I can see :( >>> >>> Best regards >>> > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [dpdk-users] mellanox connect-x 3 pro dpdk with openvswitch 2018-06-11 11:34 ` Tobias Urdin @ 2018-06-11 11:37 ` Raslan Darawsheh 2018-06-11 11:45 ` Tobias Urdin 0 siblings, 1 reply; 14+ messages in thread From: Raslan Darawsheh @ 2018-06-11 11:37 UTC (permalink / raw) To: Tobias Urdin; +Cc: users You can refer to this community page for compiling OVS-DPDK with MLX pmds enabled: https://community.mellanox.com/docs/DOC-2637 Kindest regards, Raslan Darawsheh -----Original Message----- From: Tobias Urdin [mailto:tobias.urdin@crystone.com] Sent: Monday, June 11, 2018 2:35 PM To: Raslan Darawsheh <rasland@mellanox.com> Cc: users@dpdk.org Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch I haven't compiled any DPDK or Mellanox related modules/packages. What I have is ovs, ovs-dpdk, mlx4 module, I have also loaded the ib_uverbs driver. Is it the librte_pmd_mlx4 I have to compile manually? Do you know any updated documentation about this, there isn't much relevant and searching for documentation and the correct mlnx driver and code is a struggle. Best regards On 06/11/2018 01:20 PM, Raslan Darawsheh wrote: > From what I'm seeing it looks like it didn't prop the MLX4 device at all. > Can you make sure you have it enabled in your DPDK package when you compiled DPDK ? > > Kindest regards, > Raslan Darawsheh > > -----Original Message----- > From: Tobias Urdin [mailto:tobias.urdin@crystone.com] > Sent: Monday, June 11, 2018 2:16 PM > To: Raslan Darawsheh <rasland@mellanox.com> > Cc: users@dpdk.org > Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch > > Ok, here is the complete log when restarting the openvswitch-switch service, see below. > > Also here is the kernel for information: > $uname -r > 4.4.0-127-generic > > 2018-06-11T11:13:42.729Z|00176|bridge|INFO|bridge br-mlx: deleted > interface br-mlx on port 65534 > 2018-06-11T11:13:43.274Z|00177|bridge|INFO|bridge br-int: deleted > interface br-int on port 65534 > 2018-06-11T11:13:43.275Z|00178|bridge|INFO|bridge br-int: deleted > interface int-br-ex on port 1 > 2018-06-11T11:13:43.275Z|00179|bridge|INFO|bridge br-int: deleted > interface patch-tun on port 2 > 2018-06-11T11:13:43.762Z|00180|bridge|INFO|bridge br-tun: deleted > interface patch-int on port 1 > 2018-06-11T11:13:43.762Z|00181|bridge|INFO|bridge br-tun: deleted > interface br-tun on port 65534 > 2018-06-11T11:13:44.231Z|00182|bridge|INFO|bridge br-ex: deleted > interface br-ex on port 65534 > 2018-06-11T11:13:44.232Z|00183|bridge|INFO|bridge br-ex: deleted > interface phy-br-ex on port 1 > 2018-06-11T11:13:45.245Z|00001|vlog|INFO|opened log file > /var/log/openvswitch/ovs-vswitchd.log > 2018-06-11T11:13:45.252Z|00002|ovs_numa|INFO|Discovered 40 CPU cores > on NUMA node 0 2018-06-11T11:13:45.252Z|00003|ovs_numa|INFO|Discovered > 40 CPU cores on NUMA node 1 > 2018-06-11T11:13:45.252Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes > and > 80 CPU cores > 2018-06-11T11:13:45.252Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: > connecting... > 2018-06-11T11:13:45.252Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: > connected > 2018-06-11T11:13:45.256Z|00007|dpdk|INFO|Using DPDK 17.11.0 2018-06-11T11:13:45.256Z|00008|dpdk|INFO|DPDK Enabled - initializing... > 2018-06-11T11:13:45.256Z|00009|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch 2018-06-11T11:13:45.256Z|00010|dpdk|INFO|IOMMU support for vhost-user-client disabled. > 2018-06-11T11:13:45.256Z|00011|dpdk|INFO|EAL ARGS: ovs-vswitchd > --socket-mem 1024,0 -c 0x00000001 > 2018-06-11T11:13:45.258Z|00012|dpdk|INFO|EAL: Detected 80 lcore(s) > 2018-06-11T11:13:45.285Z|00013|dpdk|INFO|EAL: Probing VFIO support... > 2018-06-11T11:13:45.285Z|00014|dpdk|INFO|EAL: VFIO support initialized > 2018-06-11T11:13:48.214Z|00015|dpdk|INFO|EAL: PCI device 0000:01:00.0 on NUMA socket 0 > 2018-06-11T11:13:48.214Z|00016|dpdk|INFO|EAL: probe driver: 8086:1521 > net_e1000_igb > 2018-06-11T11:13:48.214Z|00017|dpdk|INFO|EAL: PCI device 0000:01:00.1 on NUMA socket 0 > 2018-06-11T11:13:48.214Z|00018|dpdk|INFO|EAL: probe driver: 8086:1521 > net_e1000_igb > 2018-06-11T11:13:48.214Z|00019|dpdk|INFO|EAL: PCI device 0000:18:00.0 on NUMA socket 0 > 2018-06-11T11:13:48.214Z|00020|dpdk|INFO|EAL: probe driver: 8086:1572 > net_i40e > 2018-06-11T11:13:48.214Z|00021|dpdk|INFO|EAL: PCI device 0000:18:00.1 on NUMA socket 0 > 2018-06-11T11:13:48.214Z|00022|dpdk|INFO|EAL: probe driver: 8086:1572 > net_i40e > 2018-06-11T11:13:48.214Z|00023|dpdk|INFO|DPDK pdump packet capture > enabled 2018-06-11T11:13:48.224Z|00024|dpdk|INFO|DPDK Enabled - > initialized > 2018-06-11T11:13:48.235Z|00025|ofproto_dpif|INFO|netdev@ovs-netdev: > Datapath supports recirculation > 2018-06-11T11:13:48.235Z|00026|ofproto_dpif|INFO|netdev@ovs-netdev: > VLAN header stack length probed as 1 > 2018-06-11T11:13:48.235Z|00027|ofproto_dpif|INFO|netdev@ovs-netdev: > MPLS label stack length probed as 3 > 2018-06-11T11:13:48.235Z|00028|ofproto_dpif|INFO|netdev@ovs-netdev: > Datapath supports truncate action > 2018-06-11T11:13:48.235Z|00029|ofproto_dpif|INFO|netdev@ovs-netdev: > Datapath supports unique flow ids > 2018-06-11T11:13:48.235Z|00030|ofproto_dpif|INFO|netdev@ovs-netdev: > Datapath supports clone action > 2018-06-11T11:13:48.235Z|00031|ofproto_dpif|INFO|netdev@ovs-netdev: > Max sample nesting level probed as 10 > 2018-06-11T11:13:48.235Z|00032|ofproto_dpif|INFO|netdev@ovs-netdev: > Datapath supports eventmask in conntrack action > 2018-06-11T11:13:48.235Z|00033|ofproto_dpif|INFO|netdev@ovs-netdev: > Datapath supports ct_clear action > 2018-06-11T11:13:48.235Z|00034|ofproto_dpif|INFO|netdev@ovs-netdev: > Datapath supports ct_state > 2018-06-11T11:13:48.235Z|00035|ofproto_dpif|INFO|netdev@ovs-netdev: > Datapath supports ct_zone > 2018-06-11T11:13:48.235Z|00036|ofproto_dpif|INFO|netdev@ovs-netdev: > Datapath supports ct_mark > 2018-06-11T11:13:48.235Z|00037|ofproto_dpif|INFO|netdev@ovs-netdev: > Datapath supports ct_label > 2018-06-11T11:13:48.235Z|00038|ofproto_dpif|INFO|netdev@ovs-netdev: > Datapath supports ct_state_nat > 2018-06-11T11:13:48.235Z|00039|ofproto_dpif|INFO|netdev@ovs-netdev: > Datapath supports ct_orig_tuple > 2018-06-11T11:13:48.235Z|00040|ofproto_dpif|INFO|netdev@ovs-netdev: > Datapath supports ct_orig_tuple6 > 2018-06-11T11:13:48.605Z|00041|ofproto_dpif|INFO|system@ovs-system: > Datapath supports recirculation > 2018-06-11T11:13:48.605Z|00042|ofproto_dpif|INFO|system@ovs-system: > VLAN header stack length probed as 1 > 2018-06-11T11:13:48.605Z|00043|ofproto_dpif|INFO|system@ovs-system: > MPLS label stack length probed as 1 > 2018-06-11T11:13:48.605Z|00044|ofproto_dpif|INFO|system@ovs-system: > Datapath does not support truncate action > 2018-06-11T11:13:48.605Z|00045|ofproto_dpif|INFO|system@ovs-system: > Datapath supports unique flow ids > 2018-06-11T11:13:48.605Z|00046|ofproto_dpif|INFO|system@ovs-system: > Datapath does not support clone action > 2018-06-11T11:13:48.605Z|00047|ofproto_dpif|INFO|system@ovs-system: > Max sample nesting level probed as 3 > 2018-06-11T11:13:48.605Z|00048|ofproto_dpif|INFO|system@ovs-system: > Datapath does not support eventmask in conntrack action > 2018-06-11T11:13:48.605Z|00049|ofproto_dpif|INFO|system@ovs-system: > Datapath does not support ct_clear action > 2018-06-11T11:13:48.605Z|00050|ofproto_dpif|INFO|system@ovs-system: > Datapath supports ct_state > 2018-06-11T11:13:48.605Z|00051|ofproto_dpif|INFO|system@ovs-system: > Datapath supports ct_zone > 2018-06-11T11:13:48.605Z|00052|ofproto_dpif|INFO|system@ovs-system: > Datapath supports ct_mark > 2018-06-11T11:13:48.605Z|00053|ofproto_dpif|INFO|system@ovs-system: > Datapath supports ct_label > 2018-06-11T11:13:48.605Z|00054|ofproto_dpif|INFO|system@ovs-system: > Datapath does not support ct_state_nat > 2018-06-11T11:13:48.605Z|00055|ofproto_dpif|INFO|system@ovs-system: > Datapath does not support ct_orig_tuple > 2018-06-11T11:13:48.605Z|00056|ofproto_dpif|INFO|system@ovs-system: > Datapath does not support ct_orig_tuple6 > 2018-06-11T11:13:48.981Z|00057|bridge|INFO|bridge br-mlx: added > interface br-mlx on port 65534 > 2018-06-11T11:13:48.981Z|00058|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T11:13:48.981Z|00059|netdev_dpdk|WARN|Error attaching device > 'mlx4_0' to DPDK > 2018-06-11T11:13:48.981Z|00060|netdev|WARN|enp59s0d1: could not set > configuration (Invalid argument) > 2018-06-11T11:13:48.982Z|00061|bridge|INFO|bridge br-int: added > interface br-int on port 65534 > 2018-06-11T11:13:48.982Z|00062|bridge|INFO|bridge br-int: added > interface int-br-ex on port 1 > 2018-06-11T11:13:48.982Z|00063|bridge|INFO|bridge br-int: added > interface patch-tun on port 2 > 2018-06-11T11:13:48.982Z|00064|bridge|INFO|bridge br-tun: added > interface patch-int on port 1 > 2018-06-11T11:13:48.982Z|00065|bridge|INFO|bridge br-tun: added > interface br-tun on port 65534 > 2018-06-11T11:13:48.982Z|00066|bridge|INFO|bridge br-ex: added > interface br-ex on port 65534 > 2018-06-11T11:13:48.982Z|00067|bridge|INFO|bridge br-ex: added > interface phy-br-ex on port 1 > 2018-06-11T11:13:48.982Z|00068|bridge|INFO|bridge br-mlx: using > datapath ID 000072528484a642 > 2018-06-11T11:13:48.982Z|00069|connmgr|INFO|br-mlx: added service controller "punix:/var/run/openvswitch/br-mlx.mgmt" > 2018-06-11T11:13:49.221Z|00070|bridge|INFO|bridge br-int: using > datapath ID 00006680927b6b40 > 2018-06-11T11:13:49.221Z|00071|connmgr|INFO|br-int: added service controller "punix:/var/run/openvswitch/br-int.mgmt" > 2018-06-11T11:13:49.221Z|00072|connmgr|INFO|br-int: added primary controller "tcp:127.0.0.1:6633" > 2018-06-11T11:13:49.221Z|00073|rconn|INFO|br-int<->tcp:127.0.0.1:6633: > connecting... > 2018-06-11T11:13:49.460Z|00074|bridge|INFO|bridge br-tun: using > datapath ID 000066532e5f0443 > 2018-06-11T11:13:49.461Z|00075|connmgr|INFO|br-tun: added service controller "punix:/var/run/openvswitch/br-tun.mgmt" > 2018-06-11T11:13:49.461Z|00076|connmgr|INFO|br-tun: added primary controller "tcp:127.0.0.1:6633" > 2018-06-11T11:13:49.461Z|00077|rconn|INFO|br-tun<->tcp:127.0.0.1:6633: > connecting... > 2018-06-11T11:13:49.692Z|00078|bridge|INFO|bridge br-ex: using > datapath ID 0000e2d1bdbe8e45 > 2018-06-11T11:13:49.692Z|00079|connmgr|INFO|br-ex: added service controller "punix:/var/run/openvswitch/br-ex.mgmt" > 2018-06-11T11:13:49.692Z|00080|connmgr|INFO|br-ex: added primary controller "tcp:127.0.0.1:6633" > 2018-06-11T11:13:49.692Z|00081|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: > connecting... > 2018-06-11T11:13:49.916Z|00082|timeval|WARN|Unreasonably long 4661ms > poll interval (16ms user, 4472ms system) > 2018-06-11T11:13:49.916Z|00083|timeval|WARN|faults: 3584706 minor, 0 > major > 2018-06-11T11:13:49.916Z|00084|timeval|WARN|disk: 0 reads, 16 writes 2018-06-11T11:13:49.916Z|00085|timeval|WARN|context switches: 147 voluntary, 25 involuntary 2018-06-11T11:13:49.916Z|00086|coverage|INFO|Event coverage, avg rate over last: 5 seconds, last minute, last hour, hash=2ba35161: > 2018-06-11T11:13:49.916Z|00087|coverage|INFO|bridge_reconfigure > 0.0/sec 0.000/sec 0.0000/sec total: 1 > 2018-06-11T11:13:49.916Z|00088|coverage|INFO|ofproto_flush > 0.0/sec 0.000/sec 0.0000/sec total: 4 > 2018-06-11T11:13:49.916Z|00089|coverage|INFO|ofproto_update_port > 0.0/sec 0.000/sec 0.0000/sec total: 8 > 2018-06-11T11:13:49.916Z|00090|coverage|INFO|rev_flow_table > 0.0/sec 0.000/sec 0.0000/sec total: 1 > 2018-06-11T11:13:49.916Z|00091|coverage|INFO|cmap_expand > 0.0/sec 0.000/sec 0.0000/sec total: 52 > 2018-06-11T11:13:49.916Z|00092|coverage|INFO|cmap_shrink > 0.0/sec 0.000/sec 0.0000/sec total: 23 > 2018-06-11T11:13:49.916Z|00093|coverage|INFO|dpif_port_add > 0.0/sec 0.000/sec 0.0000/sec total: 1 > 2018-06-11T11:13:49.916Z|00094|coverage|INFO|dpif_flow_flush > 0.0/sec 0.000/sec 0.0000/sec total: 6 > 2018-06-11T11:13:49.916Z|00095|coverage|INFO|dpif_flow_get > 0.0/sec 0.000/sec 0.0000/sec total: 29 > 2018-06-11T11:13:49.916Z|00096|coverage|INFO|dpif_flow_put > 0.0/sec 0.000/sec 0.0000/sec total: 37 > 2018-06-11T11:13:49.916Z|00097|coverage|INFO|dpif_flow_del > 0.0/sec 0.000/sec 0.0000/sec total: 29 > 2018-06-11T11:13:49.916Z|00098|coverage|INFO|dpif_execute > 0.0/sec 0.000/sec 0.0000/sec total: 8 > 2018-06-11T11:13:49.916Z|00099|coverage|INFO|flow_extract > 0.0/sec 0.000/sec 0.0000/sec total: 6 > 2018-06-11T11:13:49.916Z|00100|coverage|INFO|miniflow_malloc > 0.0/sec 0.000/sec 0.0000/sec total: 34 > 2018-06-11T11:13:49.916Z|00101|coverage|INFO|hmap_expand > 0.0/sec 0.000/sec 0.0000/sec total: 498 > 2018-06-11T11:13:49.916Z|00102|coverage|INFO|netdev_received > 0.0/sec 0.000/sec 0.0000/sec total: 1 > 2018-06-11T11:13:49.916Z|00103|coverage|INFO|netdev_get_stats > 0.0/sec 0.000/sec 0.0000/sec total: 8 > 2018-06-11T11:13:49.916Z|00104|coverage|INFO|poll_create_node > 0.0/sec 0.000/sec 0.0000/sec total: 23 > 2018-06-11T11:13:49.916Z|00105|coverage|INFO|seq_change > 0.0/sec 0.000/sec 0.0000/sec total: 2193 > 2018-06-11T11:13:49.916Z|00106|coverage|INFO|pstream_open > 0.0/sec 0.000/sec 0.0000/sec total: 9 > 2018-06-11T11:13:49.916Z|00107|coverage|INFO|stream_open > 0.0/sec 0.000/sec 0.0000/sec total: 4 > 2018-06-11T11:13:49.916Z|00108|coverage|INFO|util_xalloc > 0.0/sec 0.000/sec 0.0000/sec total: 12803 > 2018-06-11T11:13:49.916Z|00109|coverage|INFO|vconn_open > 0.0/sec 0.000/sec 0.0000/sec total: 3 > 2018-06-11T11:13:49.916Z|00110|coverage|INFO|netdev_set_policing > 0.0/sec 0.000/sec 0.0000/sec total: 4 > 2018-06-11T11:13:49.916Z|00111|coverage|INFO|netdev_get_ifindex > 0.0/sec 0.000/sec 0.0000/sec total: 4 > 2018-06-11T11:13:49.916Z|00112|coverage|INFO|netdev_get_hwaddr > 0.0/sec 0.000/sec 0.0000/sec total: 9 > 2018-06-11T11:13:49.916Z|00113|coverage|INFO|netdev_get_ethtool > 0.0/sec 0.000/sec 0.0000/sec total: 2 > 2018-06-11T11:13:49.916Z|00114|coverage|INFO|netlink_received > 0.0/sec 0.000/sec 0.0000/sec total: 85 > 2018-06-11T11:13:49.916Z|00115|coverage|INFO|netlink_recv_jumbo > 0.0/sec 0.000/sec 0.0000/sec total: 1 > 2018-06-11T11:13:49.916Z|00116|coverage|INFO|netlink_sent > 0.0/sec 0.000/sec 0.0000/sec total: 102 > 2018-06-11T11:13:49.916Z|00117|coverage|INFO|74 events never hit > 2018-06-11T11:13:49.916Z|00118|poll_loop|INFO|wakeup due to [POLLIN] > on fd 433 (FIFO pipe:[1025715]) at ../lib/ovs-rcu.c:229 (97% CPU > usage) 2018-06-11T11:13:49.917Z|00119|poll_loop|INFO|wakeup due to > [POLLIN] on fd 433 (FIFO pipe:[1025715]) at ../lib/ovs-rcu.c:229 (97% > CPU usage) 2018-06-11T11:13:49.917Z|00120|poll_loop|INFO|wakeup due to > 0-ms timeout at ../lib/ovs-rcu.c:229 (97% CPU usage) > 2018-06-11T11:13:49.917Z|00121|poll_loop|INFO|wakeup due to 0-ms > timeout at ../lib/ovs-rcu.c:229 (97% CPU usage) > 2018-06-11T11:13:49.920Z|00122|rconn|INFO|br-int<->tcp:127.0.0.1:6633: > connected > 2018-06-11T11:13:49.920Z|00123|rconn|INFO|br-tun<->tcp:127.0.0.1:6633: > connected > 2018-06-11T11:13:49.920Z|00124|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: > connected > 2018-06-11T11:13:49.926Z|00125|poll_loop|INFO|wakeup due to [POLLIN] > on fd 10 (NETLINK_ROUTE<->NETLINK_ROUTE) at > ../lib/netlink-socket.c:1331 (97% CPU usage) > 2018-06-11T11:13:49.926Z|00126|poll_loop|INFO|wakeup due to [POLLIN] > on fd 421 (character device /dev/net/tun) at > ../lib/netdev-linux.c:1171 (97% CPU usage) > 2018-06-11T11:13:49.926Z|00127|poll_loop|INFO|wakeup due to [POLLIN] > on fd 423 (127.0.0.1:33712<->127.0.0.1:6633) at ../lib/stream-fd.c:157 > (97% CPU usage) 2018-06-11T11:13:49.926Z|00128|poll_loop|INFO|wakeup > due to [POLLIN] on fd 428 (127.0.0.1:33714<->127.0.0.1:6633) at > ../lib/stream-fd.c:157 (97% CPU usage) > 2018-06-11T11:13:49.926Z|00129|poll_loop|INFO|wakeup due to [POLLIN] > on fd 11 (<->/var/run/openvswitch/db.sock) at ../lib/stream-fd.c:157 > (97% CPU usage) 2018-06-11T11:13:49.926Z|00130|poll_loop|INFO|wakeup > due to [POLLIN] on fd 429 (127.0.0.1:33716<->127.0.0.1:6633) at > ../lib/stream-fd.c:157 (97% CPU usage) > 2018-06-11T11:13:49.930Z|00131|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T11:13:49.930Z|00132|netdev_dpdk|WARN|Error attaching device > 'mlx4_0' to DPDK > 2018-06-11T11:13:49.930Z|00133|netdev|WARN|enp59s0d1: could not set > configuration (Invalid argument) > 2018-06-11T11:13:49.931Z|00134|bridge|INFO|ovs-vswitchd (Open vSwitch) > 2.9.0 > 2018-06-11T11:13:53.662Z|00135|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T11:13:53.662Z|00136|netdev_dpdk|WARN|Error attaching device > 'mlx4_0' to DPDK > 2018-06-11T11:13:53.662Z|00137|netdev|WARN|enp59s0d1: could not set > configuration (Invalid argument) > 2018-06-11T11:13:53.675Z|00138|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T11:13:53.675Z|00139|netdev_dpdk|WARN|Error attaching device > 'mlx4_0' to DPDK > 2018-06-11T11:13:53.675Z|00140|netdev|WARN|enp59s0d1: could not set > configuration (Invalid argument) > 2018-06-11T11:13:53.682Z|00141|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T11:13:53.682Z|00142|netdev_dpdk|WARN|Error attaching device > 'mlx4_0' to DPDK > 2018-06-11T11:13:53.682Z|00143|netdev|WARN|enp59s0d1: could not set > configuration (Invalid argument) > 2018-06-11T11:13:53.693Z|00144|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T11:13:53.693Z|00145|netdev_dpdk|WARN|Error attaching device > 'mlx4_0' to DPDK > 2018-06-11T11:13:53.693Z|00146|netdev|WARN|enp59s0d1: could not set > configuration (Invalid argument) > 2018-06-11T11:13:53.705Z|00147|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T11:13:53.705Z|00148|netdev_dpdk|WARN|Error attaching device > 'mlx4_0' to DPDK > 2018-06-11T11:13:53.705Z|00149|netdev|WARN|enp59s0d1: could not set > configuration (Invalid argument) > 2018-06-11T11:13:53.711Z|00150|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T11:13:53.711Z|00151|netdev_dpdk|WARN|Error attaching device > 'mlx4_0' to DPDK > 2018-06-11T11:13:53.711Z|00152|netdev|WARN|enp59s0d1: could not set > configuration (Invalid argument) > 2018-06-11T11:13:53.717Z|00153|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T11:13:53.717Z|00154|netdev_dpdk|WARN|Error attaching device > 'mlx4_0' to DPDK > 2018-06-11T11:13:53.717Z|00155|netdev|WARN|enp59s0d1: could not set > configuration (Invalid argument) > 2018-06-11T11:13:53.728Z|00156|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T11:13:53.728Z|00157|netdev_dpdk|WARN|Error attaching device > 'mlx4_0' to DPDK > 2018-06-11T11:13:53.728Z|00158|netdev|WARN|enp59s0d1: could not set > configuration (Invalid argument) > 2018-06-11T11:13:53.743Z|00159|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T11:13:53.743Z|00160|netdev_dpdk|WARN|Error attaching device > 'mlx4_0' to DPDK > 2018-06-11T11:13:53.743Z|00161|netdev|WARN|enp59s0d1: could not set > configuration (Invalid argument) > 2018-06-11T11:13:53.749Z|00162|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T11:13:53.749Z|00163|netdev_dpdk|WARN|Error attaching device > 'mlx4_0' to DPDK > 2018-06-11T11:13:53.749Z|00164|netdev|WARN|enp59s0d1: could not set > configuration (Invalid argument) > 2018-06-11T11:13:53.756Z|00165|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T11:13:53.756Z|00166|netdev_dpdk|WARN|Error attaching device > 'mlx4_0' to DPDK > 2018-06-11T11:13:53.756Z|00167|netdev|WARN|enp59s0d1: could not set > configuration (Invalid argument) > 2018-06-11T11:13:53.766Z|00168|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' > 2018-06-11T11:13:53.766Z|00169|netdev_dpdk|WARN|Error attaching device > 'mlx4_0' to DPDK > 2018-06-11T11:13:53.766Z|00170|netdev|WARN|enp59s0d1: could not set > configuration (Invalid argument) > 2018-06-11T11:13:55.270Z|00171|memory|INFO|1406100 kB peak resident > set size after 10.0 seconds > 2018-06-11T11:13:55.270Z|00172|memory|INFO|handlers:118 ofconns:3 > ports:8 revalidators:42 rules:36 udpif keys:1 > 2018-06-11T11:14:03.699Z|00173|connmgr|INFO|br-int<->tcp:127.0.0.1:6633: > 5 flow_mods 10 s ago (5 adds) > 2018-06-11T11:14:03.737Z|00174|connmgr|INFO|br-ex<->tcp:127.0.0.1:6633: > 2 flow_mods 10 s ago (2 adds) > 2018-06-11T11:14:03.773Z|00175|connmgr|INFO|br-tun<->tcp:127.0.0.1:6633: > 12 flow_mods 10 s ago (12 adds) > > Best regards > > On 06/11/2018 12:57 PM, Raslan Darawsheh wrote: >> What I really want to see is the init process for OVS-DPDK that dumps the DPDK initialization. >> I think you will find it at the beginning of the log that you just sent. >> >> Kindest regards, >> Raslan Darawsheh >> >> -----Original Message----- >> From: Tobias Urdin [mailto:tobias.urdin@crystone.com] >> Sent: Monday, June 11, 2018 1:00 PM >> To: Raslan Darawsheh <rasland@mellanox.com> >> Cc: users@dpdk.org >> Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch >> >> Hello Raslan, >> Thank you for your reply. >> >> For reference the br-mlx bridge was created like this: >> $ovs-vsctl add-br br-mlx -- set bridge br-mlx datapath_type=netdev >> >> $ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 >> type=dpdk options:dpdk-devargs=mlx4_0 >> ovs-vsctl: Error detected while setting up 'enp59s0d1': Error attaching device 'mlx4_0' to DPDK. See ovs-vswitchd log for details. >> ovs-vsctl: The default log directory is "/var/log/openvswitch". >> >> In the ovs-vswitchd.log >> 2018-06-11T09:56:20.613Z|00185|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T09:56:20.613Z|00186|netdev_dpdk|WARN|Error attaching >> device 'mlx4_0' to DPDK >> 2018-06-11T09:56:20.613Z|00187|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> >> Here is the complete log after restarting openvswitch-switch service: >> 2018-06-11T09:57:49.380Z|00134|bridge|INFO|ovs-vswitchd (Open >> vSwitch) >> 2.9.0 >> 2018-06-11T09:57:52.812Z|00135|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T09:57:52.812Z|00136|netdev_dpdk|WARN|Error attaching >> device 'mlx4_0' to DPDK >> 2018-06-11T09:57:52.812Z|00137|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T09:57:52.825Z|00138|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T09:57:52.825Z|00139|netdev_dpdk|WARN|Error attaching >> device 'mlx4_0' to DPDK >> 2018-06-11T09:57:52.825Z|00140|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T09:57:52.832Z|00141|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T09:57:52.832Z|00142|netdev_dpdk|WARN|Error attaching >> device 'mlx4_0' to DPDK >> 2018-06-11T09:57:52.832Z|00143|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T09:57:52.843Z|00144|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T09:57:52.843Z|00145|netdev_dpdk|WARN|Error attaching >> device 'mlx4_0' to DPDK >> 2018-06-11T09:57:52.843Z|00146|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T09:57:52.856Z|00147|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T09:57:52.856Z|00148|netdev_dpdk|WARN|Error attaching >> device 'mlx4_0' to DPDK >> 2018-06-11T09:57:52.856Z|00149|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T09:57:52.862Z|00150|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T09:57:52.862Z|00151|netdev_dpdk|WARN|Error attaching >> device 'mlx4_0' to DPDK >> 2018-06-11T09:57:52.862Z|00152|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T09:57:52.870Z|00153|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T09:57:52.870Z|00154|netdev_dpdk|WARN|Error attaching >> device 'mlx4_0' to DPDK >> 2018-06-11T09:57:52.870Z|00155|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T09:57:52.889Z|00156|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T09:57:52.889Z|00157|netdev_dpdk|WARN|Error attaching >> device 'mlx4_0' to DPDK >> 2018-06-11T09:57:52.889Z|00158|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T09:57:52.903Z|00159|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T09:57:52.903Z|00160|netdev_dpdk|WARN|Error attaching >> device 'mlx4_0' to DPDK >> 2018-06-11T09:57:52.903Z|00161|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T09:57:52.908Z|00162|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T09:57:52.908Z|00163|netdev_dpdk|WARN|Error attaching >> device 'mlx4_0' to DPDK >> 2018-06-11T09:57:52.908Z|00164|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T09:57:52.914Z|00165|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T09:57:52.914Z|00166|netdev_dpdk|WARN|Error attaching >> device 'mlx4_0' to DPDK >> 2018-06-11T09:57:52.914Z|00167|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T09:57:52.923Z|00168|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T09:57:52.923Z|00169|netdev_dpdk|WARN|Error attaching >> device 'mlx4_0' to DPDK >> 2018-06-11T09:57:52.923Z|00170|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T09:57:54.808Z|00171|memory|INFO|1406100 kB peak resident >> set size after 10.1 seconds >> 2018-06-11T09:57:54.808Z|00172|memory|INFO|handlers:118 ofconns:3 >> ports:8 revalidators:42 rules:36 udpif keys:1 >> 2018-06-11T09:58:02.850Z|00173|connmgr|INFO|br-int<->tcp:127.0.0.1:6633: >> 5 flow_mods 10 s ago (5 adds) >> 2018-06-11T09:58:02.898Z|00174|connmgr|INFO|br-ex<->tcp:127.0.0.1:6633: >> 2 flow_mods 10 s ago (2 adds) >> 2018-06-11T09:58:02.930Z|00175|connmgr|INFO|br-tun<->tcp:127.0.0.1:6633: >> 12 flow_mods 10 s ago (12 adds) >> >> Was that what you mean by init log? >> Best regards >> >> On 06/11/2018 11:51 AM, Raslan Darawsheh wrote: >>> Hi Tobias, >>> >>> Can you try using the following ? >>> >>> ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 >>> type=dpdk options:dpdk-devargs=mlx4_0 >>> >>> if it didn't work can you send the OVS-DPDK init log ? >>> >>> Kindest regards, >>> Raslan Darawsheh >>> >>> -----Original Message----- >>> From: users [mailto:users-bounces@dpdk.org] On Behalf Of Tobias >>> Urdin >>> Sent: Monday, June 11, 2018 12:15 PM >>> To: users@dpdk.org >>> Subject: [dpdk-users] mellanox connect-x 3 pro dpdk with openvswitch >>> >>> Hello, >>> >>> I'm stuck trying to test DPDK with a Mellanox Connect-X 3 Pro card >>> on Ubuntu 16.04 >>> >>> I've enabled iommu, fixed hugepages and has installed and are running the openvswitch-switch-dpdk alternative. >>> >>> $lspci | grep -i mellanox >>> 3b:00.0 Ethernet controller: Mellanox Technologies MT27520 Family >>> [ConnectX-3 Pro] >>> >>> $lsmod | grep -i mlx >>> mlx4_en 110592 0 >>> vxlan 49152 2 i40e,mlx4_en >>> mlx4_core 286720 1 mlx4_en >>> ptp 20480 3 igb,i40e,mlx4_en >>> >>> $cat /etc/modprobe.d/mlx4_core.conf >>> options mlx4_core log_num_mgm_entry_size=-1 >>> >>> >>> $ovs-vsctl add-port br-mlx dpdk0 -- set Interface dpdk0 type=dpdk >>> >>> Bridge br-mlx >>> Port "dpdk0" >>> Interface "dpdk0" >>> type: dpdk >>> error: "'dpdk0' is missing 'options:dpdk-devargs'. The old 'dpdk<port_id>' names are not supported" >>> >>> >>> $ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 >>> type=dpdk options:dpdk-devargs=0000:3b:00.0,1 >>> ovs-vsctl: Error detected while setting up 'enp59s0d1': Error attaching device '0000:3b:00.0,1' to DPDK. See ovs-vswitchd log for details. >>> ovs-vsctl: The default log directory is "/var/log/openvswitch". >>> >>> 2018-06-11T09:10:38.696Z|00177|netdev_dpdk|WARN|Error attaching >>> device '0000:3b:00.0,1' to DPDK >>> 2018-06-11T09:10:38.696Z|00178|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> >>> 2018-06-11T09:12:50.994Z|00181|dpdk|ERR|EAL: Driver cannot attach >>> the device (0000:3b:00.0) >>> 2018-06-11T09:12:50.994Z|00182|dpdk|ERR|EAL: No port found for >>> device >>> (0000:3b:00.0) 2018-06-11T09:12:50.994Z|00183|netdev_dpdk|WARN|Error >>> attaching device '0000:3b:00.0,1' to DPDK >>> 2018-06-11T09:12:50.994Z|00184|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> >>> $ovs-vsctl add-port br-mlx dpdk0 -- set Interface dpdk0 type=dpdk options:dpdk-devargs="class=eth,mac=ec:0d:9a:b0:13:21" >>> ovs-vsctl: Error detected while setting up 'dpdk0': Error attaching device 'class=eth,mac=ec:0d:9a:b0:13:21' to DPDK. See ovs-vswitchd log for details. >>> ovs-vsctl: The default log directory is "/var/log/openvswitch". >>> >>> Port "dpdk0" >>> Interface "dpdk0" >>> type: dpdk >>> options: {dpdk-devargs="class=eth,mac=ec:0d:9a:b0:13:21"} >>> error: "Error attaching device 'class=eth,mac=ec:0d:9a:b0:13:21' to DPDK" >>> >>> >>> Anybody know what I might have missed? I have tried restarting the openvswitch services after adding the interfaces as well but no difference. >>> >>> The logs doesn't say anything interesting from what I can see :( >>> >>> Best regards >>> > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [dpdk-users] mellanox connect-x 3 pro dpdk with openvswitch 2018-06-11 11:37 ` Raslan Darawsheh @ 2018-06-11 11:45 ` Tobias Urdin 2018-06-11 11:51 ` Raslan Darawsheh 0 siblings, 1 reply; 14+ messages in thread From: Tobias Urdin @ 2018-06-11 11:45 UTC (permalink / raw) To: Raslan Darawsheh; +Cc: users Is there any to skip compiling DPDK and OVS+OVS-DPDK and instead use the ones I have installed? Only compiling the Mellanox PMD module/binary or something similar to that? I can see that other drivers supply pmd packages like below. $dpkg -l | grep -E "openvswitch|pmd|dpdk" ii dpdk 17.11.1-6~cloud0 amd64 Data Plane Development Kit (runtime) ii librte-pmd-af-packet17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-af-packet runtime library) ii librte-pmd-ark17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte_pmd_ark runtime library) ii librte-pmd-avp17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte_pmd_avp runtime library) ii librte-pmd-bnxt17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-bnxt runtime library) ii librte-pmd-bond17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-bond runtime library) ii librte-pmd-crypto-scheduler17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte_pmd_crypto_scheduler runtime library) ii librte-pmd-cxgbe17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-cxgbe runtime library) ii librte-pmd-e1000-17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-e1000 runtime library) ii librte-pmd-ena17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-ena runtime library) ii librte-pmd-enic17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-enic runtime library) ii librte-pmd-failsafe17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-failsafe runtime library) ii librte-pmd-fm10k17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-fm10k runtime library) ii librte-pmd-i40e17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-i40e runtime library) ii librte-pmd-ixgbe17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-ixgbe runtime library) ii librte-pmd-kni17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte_pmd_kni runtime library) ii librte-pmd-lio17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte_pmd_lio runtime library) ii librte-pmd-nfp17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte_pmd_nfp runtime library) ii librte-pmd-null-crypto17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-null-crypto runtime library) ii librte-pmd-null17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-null runtime library) ii librte-pmd-octeontx-ssovf17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte_pmd_octeontx_ssovf runtime library) ii librte-pmd-octeontx17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte_pmd_octeontx runtime library) ii librte-pmd-pcap17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-pcap runtime library) ii librte-pmd-qede17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-qede runtime library) ii librte-pmd-ring17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-ring runtime library) ii librte-pmd-sfc-efx17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte_pmd_sfc_efx runtime library) ii librte-pmd-skeleton-event17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte_pmd_skeleton_event runtime library) ii librte-pmd-softnic17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-softnic runtime library) ii librte-pmd-sw-event17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte_pmd_sw_event runtime library) ii librte-pmd-tap17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte_pmd_tap runtime library) ii librte-pmd-thunderx-nicvf17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte_pmd_thunderx_nicvf runtime library) ii librte-pmd-vhost17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-vhost runtime library) ii librte-pmd-virtio17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-virtio runtime library) ii librte-pmd-vmxnet3-uio17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-vmxnet3-uio runtime library) ii openvswitch-common 2.9.0-0ubuntu1~cloud0 amd64 Open vSwitch common components ii openvswitch-switch 2.9.0-0ubuntu1~cloud0 amd64 Open vSwitch switch implementations ii openvswitch-switch-dpdk 2.9.0-0ubuntu1~cloud0 amd64 DPDK enabled Open vSwitch switch implementation ii python-openvswitch 2.9.0-0ubuntu1~cloud0 all Python bindings for Open vSwitch Thanks for all the help! Best regards On 06/11/2018 01:37 PM, Raslan Darawsheh wrote: > You can refer to this community page for compiling OVS-DPDK with MLX pmds enabled: > https://community.mellanox.com/docs/DOC-2637 > > > Kindest regards, > Raslan Darawsheh > > -----Original Message----- > From: Tobias Urdin [mailto:tobias.urdin@crystone.com] > Sent: Monday, June 11, 2018 2:35 PM > To: Raslan Darawsheh <rasland@mellanox.com> > Cc: users@dpdk.org > Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch > > I haven't compiled any DPDK or Mellanox related modules/packages. > What I have is ovs, ovs-dpdk, mlx4 module, I have also loaded the ib_uverbs driver. > > Is it the librte_pmd_mlx4 I have to compile manually? > > Do you know any updated documentation about this, there isn't much relevant and searching for documentation and the correct mlnx driver and code is a struggle. > > Best regards > > On 06/11/2018 01:20 PM, Raslan Darawsheh wrote: >> From what I'm seeing it looks like it didn't prop the MLX4 device at all. >> Can you make sure you have it enabled in your DPDK package when you compiled DPDK ? >> >> Kindest regards, >> Raslan Darawsheh >> >> -----Original Message----- >> From: Tobias Urdin [mailto:tobias.urdin@crystone.com] >> Sent: Monday, June 11, 2018 2:16 PM >> To: Raslan Darawsheh <rasland@mellanox.com> >> Cc: users@dpdk.org >> Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch >> >> Ok, here is the complete log when restarting the openvswitch-switch service, see below. >> >> Also here is the kernel for information: >> $uname -r >> 4.4.0-127-generic >> >> 2018-06-11T11:13:42.729Z|00176|bridge|INFO|bridge br-mlx: deleted >> interface br-mlx on port 65534 >> 2018-06-11T11:13:43.274Z|00177|bridge|INFO|bridge br-int: deleted >> interface br-int on port 65534 >> 2018-06-11T11:13:43.275Z|00178|bridge|INFO|bridge br-int: deleted >> interface int-br-ex on port 1 >> 2018-06-11T11:13:43.275Z|00179|bridge|INFO|bridge br-int: deleted >> interface patch-tun on port 2 >> 2018-06-11T11:13:43.762Z|00180|bridge|INFO|bridge br-tun: deleted >> interface patch-int on port 1 >> 2018-06-11T11:13:43.762Z|00181|bridge|INFO|bridge br-tun: deleted >> interface br-tun on port 65534 >> 2018-06-11T11:13:44.231Z|00182|bridge|INFO|bridge br-ex: deleted >> interface br-ex on port 65534 >> 2018-06-11T11:13:44.232Z|00183|bridge|INFO|bridge br-ex: deleted >> interface phy-br-ex on port 1 >> 2018-06-11T11:13:45.245Z|00001|vlog|INFO|opened log file >> /var/log/openvswitch/ovs-vswitchd.log >> 2018-06-11T11:13:45.252Z|00002|ovs_numa|INFO|Discovered 40 CPU cores >> on NUMA node 0 2018-06-11T11:13:45.252Z|00003|ovs_numa|INFO|Discovered >> 40 CPU cores on NUMA node 1 >> 2018-06-11T11:13:45.252Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes >> and >> 80 CPU cores >> 2018-06-11T11:13:45.252Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: >> connecting... >> 2018-06-11T11:13:45.252Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: >> connected >> 2018-06-11T11:13:45.256Z|00007|dpdk|INFO|Using DPDK 17.11.0 2018-06-11T11:13:45.256Z|00008|dpdk|INFO|DPDK Enabled - initializing... >> 2018-06-11T11:13:45.256Z|00009|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch 2018-06-11T11:13:45.256Z|00010|dpdk|INFO|IOMMU support for vhost-user-client disabled. >> 2018-06-11T11:13:45.256Z|00011|dpdk|INFO|EAL ARGS: ovs-vswitchd >> --socket-mem 1024,0 -c 0x00000001 >> 2018-06-11T11:13:45.258Z|00012|dpdk|INFO|EAL: Detected 80 lcore(s) >> 2018-06-11T11:13:45.285Z|00013|dpdk|INFO|EAL: Probing VFIO support... >> 2018-06-11T11:13:45.285Z|00014|dpdk|INFO|EAL: VFIO support initialized >> 2018-06-11T11:13:48.214Z|00015|dpdk|INFO|EAL: PCI device 0000:01:00.0 on NUMA socket 0 >> 2018-06-11T11:13:48.214Z|00016|dpdk|INFO|EAL: probe driver: 8086:1521 >> net_e1000_igb >> 2018-06-11T11:13:48.214Z|00017|dpdk|INFO|EAL: PCI device 0000:01:00.1 on NUMA socket 0 >> 2018-06-11T11:13:48.214Z|00018|dpdk|INFO|EAL: probe driver: 8086:1521 >> net_e1000_igb >> 2018-06-11T11:13:48.214Z|00019|dpdk|INFO|EAL: PCI device 0000:18:00.0 on NUMA socket 0 >> 2018-06-11T11:13:48.214Z|00020|dpdk|INFO|EAL: probe driver: 8086:1572 >> net_i40e >> 2018-06-11T11:13:48.214Z|00021|dpdk|INFO|EAL: PCI device 0000:18:00.1 on NUMA socket 0 >> 2018-06-11T11:13:48.214Z|00022|dpdk|INFO|EAL: probe driver: 8086:1572 >> net_i40e >> 2018-06-11T11:13:48.214Z|00023|dpdk|INFO|DPDK pdump packet capture >> enabled 2018-06-11T11:13:48.224Z|00024|dpdk|INFO|DPDK Enabled - >> initialized >> 2018-06-11T11:13:48.235Z|00025|ofproto_dpif|INFO|netdev@ovs-netdev: >> Datapath supports recirculation >> 2018-06-11T11:13:48.235Z|00026|ofproto_dpif|INFO|netdev@ovs-netdev: >> VLAN header stack length probed as 1 >> 2018-06-11T11:13:48.235Z|00027|ofproto_dpif|INFO|netdev@ovs-netdev: >> MPLS label stack length probed as 3 >> 2018-06-11T11:13:48.235Z|00028|ofproto_dpif|INFO|netdev@ovs-netdev: >> Datapath supports truncate action >> 2018-06-11T11:13:48.235Z|00029|ofproto_dpif|INFO|netdev@ovs-netdev: >> Datapath supports unique flow ids >> 2018-06-11T11:13:48.235Z|00030|ofproto_dpif|INFO|netdev@ovs-netdev: >> Datapath supports clone action >> 2018-06-11T11:13:48.235Z|00031|ofproto_dpif|INFO|netdev@ovs-netdev: >> Max sample nesting level probed as 10 >> 2018-06-11T11:13:48.235Z|00032|ofproto_dpif|INFO|netdev@ovs-netdev: >> Datapath supports eventmask in conntrack action >> 2018-06-11T11:13:48.235Z|00033|ofproto_dpif|INFO|netdev@ovs-netdev: >> Datapath supports ct_clear action >> 2018-06-11T11:13:48.235Z|00034|ofproto_dpif|INFO|netdev@ovs-netdev: >> Datapath supports ct_state >> 2018-06-11T11:13:48.235Z|00035|ofproto_dpif|INFO|netdev@ovs-netdev: >> Datapath supports ct_zone >> 2018-06-11T11:13:48.235Z|00036|ofproto_dpif|INFO|netdev@ovs-netdev: >> Datapath supports ct_mark >> 2018-06-11T11:13:48.235Z|00037|ofproto_dpif|INFO|netdev@ovs-netdev: >> Datapath supports ct_label >> 2018-06-11T11:13:48.235Z|00038|ofproto_dpif|INFO|netdev@ovs-netdev: >> Datapath supports ct_state_nat >> 2018-06-11T11:13:48.235Z|00039|ofproto_dpif|INFO|netdev@ovs-netdev: >> Datapath supports ct_orig_tuple >> 2018-06-11T11:13:48.235Z|00040|ofproto_dpif|INFO|netdev@ovs-netdev: >> Datapath supports ct_orig_tuple6 >> 2018-06-11T11:13:48.605Z|00041|ofproto_dpif|INFO|system@ovs-system: >> Datapath supports recirculation >> 2018-06-11T11:13:48.605Z|00042|ofproto_dpif|INFO|system@ovs-system: >> VLAN header stack length probed as 1 >> 2018-06-11T11:13:48.605Z|00043|ofproto_dpif|INFO|system@ovs-system: >> MPLS label stack length probed as 1 >> 2018-06-11T11:13:48.605Z|00044|ofproto_dpif|INFO|system@ovs-system: >> Datapath does not support truncate action >> 2018-06-11T11:13:48.605Z|00045|ofproto_dpif|INFO|system@ovs-system: >> Datapath supports unique flow ids >> 2018-06-11T11:13:48.605Z|00046|ofproto_dpif|INFO|system@ovs-system: >> Datapath does not support clone action >> 2018-06-11T11:13:48.605Z|00047|ofproto_dpif|INFO|system@ovs-system: >> Max sample nesting level probed as 3 >> 2018-06-11T11:13:48.605Z|00048|ofproto_dpif|INFO|system@ovs-system: >> Datapath does not support eventmask in conntrack action >> 2018-06-11T11:13:48.605Z|00049|ofproto_dpif|INFO|system@ovs-system: >> Datapath does not support ct_clear action >> 2018-06-11T11:13:48.605Z|00050|ofproto_dpif|INFO|system@ovs-system: >> Datapath supports ct_state >> 2018-06-11T11:13:48.605Z|00051|ofproto_dpif|INFO|system@ovs-system: >> Datapath supports ct_zone >> 2018-06-11T11:13:48.605Z|00052|ofproto_dpif|INFO|system@ovs-system: >> Datapath supports ct_mark >> 2018-06-11T11:13:48.605Z|00053|ofproto_dpif|INFO|system@ovs-system: >> Datapath supports ct_label >> 2018-06-11T11:13:48.605Z|00054|ofproto_dpif|INFO|system@ovs-system: >> Datapath does not support ct_state_nat >> 2018-06-11T11:13:48.605Z|00055|ofproto_dpif|INFO|system@ovs-system: >> Datapath does not support ct_orig_tuple >> 2018-06-11T11:13:48.605Z|00056|ofproto_dpif|INFO|system@ovs-system: >> Datapath does not support ct_orig_tuple6 >> 2018-06-11T11:13:48.981Z|00057|bridge|INFO|bridge br-mlx: added >> interface br-mlx on port 65534 >> 2018-06-11T11:13:48.981Z|00058|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T11:13:48.981Z|00059|netdev_dpdk|WARN|Error attaching device >> 'mlx4_0' to DPDK >> 2018-06-11T11:13:48.981Z|00060|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T11:13:48.982Z|00061|bridge|INFO|bridge br-int: added >> interface br-int on port 65534 >> 2018-06-11T11:13:48.982Z|00062|bridge|INFO|bridge br-int: added >> interface int-br-ex on port 1 >> 2018-06-11T11:13:48.982Z|00063|bridge|INFO|bridge br-int: added >> interface patch-tun on port 2 >> 2018-06-11T11:13:48.982Z|00064|bridge|INFO|bridge br-tun: added >> interface patch-int on port 1 >> 2018-06-11T11:13:48.982Z|00065|bridge|INFO|bridge br-tun: added >> interface br-tun on port 65534 >> 2018-06-11T11:13:48.982Z|00066|bridge|INFO|bridge br-ex: added >> interface br-ex on port 65534 >> 2018-06-11T11:13:48.982Z|00067|bridge|INFO|bridge br-ex: added >> interface phy-br-ex on port 1 >> 2018-06-11T11:13:48.982Z|00068|bridge|INFO|bridge br-mlx: using >> datapath ID 000072528484a642 >> 2018-06-11T11:13:48.982Z|00069|connmgr|INFO|br-mlx: added service controller "punix:/var/run/openvswitch/br-mlx.mgmt" >> 2018-06-11T11:13:49.221Z|00070|bridge|INFO|bridge br-int: using >> datapath ID 00006680927b6b40 >> 2018-06-11T11:13:49.221Z|00071|connmgr|INFO|br-int: added service controller "punix:/var/run/openvswitch/br-int.mgmt" >> 2018-06-11T11:13:49.221Z|00072|connmgr|INFO|br-int: added primary controller "tcp:127.0.0.1:6633" >> 2018-06-11T11:13:49.221Z|00073|rconn|INFO|br-int<->tcp:127.0.0.1:6633: >> connecting... >> 2018-06-11T11:13:49.460Z|00074|bridge|INFO|bridge br-tun: using >> datapath ID 000066532e5f0443 >> 2018-06-11T11:13:49.461Z|00075|connmgr|INFO|br-tun: added service controller "punix:/var/run/openvswitch/br-tun.mgmt" >> 2018-06-11T11:13:49.461Z|00076|connmgr|INFO|br-tun: added primary controller "tcp:127.0.0.1:6633" >> 2018-06-11T11:13:49.461Z|00077|rconn|INFO|br-tun<->tcp:127.0.0.1:6633: >> connecting... >> 2018-06-11T11:13:49.692Z|00078|bridge|INFO|bridge br-ex: using >> datapath ID 0000e2d1bdbe8e45 >> 2018-06-11T11:13:49.692Z|00079|connmgr|INFO|br-ex: added service controller "punix:/var/run/openvswitch/br-ex.mgmt" >> 2018-06-11T11:13:49.692Z|00080|connmgr|INFO|br-ex: added primary controller "tcp:127.0.0.1:6633" >> 2018-06-11T11:13:49.692Z|00081|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: >> connecting... >> 2018-06-11T11:13:49.916Z|00082|timeval|WARN|Unreasonably long 4661ms >> poll interval (16ms user, 4472ms system) >> 2018-06-11T11:13:49.916Z|00083|timeval|WARN|faults: 3584706 minor, 0 >> major >> 2018-06-11T11:13:49.916Z|00084|timeval|WARN|disk: 0 reads, 16 writes 2018-06-11T11:13:49.916Z|00085|timeval|WARN|context switches: 147 voluntary, 25 involuntary 2018-06-11T11:13:49.916Z|00086|coverage|INFO|Event coverage, avg rate over last: 5 seconds, last minute, last hour, hash=2ba35161: >> 2018-06-11T11:13:49.916Z|00087|coverage|INFO|bridge_reconfigure >> 0.0/sec 0.000/sec 0.0000/sec total: 1 >> 2018-06-11T11:13:49.916Z|00088|coverage|INFO|ofproto_flush >> 0.0/sec 0.000/sec 0.0000/sec total: 4 >> 2018-06-11T11:13:49.916Z|00089|coverage|INFO|ofproto_update_port >> 0.0/sec 0.000/sec 0.0000/sec total: 8 >> 2018-06-11T11:13:49.916Z|00090|coverage|INFO|rev_flow_table >> 0.0/sec 0.000/sec 0.0000/sec total: 1 >> 2018-06-11T11:13:49.916Z|00091|coverage|INFO|cmap_expand >> 0.0/sec 0.000/sec 0.0000/sec total: 52 >> 2018-06-11T11:13:49.916Z|00092|coverage|INFO|cmap_shrink >> 0.0/sec 0.000/sec 0.0000/sec total: 23 >> 2018-06-11T11:13:49.916Z|00093|coverage|INFO|dpif_port_add >> 0.0/sec 0.000/sec 0.0000/sec total: 1 >> 2018-06-11T11:13:49.916Z|00094|coverage|INFO|dpif_flow_flush >> 0.0/sec 0.000/sec 0.0000/sec total: 6 >> 2018-06-11T11:13:49.916Z|00095|coverage|INFO|dpif_flow_get >> 0.0/sec 0.000/sec 0.0000/sec total: 29 >> 2018-06-11T11:13:49.916Z|00096|coverage|INFO|dpif_flow_put >> 0.0/sec 0.000/sec 0.0000/sec total: 37 >> 2018-06-11T11:13:49.916Z|00097|coverage|INFO|dpif_flow_del >> 0.0/sec 0.000/sec 0.0000/sec total: 29 >> 2018-06-11T11:13:49.916Z|00098|coverage|INFO|dpif_execute >> 0.0/sec 0.000/sec 0.0000/sec total: 8 >> 2018-06-11T11:13:49.916Z|00099|coverage|INFO|flow_extract >> 0.0/sec 0.000/sec 0.0000/sec total: 6 >> 2018-06-11T11:13:49.916Z|00100|coverage|INFO|miniflow_malloc >> 0.0/sec 0.000/sec 0.0000/sec total: 34 >> 2018-06-11T11:13:49.916Z|00101|coverage|INFO|hmap_expand >> 0.0/sec 0.000/sec 0.0000/sec total: 498 >> 2018-06-11T11:13:49.916Z|00102|coverage|INFO|netdev_received >> 0.0/sec 0.000/sec 0.0000/sec total: 1 >> 2018-06-11T11:13:49.916Z|00103|coverage|INFO|netdev_get_stats >> 0.0/sec 0.000/sec 0.0000/sec total: 8 >> 2018-06-11T11:13:49.916Z|00104|coverage|INFO|poll_create_node >> 0.0/sec 0.000/sec 0.0000/sec total: 23 >> 2018-06-11T11:13:49.916Z|00105|coverage|INFO|seq_change >> 0.0/sec 0.000/sec 0.0000/sec total: 2193 >> 2018-06-11T11:13:49.916Z|00106|coverage|INFO|pstream_open >> 0.0/sec 0.000/sec 0.0000/sec total: 9 >> 2018-06-11T11:13:49.916Z|00107|coverage|INFO|stream_open >> 0.0/sec 0.000/sec 0.0000/sec total: 4 >> 2018-06-11T11:13:49.916Z|00108|coverage|INFO|util_xalloc >> 0.0/sec 0.000/sec 0.0000/sec total: 12803 >> 2018-06-11T11:13:49.916Z|00109|coverage|INFO|vconn_open >> 0.0/sec 0.000/sec 0.0000/sec total: 3 >> 2018-06-11T11:13:49.916Z|00110|coverage|INFO|netdev_set_policing >> 0.0/sec 0.000/sec 0.0000/sec total: 4 >> 2018-06-11T11:13:49.916Z|00111|coverage|INFO|netdev_get_ifindex >> 0.0/sec 0.000/sec 0.0000/sec total: 4 >> 2018-06-11T11:13:49.916Z|00112|coverage|INFO|netdev_get_hwaddr >> 0.0/sec 0.000/sec 0.0000/sec total: 9 >> 2018-06-11T11:13:49.916Z|00113|coverage|INFO|netdev_get_ethtool >> 0.0/sec 0.000/sec 0.0000/sec total: 2 >> 2018-06-11T11:13:49.916Z|00114|coverage|INFO|netlink_received >> 0.0/sec 0.000/sec 0.0000/sec total: 85 >> 2018-06-11T11:13:49.916Z|00115|coverage|INFO|netlink_recv_jumbo >> 0.0/sec 0.000/sec 0.0000/sec total: 1 >> 2018-06-11T11:13:49.916Z|00116|coverage|INFO|netlink_sent >> 0.0/sec 0.000/sec 0.0000/sec total: 102 >> 2018-06-11T11:13:49.916Z|00117|coverage|INFO|74 events never hit >> 2018-06-11T11:13:49.916Z|00118|poll_loop|INFO|wakeup due to [POLLIN] >> on fd 433 (FIFO pipe:[1025715]) at ../lib/ovs-rcu.c:229 (97% CPU >> usage) 2018-06-11T11:13:49.917Z|00119|poll_loop|INFO|wakeup due to >> [POLLIN] on fd 433 (FIFO pipe:[1025715]) at ../lib/ovs-rcu.c:229 (97% >> CPU usage) 2018-06-11T11:13:49.917Z|00120|poll_loop|INFO|wakeup due to >> 0-ms timeout at ../lib/ovs-rcu.c:229 (97% CPU usage) >> 2018-06-11T11:13:49.917Z|00121|poll_loop|INFO|wakeup due to 0-ms >> timeout at ../lib/ovs-rcu.c:229 (97% CPU usage) >> 2018-06-11T11:13:49.920Z|00122|rconn|INFO|br-int<->tcp:127.0.0.1:6633: >> connected >> 2018-06-11T11:13:49.920Z|00123|rconn|INFO|br-tun<->tcp:127.0.0.1:6633: >> connected >> 2018-06-11T11:13:49.920Z|00124|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: >> connected >> 2018-06-11T11:13:49.926Z|00125|poll_loop|INFO|wakeup due to [POLLIN] >> on fd 10 (NETLINK_ROUTE<->NETLINK_ROUTE) at >> ../lib/netlink-socket.c:1331 (97% CPU usage) >> 2018-06-11T11:13:49.926Z|00126|poll_loop|INFO|wakeup due to [POLLIN] >> on fd 421 (character device /dev/net/tun) at >> ../lib/netdev-linux.c:1171 (97% CPU usage) >> 2018-06-11T11:13:49.926Z|00127|poll_loop|INFO|wakeup due to [POLLIN] >> on fd 423 (127.0.0.1:33712<->127.0.0.1:6633) at ../lib/stream-fd.c:157 >> (97% CPU usage) 2018-06-11T11:13:49.926Z|00128|poll_loop|INFO|wakeup >> due to [POLLIN] on fd 428 (127.0.0.1:33714<->127.0.0.1:6633) at >> ../lib/stream-fd.c:157 (97% CPU usage) >> 2018-06-11T11:13:49.926Z|00129|poll_loop|INFO|wakeup due to [POLLIN] >> on fd 11 (<->/var/run/openvswitch/db.sock) at ../lib/stream-fd.c:157 >> (97% CPU usage) 2018-06-11T11:13:49.926Z|00130|poll_loop|INFO|wakeup >> due to [POLLIN] on fd 429 (127.0.0.1:33716<->127.0.0.1:6633) at >> ../lib/stream-fd.c:157 (97% CPU usage) >> 2018-06-11T11:13:49.930Z|00131|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T11:13:49.930Z|00132|netdev_dpdk|WARN|Error attaching device >> 'mlx4_0' to DPDK >> 2018-06-11T11:13:49.930Z|00133|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T11:13:49.931Z|00134|bridge|INFO|ovs-vswitchd (Open vSwitch) >> 2.9.0 >> 2018-06-11T11:13:53.662Z|00135|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T11:13:53.662Z|00136|netdev_dpdk|WARN|Error attaching device >> 'mlx4_0' to DPDK >> 2018-06-11T11:13:53.662Z|00137|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T11:13:53.675Z|00138|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T11:13:53.675Z|00139|netdev_dpdk|WARN|Error attaching device >> 'mlx4_0' to DPDK >> 2018-06-11T11:13:53.675Z|00140|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T11:13:53.682Z|00141|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T11:13:53.682Z|00142|netdev_dpdk|WARN|Error attaching device >> 'mlx4_0' to DPDK >> 2018-06-11T11:13:53.682Z|00143|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T11:13:53.693Z|00144|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T11:13:53.693Z|00145|netdev_dpdk|WARN|Error attaching device >> 'mlx4_0' to DPDK >> 2018-06-11T11:13:53.693Z|00146|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T11:13:53.705Z|00147|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T11:13:53.705Z|00148|netdev_dpdk|WARN|Error attaching device >> 'mlx4_0' to DPDK >> 2018-06-11T11:13:53.705Z|00149|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T11:13:53.711Z|00150|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T11:13:53.711Z|00151|netdev_dpdk|WARN|Error attaching device >> 'mlx4_0' to DPDK >> 2018-06-11T11:13:53.711Z|00152|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T11:13:53.717Z|00153|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T11:13:53.717Z|00154|netdev_dpdk|WARN|Error attaching device >> 'mlx4_0' to DPDK >> 2018-06-11T11:13:53.717Z|00155|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T11:13:53.728Z|00156|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T11:13:53.728Z|00157|netdev_dpdk|WARN|Error attaching device >> 'mlx4_0' to DPDK >> 2018-06-11T11:13:53.728Z|00158|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T11:13:53.743Z|00159|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T11:13:53.743Z|00160|netdev_dpdk|WARN|Error attaching device >> 'mlx4_0' to DPDK >> 2018-06-11T11:13:53.743Z|00161|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T11:13:53.749Z|00162|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T11:13:53.749Z|00163|netdev_dpdk|WARN|Error attaching device >> 'mlx4_0' to DPDK >> 2018-06-11T11:13:53.749Z|00164|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T11:13:53.756Z|00165|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T11:13:53.756Z|00166|netdev_dpdk|WARN|Error attaching device >> 'mlx4_0' to DPDK >> 2018-06-11T11:13:53.756Z|00167|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T11:13:53.766Z|00168|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T11:13:53.766Z|00169|netdev_dpdk|WARN|Error attaching device >> 'mlx4_0' to DPDK >> 2018-06-11T11:13:53.766Z|00170|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T11:13:55.270Z|00171|memory|INFO|1406100 kB peak resident >> set size after 10.0 seconds >> 2018-06-11T11:13:55.270Z|00172|memory|INFO|handlers:118 ofconns:3 >> ports:8 revalidators:42 rules:36 udpif keys:1 >> 2018-06-11T11:14:03.699Z|00173|connmgr|INFO|br-int<->tcp:127.0.0.1:6633: >> 5 flow_mods 10 s ago (5 adds) >> 2018-06-11T11:14:03.737Z|00174|connmgr|INFO|br-ex<->tcp:127.0.0.1:6633: >> 2 flow_mods 10 s ago (2 adds) >> 2018-06-11T11:14:03.773Z|00175|connmgr|INFO|br-tun<->tcp:127.0.0.1:6633: >> 12 flow_mods 10 s ago (12 adds) >> >> Best regards >> >> On 06/11/2018 12:57 PM, Raslan Darawsheh wrote: >>> What I really want to see is the init process for OVS-DPDK that dumps the DPDK initialization. >>> I think you will find it at the beginning of the log that you just sent. >>> >>> Kindest regards, >>> Raslan Darawsheh >>> >>> -----Original Message----- >>> From: Tobias Urdin [mailto:tobias.urdin@crystone.com] >>> Sent: Monday, June 11, 2018 1:00 PM >>> To: Raslan Darawsheh <rasland@mellanox.com> >>> Cc: users@dpdk.org >>> Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch >>> >>> Hello Raslan, >>> Thank you for your reply. >>> >>> For reference the br-mlx bridge was created like this: >>> $ovs-vsctl add-br br-mlx -- set bridge br-mlx datapath_type=netdev >>> >>> $ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 >>> type=dpdk options:dpdk-devargs=mlx4_0 >>> ovs-vsctl: Error detected while setting up 'enp59s0d1': Error attaching device 'mlx4_0' to DPDK. See ovs-vswitchd log for details. >>> ovs-vsctl: The default log directory is "/var/log/openvswitch". >>> >>> In the ovs-vswitchd.log >>> 2018-06-11T09:56:20.613Z|00185|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T09:56:20.613Z|00186|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T09:56:20.613Z|00187|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> >>> Here is the complete log after restarting openvswitch-switch service: >>> 2018-06-11T09:57:49.380Z|00134|bridge|INFO|ovs-vswitchd (Open >>> vSwitch) >>> 2.9.0 >>> 2018-06-11T09:57:52.812Z|00135|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T09:57:52.812Z|00136|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T09:57:52.812Z|00137|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T09:57:52.825Z|00138|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T09:57:52.825Z|00139|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T09:57:52.825Z|00140|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T09:57:52.832Z|00141|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T09:57:52.832Z|00142|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T09:57:52.832Z|00143|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T09:57:52.843Z|00144|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T09:57:52.843Z|00145|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T09:57:52.843Z|00146|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T09:57:52.856Z|00147|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T09:57:52.856Z|00148|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T09:57:52.856Z|00149|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T09:57:52.862Z|00150|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T09:57:52.862Z|00151|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T09:57:52.862Z|00152|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T09:57:52.870Z|00153|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T09:57:52.870Z|00154|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T09:57:52.870Z|00155|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T09:57:52.889Z|00156|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T09:57:52.889Z|00157|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T09:57:52.889Z|00158|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T09:57:52.903Z|00159|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T09:57:52.903Z|00160|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T09:57:52.903Z|00161|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T09:57:52.908Z|00162|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T09:57:52.908Z|00163|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T09:57:52.908Z|00164|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T09:57:52.914Z|00165|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T09:57:52.914Z|00166|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T09:57:52.914Z|00167|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T09:57:52.923Z|00168|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T09:57:52.923Z|00169|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T09:57:52.923Z|00170|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T09:57:54.808Z|00171|memory|INFO|1406100 kB peak resident >>> set size after 10.1 seconds >>> 2018-06-11T09:57:54.808Z|00172|memory|INFO|handlers:118 ofconns:3 >>> ports:8 revalidators:42 rules:36 udpif keys:1 >>> 2018-06-11T09:58:02.850Z|00173|connmgr|INFO|br-int<->tcp:127.0.0.1:6633: >>> 5 flow_mods 10 s ago (5 adds) >>> 2018-06-11T09:58:02.898Z|00174|connmgr|INFO|br-ex<->tcp:127.0.0.1:6633: >>> 2 flow_mods 10 s ago (2 adds) >>> 2018-06-11T09:58:02.930Z|00175|connmgr|INFO|br-tun<->tcp:127.0.0.1:6633: >>> 12 flow_mods 10 s ago (12 adds) >>> >>> Was that what you mean by init log? >>> Best regards >>> >>> On 06/11/2018 11:51 AM, Raslan Darawsheh wrote: >>>> Hi Tobias, >>>> >>>> Can you try using the following ? >>>> >>>> ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 >>>> type=dpdk options:dpdk-devargs=mlx4_0 >>>> >>>> if it didn't work can you send the OVS-DPDK init log ? >>>> >>>> Kindest regards, >>>> Raslan Darawsheh >>>> >>>> -----Original Message----- >>>> From: users [mailto:users-bounces@dpdk.org] On Behalf Of Tobias >>>> Urdin >>>> Sent: Monday, June 11, 2018 12:15 PM >>>> To: users@dpdk.org >>>> Subject: [dpdk-users] mellanox connect-x 3 pro dpdk with openvswitch >>>> >>>> Hello, >>>> >>>> I'm stuck trying to test DPDK with a Mellanox Connect-X 3 Pro card >>>> on Ubuntu 16.04 >>>> >>>> I've enabled iommu, fixed hugepages and has installed and are running the openvswitch-switch-dpdk alternative. >>>> >>>> $lspci | grep -i mellanox >>>> 3b:00.0 Ethernet controller: Mellanox Technologies MT27520 Family >>>> [ConnectX-3 Pro] >>>> >>>> $lsmod | grep -i mlx >>>> mlx4_en 110592 0 >>>> vxlan 49152 2 i40e,mlx4_en >>>> mlx4_core 286720 1 mlx4_en >>>> ptp 20480 3 igb,i40e,mlx4_en >>>> >>>> $cat /etc/modprobe.d/mlx4_core.conf >>>> options mlx4_core log_num_mgm_entry_size=-1 >>>> >>>> >>>> $ovs-vsctl add-port br-mlx dpdk0 -- set Interface dpdk0 type=dpdk >>>> >>>> Bridge br-mlx >>>> Port "dpdk0" >>>> Interface "dpdk0" >>>> type: dpdk >>>> error: "'dpdk0' is missing 'options:dpdk-devargs'. The old 'dpdk<port_id>' names are not supported" >>>> >>>> >>>> $ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 >>>> type=dpdk options:dpdk-devargs=0000:3b:00.0,1 >>>> ovs-vsctl: Error detected while setting up 'enp59s0d1': Error attaching device '0000:3b:00.0,1' to DPDK. See ovs-vswitchd log for details. >>>> ovs-vsctl: The default log directory is "/var/log/openvswitch". >>>> >>>> 2018-06-11T09:10:38.696Z|00177|netdev_dpdk|WARN|Error attaching >>>> device '0000:3b:00.0,1' to DPDK >>>> 2018-06-11T09:10:38.696Z|00178|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> >>>> 2018-06-11T09:12:50.994Z|00181|dpdk|ERR|EAL: Driver cannot attach >>>> the device (0000:3b:00.0) >>>> 2018-06-11T09:12:50.994Z|00182|dpdk|ERR|EAL: No port found for >>>> device >>>> (0000:3b:00.0) 2018-06-11T09:12:50.994Z|00183|netdev_dpdk|WARN|Error >>>> attaching device '0000:3b:00.0,1' to DPDK >>>> 2018-06-11T09:12:50.994Z|00184|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> >>>> $ovs-vsctl add-port br-mlx dpdk0 -- set Interface dpdk0 type=dpdk options:dpdk-devargs="class=eth,mac=ec:0d:9a:b0:13:21" >>>> ovs-vsctl: Error detected while setting up 'dpdk0': Error attaching device 'class=eth,mac=ec:0d:9a:b0:13:21' to DPDK. See ovs-vswitchd log for details. >>>> ovs-vsctl: The default log directory is "/var/log/openvswitch". >>>> >>>> Port "dpdk0" >>>> Interface "dpdk0" >>>> type: dpdk >>>> options: {dpdk-devargs="class=eth,mac=ec:0d:9a:b0:13:21"} >>>> error: "Error attaching device 'class=eth,mac=ec:0d:9a:b0:13:21' to DPDK" >>>> >>>> >>>> Anybody know what I might have missed? I have tried restarting the openvswitch services after adding the interfaces as well but no difference. >>>> >>>> The logs doesn't say anything interesting from what I can see :( >>>> >>>> Best regards >>>> > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [dpdk-users] mellanox connect-x 3 pro dpdk with openvswitch 2018-06-11 11:45 ` Tobias Urdin @ 2018-06-11 11:51 ` Raslan Darawsheh 2018-06-11 12:05 ` Tobias Urdin 0 siblings, 1 reply; 14+ messages in thread From: Raslan Darawsheh @ 2018-06-11 11:51 UTC (permalink / raw) To: Tobias Urdin; +Cc: users I think you can upgrade the current version that you have as seen in this article: http://docs.openvswitch.org/en/latest/intro/install/general/#hot-upgrading but, I haven't tried it before. Kindest regards, Raslan Darawsheh -----Original Message----- From: Tobias Urdin [mailto:tobias.urdin@crystone.com] Sent: Monday, June 11, 2018 2:45 PM To: Raslan Darawsheh <rasland@mellanox.com> Cc: users@dpdk.org Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch Is there any to skip compiling DPDK and OVS+OVS-DPDK and instead use the ones I have installed? Only compiling the Mellanox PMD module/binary or something similar to that? I can see that other drivers supply pmd packages like below. $dpkg -l | grep -E "openvswitch|pmd|dpdk" ii dpdk 17.11.1-6~cloud0 amd64 Data Plane Development Kit (runtime) ii librte-pmd-af-packet17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-af-packet runtime library) ii librte-pmd-ark17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte_pmd_ark runtime library) ii librte-pmd-avp17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte_pmd_avp runtime library) ii librte-pmd-bnxt17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-bnxt runtime library) ii librte-pmd-bond17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-bond runtime library) ii librte-pmd-crypto-scheduler17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte_pmd_crypto_scheduler runtime library) ii librte-pmd-cxgbe17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-cxgbe runtime library) ii librte-pmd-e1000-17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-e1000 runtime library) ii librte-pmd-ena17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-ena runtime library) ii librte-pmd-enic17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-enic runtime library) ii librte-pmd-failsafe17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-failsafe runtime library) ii librte-pmd-fm10k17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-fm10k runtime library) ii librte-pmd-i40e17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-i40e runtime library) ii librte-pmd-ixgbe17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-ixgbe runtime library) ii librte-pmd-kni17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte_pmd_kni runtime library) ii librte-pmd-lio17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte_pmd_lio runtime library) ii librte-pmd-nfp17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte_pmd_nfp runtime library) ii librte-pmd-null-crypto17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-null-crypto runtime library) ii librte-pmd-null17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-null runtime library) ii librte-pmd-octeontx-ssovf17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte_pmd_octeontx_ssovf runtime library) ii librte-pmd-octeontx17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte_pmd_octeontx runtime library) ii librte-pmd-pcap17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-pcap runtime library) ii librte-pmd-qede17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-qede runtime library) ii librte-pmd-ring17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-ring runtime library) ii librte-pmd-sfc-efx17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte_pmd_sfc_efx runtime library) ii librte-pmd-skeleton-event17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte_pmd_skeleton_event runtime library) ii librte-pmd-softnic17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-softnic runtime library) ii librte-pmd-sw-event17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte_pmd_sw_event runtime library) ii librte-pmd-tap17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte_pmd_tap runtime library) ii librte-pmd-thunderx-nicvf17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte_pmd_thunderx_nicvf runtime library) ii librte-pmd-vhost17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-vhost runtime library) ii librte-pmd-virtio17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-virtio runtime library) ii librte-pmd-vmxnet3-uio17.11:amd64 17.11.1-6~cloud0 amd64 Data Plane Development Kit (librte-pmd-vmxnet3-uio runtime library) ii openvswitch-common 2.9.0-0ubuntu1~cloud0 amd64 Open vSwitch common components ii openvswitch-switch 2.9.0-0ubuntu1~cloud0 amd64 Open vSwitch switch implementations ii openvswitch-switch-dpdk 2.9.0-0ubuntu1~cloud0 amd64 DPDK enabled Open vSwitch switch implementation ii python-openvswitch 2.9.0-0ubuntu1~cloud0 all Python bindings for Open vSwitch Thanks for all the help! Best regards On 06/11/2018 01:37 PM, Raslan Darawsheh wrote: > You can refer to this community page for compiling OVS-DPDK with MLX pmds enabled: > https://community.mellanox.com/docs/DOC-2637 > > > Kindest regards, > Raslan Darawsheh > > -----Original Message----- > From: Tobias Urdin [mailto:tobias.urdin@crystone.com] > Sent: Monday, June 11, 2018 2:35 PM > To: Raslan Darawsheh <rasland@mellanox.com> > Cc: users@dpdk.org > Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch > > I haven't compiled any DPDK or Mellanox related modules/packages. > What I have is ovs, ovs-dpdk, mlx4 module, I have also loaded the ib_uverbs driver. > > Is it the librte_pmd_mlx4 I have to compile manually? > > Do you know any updated documentation about this, there isn't much relevant and searching for documentation and the correct mlnx driver and code is a struggle. > > Best regards > > On 06/11/2018 01:20 PM, Raslan Darawsheh wrote: >> From what I'm seeing it looks like it didn't prop the MLX4 device at all. >> Can you make sure you have it enabled in your DPDK package when you compiled DPDK ? >> >> Kindest regards, >> Raslan Darawsheh >> >> -----Original Message----- >> From: Tobias Urdin [mailto:tobias.urdin@crystone.com] >> Sent: Monday, June 11, 2018 2:16 PM >> To: Raslan Darawsheh <rasland@mellanox.com> >> Cc: users@dpdk.org >> Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch >> >> Ok, here is the complete log when restarting the openvswitch-switch service, see below. >> >> Also here is the kernel for information: >> $uname -r >> 4.4.0-127-generic >> >> 2018-06-11T11:13:42.729Z|00176|bridge|INFO|bridge br-mlx: deleted >> interface br-mlx on port 65534 >> 2018-06-11T11:13:43.274Z|00177|bridge|INFO|bridge br-int: deleted >> interface br-int on port 65534 >> 2018-06-11T11:13:43.275Z|00178|bridge|INFO|bridge br-int: deleted >> interface int-br-ex on port 1 >> 2018-06-11T11:13:43.275Z|00179|bridge|INFO|bridge br-int: deleted >> interface patch-tun on port 2 >> 2018-06-11T11:13:43.762Z|00180|bridge|INFO|bridge br-tun: deleted >> interface patch-int on port 1 >> 2018-06-11T11:13:43.762Z|00181|bridge|INFO|bridge br-tun: deleted >> interface br-tun on port 65534 >> 2018-06-11T11:13:44.231Z|00182|bridge|INFO|bridge br-ex: deleted >> interface br-ex on port 65534 >> 2018-06-11T11:13:44.232Z|00183|bridge|INFO|bridge br-ex: deleted >> interface phy-br-ex on port 1 >> 2018-06-11T11:13:45.245Z|00001|vlog|INFO|opened log file >> /var/log/openvswitch/ovs-vswitchd.log >> 2018-06-11T11:13:45.252Z|00002|ovs_numa|INFO|Discovered 40 CPU cores >> on NUMA node 0 >> 2018-06-11T11:13:45.252Z|00003|ovs_numa|INFO|Discovered >> 40 CPU cores on NUMA node 1 >> 2018-06-11T11:13:45.252Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes >> and >> 80 CPU cores >> 2018-06-11T11:13:45.252Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: >> connecting... >> 2018-06-11T11:13:45.252Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: >> connected >> 2018-06-11T11:13:45.256Z|00007|dpdk|INFO|Using DPDK 17.11.0 2018-06-11T11:13:45.256Z|00008|dpdk|INFO|DPDK Enabled - initializing... >> 2018-06-11T11:13:45.256Z|00009|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch 2018-06-11T11:13:45.256Z|00010|dpdk|INFO|IOMMU support for vhost-user-client disabled. >> 2018-06-11T11:13:45.256Z|00011|dpdk|INFO|EAL ARGS: ovs-vswitchd >> --socket-mem 1024,0 -c 0x00000001 >> 2018-06-11T11:13:45.258Z|00012|dpdk|INFO|EAL: Detected 80 lcore(s) >> 2018-06-11T11:13:45.285Z|00013|dpdk|INFO|EAL: Probing VFIO support... >> 2018-06-11T11:13:45.285Z|00014|dpdk|INFO|EAL: VFIO support >> initialized >> 2018-06-11T11:13:48.214Z|00015|dpdk|INFO|EAL: PCI device 0000:01:00.0 on NUMA socket 0 >> 2018-06-11T11:13:48.214Z|00016|dpdk|INFO|EAL: probe driver: 8086:1521 >> net_e1000_igb >> 2018-06-11T11:13:48.214Z|00017|dpdk|INFO|EAL: PCI device 0000:01:00.1 on NUMA socket 0 >> 2018-06-11T11:13:48.214Z|00018|dpdk|INFO|EAL: probe driver: 8086:1521 >> net_e1000_igb >> 2018-06-11T11:13:48.214Z|00019|dpdk|INFO|EAL: PCI device 0000:18:00.0 on NUMA socket 0 >> 2018-06-11T11:13:48.214Z|00020|dpdk|INFO|EAL: probe driver: 8086:1572 >> net_i40e >> 2018-06-11T11:13:48.214Z|00021|dpdk|INFO|EAL: PCI device 0000:18:00.1 on NUMA socket 0 >> 2018-06-11T11:13:48.214Z|00022|dpdk|INFO|EAL: probe driver: 8086:1572 >> net_i40e >> 2018-06-11T11:13:48.214Z|00023|dpdk|INFO|DPDK pdump packet capture >> enabled 2018-06-11T11:13:48.224Z|00024|dpdk|INFO|DPDK Enabled - >> initialized >> 2018-06-11T11:13:48.235Z|00025|ofproto_dpif|INFO|netdev@ovs-netdev: >> Datapath supports recirculation >> 2018-06-11T11:13:48.235Z|00026|ofproto_dpif|INFO|netdev@ovs-netdev: >> VLAN header stack length probed as 1 >> 2018-06-11T11:13:48.235Z|00027|ofproto_dpif|INFO|netdev@ovs-netdev: >> MPLS label stack length probed as 3 >> 2018-06-11T11:13:48.235Z|00028|ofproto_dpif|INFO|netdev@ovs-netdev: >> Datapath supports truncate action >> 2018-06-11T11:13:48.235Z|00029|ofproto_dpif|INFO|netdev@ovs-netdev: >> Datapath supports unique flow ids >> 2018-06-11T11:13:48.235Z|00030|ofproto_dpif|INFO|netdev@ovs-netdev: >> Datapath supports clone action >> 2018-06-11T11:13:48.235Z|00031|ofproto_dpif|INFO|netdev@ovs-netdev: >> Max sample nesting level probed as 10 >> 2018-06-11T11:13:48.235Z|00032|ofproto_dpif|INFO|netdev@ovs-netdev: >> Datapath supports eventmask in conntrack action >> 2018-06-11T11:13:48.235Z|00033|ofproto_dpif|INFO|netdev@ovs-netdev: >> Datapath supports ct_clear action >> 2018-06-11T11:13:48.235Z|00034|ofproto_dpif|INFO|netdev@ovs-netdev: >> Datapath supports ct_state >> 2018-06-11T11:13:48.235Z|00035|ofproto_dpif|INFO|netdev@ovs-netdev: >> Datapath supports ct_zone >> 2018-06-11T11:13:48.235Z|00036|ofproto_dpif|INFO|netdev@ovs-netdev: >> Datapath supports ct_mark >> 2018-06-11T11:13:48.235Z|00037|ofproto_dpif|INFO|netdev@ovs-netdev: >> Datapath supports ct_label >> 2018-06-11T11:13:48.235Z|00038|ofproto_dpif|INFO|netdev@ovs-netdev: >> Datapath supports ct_state_nat >> 2018-06-11T11:13:48.235Z|00039|ofproto_dpif|INFO|netdev@ovs-netdev: >> Datapath supports ct_orig_tuple >> 2018-06-11T11:13:48.235Z|00040|ofproto_dpif|INFO|netdev@ovs-netdev: >> Datapath supports ct_orig_tuple6 >> 2018-06-11T11:13:48.605Z|00041|ofproto_dpif|INFO|system@ovs-system: >> Datapath supports recirculation >> 2018-06-11T11:13:48.605Z|00042|ofproto_dpif|INFO|system@ovs-system: >> VLAN header stack length probed as 1 >> 2018-06-11T11:13:48.605Z|00043|ofproto_dpif|INFO|system@ovs-system: >> MPLS label stack length probed as 1 >> 2018-06-11T11:13:48.605Z|00044|ofproto_dpif|INFO|system@ovs-system: >> Datapath does not support truncate action >> 2018-06-11T11:13:48.605Z|00045|ofproto_dpif|INFO|system@ovs-system: >> Datapath supports unique flow ids >> 2018-06-11T11:13:48.605Z|00046|ofproto_dpif|INFO|system@ovs-system: >> Datapath does not support clone action >> 2018-06-11T11:13:48.605Z|00047|ofproto_dpif|INFO|system@ovs-system: >> Max sample nesting level probed as 3 >> 2018-06-11T11:13:48.605Z|00048|ofproto_dpif|INFO|system@ovs-system: >> Datapath does not support eventmask in conntrack action >> 2018-06-11T11:13:48.605Z|00049|ofproto_dpif|INFO|system@ovs-system: >> Datapath does not support ct_clear action >> 2018-06-11T11:13:48.605Z|00050|ofproto_dpif|INFO|system@ovs-system: >> Datapath supports ct_state >> 2018-06-11T11:13:48.605Z|00051|ofproto_dpif|INFO|system@ovs-system: >> Datapath supports ct_zone >> 2018-06-11T11:13:48.605Z|00052|ofproto_dpif|INFO|system@ovs-system: >> Datapath supports ct_mark >> 2018-06-11T11:13:48.605Z|00053|ofproto_dpif|INFO|system@ovs-system: >> Datapath supports ct_label >> 2018-06-11T11:13:48.605Z|00054|ofproto_dpif|INFO|system@ovs-system: >> Datapath does not support ct_state_nat >> 2018-06-11T11:13:48.605Z|00055|ofproto_dpif|INFO|system@ovs-system: >> Datapath does not support ct_orig_tuple >> 2018-06-11T11:13:48.605Z|00056|ofproto_dpif|INFO|system@ovs-system: >> Datapath does not support ct_orig_tuple6 >> 2018-06-11T11:13:48.981Z|00057|bridge|INFO|bridge br-mlx: added >> interface br-mlx on port 65534 >> 2018-06-11T11:13:48.981Z|00058|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T11:13:48.981Z|00059|netdev_dpdk|WARN|Error attaching >> device 'mlx4_0' to DPDK >> 2018-06-11T11:13:48.981Z|00060|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T11:13:48.982Z|00061|bridge|INFO|bridge br-int: added >> interface br-int on port 65534 >> 2018-06-11T11:13:48.982Z|00062|bridge|INFO|bridge br-int: added >> interface int-br-ex on port 1 >> 2018-06-11T11:13:48.982Z|00063|bridge|INFO|bridge br-int: added >> interface patch-tun on port 2 >> 2018-06-11T11:13:48.982Z|00064|bridge|INFO|bridge br-tun: added >> interface patch-int on port 1 >> 2018-06-11T11:13:48.982Z|00065|bridge|INFO|bridge br-tun: added >> interface br-tun on port 65534 >> 2018-06-11T11:13:48.982Z|00066|bridge|INFO|bridge br-ex: added >> interface br-ex on port 65534 >> 2018-06-11T11:13:48.982Z|00067|bridge|INFO|bridge br-ex: added >> interface phy-br-ex on port 1 >> 2018-06-11T11:13:48.982Z|00068|bridge|INFO|bridge br-mlx: using >> datapath ID 000072528484a642 >> 2018-06-11T11:13:48.982Z|00069|connmgr|INFO|br-mlx: added service controller "punix:/var/run/openvswitch/br-mlx.mgmt" >> 2018-06-11T11:13:49.221Z|00070|bridge|INFO|bridge br-int: using >> datapath ID 00006680927b6b40 >> 2018-06-11T11:13:49.221Z|00071|connmgr|INFO|br-int: added service controller "punix:/var/run/openvswitch/br-int.mgmt" >> 2018-06-11T11:13:49.221Z|00072|connmgr|INFO|br-int: added primary controller "tcp:127.0.0.1:6633" >> 2018-06-11T11:13:49.221Z|00073|rconn|INFO|br-int<->tcp:127.0.0.1:6633: >> connecting... >> 2018-06-11T11:13:49.460Z|00074|bridge|INFO|bridge br-tun: using >> datapath ID 000066532e5f0443 >> 2018-06-11T11:13:49.461Z|00075|connmgr|INFO|br-tun: added service controller "punix:/var/run/openvswitch/br-tun.mgmt" >> 2018-06-11T11:13:49.461Z|00076|connmgr|INFO|br-tun: added primary controller "tcp:127.0.0.1:6633" >> 2018-06-11T11:13:49.461Z|00077|rconn|INFO|br-tun<->tcp:127.0.0.1:6633: >> connecting... >> 2018-06-11T11:13:49.692Z|00078|bridge|INFO|bridge br-ex: using >> datapath ID 0000e2d1bdbe8e45 >> 2018-06-11T11:13:49.692Z|00079|connmgr|INFO|br-ex: added service controller "punix:/var/run/openvswitch/br-ex.mgmt" >> 2018-06-11T11:13:49.692Z|00080|connmgr|INFO|br-ex: added primary controller "tcp:127.0.0.1:6633" >> 2018-06-11T11:13:49.692Z|00081|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: >> connecting... >> 2018-06-11T11:13:49.916Z|00082|timeval|WARN|Unreasonably long 4661ms >> poll interval (16ms user, 4472ms system) >> 2018-06-11T11:13:49.916Z|00083|timeval|WARN|faults: 3584706 minor, 0 >> major >> 2018-06-11T11:13:49.916Z|00084|timeval|WARN|disk: 0 reads, 16 writes 2018-06-11T11:13:49.916Z|00085|timeval|WARN|context switches: 147 voluntary, 25 involuntary 2018-06-11T11:13:49.916Z|00086|coverage|INFO|Event coverage, avg rate over last: 5 seconds, last minute, last hour, hash=2ba35161: >> 2018-06-11T11:13:49.916Z|00087|coverage|INFO|bridge_reconfigure >> 0.0/sec 0.000/sec 0.0000/sec total: 1 >> 2018-06-11T11:13:49.916Z|00088|coverage|INFO|ofproto_flush >> 0.0/sec 0.000/sec 0.0000/sec total: 4 >> 2018-06-11T11:13:49.916Z|00089|coverage|INFO|ofproto_update_port >> 0.0/sec 0.000/sec 0.0000/sec total: 8 >> 2018-06-11T11:13:49.916Z|00090|coverage|INFO|rev_flow_table >> 0.0/sec 0.000/sec 0.0000/sec total: 1 >> 2018-06-11T11:13:49.916Z|00091|coverage|INFO|cmap_expand >> 0.0/sec 0.000/sec 0.0000/sec total: 52 >> 2018-06-11T11:13:49.916Z|00092|coverage|INFO|cmap_shrink >> 0.0/sec 0.000/sec 0.0000/sec total: 23 >> 2018-06-11T11:13:49.916Z|00093|coverage|INFO|dpif_port_add >> 0.0/sec 0.000/sec 0.0000/sec total: 1 >> 2018-06-11T11:13:49.916Z|00094|coverage|INFO|dpif_flow_flush >> 0.0/sec 0.000/sec 0.0000/sec total: 6 >> 2018-06-11T11:13:49.916Z|00095|coverage|INFO|dpif_flow_get >> 0.0/sec 0.000/sec 0.0000/sec total: 29 >> 2018-06-11T11:13:49.916Z|00096|coverage|INFO|dpif_flow_put >> 0.0/sec 0.000/sec 0.0000/sec total: 37 >> 2018-06-11T11:13:49.916Z|00097|coverage|INFO|dpif_flow_del >> 0.0/sec 0.000/sec 0.0000/sec total: 29 >> 2018-06-11T11:13:49.916Z|00098|coverage|INFO|dpif_execute >> 0.0/sec 0.000/sec 0.0000/sec total: 8 >> 2018-06-11T11:13:49.916Z|00099|coverage|INFO|flow_extract >> 0.0/sec 0.000/sec 0.0000/sec total: 6 >> 2018-06-11T11:13:49.916Z|00100|coverage|INFO|miniflow_malloc >> 0.0/sec 0.000/sec 0.0000/sec total: 34 >> 2018-06-11T11:13:49.916Z|00101|coverage|INFO|hmap_expand >> 0.0/sec 0.000/sec 0.0000/sec total: 498 >> 2018-06-11T11:13:49.916Z|00102|coverage|INFO|netdev_received >> 0.0/sec 0.000/sec 0.0000/sec total: 1 >> 2018-06-11T11:13:49.916Z|00103|coverage|INFO|netdev_get_stats >> 0.0/sec 0.000/sec 0.0000/sec total: 8 >> 2018-06-11T11:13:49.916Z|00104|coverage|INFO|poll_create_node >> 0.0/sec 0.000/sec 0.0000/sec total: 23 >> 2018-06-11T11:13:49.916Z|00105|coverage|INFO|seq_change >> 0.0/sec 0.000/sec 0.0000/sec total: 2193 >> 2018-06-11T11:13:49.916Z|00106|coverage|INFO|pstream_open >> 0.0/sec 0.000/sec 0.0000/sec total: 9 >> 2018-06-11T11:13:49.916Z|00107|coverage|INFO|stream_open >> 0.0/sec 0.000/sec 0.0000/sec total: 4 >> 2018-06-11T11:13:49.916Z|00108|coverage|INFO|util_xalloc >> 0.0/sec 0.000/sec 0.0000/sec total: 12803 >> 2018-06-11T11:13:49.916Z|00109|coverage|INFO|vconn_open >> 0.0/sec 0.000/sec 0.0000/sec total: 3 >> 2018-06-11T11:13:49.916Z|00110|coverage|INFO|netdev_set_policing >> 0.0/sec 0.000/sec 0.0000/sec total: 4 >> 2018-06-11T11:13:49.916Z|00111|coverage|INFO|netdev_get_ifindex >> 0.0/sec 0.000/sec 0.0000/sec total: 4 >> 2018-06-11T11:13:49.916Z|00112|coverage|INFO|netdev_get_hwaddr >> 0.0/sec 0.000/sec 0.0000/sec total: 9 >> 2018-06-11T11:13:49.916Z|00113|coverage|INFO|netdev_get_ethtool >> 0.0/sec 0.000/sec 0.0000/sec total: 2 >> 2018-06-11T11:13:49.916Z|00114|coverage|INFO|netlink_received >> 0.0/sec 0.000/sec 0.0000/sec total: 85 >> 2018-06-11T11:13:49.916Z|00115|coverage|INFO|netlink_recv_jumbo >> 0.0/sec 0.000/sec 0.0000/sec total: 1 >> 2018-06-11T11:13:49.916Z|00116|coverage|INFO|netlink_sent >> 0.0/sec 0.000/sec 0.0000/sec total: 102 >> 2018-06-11T11:13:49.916Z|00117|coverage|INFO|74 events never hit >> 2018-06-11T11:13:49.916Z|00118|poll_loop|INFO|wakeup due to [POLLIN] >> on fd 433 (FIFO pipe:[1025715]) at ../lib/ovs-rcu.c:229 (97% CPU >> usage) 2018-06-11T11:13:49.917Z|00119|poll_loop|INFO|wakeup due to >> [POLLIN] on fd 433 (FIFO pipe:[1025715]) at ../lib/ovs-rcu.c:229 (97% >> CPU usage) 2018-06-11T11:13:49.917Z|00120|poll_loop|INFO|wakeup due >> to 0-ms timeout at ../lib/ovs-rcu.c:229 (97% CPU usage) >> 2018-06-11T11:13:49.917Z|00121|poll_loop|INFO|wakeup due to 0-ms >> timeout at ../lib/ovs-rcu.c:229 (97% CPU usage) >> 2018-06-11T11:13:49.920Z|00122|rconn|INFO|br-int<->tcp:127.0.0.1:6633: >> connected >> 2018-06-11T11:13:49.920Z|00123|rconn|INFO|br-tun<->tcp:127.0.0.1:6633: >> connected >> 2018-06-11T11:13:49.920Z|00124|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: >> connected >> 2018-06-11T11:13:49.926Z|00125|poll_loop|INFO|wakeup due to [POLLIN] >> on fd 10 (NETLINK_ROUTE<->NETLINK_ROUTE) at >> ../lib/netlink-socket.c:1331 (97% CPU usage) >> 2018-06-11T11:13:49.926Z|00126|poll_loop|INFO|wakeup due to [POLLIN] >> on fd 421 (character device /dev/net/tun) at >> ../lib/netdev-linux.c:1171 (97% CPU usage) >> 2018-06-11T11:13:49.926Z|00127|poll_loop|INFO|wakeup due to [POLLIN] >> on fd 423 (127.0.0.1:33712<->127.0.0.1:6633) at >> ../lib/stream-fd.c:157 (97% CPU usage) >> 2018-06-11T11:13:49.926Z|00128|poll_loop|INFO|wakeup >> due to [POLLIN] on fd 428 (127.0.0.1:33714<->127.0.0.1:6633) at >> ../lib/stream-fd.c:157 (97% CPU usage) >> 2018-06-11T11:13:49.926Z|00129|poll_loop|INFO|wakeup due to [POLLIN] >> on fd 11 (<->/var/run/openvswitch/db.sock) at ../lib/stream-fd.c:157 >> (97% CPU usage) 2018-06-11T11:13:49.926Z|00130|poll_loop|INFO|wakeup >> due to [POLLIN] on fd 429 (127.0.0.1:33716<->127.0.0.1:6633) at >> ../lib/stream-fd.c:157 (97% CPU usage) >> 2018-06-11T11:13:49.930Z|00131|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T11:13:49.930Z|00132|netdev_dpdk|WARN|Error attaching >> device 'mlx4_0' to DPDK >> 2018-06-11T11:13:49.930Z|00133|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T11:13:49.931Z|00134|bridge|INFO|ovs-vswitchd (Open >> vSwitch) >> 2.9.0 >> 2018-06-11T11:13:53.662Z|00135|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T11:13:53.662Z|00136|netdev_dpdk|WARN|Error attaching >> device 'mlx4_0' to DPDK >> 2018-06-11T11:13:53.662Z|00137|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T11:13:53.675Z|00138|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T11:13:53.675Z|00139|netdev_dpdk|WARN|Error attaching >> device 'mlx4_0' to DPDK >> 2018-06-11T11:13:53.675Z|00140|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T11:13:53.682Z|00141|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T11:13:53.682Z|00142|netdev_dpdk|WARN|Error attaching >> device 'mlx4_0' to DPDK >> 2018-06-11T11:13:53.682Z|00143|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T11:13:53.693Z|00144|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T11:13:53.693Z|00145|netdev_dpdk|WARN|Error attaching >> device 'mlx4_0' to DPDK >> 2018-06-11T11:13:53.693Z|00146|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T11:13:53.705Z|00147|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T11:13:53.705Z|00148|netdev_dpdk|WARN|Error attaching >> device 'mlx4_0' to DPDK >> 2018-06-11T11:13:53.705Z|00149|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T11:13:53.711Z|00150|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T11:13:53.711Z|00151|netdev_dpdk|WARN|Error attaching >> device 'mlx4_0' to DPDK >> 2018-06-11T11:13:53.711Z|00152|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T11:13:53.717Z|00153|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T11:13:53.717Z|00154|netdev_dpdk|WARN|Error attaching >> device 'mlx4_0' to DPDK >> 2018-06-11T11:13:53.717Z|00155|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T11:13:53.728Z|00156|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T11:13:53.728Z|00157|netdev_dpdk|WARN|Error attaching >> device 'mlx4_0' to DPDK >> 2018-06-11T11:13:53.728Z|00158|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T11:13:53.743Z|00159|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T11:13:53.743Z|00160|netdev_dpdk|WARN|Error attaching >> device 'mlx4_0' to DPDK >> 2018-06-11T11:13:53.743Z|00161|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T11:13:53.749Z|00162|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T11:13:53.749Z|00163|netdev_dpdk|WARN|Error attaching >> device 'mlx4_0' to DPDK >> 2018-06-11T11:13:53.749Z|00164|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T11:13:53.756Z|00165|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T11:13:53.756Z|00166|netdev_dpdk|WARN|Error attaching >> device 'mlx4_0' to DPDK >> 2018-06-11T11:13:53.756Z|00167|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T11:13:53.766Z|00168|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >> 2018-06-11T11:13:53.766Z|00169|netdev_dpdk|WARN|Error attaching >> device 'mlx4_0' to DPDK >> 2018-06-11T11:13:53.766Z|00170|netdev|WARN|enp59s0d1: could not set >> configuration (Invalid argument) >> 2018-06-11T11:13:55.270Z|00171|memory|INFO|1406100 kB peak resident >> set size after 10.0 seconds >> 2018-06-11T11:13:55.270Z|00172|memory|INFO|handlers:118 ofconns:3 >> ports:8 revalidators:42 rules:36 udpif keys:1 >> 2018-06-11T11:14:03.699Z|00173|connmgr|INFO|br-int<->tcp:127.0.0.1:6633: >> 5 flow_mods 10 s ago (5 adds) >> 2018-06-11T11:14:03.737Z|00174|connmgr|INFO|br-ex<->tcp:127.0.0.1:6633: >> 2 flow_mods 10 s ago (2 adds) >> 2018-06-11T11:14:03.773Z|00175|connmgr|INFO|br-tun<->tcp:127.0.0.1:6633: >> 12 flow_mods 10 s ago (12 adds) >> >> Best regards >> >> On 06/11/2018 12:57 PM, Raslan Darawsheh wrote: >>> What I really want to see is the init process for OVS-DPDK that dumps the DPDK initialization. >>> I think you will find it at the beginning of the log that you just sent. >>> >>> Kindest regards, >>> Raslan Darawsheh >>> >>> -----Original Message----- >>> From: Tobias Urdin [mailto:tobias.urdin@crystone.com] >>> Sent: Monday, June 11, 2018 1:00 PM >>> To: Raslan Darawsheh <rasland@mellanox.com> >>> Cc: users@dpdk.org >>> Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch >>> >>> Hello Raslan, >>> Thank you for your reply. >>> >>> For reference the br-mlx bridge was created like this: >>> $ovs-vsctl add-br br-mlx -- set bridge br-mlx datapath_type=netdev >>> >>> $ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 >>> type=dpdk options:dpdk-devargs=mlx4_0 >>> ovs-vsctl: Error detected while setting up 'enp59s0d1': Error attaching device 'mlx4_0' to DPDK. See ovs-vswitchd log for details. >>> ovs-vsctl: The default log directory is "/var/log/openvswitch". >>> >>> In the ovs-vswitchd.log >>> 2018-06-11T09:56:20.613Z|00185|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T09:56:20.613Z|00186|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T09:56:20.613Z|00187|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> >>> Here is the complete log after restarting openvswitch-switch service: >>> 2018-06-11T09:57:49.380Z|00134|bridge|INFO|ovs-vswitchd (Open >>> vSwitch) >>> 2.9.0 >>> 2018-06-11T09:57:52.812Z|00135|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T09:57:52.812Z|00136|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T09:57:52.812Z|00137|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T09:57:52.825Z|00138|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T09:57:52.825Z|00139|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T09:57:52.825Z|00140|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T09:57:52.832Z|00141|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T09:57:52.832Z|00142|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T09:57:52.832Z|00143|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T09:57:52.843Z|00144|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T09:57:52.843Z|00145|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T09:57:52.843Z|00146|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T09:57:52.856Z|00147|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T09:57:52.856Z|00148|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T09:57:52.856Z|00149|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T09:57:52.862Z|00150|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T09:57:52.862Z|00151|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T09:57:52.862Z|00152|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T09:57:52.870Z|00153|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T09:57:52.870Z|00154|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T09:57:52.870Z|00155|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T09:57:52.889Z|00156|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T09:57:52.889Z|00157|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T09:57:52.889Z|00158|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T09:57:52.903Z|00159|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T09:57:52.903Z|00160|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T09:57:52.903Z|00161|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T09:57:52.908Z|00162|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T09:57:52.908Z|00163|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T09:57:52.908Z|00164|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T09:57:52.914Z|00165|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T09:57:52.914Z|00166|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T09:57:52.914Z|00167|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T09:57:52.923Z|00168|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T09:57:52.923Z|00169|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T09:57:52.923Z|00170|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T09:57:54.808Z|00171|memory|INFO|1406100 kB peak resident >>> set size after 10.1 seconds >>> 2018-06-11T09:57:54.808Z|00172|memory|INFO|handlers:118 ofconns:3 >>> ports:8 revalidators:42 rules:36 udpif keys:1 >>> 2018-06-11T09:58:02.850Z|00173|connmgr|INFO|br-int<->tcp:127.0.0.1:6633: >>> 5 flow_mods 10 s ago (5 adds) >>> 2018-06-11T09:58:02.898Z|00174|connmgr|INFO|br-ex<->tcp:127.0.0.1:6633: >>> 2 flow_mods 10 s ago (2 adds) >>> 2018-06-11T09:58:02.930Z|00175|connmgr|INFO|br-tun<->tcp:127.0.0.1:6633: >>> 12 flow_mods 10 s ago (12 adds) >>> >>> Was that what you mean by init log? >>> Best regards >>> >>> On 06/11/2018 11:51 AM, Raslan Darawsheh wrote: >>>> Hi Tobias, >>>> >>>> Can you try using the following ? >>>> >>>> ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 >>>> type=dpdk options:dpdk-devargs=mlx4_0 >>>> >>>> if it didn't work can you send the OVS-DPDK init log ? >>>> >>>> Kindest regards, >>>> Raslan Darawsheh >>>> >>>> -----Original Message----- >>>> From: users [mailto:users-bounces@dpdk.org] On Behalf Of Tobias >>>> Urdin >>>> Sent: Monday, June 11, 2018 12:15 PM >>>> To: users@dpdk.org >>>> Subject: [dpdk-users] mellanox connect-x 3 pro dpdk with >>>> openvswitch >>>> >>>> Hello, >>>> >>>> I'm stuck trying to test DPDK with a Mellanox Connect-X 3 Pro card >>>> on Ubuntu 16.04 >>>> >>>> I've enabled iommu, fixed hugepages and has installed and are running the openvswitch-switch-dpdk alternative. >>>> >>>> $lspci | grep -i mellanox >>>> 3b:00.0 Ethernet controller: Mellanox Technologies MT27520 Family >>>> [ConnectX-3 Pro] >>>> >>>> $lsmod | grep -i mlx >>>> mlx4_en 110592 0 >>>> vxlan 49152 2 i40e,mlx4_en >>>> mlx4_core 286720 1 mlx4_en >>>> ptp 20480 3 igb,i40e,mlx4_en >>>> >>>> $cat /etc/modprobe.d/mlx4_core.conf options mlx4_core >>>> log_num_mgm_entry_size=-1 >>>> >>>> >>>> $ovs-vsctl add-port br-mlx dpdk0 -- set Interface dpdk0 type=dpdk >>>> >>>> Bridge br-mlx >>>> Port "dpdk0" >>>> Interface "dpdk0" >>>> type: dpdk >>>> error: "'dpdk0' is missing 'options:dpdk-devargs'. The old 'dpdk<port_id>' names are not supported" >>>> >>>> >>>> $ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 >>>> type=dpdk options:dpdk-devargs=0000:3b:00.0,1 >>>> ovs-vsctl: Error detected while setting up 'enp59s0d1': Error attaching device '0000:3b:00.0,1' to DPDK. See ovs-vswitchd log for details. >>>> ovs-vsctl: The default log directory is "/var/log/openvswitch". >>>> >>>> 2018-06-11T09:10:38.696Z|00177|netdev_dpdk|WARN|Error attaching >>>> device '0000:3b:00.0,1' to DPDK >>>> 2018-06-11T09:10:38.696Z|00178|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> >>>> 2018-06-11T09:12:50.994Z|00181|dpdk|ERR|EAL: Driver cannot attach >>>> the device (0000:3b:00.0) >>>> 2018-06-11T09:12:50.994Z|00182|dpdk|ERR|EAL: No port found for >>>> device >>>> (0000:3b:00.0) >>>> 2018-06-11T09:12:50.994Z|00183|netdev_dpdk|WARN|Error >>>> attaching device '0000:3b:00.0,1' to DPDK >>>> 2018-06-11T09:12:50.994Z|00184|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> >>>> $ovs-vsctl add-port br-mlx dpdk0 -- set Interface dpdk0 type=dpdk options:dpdk-devargs="class=eth,mac=ec:0d:9a:b0:13:21" >>>> ovs-vsctl: Error detected while setting up 'dpdk0': Error attaching device 'class=eth,mac=ec:0d:9a:b0:13:21' to DPDK. See ovs-vswitchd log for details. >>>> ovs-vsctl: The default log directory is "/var/log/openvswitch". >>>> >>>> Port "dpdk0" >>>> Interface "dpdk0" >>>> type: dpdk >>>> options: {dpdk-devargs="class=eth,mac=ec:0d:9a:b0:13:21"} >>>> error: "Error attaching device 'class=eth,mac=ec:0d:9a:b0:13:21' to DPDK" >>>> >>>> >>>> Anybody know what I might have missed? I have tried restarting the openvswitch services after adding the interfaces as well but no difference. >>>> >>>> The logs doesn't say anything interesting from what I can see :( >>>> >>>> Best regards >>>> > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [dpdk-users] mellanox connect-x 3 pro dpdk with openvswitch 2018-06-11 11:51 ` Raslan Darawsheh @ 2018-06-11 12:05 ` Tobias Urdin 2018-06-11 12:07 ` Raslan Darawsheh 0 siblings, 1 reply; 14+ messages in thread From: Tobias Urdin @ 2018-06-11 12:05 UTC (permalink / raw) To: Raslan Darawsheh; +Cc: users Ok, I'm more wondering if there is any way I can build a librte-pmd-mlx4 package that supplies the compiled PMD or something like that instead of having to rebuild all the major parts? Thanks! Best regards On 06/11/2018 01:51 PM, Raslan Darawsheh wrote: > I think you can upgrade the current version that you have as seen in this article: > http://docs.openvswitch.org/en/latest/intro/install/general/#hot-upgrading > > but, I haven't tried it before. > > Kindest regards, > Raslan Darawsheh > > -----Original Message----- > From: Tobias Urdin [mailto:tobias.urdin@crystone.com] > Sent: Monday, June 11, 2018 2:45 PM > To: Raslan Darawsheh <rasland@mellanox.com> > Cc: users@dpdk.org > Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch > > Is there any to skip compiling DPDK and OVS+OVS-DPDK and instead use the ones I have installed? > Only compiling the Mellanox PMD module/binary or something similar to that? > > I can see that other drivers supply pmd packages like below. > > $dpkg -l | grep -E "openvswitch|pmd|dpdk" > ii dpdk > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (runtime) > ii librte-pmd-af-packet17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-af-packet runtime library) > ii librte-pmd-ark17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte_pmd_ark runtime library) > ii librte-pmd-avp17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte_pmd_avp runtime library) > ii librte-pmd-bnxt17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-bnxt runtime library) > ii librte-pmd-bond17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-bond runtime library) ii librte-pmd-crypto-scheduler17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte_pmd_crypto_scheduler runtime library) > ii librte-pmd-cxgbe17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-cxgbe runtime library) > ii librte-pmd-e1000-17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-e1000 runtime library) > ii librte-pmd-ena17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-ena runtime library) > ii librte-pmd-enic17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-enic runtime library) > ii librte-pmd-failsafe17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-failsafe runtime library) > ii librte-pmd-fm10k17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-fm10k runtime library) > ii librte-pmd-i40e17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-i40e runtime library) > ii librte-pmd-ixgbe17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-ixgbe runtime library) > ii librte-pmd-kni17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte_pmd_kni runtime library) > ii librte-pmd-lio17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte_pmd_lio runtime library) > ii librte-pmd-nfp17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte_pmd_nfp runtime library) > ii librte-pmd-null-crypto17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-null-crypto runtime library) > ii librte-pmd-null17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-null runtime library) ii librte-pmd-octeontx-ssovf17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte_pmd_octeontx_ssovf runtime library) > ii librte-pmd-octeontx17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte_pmd_octeontx runtime library) > ii librte-pmd-pcap17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-pcap runtime library) > ii librte-pmd-qede17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-qede runtime library) > ii librte-pmd-ring17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-ring runtime library) > ii librte-pmd-sfc-efx17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte_pmd_sfc_efx runtime library) ii librte-pmd-skeleton-event17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte_pmd_skeleton_event runtime library) > ii librte-pmd-softnic17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-softnic runtime library) > ii librte-pmd-sw-event17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte_pmd_sw_event runtime library) > ii librte-pmd-tap17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte_pmd_tap runtime library) ii librte-pmd-thunderx-nicvf17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte_pmd_thunderx_nicvf runtime library) > ii librte-pmd-vhost17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-vhost runtime library) > ii librte-pmd-virtio17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-virtio runtime library) > ii librte-pmd-vmxnet3-uio17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-vmxnet3-uio runtime library) > ii openvswitch-common > 2.9.0-0ubuntu1~cloud0 amd64 Open vSwitch > common components > ii openvswitch-switch > 2.9.0-0ubuntu1~cloud0 amd64 Open vSwitch > switch implementations > ii openvswitch-switch-dpdk > 2.9.0-0ubuntu1~cloud0 amd64 DPDK enabled > Open vSwitch switch implementation > ii python-openvswitch > 2.9.0-0ubuntu1~cloud0 all Python bindings > for Open vSwitch > > Thanks for all the help! > Best regards > > > On 06/11/2018 01:37 PM, Raslan Darawsheh wrote: >> You can refer to this community page for compiling OVS-DPDK with MLX pmds enabled: >> https://community.mellanox.com/docs/DOC-2637 >> >> >> Kindest regards, >> Raslan Darawsheh >> >> -----Original Message----- >> From: Tobias Urdin [mailto:tobias.urdin@crystone.com] >> Sent: Monday, June 11, 2018 2:35 PM >> To: Raslan Darawsheh <rasland@mellanox.com> >> Cc: users@dpdk.org >> Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch >> >> I haven't compiled any DPDK or Mellanox related modules/packages. >> What I have is ovs, ovs-dpdk, mlx4 module, I have also loaded the ib_uverbs driver. >> >> Is it the librte_pmd_mlx4 I have to compile manually? >> >> Do you know any updated documentation about this, there isn't much relevant and searching for documentation and the correct mlnx driver and code is a struggle. >> >> Best regards >> >> On 06/11/2018 01:20 PM, Raslan Darawsheh wrote: >>> From what I'm seeing it looks like it didn't prop the MLX4 device at all. >>> Can you make sure you have it enabled in your DPDK package when you compiled DPDK ? >>> >>> Kindest regards, >>> Raslan Darawsheh >>> >>> -----Original Message----- >>> From: Tobias Urdin [mailto:tobias.urdin@crystone.com] >>> Sent: Monday, June 11, 2018 2:16 PM >>> To: Raslan Darawsheh <rasland@mellanox.com> >>> Cc: users@dpdk.org >>> Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch >>> >>> Ok, here is the complete log when restarting the openvswitch-switch service, see below. >>> >>> Also here is the kernel for information: >>> $uname -r >>> 4.4.0-127-generic >>> >>> 2018-06-11T11:13:42.729Z|00176|bridge|INFO|bridge br-mlx: deleted >>> interface br-mlx on port 65534 >>> 2018-06-11T11:13:43.274Z|00177|bridge|INFO|bridge br-int: deleted >>> interface br-int on port 65534 >>> 2018-06-11T11:13:43.275Z|00178|bridge|INFO|bridge br-int: deleted >>> interface int-br-ex on port 1 >>> 2018-06-11T11:13:43.275Z|00179|bridge|INFO|bridge br-int: deleted >>> interface patch-tun on port 2 >>> 2018-06-11T11:13:43.762Z|00180|bridge|INFO|bridge br-tun: deleted >>> interface patch-int on port 1 >>> 2018-06-11T11:13:43.762Z|00181|bridge|INFO|bridge br-tun: deleted >>> interface br-tun on port 65534 >>> 2018-06-11T11:13:44.231Z|00182|bridge|INFO|bridge br-ex: deleted >>> interface br-ex on port 65534 >>> 2018-06-11T11:13:44.232Z|00183|bridge|INFO|bridge br-ex: deleted >>> interface phy-br-ex on port 1 >>> 2018-06-11T11:13:45.245Z|00001|vlog|INFO|opened log file >>> /var/log/openvswitch/ovs-vswitchd.log >>> 2018-06-11T11:13:45.252Z|00002|ovs_numa|INFO|Discovered 40 CPU cores >>> on NUMA node 0 >>> 2018-06-11T11:13:45.252Z|00003|ovs_numa|INFO|Discovered >>> 40 CPU cores on NUMA node 1 >>> 2018-06-11T11:13:45.252Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes >>> and >>> 80 CPU cores >>> 2018-06-11T11:13:45.252Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: >>> connecting... >>> 2018-06-11T11:13:45.252Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: >>> connected >>> 2018-06-11T11:13:45.256Z|00007|dpdk|INFO|Using DPDK 17.11.0 2018-06-11T11:13:45.256Z|00008|dpdk|INFO|DPDK Enabled - initializing... >>> 2018-06-11T11:13:45.256Z|00009|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch 2018-06-11T11:13:45.256Z|00010|dpdk|INFO|IOMMU support for vhost-user-client disabled. >>> 2018-06-11T11:13:45.256Z|00011|dpdk|INFO|EAL ARGS: ovs-vswitchd >>> --socket-mem 1024,0 -c 0x00000001 >>> 2018-06-11T11:13:45.258Z|00012|dpdk|INFO|EAL: Detected 80 lcore(s) >>> 2018-06-11T11:13:45.285Z|00013|dpdk|INFO|EAL: Probing VFIO support... >>> 2018-06-11T11:13:45.285Z|00014|dpdk|INFO|EAL: VFIO support >>> initialized >>> 2018-06-11T11:13:48.214Z|00015|dpdk|INFO|EAL: PCI device 0000:01:00.0 on NUMA socket 0 >>> 2018-06-11T11:13:48.214Z|00016|dpdk|INFO|EAL: probe driver: 8086:1521 >>> net_e1000_igb >>> 2018-06-11T11:13:48.214Z|00017|dpdk|INFO|EAL: PCI device 0000:01:00.1 on NUMA socket 0 >>> 2018-06-11T11:13:48.214Z|00018|dpdk|INFO|EAL: probe driver: 8086:1521 >>> net_e1000_igb >>> 2018-06-11T11:13:48.214Z|00019|dpdk|INFO|EAL: PCI device 0000:18:00.0 on NUMA socket 0 >>> 2018-06-11T11:13:48.214Z|00020|dpdk|INFO|EAL: probe driver: 8086:1572 >>> net_i40e >>> 2018-06-11T11:13:48.214Z|00021|dpdk|INFO|EAL: PCI device 0000:18:00.1 on NUMA socket 0 >>> 2018-06-11T11:13:48.214Z|00022|dpdk|INFO|EAL: probe driver: 8086:1572 >>> net_i40e >>> 2018-06-11T11:13:48.214Z|00023|dpdk|INFO|DPDK pdump packet capture >>> enabled 2018-06-11T11:13:48.224Z|00024|dpdk|INFO|DPDK Enabled - >>> initialized >>> 2018-06-11T11:13:48.235Z|00025|ofproto_dpif|INFO|netdev@ovs-netdev: >>> Datapath supports recirculation >>> 2018-06-11T11:13:48.235Z|00026|ofproto_dpif|INFO|netdev@ovs-netdev: >>> VLAN header stack length probed as 1 >>> 2018-06-11T11:13:48.235Z|00027|ofproto_dpif|INFO|netdev@ovs-netdev: >>> MPLS label stack length probed as 3 >>> 2018-06-11T11:13:48.235Z|00028|ofproto_dpif|INFO|netdev@ovs-netdev: >>> Datapath supports truncate action >>> 2018-06-11T11:13:48.235Z|00029|ofproto_dpif|INFO|netdev@ovs-netdev: >>> Datapath supports unique flow ids >>> 2018-06-11T11:13:48.235Z|00030|ofproto_dpif|INFO|netdev@ovs-netdev: >>> Datapath supports clone action >>> 2018-06-11T11:13:48.235Z|00031|ofproto_dpif|INFO|netdev@ovs-netdev: >>> Max sample nesting level probed as 10 >>> 2018-06-11T11:13:48.235Z|00032|ofproto_dpif|INFO|netdev@ovs-netdev: >>> Datapath supports eventmask in conntrack action >>> 2018-06-11T11:13:48.235Z|00033|ofproto_dpif|INFO|netdev@ovs-netdev: >>> Datapath supports ct_clear action >>> 2018-06-11T11:13:48.235Z|00034|ofproto_dpif|INFO|netdev@ovs-netdev: >>> Datapath supports ct_state >>> 2018-06-11T11:13:48.235Z|00035|ofproto_dpif|INFO|netdev@ovs-netdev: >>> Datapath supports ct_zone >>> 2018-06-11T11:13:48.235Z|00036|ofproto_dpif|INFO|netdev@ovs-netdev: >>> Datapath supports ct_mark >>> 2018-06-11T11:13:48.235Z|00037|ofproto_dpif|INFO|netdev@ovs-netdev: >>> Datapath supports ct_label >>> 2018-06-11T11:13:48.235Z|00038|ofproto_dpif|INFO|netdev@ovs-netdev: >>> Datapath supports ct_state_nat >>> 2018-06-11T11:13:48.235Z|00039|ofproto_dpif|INFO|netdev@ovs-netdev: >>> Datapath supports ct_orig_tuple >>> 2018-06-11T11:13:48.235Z|00040|ofproto_dpif|INFO|netdev@ovs-netdev: >>> Datapath supports ct_orig_tuple6 >>> 2018-06-11T11:13:48.605Z|00041|ofproto_dpif|INFO|system@ovs-system: >>> Datapath supports recirculation >>> 2018-06-11T11:13:48.605Z|00042|ofproto_dpif|INFO|system@ovs-system: >>> VLAN header stack length probed as 1 >>> 2018-06-11T11:13:48.605Z|00043|ofproto_dpif|INFO|system@ovs-system: >>> MPLS label stack length probed as 1 >>> 2018-06-11T11:13:48.605Z|00044|ofproto_dpif|INFO|system@ovs-system: >>> Datapath does not support truncate action >>> 2018-06-11T11:13:48.605Z|00045|ofproto_dpif|INFO|system@ovs-system: >>> Datapath supports unique flow ids >>> 2018-06-11T11:13:48.605Z|00046|ofproto_dpif|INFO|system@ovs-system: >>> Datapath does not support clone action >>> 2018-06-11T11:13:48.605Z|00047|ofproto_dpif|INFO|system@ovs-system: >>> Max sample nesting level probed as 3 >>> 2018-06-11T11:13:48.605Z|00048|ofproto_dpif|INFO|system@ovs-system: >>> Datapath does not support eventmask in conntrack action >>> 2018-06-11T11:13:48.605Z|00049|ofproto_dpif|INFO|system@ovs-system: >>> Datapath does not support ct_clear action >>> 2018-06-11T11:13:48.605Z|00050|ofproto_dpif|INFO|system@ovs-system: >>> Datapath supports ct_state >>> 2018-06-11T11:13:48.605Z|00051|ofproto_dpif|INFO|system@ovs-system: >>> Datapath supports ct_zone >>> 2018-06-11T11:13:48.605Z|00052|ofproto_dpif|INFO|system@ovs-system: >>> Datapath supports ct_mark >>> 2018-06-11T11:13:48.605Z|00053|ofproto_dpif|INFO|system@ovs-system: >>> Datapath supports ct_label >>> 2018-06-11T11:13:48.605Z|00054|ofproto_dpif|INFO|system@ovs-system: >>> Datapath does not support ct_state_nat >>> 2018-06-11T11:13:48.605Z|00055|ofproto_dpif|INFO|system@ovs-system: >>> Datapath does not support ct_orig_tuple >>> 2018-06-11T11:13:48.605Z|00056|ofproto_dpif|INFO|system@ovs-system: >>> Datapath does not support ct_orig_tuple6 >>> 2018-06-11T11:13:48.981Z|00057|bridge|INFO|bridge br-mlx: added >>> interface br-mlx on port 65534 >>> 2018-06-11T11:13:48.981Z|00058|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T11:13:48.981Z|00059|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T11:13:48.981Z|00060|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T11:13:48.982Z|00061|bridge|INFO|bridge br-int: added >>> interface br-int on port 65534 >>> 2018-06-11T11:13:48.982Z|00062|bridge|INFO|bridge br-int: added >>> interface int-br-ex on port 1 >>> 2018-06-11T11:13:48.982Z|00063|bridge|INFO|bridge br-int: added >>> interface patch-tun on port 2 >>> 2018-06-11T11:13:48.982Z|00064|bridge|INFO|bridge br-tun: added >>> interface patch-int on port 1 >>> 2018-06-11T11:13:48.982Z|00065|bridge|INFO|bridge br-tun: added >>> interface br-tun on port 65534 >>> 2018-06-11T11:13:48.982Z|00066|bridge|INFO|bridge br-ex: added >>> interface br-ex on port 65534 >>> 2018-06-11T11:13:48.982Z|00067|bridge|INFO|bridge br-ex: added >>> interface phy-br-ex on port 1 >>> 2018-06-11T11:13:48.982Z|00068|bridge|INFO|bridge br-mlx: using >>> datapath ID 000072528484a642 >>> 2018-06-11T11:13:48.982Z|00069|connmgr|INFO|br-mlx: added service controller "punix:/var/run/openvswitch/br-mlx.mgmt" >>> 2018-06-11T11:13:49.221Z|00070|bridge|INFO|bridge br-int: using >>> datapath ID 00006680927b6b40 >>> 2018-06-11T11:13:49.221Z|00071|connmgr|INFO|br-int: added service controller "punix:/var/run/openvswitch/br-int.mgmt" >>> 2018-06-11T11:13:49.221Z|00072|connmgr|INFO|br-int: added primary controller "tcp:127.0.0.1:6633" >>> 2018-06-11T11:13:49.221Z|00073|rconn|INFO|br-int<->tcp:127.0.0.1:6633: >>> connecting... >>> 2018-06-11T11:13:49.460Z|00074|bridge|INFO|bridge br-tun: using >>> datapath ID 000066532e5f0443 >>> 2018-06-11T11:13:49.461Z|00075|connmgr|INFO|br-tun: added service controller "punix:/var/run/openvswitch/br-tun.mgmt" >>> 2018-06-11T11:13:49.461Z|00076|connmgr|INFO|br-tun: added primary controller "tcp:127.0.0.1:6633" >>> 2018-06-11T11:13:49.461Z|00077|rconn|INFO|br-tun<->tcp:127.0.0.1:6633: >>> connecting... >>> 2018-06-11T11:13:49.692Z|00078|bridge|INFO|bridge br-ex: using >>> datapath ID 0000e2d1bdbe8e45 >>> 2018-06-11T11:13:49.692Z|00079|connmgr|INFO|br-ex: added service controller "punix:/var/run/openvswitch/br-ex.mgmt" >>> 2018-06-11T11:13:49.692Z|00080|connmgr|INFO|br-ex: added primary controller "tcp:127.0.0.1:6633" >>> 2018-06-11T11:13:49.692Z|00081|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: >>> connecting... >>> 2018-06-11T11:13:49.916Z|00082|timeval|WARN|Unreasonably long 4661ms >>> poll interval (16ms user, 4472ms system) >>> 2018-06-11T11:13:49.916Z|00083|timeval|WARN|faults: 3584706 minor, 0 >>> major >>> 2018-06-11T11:13:49.916Z|00084|timeval|WARN|disk: 0 reads, 16 writes 2018-06-11T11:13:49.916Z|00085|timeval|WARN|context switches: 147 voluntary, 25 involuntary 2018-06-11T11:13:49.916Z|00086|coverage|INFO|Event coverage, avg rate over last: 5 seconds, last minute, last hour, hash=2ba35161: >>> 2018-06-11T11:13:49.916Z|00087|coverage|INFO|bridge_reconfigure >>> 0.0/sec 0.000/sec 0.0000/sec total: 1 >>> 2018-06-11T11:13:49.916Z|00088|coverage|INFO|ofproto_flush >>> 0.0/sec 0.000/sec 0.0000/sec total: 4 >>> 2018-06-11T11:13:49.916Z|00089|coverage|INFO|ofproto_update_port >>> 0.0/sec 0.000/sec 0.0000/sec total: 8 >>> 2018-06-11T11:13:49.916Z|00090|coverage|INFO|rev_flow_table >>> 0.0/sec 0.000/sec 0.0000/sec total: 1 >>> 2018-06-11T11:13:49.916Z|00091|coverage|INFO|cmap_expand >>> 0.0/sec 0.000/sec 0.0000/sec total: 52 >>> 2018-06-11T11:13:49.916Z|00092|coverage|INFO|cmap_shrink >>> 0.0/sec 0.000/sec 0.0000/sec total: 23 >>> 2018-06-11T11:13:49.916Z|00093|coverage|INFO|dpif_port_add >>> 0.0/sec 0.000/sec 0.0000/sec total: 1 >>> 2018-06-11T11:13:49.916Z|00094|coverage|INFO|dpif_flow_flush >>> 0.0/sec 0.000/sec 0.0000/sec total: 6 >>> 2018-06-11T11:13:49.916Z|00095|coverage|INFO|dpif_flow_get >>> 0.0/sec 0.000/sec 0.0000/sec total: 29 >>> 2018-06-11T11:13:49.916Z|00096|coverage|INFO|dpif_flow_put >>> 0.0/sec 0.000/sec 0.0000/sec total: 37 >>> 2018-06-11T11:13:49.916Z|00097|coverage|INFO|dpif_flow_del >>> 0.0/sec 0.000/sec 0.0000/sec total: 29 >>> 2018-06-11T11:13:49.916Z|00098|coverage|INFO|dpif_execute >>> 0.0/sec 0.000/sec 0.0000/sec total: 8 >>> 2018-06-11T11:13:49.916Z|00099|coverage|INFO|flow_extract >>> 0.0/sec 0.000/sec 0.0000/sec total: 6 >>> 2018-06-11T11:13:49.916Z|00100|coverage|INFO|miniflow_malloc >>> 0.0/sec 0.000/sec 0.0000/sec total: 34 >>> 2018-06-11T11:13:49.916Z|00101|coverage|INFO|hmap_expand >>> 0.0/sec 0.000/sec 0.0000/sec total: 498 >>> 2018-06-11T11:13:49.916Z|00102|coverage|INFO|netdev_received >>> 0.0/sec 0.000/sec 0.0000/sec total: 1 >>> 2018-06-11T11:13:49.916Z|00103|coverage|INFO|netdev_get_stats >>> 0.0/sec 0.000/sec 0.0000/sec total: 8 >>> 2018-06-11T11:13:49.916Z|00104|coverage|INFO|poll_create_node >>> 0.0/sec 0.000/sec 0.0000/sec total: 23 >>> 2018-06-11T11:13:49.916Z|00105|coverage|INFO|seq_change >>> 0.0/sec 0.000/sec 0.0000/sec total: 2193 >>> 2018-06-11T11:13:49.916Z|00106|coverage|INFO|pstream_open >>> 0.0/sec 0.000/sec 0.0000/sec total: 9 >>> 2018-06-11T11:13:49.916Z|00107|coverage|INFO|stream_open >>> 0.0/sec 0.000/sec 0.0000/sec total: 4 >>> 2018-06-11T11:13:49.916Z|00108|coverage|INFO|util_xalloc >>> 0.0/sec 0.000/sec 0.0000/sec total: 12803 >>> 2018-06-11T11:13:49.916Z|00109|coverage|INFO|vconn_open >>> 0.0/sec 0.000/sec 0.0000/sec total: 3 >>> 2018-06-11T11:13:49.916Z|00110|coverage|INFO|netdev_set_policing >>> 0.0/sec 0.000/sec 0.0000/sec total: 4 >>> 2018-06-11T11:13:49.916Z|00111|coverage|INFO|netdev_get_ifindex >>> 0.0/sec 0.000/sec 0.0000/sec total: 4 >>> 2018-06-11T11:13:49.916Z|00112|coverage|INFO|netdev_get_hwaddr >>> 0.0/sec 0.000/sec 0.0000/sec total: 9 >>> 2018-06-11T11:13:49.916Z|00113|coverage|INFO|netdev_get_ethtool >>> 0.0/sec 0.000/sec 0.0000/sec total: 2 >>> 2018-06-11T11:13:49.916Z|00114|coverage|INFO|netlink_received >>> 0.0/sec 0.000/sec 0.0000/sec total: 85 >>> 2018-06-11T11:13:49.916Z|00115|coverage|INFO|netlink_recv_jumbo >>> 0.0/sec 0.000/sec 0.0000/sec total: 1 >>> 2018-06-11T11:13:49.916Z|00116|coverage|INFO|netlink_sent >>> 0.0/sec 0.000/sec 0.0000/sec total: 102 >>> 2018-06-11T11:13:49.916Z|00117|coverage|INFO|74 events never hit >>> 2018-06-11T11:13:49.916Z|00118|poll_loop|INFO|wakeup due to [POLLIN] >>> on fd 433 (FIFO pipe:[1025715]) at ../lib/ovs-rcu.c:229 (97% CPU >>> usage) 2018-06-11T11:13:49.917Z|00119|poll_loop|INFO|wakeup due to >>> [POLLIN] on fd 433 (FIFO pipe:[1025715]) at ../lib/ovs-rcu.c:229 (97% >>> CPU usage) 2018-06-11T11:13:49.917Z|00120|poll_loop|INFO|wakeup due >>> to 0-ms timeout at ../lib/ovs-rcu.c:229 (97% CPU usage) >>> 2018-06-11T11:13:49.917Z|00121|poll_loop|INFO|wakeup due to 0-ms >>> timeout at ../lib/ovs-rcu.c:229 (97% CPU usage) >>> 2018-06-11T11:13:49.920Z|00122|rconn|INFO|br-int<->tcp:127.0.0.1:6633: >>> connected >>> 2018-06-11T11:13:49.920Z|00123|rconn|INFO|br-tun<->tcp:127.0.0.1:6633: >>> connected >>> 2018-06-11T11:13:49.920Z|00124|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: >>> connected >>> 2018-06-11T11:13:49.926Z|00125|poll_loop|INFO|wakeup due to [POLLIN] >>> on fd 10 (NETLINK_ROUTE<->NETLINK_ROUTE) at >>> ../lib/netlink-socket.c:1331 (97% CPU usage) >>> 2018-06-11T11:13:49.926Z|00126|poll_loop|INFO|wakeup due to [POLLIN] >>> on fd 421 (character device /dev/net/tun) at >>> ../lib/netdev-linux.c:1171 (97% CPU usage) >>> 2018-06-11T11:13:49.926Z|00127|poll_loop|INFO|wakeup due to [POLLIN] >>> on fd 423 (127.0.0.1:33712<->127.0.0.1:6633) at >>> ../lib/stream-fd.c:157 (97% CPU usage) >>> 2018-06-11T11:13:49.926Z|00128|poll_loop|INFO|wakeup >>> due to [POLLIN] on fd 428 (127.0.0.1:33714<->127.0.0.1:6633) at >>> ../lib/stream-fd.c:157 (97% CPU usage) >>> 2018-06-11T11:13:49.926Z|00129|poll_loop|INFO|wakeup due to [POLLIN] >>> on fd 11 (<->/var/run/openvswitch/db.sock) at ../lib/stream-fd.c:157 >>> (97% CPU usage) 2018-06-11T11:13:49.926Z|00130|poll_loop|INFO|wakeup >>> due to [POLLIN] on fd 429 (127.0.0.1:33716<->127.0.0.1:6633) at >>> ../lib/stream-fd.c:157 (97% CPU usage) >>> 2018-06-11T11:13:49.930Z|00131|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T11:13:49.930Z|00132|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T11:13:49.930Z|00133|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T11:13:49.931Z|00134|bridge|INFO|ovs-vswitchd (Open >>> vSwitch) >>> 2.9.0 >>> 2018-06-11T11:13:53.662Z|00135|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T11:13:53.662Z|00136|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T11:13:53.662Z|00137|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T11:13:53.675Z|00138|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T11:13:53.675Z|00139|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T11:13:53.675Z|00140|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T11:13:53.682Z|00141|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T11:13:53.682Z|00142|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T11:13:53.682Z|00143|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T11:13:53.693Z|00144|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T11:13:53.693Z|00145|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T11:13:53.693Z|00146|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T11:13:53.705Z|00147|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T11:13:53.705Z|00148|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T11:13:53.705Z|00149|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T11:13:53.711Z|00150|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T11:13:53.711Z|00151|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T11:13:53.711Z|00152|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T11:13:53.717Z|00153|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T11:13:53.717Z|00154|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T11:13:53.717Z|00155|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T11:13:53.728Z|00156|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T11:13:53.728Z|00157|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T11:13:53.728Z|00158|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T11:13:53.743Z|00159|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T11:13:53.743Z|00160|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T11:13:53.743Z|00161|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T11:13:53.749Z|00162|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T11:13:53.749Z|00163|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T11:13:53.749Z|00164|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T11:13:53.756Z|00165|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T11:13:53.756Z|00166|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T11:13:53.756Z|00167|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T11:13:53.766Z|00168|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T11:13:53.766Z|00169|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T11:13:53.766Z|00170|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T11:13:55.270Z|00171|memory|INFO|1406100 kB peak resident >>> set size after 10.0 seconds >>> 2018-06-11T11:13:55.270Z|00172|memory|INFO|handlers:118 ofconns:3 >>> ports:8 revalidators:42 rules:36 udpif keys:1 >>> 2018-06-11T11:14:03.699Z|00173|connmgr|INFO|br-int<->tcp:127.0.0.1:6633: >>> 5 flow_mods 10 s ago (5 adds) >>> 2018-06-11T11:14:03.737Z|00174|connmgr|INFO|br-ex<->tcp:127.0.0.1:6633: >>> 2 flow_mods 10 s ago (2 adds) >>> 2018-06-11T11:14:03.773Z|00175|connmgr|INFO|br-tun<->tcp:127.0.0.1:6633: >>> 12 flow_mods 10 s ago (12 adds) >>> >>> Best regards >>> >>> On 06/11/2018 12:57 PM, Raslan Darawsheh wrote: >>>> What I really want to see is the init process for OVS-DPDK that dumps the DPDK initialization. >>>> I think you will find it at the beginning of the log that you just sent. >>>> >>>> Kindest regards, >>>> Raslan Darawsheh >>>> >>>> -----Original Message----- >>>> From: Tobias Urdin [mailto:tobias.urdin@crystone.com] >>>> Sent: Monday, June 11, 2018 1:00 PM >>>> To: Raslan Darawsheh <rasland@mellanox.com> >>>> Cc: users@dpdk.org >>>> Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch >>>> >>>> Hello Raslan, >>>> Thank you for your reply. >>>> >>>> For reference the br-mlx bridge was created like this: >>>> $ovs-vsctl add-br br-mlx -- set bridge br-mlx datapath_type=netdev >>>> >>>> $ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 >>>> type=dpdk options:dpdk-devargs=mlx4_0 >>>> ovs-vsctl: Error detected while setting up 'enp59s0d1': Error attaching device 'mlx4_0' to DPDK. See ovs-vswitchd log for details. >>>> ovs-vsctl: The default log directory is "/var/log/openvswitch". >>>> >>>> In the ovs-vswitchd.log >>>> 2018-06-11T09:56:20.613Z|00185|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T09:56:20.613Z|00186|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T09:56:20.613Z|00187|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> >>>> Here is the complete log after restarting openvswitch-switch service: >>>> 2018-06-11T09:57:49.380Z|00134|bridge|INFO|ovs-vswitchd (Open >>>> vSwitch) >>>> 2.9.0 >>>> 2018-06-11T09:57:52.812Z|00135|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T09:57:52.812Z|00136|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T09:57:52.812Z|00137|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T09:57:52.825Z|00138|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T09:57:52.825Z|00139|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T09:57:52.825Z|00140|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T09:57:52.832Z|00141|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T09:57:52.832Z|00142|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T09:57:52.832Z|00143|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T09:57:52.843Z|00144|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T09:57:52.843Z|00145|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T09:57:52.843Z|00146|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T09:57:52.856Z|00147|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T09:57:52.856Z|00148|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T09:57:52.856Z|00149|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T09:57:52.862Z|00150|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T09:57:52.862Z|00151|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T09:57:52.862Z|00152|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T09:57:52.870Z|00153|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T09:57:52.870Z|00154|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T09:57:52.870Z|00155|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T09:57:52.889Z|00156|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T09:57:52.889Z|00157|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T09:57:52.889Z|00158|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T09:57:52.903Z|00159|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T09:57:52.903Z|00160|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T09:57:52.903Z|00161|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T09:57:52.908Z|00162|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T09:57:52.908Z|00163|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T09:57:52.908Z|00164|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T09:57:52.914Z|00165|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T09:57:52.914Z|00166|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T09:57:52.914Z|00167|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T09:57:52.923Z|00168|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T09:57:52.923Z|00169|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T09:57:52.923Z|00170|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T09:57:54.808Z|00171|memory|INFO|1406100 kB peak resident >>>> set size after 10.1 seconds >>>> 2018-06-11T09:57:54.808Z|00172|memory|INFO|handlers:118 ofconns:3 >>>> ports:8 revalidators:42 rules:36 udpif keys:1 >>>> 2018-06-11T09:58:02.850Z|00173|connmgr|INFO|br-int<->tcp:127.0.0.1:6633: >>>> 5 flow_mods 10 s ago (5 adds) >>>> 2018-06-11T09:58:02.898Z|00174|connmgr|INFO|br-ex<->tcp:127.0.0.1:6633: >>>> 2 flow_mods 10 s ago (2 adds) >>>> 2018-06-11T09:58:02.930Z|00175|connmgr|INFO|br-tun<->tcp:127.0.0.1:6633: >>>> 12 flow_mods 10 s ago (12 adds) >>>> >>>> Was that what you mean by init log? >>>> Best regards >>>> >>>> On 06/11/2018 11:51 AM, Raslan Darawsheh wrote: >>>>> Hi Tobias, >>>>> >>>>> Can you try using the following ? >>>>> >>>>> ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 >>>>> type=dpdk options:dpdk-devargs=mlx4_0 >>>>> >>>>> if it didn't work can you send the OVS-DPDK init log ? >>>>> >>>>> Kindest regards, >>>>> Raslan Darawsheh >>>>> >>>>> -----Original Message----- >>>>> From: users [mailto:users-bounces@dpdk.org] On Behalf Of Tobias >>>>> Urdin >>>>> Sent: Monday, June 11, 2018 12:15 PM >>>>> To: users@dpdk.org >>>>> Subject: [dpdk-users] mellanox connect-x 3 pro dpdk with >>>>> openvswitch >>>>> >>>>> Hello, >>>>> >>>>> I'm stuck trying to test DPDK with a Mellanox Connect-X 3 Pro card >>>>> on Ubuntu 16.04 >>>>> >>>>> I've enabled iommu, fixed hugepages and has installed and are running the openvswitch-switch-dpdk alternative. >>>>> >>>>> $lspci | grep -i mellanox >>>>> 3b:00.0 Ethernet controller: Mellanox Technologies MT27520 Family >>>>> [ConnectX-3 Pro] >>>>> >>>>> $lsmod | grep -i mlx >>>>> mlx4_en 110592 0 >>>>> vxlan 49152 2 i40e,mlx4_en >>>>> mlx4_core 286720 1 mlx4_en >>>>> ptp 20480 3 igb,i40e,mlx4_en >>>>> >>>>> $cat /etc/modprobe.d/mlx4_core.conf options mlx4_core >>>>> log_num_mgm_entry_size=-1 >>>>> >>>>> >>>>> $ovs-vsctl add-port br-mlx dpdk0 -- set Interface dpdk0 type=dpdk >>>>> >>>>> Bridge br-mlx >>>>> Port "dpdk0" >>>>> Interface "dpdk0" >>>>> type: dpdk >>>>> error: "'dpdk0' is missing 'options:dpdk-devargs'. The old 'dpdk<port_id>' names are not supported" >>>>> >>>>> >>>>> $ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 >>>>> type=dpdk options:dpdk-devargs=0000:3b:00.0,1 >>>>> ovs-vsctl: Error detected while setting up 'enp59s0d1': Error attaching device '0000:3b:00.0,1' to DPDK. See ovs-vswitchd log for details. >>>>> ovs-vsctl: The default log directory is "/var/log/openvswitch". >>>>> >>>>> 2018-06-11T09:10:38.696Z|00177|netdev_dpdk|WARN|Error attaching >>>>> device '0000:3b:00.0,1' to DPDK >>>>> 2018-06-11T09:10:38.696Z|00178|netdev|WARN|enp59s0d1: could not set >>>>> configuration (Invalid argument) >>>>> >>>>> 2018-06-11T09:12:50.994Z|00181|dpdk|ERR|EAL: Driver cannot attach >>>>> the device (0000:3b:00.0) >>>>> 2018-06-11T09:12:50.994Z|00182|dpdk|ERR|EAL: No port found for >>>>> device >>>>> (0000:3b:00.0) >>>>> 2018-06-11T09:12:50.994Z|00183|netdev_dpdk|WARN|Error >>>>> attaching device '0000:3b:00.0,1' to DPDK >>>>> 2018-06-11T09:12:50.994Z|00184|netdev|WARN|enp59s0d1: could not set >>>>> configuration (Invalid argument) >>>>> >>>>> $ovs-vsctl add-port br-mlx dpdk0 -- set Interface dpdk0 type=dpdk options:dpdk-devargs="class=eth,mac=ec:0d:9a:b0:13:21" >>>>> ovs-vsctl: Error detected while setting up 'dpdk0': Error attaching device 'class=eth,mac=ec:0d:9a:b0:13:21' to DPDK. See ovs-vswitchd log for details. >>>>> ovs-vsctl: The default log directory is "/var/log/openvswitch". >>>>> >>>>> Port "dpdk0" >>>>> Interface "dpdk0" >>>>> type: dpdk >>>>> options: {dpdk-devargs="class=eth,mac=ec:0d:9a:b0:13:21"} >>>>> error: "Error attaching device 'class=eth,mac=ec:0d:9a:b0:13:21' to DPDK" >>>>> >>>>> >>>>> Anybody know what I might have missed? I have tried restarting the openvswitch services after adding the interfaces as well but no difference. >>>>> >>>>> The logs doesn't say anything interesting from what I can see :( >>>>> >>>>> Best regards >>>>> > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [dpdk-users] mellanox connect-x 3 pro dpdk with openvswitch 2018-06-11 12:05 ` Tobias Urdin @ 2018-06-11 12:07 ` Raslan Darawsheh 2018-06-11 12:14 ` Tobias Urdin 0 siblings, 1 reply; 14+ messages in thread From: Raslan Darawsheh @ 2018-06-11 12:07 UTC (permalink / raw) To: Tobias Urdin; +Cc: users I think OVS-DPDK has static linking for dpdk and I'm not sure you can do it this way Kindest regards, Raslan Darawsheh -----Original Message----- From: Tobias Urdin [mailto:tobias.urdin@crystone.com] Sent: Monday, June 11, 2018 3:05 PM To: Raslan Darawsheh <rasland@mellanox.com> Cc: users@dpdk.org Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch Ok, I'm more wondering if there is any way I can build a librte-pmd-mlx4 package that supplies the compiled PMD or something like that instead of having to rebuild all the major parts? Thanks! Best regards On 06/11/2018 01:51 PM, Raslan Darawsheh wrote: > I think you can upgrade the current version that you have as seen in this article: > https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdocs > .openvswitch.org%2Fen%2Flatest%2Fintro%2Finstall%2Fgeneral%2F%23hot-up > grading&data=02%7C01%7Crasland%40mellanox.com%7Cfdaa105976c744dba2f908 > d5cf9394cc%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C1%7C63664315514160 > 5703&sdata=VhpqI3M4RdnD5Shpbt6O8YmufIKiyoYK9kxO9TR1Yrk%3D&reserved=0 > > but, I haven't tried it before. > > Kindest regards, > Raslan Darawsheh > > -----Original Message----- > From: Tobias Urdin [mailto:tobias.urdin@crystone.com] > Sent: Monday, June 11, 2018 2:45 PM > To: Raslan Darawsheh <rasland@mellanox.com> > Cc: users@dpdk.org > Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch > > Is there any to skip compiling DPDK and OVS+OVS-DPDK and instead use the ones I have installed? > Only compiling the Mellanox PMD module/binary or something similar to that? > > I can see that other drivers supply pmd packages like below. > > $dpkg -l | grep -E "openvswitch|pmd|dpdk" > ii dpdk > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (runtime) > ii librte-pmd-af-packet17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-af-packet runtime library) > ii librte-pmd-ark17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte_pmd_ark runtime library) > ii librte-pmd-avp17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte_pmd_avp runtime library) > ii librte-pmd-bnxt17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-bnxt runtime library) > ii librte-pmd-bond17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-bond runtime library) ii librte-pmd-crypto-scheduler17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte_pmd_crypto_scheduler runtime library) > ii librte-pmd-cxgbe17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-cxgbe runtime library) > ii librte-pmd-e1000-17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-e1000 runtime library) > ii librte-pmd-ena17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-ena runtime library) > ii librte-pmd-enic17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-enic runtime library) > ii librte-pmd-failsafe17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-failsafe runtime library) > ii librte-pmd-fm10k17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-fm10k runtime library) > ii librte-pmd-i40e17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-i40e runtime library) > ii librte-pmd-ixgbe17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-ixgbe runtime library) > ii librte-pmd-kni17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte_pmd_kni runtime library) > ii librte-pmd-lio17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte_pmd_lio runtime library) > ii librte-pmd-nfp17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte_pmd_nfp runtime library) > ii librte-pmd-null-crypto17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-null-crypto runtime library) > ii librte-pmd-null17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-null runtime library) ii librte-pmd-octeontx-ssovf17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte_pmd_octeontx_ssovf runtime library) > ii librte-pmd-octeontx17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte_pmd_octeontx runtime library) > ii librte-pmd-pcap17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-pcap runtime library) > ii librte-pmd-qede17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-qede runtime library) > ii librte-pmd-ring17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-ring runtime library) > ii librte-pmd-sfc-efx17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte_pmd_sfc_efx runtime library) ii librte-pmd-skeleton-event17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte_pmd_skeleton_event runtime library) > ii librte-pmd-softnic17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-softnic runtime library) > ii librte-pmd-sw-event17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte_pmd_sw_event runtime library) > ii librte-pmd-tap17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte_pmd_tap runtime library) ii librte-pmd-thunderx-nicvf17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte_pmd_thunderx_nicvf runtime library) > ii librte-pmd-vhost17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-vhost runtime library) > ii librte-pmd-virtio17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-virtio runtime library) > ii librte-pmd-vmxnet3-uio17.11:amd64 > 17.11.1-6~cloud0 amd64 Data Plane > Development Kit (librte-pmd-vmxnet3-uio runtime library) > ii openvswitch-common > 2.9.0-0ubuntu1~cloud0 amd64 Open vSwitch > common components > ii openvswitch-switch > 2.9.0-0ubuntu1~cloud0 amd64 Open vSwitch > switch implementations > ii openvswitch-switch-dpdk > 2.9.0-0ubuntu1~cloud0 amd64 DPDK enabled > Open vSwitch switch implementation > ii python-openvswitch > 2.9.0-0ubuntu1~cloud0 all Python bindings > for Open vSwitch > > Thanks for all the help! > Best regards > > > On 06/11/2018 01:37 PM, Raslan Darawsheh wrote: >> You can refer to this community page for compiling OVS-DPDK with MLX pmds enabled: >> https://community.mellanox.com/docs/DOC-2637 >> >> >> Kindest regards, >> Raslan Darawsheh >> >> -----Original Message----- >> From: Tobias Urdin [mailto:tobias.urdin@crystone.com] >> Sent: Monday, June 11, 2018 2:35 PM >> To: Raslan Darawsheh <rasland@mellanox.com> >> Cc: users@dpdk.org >> Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch >> >> I haven't compiled any DPDK or Mellanox related modules/packages. >> What I have is ovs, ovs-dpdk, mlx4 module, I have also loaded the ib_uverbs driver. >> >> Is it the librte_pmd_mlx4 I have to compile manually? >> >> Do you know any updated documentation about this, there isn't much relevant and searching for documentation and the correct mlnx driver and code is a struggle. >> >> Best regards >> >> On 06/11/2018 01:20 PM, Raslan Darawsheh wrote: >>> From what I'm seeing it looks like it didn't prop the MLX4 device at all. >>> Can you make sure you have it enabled in your DPDK package when you compiled DPDK ? >>> >>> Kindest regards, >>> Raslan Darawsheh >>> >>> -----Original Message----- >>> From: Tobias Urdin [mailto:tobias.urdin@crystone.com] >>> Sent: Monday, June 11, 2018 2:16 PM >>> To: Raslan Darawsheh <rasland@mellanox.com> >>> Cc: users@dpdk.org >>> Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch >>> >>> Ok, here is the complete log when restarting the openvswitch-switch service, see below. >>> >>> Also here is the kernel for information: >>> $uname -r >>> 4.4.0-127-generic >>> >>> 2018-06-11T11:13:42.729Z|00176|bridge|INFO|bridge br-mlx: deleted >>> interface br-mlx on port 65534 >>> 2018-06-11T11:13:43.274Z|00177|bridge|INFO|bridge br-int: deleted >>> interface br-int on port 65534 >>> 2018-06-11T11:13:43.275Z|00178|bridge|INFO|bridge br-int: deleted >>> interface int-br-ex on port 1 >>> 2018-06-11T11:13:43.275Z|00179|bridge|INFO|bridge br-int: deleted >>> interface patch-tun on port 2 >>> 2018-06-11T11:13:43.762Z|00180|bridge|INFO|bridge br-tun: deleted >>> interface patch-int on port 1 >>> 2018-06-11T11:13:43.762Z|00181|bridge|INFO|bridge br-tun: deleted >>> interface br-tun on port 65534 >>> 2018-06-11T11:13:44.231Z|00182|bridge|INFO|bridge br-ex: deleted >>> interface br-ex on port 65534 >>> 2018-06-11T11:13:44.232Z|00183|bridge|INFO|bridge br-ex: deleted >>> interface phy-br-ex on port 1 >>> 2018-06-11T11:13:45.245Z|00001|vlog|INFO|opened log file >>> /var/log/openvswitch/ovs-vswitchd.log >>> 2018-06-11T11:13:45.252Z|00002|ovs_numa|INFO|Discovered 40 CPU cores >>> on NUMA node 0 >>> 2018-06-11T11:13:45.252Z|00003|ovs_numa|INFO|Discovered >>> 40 CPU cores on NUMA node 1 >>> 2018-06-11T11:13:45.252Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes >>> and >>> 80 CPU cores >>> 2018-06-11T11:13:45.252Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: >>> connecting... >>> 2018-06-11T11:13:45.252Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: >>> connected >>> 2018-06-11T11:13:45.256Z|00007|dpdk|INFO|Using DPDK 17.11.0 2018-06-11T11:13:45.256Z|00008|dpdk|INFO|DPDK Enabled - initializing... >>> 2018-06-11T11:13:45.256Z|00009|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch 2018-06-11T11:13:45.256Z|00010|dpdk|INFO|IOMMU support for vhost-user-client disabled. >>> 2018-06-11T11:13:45.256Z|00011|dpdk|INFO|EAL ARGS: ovs-vswitchd >>> --socket-mem 1024,0 -c 0x00000001 >>> 2018-06-11T11:13:45.258Z|00012|dpdk|INFO|EAL: Detected 80 lcore(s) >>> 2018-06-11T11:13:45.285Z|00013|dpdk|INFO|EAL: Probing VFIO support... >>> 2018-06-11T11:13:45.285Z|00014|dpdk|INFO|EAL: VFIO support >>> initialized >>> 2018-06-11T11:13:48.214Z|00015|dpdk|INFO|EAL: PCI device 0000:01:00.0 on NUMA socket 0 >>> 2018-06-11T11:13:48.214Z|00016|dpdk|INFO|EAL: probe driver: 8086:1521 >>> net_e1000_igb >>> 2018-06-11T11:13:48.214Z|00017|dpdk|INFO|EAL: PCI device 0000:01:00.1 on NUMA socket 0 >>> 2018-06-11T11:13:48.214Z|00018|dpdk|INFO|EAL: probe driver: 8086:1521 >>> net_e1000_igb >>> 2018-06-11T11:13:48.214Z|00019|dpdk|INFO|EAL: PCI device 0000:18:00.0 on NUMA socket 0 >>> 2018-06-11T11:13:48.214Z|00020|dpdk|INFO|EAL: probe driver: 8086:1572 >>> net_i40e >>> 2018-06-11T11:13:48.214Z|00021|dpdk|INFO|EAL: PCI device 0000:18:00.1 on NUMA socket 0 >>> 2018-06-11T11:13:48.214Z|00022|dpdk|INFO|EAL: probe driver: 8086:1572 >>> net_i40e >>> 2018-06-11T11:13:48.214Z|00023|dpdk|INFO|DPDK pdump packet capture >>> enabled 2018-06-11T11:13:48.224Z|00024|dpdk|INFO|DPDK Enabled - >>> initialized >>> 2018-06-11T11:13:48.235Z|00025|ofproto_dpif|INFO|netdev@ovs-netdev: >>> Datapath supports recirculation >>> 2018-06-11T11:13:48.235Z|00026|ofproto_dpif|INFO|netdev@ovs-netdev: >>> VLAN header stack length probed as 1 >>> 2018-06-11T11:13:48.235Z|00027|ofproto_dpif|INFO|netdev@ovs-netdev: >>> MPLS label stack length probed as 3 >>> 2018-06-11T11:13:48.235Z|00028|ofproto_dpif|INFO|netdev@ovs-netdev: >>> Datapath supports truncate action >>> 2018-06-11T11:13:48.235Z|00029|ofproto_dpif|INFO|netdev@ovs-netdev: >>> Datapath supports unique flow ids >>> 2018-06-11T11:13:48.235Z|00030|ofproto_dpif|INFO|netdev@ovs-netdev: >>> Datapath supports clone action >>> 2018-06-11T11:13:48.235Z|00031|ofproto_dpif|INFO|netdev@ovs-netdev: >>> Max sample nesting level probed as 10 >>> 2018-06-11T11:13:48.235Z|00032|ofproto_dpif|INFO|netdev@ovs-netdev: >>> Datapath supports eventmask in conntrack action >>> 2018-06-11T11:13:48.235Z|00033|ofproto_dpif|INFO|netdev@ovs-netdev: >>> Datapath supports ct_clear action >>> 2018-06-11T11:13:48.235Z|00034|ofproto_dpif|INFO|netdev@ovs-netdev: >>> Datapath supports ct_state >>> 2018-06-11T11:13:48.235Z|00035|ofproto_dpif|INFO|netdev@ovs-netdev: >>> Datapath supports ct_zone >>> 2018-06-11T11:13:48.235Z|00036|ofproto_dpif|INFO|netdev@ovs-netdev: >>> Datapath supports ct_mark >>> 2018-06-11T11:13:48.235Z|00037|ofproto_dpif|INFO|netdev@ovs-netdev: >>> Datapath supports ct_label >>> 2018-06-11T11:13:48.235Z|00038|ofproto_dpif|INFO|netdev@ovs-netdev: >>> Datapath supports ct_state_nat >>> 2018-06-11T11:13:48.235Z|00039|ofproto_dpif|INFO|netdev@ovs-netdev: >>> Datapath supports ct_orig_tuple >>> 2018-06-11T11:13:48.235Z|00040|ofproto_dpif|INFO|netdev@ovs-netdev: >>> Datapath supports ct_orig_tuple6 >>> 2018-06-11T11:13:48.605Z|00041|ofproto_dpif|INFO|system@ovs-system: >>> Datapath supports recirculation >>> 2018-06-11T11:13:48.605Z|00042|ofproto_dpif|INFO|system@ovs-system: >>> VLAN header stack length probed as 1 >>> 2018-06-11T11:13:48.605Z|00043|ofproto_dpif|INFO|system@ovs-system: >>> MPLS label stack length probed as 1 >>> 2018-06-11T11:13:48.605Z|00044|ofproto_dpif|INFO|system@ovs-system: >>> Datapath does not support truncate action >>> 2018-06-11T11:13:48.605Z|00045|ofproto_dpif|INFO|system@ovs-system: >>> Datapath supports unique flow ids >>> 2018-06-11T11:13:48.605Z|00046|ofproto_dpif|INFO|system@ovs-system: >>> Datapath does not support clone action >>> 2018-06-11T11:13:48.605Z|00047|ofproto_dpif|INFO|system@ovs-system: >>> Max sample nesting level probed as 3 >>> 2018-06-11T11:13:48.605Z|00048|ofproto_dpif|INFO|system@ovs-system: >>> Datapath does not support eventmask in conntrack action >>> 2018-06-11T11:13:48.605Z|00049|ofproto_dpif|INFO|system@ovs-system: >>> Datapath does not support ct_clear action >>> 2018-06-11T11:13:48.605Z|00050|ofproto_dpif|INFO|system@ovs-system: >>> Datapath supports ct_state >>> 2018-06-11T11:13:48.605Z|00051|ofproto_dpif|INFO|system@ovs-system: >>> Datapath supports ct_zone >>> 2018-06-11T11:13:48.605Z|00052|ofproto_dpif|INFO|system@ovs-system: >>> Datapath supports ct_mark >>> 2018-06-11T11:13:48.605Z|00053|ofproto_dpif|INFO|system@ovs-system: >>> Datapath supports ct_label >>> 2018-06-11T11:13:48.605Z|00054|ofproto_dpif|INFO|system@ovs-system: >>> Datapath does not support ct_state_nat >>> 2018-06-11T11:13:48.605Z|00055|ofproto_dpif|INFO|system@ovs-system: >>> Datapath does not support ct_orig_tuple >>> 2018-06-11T11:13:48.605Z|00056|ofproto_dpif|INFO|system@ovs-system: >>> Datapath does not support ct_orig_tuple6 >>> 2018-06-11T11:13:48.981Z|00057|bridge|INFO|bridge br-mlx: added >>> interface br-mlx on port 65534 >>> 2018-06-11T11:13:48.981Z|00058|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T11:13:48.981Z|00059|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T11:13:48.981Z|00060|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T11:13:48.982Z|00061|bridge|INFO|bridge br-int: added >>> interface br-int on port 65534 >>> 2018-06-11T11:13:48.982Z|00062|bridge|INFO|bridge br-int: added >>> interface int-br-ex on port 1 >>> 2018-06-11T11:13:48.982Z|00063|bridge|INFO|bridge br-int: added >>> interface patch-tun on port 2 >>> 2018-06-11T11:13:48.982Z|00064|bridge|INFO|bridge br-tun: added >>> interface patch-int on port 1 >>> 2018-06-11T11:13:48.982Z|00065|bridge|INFO|bridge br-tun: added >>> interface br-tun on port 65534 >>> 2018-06-11T11:13:48.982Z|00066|bridge|INFO|bridge br-ex: added >>> interface br-ex on port 65534 >>> 2018-06-11T11:13:48.982Z|00067|bridge|INFO|bridge br-ex: added >>> interface phy-br-ex on port 1 >>> 2018-06-11T11:13:48.982Z|00068|bridge|INFO|bridge br-mlx: using >>> datapath ID 000072528484a642 >>> 2018-06-11T11:13:48.982Z|00069|connmgr|INFO|br-mlx: added service controller "punix:/var/run/openvswitch/br-mlx.mgmt" >>> 2018-06-11T11:13:49.221Z|00070|bridge|INFO|bridge br-int: using >>> datapath ID 00006680927b6b40 >>> 2018-06-11T11:13:49.221Z|00071|connmgr|INFO|br-int: added service controller "punix:/var/run/openvswitch/br-int.mgmt" >>> 2018-06-11T11:13:49.221Z|00072|connmgr|INFO|br-int: added primary controller "tcp:127.0.0.1:6633" >>> 2018-06-11T11:13:49.221Z|00073|rconn|INFO|br-int<->tcp:127.0.0.1:6633: >>> connecting... >>> 2018-06-11T11:13:49.460Z|00074|bridge|INFO|bridge br-tun: using >>> datapath ID 000066532e5f0443 >>> 2018-06-11T11:13:49.461Z|00075|connmgr|INFO|br-tun: added service controller "punix:/var/run/openvswitch/br-tun.mgmt" >>> 2018-06-11T11:13:49.461Z|00076|connmgr|INFO|br-tun: added primary controller "tcp:127.0.0.1:6633" >>> 2018-06-11T11:13:49.461Z|00077|rconn|INFO|br-tun<->tcp:127.0.0.1:6633: >>> connecting... >>> 2018-06-11T11:13:49.692Z|00078|bridge|INFO|bridge br-ex: using >>> datapath ID 0000e2d1bdbe8e45 >>> 2018-06-11T11:13:49.692Z|00079|connmgr|INFO|br-ex: added service controller "punix:/var/run/openvswitch/br-ex.mgmt" >>> 2018-06-11T11:13:49.692Z|00080|connmgr|INFO|br-ex: added primary controller "tcp:127.0.0.1:6633" >>> 2018-06-11T11:13:49.692Z|00081|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: >>> connecting... >>> 2018-06-11T11:13:49.916Z|00082|timeval|WARN|Unreasonably long 4661ms >>> poll interval (16ms user, 4472ms system) >>> 2018-06-11T11:13:49.916Z|00083|timeval|WARN|faults: 3584706 minor, 0 >>> major >>> 2018-06-11T11:13:49.916Z|00084|timeval|WARN|disk: 0 reads, 16 writes 2018-06-11T11:13:49.916Z|00085|timeval|WARN|context switches: 147 voluntary, 25 involuntary 2018-06-11T11:13:49.916Z|00086|coverage|INFO|Event coverage, avg rate over last: 5 seconds, last minute, last hour, hash=2ba35161: >>> 2018-06-11T11:13:49.916Z|00087|coverage|INFO|bridge_reconfigure >>> 0.0/sec 0.000/sec 0.0000/sec total: 1 >>> 2018-06-11T11:13:49.916Z|00088|coverage|INFO|ofproto_flush >>> 0.0/sec 0.000/sec 0.0000/sec total: 4 >>> 2018-06-11T11:13:49.916Z|00089|coverage|INFO|ofproto_update_port >>> 0.0/sec 0.000/sec 0.0000/sec total: 8 >>> 2018-06-11T11:13:49.916Z|00090|coverage|INFO|rev_flow_table >>> 0.0/sec 0.000/sec 0.0000/sec total: 1 >>> 2018-06-11T11:13:49.916Z|00091|coverage|INFO|cmap_expand >>> 0.0/sec 0.000/sec 0.0000/sec total: 52 >>> 2018-06-11T11:13:49.916Z|00092|coverage|INFO|cmap_shrink >>> 0.0/sec 0.000/sec 0.0000/sec total: 23 >>> 2018-06-11T11:13:49.916Z|00093|coverage|INFO|dpif_port_add >>> 0.0/sec 0.000/sec 0.0000/sec total: 1 >>> 2018-06-11T11:13:49.916Z|00094|coverage|INFO|dpif_flow_flush >>> 0.0/sec 0.000/sec 0.0000/sec total: 6 >>> 2018-06-11T11:13:49.916Z|00095|coverage|INFO|dpif_flow_get >>> 0.0/sec 0.000/sec 0.0000/sec total: 29 >>> 2018-06-11T11:13:49.916Z|00096|coverage|INFO|dpif_flow_put >>> 0.0/sec 0.000/sec 0.0000/sec total: 37 >>> 2018-06-11T11:13:49.916Z|00097|coverage|INFO|dpif_flow_del >>> 0.0/sec 0.000/sec 0.0000/sec total: 29 >>> 2018-06-11T11:13:49.916Z|00098|coverage|INFO|dpif_execute >>> 0.0/sec 0.000/sec 0.0000/sec total: 8 >>> 2018-06-11T11:13:49.916Z|00099|coverage|INFO|flow_extract >>> 0.0/sec 0.000/sec 0.0000/sec total: 6 >>> 2018-06-11T11:13:49.916Z|00100|coverage|INFO|miniflow_malloc >>> 0.0/sec 0.000/sec 0.0000/sec total: 34 >>> 2018-06-11T11:13:49.916Z|00101|coverage|INFO|hmap_expand >>> 0.0/sec 0.000/sec 0.0000/sec total: 498 >>> 2018-06-11T11:13:49.916Z|00102|coverage|INFO|netdev_received >>> 0.0/sec 0.000/sec 0.0000/sec total: 1 >>> 2018-06-11T11:13:49.916Z|00103|coverage|INFO|netdev_get_stats >>> 0.0/sec 0.000/sec 0.0000/sec total: 8 >>> 2018-06-11T11:13:49.916Z|00104|coverage|INFO|poll_create_node >>> 0.0/sec 0.000/sec 0.0000/sec total: 23 >>> 2018-06-11T11:13:49.916Z|00105|coverage|INFO|seq_change >>> 0.0/sec 0.000/sec 0.0000/sec total: 2193 >>> 2018-06-11T11:13:49.916Z|00106|coverage|INFO|pstream_open >>> 0.0/sec 0.000/sec 0.0000/sec total: 9 >>> 2018-06-11T11:13:49.916Z|00107|coverage|INFO|stream_open >>> 0.0/sec 0.000/sec 0.0000/sec total: 4 >>> 2018-06-11T11:13:49.916Z|00108|coverage|INFO|util_xalloc >>> 0.0/sec 0.000/sec 0.0000/sec total: 12803 >>> 2018-06-11T11:13:49.916Z|00109|coverage|INFO|vconn_open >>> 0.0/sec 0.000/sec 0.0000/sec total: 3 >>> 2018-06-11T11:13:49.916Z|00110|coverage|INFO|netdev_set_policing >>> 0.0/sec 0.000/sec 0.0000/sec total: 4 >>> 2018-06-11T11:13:49.916Z|00111|coverage|INFO|netdev_get_ifindex >>> 0.0/sec 0.000/sec 0.0000/sec total: 4 >>> 2018-06-11T11:13:49.916Z|00112|coverage|INFO|netdev_get_hwaddr >>> 0.0/sec 0.000/sec 0.0000/sec total: 9 >>> 2018-06-11T11:13:49.916Z|00113|coverage|INFO|netdev_get_ethtool >>> 0.0/sec 0.000/sec 0.0000/sec total: 2 >>> 2018-06-11T11:13:49.916Z|00114|coverage|INFO|netlink_received >>> 0.0/sec 0.000/sec 0.0000/sec total: 85 >>> 2018-06-11T11:13:49.916Z|00115|coverage|INFO|netlink_recv_jumbo >>> 0.0/sec 0.000/sec 0.0000/sec total: 1 >>> 2018-06-11T11:13:49.916Z|00116|coverage|INFO|netlink_sent >>> 0.0/sec 0.000/sec 0.0000/sec total: 102 >>> 2018-06-11T11:13:49.916Z|00117|coverage|INFO|74 events never hit >>> 2018-06-11T11:13:49.916Z|00118|poll_loop|INFO|wakeup due to [POLLIN] >>> on fd 433 (FIFO pipe:[1025715]) at ../lib/ovs-rcu.c:229 (97% CPU >>> usage) 2018-06-11T11:13:49.917Z|00119|poll_loop|INFO|wakeup due to >>> [POLLIN] on fd 433 (FIFO pipe:[1025715]) at ../lib/ovs-rcu.c:229 >>> (97% CPU usage) 2018-06-11T11:13:49.917Z|00120|poll_loop|INFO|wakeup >>> due to 0-ms timeout at ../lib/ovs-rcu.c:229 (97% CPU usage) >>> 2018-06-11T11:13:49.917Z|00121|poll_loop|INFO|wakeup due to 0-ms >>> timeout at ../lib/ovs-rcu.c:229 (97% CPU usage) >>> 2018-06-11T11:13:49.920Z|00122|rconn|INFO|br-int<->tcp:127.0.0.1:6633: >>> connected >>> 2018-06-11T11:13:49.920Z|00123|rconn|INFO|br-tun<->tcp:127.0.0.1:6633: >>> connected >>> 2018-06-11T11:13:49.920Z|00124|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: >>> connected >>> 2018-06-11T11:13:49.926Z|00125|poll_loop|INFO|wakeup due to [POLLIN] >>> on fd 10 (NETLINK_ROUTE<->NETLINK_ROUTE) at >>> ../lib/netlink-socket.c:1331 (97% CPU usage) >>> 2018-06-11T11:13:49.926Z|00126|poll_loop|INFO|wakeup due to [POLLIN] >>> on fd 421 (character device /dev/net/tun) at >>> ../lib/netdev-linux.c:1171 (97% CPU usage) >>> 2018-06-11T11:13:49.926Z|00127|poll_loop|INFO|wakeup due to [POLLIN] >>> on fd 423 (127.0.0.1:33712<->127.0.0.1:6633) at >>> ../lib/stream-fd.c:157 (97% CPU usage) >>> 2018-06-11T11:13:49.926Z|00128|poll_loop|INFO|wakeup >>> due to [POLLIN] on fd 428 (127.0.0.1:33714<->127.0.0.1:6633) at >>> ../lib/stream-fd.c:157 (97% CPU usage) >>> 2018-06-11T11:13:49.926Z|00129|poll_loop|INFO|wakeup due to [POLLIN] >>> on fd 11 (<->/var/run/openvswitch/db.sock) at ../lib/stream-fd.c:157 >>> (97% CPU usage) 2018-06-11T11:13:49.926Z|00130|poll_loop|INFO|wakeup >>> due to [POLLIN] on fd 429 (127.0.0.1:33716<->127.0.0.1:6633) at >>> ../lib/stream-fd.c:157 (97% CPU usage) >>> 2018-06-11T11:13:49.930Z|00131|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T11:13:49.930Z|00132|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T11:13:49.930Z|00133|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T11:13:49.931Z|00134|bridge|INFO|ovs-vswitchd (Open >>> vSwitch) >>> 2.9.0 >>> 2018-06-11T11:13:53.662Z|00135|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T11:13:53.662Z|00136|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T11:13:53.662Z|00137|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T11:13:53.675Z|00138|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T11:13:53.675Z|00139|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T11:13:53.675Z|00140|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T11:13:53.682Z|00141|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T11:13:53.682Z|00142|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T11:13:53.682Z|00143|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T11:13:53.693Z|00144|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T11:13:53.693Z|00145|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T11:13:53.693Z|00146|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T11:13:53.705Z|00147|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T11:13:53.705Z|00148|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T11:13:53.705Z|00149|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T11:13:53.711Z|00150|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T11:13:53.711Z|00151|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T11:13:53.711Z|00152|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T11:13:53.717Z|00153|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T11:13:53.717Z|00154|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T11:13:53.717Z|00155|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T11:13:53.728Z|00156|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T11:13:53.728Z|00157|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T11:13:53.728Z|00158|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T11:13:53.743Z|00159|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T11:13:53.743Z|00160|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T11:13:53.743Z|00161|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T11:13:53.749Z|00162|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T11:13:53.749Z|00163|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T11:13:53.749Z|00164|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T11:13:53.756Z|00165|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T11:13:53.756Z|00166|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T11:13:53.756Z|00167|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T11:13:53.766Z|00168|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>> 2018-06-11T11:13:53.766Z|00169|netdev_dpdk|WARN|Error attaching >>> device 'mlx4_0' to DPDK >>> 2018-06-11T11:13:53.766Z|00170|netdev|WARN|enp59s0d1: could not set >>> configuration (Invalid argument) >>> 2018-06-11T11:13:55.270Z|00171|memory|INFO|1406100 kB peak resident >>> set size after 10.0 seconds >>> 2018-06-11T11:13:55.270Z|00172|memory|INFO|handlers:118 ofconns:3 >>> ports:8 revalidators:42 rules:36 udpif keys:1 >>> 2018-06-11T11:14:03.699Z|00173|connmgr|INFO|br-int<->tcp:127.0.0.1:6633: >>> 5 flow_mods 10 s ago (5 adds) >>> 2018-06-11T11:14:03.737Z|00174|connmgr|INFO|br-ex<->tcp:127.0.0.1:6633: >>> 2 flow_mods 10 s ago (2 adds) >>> 2018-06-11T11:14:03.773Z|00175|connmgr|INFO|br-tun<->tcp:127.0.0.1:6633: >>> 12 flow_mods 10 s ago (12 adds) >>> >>> Best regards >>> >>> On 06/11/2018 12:57 PM, Raslan Darawsheh wrote: >>>> What I really want to see is the init process for OVS-DPDK that dumps the DPDK initialization. >>>> I think you will find it at the beginning of the log that you just sent. >>>> >>>> Kindest regards, >>>> Raslan Darawsheh >>>> >>>> -----Original Message----- >>>> From: Tobias Urdin [mailto:tobias.urdin@crystone.com] >>>> Sent: Monday, June 11, 2018 1:00 PM >>>> To: Raslan Darawsheh <rasland@mellanox.com> >>>> Cc: users@dpdk.org >>>> Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch >>>> >>>> Hello Raslan, >>>> Thank you for your reply. >>>> >>>> For reference the br-mlx bridge was created like this: >>>> $ovs-vsctl add-br br-mlx -- set bridge br-mlx datapath_type=netdev >>>> >>>> $ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 >>>> type=dpdk options:dpdk-devargs=mlx4_0 >>>> ovs-vsctl: Error detected while setting up 'enp59s0d1': Error attaching device 'mlx4_0' to DPDK. See ovs-vswitchd log for details. >>>> ovs-vsctl: The default log directory is "/var/log/openvswitch". >>>> >>>> In the ovs-vswitchd.log >>>> 2018-06-11T09:56:20.613Z|00185|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T09:56:20.613Z|00186|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T09:56:20.613Z|00187|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> >>>> Here is the complete log after restarting openvswitch-switch service: >>>> 2018-06-11T09:57:49.380Z|00134|bridge|INFO|ovs-vswitchd (Open >>>> vSwitch) >>>> 2.9.0 >>>> 2018-06-11T09:57:52.812Z|00135|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T09:57:52.812Z|00136|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T09:57:52.812Z|00137|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T09:57:52.825Z|00138|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T09:57:52.825Z|00139|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T09:57:52.825Z|00140|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T09:57:52.832Z|00141|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T09:57:52.832Z|00142|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T09:57:52.832Z|00143|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T09:57:52.843Z|00144|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T09:57:52.843Z|00145|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T09:57:52.843Z|00146|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T09:57:52.856Z|00147|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T09:57:52.856Z|00148|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T09:57:52.856Z|00149|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T09:57:52.862Z|00150|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T09:57:52.862Z|00151|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T09:57:52.862Z|00152|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T09:57:52.870Z|00153|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T09:57:52.870Z|00154|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T09:57:52.870Z|00155|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T09:57:52.889Z|00156|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T09:57:52.889Z|00157|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T09:57:52.889Z|00158|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T09:57:52.903Z|00159|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T09:57:52.903Z|00160|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T09:57:52.903Z|00161|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T09:57:52.908Z|00162|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T09:57:52.908Z|00163|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T09:57:52.908Z|00164|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T09:57:52.914Z|00165|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T09:57:52.914Z|00166|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T09:57:52.914Z|00167|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T09:57:52.923Z|00168|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T09:57:52.923Z|00169|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T09:57:52.923Z|00170|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T09:57:54.808Z|00171|memory|INFO|1406100 kB peak resident >>>> set size after 10.1 seconds >>>> 2018-06-11T09:57:54.808Z|00172|memory|INFO|handlers:118 ofconns:3 >>>> ports:8 revalidators:42 rules:36 udpif keys:1 >>>> 2018-06-11T09:58:02.850Z|00173|connmgr|INFO|br-int<->tcp:127.0.0.1:6633: >>>> 5 flow_mods 10 s ago (5 adds) >>>> 2018-06-11T09:58:02.898Z|00174|connmgr|INFO|br-ex<->tcp:127.0.0.1:6633: >>>> 2 flow_mods 10 s ago (2 adds) >>>> 2018-06-11T09:58:02.930Z|00175|connmgr|INFO|br-tun<->tcp:127.0.0.1:6633: >>>> 12 flow_mods 10 s ago (12 adds) >>>> >>>> Was that what you mean by init log? >>>> Best regards >>>> >>>> On 06/11/2018 11:51 AM, Raslan Darawsheh wrote: >>>>> Hi Tobias, >>>>> >>>>> Can you try using the following ? >>>>> >>>>> ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 >>>>> type=dpdk options:dpdk-devargs=mlx4_0 >>>>> >>>>> if it didn't work can you send the OVS-DPDK init log ? >>>>> >>>>> Kindest regards, >>>>> Raslan Darawsheh >>>>> >>>>> -----Original Message----- >>>>> From: users [mailto:users-bounces@dpdk.org] On Behalf Of Tobias >>>>> Urdin >>>>> Sent: Monday, June 11, 2018 12:15 PM >>>>> To: users@dpdk.org >>>>> Subject: [dpdk-users] mellanox connect-x 3 pro dpdk with >>>>> openvswitch >>>>> >>>>> Hello, >>>>> >>>>> I'm stuck trying to test DPDK with a Mellanox Connect-X 3 Pro card >>>>> on Ubuntu 16.04 >>>>> >>>>> I've enabled iommu, fixed hugepages and has installed and are running the openvswitch-switch-dpdk alternative. >>>>> >>>>> $lspci | grep -i mellanox >>>>> 3b:00.0 Ethernet controller: Mellanox Technologies MT27520 Family >>>>> [ConnectX-3 Pro] >>>>> >>>>> $lsmod | grep -i mlx >>>>> mlx4_en 110592 0 >>>>> vxlan 49152 2 i40e,mlx4_en >>>>> mlx4_core 286720 1 mlx4_en >>>>> ptp 20480 3 igb,i40e,mlx4_en >>>>> >>>>> $cat /etc/modprobe.d/mlx4_core.conf options mlx4_core >>>>> log_num_mgm_entry_size=-1 >>>>> >>>>> >>>>> $ovs-vsctl add-port br-mlx dpdk0 -- set Interface dpdk0 type=dpdk >>>>> >>>>> Bridge br-mlx >>>>> Port "dpdk0" >>>>> Interface "dpdk0" >>>>> type: dpdk >>>>> error: "'dpdk0' is missing 'options:dpdk-devargs'. The old 'dpdk<port_id>' names are not supported" >>>>> >>>>> >>>>> $ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 >>>>> type=dpdk options:dpdk-devargs=0000:3b:00.0,1 >>>>> ovs-vsctl: Error detected while setting up 'enp59s0d1': Error attaching device '0000:3b:00.0,1' to DPDK. See ovs-vswitchd log for details. >>>>> ovs-vsctl: The default log directory is "/var/log/openvswitch". >>>>> >>>>> 2018-06-11T09:10:38.696Z|00177|netdev_dpdk|WARN|Error attaching >>>>> device '0000:3b:00.0,1' to DPDK >>>>> 2018-06-11T09:10:38.696Z|00178|netdev|WARN|enp59s0d1: could not >>>>> set configuration (Invalid argument) >>>>> >>>>> 2018-06-11T09:12:50.994Z|00181|dpdk|ERR|EAL: Driver cannot attach >>>>> the device (0000:3b:00.0) >>>>> 2018-06-11T09:12:50.994Z|00182|dpdk|ERR|EAL: No port found for >>>>> device >>>>> (0000:3b:00.0) >>>>> 2018-06-11T09:12:50.994Z|00183|netdev_dpdk|WARN|Error >>>>> attaching device '0000:3b:00.0,1' to DPDK >>>>> 2018-06-11T09:12:50.994Z|00184|netdev|WARN|enp59s0d1: could not >>>>> set configuration (Invalid argument) >>>>> >>>>> $ovs-vsctl add-port br-mlx dpdk0 -- set Interface dpdk0 type=dpdk options:dpdk-devargs="class=eth,mac=ec:0d:9a:b0:13:21" >>>>> ovs-vsctl: Error detected while setting up 'dpdk0': Error attaching device 'class=eth,mac=ec:0d:9a:b0:13:21' to DPDK. See ovs-vswitchd log for details. >>>>> ovs-vsctl: The default log directory is "/var/log/openvswitch". >>>>> >>>>> Port "dpdk0" >>>>> Interface "dpdk0" >>>>> type: dpdk >>>>> options: {dpdk-devargs="class=eth,mac=ec:0d:9a:b0:13:21"} >>>>> error: "Error attaching device 'class=eth,mac=ec:0d:9a:b0:13:21' to DPDK" >>>>> >>>>> >>>>> Anybody know what I might have missed? I have tried restarting the openvswitch services after adding the interfaces as well but no difference. >>>>> >>>>> The logs doesn't say anything interesting from what I can see :( >>>>> >>>>> Best regards >>>>> > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [dpdk-users] mellanox connect-x 3 pro dpdk with openvswitch 2018-06-11 12:07 ` Raslan Darawsheh @ 2018-06-11 12:14 ` Tobias Urdin 2018-06-12 7:13 ` Raslan Darawsheh 0 siblings, 1 reply; 14+ messages in thread From: Tobias Urdin @ 2018-06-11 12:14 UTC (permalink / raw) To: Raslan Darawsheh; +Cc: users I found this librte-pmd-mlx4 driver but it's for 18.02 and I have dpdk 17.11.1 installed. It contains files: ./lib/x86_64-linux-gnu/dpdk-18.02-drivers/librte_pmd_mlx4.so.18.02 ./lib/x86_64-linux-gnu/librte_pmd_mlx4.so.18.02 I assume I can compile librte-pmd-mlx4 manually for 17.11 instead and place it in that folder. It looks like this: $ls -l /usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/ | head -n 5 total 0 lrwxrwxrwx 1 root root 35 Mar 22 10:23 librte_mempool_octeontx.so.17.11 -> ../librte_mempool_octeontx.so.17.11 lrwxrwxrwx 1 root root 31 Mar 22 10:23 librte_mempool_ring.so.17.11 -> ../librte_mempool_ring.so.17.11 lrwxrwxrwx 1 root root 32 Mar 22 10:23 librte_mempool_stack.so.17.11 -> ../librte_mempool_stack.so.17.11 lrwxrwxrwx 1 root root 32 Mar 22 10:23 librte_pmd_af_packet.so.17.11 -> ../librte_pmd_af_packet.so.17.11 ... Would that be enough or is there anything else I need? Best regards On 06/11/2018 02:07 PM, Raslan Darawsheh wrote: > I think OVS-DPDK has static linking for dpdk and I'm not sure you can do it this way > > Kindest regards, > Raslan Darawsheh > > -----Original Message----- > From: Tobias Urdin [mailto:tobias.urdin@crystone.com] > Sent: Monday, June 11, 2018 3:05 PM > To: Raslan Darawsheh <rasland@mellanox.com> > Cc: users@dpdk.org > Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch > > Ok, I'm more wondering if there is any way I can build a librte-pmd-mlx4 package that supplies the compiled PMD or something like that instead of having to rebuild all the major parts? > > Thanks! > Best regards > > On 06/11/2018 01:51 PM, Raslan Darawsheh wrote: >> I think you can upgrade the current version that you have as seen in this article: >> https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdocs >> .openvswitch.org%2Fen%2Flatest%2Fintro%2Finstall%2Fgeneral%2F%23hot-up >> grading&data=02%7C01%7Crasland%40mellanox.com%7Cfdaa105976c744dba2f908 >> d5cf9394cc%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C1%7C63664315514160 >> 5703&sdata=VhpqI3M4RdnD5Shpbt6O8YmufIKiyoYK9kxO9TR1Yrk%3D&reserved=0 >> >> but, I haven't tried it before. >> >> Kindest regards, >> Raslan Darawsheh >> >> -----Original Message----- >> From: Tobias Urdin [mailto:tobias.urdin@crystone.com] >> Sent: Monday, June 11, 2018 2:45 PM >> To: Raslan Darawsheh <rasland@mellanox.com> >> Cc: users@dpdk.org >> Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch >> >> Is there any to skip compiling DPDK and OVS+OVS-DPDK and instead use the ones I have installed? >> Only compiling the Mellanox PMD module/binary or something similar to that? >> >> I can see that other drivers supply pmd packages like below. >> >> $dpkg -l | grep -E "openvswitch|pmd|dpdk" >> ii dpdk >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (runtime) >> ii librte-pmd-af-packet17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-af-packet runtime library) >> ii librte-pmd-ark17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte_pmd_ark runtime library) >> ii librte-pmd-avp17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte_pmd_avp runtime library) >> ii librte-pmd-bnxt17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-bnxt runtime library) >> ii librte-pmd-bond17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-bond runtime library) ii librte-pmd-crypto-scheduler17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte_pmd_crypto_scheduler runtime library) >> ii librte-pmd-cxgbe17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-cxgbe runtime library) >> ii librte-pmd-e1000-17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-e1000 runtime library) >> ii librte-pmd-ena17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-ena runtime library) >> ii librte-pmd-enic17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-enic runtime library) >> ii librte-pmd-failsafe17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-failsafe runtime library) >> ii librte-pmd-fm10k17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-fm10k runtime library) >> ii librte-pmd-i40e17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-i40e runtime library) >> ii librte-pmd-ixgbe17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-ixgbe runtime library) >> ii librte-pmd-kni17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte_pmd_kni runtime library) >> ii librte-pmd-lio17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte_pmd_lio runtime library) >> ii librte-pmd-nfp17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte_pmd_nfp runtime library) >> ii librte-pmd-null-crypto17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-null-crypto runtime library) >> ii librte-pmd-null17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-null runtime library) ii librte-pmd-octeontx-ssovf17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte_pmd_octeontx_ssovf runtime library) >> ii librte-pmd-octeontx17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte_pmd_octeontx runtime library) >> ii librte-pmd-pcap17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-pcap runtime library) >> ii librte-pmd-qede17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-qede runtime library) >> ii librte-pmd-ring17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-ring runtime library) >> ii librte-pmd-sfc-efx17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte_pmd_sfc_efx runtime library) ii librte-pmd-skeleton-event17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte_pmd_skeleton_event runtime library) >> ii librte-pmd-softnic17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-softnic runtime library) >> ii librte-pmd-sw-event17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte_pmd_sw_event runtime library) >> ii librte-pmd-tap17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte_pmd_tap runtime library) ii librte-pmd-thunderx-nicvf17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte_pmd_thunderx_nicvf runtime library) >> ii librte-pmd-vhost17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-vhost runtime library) >> ii librte-pmd-virtio17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-virtio runtime library) >> ii librte-pmd-vmxnet3-uio17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-vmxnet3-uio runtime library) >> ii openvswitch-common >> 2.9.0-0ubuntu1~cloud0 amd64 Open vSwitch >> common components >> ii openvswitch-switch >> 2.9.0-0ubuntu1~cloud0 amd64 Open vSwitch >> switch implementations >> ii openvswitch-switch-dpdk >> 2.9.0-0ubuntu1~cloud0 amd64 DPDK enabled >> Open vSwitch switch implementation >> ii python-openvswitch >> 2.9.0-0ubuntu1~cloud0 all Python bindings >> for Open vSwitch >> >> Thanks for all the help! >> Best regards >> >> >> On 06/11/2018 01:37 PM, Raslan Darawsheh wrote: >>> You can refer to this community page for compiling OVS-DPDK with MLX pmds enabled: >>> https://community.mellanox.com/docs/DOC-2637 >>> >>> >>> Kindest regards, >>> Raslan Darawsheh >>> >>> -----Original Message----- >>> From: Tobias Urdin [mailto:tobias.urdin@crystone.com] >>> Sent: Monday, June 11, 2018 2:35 PM >>> To: Raslan Darawsheh <rasland@mellanox.com> >>> Cc: users@dpdk.org >>> Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch >>> >>> I haven't compiled any DPDK or Mellanox related modules/packages. >>> What I have is ovs, ovs-dpdk, mlx4 module, I have also loaded the ib_uverbs driver. >>> >>> Is it the librte_pmd_mlx4 I have to compile manually? >>> >>> Do you know any updated documentation about this, there isn't much relevant and searching for documentation and the correct mlnx driver and code is a struggle. >>> >>> Best regards >>> >>> On 06/11/2018 01:20 PM, Raslan Darawsheh wrote: >>>> From what I'm seeing it looks like it didn't prop the MLX4 device at all. >>>> Can you make sure you have it enabled in your DPDK package when you compiled DPDK ? >>>> >>>> Kindest regards, >>>> Raslan Darawsheh >>>> >>>> -----Original Message----- >>>> From: Tobias Urdin [mailto:tobias.urdin@crystone.com] >>>> Sent: Monday, June 11, 2018 2:16 PM >>>> To: Raslan Darawsheh <rasland@mellanox.com> >>>> Cc: users@dpdk.org >>>> Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch >>>> >>>> Ok, here is the complete log when restarting the openvswitch-switch service, see below. >>>> >>>> Also here is the kernel for information: >>>> $uname -r >>>> 4.4.0-127-generic >>>> >>>> 2018-06-11T11:13:42.729Z|00176|bridge|INFO|bridge br-mlx: deleted >>>> interface br-mlx on port 65534 >>>> 2018-06-11T11:13:43.274Z|00177|bridge|INFO|bridge br-int: deleted >>>> interface br-int on port 65534 >>>> 2018-06-11T11:13:43.275Z|00178|bridge|INFO|bridge br-int: deleted >>>> interface int-br-ex on port 1 >>>> 2018-06-11T11:13:43.275Z|00179|bridge|INFO|bridge br-int: deleted >>>> interface patch-tun on port 2 >>>> 2018-06-11T11:13:43.762Z|00180|bridge|INFO|bridge br-tun: deleted >>>> interface patch-int on port 1 >>>> 2018-06-11T11:13:43.762Z|00181|bridge|INFO|bridge br-tun: deleted >>>> interface br-tun on port 65534 >>>> 2018-06-11T11:13:44.231Z|00182|bridge|INFO|bridge br-ex: deleted >>>> interface br-ex on port 65534 >>>> 2018-06-11T11:13:44.232Z|00183|bridge|INFO|bridge br-ex: deleted >>>> interface phy-br-ex on port 1 >>>> 2018-06-11T11:13:45.245Z|00001|vlog|INFO|opened log file >>>> /var/log/openvswitch/ovs-vswitchd.log >>>> 2018-06-11T11:13:45.252Z|00002|ovs_numa|INFO|Discovered 40 CPU cores >>>> on NUMA node 0 >>>> 2018-06-11T11:13:45.252Z|00003|ovs_numa|INFO|Discovered >>>> 40 CPU cores on NUMA node 1 >>>> 2018-06-11T11:13:45.252Z|00004|ovs_numa|INFO|Discovered 2 NUMA nodes >>>> and >>>> 80 CPU cores >>>> 2018-06-11T11:13:45.252Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: >>>> connecting... >>>> 2018-06-11T11:13:45.252Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: >>>> connected >>>> 2018-06-11T11:13:45.256Z|00007|dpdk|INFO|Using DPDK 17.11.0 2018-06-11T11:13:45.256Z|00008|dpdk|INFO|DPDK Enabled - initializing... >>>> 2018-06-11T11:13:45.256Z|00009|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch 2018-06-11T11:13:45.256Z|00010|dpdk|INFO|IOMMU support for vhost-user-client disabled. >>>> 2018-06-11T11:13:45.256Z|00011|dpdk|INFO|EAL ARGS: ovs-vswitchd >>>> --socket-mem 1024,0 -c 0x00000001 >>>> 2018-06-11T11:13:45.258Z|00012|dpdk|INFO|EAL: Detected 80 lcore(s) >>>> 2018-06-11T11:13:45.285Z|00013|dpdk|INFO|EAL: Probing VFIO support... >>>> 2018-06-11T11:13:45.285Z|00014|dpdk|INFO|EAL: VFIO support >>>> initialized >>>> 2018-06-11T11:13:48.214Z|00015|dpdk|INFO|EAL: PCI device 0000:01:00.0 on NUMA socket 0 >>>> 2018-06-11T11:13:48.214Z|00016|dpdk|INFO|EAL: probe driver: 8086:1521 >>>> net_e1000_igb >>>> 2018-06-11T11:13:48.214Z|00017|dpdk|INFO|EAL: PCI device 0000:01:00.1 on NUMA socket 0 >>>> 2018-06-11T11:13:48.214Z|00018|dpdk|INFO|EAL: probe driver: 8086:1521 >>>> net_e1000_igb >>>> 2018-06-11T11:13:48.214Z|00019|dpdk|INFO|EAL: PCI device 0000:18:00.0 on NUMA socket 0 >>>> 2018-06-11T11:13:48.214Z|00020|dpdk|INFO|EAL: probe driver: 8086:1572 >>>> net_i40e >>>> 2018-06-11T11:13:48.214Z|00021|dpdk|INFO|EAL: PCI device 0000:18:00.1 on NUMA socket 0 >>>> 2018-06-11T11:13:48.214Z|00022|dpdk|INFO|EAL: probe driver: 8086:1572 >>>> net_i40e >>>> 2018-06-11T11:13:48.214Z|00023|dpdk|INFO|DPDK pdump packet capture >>>> enabled 2018-06-11T11:13:48.224Z|00024|dpdk|INFO|DPDK Enabled - >>>> initialized >>>> 2018-06-11T11:13:48.235Z|00025|ofproto_dpif|INFO|netdev@ovs-netdev: >>>> Datapath supports recirculation >>>> 2018-06-11T11:13:48.235Z|00026|ofproto_dpif|INFO|netdev@ovs-netdev: >>>> VLAN header stack length probed as 1 >>>> 2018-06-11T11:13:48.235Z|00027|ofproto_dpif|INFO|netdev@ovs-netdev: >>>> MPLS label stack length probed as 3 >>>> 2018-06-11T11:13:48.235Z|00028|ofproto_dpif|INFO|netdev@ovs-netdev: >>>> Datapath supports truncate action >>>> 2018-06-11T11:13:48.235Z|00029|ofproto_dpif|INFO|netdev@ovs-netdev: >>>> Datapath supports unique flow ids >>>> 2018-06-11T11:13:48.235Z|00030|ofproto_dpif|INFO|netdev@ovs-netdev: >>>> Datapath supports clone action >>>> 2018-06-11T11:13:48.235Z|00031|ofproto_dpif|INFO|netdev@ovs-netdev: >>>> Max sample nesting level probed as 10 >>>> 2018-06-11T11:13:48.235Z|00032|ofproto_dpif|INFO|netdev@ovs-netdev: >>>> Datapath supports eventmask in conntrack action >>>> 2018-06-11T11:13:48.235Z|00033|ofproto_dpif|INFO|netdev@ovs-netdev: >>>> Datapath supports ct_clear action >>>> 2018-06-11T11:13:48.235Z|00034|ofproto_dpif|INFO|netdev@ovs-netdev: >>>> Datapath supports ct_state >>>> 2018-06-11T11:13:48.235Z|00035|ofproto_dpif|INFO|netdev@ovs-netdev: >>>> Datapath supports ct_zone >>>> 2018-06-11T11:13:48.235Z|00036|ofproto_dpif|INFO|netdev@ovs-netdev: >>>> Datapath supports ct_mark >>>> 2018-06-11T11:13:48.235Z|00037|ofproto_dpif|INFO|netdev@ovs-netdev: >>>> Datapath supports ct_label >>>> 2018-06-11T11:13:48.235Z|00038|ofproto_dpif|INFO|netdev@ovs-netdev: >>>> Datapath supports ct_state_nat >>>> 2018-06-11T11:13:48.235Z|00039|ofproto_dpif|INFO|netdev@ovs-netdev: >>>> Datapath supports ct_orig_tuple >>>> 2018-06-11T11:13:48.235Z|00040|ofproto_dpif|INFO|netdev@ovs-netdev: >>>> Datapath supports ct_orig_tuple6 >>>> 2018-06-11T11:13:48.605Z|00041|ofproto_dpif|INFO|system@ovs-system: >>>> Datapath supports recirculation >>>> 2018-06-11T11:13:48.605Z|00042|ofproto_dpif|INFO|system@ovs-system: >>>> VLAN header stack length probed as 1 >>>> 2018-06-11T11:13:48.605Z|00043|ofproto_dpif|INFO|system@ovs-system: >>>> MPLS label stack length probed as 1 >>>> 2018-06-11T11:13:48.605Z|00044|ofproto_dpif|INFO|system@ovs-system: >>>> Datapath does not support truncate action >>>> 2018-06-11T11:13:48.605Z|00045|ofproto_dpif|INFO|system@ovs-system: >>>> Datapath supports unique flow ids >>>> 2018-06-11T11:13:48.605Z|00046|ofproto_dpif|INFO|system@ovs-system: >>>> Datapath does not support clone action >>>> 2018-06-11T11:13:48.605Z|00047|ofproto_dpif|INFO|system@ovs-system: >>>> Max sample nesting level probed as 3 >>>> 2018-06-11T11:13:48.605Z|00048|ofproto_dpif|INFO|system@ovs-system: >>>> Datapath does not support eventmask in conntrack action >>>> 2018-06-11T11:13:48.605Z|00049|ofproto_dpif|INFO|system@ovs-system: >>>> Datapath does not support ct_clear action >>>> 2018-06-11T11:13:48.605Z|00050|ofproto_dpif|INFO|system@ovs-system: >>>> Datapath supports ct_state >>>> 2018-06-11T11:13:48.605Z|00051|ofproto_dpif|INFO|system@ovs-system: >>>> Datapath supports ct_zone >>>> 2018-06-11T11:13:48.605Z|00052|ofproto_dpif|INFO|system@ovs-system: >>>> Datapath supports ct_mark >>>> 2018-06-11T11:13:48.605Z|00053|ofproto_dpif|INFO|system@ovs-system: >>>> Datapath supports ct_label >>>> 2018-06-11T11:13:48.605Z|00054|ofproto_dpif|INFO|system@ovs-system: >>>> Datapath does not support ct_state_nat >>>> 2018-06-11T11:13:48.605Z|00055|ofproto_dpif|INFO|system@ovs-system: >>>> Datapath does not support ct_orig_tuple >>>> 2018-06-11T11:13:48.605Z|00056|ofproto_dpif|INFO|system@ovs-system: >>>> Datapath does not support ct_orig_tuple6 >>>> 2018-06-11T11:13:48.981Z|00057|bridge|INFO|bridge br-mlx: added >>>> interface br-mlx on port 65534 >>>> 2018-06-11T11:13:48.981Z|00058|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T11:13:48.981Z|00059|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T11:13:48.981Z|00060|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T11:13:48.982Z|00061|bridge|INFO|bridge br-int: added >>>> interface br-int on port 65534 >>>> 2018-06-11T11:13:48.982Z|00062|bridge|INFO|bridge br-int: added >>>> interface int-br-ex on port 1 >>>> 2018-06-11T11:13:48.982Z|00063|bridge|INFO|bridge br-int: added >>>> interface patch-tun on port 2 >>>> 2018-06-11T11:13:48.982Z|00064|bridge|INFO|bridge br-tun: added >>>> interface patch-int on port 1 >>>> 2018-06-11T11:13:48.982Z|00065|bridge|INFO|bridge br-tun: added >>>> interface br-tun on port 65534 >>>> 2018-06-11T11:13:48.982Z|00066|bridge|INFO|bridge br-ex: added >>>> interface br-ex on port 65534 >>>> 2018-06-11T11:13:48.982Z|00067|bridge|INFO|bridge br-ex: added >>>> interface phy-br-ex on port 1 >>>> 2018-06-11T11:13:48.982Z|00068|bridge|INFO|bridge br-mlx: using >>>> datapath ID 000072528484a642 >>>> 2018-06-11T11:13:48.982Z|00069|connmgr|INFO|br-mlx: added service controller "punix:/var/run/openvswitch/br-mlx.mgmt" >>>> 2018-06-11T11:13:49.221Z|00070|bridge|INFO|bridge br-int: using >>>> datapath ID 00006680927b6b40 >>>> 2018-06-11T11:13:49.221Z|00071|connmgr|INFO|br-int: added service controller "punix:/var/run/openvswitch/br-int.mgmt" >>>> 2018-06-11T11:13:49.221Z|00072|connmgr|INFO|br-int: added primary controller "tcp:127.0.0.1:6633" >>>> 2018-06-11T11:13:49.221Z|00073|rconn|INFO|br-int<->tcp:127.0.0.1:6633: >>>> connecting... >>>> 2018-06-11T11:13:49.460Z|00074|bridge|INFO|bridge br-tun: using >>>> datapath ID 000066532e5f0443 >>>> 2018-06-11T11:13:49.461Z|00075|connmgr|INFO|br-tun: added service controller "punix:/var/run/openvswitch/br-tun.mgmt" >>>> 2018-06-11T11:13:49.461Z|00076|connmgr|INFO|br-tun: added primary controller "tcp:127.0.0.1:6633" >>>> 2018-06-11T11:13:49.461Z|00077|rconn|INFO|br-tun<->tcp:127.0.0.1:6633: >>>> connecting... >>>> 2018-06-11T11:13:49.692Z|00078|bridge|INFO|bridge br-ex: using >>>> datapath ID 0000e2d1bdbe8e45 >>>> 2018-06-11T11:13:49.692Z|00079|connmgr|INFO|br-ex: added service controller "punix:/var/run/openvswitch/br-ex.mgmt" >>>> 2018-06-11T11:13:49.692Z|00080|connmgr|INFO|br-ex: added primary controller "tcp:127.0.0.1:6633" >>>> 2018-06-11T11:13:49.692Z|00081|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: >>>> connecting... >>>> 2018-06-11T11:13:49.916Z|00082|timeval|WARN|Unreasonably long 4661ms >>>> poll interval (16ms user, 4472ms system) >>>> 2018-06-11T11:13:49.916Z|00083|timeval|WARN|faults: 3584706 minor, 0 >>>> major >>>> 2018-06-11T11:13:49.916Z|00084|timeval|WARN|disk: 0 reads, 16 writes 2018-06-11T11:13:49.916Z|00085|timeval|WARN|context switches: 147 voluntary, 25 involuntary 2018-06-11T11:13:49.916Z|00086|coverage|INFO|Event coverage, avg rate over last: 5 seconds, last minute, last hour, hash=2ba35161: >>>> 2018-06-11T11:13:49.916Z|00087|coverage|INFO|bridge_reconfigure >>>> 0.0/sec 0.000/sec 0.0000/sec total: 1 >>>> 2018-06-11T11:13:49.916Z|00088|coverage|INFO|ofproto_flush >>>> 0.0/sec 0.000/sec 0.0000/sec total: 4 >>>> 2018-06-11T11:13:49.916Z|00089|coverage|INFO|ofproto_update_port >>>> 0.0/sec 0.000/sec 0.0000/sec total: 8 >>>> 2018-06-11T11:13:49.916Z|00090|coverage|INFO|rev_flow_table >>>> 0.0/sec 0.000/sec 0.0000/sec total: 1 >>>> 2018-06-11T11:13:49.916Z|00091|coverage|INFO|cmap_expand >>>> 0.0/sec 0.000/sec 0.0000/sec total: 52 >>>> 2018-06-11T11:13:49.916Z|00092|coverage|INFO|cmap_shrink >>>> 0.0/sec 0.000/sec 0.0000/sec total: 23 >>>> 2018-06-11T11:13:49.916Z|00093|coverage|INFO|dpif_port_add >>>> 0.0/sec 0.000/sec 0.0000/sec total: 1 >>>> 2018-06-11T11:13:49.916Z|00094|coverage|INFO|dpif_flow_flush >>>> 0.0/sec 0.000/sec 0.0000/sec total: 6 >>>> 2018-06-11T11:13:49.916Z|00095|coverage|INFO|dpif_flow_get >>>> 0.0/sec 0.000/sec 0.0000/sec total: 29 >>>> 2018-06-11T11:13:49.916Z|00096|coverage|INFO|dpif_flow_put >>>> 0.0/sec 0.000/sec 0.0000/sec total: 37 >>>> 2018-06-11T11:13:49.916Z|00097|coverage|INFO|dpif_flow_del >>>> 0.0/sec 0.000/sec 0.0000/sec total: 29 >>>> 2018-06-11T11:13:49.916Z|00098|coverage|INFO|dpif_execute >>>> 0.0/sec 0.000/sec 0.0000/sec total: 8 >>>> 2018-06-11T11:13:49.916Z|00099|coverage|INFO|flow_extract >>>> 0.0/sec 0.000/sec 0.0000/sec total: 6 >>>> 2018-06-11T11:13:49.916Z|00100|coverage|INFO|miniflow_malloc >>>> 0.0/sec 0.000/sec 0.0000/sec total: 34 >>>> 2018-06-11T11:13:49.916Z|00101|coverage|INFO|hmap_expand >>>> 0.0/sec 0.000/sec 0.0000/sec total: 498 >>>> 2018-06-11T11:13:49.916Z|00102|coverage|INFO|netdev_received >>>> 0.0/sec 0.000/sec 0.0000/sec total: 1 >>>> 2018-06-11T11:13:49.916Z|00103|coverage|INFO|netdev_get_stats >>>> 0.0/sec 0.000/sec 0.0000/sec total: 8 >>>> 2018-06-11T11:13:49.916Z|00104|coverage|INFO|poll_create_node >>>> 0.0/sec 0.000/sec 0.0000/sec total: 23 >>>> 2018-06-11T11:13:49.916Z|00105|coverage|INFO|seq_change >>>> 0.0/sec 0.000/sec 0.0000/sec total: 2193 >>>> 2018-06-11T11:13:49.916Z|00106|coverage|INFO|pstream_open >>>> 0.0/sec 0.000/sec 0.0000/sec total: 9 >>>> 2018-06-11T11:13:49.916Z|00107|coverage|INFO|stream_open >>>> 0.0/sec 0.000/sec 0.0000/sec total: 4 >>>> 2018-06-11T11:13:49.916Z|00108|coverage|INFO|util_xalloc >>>> 0.0/sec 0.000/sec 0.0000/sec total: 12803 >>>> 2018-06-11T11:13:49.916Z|00109|coverage|INFO|vconn_open >>>> 0.0/sec 0.000/sec 0.0000/sec total: 3 >>>> 2018-06-11T11:13:49.916Z|00110|coverage|INFO|netdev_set_policing >>>> 0.0/sec 0.000/sec 0.0000/sec total: 4 >>>> 2018-06-11T11:13:49.916Z|00111|coverage|INFO|netdev_get_ifindex >>>> 0.0/sec 0.000/sec 0.0000/sec total: 4 >>>> 2018-06-11T11:13:49.916Z|00112|coverage|INFO|netdev_get_hwaddr >>>> 0.0/sec 0.000/sec 0.0000/sec total: 9 >>>> 2018-06-11T11:13:49.916Z|00113|coverage|INFO|netdev_get_ethtool >>>> 0.0/sec 0.000/sec 0.0000/sec total: 2 >>>> 2018-06-11T11:13:49.916Z|00114|coverage|INFO|netlink_received >>>> 0.0/sec 0.000/sec 0.0000/sec total: 85 >>>> 2018-06-11T11:13:49.916Z|00115|coverage|INFO|netlink_recv_jumbo >>>> 0.0/sec 0.000/sec 0.0000/sec total: 1 >>>> 2018-06-11T11:13:49.916Z|00116|coverage|INFO|netlink_sent >>>> 0.0/sec 0.000/sec 0.0000/sec total: 102 >>>> 2018-06-11T11:13:49.916Z|00117|coverage|INFO|74 events never hit >>>> 2018-06-11T11:13:49.916Z|00118|poll_loop|INFO|wakeup due to [POLLIN] >>>> on fd 433 (FIFO pipe:[1025715]) at ../lib/ovs-rcu.c:229 (97% CPU >>>> usage) 2018-06-11T11:13:49.917Z|00119|poll_loop|INFO|wakeup due to >>>> [POLLIN] on fd 433 (FIFO pipe:[1025715]) at ../lib/ovs-rcu.c:229 >>>> (97% CPU usage) 2018-06-11T11:13:49.917Z|00120|poll_loop|INFO|wakeup >>>> due to 0-ms timeout at ../lib/ovs-rcu.c:229 (97% CPU usage) >>>> 2018-06-11T11:13:49.917Z|00121|poll_loop|INFO|wakeup due to 0-ms >>>> timeout at ../lib/ovs-rcu.c:229 (97% CPU usage) >>>> 2018-06-11T11:13:49.920Z|00122|rconn|INFO|br-int<->tcp:127.0.0.1:6633: >>>> connected >>>> 2018-06-11T11:13:49.920Z|00123|rconn|INFO|br-tun<->tcp:127.0.0.1:6633: >>>> connected >>>> 2018-06-11T11:13:49.920Z|00124|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: >>>> connected >>>> 2018-06-11T11:13:49.926Z|00125|poll_loop|INFO|wakeup due to [POLLIN] >>>> on fd 10 (NETLINK_ROUTE<->NETLINK_ROUTE) at >>>> ../lib/netlink-socket.c:1331 (97% CPU usage) >>>> 2018-06-11T11:13:49.926Z|00126|poll_loop|INFO|wakeup due to [POLLIN] >>>> on fd 421 (character device /dev/net/tun) at >>>> ../lib/netdev-linux.c:1171 (97% CPU usage) >>>> 2018-06-11T11:13:49.926Z|00127|poll_loop|INFO|wakeup due to [POLLIN] >>>> on fd 423 (127.0.0.1:33712<->127.0.0.1:6633) at >>>> ../lib/stream-fd.c:157 (97% CPU usage) >>>> 2018-06-11T11:13:49.926Z|00128|poll_loop|INFO|wakeup >>>> due to [POLLIN] on fd 428 (127.0.0.1:33714<->127.0.0.1:6633) at >>>> ../lib/stream-fd.c:157 (97% CPU usage) >>>> 2018-06-11T11:13:49.926Z|00129|poll_loop|INFO|wakeup due to [POLLIN] >>>> on fd 11 (<->/var/run/openvswitch/db.sock) at ../lib/stream-fd.c:157 >>>> (97% CPU usage) 2018-06-11T11:13:49.926Z|00130|poll_loop|INFO|wakeup >>>> due to [POLLIN] on fd 429 (127.0.0.1:33716<->127.0.0.1:6633) at >>>> ../lib/stream-fd.c:157 (97% CPU usage) >>>> 2018-06-11T11:13:49.930Z|00131|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T11:13:49.930Z|00132|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T11:13:49.930Z|00133|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T11:13:49.931Z|00134|bridge|INFO|ovs-vswitchd (Open >>>> vSwitch) >>>> 2.9.0 >>>> 2018-06-11T11:13:53.662Z|00135|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T11:13:53.662Z|00136|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T11:13:53.662Z|00137|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T11:13:53.675Z|00138|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T11:13:53.675Z|00139|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T11:13:53.675Z|00140|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T11:13:53.682Z|00141|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T11:13:53.682Z|00142|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T11:13:53.682Z|00143|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T11:13:53.693Z|00144|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T11:13:53.693Z|00145|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T11:13:53.693Z|00146|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T11:13:53.705Z|00147|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T11:13:53.705Z|00148|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T11:13:53.705Z|00149|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T11:13:53.711Z|00150|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T11:13:53.711Z|00151|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T11:13:53.711Z|00152|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T11:13:53.717Z|00153|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T11:13:53.717Z|00154|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T11:13:53.717Z|00155|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T11:13:53.728Z|00156|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T11:13:53.728Z|00157|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T11:13:53.728Z|00158|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T11:13:53.743Z|00159|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T11:13:53.743Z|00160|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T11:13:53.743Z|00161|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T11:13:53.749Z|00162|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T11:13:53.749Z|00163|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T11:13:53.749Z|00164|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T11:13:53.756Z|00165|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T11:13:53.756Z|00166|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T11:13:53.756Z|00167|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T11:13:53.766Z|00168|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T11:13:53.766Z|00169|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T11:13:53.766Z|00170|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T11:13:55.270Z|00171|memory|INFO|1406100 kB peak resident >>>> set size after 10.0 seconds >>>> 2018-06-11T11:13:55.270Z|00172|memory|INFO|handlers:118 ofconns:3 >>>> ports:8 revalidators:42 rules:36 udpif keys:1 >>>> 2018-06-11T11:14:03.699Z|00173|connmgr|INFO|br-int<->tcp:127.0.0.1:6633: >>>> 5 flow_mods 10 s ago (5 adds) >>>> 2018-06-11T11:14:03.737Z|00174|connmgr|INFO|br-ex<->tcp:127.0.0.1:6633: >>>> 2 flow_mods 10 s ago (2 adds) >>>> 2018-06-11T11:14:03.773Z|00175|connmgr|INFO|br-tun<->tcp:127.0.0.1:6633: >>>> 12 flow_mods 10 s ago (12 adds) >>>> >>>> Best regards >>>> >>>> On 06/11/2018 12:57 PM, Raslan Darawsheh wrote: >>>>> What I really want to see is the init process for OVS-DPDK that dumps the DPDK initialization. >>>>> I think you will find it at the beginning of the log that you just sent. >>>>> >>>>> Kindest regards, >>>>> Raslan Darawsheh >>>>> >>>>> -----Original Message----- >>>>> From: Tobias Urdin [mailto:tobias.urdin@crystone.com] >>>>> Sent: Monday, June 11, 2018 1:00 PM >>>>> To: Raslan Darawsheh <rasland@mellanox.com> >>>>> Cc: users@dpdk.org >>>>> Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch >>>>> >>>>> Hello Raslan, >>>>> Thank you for your reply. >>>>> >>>>> For reference the br-mlx bridge was created like this: >>>>> $ovs-vsctl add-br br-mlx -- set bridge br-mlx datapath_type=netdev >>>>> >>>>> $ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 >>>>> type=dpdk options:dpdk-devargs=mlx4_0 >>>>> ovs-vsctl: Error detected while setting up 'enp59s0d1': Error attaching device 'mlx4_0' to DPDK. See ovs-vswitchd log for details. >>>>> ovs-vsctl: The default log directory is "/var/log/openvswitch". >>>>> >>>>> In the ovs-vswitchd.log >>>>> 2018-06-11T09:56:20.613Z|00185|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>>> 2018-06-11T09:56:20.613Z|00186|netdev_dpdk|WARN|Error attaching >>>>> device 'mlx4_0' to DPDK >>>>> 2018-06-11T09:56:20.613Z|00187|netdev|WARN|enp59s0d1: could not set >>>>> configuration (Invalid argument) >>>>> >>>>> Here is the complete log after restarting openvswitch-switch service: >>>>> 2018-06-11T09:57:49.380Z|00134|bridge|INFO|ovs-vswitchd (Open >>>>> vSwitch) >>>>> 2.9.0 >>>>> 2018-06-11T09:57:52.812Z|00135|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>>> 2018-06-11T09:57:52.812Z|00136|netdev_dpdk|WARN|Error attaching >>>>> device 'mlx4_0' to DPDK >>>>> 2018-06-11T09:57:52.812Z|00137|netdev|WARN|enp59s0d1: could not set >>>>> configuration (Invalid argument) >>>>> 2018-06-11T09:57:52.825Z|00138|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>>> 2018-06-11T09:57:52.825Z|00139|netdev_dpdk|WARN|Error attaching >>>>> device 'mlx4_0' to DPDK >>>>> 2018-06-11T09:57:52.825Z|00140|netdev|WARN|enp59s0d1: could not set >>>>> configuration (Invalid argument) >>>>> 2018-06-11T09:57:52.832Z|00141|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>>> 2018-06-11T09:57:52.832Z|00142|netdev_dpdk|WARN|Error attaching >>>>> device 'mlx4_0' to DPDK >>>>> 2018-06-11T09:57:52.832Z|00143|netdev|WARN|enp59s0d1: could not set >>>>> configuration (Invalid argument) >>>>> 2018-06-11T09:57:52.843Z|00144|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>>> 2018-06-11T09:57:52.843Z|00145|netdev_dpdk|WARN|Error attaching >>>>> device 'mlx4_0' to DPDK >>>>> 2018-06-11T09:57:52.843Z|00146|netdev|WARN|enp59s0d1: could not set >>>>> configuration (Invalid argument) >>>>> 2018-06-11T09:57:52.856Z|00147|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>>> 2018-06-11T09:57:52.856Z|00148|netdev_dpdk|WARN|Error attaching >>>>> device 'mlx4_0' to DPDK >>>>> 2018-06-11T09:57:52.856Z|00149|netdev|WARN|enp59s0d1: could not set >>>>> configuration (Invalid argument) >>>>> 2018-06-11T09:57:52.862Z|00150|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>>> 2018-06-11T09:57:52.862Z|00151|netdev_dpdk|WARN|Error attaching >>>>> device 'mlx4_0' to DPDK >>>>> 2018-06-11T09:57:52.862Z|00152|netdev|WARN|enp59s0d1: could not set >>>>> configuration (Invalid argument) >>>>> 2018-06-11T09:57:52.870Z|00153|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>>> 2018-06-11T09:57:52.870Z|00154|netdev_dpdk|WARN|Error attaching >>>>> device 'mlx4_0' to DPDK >>>>> 2018-06-11T09:57:52.870Z|00155|netdev|WARN|enp59s0d1: could not set >>>>> configuration (Invalid argument) >>>>> 2018-06-11T09:57:52.889Z|00156|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>>> 2018-06-11T09:57:52.889Z|00157|netdev_dpdk|WARN|Error attaching >>>>> device 'mlx4_0' to DPDK >>>>> 2018-06-11T09:57:52.889Z|00158|netdev|WARN|enp59s0d1: could not set >>>>> configuration (Invalid argument) >>>>> 2018-06-11T09:57:52.903Z|00159|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>>> 2018-06-11T09:57:52.903Z|00160|netdev_dpdk|WARN|Error attaching >>>>> device 'mlx4_0' to DPDK >>>>> 2018-06-11T09:57:52.903Z|00161|netdev|WARN|enp59s0d1: could not set >>>>> configuration (Invalid argument) >>>>> 2018-06-11T09:57:52.908Z|00162|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>>> 2018-06-11T09:57:52.908Z|00163|netdev_dpdk|WARN|Error attaching >>>>> device 'mlx4_0' to DPDK >>>>> 2018-06-11T09:57:52.908Z|00164|netdev|WARN|enp59s0d1: could not set >>>>> configuration (Invalid argument) >>>>> 2018-06-11T09:57:52.914Z|00165|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>>> 2018-06-11T09:57:52.914Z|00166|netdev_dpdk|WARN|Error attaching >>>>> device 'mlx4_0' to DPDK >>>>> 2018-06-11T09:57:52.914Z|00167|netdev|WARN|enp59s0d1: could not set >>>>> configuration (Invalid argument) >>>>> 2018-06-11T09:57:52.923Z|00168|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>>> 2018-06-11T09:57:52.923Z|00169|netdev_dpdk|WARN|Error attaching >>>>> device 'mlx4_0' to DPDK >>>>> 2018-06-11T09:57:52.923Z|00170|netdev|WARN|enp59s0d1: could not set >>>>> configuration (Invalid argument) >>>>> 2018-06-11T09:57:54.808Z|00171|memory|INFO|1406100 kB peak resident >>>>> set size after 10.1 seconds >>>>> 2018-06-11T09:57:54.808Z|00172|memory|INFO|handlers:118 ofconns:3 >>>>> ports:8 revalidators:42 rules:36 udpif keys:1 >>>>> 2018-06-11T09:58:02.850Z|00173|connmgr|INFO|br-int<->tcp:127.0.0.1:6633: >>>>> 5 flow_mods 10 s ago (5 adds) >>>>> 2018-06-11T09:58:02.898Z|00174|connmgr|INFO|br-ex<->tcp:127.0.0.1:6633: >>>>> 2 flow_mods 10 s ago (2 adds) >>>>> 2018-06-11T09:58:02.930Z|00175|connmgr|INFO|br-tun<->tcp:127.0.0.1:6633: >>>>> 12 flow_mods 10 s ago (12 adds) >>>>> >>>>> Was that what you mean by init log? >>>>> Best regards >>>>> >>>>> On 06/11/2018 11:51 AM, Raslan Darawsheh wrote: >>>>>> Hi Tobias, >>>>>> >>>>>> Can you try using the following ? >>>>>> >>>>>> ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 >>>>>> type=dpdk options:dpdk-devargs=mlx4_0 >>>>>> >>>>>> if it didn't work can you send the OVS-DPDK init log ? >>>>>> >>>>>> Kindest regards, >>>>>> Raslan Darawsheh >>>>>> >>>>>> -----Original Message----- >>>>>> From: users [mailto:users-bounces@dpdk.org] On Behalf Of Tobias >>>>>> Urdin >>>>>> Sent: Monday, June 11, 2018 12:15 PM >>>>>> To: users@dpdk.org >>>>>> Subject: [dpdk-users] mellanox connect-x 3 pro dpdk with >>>>>> openvswitch >>>>>> >>>>>> Hello, >>>>>> >>>>>> I'm stuck trying to test DPDK with a Mellanox Connect-X 3 Pro card >>>>>> on Ubuntu 16.04 >>>>>> >>>>>> I've enabled iommu, fixed hugepages and has installed and are running the openvswitch-switch-dpdk alternative. >>>>>> >>>>>> $lspci | grep -i mellanox >>>>>> 3b:00.0 Ethernet controller: Mellanox Technologies MT27520 Family >>>>>> [ConnectX-3 Pro] >>>>>> >>>>>> $lsmod | grep -i mlx >>>>>> mlx4_en 110592 0 >>>>>> vxlan 49152 2 i40e,mlx4_en >>>>>> mlx4_core 286720 1 mlx4_en >>>>>> ptp 20480 3 igb,i40e,mlx4_en >>>>>> >>>>>> $cat /etc/modprobe.d/mlx4_core.conf options mlx4_core >>>>>> log_num_mgm_entry_size=-1 >>>>>> >>>>>> >>>>>> $ovs-vsctl add-port br-mlx dpdk0 -- set Interface dpdk0 type=dpdk >>>>>> >>>>>> Bridge br-mlx >>>>>> Port "dpdk0" >>>>>> Interface "dpdk0" >>>>>> type: dpdk >>>>>> error: "'dpdk0' is missing 'options:dpdk-devargs'. The old 'dpdk<port_id>' names are not supported" >>>>>> >>>>>> >>>>>> $ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 >>>>>> type=dpdk options:dpdk-devargs=0000:3b:00.0,1 >>>>>> ovs-vsctl: Error detected while setting up 'enp59s0d1': Error attaching device '0000:3b:00.0,1' to DPDK. See ovs-vswitchd log for details. >>>>>> ovs-vsctl: The default log directory is "/var/log/openvswitch". >>>>>> >>>>>> 2018-06-11T09:10:38.696Z|00177|netdev_dpdk|WARN|Error attaching >>>>>> device '0000:3b:00.0,1' to DPDK >>>>>> 2018-06-11T09:10:38.696Z|00178|netdev|WARN|enp59s0d1: could not >>>>>> set configuration (Invalid argument) >>>>>> >>>>>> 2018-06-11T09:12:50.994Z|00181|dpdk|ERR|EAL: Driver cannot attach >>>>>> the device (0000:3b:00.0) >>>>>> 2018-06-11T09:12:50.994Z|00182|dpdk|ERR|EAL: No port found for >>>>>> device >>>>>> (0000:3b:00.0) >>>>>> 2018-06-11T09:12:50.994Z|00183|netdev_dpdk|WARN|Error >>>>>> attaching device '0000:3b:00.0,1' to DPDK >>>>>> 2018-06-11T09:12:50.994Z|00184|netdev|WARN|enp59s0d1: could not >>>>>> set configuration (Invalid argument) >>>>>> >>>>>> $ovs-vsctl add-port br-mlx dpdk0 -- set Interface dpdk0 type=dpdk options:dpdk-devargs="class=eth,mac=ec:0d:9a:b0:13:21" >>>>>> ovs-vsctl: Error detected while setting up 'dpdk0': Error attaching device 'class=eth,mac=ec:0d:9a:b0:13:21' to DPDK. See ovs-vswitchd log for details. >>>>>> ovs-vsctl: The default log directory is "/var/log/openvswitch". >>>>>> >>>>>> Port "dpdk0" >>>>>> Interface "dpdk0" >>>>>> type: dpdk >>>>>> options: {dpdk-devargs="class=eth,mac=ec:0d:9a:b0:13:21"} >>>>>> error: "Error attaching device 'class=eth,mac=ec:0d:9a:b0:13:21' to DPDK" >>>>>> >>>>>> >>>>>> Anybody know what I might have missed? I have tried restarting the openvswitch services after adding the interfaces as well but no difference. >>>>>> >>>>>> The logs doesn't say anything interesting from what I can see :( >>>>>> >>>>>> Best regards >>>>>> > ^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [dpdk-users] mellanox connect-x 3 pro dpdk with openvswitch 2018-06-11 12:14 ` Tobias Urdin @ 2018-06-12 7:13 ` Raslan Darawsheh 0 siblings, 0 replies; 14+ messages in thread From: Raslan Darawsheh @ 2018-06-12 7:13 UTC (permalink / raw) To: Tobias Urdin; +Cc: users I'm not sure about that since as I mentioned before I haven't tried it before, so give it a try and let me know if it worked for you. Kindest regards, Raslan Darawsheh -----Original Message----- From: Tobias Urdin [mailto:tobias.urdin@crystone.com] Sent: Monday, June 11, 2018 3:14 PM To: Raslan Darawsheh <rasland@mellanox.com> Cc: users@dpdk.org Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch I found this librte-pmd-mlx4 driver but it's for 18.02 and I have dpdk 17.11.1 installed. It contains files: ./lib/x86_64-linux-gnu/dpdk-18.02-drivers/librte_pmd_mlx4.so.18.02 ./lib/x86_64-linux-gnu/librte_pmd_mlx4.so.18.02 I assume I can compile librte-pmd-mlx4 manually for 17.11 instead and place it in that folder. It looks like this: $ls -l /usr/lib/x86_64-linux-gnu/dpdk-17.11-drivers/ | head -n 5 total 0 lrwxrwxrwx 1 root root 35 Mar 22 10:23 librte_mempool_octeontx.so.17.11 -> ../librte_mempool_octeontx.so.17.11 lrwxrwxrwx 1 root root 31 Mar 22 10:23 librte_mempool_ring.so.17.11 -> ../librte_mempool_ring.so.17.11 lrwxrwxrwx 1 root root 32 Mar 22 10:23 librte_mempool_stack.so.17.11 -> ../librte_mempool_stack.so.17.11 lrwxrwxrwx 1 root root 32 Mar 22 10:23 librte_pmd_af_packet.so.17.11 -> ../librte_pmd_af_packet.so.17.11 ... Would that be enough or is there anything else I need? Best regards On 06/11/2018 02:07 PM, Raslan Darawsheh wrote: > I think OVS-DPDK has static linking for dpdk and I'm not sure you can > do it this way > > Kindest regards, > Raslan Darawsheh > > -----Original Message----- > From: Tobias Urdin [mailto:tobias.urdin@crystone.com] > Sent: Monday, June 11, 2018 3:05 PM > To: Raslan Darawsheh <rasland@mellanox.com> > Cc: users@dpdk.org > Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch > > Ok, I'm more wondering if there is any way I can build a librte-pmd-mlx4 package that supplies the compiled PMD or something like that instead of having to rebuild all the major parts? > > Thanks! > Best regards > > On 06/11/2018 01:51 PM, Raslan Darawsheh wrote: >> I think you can upgrade the current version that you have as seen in this article: >> https://emea01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fdoc >> s >> .openvswitch.org%2Fen%2Flatest%2Fintro%2Finstall%2Fgeneral%2F%23hot-u >> p >> grading&data=02%7C01%7Crasland%40mellanox.com%7Cfdaa105976c744dba2f90 >> 8 >> d5cf9394cc%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C1%7C6366431551416 >> 0 >> 5703&sdata=VhpqI3M4RdnD5Shpbt6O8YmufIKiyoYK9kxO9TR1Yrk%3D&reserved=0 >> >> but, I haven't tried it before. >> >> Kindest regards, >> Raslan Darawsheh >> >> -----Original Message----- >> From: Tobias Urdin [mailto:tobias.urdin@crystone.com] >> Sent: Monday, June 11, 2018 2:45 PM >> To: Raslan Darawsheh <rasland@mellanox.com> >> Cc: users@dpdk.org >> Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch >> >> Is there any to skip compiling DPDK and OVS+OVS-DPDK and instead use the ones I have installed? >> Only compiling the Mellanox PMD module/binary or something similar to that? >> >> I can see that other drivers supply pmd packages like below. >> >> $dpkg -l | grep -E "openvswitch|pmd|dpdk" >> ii dpdk >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (runtime) >> ii librte-pmd-af-packet17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-af-packet runtime library) >> ii librte-pmd-ark17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte_pmd_ark runtime library) >> ii librte-pmd-avp17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte_pmd_avp runtime library) >> ii librte-pmd-bnxt17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-bnxt runtime library) >> ii librte-pmd-bond17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-bond runtime library) ii librte-pmd-crypto-scheduler17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte_pmd_crypto_scheduler runtime library) >> ii librte-pmd-cxgbe17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-cxgbe runtime library) >> ii librte-pmd-e1000-17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-e1000 runtime library) >> ii librte-pmd-ena17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-ena runtime library) >> ii librte-pmd-enic17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-enic runtime library) >> ii librte-pmd-failsafe17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-failsafe runtime library) >> ii librte-pmd-fm10k17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-fm10k runtime library) >> ii librte-pmd-i40e17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-i40e runtime library) >> ii librte-pmd-ixgbe17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-ixgbe runtime library) >> ii librte-pmd-kni17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte_pmd_kni runtime library) >> ii librte-pmd-lio17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte_pmd_lio runtime library) >> ii librte-pmd-nfp17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte_pmd_nfp runtime library) >> ii librte-pmd-null-crypto17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-null-crypto runtime library) >> ii librte-pmd-null17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-null runtime library) ii librte-pmd-octeontx-ssovf17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte_pmd_octeontx_ssovf runtime library) >> ii librte-pmd-octeontx17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte_pmd_octeontx runtime library) >> ii librte-pmd-pcap17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-pcap runtime library) >> ii librte-pmd-qede17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-qede runtime library) >> ii librte-pmd-ring17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-ring runtime library) >> ii librte-pmd-sfc-efx17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte_pmd_sfc_efx runtime library) ii librte-pmd-skeleton-event17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte_pmd_skeleton_event runtime library) >> ii librte-pmd-softnic17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-softnic runtime library) >> ii librte-pmd-sw-event17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte_pmd_sw_event runtime library) >> ii librte-pmd-tap17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte_pmd_tap runtime library) ii librte-pmd-thunderx-nicvf17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte_pmd_thunderx_nicvf runtime library) >> ii librte-pmd-vhost17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-vhost runtime library) >> ii librte-pmd-virtio17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-virtio runtime library) >> ii librte-pmd-vmxnet3-uio17.11:amd64 >> 17.11.1-6~cloud0 amd64 Data Plane >> Development Kit (librte-pmd-vmxnet3-uio runtime library) >> ii openvswitch-common >> 2.9.0-0ubuntu1~cloud0 amd64 Open vSwitch >> common components >> ii openvswitch-switch >> 2.9.0-0ubuntu1~cloud0 amd64 Open vSwitch >> switch implementations >> ii openvswitch-switch-dpdk >> 2.9.0-0ubuntu1~cloud0 amd64 DPDK enabled >> Open vSwitch switch implementation >> ii python-openvswitch >> 2.9.0-0ubuntu1~cloud0 all Python bindings >> for Open vSwitch >> >> Thanks for all the help! >> Best regards >> >> >> On 06/11/2018 01:37 PM, Raslan Darawsheh wrote: >>> You can refer to this community page for compiling OVS-DPDK with MLX pmds enabled: >>> https://community.mellanox.com/docs/DOC-2637 >>> >>> >>> Kindest regards, >>> Raslan Darawsheh >>> >>> -----Original Message----- >>> From: Tobias Urdin [mailto:tobias.urdin@crystone.com] >>> Sent: Monday, June 11, 2018 2:35 PM >>> To: Raslan Darawsheh <rasland@mellanox.com> >>> Cc: users@dpdk.org >>> Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch >>> >>> I haven't compiled any DPDK or Mellanox related modules/packages. >>> What I have is ovs, ovs-dpdk, mlx4 module, I have also loaded the ib_uverbs driver. >>> >>> Is it the librte_pmd_mlx4 I have to compile manually? >>> >>> Do you know any updated documentation about this, there isn't much relevant and searching for documentation and the correct mlnx driver and code is a struggle. >>> >>> Best regards >>> >>> On 06/11/2018 01:20 PM, Raslan Darawsheh wrote: >>>> From what I'm seeing it looks like it didn't prop the MLX4 device at all. >>>> Can you make sure you have it enabled in your DPDK package when you compiled DPDK ? >>>> >>>> Kindest regards, >>>> Raslan Darawsheh >>>> >>>> -----Original Message----- >>>> From: Tobias Urdin [mailto:tobias.urdin@crystone.com] >>>> Sent: Monday, June 11, 2018 2:16 PM >>>> To: Raslan Darawsheh <rasland@mellanox.com> >>>> Cc: users@dpdk.org >>>> Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch >>>> >>>> Ok, here is the complete log when restarting the openvswitch-switch service, see below. >>>> >>>> Also here is the kernel for information: >>>> $uname -r >>>> 4.4.0-127-generic >>>> >>>> 2018-06-11T11:13:42.729Z|00176|bridge|INFO|bridge br-mlx: deleted >>>> interface br-mlx on port 65534 >>>> 2018-06-11T11:13:43.274Z|00177|bridge|INFO|bridge br-int: deleted >>>> interface br-int on port 65534 >>>> 2018-06-11T11:13:43.275Z|00178|bridge|INFO|bridge br-int: deleted >>>> interface int-br-ex on port 1 >>>> 2018-06-11T11:13:43.275Z|00179|bridge|INFO|bridge br-int: deleted >>>> interface patch-tun on port 2 >>>> 2018-06-11T11:13:43.762Z|00180|bridge|INFO|bridge br-tun: deleted >>>> interface patch-int on port 1 >>>> 2018-06-11T11:13:43.762Z|00181|bridge|INFO|bridge br-tun: deleted >>>> interface br-tun on port 65534 >>>> 2018-06-11T11:13:44.231Z|00182|bridge|INFO|bridge br-ex: deleted >>>> interface br-ex on port 65534 >>>> 2018-06-11T11:13:44.232Z|00183|bridge|INFO|bridge br-ex: deleted >>>> interface phy-br-ex on port 1 >>>> 2018-06-11T11:13:45.245Z|00001|vlog|INFO|opened log file >>>> /var/log/openvswitch/ovs-vswitchd.log >>>> 2018-06-11T11:13:45.252Z|00002|ovs_numa|INFO|Discovered 40 CPU >>>> cores on NUMA node 0 >>>> 2018-06-11T11:13:45.252Z|00003|ovs_numa|INFO|Discovered >>>> 40 CPU cores on NUMA node 1 >>>> 2018-06-11T11:13:45.252Z|00004|ovs_numa|INFO|Discovered 2 NUMA >>>> nodes and >>>> 80 CPU cores >>>> 2018-06-11T11:13:45.252Z|00005|reconnect|INFO|unix:/var/run/openvswitch/db.sock: >>>> connecting... >>>> 2018-06-11T11:13:45.252Z|00006|reconnect|INFO|unix:/var/run/openvswitch/db.sock: >>>> connected >>>> 2018-06-11T11:13:45.256Z|00007|dpdk|INFO|Using DPDK 17.11.0 2018-06-11T11:13:45.256Z|00008|dpdk|INFO|DPDK Enabled - initializing... >>>> 2018-06-11T11:13:45.256Z|00009|dpdk|INFO|No vhost-sock-dir provided - defaulting to /var/run/openvswitch 2018-06-11T11:13:45.256Z|00010|dpdk|INFO|IOMMU support for vhost-user-client disabled. >>>> 2018-06-11T11:13:45.256Z|00011|dpdk|INFO|EAL ARGS: ovs-vswitchd >>>> --socket-mem 1024,0 -c 0x00000001 >>>> 2018-06-11T11:13:45.258Z|00012|dpdk|INFO|EAL: Detected 80 lcore(s) >>>> 2018-06-11T11:13:45.285Z|00013|dpdk|INFO|EAL: Probing VFIO support... >>>> 2018-06-11T11:13:45.285Z|00014|dpdk|INFO|EAL: VFIO support >>>> initialized >>>> 2018-06-11T11:13:48.214Z|00015|dpdk|INFO|EAL: PCI device 0000:01:00.0 on NUMA socket 0 >>>> 2018-06-11T11:13:48.214Z|00016|dpdk|INFO|EAL: probe driver: 8086:1521 >>>> net_e1000_igb >>>> 2018-06-11T11:13:48.214Z|00017|dpdk|INFO|EAL: PCI device 0000:01:00.1 on NUMA socket 0 >>>> 2018-06-11T11:13:48.214Z|00018|dpdk|INFO|EAL: probe driver: 8086:1521 >>>> net_e1000_igb >>>> 2018-06-11T11:13:48.214Z|00019|dpdk|INFO|EAL: PCI device 0000:18:00.0 on NUMA socket 0 >>>> 2018-06-11T11:13:48.214Z|00020|dpdk|INFO|EAL: probe driver: 8086:1572 >>>> net_i40e >>>> 2018-06-11T11:13:48.214Z|00021|dpdk|INFO|EAL: PCI device 0000:18:00.1 on NUMA socket 0 >>>> 2018-06-11T11:13:48.214Z|00022|dpdk|INFO|EAL: probe driver: 8086:1572 >>>> net_i40e >>>> 2018-06-11T11:13:48.214Z|00023|dpdk|INFO|DPDK pdump packet capture >>>> enabled 2018-06-11T11:13:48.224Z|00024|dpdk|INFO|DPDK Enabled - >>>> initialized >>>> 2018-06-11T11:13:48.235Z|00025|ofproto_dpif|INFO|netdev@ovs-netdev: >>>> Datapath supports recirculation >>>> 2018-06-11T11:13:48.235Z|00026|ofproto_dpif|INFO|netdev@ovs-netdev: >>>> VLAN header stack length probed as 1 >>>> 2018-06-11T11:13:48.235Z|00027|ofproto_dpif|INFO|netdev@ovs-netdev: >>>> MPLS label stack length probed as 3 >>>> 2018-06-11T11:13:48.235Z|00028|ofproto_dpif|INFO|netdev@ovs-netdev: >>>> Datapath supports truncate action >>>> 2018-06-11T11:13:48.235Z|00029|ofproto_dpif|INFO|netdev@ovs-netdev: >>>> Datapath supports unique flow ids >>>> 2018-06-11T11:13:48.235Z|00030|ofproto_dpif|INFO|netdev@ovs-netdev: >>>> Datapath supports clone action >>>> 2018-06-11T11:13:48.235Z|00031|ofproto_dpif|INFO|netdev@ovs-netdev: >>>> Max sample nesting level probed as 10 >>>> 2018-06-11T11:13:48.235Z|00032|ofproto_dpif|INFO|netdev@ovs-netdev: >>>> Datapath supports eventmask in conntrack action >>>> 2018-06-11T11:13:48.235Z|00033|ofproto_dpif|INFO|netdev@ovs-netdev: >>>> Datapath supports ct_clear action >>>> 2018-06-11T11:13:48.235Z|00034|ofproto_dpif|INFO|netdev@ovs-netdev: >>>> Datapath supports ct_state >>>> 2018-06-11T11:13:48.235Z|00035|ofproto_dpif|INFO|netdev@ovs-netdev: >>>> Datapath supports ct_zone >>>> 2018-06-11T11:13:48.235Z|00036|ofproto_dpif|INFO|netdev@ovs-netdev: >>>> Datapath supports ct_mark >>>> 2018-06-11T11:13:48.235Z|00037|ofproto_dpif|INFO|netdev@ovs-netdev: >>>> Datapath supports ct_label >>>> 2018-06-11T11:13:48.235Z|00038|ofproto_dpif|INFO|netdev@ovs-netdev: >>>> Datapath supports ct_state_nat >>>> 2018-06-11T11:13:48.235Z|00039|ofproto_dpif|INFO|netdev@ovs-netdev: >>>> Datapath supports ct_orig_tuple >>>> 2018-06-11T11:13:48.235Z|00040|ofproto_dpif|INFO|netdev@ovs-netdev: >>>> Datapath supports ct_orig_tuple6 >>>> 2018-06-11T11:13:48.605Z|00041|ofproto_dpif|INFO|system@ovs-system: >>>> Datapath supports recirculation >>>> 2018-06-11T11:13:48.605Z|00042|ofproto_dpif|INFO|system@ovs-system: >>>> VLAN header stack length probed as 1 >>>> 2018-06-11T11:13:48.605Z|00043|ofproto_dpif|INFO|system@ovs-system: >>>> MPLS label stack length probed as 1 >>>> 2018-06-11T11:13:48.605Z|00044|ofproto_dpif|INFO|system@ovs-system: >>>> Datapath does not support truncate action >>>> 2018-06-11T11:13:48.605Z|00045|ofproto_dpif|INFO|system@ovs-system: >>>> Datapath supports unique flow ids >>>> 2018-06-11T11:13:48.605Z|00046|ofproto_dpif|INFO|system@ovs-system: >>>> Datapath does not support clone action >>>> 2018-06-11T11:13:48.605Z|00047|ofproto_dpif|INFO|system@ovs-system: >>>> Max sample nesting level probed as 3 >>>> 2018-06-11T11:13:48.605Z|00048|ofproto_dpif|INFO|system@ovs-system: >>>> Datapath does not support eventmask in conntrack action >>>> 2018-06-11T11:13:48.605Z|00049|ofproto_dpif|INFO|system@ovs-system: >>>> Datapath does not support ct_clear action >>>> 2018-06-11T11:13:48.605Z|00050|ofproto_dpif|INFO|system@ovs-system: >>>> Datapath supports ct_state >>>> 2018-06-11T11:13:48.605Z|00051|ofproto_dpif|INFO|system@ovs-system: >>>> Datapath supports ct_zone >>>> 2018-06-11T11:13:48.605Z|00052|ofproto_dpif|INFO|system@ovs-system: >>>> Datapath supports ct_mark >>>> 2018-06-11T11:13:48.605Z|00053|ofproto_dpif|INFO|system@ovs-system: >>>> Datapath supports ct_label >>>> 2018-06-11T11:13:48.605Z|00054|ofproto_dpif|INFO|system@ovs-system: >>>> Datapath does not support ct_state_nat >>>> 2018-06-11T11:13:48.605Z|00055|ofproto_dpif|INFO|system@ovs-system: >>>> Datapath does not support ct_orig_tuple >>>> 2018-06-11T11:13:48.605Z|00056|ofproto_dpif|INFO|system@ovs-system: >>>> Datapath does not support ct_orig_tuple6 >>>> 2018-06-11T11:13:48.981Z|00057|bridge|INFO|bridge br-mlx: added >>>> interface br-mlx on port 65534 >>>> 2018-06-11T11:13:48.981Z|00058|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T11:13:48.981Z|00059|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T11:13:48.981Z|00060|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T11:13:48.982Z|00061|bridge|INFO|bridge br-int: added >>>> interface br-int on port 65534 >>>> 2018-06-11T11:13:48.982Z|00062|bridge|INFO|bridge br-int: added >>>> interface int-br-ex on port 1 >>>> 2018-06-11T11:13:48.982Z|00063|bridge|INFO|bridge br-int: added >>>> interface patch-tun on port 2 >>>> 2018-06-11T11:13:48.982Z|00064|bridge|INFO|bridge br-tun: added >>>> interface patch-int on port 1 >>>> 2018-06-11T11:13:48.982Z|00065|bridge|INFO|bridge br-tun: added >>>> interface br-tun on port 65534 >>>> 2018-06-11T11:13:48.982Z|00066|bridge|INFO|bridge br-ex: added >>>> interface br-ex on port 65534 >>>> 2018-06-11T11:13:48.982Z|00067|bridge|INFO|bridge br-ex: added >>>> interface phy-br-ex on port 1 >>>> 2018-06-11T11:13:48.982Z|00068|bridge|INFO|bridge br-mlx: using >>>> datapath ID 000072528484a642 >>>> 2018-06-11T11:13:48.982Z|00069|connmgr|INFO|br-mlx: added service controller "punix:/var/run/openvswitch/br-mlx.mgmt" >>>> 2018-06-11T11:13:49.221Z|00070|bridge|INFO|bridge br-int: using >>>> datapath ID 00006680927b6b40 >>>> 2018-06-11T11:13:49.221Z|00071|connmgr|INFO|br-int: added service controller "punix:/var/run/openvswitch/br-int.mgmt" >>>> 2018-06-11T11:13:49.221Z|00072|connmgr|INFO|br-int: added primary controller "tcp:127.0.0.1:6633" >>>> 2018-06-11T11:13:49.221Z|00073|rconn|INFO|br-int<->tcp:127.0.0.1:6633: >>>> connecting... >>>> 2018-06-11T11:13:49.460Z|00074|bridge|INFO|bridge br-tun: using >>>> datapath ID 000066532e5f0443 >>>> 2018-06-11T11:13:49.461Z|00075|connmgr|INFO|br-tun: added service controller "punix:/var/run/openvswitch/br-tun.mgmt" >>>> 2018-06-11T11:13:49.461Z|00076|connmgr|INFO|br-tun: added primary controller "tcp:127.0.0.1:6633" >>>> 2018-06-11T11:13:49.461Z|00077|rconn|INFO|br-tun<->tcp:127.0.0.1:6633: >>>> connecting... >>>> 2018-06-11T11:13:49.692Z|00078|bridge|INFO|bridge br-ex: using >>>> datapath ID 0000e2d1bdbe8e45 >>>> 2018-06-11T11:13:49.692Z|00079|connmgr|INFO|br-ex: added service controller "punix:/var/run/openvswitch/br-ex.mgmt" >>>> 2018-06-11T11:13:49.692Z|00080|connmgr|INFO|br-ex: added primary controller "tcp:127.0.0.1:6633" >>>> 2018-06-11T11:13:49.692Z|00081|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: >>>> connecting... >>>> 2018-06-11T11:13:49.916Z|00082|timeval|WARN|Unreasonably long >>>> 4661ms poll interval (16ms user, 4472ms system) >>>> 2018-06-11T11:13:49.916Z|00083|timeval|WARN|faults: 3584706 minor, >>>> 0 major >>>> 2018-06-11T11:13:49.916Z|00084|timeval|WARN|disk: 0 reads, 16 writes 2018-06-11T11:13:49.916Z|00085|timeval|WARN|context switches: 147 voluntary, 25 involuntary 2018-06-11T11:13:49.916Z|00086|coverage|INFO|Event coverage, avg rate over last: 5 seconds, last minute, last hour, hash=2ba35161: >>>> 2018-06-11T11:13:49.916Z|00087|coverage|INFO|bridge_reconfigure >>>> 0.0/sec 0.000/sec 0.0000/sec total: 1 >>>> 2018-06-11T11:13:49.916Z|00088|coverage|INFO|ofproto_flush >>>> 0.0/sec 0.000/sec 0.0000/sec total: 4 >>>> 2018-06-11T11:13:49.916Z|00089|coverage|INFO|ofproto_update_port >>>> 0.0/sec 0.000/sec 0.0000/sec total: 8 >>>> 2018-06-11T11:13:49.916Z|00090|coverage|INFO|rev_flow_table >>>> 0.0/sec 0.000/sec 0.0000/sec total: 1 >>>> 2018-06-11T11:13:49.916Z|00091|coverage|INFO|cmap_expand >>>> 0.0/sec 0.000/sec 0.0000/sec total: 52 >>>> 2018-06-11T11:13:49.916Z|00092|coverage|INFO|cmap_shrink >>>> 0.0/sec 0.000/sec 0.0000/sec total: 23 >>>> 2018-06-11T11:13:49.916Z|00093|coverage|INFO|dpif_port_add >>>> 0.0/sec 0.000/sec 0.0000/sec total: 1 >>>> 2018-06-11T11:13:49.916Z|00094|coverage|INFO|dpif_flow_flush >>>> 0.0/sec 0.000/sec 0.0000/sec total: 6 >>>> 2018-06-11T11:13:49.916Z|00095|coverage|INFO|dpif_flow_get >>>> 0.0/sec 0.000/sec 0.0000/sec total: 29 >>>> 2018-06-11T11:13:49.916Z|00096|coverage|INFO|dpif_flow_put >>>> 0.0/sec 0.000/sec 0.0000/sec total: 37 >>>> 2018-06-11T11:13:49.916Z|00097|coverage|INFO|dpif_flow_del >>>> 0.0/sec 0.000/sec 0.0000/sec total: 29 >>>> 2018-06-11T11:13:49.916Z|00098|coverage|INFO|dpif_execute >>>> 0.0/sec 0.000/sec 0.0000/sec total: 8 >>>> 2018-06-11T11:13:49.916Z|00099|coverage|INFO|flow_extract >>>> 0.0/sec 0.000/sec 0.0000/sec total: 6 >>>> 2018-06-11T11:13:49.916Z|00100|coverage|INFO|miniflow_malloc >>>> 0.0/sec 0.000/sec 0.0000/sec total: 34 >>>> 2018-06-11T11:13:49.916Z|00101|coverage|INFO|hmap_expand >>>> 0.0/sec 0.000/sec 0.0000/sec total: 498 >>>> 2018-06-11T11:13:49.916Z|00102|coverage|INFO|netdev_received >>>> 0.0/sec 0.000/sec 0.0000/sec total: 1 >>>> 2018-06-11T11:13:49.916Z|00103|coverage|INFO|netdev_get_stats >>>> 0.0/sec 0.000/sec 0.0000/sec total: 8 >>>> 2018-06-11T11:13:49.916Z|00104|coverage|INFO|poll_create_node >>>> 0.0/sec 0.000/sec 0.0000/sec total: 23 >>>> 2018-06-11T11:13:49.916Z|00105|coverage|INFO|seq_change >>>> 0.0/sec 0.000/sec 0.0000/sec total: 2193 >>>> 2018-06-11T11:13:49.916Z|00106|coverage|INFO|pstream_open >>>> 0.0/sec 0.000/sec 0.0000/sec total: 9 >>>> 2018-06-11T11:13:49.916Z|00107|coverage|INFO|stream_open >>>> 0.0/sec 0.000/sec 0.0000/sec total: 4 >>>> 2018-06-11T11:13:49.916Z|00108|coverage|INFO|util_xalloc >>>> 0.0/sec 0.000/sec 0.0000/sec total: 12803 >>>> 2018-06-11T11:13:49.916Z|00109|coverage|INFO|vconn_open >>>> 0.0/sec 0.000/sec 0.0000/sec total: 3 >>>> 2018-06-11T11:13:49.916Z|00110|coverage|INFO|netdev_set_policing >>>> 0.0/sec 0.000/sec 0.0000/sec total: 4 >>>> 2018-06-11T11:13:49.916Z|00111|coverage|INFO|netdev_get_ifindex >>>> 0.0/sec 0.000/sec 0.0000/sec total: 4 >>>> 2018-06-11T11:13:49.916Z|00112|coverage|INFO|netdev_get_hwaddr >>>> 0.0/sec 0.000/sec 0.0000/sec total: 9 >>>> 2018-06-11T11:13:49.916Z|00113|coverage|INFO|netdev_get_ethtool >>>> 0.0/sec 0.000/sec 0.0000/sec total: 2 >>>> 2018-06-11T11:13:49.916Z|00114|coverage|INFO|netlink_received >>>> 0.0/sec 0.000/sec 0.0000/sec total: 85 >>>> 2018-06-11T11:13:49.916Z|00115|coverage|INFO|netlink_recv_jumbo >>>> 0.0/sec 0.000/sec 0.0000/sec total: 1 >>>> 2018-06-11T11:13:49.916Z|00116|coverage|INFO|netlink_sent >>>> 0.0/sec 0.000/sec 0.0000/sec total: 102 >>>> 2018-06-11T11:13:49.916Z|00117|coverage|INFO|74 events never hit >>>> 2018-06-11T11:13:49.916Z|00118|poll_loop|INFO|wakeup due to >>>> [POLLIN] on fd 433 (FIFO pipe:[1025715]) at ../lib/ovs-rcu.c:229 >>>> (97% CPU >>>> usage) 2018-06-11T11:13:49.917Z|00119|poll_loop|INFO|wakeup due to >>>> [POLLIN] on fd 433 (FIFO pipe:[1025715]) at ../lib/ovs-rcu.c:229 >>>> (97% CPU usage) >>>> 2018-06-11T11:13:49.917Z|00120|poll_loop|INFO|wakeup >>>> due to 0-ms timeout at ../lib/ovs-rcu.c:229 (97% CPU usage) >>>> 2018-06-11T11:13:49.917Z|00121|poll_loop|INFO|wakeup due to 0-ms >>>> timeout at ../lib/ovs-rcu.c:229 (97% CPU usage) >>>> 2018-06-11T11:13:49.920Z|00122|rconn|INFO|br-int<->tcp:127.0.0.1:6633: >>>> connected >>>> 2018-06-11T11:13:49.920Z|00123|rconn|INFO|br-tun<->tcp:127.0.0.1:6633: >>>> connected >>>> 2018-06-11T11:13:49.920Z|00124|rconn|INFO|br-ex<->tcp:127.0.0.1:6633: >>>> connected >>>> 2018-06-11T11:13:49.926Z|00125|poll_loop|INFO|wakeup due to >>>> [POLLIN] on fd 10 (NETLINK_ROUTE<->NETLINK_ROUTE) at >>>> ../lib/netlink-socket.c:1331 (97% CPU usage) >>>> 2018-06-11T11:13:49.926Z|00126|poll_loop|INFO|wakeup due to >>>> [POLLIN] on fd 421 (character device /dev/net/tun) at >>>> ../lib/netdev-linux.c:1171 (97% CPU usage) >>>> 2018-06-11T11:13:49.926Z|00127|poll_loop|INFO|wakeup due to >>>> [POLLIN] on fd 423 (127.0.0.1:33712<->127.0.0.1:6633) at >>>> ../lib/stream-fd.c:157 (97% CPU usage) >>>> 2018-06-11T11:13:49.926Z|00128|poll_loop|INFO|wakeup >>>> due to [POLLIN] on fd 428 (127.0.0.1:33714<->127.0.0.1:6633) at >>>> ../lib/stream-fd.c:157 (97% CPU usage) >>>> 2018-06-11T11:13:49.926Z|00129|poll_loop|INFO|wakeup due to >>>> [POLLIN] on fd 11 (<->/var/run/openvswitch/db.sock) at >>>> ../lib/stream-fd.c:157 (97% CPU usage) >>>> 2018-06-11T11:13:49.926Z|00130|poll_loop|INFO|wakeup >>>> due to [POLLIN] on fd 429 (127.0.0.1:33716<->127.0.0.1:6633) at >>>> ../lib/stream-fd.c:157 (97% CPU usage) >>>> 2018-06-11T11:13:49.930Z|00131|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T11:13:49.930Z|00132|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T11:13:49.930Z|00133|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T11:13:49.931Z|00134|bridge|INFO|ovs-vswitchd (Open >>>> vSwitch) >>>> 2.9.0 >>>> 2018-06-11T11:13:53.662Z|00135|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T11:13:53.662Z|00136|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T11:13:53.662Z|00137|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T11:13:53.675Z|00138|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T11:13:53.675Z|00139|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T11:13:53.675Z|00140|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T11:13:53.682Z|00141|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T11:13:53.682Z|00142|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T11:13:53.682Z|00143|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T11:13:53.693Z|00144|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T11:13:53.693Z|00145|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T11:13:53.693Z|00146|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T11:13:53.705Z|00147|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T11:13:53.705Z|00148|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T11:13:53.705Z|00149|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T11:13:53.711Z|00150|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T11:13:53.711Z|00151|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T11:13:53.711Z|00152|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T11:13:53.717Z|00153|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T11:13:53.717Z|00154|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T11:13:53.717Z|00155|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T11:13:53.728Z|00156|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T11:13:53.728Z|00157|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T11:13:53.728Z|00158|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T11:13:53.743Z|00159|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T11:13:53.743Z|00160|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T11:13:53.743Z|00161|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T11:13:53.749Z|00162|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T11:13:53.749Z|00163|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T11:13:53.749Z|00164|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T11:13:53.756Z|00165|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T11:13:53.756Z|00166|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T11:13:53.756Z|00167|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T11:13:53.766Z|00168|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>> 2018-06-11T11:13:53.766Z|00169|netdev_dpdk|WARN|Error attaching >>>> device 'mlx4_0' to DPDK >>>> 2018-06-11T11:13:53.766Z|00170|netdev|WARN|enp59s0d1: could not set >>>> configuration (Invalid argument) >>>> 2018-06-11T11:13:55.270Z|00171|memory|INFO|1406100 kB peak resident >>>> set size after 10.0 seconds >>>> 2018-06-11T11:13:55.270Z|00172|memory|INFO|handlers:118 ofconns:3 >>>> ports:8 revalidators:42 rules:36 udpif keys:1 >>>> 2018-06-11T11:14:03.699Z|00173|connmgr|INFO|br-int<->tcp:127.0.0.1:6633: >>>> 5 flow_mods 10 s ago (5 adds) >>>> 2018-06-11T11:14:03.737Z|00174|connmgr|INFO|br-ex<->tcp:127.0.0.1:6633: >>>> 2 flow_mods 10 s ago (2 adds) >>>> 2018-06-11T11:14:03.773Z|00175|connmgr|INFO|br-tun<->tcp:127.0.0.1:6633: >>>> 12 flow_mods 10 s ago (12 adds) >>>> >>>> Best regards >>>> >>>> On 06/11/2018 12:57 PM, Raslan Darawsheh wrote: >>>>> What I really want to see is the init process for OVS-DPDK that dumps the DPDK initialization. >>>>> I think you will find it at the beginning of the log that you just sent. >>>>> >>>>> Kindest regards, >>>>> Raslan Darawsheh >>>>> >>>>> -----Original Message----- >>>>> From: Tobias Urdin [mailto:tobias.urdin@crystone.com] >>>>> Sent: Monday, June 11, 2018 1:00 PM >>>>> To: Raslan Darawsheh <rasland@mellanox.com> >>>>> Cc: users@dpdk.org >>>>> Subject: Re: mellanox connect-x 3 pro dpdk with openvswitch >>>>> >>>>> Hello Raslan, >>>>> Thank you for your reply. >>>>> >>>>> For reference the br-mlx bridge was created like this: >>>>> $ovs-vsctl add-br br-mlx -- set bridge br-mlx datapath_type=netdev >>>>> >>>>> $ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 >>>>> type=dpdk options:dpdk-devargs=mlx4_0 >>>>> ovs-vsctl: Error detected while setting up 'enp59s0d1': Error attaching device 'mlx4_0' to DPDK. See ovs-vswitchd log for details. >>>>> ovs-vsctl: The default log directory is "/var/log/openvswitch". >>>>> >>>>> In the ovs-vswitchd.log >>>>> 2018-06-11T09:56:20.613Z|00185|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>>> 2018-06-11T09:56:20.613Z|00186|netdev_dpdk|WARN|Error attaching >>>>> device 'mlx4_0' to DPDK >>>>> 2018-06-11T09:56:20.613Z|00187|netdev|WARN|enp59s0d1: could not >>>>> set configuration (Invalid argument) >>>>> >>>>> Here is the complete log after restarting openvswitch-switch service: >>>>> 2018-06-11T09:57:49.380Z|00134|bridge|INFO|ovs-vswitchd (Open >>>>> vSwitch) >>>>> 2.9.0 >>>>> 2018-06-11T09:57:52.812Z|00135|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>>> 2018-06-11T09:57:52.812Z|00136|netdev_dpdk|WARN|Error attaching >>>>> device 'mlx4_0' to DPDK >>>>> 2018-06-11T09:57:52.812Z|00137|netdev|WARN|enp59s0d1: could not >>>>> set configuration (Invalid argument) >>>>> 2018-06-11T09:57:52.825Z|00138|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>>> 2018-06-11T09:57:52.825Z|00139|netdev_dpdk|WARN|Error attaching >>>>> device 'mlx4_0' to DPDK >>>>> 2018-06-11T09:57:52.825Z|00140|netdev|WARN|enp59s0d1: could not >>>>> set configuration (Invalid argument) >>>>> 2018-06-11T09:57:52.832Z|00141|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>>> 2018-06-11T09:57:52.832Z|00142|netdev_dpdk|WARN|Error attaching >>>>> device 'mlx4_0' to DPDK >>>>> 2018-06-11T09:57:52.832Z|00143|netdev|WARN|enp59s0d1: could not >>>>> set configuration (Invalid argument) >>>>> 2018-06-11T09:57:52.843Z|00144|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>>> 2018-06-11T09:57:52.843Z|00145|netdev_dpdk|WARN|Error attaching >>>>> device 'mlx4_0' to DPDK >>>>> 2018-06-11T09:57:52.843Z|00146|netdev|WARN|enp59s0d1: could not >>>>> set configuration (Invalid argument) >>>>> 2018-06-11T09:57:52.856Z|00147|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>>> 2018-06-11T09:57:52.856Z|00148|netdev_dpdk|WARN|Error attaching >>>>> device 'mlx4_0' to DPDK >>>>> 2018-06-11T09:57:52.856Z|00149|netdev|WARN|enp59s0d1: could not >>>>> set configuration (Invalid argument) >>>>> 2018-06-11T09:57:52.862Z|00150|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>>> 2018-06-11T09:57:52.862Z|00151|netdev_dpdk|WARN|Error attaching >>>>> device 'mlx4_0' to DPDK >>>>> 2018-06-11T09:57:52.862Z|00152|netdev|WARN|enp59s0d1: could not >>>>> set configuration (Invalid argument) >>>>> 2018-06-11T09:57:52.870Z|00153|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>>> 2018-06-11T09:57:52.870Z|00154|netdev_dpdk|WARN|Error attaching >>>>> device 'mlx4_0' to DPDK >>>>> 2018-06-11T09:57:52.870Z|00155|netdev|WARN|enp59s0d1: could not >>>>> set configuration (Invalid argument) >>>>> 2018-06-11T09:57:52.889Z|00156|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>>> 2018-06-11T09:57:52.889Z|00157|netdev_dpdk|WARN|Error attaching >>>>> device 'mlx4_0' to DPDK >>>>> 2018-06-11T09:57:52.889Z|00158|netdev|WARN|enp59s0d1: could not >>>>> set configuration (Invalid argument) >>>>> 2018-06-11T09:57:52.903Z|00159|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>>> 2018-06-11T09:57:52.903Z|00160|netdev_dpdk|WARN|Error attaching >>>>> device 'mlx4_0' to DPDK >>>>> 2018-06-11T09:57:52.903Z|00161|netdev|WARN|enp59s0d1: could not >>>>> set configuration (Invalid argument) >>>>> 2018-06-11T09:57:52.908Z|00162|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>>> 2018-06-11T09:57:52.908Z|00163|netdev_dpdk|WARN|Error attaching >>>>> device 'mlx4_0' to DPDK >>>>> 2018-06-11T09:57:52.908Z|00164|netdev|WARN|enp59s0d1: could not >>>>> set configuration (Invalid argument) >>>>> 2018-06-11T09:57:52.914Z|00165|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>>> 2018-06-11T09:57:52.914Z|00166|netdev_dpdk|WARN|Error attaching >>>>> device 'mlx4_0' to DPDK >>>>> 2018-06-11T09:57:52.914Z|00167|netdev|WARN|enp59s0d1: could not >>>>> set configuration (Invalid argument) >>>>> 2018-06-11T09:57:52.923Z|00168|dpdk|ERR|EAL: Unable to find a bus for the device 'mlx4_0' >>>>> 2018-06-11T09:57:52.923Z|00169|netdev_dpdk|WARN|Error attaching >>>>> device 'mlx4_0' to DPDK >>>>> 2018-06-11T09:57:52.923Z|00170|netdev|WARN|enp59s0d1: could not >>>>> set configuration (Invalid argument) >>>>> 2018-06-11T09:57:54.808Z|00171|memory|INFO|1406100 kB peak >>>>> resident set size after 10.1 seconds >>>>> 2018-06-11T09:57:54.808Z|00172|memory|INFO|handlers:118 ofconns:3 >>>>> ports:8 revalidators:42 rules:36 udpif keys:1 >>>>> 2018-06-11T09:58:02.850Z|00173|connmgr|INFO|br-int<->tcp:127.0.0.1:6633: >>>>> 5 flow_mods 10 s ago (5 adds) >>>>> 2018-06-11T09:58:02.898Z|00174|connmgr|INFO|br-ex<->tcp:127.0.0.1:6633: >>>>> 2 flow_mods 10 s ago (2 adds) >>>>> 2018-06-11T09:58:02.930Z|00175|connmgr|INFO|br-tun<->tcp:127.0.0.1:6633: >>>>> 12 flow_mods 10 s ago (12 adds) >>>>> >>>>> Was that what you mean by init log? >>>>> Best regards >>>>> >>>>> On 06/11/2018 11:51 AM, Raslan Darawsheh wrote: >>>>>> Hi Tobias, >>>>>> >>>>>> Can you try using the following ? >>>>>> >>>>>> ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 >>>>>> type=dpdk options:dpdk-devargs=mlx4_0 >>>>>> >>>>>> if it didn't work can you send the OVS-DPDK init log ? >>>>>> >>>>>> Kindest regards, >>>>>> Raslan Darawsheh >>>>>> >>>>>> -----Original Message----- >>>>>> From: users [mailto:users-bounces@dpdk.org] On Behalf Of Tobias >>>>>> Urdin >>>>>> Sent: Monday, June 11, 2018 12:15 PM >>>>>> To: users@dpdk.org >>>>>> Subject: [dpdk-users] mellanox connect-x 3 pro dpdk with >>>>>> openvswitch >>>>>> >>>>>> Hello, >>>>>> >>>>>> I'm stuck trying to test DPDK with a Mellanox Connect-X 3 Pro >>>>>> card on Ubuntu 16.04 >>>>>> >>>>>> I've enabled iommu, fixed hugepages and has installed and are running the openvswitch-switch-dpdk alternative. >>>>>> >>>>>> $lspci | grep -i mellanox >>>>>> 3b:00.0 Ethernet controller: Mellanox Technologies MT27520 Family >>>>>> [ConnectX-3 Pro] >>>>>> >>>>>> $lsmod | grep -i mlx >>>>>> mlx4_en 110592 0 >>>>>> vxlan 49152 2 i40e,mlx4_en >>>>>> mlx4_core 286720 1 mlx4_en >>>>>> ptp 20480 3 igb,i40e,mlx4_en >>>>>> >>>>>> $cat /etc/modprobe.d/mlx4_core.conf options mlx4_core >>>>>> log_num_mgm_entry_size=-1 >>>>>> >>>>>> >>>>>> $ovs-vsctl add-port br-mlx dpdk0 -- set Interface dpdk0 type=dpdk >>>>>> >>>>>> Bridge br-mlx >>>>>> Port "dpdk0" >>>>>> Interface "dpdk0" >>>>>> type: dpdk >>>>>> error: "'dpdk0' is missing 'options:dpdk-devargs'. The old 'dpdk<port_id>' names are not supported" >>>>>> >>>>>> >>>>>> $ovs-vsctl add-port br-mlx enp59s0d1 -- set Interface enp59s0d1 >>>>>> type=dpdk options:dpdk-devargs=0000:3b:00.0,1 >>>>>> ovs-vsctl: Error detected while setting up 'enp59s0d1': Error attaching device '0000:3b:00.0,1' to DPDK. See ovs-vswitchd log for details. >>>>>> ovs-vsctl: The default log directory is "/var/log/openvswitch". >>>>>> >>>>>> 2018-06-11T09:10:38.696Z|00177|netdev_dpdk|WARN|Error attaching >>>>>> device '0000:3b:00.0,1' to DPDK >>>>>> 2018-06-11T09:10:38.696Z|00178|netdev|WARN|enp59s0d1: could not >>>>>> set configuration (Invalid argument) >>>>>> >>>>>> 2018-06-11T09:12:50.994Z|00181|dpdk|ERR|EAL: Driver cannot attach >>>>>> the device (0000:3b:00.0) >>>>>> 2018-06-11T09:12:50.994Z|00182|dpdk|ERR|EAL: No port found for >>>>>> device >>>>>> (0000:3b:00.0) >>>>>> 2018-06-11T09:12:50.994Z|00183|netdev_dpdk|WARN|Error >>>>>> attaching device '0000:3b:00.0,1' to DPDK >>>>>> 2018-06-11T09:12:50.994Z|00184|netdev|WARN|enp59s0d1: could not >>>>>> set configuration (Invalid argument) >>>>>> >>>>>> $ovs-vsctl add-port br-mlx dpdk0 -- set Interface dpdk0 type=dpdk options:dpdk-devargs="class=eth,mac=ec:0d:9a:b0:13:21" >>>>>> ovs-vsctl: Error detected while setting up 'dpdk0': Error attaching device 'class=eth,mac=ec:0d:9a:b0:13:21' to DPDK. See ovs-vswitchd log for details. >>>>>> ovs-vsctl: The default log directory is "/var/log/openvswitch". >>>>>> >>>>>> Port "dpdk0" >>>>>> Interface "dpdk0" >>>>>> type: dpdk >>>>>> options: {dpdk-devargs="class=eth,mac=ec:0d:9a:b0:13:21"} >>>>>> error: "Error attaching device 'class=eth,mac=ec:0d:9a:b0:13:21' to DPDK" >>>>>> >>>>>> >>>>>> Anybody know what I might have missed? I have tried restarting the openvswitch services after adding the interfaces as well but no difference. >>>>>> >>>>>> The logs doesn't say anything interesting from what I can see :( >>>>>> >>>>>> Best regards >>>>>> > ^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2018-06-12 7:13 UTC | newest] Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-06-11 9:15 [dpdk-users] mellanox connect-x 3 pro dpdk with openvswitch Tobias Urdin 2018-06-11 9:51 ` Raslan Darawsheh 2018-06-11 9:59 ` Tobias Urdin 2018-06-11 10:57 ` Raslan Darawsheh 2018-06-11 11:15 ` Tobias Urdin 2018-06-11 11:19 ` Raslan Darawsheh 2018-06-11 11:34 ` Tobias Urdin 2018-06-11 11:37 ` Raslan Darawsheh 2018-06-11 11:45 ` Tobias Urdin 2018-06-11 11:51 ` Raslan Darawsheh 2018-06-11 12:05 ` Tobias Urdin 2018-06-11 12:07 ` Raslan Darawsheh 2018-06-11 12:14 ` Tobias Urdin 2018-06-12 7:13 ` Raslan Darawsheh
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).