From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 37590A034E for ; Wed, 6 May 2020 22:51:32 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 1BBE21DB0F; Wed, 6 May 2020 22:51:32 +0200 (CEST) Received: from mx0a-002b3901.pphosted.com (mx0a-002b3901.pphosted.com [148.163.151.18]) by dpdk.org (Postfix) with ESMTP id 66FD01D52F for ; Mon, 4 May 2020 23:18:11 +0200 (CEST) Received: from pps.filterd (m0121840.ppops.net [127.0.0.1]) by mx0a-002b3901.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 044LEi6X017084 for ; Mon, 4 May 2020 17:18:10 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cox.com; h=from : to : subject : date : message-id : content-type : mime-version; s=DKIM1; bh=a0SgNogn4TAFKY8PAqctfVwV+JdIEYkjp7XZdSHygx8=; b=IKE+5ZcPUEwPzU2xzDBC8V2DyxQ5f2BCGX2qCkFuArK3AF0iPqN2l3CVk9Y8AZ5BckQd CTDozYYac0Kcyjqgq2DVT+A2Gielr8CdqLJmv4JsFV2pKYWf2vLATcujruDeL4LLNhxQ zkn7viM6GFulV0MCecRGdzqVzIzidOYlofbGSkwQVMc8Xn+F0st6RZjBAuteRCjCf7Gr 3FO1DNi91aJdB5UUw6llIFOblhdgP/6CoKChkAKIG3GWOra8OFSTaEv4DywbVtMsTPae RMlrzoHwt4NwO5NwNQXcengx9ue+IGh70Q4PanpaE95S49UFefdgWt/j0TT8dbJpgRjw qQ== Received: from nam12-mw2-obe.outbound.protection.outlook.com ([24.248.74.108]) by mx0a-002b3901.pphosted.com with ESMTP id 30s5jk4k2g-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-SHA384 bits=256 verify=NOT) for ; Mon, 04 May 2020 17:18:09 -0400 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=lKrjIFDmude2jTJ6FIrV3as9Gjb5Zx/B8jltDr0y9PhWBRlRNy3fzjDCm0Mcwu0gGdCXQH1KsO7zoz1rpNBzurtFHMJlTyNsH8rfCoJm73ekfaAg9DU8fEnKdyaiB0FkPTw5kwUy3Mzyuf8NMJXhHY0QR6kTkHBfm92f/xS6yNiu3Fx+jiOrgr3VGo87oT7DR3v+6MKETC/DD8YHcxLXKPxW4VVfFY1vpoVVYYxtTYuVOEUm1s7XI2xX7KjT6O+vWf0LvmpJNcDPsH0u3Q6YYLJA+iSlQVkzhqDJLn0KHs9fKfxgN6mdr74DVEDXa34tnFIaYYi4SOYnScmXoUJgcg== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=a0SgNogn4TAFKY8PAqctfVwV+JdIEYkjp7XZdSHygx8=; b=AOr9uqPtoTjG2BWDUBBl4D4SfUzv/avkxxlCPO+QkiLLqDvDIzHRgU0R/wTivFjUGaerWE6OBsC9SUvQnFE0fe+FUcwla6yKafkP1/cG+w1zLCy5YVzawacyBYicQ1oZlE1AaF+ZfevQIdD9zNqsB4lSAleJTLrU5wrudcsZIESNfdtlaHPX/QM1owF9A70UUe/DulTXjJCseWwLUtgn2D4Bq+PMk0o8bRxJtUmH6ByM18gxQDmCzunsStecfIc0LRWR0vmPm+Lo6mMFyDrht143YUo7XsfS1k1n9V4kMilrNHynqVa5mKEhdOjx0og8H1h/qKxSi7lOGQ3BtMU+7g== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=cox.com; dmarc=pass action=none header.from=cox.com; dkim=pass header.d=cox.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cox.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=a0SgNogn4TAFKY8PAqctfVwV+JdIEYkjp7XZdSHygx8=; b=nIx7gpFkZ+USpiHLIcBws1sCF9oosVX0Fdee3cyxZ2NqNpLycNFk7VRgwljkz7LwJBg9K/hQxMKacnwhzWybuUmabiu8NafsE5HPXpAOoyqWvWYPN6OsYgau5NxvTZRCDmdJ8aqfhK/xZhRr/udPQfZerqMa+jH/rrejfZgRSLI= Received: from BN6PR0101MB2850.prod.exchangelabs.com (2603:10b6:405:2c::11) by BN6SPR01MB0021.prod.exchangelabs.com (2603:10b6:405:78::11) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2958.20; Mon, 4 May 2020 21:18:05 +0000 Received: from BN6PR0101MB2850.prod.exchangelabs.com ([fe80::910e:446e:4b05:a5e8]) by BN6PR0101MB2850.prod.exchangelabs.com ([fe80::910e:446e:4b05:a5e8%7]) with mapi id 15.20.2958.029; Mon, 4 May 2020 21:18:05 +0000 From: "Wittling, Mark (CCI-Atlanta)" To: "users@dpdk.org" Thread-Topic: Assistance Request: Error adding DPDK VFIO NIC to OVS+DPDK Bridge Thread-Index: AdYiWYG1bjxMeAO+QVGQWqPto2EuAg== Date: Mon, 4 May 2020 21:18:05 +0000 Message-ID: Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: dpdk.org; dkim=none (message not signed) header.d=none;dpdk.org; dmarc=none action=none header.from=cox.com; x-originating-ip: [97.81.125.202] x-ms-publictraffictype: Email x-ms-office365-filtering-ht: Tenant x-ms-office365-filtering-correlation-id: 8f522951-c487-494c-03cb-08d7f070a2e2 x-ms-traffictypediagnostic: BN6SPR01MB0021: x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:6430; x-forefront-prvs: 03932714EB x-ms-exchange-senderadcheck: 1 x-microsoft-antispam: BCL:0; x-microsoft-antispam-message-info: Flmd6HaWKkv95z9tpyQxBYKv62KLnujXuxQ1IIGcXIeBnG6SKeHbuZP7puQLaaQ5eCXiryw4JXeGn6Rlov/xBb479iH7De4XCjAuNypVHNj42BVDl2TAH4HyRCe248louYUY5AI+c3o1rWU37LOH0qS/ETKh4gwbj2aikev+JCSndjs9S/Ul+fDaWJJydTNhBlNa166NBVEC3t/lJdD9gFhvpHrjq1LD+rDIEJ7r6QTs8vrs90ulSrGbpKvLPjYzLHUbItPc30R29HsRMNmhzunKrQCQLds1v5EvC2hEhfMoqwPEzGrQGBY+hxCyM7YPy9oREOoECsL4k1pFLgs9kogqCOnQldP8Hg3oeYgaiPIWRRRqIRK0Gm47IyKYijWThWx7hxoH6HrfQKL+9HR6deYRoMN5XsJAS+Sn7RRTBIWqOMswFc+GwDNEiWTiaymex3FnqV9IElz+A04YhB/FTYMrUlUA+/HzSBkiXVbXnjExVcI5xI3u7S8uT04QwOgu x-forefront-antispam-report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:; IPV:NLI; SFV:NSPM; H:BN6PR0101MB2850.prod.exchangelabs.com; PTR:; CAT:NONE; SFTY:; SFS:(366004)(39860400002)(376002)(346002)(136003)(396003)(7696005)(6506007)(316002)(86362001)(26005)(71200400001)(76116006)(66446008)(8676002)(66946007)(8936002)(52536014)(66476007)(66556008)(64756008)(33656002)(6916009)(186003)(55016002)(478600001)(5660300002)(9686003)(2906002)(473944003); DIR:OUT; SFP:1101; x-ms-exchange-antispam-messagedata: JXdbWq4Yl6yOVFP2Nx+PJ1WkZVuTIEiyZLtpeLCCfsRKTHsydtdfG/zesaCmEMu/TVgtIep/sSG00JQ/qNMfPJJXAj/uK88JXruldBtwnQEe4eqPeFRXer8/GaTUa5w400BOjo3UaTsUmhdQoAyf/RX7tH/exNqBSDn30QpkQU8/F+cVjJijRWmbcnpJ/OvNtd/EpdWLkSZMe3W3hMZbq0e1yJICorHNpd+MIyQUFyl7NLs0abJSM0EQ1mT6nI6ZOvwIplfYRBl1jYkkCR4USNEDDCbk2AEcxXW2neZn4emJULEDbbiFqipiRQSf0nGQ5QnkBFSPRHT9jm3OwZMdObsvP8IMXUtRhjMdiQRUjyT1K08FDDWLkk0TTy2qEsW8tEAYqXThqrQrdR+vQI6bWtX+UpsBcubu1U5qWFmX2Pnpq5mXl/t4QUu1jK3PJkKnE/BlH27zHdMnpci9dLtd7cc7w9NwrwBJhuPehxrs7kuGnWKaEZS6dTY7jaqJuKnBOzCTvzMKIj2By9i+nX5OFwpxexQZFdOFScPAKJ/1RBillAzyX5qZ+heb+LaMZHP24C4NRRY7SVRYX8ilQ5p/7ER+1I8GxaibUQ7ajMU7hgm0UZh0ddMLSy6gqIXhdT/Y4XZSckeidR5ygjYehajvxLUfxL8niY6NZS8KzMPy3eA61uu7IVg7tAvqlcUn6NnkagwizdaSgxzKAsbb0SCc+S+0tYc/J3Iz6Xk6cwNfQ1Lvxog2qPvuRg0SOBIQvxYEEtSeHVIqYHO6YnAXHTr2UBRlM4AgvVM2CxVfj79yKjs= x-ms-exchange-transport-forked: True MIME-Version: 1.0 X-OriginatorOrg: cox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 8f522951-c487-494c-03cb-08d7f070a2e2 X-MS-Exchange-CrossTenant-originalarrivaltime: 04 May 2020 21:18:05.3610 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: 9feebc97-ff04-42c9-a152-767073872118 X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: O+cc68ANV2LUmSkCwGUANWm4xRWjNXfHTT35Z3Z4d1GvcEndbeyzLMJ2DyLvOJ9bFuR54CiitEBGc1DX8lR+hw== X-MS-Exchange-Transport-CrossTenantHeadersStamped: BN6SPR01MB0021 X-DP-BB: PSTHR X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138, 18.0.676 definitions=2020-05-04_11:2020-05-04, 2020-05-04 signatures=0 X-Proofpoint-Spam-Details: rule=outbound_notspam policy=outbound score=0 mlxscore=0 lowpriorityscore=0 malwarescore=0 mlxlogscore=999 impostorscore=0 phishscore=0 adultscore=0 bulkscore=0 clxscore=1011 priorityscore=1501 suspectscore=0 spamscore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2003020000 definitions=main-2005040166 X-Mailman-Approved-At: Wed, 06 May 2020 22:51:31 +0200 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: [dpdk-users] Assistance Request: Error adding DPDK VFIO NIC to OVS+DPDK Bridge X-BeenThere: users@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK usage discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: users-bounces@dpdk.org Sender: "users" Greetings, I cannot get OpenVSwitch to add my DPDK-bound VFIO port (e1000 D= PDK-compatible NIC) to the bridge without an error. The error is at the end, and I am supplying all of the info I know the comm= unity would typically ask me for, before I show the error at the bottom. Any help would be greatly appreciated. Hardware: Model: Dell Precision T-1700 CPU: # lscpu Architecture: x86_64 CPU op-mode(s): 32-bit, 64-bit Byte Order: Little Endian CPU(s): 4 On-line CPU(s) list: 0-3 Thread(s) per core: 1 Core(s) per socket: 4 Socket(s): 1 NUMA node(s): 1 Vendor ID: GenuineIntel CPU family: 6 Model: 60 Model name: Intel(R) Core(TM) i5-4690 CPU @ 3.50GHz Stepping: 3 CPU MHz: 1183.471 CPU max MHz: 3900.0000 CPU min MHz: 800.0000 BogoMIPS: 6983.91 Virtualization: VT-x L1d cache: 32K L1i cache: 32K L2 cache: 256K L3 cache: 6144K NUMA node0 CPU(s): 0-3 Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge= mca cmov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pbe syscall= nx pdpe1gb rdtscp lm constant_tsc arch_perfmon pebs bts rep_good nopl xtop= ology nonstop_tsc aperfmperf eagerfpu pni pclmulqdq dtes64 monitor ds_cpl v= mx smx est tm2 ssse3 sdbg fma cx16 xtpr pdcm pcid sse4_1 sse4_2 x2apic movb= e popcnt tsc_deadline_timer aes xsave avx f16c rdrand lahf_lm abm epb invpc= id_single ssbd ibrs ibpb stibp tpr_shadow vnmi flexpriority ept vpid fsgsba= se tsc_adjust bmi1 avx2 smep bmi2 erms invpcid xsaveopt dtherm ida arat pln= pts md_clear spec_ctrl intel_stibp flush_l1d # numactl -H available: 1 nodes (0) node 0 cpus: 0 1 2 3 node 0 size: 16019 MB node 0 free: 7554 MB node distances: node 0 0: 10 NOTE: I only have a single 4 core CPU, but it is Numa-enabled and Numa comp= atible, giving me one Numa Node with 4 cores. Memory: # lsmem --summary Memory block size: 128M Total online memory: 16G Total offline memory: 0B # cat /proc/cmdline BOOT_IMAGE=3D/vmlinuz-3.10.0-1127.el7.x86_64 root=3DUUID=3D4102ab69-f71a-4d= d0-a14e-8695aa230a0d ro rhgb quiet iommu=3Dpt intel_iommu=3Don default_huge= pagesz=3D1G hugepagesz=3D1G hugepages=3D4 transparent_hugepage=3Dnever LANG= =3Den_US.UTF-8 A look at the kernel command line, which has iommu and hugepage directives # cat /proc/meminfo | grep Huge AnonHugePages: 0 kB HugePages_Total: 4 HugePages_Free: 3 HugePages_Rsvd: 0 HugePages_Surp: 0 Hugepagesize: 1048576 kB A look at how HugePages are allocated # lsmod | grep vfio vfio_pci 41412 0 vfio_iommu_type1 22440 0 vfio 32657 3 vfio_iommu_type1,vfio_pci irqbypass 13503 2 kvm,vfio_pci Kernel Modules successfully loaded. # /usr/share/dpdk/usertools/dpdk-devbind.py --status Network devices using DPDK-compatible driver =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D 0000:01:00.0 '82571EB/82571GB Gigabit Ethernet Controller D0/D1 (copper app= lications) 105e' drv=3Dvfio-pci unused=3De1000e 0000:01:00.1 '82571EB/82571GB Gigabit Ethernet Controller D0/D1 (copper app= lications) 105e' drv=3Dvfio-pci unused=3De1000e Our DPDK-compatible NICs (p2p1 and p2p2 when not overridden with vfio) have= been properly bound to DPDK. Network devices using kernel driver =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D 0000:00:19.0 'Ethernet Connection I217-LM 153a' if=3Dem1 drv=3De1000e unuse= d=3Dvfio-pci 0000:03:00.0 '82571EB/82571GB Gigabit Ethernet Controller D0/D1 (copper app= lications) 105e' if=3Dp1p1 drv=3De1000e unused=3Dvfio-pci 0000:03:00.1 '82571EB/82571GB Gigabit Ethernet Controller D0/D1 (copper app= lications) 105e' if=3Dp1p2 drv=3De1000e unused=3Dvfio-pci These are the devices not bound to DPDK, em1, p1p1 and p1p2. # ovs-vsctl list Open_vSwitch _uuid : ace95756-927f-4ceb-be27-76d0d5374461 bridges : [2382d0c5-8955-4a27-b001-5fc606aabab8, 518c4675-a34a-= 43de-aedf-0eb9b44a7195, 77d5b674-2e0f-4311-9529-c7d1e2f1c344, 8ee6f680-63b1= -42f0-b0ed-48807cbe49af, d0d3adca-e359-4b12-a775-b75647b98e47] cur_cfg : 369 datapath_types : [netdev, system] db_version : "7.16.1" dpdk_initialized : true dpdk_version : "DPDK 18.11.0" external_ids : {hostname=3Dmaschinen, ovn-encap-ip=3D"192.168.20.201= ", ovn-encap-type=3D"geneve,vxlan", ovn-remote=3D"tcp:192.168.20.200:6642",= rundir=3D"/var/run/openvswitch", system-id=3D"35b95ef5-fd71-491f-8623-5ccb= bc1eca6b"} iface_types : [dpdk, dpdkr, dpdkvhostuser, dpdkvhostuserclient, ers= pan, geneve, gre, internal, "ip6erspan", "ip6gre", lisp, patch, stt, system= , tap, vxlan] manager_options : [8e30d191-ed8c-4f26-bc13-9f1087c0db25] next_cfg : 369 other_config : {dpdk-init=3D"true", dpdk-socket-limit=3D"1024", dpdk= -socket-mem=3D"1024", pmd-cpu-mask=3D"0x8"} ovs_version : "2.11.0" ssl : [] statistics : {} system_type : centos system_version : "7" OpenVSwitch initialized with dpdk (init=3Dtrue), and our socket parms (1 Hu= gepage), and I even set a mdp-cpu-mask to 0x8 to ensure that I am using CPU= core of 3 in the array [0,1,2,3]. # ovs-vsctl add-br br-testdpdk -- set bridge br-testdpdk datapath_type=3Dne= tdev We add a bridge in OpenVSwitch. No issue with this. # ovs-vsctl add-port br-testdpdk p2p1 -- set Interface p2p1 type=3Ddpdk opt= ions:dpdk-devargs=3D0000:01:00.0 ovs-vsctl: Error detected while setting up 'p2p1': Error attaching device '= 0000:01:00.0' to DPDK. See ovs-vswitchd log for details. ovs-vsctl: The default log directory is "/var/log/openvswitch". Adding the PCI 0000:01:00.0 (p2p1) to the bridge fails. Also tried adding 0000:01:00.1 (p2p2) which also fails identical way. # cat ovs-vswitchd.log 2020-05-04T21:12:11.071Z|00291|dpdk|ERR|EAL: Driver cannot attach the devic= e (0000:01:00.0) 2020-05-04T21:12:11.071Z|00292|dpdk|ERR|EAL: Failed to attach device on pri= mary process 2020-05-04T21:12:11.071Z|00293|netdev_dpdk|WARN|Error attaching device '000= 0:01:00.0' to DPDK 2020-05-04T21:12:11.071Z|00294|netdev|WARN|p2p1: could not set configuratio= n (Invalid argument) 2020-05-04T21:12:11.071Z|00295|dpdk|ERR|Invalid port_id=3D32 Looking on the web for an answer to this, has not provided me any results. = Everything up to this point looked just fine. NOTE: I did a full yum update on the box, uninstalled openvswitch, and re-i= nstalled openvswitch. NOTE: I did *not* do a custom build of DPDK or OpenVSwitch. I used yum to i= nstall dpdk, dpdk-tools, and openvswitch. Regards, Mark Wittling NFV Cloud Operations Cox Communications Inc CTECH A08-150D 6305-A Peachtree Dunwoody Road, Atlanta GA 30328 1-770-849-9696