From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR01-VE1-obe.outbound.protection.outlook.com (mail-eopbgr140057.outbound.protection.outlook.com [40.107.14.57]) by dpdk.org (Postfix) with ESMTP id 3D68B2C24 for ; Sun, 10 Mar 2019 06:57:13 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=jUH/qZKsf0JiooP2Wov31Udyg7AzfCerxjmQTIRAJIU=; b=NF+BjM1zCaUodnh0wmGZGhmFKv5oHIjdn39tx18RSoNBC+zQRebHvfU4MprHJJX2YL6Hv7bpIp3NtzqCYYtaK5RFTgzrsrPrUNEFPZkENc3+NKdllpOwtTjjSq2XJfSXQWYUQyemsdcv75ANABSlIBdFiXuzrzS0r6cMuVHB1x0= Received: from AM0PR0502MB3795.eurprd05.prod.outlook.com (52.133.45.150) by AM0PR0502MB4033.eurprd05.prod.outlook.com (52.133.34.154) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1686.18; Sun, 10 Mar 2019 05:57:11 +0000 Received: from AM0PR0502MB3795.eurprd05.prod.outlook.com ([fe80::84f3:7e92:7a51:1003]) by AM0PR0502MB3795.eurprd05.prod.outlook.com ([fe80::84f3:7e92:7a51:1003%2]) with mapi id 15.20.1686.021; Sun, 10 Mar 2019 05:57:11 +0000 From: Shahaf Shuler To: Mit Matelske CC: stephen , Thomas Monjalon , dev , Yongseok Koh Thread-Topic: mlx5 under FreeBSD Thread-Index: AQHU1F/SgD54hViq1kmIc/XC6UC7qKX/s/5g1OXRg7mrHt1zkA== Date: Sun, 10 Mar 2019 05:57:10 +0000 Message-ID: References: <1203238912.103536.1542300483728.JavaMail.zimbra@perftech.com> <20181121121204.297ef328@xeon-e3> <832315875.3148291.1551455882901.JavaMail.zimbra@perftech.com> <873778686.3682315.1551717896130.JavaMail.zimbra@perftech.com> <1452194944.4333375.1551906099316.JavaMail.zimbra@perftech.com> <1991170839.4488451.1551972473124.JavaMail.zimbra@perftech.com> In-Reply-To: <1991170839.4488451.1551972473124.JavaMail.zimbra@perftech.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=shahafs@mellanox.com; x-originating-ip: [31.154.10.105] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: c324548f-2167-401a-26e2-08d6a51d3cb7 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(5600127)(711020)(4605104)(4618075)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(2017052603328)(7153060)(7193020); SRVR:AM0PR0502MB4033; x-ms-traffictypediagnostic: AM0PR0502MB4033: x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr x-microsoft-exchange-diagnostics: =?us-ascii?Q?1; AM0PR0502MB4033; 23:wKKCEPS/S/m3KEqxXTLq/b2/mikccS02Nw7N24I?= =?us-ascii?Q?jsZT1Sj6elRAa8GchREVYf1mEuSvtFzgdXJg0ZdV6Y0XgkDeGaz/8SO/FNeH?= =?us-ascii?Q?CYb2NZTvjzWzCPm6BjkcmsGKVEPuFz4qauIDvnYcD/n9doQMI/XNatg6kM+k?= =?us-ascii?Q?gb4Xb/1lfSFUHmJt1gqTg5/9cW2e3JeeyoYFp4BlCb2eQLN6O/VZaZN5aaVb?= =?us-ascii?Q?iXQqPialUSGKrM6Vn7YT3nfqMkoGWlKwRyjTMDuTyFz+35vOwLtSckECEQdQ?= =?us-ascii?Q?peJAFvd8+JHZd8GD4F7PUK0BTFKG7o5MiAIvKWq/b70188fbfyjCGBW6ozk9?= =?us-ascii?Q?0ulHdxuA1N63SCPPFJIK8w/NhkclVNv3HH0D4X8OTjHuAQ1zV987bw7aDQ5r?= =?us-ascii?Q?lcXtehPV49ffSCY+MfnjwzSzSIBIZS0Y3fGU41uEEOYpeXuJWlqQfl3IVrlh?= =?us-ascii?Q?EWlE1jutj3KdRumeSwCheAENXR8FWlZLN4xZ8T+P3XappMSaCxORl4CrfFN9?= =?us-ascii?Q?ggXd2FzMtl2WbmkeyZnytBIUsv32llyMX6tNui9soxuAuD9UbUdM8mf6m6m2?= =?us-ascii?Q?LL0Vl9/cgKMefFuy8i4oeMwY3xV2q3e44r+5Q11T9QpzFAGC7mJQyShLMVRo?= =?us-ascii?Q?jeH49W6gJgRsu1h5QqyuLdE1G5svDAzTatEk+u+NQgDlmP8yH7pwEGCim/JL?= =?us-ascii?Q?aULhtMLPwWLVCDJlGvo5zt1eNTJFFYzhVAnea54Ep9fTdt8ICNhEyZya+7kd?= =?us-ascii?Q?zv0XZRn7uhttYDPLydPI2kMcZltdme9Iln84hBeQ999D3frzNKCW7UM+tcwJ?= =?us-ascii?Q?KwCWq7bWo/mtWkIEEJm/HyKoEy0Td4gdssC0JKYJ3W5ylK1YoF2yWRjUrvsX?= =?us-ascii?Q?dporJ1EHHDewNX2QJEVZr+NV2lh0eQX/KEi7G2XsdsgtrUx67Yq9ytyiO6U6?= =?us-ascii?Q?qqwjjhNu5e6jbM2ZY1NygaleqgtO5glG7TXBwxEEzU+CzgOCCjChWd1RI9OX?= =?us-ascii?Q?sz2hGgG7MxHRtbcl3n6ApNRXMB47z/T090r+MGCValgm0UISX3wJmSudAjgG?= =?us-ascii?Q?S99EM7TULdzefZmZCgLFBYJjq3A2MLnoZPrNzlUoiP2Un+79vXlEv4wc4Jzd?= =?us-ascii?Q?Z+A99MNKM1lFtuINw56/qqyUgp1DwSjFEZNunBRUtVU8s2UdHF6qNIlLWDmK?= =?us-ascii?Q?ruDrpsvONR7GC5Ks8hYCUHPdhWf2qk9azCAiHuSyT7CqaoeWV747yi2eCspW?= =?us-ascii?Q?R5rg9MYTBy5X5vK19eWxStcgWCKI4f0DDEX3lfTFCLsNBjlwZxeN4BF7eE0b?= =?us-ascii?Q?A714bGydYwyGD0jgZW9EhjCM=3D?= x-microsoft-antispam-prvs: x-forefront-prvs: 0972DEC1D9 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(346002)(376002)(366004)(39850400004)(136003)(396003)(13464003)(189003)(199004)(53936002)(55016002)(97736004)(33656002)(6436002)(5660300002)(8676002)(81156014)(81166006)(86362001)(229853002)(6116002)(3846002)(186003)(9686003)(11346002)(446003)(476003)(486006)(14444005)(26005)(66066001)(6246003)(256004)(6506007)(102836004)(316002)(93886005)(14454004)(68736007)(7116003)(74316002)(52536013)(478600001)(76176011)(54906003)(106356001)(7696005)(71200400001)(71190400001)(99286004)(105586002)(6916009)(2906002)(7736002)(305945005)(107886003)(25786009)(8936002)(4326008); DIR:OUT; SFP:1101; SCL:1; SRVR:AM0PR0502MB4033; H:AM0PR0502MB3795.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: y3G/L5R7R0UyHSU1E9Sgq1RJUfFUKzQfTyb8wO1hxzrU5YyaeQvYxcKE85fsaqngpyLVUuqn6xgcAnH+uRWfnFnJiEv5sbEJY57RFlOr+Ph9cXBrx5wpJbLmqjZgwVm1k9NRbCS/D6MEPNTIpAfW9/WZeUUtalg7+GAkagghp5FBhk4pCp/WPdOqY7zM5YFym2jXqoBcVAmi57a5v3RkX2HOJtlQVMbJxdk6W+iISG+MynaW8PRyecWZinYwoCH7bK0bX387yGaTd1jdMy3XEHkzWgOjJ3k0z/oRiRfAFlD2nWgXlporygsresHnJQpXwrgFQ3vP1dAT8Y5DzyftJ21hv6tVYakvmj9779efnrS8kITI7hZbUWSobnoGaQWTpsc33U6RYiMngqOqLX0TWzTRMPP3ojBVtcdryzrptCs= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: c324548f-2167-401a-26e2-08d6a51d3cb7 X-MS-Exchange-CrossTenant-originalarrivaltime: 10 Mar 2019 05:57:10.9323 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR0502MB4033 Subject: Re: [dpdk-dev] mlx5 under FreeBSD X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Sun, 10 Mar 2019 05:57:13 -0000 Thursday, March 7, 2019 5:28 PM, Mit Matelske: > Subject: Re: [dpdk-dev] mlx5 under FreeBSD >=20 >=20 > > > > ----- Original Message ----- > > From: "shahafs" > > To: "Mit Matelske" > > > > What you print here is the verbs layer, and looks OK. I am more interes= ted > on the values before the command reach the device. > > Can you print the values of: rx_hash_fn and rx_hash_toeplitz_key from t= he > CREATE_TIR input mailbox command? > > > > Relevant function on mlx5_ib kernel module are: mlx5_core_create_tir o= r > create_rss_raw_qp_tir. >=20 > In create_rss_raw_qp_tir, starting at line 1374 I added the following pri= ntfs: >=20 > switch (ucmd.rx_hash_function) { > case MLX5_RX_HASH_FUNC_TOEPLITZ: > { > void *rss_key =3D MLX5_ADDR_OF(tirc, tirc, rx_hash_toeplitz_key); > size_t len =3D MLX5_FLD_SZ_BYTES(tirc, rx_hash_toeplitz_key); >=20 > if (len !=3D ucmd.rx_key_len) { > err =3D -EINVAL; > goto err; > } >=20 > printf("setting MLX5_RX_HASH_FUNC_TOEPLITZ\n"); > MLX5_SET(tirc, tirc, rx_hash_fn, MLX5_RX_HASH_FUNC_TOEPLITZ); > MLX5_SET(tirc, tirc, rx_hash_symmetric, 1); > memcpy(rss_key, ucmd.rx_hash_key, len); > printf("rx_hash_toeplitz_key(%ld):\n", len); for(int ix =3D 0; ix < len; = ++ix) { > printf("%02X ", ((u_char *)rss_key)[ix]); > if (((ix + 1) % 10) =3D=3D 0) printf("\n"); } > break; > } > default: > err =3D -EOPNOTSUPP; > goto err; > } >=20 >=20 > Which resulted in: >=20 > Mar 7 09:25:56 fb100g1 kernel: setting MLX5_RX_HASH_FUNC_TOEPLITZ > Mar 7 09:25:56 fb100g1 kernel: rx_hash_toeplitz_key(40): > Mar 7 09:25:56 fb100g1 kernel: 2C C6 81 D1 5B DB F4 F7 FC A2 Mar 7 09:2= 5:56 > fb100g1 kernel: 83 19 DB 1A 3E 94 6B 9E 38 D9 Mar 7 09:25:56 fb100g1 ker= nel: > 2C 9C 03 D1 AD 99 44 A7 D9 56 Mar 7 09:25:56 fb100g1 kernel: 3D 59 06 3C= 25 > F3 FC 1F DC 2A >=20 > So, to me, everything looks good? Yes it looks good. Wired, because it doesn't match the syndrome you got fro= m the device. I don't have any more good ideas here.=20 Have you tried to compare the input parameters for this command between a w= orking version of Mellanox PMD over Linux and this version?=20 >=20 > Thanks again! >=20 > Mit Matelske >=20 >=20 >=20 > > > > All this is "stock" FB 12.0 and DPDK 18.08 code, FWIW. > > > > Any and all help is appreciated. I'm a little out of my depths with th= is. > > > > Thanks! > > > > Mit Matelske > > > > > > > > Thanks! > > > > > > Mit Matelske > > > > > > ----- Original Message ----- > > > From: "shahafs" > > > To: "Mit Matelske" , "stephen" > > > > > > Cc: "Thomas Monjalon" , "dev" > > , > > > "Yongseok Koh" > > > Sent: Sunday, March 3, 2019 2:43:41 AM > > > Subject: RE: [dpdk-dev] mlx5 under FreeBSD > > > > > > Friday, March 1, 2019 5:58 PM, Mit Matelske: > > > > Subject: Re: [dpdk-dev] mlx5 under FreeBSD > > > > > > > > I was wondering if I could lean on the experts to help me out with > > > > an > > issue? > > > > I'm finally testing this port but don't understand why the > > > > following error is > > > > happening: > > > > > > > > net_mlx5: mlx5_rxq.c:2151: mlx5_hrxq_drop_new(): port 0 cannot > > > > allocate QP for drop queue > > > > > > > > The driver is giving the following error corresponding with that > > > > DPDK > > > > message: > > > > > > > > Feb 28 15:00:44 fb100g1 kernel: mlx5_core0: ERR: > > > > mlx5_cmd_check:701:(pid > > > > 6819): CREATE_TIR(0x900) op_mod(0x0) failed, status bad > > > > parameter(0x3), syndrome (0x569ee6) > > > > > > Hard to say why. I tried to look for the syndrome above on FW but > > > couldn't find. Are you sure this is correct one? > > > Which FW version you use? > > > > > > > > > > > The mlx5_core_create_tir function completes successfully for the > > > > en driver, just not the ib driver when trying to build the hash tab= les. > > > > Any > > > ideas? > > > > > > > > I've instrumented libmlx5.so.1, mlx5.ko and mlx5ib.ko to debug > > > > this and everything looks fine according to my limited knowledge. > > > > Not sure why the card is balking on this action. > > > > > > > > Thanks! > > > > > > > > Mit Matelske > > > > > > > > ----- Original Message ----- > > > > From: "stephen" > > > > To: "Mit Matelske" > > > > Cc: "shahafs" , "Thomas Monjalon" > > > > , "dev" , "Yongseok Koh" > > > > > > > > Sent: Wednesday, November 21, 2018 2:12:04 PM > > > > Subject: Re: [dpdk-dev] mlx5 under FreeBSD > > > > > > > > On Wed, 21 Nov 2018 11:11:51 -0600 (CST) Mit Matelske > > > > wrote: > > > > > > > > > > > > > > > > Rather than building a complex shim, why not just have an OS > > > > > > dependent section of the driver. The bigger issue is that the > > > > > > kernel side functionality in BSD is different or missing. For > > > > > > example TAP > > > > API is different and not multi-queue. > > > > > > > > > > Once I started working on it, I had to go this route. I haven't > > > > > been able to but much time into it yet, but it is coming along: > > > > > > > > > > EAL: PCI device 0000:b3:00.0 on NUMA socket 0 > > > > > EAL: probe driver: 15b3:1013 net_mlx5 > > > > > net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_0" > > > > > uverbs0 /sys/class/infiniband_verbs/uverbs0 > > > > > /sys/class/infiniband/mlx5_0 > > > > > net_mlx5: mlx5.c:1325: mlx5_pci_probe(): PCI information matches > > > > > for > > > > device "mlx5_0" > > > > > net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_1" > > > > > uverbs1 /sys/class/infiniband_verbs/uverbs1 > > > > > /sys/class/infiniband/mlx5_1 > > > > > net_mlx5: mlx5.c:1382: mlx5_pci_probe(): no switch support > > > > > detected > > > > > net_mlx5: mlx5.c:796: mlx5_dev_spawn(): MPW isn't supported > > > > > net_mlx5: mlx5.c:852: mlx5_dev_spawn(): tunnel offloading > > > > > disabled due to old OFED/rdma-core version > > > > > net_mlx5: mlx5.c:864: mlx5_dev_spawn(): MPLS over GRE/UDP > tunnel > > > > > offloading disabled due to old OFED/rdma-core version or > > > > > firmware configuration > > > > > net_mlx5: mlx5.c:877: mlx5_dev_spawn(): naming Ethernet device > > > > "0000:b3:00.0" > > > > > net_mlx5: mlx5.c:1007: mlx5_dev_spawn(): checksum offloading is > > > > > supported > > > > > net_mlx5: mlx5.c:1024: mlx5_dev_spawn(): maximum Rx indirection > > > > > table size is 512 > > > > > net_mlx5: mlx5.c:1028: mlx5_dev_spawn(): VLAN stripping is not > > > > > supported > > > > > net_mlx5: mlx5.c:1032: mlx5_dev_spawn(): FCS stripping > > > > > configuration is not supported > > > > > net_mlx5: mlx5.c:1037: mlx5_dev_spawn(): hardware Rx end > > > > > alignment padding is not supported > > > > > net_mlx5: mlx5.c:1052: mlx5_dev_spawn(): MPS is disabled > > > > > net_mlx5: mlx5.c:628: mlx5_uar_init_primary(): port 0 reserved > > > > > UAR address space: 0x880000000 > > > > > net_mlx5: mlx5.c:1107: mlx5_dev_spawn(): port 0 MAC address is > > > > > 00:90:0b:78:2a:ea > > > > > net_mlx5: mlx5.c:1114: mlx5_dev_spawn(): port 0 ifname is "mce0" > > > > > net_mlx5: mlx5.c:1127: mlx5_dev_spawn(): port 0 MTU is 1500 > > > > > net_mlx5: mlx5.c:1142: mlx5_dev_spawn(): flow rules relying on > > > > > switch offloads will not be supported: cannot open libmnl socket: > > > > > Address family not supported by protocol family > > > > > net_mlx5: mlx5.c:1176: mlx5_dev_spawn(): port 0 forcing Ethernet > > > > > interface up > > > > > net_mlx5: mlx5_ethdev.c:735: mlx5_link_update_unlocked_gs(): > > > > > port > > > > > 0 > > > > > ioctl(SIOCGIFFLAGS) failed: No such file or directory > > > > > net_mlx5: mlx5_ethdev.c:664: mlx5_link_update_unlocked_gset(): > > > > > port > > > > > 0 > > > > > ioctl(SIOCGIFFLAGS) failed: No such file or directory > > > > > net_mlx5: mlx5_rxq.c:2150: mlx5_hrxq_drop_new(): port 0 cannot > > > > > allocate QP for drop queue Segmentation fault (core dumped) > > > > > > > > > > Working through mlx5_ethdev now... > > > > > > > > > > Mit > > > > > > > > Also if you are targeting Azure or Hyper-V, then don't bother > > > > porting TAP/VDEV_NETVSC/Failsafe model because that is being > > > > replaced by netvsc native PMD. It would be easier to just get > > > > that > > working. > > > > It would mean writing a uio equivalent driver for FreeBSD.