From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR03-AM5-obe.outbound.protection.outlook.com (mail-eopbgr30051.outbound.protection.outlook.com [40.107.3.51]) by dpdk.org (Postfix) with ESMTP id 2B5922BD5 for ; Thu, 7 Mar 2019 07:28:51 +0100 (CET) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=YU7wjr/etPzpySxUwvgriySCAy6/yDdI0iVUB3pqKpo=; b=Z/wjEBHHb8FiRvn7KeMkpN6hvR/cQbJJqpMF4aKGisluPRJfaHrwp8T3+O4KhfDnd422MIrXTv3pIptjCZkQjKIm9CafZ/q4MuuNpdrsC1QEo44EdaADD/5fCQhCfEiaDuji0NTM/oUBj7oQsqcwWyHDFrJHAJRy3fTM+x2hs9s= Received: from AM0PR0502MB3795.eurprd05.prod.outlook.com (52.133.45.150) by AM0PR0502MB4084.eurprd05.prod.outlook.com (52.133.33.158) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.1665.19; Thu, 7 Mar 2019 06:28:49 +0000 Received: from AM0PR0502MB3795.eurprd05.prod.outlook.com ([fe80::84f3:7e92:7a51:1003]) by AM0PR0502MB3795.eurprd05.prod.outlook.com ([fe80::84f3:7e92:7a51:1003%2]) with mapi id 15.20.1686.018; Thu, 7 Mar 2019 06:28:49 +0000 From: Shahaf Shuler To: Mit Matelske CC: stephen , Thomas Monjalon , dev , Yongseok Koh Thread-Topic: mlx5 under FreeBSD Thread-Index: AQHU1F/SgD54hViq1kmIc/XC6UC7qKX/s/5g Date: Thu, 7 Mar 2019 06:28:49 +0000 Message-ID: References: <1203238912.103536.1542300483728.JavaMail.zimbra@perftech.com> <20181119090810.02543f44@xeon-e3> <1145619802.1261333.1542820311675.JavaMail.zimbra@perftech.com> <20181121121204.297ef328@xeon-e3> <832315875.3148291.1551455882901.JavaMail.zimbra@perftech.com> <873778686.3682315.1551717896130.JavaMail.zimbra@perftech.com> <1452194944.4333375.1551906099316.JavaMail.zimbra@perftech.com> In-Reply-To: <1452194944.4333375.1551906099316.JavaMail.zimbra@perftech.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=shahafs@mellanox.com; x-originating-ip: [193.47.165.251] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: c3406d5b-d7bb-4f6d-c2c6-08d6a2c62908 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(5600127)(711020)(4605104)(4618075)(2017052603328)(7153060)(7193020); SRVR:AM0PR0502MB4084; x-ms-traffictypediagnostic: AM0PR0502MB4084: x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr x-microsoft-exchange-diagnostics: =?us-ascii?Q?1; AM0PR0502MB4084; 23:LpmIY0FAr7p7ft0a3FyWyj66AZqqW0LZH7I92M+?= =?us-ascii?Q?B2eXxCkm28UWzpc5EXjnK3MjeslTnNFEs57sG90R6vCnXr6UMRS9dD5Z46xt?= =?us-ascii?Q?TdKnR3HxNWNPS0K+3fn1qvQml+qi9nk1uGqUXGYt8Sc9wiX03yRV/14rn/L4?= =?us-ascii?Q?+UJR2+sMAvbsf9DYetjH5zyuxS4Q8BAXZnXo58FQb2LxZnrz3+TUDMMtjcpd?= =?us-ascii?Q?PqrxfXgbIQ5gUGnGJsgysnLWtCoL0BXLu7nToooK2BrXpg5YxtY1TrgC6o34?= =?us-ascii?Q?k3jrLIFCNbJTRbG+ruH/3Bvmo9nvwlNMMvtMoUmE88K6gz1Lk5i/mwWRLolv?= =?us-ascii?Q?CJL6D4AIC6x4qkx7ntx27yubHjyCvN4I+CvNpy6fYmE53EuM7m0Nwgev6QC0?= =?us-ascii?Q?Zc3xxGETrFe34tDJpEFSWadbMdheaLDLPaeMXrTSEz0/H7L93p6ggJ8tW0zq?= =?us-ascii?Q?9B8LKmc2DxmBM/jhAbHbINl+gZ7E5xd0/4wCH/4WzSfQ63IxE8JsjbS+r7GZ?= =?us-ascii?Q?58c0tV+0De3Q/XmDhBc5GrGRkCPHjSTgI7UOEr9AWRqlZQcoD0BVaMJWgXvz?= =?us-ascii?Q?3/O85XUtj/Wx4jsxQZAKRTT01q3FP5JFrOFJylGulBbCpksWaPS0oFzYV44p?= =?us-ascii?Q?ZLvvl/PFKeofLTMfYk4ICdv6oErb61DNaID4UB0bYfrJETLBLqiTYWmWbJM3?= =?us-ascii?Q?LTGX4TIn9rJxQ7UIqjUm6qFu4DGI4AD0z/1X8nPRlcsYyLTMZIUB0aPF1DSz?= =?us-ascii?Q?8SKhvG/dHiNSgca+zFV7moKE+AZ4aZBO51DCNkxkm29GeBt/rXWuAJmBMMEg?= =?us-ascii?Q?Fr0MBIQSeddN6wV87ny4SePEeWaXb40dOmjEO7JYCgnw1+G9ubi87jp1Euo4?= =?us-ascii?Q?/AuXo1SlSSXGRukw8xBPuTJsrQ1umkG+NeADaEfvM24JFVQoCfFoGhEBLDXD?= =?us-ascii?Q?nCKlpZEb3Xbk8DzDBs7wKlhyFSF62sGzXUeT8pBC4B5/i6Jrljy4bTVoxgJM?= =?us-ascii?Q?4A0U/03BUZAntSVFotfeoVyVYdpegeosUWeWdL18468BZDwLrqnYoFWJAmPi?= =?us-ascii?Q?Isw7rupVOjCwq1e9ajXrLcZN4PXety8IfD8x6l1bZeH+rZ9Xeoc+Monb8h9H?= =?us-ascii?Q?KwviDFVqRMT3mflUiMI2pksgdBMEcDWlyFNNBwPFif/C1IM0AeUtBEfMbWsl?= =?us-ascii?Q?jqRvcxt8ucncCNjaHKlUIHBZ0iwyPdfusm0sER02x1ZzXvU+3i0EEfhiuH3F?= =?us-ascii?Q?k10q1M4evjgIeJv1kPh/MYnAhLnn/Y3zSfmdFmv94KBNDo3nP2gck/WQAKR1?= =?us-ascii?Q?gPLvdV11fT5HxG+K26fXTngg=3D?= x-microsoft-antispam-prvs: x-forefront-prvs: 096943F07A x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(346002)(39860400002)(136003)(366004)(376002)(396003)(13464003)(199004)(189003)(102836004)(33656002)(7736002)(26005)(8936002)(71200400001)(71190400001)(305945005)(55016002)(9686003)(99286004)(256004)(14444005)(478600001)(93886005)(2906002)(86362001)(25786009)(316002)(54906003)(8676002)(81156014)(74316002)(97736004)(81166006)(66066001)(68736007)(53936002)(4326008)(186003)(6246003)(6916009)(486006)(105586002)(52536013)(6436002)(229853002)(446003)(106356001)(11346002)(476003)(7696005)(6116002)(6506007)(107886003)(7116003)(5660300002)(14454004)(76176011)(3846002); DIR:OUT; SFP:1101; SCL:1; SRVR:AM0PR0502MB4084; H:AM0PR0502MB3795.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: yxf8Cw8tUp6UNhSjQYtCHB/k5YDwp6VYH34fOyGanzlaifYaH1AUtkAGjmfyklSJ8z0qqixaLJSzxjZtVXXdI0cwVv5K92Ms8pYL27KvampZJKBi3qX0ACSJ1OcZlpFD5r+w9BeKRvsrMVSnmz+q5BeZbjfi12tmzII7RCK3KwXH54mUD1POOPQ4/bKSnhPn98cHTiPcYbPsHQrLmXMaiVaVl8UQrqVREADNOTiXp6F9z2iwmbZfsvNA356PTZhXtQpL0WE8gEnukyQ8QtVodXKOulBoMBNtnn4ZhQJowtxGSyonQBGg8lX5lUkt6NZdKf5ee7Qe77aL3G+neN8ZgktI9Zlov+tR3Ryr9bxjNEpcbZjef0aOXMJwEUZIKqrzp+OT/GLFzRo9lgm6VRDz/Te1xc95lU0JfcK36s1cPuk= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: c3406d5b-d7bb-4f6d-c2c6-08d6a2c62908 X-MS-Exchange-CrossTenant-originalarrivaltime: 07 Mar 2019 06:28:49.3737 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM0PR0502MB4084 Subject: Re: [dpdk-dev] mlx5 under FreeBSD X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Mar 2019 06:28:51 -0000 Wednesday, March 6, 2019 11:02 PM, Mit Matelske: > Subject: Re: [dpdk-dev] mlx5 under FreeBSD >=20 > > > > > > dev.mlx5_core.1.hw.board_id: LNR3270110033 > > > dev.mlx5_core.1.hw.fw_version: 12.17.2020 > > > > Getting back to you on this one. > > > > From the FW it looks like you are trying to create a TIR (QP) which doe= sn't > use Toeplitz hash function (IBV_RX_HASH_FUNC_TOEPLITZ) but the rss key > is not reserved. > > > > Makes sense? >=20 > Kinda, except I believe the DPDK code is trying to create a TIR with > IBV_RX_HASH_FUNC_TOEPLITZ set and the RSS key defined: >=20 >=20 > qp =3D mlx5_glue->create_qp_ex(priv->ctx, > &(struct ibv_qp_init_attr_ex){ > .qp_type =3D IBV_QPT_RAW_PACKET, > .comp_mask =3D > IBV_QP_INIT_ATTR_PD | > IBV_QP_INIT_ATTR_IND_TABLE | > IBV_QP_INIT_ATTR_RX_HASH, > .rx_hash_conf =3D (struct ibv_rx_hash_conf){ > .rx_hash_function =3D > IBV_RX_HASH_FUNC_TOEPLITZ, > .rx_hash_key_len =3D MLX5_RSS_HASH_KEY_LEN, > .rx_hash_key =3D rss_hash_default_key, > .rx_hash_fields_mask =3D 0, > }, > .rwq_ind_tbl =3D ind_tbl->ind_table, > .pd =3D priv->pd > }); >=20 >=20 > That call goes into create_qp in libmlx5 which eventually makes the follo= wing > call: >=20 >=20 > ret =3D ibv_cmd_create_qp_ex2(context, &qp->verbs_qp, > sizeof(qp->verbs_qp), attr, > &cmd_ex_rss.ibv_cmd, sizeof(cmd_ex_rss.ibv_cmd), > sizeof(cmd_ex_rss), &resp.ibv_resp, > sizeof(resp.ibv_resp), sizeof(resp)); >=20 > Relevant variables from that call: >=20 > cmd_ex_rss.rx_hash_function: 1 > cmd_ex_rss.rx_key_len: 40 > cmd_ex_rss.rx_hash_key: > 2C C6 81 D1 5B DB F4 F7 FC A2 > 83 19 DB 1A 3E 94 6B 9E 38 D9 > 2C 9C 03 D1 AD 99 44 A7 D9 56 > 3D 59 06 3C 25 F3 FC 1F DC 2A What you print here is the verbs layer, and looks OK. I am more interested = on the values before the command reach the device.=20 Can you print the values of: rx_hash_fn and rx_hash_toeplitz_key from the C= REATE_TIR input mailbox command? Relevant function on mlx5_ib kernel module are: mlx5_core_create_tir or cr= eate_rss_raw_qp_tir. >=20 > All this is "stock" FB 12.0 and DPDK 18.08 code, FWIW. >=20 > Any and all help is appreciated. I'm a little out of my depths with this= . >=20 > Thanks! >=20 > Mit Matelske >=20 > > > > Thanks! > > > > Mit Matelske > > > > ----- Original Message ----- > > From: "shahafs" > > To: "Mit Matelske" , "stephen" > > > > Cc: "Thomas Monjalon" , "dev" > , > > "Yongseok Koh" > > Sent: Sunday, March 3, 2019 2:43:41 AM > > Subject: RE: [dpdk-dev] mlx5 under FreeBSD > > > > Friday, March 1, 2019 5:58 PM, Mit Matelske: > > > Subject: Re: [dpdk-dev] mlx5 under FreeBSD > > > > > > I was wondering if I could lean on the experts to help me out with an > issue? > > > I'm finally testing this port but don't understand why the following > > > error is > > > happening: > > > > > > net_mlx5: mlx5_rxq.c:2151: mlx5_hrxq_drop_new(): port 0 cannot > > > allocate QP for drop queue > > > > > > The driver is giving the following error corresponding with that > > > DPDK > > > message: > > > > > > Feb 28 15:00:44 fb100g1 kernel: mlx5_core0: ERR: > > > mlx5_cmd_check:701:(pid > > > 6819): CREATE_TIR(0x900) op_mod(0x0) failed, status bad > > > parameter(0x3), syndrome (0x569ee6) > > > > Hard to say why. I tried to look for the syndrome above on FW but > > couldn't find. Are you sure this is correct one? > > Which FW version you use? > > > > > > > > The mlx5_core_create_tir function completes successfully for the en > > > driver, just not the ib driver when trying to build the hash tables. > > > Any > > ideas? > > > > > > I've instrumented libmlx5.so.1, mlx5.ko and mlx5ib.ko to debug this > > > and everything looks fine according to my limited knowledge. Not > > > sure why the card is balking on this action. > > > > > > Thanks! > > > > > > Mit Matelske > > > > > > ----- Original Message ----- > > > From: "stephen" > > > To: "Mit Matelske" > > > Cc: "shahafs" , "Thomas Monjalon" > > > , "dev" , "Yongseok Koh" > > > > > > Sent: Wednesday, November 21, 2018 2:12:04 PM > > > Subject: Re: [dpdk-dev] mlx5 under FreeBSD > > > > > > On Wed, 21 Nov 2018 11:11:51 -0600 (CST) Mit Matelske > > > wrote: > > > > > > > > > > > > > Rather than building a complex shim, why not just have an OS > > > > > dependent section of the driver. The bigger issue is that the > > > > > kernel side functionality in BSD is different or missing. For > > > > > example TAP > > > API is different and not multi-queue. > > > > > > > > Once I started working on it, I had to go this route. I haven't > > > > been able to but much time into it yet, but it is coming along: > > > > > > > > EAL: PCI device 0000:b3:00.0 on NUMA socket 0 > > > > EAL: probe driver: 15b3:1013 net_mlx5 > > > > net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_0" > > > > uverbs0 /sys/class/infiniband_verbs/uverbs0 > > > > /sys/class/infiniband/mlx5_0 > > > > net_mlx5: mlx5.c:1325: mlx5_pci_probe(): PCI information matches > > > > for > > > device "mlx5_0" > > > > net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_1" > > > > uverbs1 /sys/class/infiniband_verbs/uverbs1 > > > > /sys/class/infiniband/mlx5_1 > > > > net_mlx5: mlx5.c:1382: mlx5_pci_probe(): no switch support > > > > detected > > > > net_mlx5: mlx5.c:796: mlx5_dev_spawn(): MPW isn't supported > > > > net_mlx5: mlx5.c:852: mlx5_dev_spawn(): tunnel offloading disabled > > > > due to old OFED/rdma-core version > > > > net_mlx5: mlx5.c:864: mlx5_dev_spawn(): MPLS over GRE/UDP tunnel > > > > offloading disabled due to old OFED/rdma-core version or firmware > > > > configuration > > > > net_mlx5: mlx5.c:877: mlx5_dev_spawn(): naming Ethernet device > > > "0000:b3:00.0" > > > > net_mlx5: mlx5.c:1007: mlx5_dev_spawn(): checksum offloading is > > > > supported > > > > net_mlx5: mlx5.c:1024: mlx5_dev_spawn(): maximum Rx indirection > > > > table size is 512 > > > > net_mlx5: mlx5.c:1028: mlx5_dev_spawn(): VLAN stripping is not > > > > supported > > > > net_mlx5: mlx5.c:1032: mlx5_dev_spawn(): FCS stripping > > > > configuration is not supported > > > > net_mlx5: mlx5.c:1037: mlx5_dev_spawn(): hardware Rx end alignment > > > > padding is not supported > > > > net_mlx5: mlx5.c:1052: mlx5_dev_spawn(): MPS is disabled > > > > net_mlx5: mlx5.c:628: mlx5_uar_init_primary(): port 0 reserved UAR > > > > address space: 0x880000000 > > > > net_mlx5: mlx5.c:1107: mlx5_dev_spawn(): port 0 MAC address is > > > > 00:90:0b:78:2a:ea > > > > net_mlx5: mlx5.c:1114: mlx5_dev_spawn(): port 0 ifname is "mce0" > > > > net_mlx5: mlx5.c:1127: mlx5_dev_spawn(): port 0 MTU is 1500 > > > > net_mlx5: mlx5.c:1142: mlx5_dev_spawn(): flow rules relying on > > > > switch offloads will not be supported: cannot open libmnl socket: > > > > Address family not supported by protocol family > > > > net_mlx5: mlx5.c:1176: mlx5_dev_spawn(): port 0 forcing Ethernet > > > > interface up > > > > net_mlx5: mlx5_ethdev.c:735: mlx5_link_update_unlocked_gs(): port > > > > 0 > > > > ioctl(SIOCGIFFLAGS) failed: No such file or directory > > > > net_mlx5: mlx5_ethdev.c:664: mlx5_link_update_unlocked_gset(): > > > > port > > > > 0 > > > > ioctl(SIOCGIFFLAGS) failed: No such file or directory > > > > net_mlx5: mlx5_rxq.c:2150: mlx5_hrxq_drop_new(): port 0 cannot > > > > allocate QP for drop queue Segmentation fault (core dumped) > > > > > > > > Working through mlx5_ethdev now... > > > > > > > > Mit > > > > > > Also if you are targeting Azure or Hyper-V, then don't bother > > > porting TAP/VDEV_NETVSC/Failsafe model because that is being > > > replaced by netvsc native PMD. It would be easier to just get that > working. > > > It would mean writing a uio equivalent driver for FreeBSD.