From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp-gw.pt.net (smtp-gw.pt.net [206.210.194.15]) by dpdk.org (Postfix) with ESMTP id 342621DA4 for ; Wed, 6 Mar 2019 22:01:46 +0100 (CET) X-ASG-Debug-ID: 1551906104-09411a1181e6fa0001-TfluYd Received: from mail.pt.net (mail.pt.net [206.210.194.11]) by smtp-gw.pt.net with ESMTP id nQbWWTkt8lq9brX8 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Wed, 06 Mar 2019 15:01:44 -0600 (CST) X-Barracuda-Envelope-From: mit@perftech.com X-Barracuda-Effective-Source-IP: mail.pt.net[206.210.194.11] X-Barracuda-Apparent-Source-IP: 206.210.194.11 Received: from localhost (localhost [IPv6:::1]) by mail.pt.net (Postfix) with ESMTP id 08CA884278A; Wed, 6 Mar 2019 15:01:44 -0600 (CST) Received: from mail.pt.net ([IPv6:::1]) by localhost (mail.pt.net [IPv6:::1]) (amavisd-new, port 10032) with ESMTP id 1R3djbXkVuqU; Wed, 6 Mar 2019 15:01:39 -0600 (CST) Received: from localhost (localhost [IPv6:::1]) by mail.pt.net (Postfix) with ESMTP id B6052842785; Wed, 6 Mar 2019 15:01:39 -0600 (CST) X-Virus-Scanned: amavisd-new at pt.net Received: from mail.pt.net ([IPv6:::1]) by localhost (mail.pt.net [IPv6:::1]) (amavisd-new, port 10026) with ESMTP id eW547T03EGCT; Wed, 6 Mar 2019 15:01:39 -0600 (CST) Received: from mail.pt.net (mail.pt.net [206.210.194.11]) by mail.pt.net (Postfix) with ESMTP id 6BF778426F8; Wed, 6 Mar 2019 15:01:39 -0600 (CST) Date: Wed, 6 Mar 2019 15:01:39 -0600 (CST) From: Mit Matelske To: shahafs Cc: stephen , Thomas Monjalon , dev , Yongseok Koh Message-ID: <1452194944.4333375.1551906099316.JavaMail.zimbra@perftech.com> In-Reply-To: References: <1203238912.103536.1542300483728.JavaMail.zimbra@perftech.com> <20181119090810.02543f44@xeon-e3> <1145619802.1261333.1542820311675.JavaMail.zimbra@perftech.com> <20181121121204.297ef328@xeon-e3> <832315875.3148291.1551455882901.JavaMail.zimbra@perftech.com> <873778686.3682315.1551717896130.JavaMail.zimbra@perftech.com> MIME-Version: 1.0 X-ASG-Orig-Subj: Re: [dpdk-dev] mlx5 under FreeBSD Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Mailer: Zimbra 8.8.10_GA_3758 (ZimbraWebClient - GC72 (Linux)/8.8.10_GA_3745) Thread-Topic: mlx5 under FreeBSD Thread-Index: AQHUgBne83qEGdtAiUCv0MW4WOiaYKVXVLEAgAMlsYCAADJaAICc4ikAgAKrAAD88E9MG4MU0BngVOKWRs8= X-Barracuda-Connect: mail.pt.net[206.210.194.11] X-Barracuda-Start-Time: 1551906104 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://smtp-gw.pt.net:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at pt.net X-Barracuda-Scan-Msg-Size: 7416 X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using global scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=9.0 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.68292 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Subject: Re: [dpdk-dev] mlx5 under FreeBSD X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 06 Mar 2019 21:01:46 -0000 > > > > dev.mlx5_core.1.hw.board_id: LNR3270110033 > > dev.mlx5_core.1.hw.fw_version: 12.17.2020 > > Getting back to you on this one. > > From the FW it looks like you are trying to create a TIR (QP) which doesn't use Toeplitz hash function (IBV_RX_HASH_FUNC_TOEPLITZ) but the rss key is not reserved. > > Makes sense? Kinda, except I believe the DPDK code is trying to create a TIR with IBV_RX_HASH_FUNC_TOEPLITZ set and the RSS key defined: qp = mlx5_glue->create_qp_ex(priv->ctx, &(struct ibv_qp_init_attr_ex){ .qp_type = IBV_QPT_RAW_PACKET, .comp_mask = IBV_QP_INIT_ATTR_PD | IBV_QP_INIT_ATTR_IND_TABLE | IBV_QP_INIT_ATTR_RX_HASH, .rx_hash_conf = (struct ibv_rx_hash_conf){ .rx_hash_function = IBV_RX_HASH_FUNC_TOEPLITZ, .rx_hash_key_len = MLX5_RSS_HASH_KEY_LEN, .rx_hash_key = rss_hash_default_key, .rx_hash_fields_mask = 0, }, .rwq_ind_tbl = ind_tbl->ind_table, .pd = priv->pd }); That call goes into create_qp in libmlx5 which eventually makes the following call: ret = ibv_cmd_create_qp_ex2(context, &qp->verbs_qp, sizeof(qp->verbs_qp), attr, &cmd_ex_rss.ibv_cmd, sizeof(cmd_ex_rss.ibv_cmd), sizeof(cmd_ex_rss), &resp.ibv_resp, sizeof(resp.ibv_resp), sizeof(resp)); Relevant variables from that call: cmd_ex_rss.rx_hash_function: 1 cmd_ex_rss.rx_key_len: 40 cmd_ex_rss.rx_hash_key: 2C C6 81 D1 5B DB F4 F7 FC A2 83 19 DB 1A 3E 94 6B 9E 38 D9 2C 9C 03 D1 AD 99 44 A7 D9 56 3D 59 06 3C 25 F3 FC 1F DC 2A All this is "stock" FB 12.0 and DPDK 18.08 code, FWIW. Any and all help is appreciated. I'm a little out of my depths with this. Thanks! Mit Matelske > > Thanks! > > Mit Matelske > > ----- Original Message ----- > From: "shahafs" > To: "Mit Matelske" , "stephen" > > Cc: "Thomas Monjalon" , "dev" , > "Yongseok Koh" > Sent: Sunday, March 3, 2019 2:43:41 AM > Subject: RE: [dpdk-dev] mlx5 under FreeBSD > > Friday, March 1, 2019 5:58 PM, Mit Matelske: > > Subject: Re: [dpdk-dev] mlx5 under FreeBSD > > > > I was wondering if I could lean on the experts to help me out with an issue? > > I'm finally testing this port but don't understand why the following > > error is > > happening: > > > > net_mlx5: mlx5_rxq.c:2151: mlx5_hrxq_drop_new(): port 0 cannot > > allocate QP for drop queue > > > > The driver is giving the following error corresponding with that DPDK > > message: > > > > Feb 28 15:00:44 fb100g1 kernel: mlx5_core0: ERR: > > mlx5_cmd_check:701:(pid > > 6819): CREATE_TIR(0x900) op_mod(0x0) failed, status bad > > parameter(0x3), syndrome (0x569ee6) > > Hard to say why. I tried to look for the syndrome above on FW but couldn't > find. Are you sure this is correct one? > Which FW version you use? > > > > > The mlx5_core_create_tir function completes successfully for the en > > driver, just not the ib driver when trying to build the hash tables. Any > ideas? > > > > I've instrumented libmlx5.so.1, mlx5.ko and mlx5ib.ko to debug this > > and everything looks fine according to my limited knowledge. Not sure > > why the card is balking on this action. > > > > Thanks! > > > > Mit Matelske > > > > ----- Original Message ----- > > From: "stephen" > > To: "Mit Matelske" > > Cc: "shahafs" , "Thomas Monjalon" > > , "dev" , "Yongseok Koh" > > > > Sent: Wednesday, November 21, 2018 2:12:04 PM > > Subject: Re: [dpdk-dev] mlx5 under FreeBSD > > > > On Wed, 21 Nov 2018 11:11:51 -0600 (CST) Mit Matelske > > wrote: > > > > > > > > > > Rather than building a complex shim, why not just have an OS > > > > dependent section of the driver. The bigger issue is that the > > > > kernel side functionality in BSD is different or missing. For > > > > example TAP > > API is different and not multi-queue. > > > > > > Once I started working on it, I had to go this route. I haven't > > > been able to but much time into it yet, but it is coming along: > > > > > > EAL: PCI device 0000:b3:00.0 on NUMA socket 0 > > > EAL: probe driver: 15b3:1013 net_mlx5 > > > net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_0" > > > uverbs0 /sys/class/infiniband_verbs/uverbs0 > > > /sys/class/infiniband/mlx5_0 > > > net_mlx5: mlx5.c:1325: mlx5_pci_probe(): PCI information matches for > > device "mlx5_0" > > > net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_1" > > > uverbs1 /sys/class/infiniband_verbs/uverbs1 > > > /sys/class/infiniband/mlx5_1 > > > net_mlx5: mlx5.c:1382: mlx5_pci_probe(): no switch support detected > > > net_mlx5: mlx5.c:796: mlx5_dev_spawn(): MPW isn't supported > > > net_mlx5: mlx5.c:852: mlx5_dev_spawn(): tunnel offloading disabled > > > due to old OFED/rdma-core version > > > net_mlx5: mlx5.c:864: mlx5_dev_spawn(): MPLS over GRE/UDP tunnel > > > offloading disabled due to old OFED/rdma-core version or firmware > > > configuration > > > net_mlx5: mlx5.c:877: mlx5_dev_spawn(): naming Ethernet device > > "0000:b3:00.0" > > > net_mlx5: mlx5.c:1007: mlx5_dev_spawn(): checksum offloading is > > > supported > > > net_mlx5: mlx5.c:1024: mlx5_dev_spawn(): maximum Rx indirection > > > table size is 512 > > > net_mlx5: mlx5.c:1028: mlx5_dev_spawn(): VLAN stripping is not > > > supported > > > net_mlx5: mlx5.c:1032: mlx5_dev_spawn(): FCS stripping configuration > > > is not supported > > > net_mlx5: mlx5.c:1037: mlx5_dev_spawn(): hardware Rx end alignment > > > padding is not supported > > > net_mlx5: mlx5.c:1052: mlx5_dev_spawn(): MPS is disabled > > > net_mlx5: mlx5.c:628: mlx5_uar_init_primary(): port 0 reserved UAR > > > address space: 0x880000000 > > > net_mlx5: mlx5.c:1107: mlx5_dev_spawn(): port 0 MAC address is > > > 00:90:0b:78:2a:ea > > > net_mlx5: mlx5.c:1114: mlx5_dev_spawn(): port 0 ifname is "mce0" > > > net_mlx5: mlx5.c:1127: mlx5_dev_spawn(): port 0 MTU is 1500 > > > net_mlx5: mlx5.c:1142: mlx5_dev_spawn(): flow rules relying on > > > switch offloads will not be supported: cannot open libmnl socket: > > > Address family not supported by protocol family > > > net_mlx5: mlx5.c:1176: mlx5_dev_spawn(): port 0 forcing Ethernet > > > interface up > > > net_mlx5: mlx5_ethdev.c:735: mlx5_link_update_unlocked_gs(): port 0 > > > ioctl(SIOCGIFFLAGS) failed: No such file or directory > > > net_mlx5: mlx5_ethdev.c:664: mlx5_link_update_unlocked_gset(): port > > > 0 > > > ioctl(SIOCGIFFLAGS) failed: No such file or directory > > > net_mlx5: mlx5_rxq.c:2150: mlx5_hrxq_drop_new(): port 0 cannot > > > allocate QP for drop queue Segmentation fault (core dumped) > > > > > > Working through mlx5_ethdev now... > > > > > > Mit > > > > Also if you are targeting Azure or Hyper-V, then don't bother porting > > TAP/VDEV_NETVSC/Failsafe model because that is being replaced by > > netvsc native PMD. It would be easier to just get that working. > > It would mean writing a uio equivalent driver for FreeBSD.