From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp-gw.pt.net (smtp-gw.pt.net [206.210.194.15]) by dpdk.org (Postfix) with ESMTP id 856BA1B11F for ; Thu, 7 Mar 2019 16:27:55 +0100 (CET) X-ASG-Debug-ID: 1551972474-09411a11811050e0001-TfluYd Received: from mail.pt.net (mail.pt.net [206.210.194.11]) by smtp-gw.pt.net with ESMTP id A8EHWjZQpcAjCpDG (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Thu, 07 Mar 2019 09:27:54 -0600 (CST) X-Barracuda-Envelope-From: mit@perftech.com X-Barracuda-Effective-Source-IP: mail.pt.net[206.210.194.11] X-Barracuda-Apparent-Source-IP: 206.210.194.11 Received: from localhost (localhost [IPv6:::1]) by mail.pt.net (Postfix) with ESMTP id EECDD8427A1; Thu, 7 Mar 2019 09:27:53 -0600 (CST) Received: from mail.pt.net ([IPv6:::1]) by localhost (mail.pt.net [IPv6:::1]) (amavisd-new, port 10032) with ESMTP id PVyzoQNK7JLK; Thu, 7 Mar 2019 09:27:53 -0600 (CST) Received: from localhost (localhost [IPv6:::1]) by mail.pt.net (Postfix) with ESMTP id 7E2618427A0; Thu, 7 Mar 2019 09:27:53 -0600 (CST) X-Virus-Scanned: amavisd-new at pt.net Received: from mail.pt.net ([IPv6:::1]) by localhost (mail.pt.net [IPv6:::1]) (amavisd-new, port 10026) with ESMTP id 81GN-7wioIWI; Thu, 7 Mar 2019 09:27:53 -0600 (CST) Received: from mail.pt.net (mail.pt.net [206.210.194.11]) by mail.pt.net (Postfix) with ESMTP id 3839884279F; Thu, 7 Mar 2019 09:27:53 -0600 (CST) Date: Thu, 7 Mar 2019 09:27:53 -0600 (CST) From: Mit Matelske To: shahafs Cc: stephen , Thomas Monjalon , dev , Yongseok Koh Message-ID: <1991170839.4488451.1551972473124.JavaMail.zimbra@perftech.com> In-Reply-To: References: <1203238912.103536.1542300483728.JavaMail.zimbra@perftech.com> <20181121121204.297ef328@xeon-e3> <832315875.3148291.1551455882901.JavaMail.zimbra@perftech.com> <873778686.3682315.1551717896130.JavaMail.zimbra@perftech.com> <1452194944.4333375.1551906099316.JavaMail.zimbra@perftech.com> MIME-Version: 1.0 X-ASG-Orig-Subj: Re: mlx5 under FreeBSD Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Mailer: Zimbra 8.8.10_GA_3758 (ZimbraWebClient - GC72 (Linux)/8.8.10_GA_3745) Thread-Topic: mlx5 under FreeBSD Thread-Index: AQHU1F/SgD54hViq1kmIc/XC6UC7qKX/s/5g1OXRg7k= X-Barracuda-Connect: mail.pt.net[206.210.194.11] X-Barracuda-Start-Time: 1551972474 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://smtp-gw.pt.net:443/cgi-mod/mark.cgi X-Virus-Scanned: by bsmtpd at pt.net X-Barracuda-Scan-Msg-Size: 7831 X-Barracuda-BRTS-Status: 1 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using global scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=9.0 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.68328 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Subject: Re: [dpdk-dev] mlx5 under FreeBSD X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 07 Mar 2019 15:27:55 -0000 > > ----- Original Message ----- > From: "shahafs" > To: "Mit Matelske" > > What you print here is the verbs layer, and looks OK. I am more interested on the values before the command reach the device. > Can you print the values of: rx_hash_fn and rx_hash_toeplitz_key from the CREATE_TIR input mailbox command? > > Relevant function on mlx5_ib kernel module are: mlx5_core_create_tir or create_rss_raw_qp_tir. In create_rss_raw_qp_tir, starting at line 1374 I added the following printfs: switch (ucmd.rx_hash_function) { case MLX5_RX_HASH_FUNC_TOEPLITZ: { void *rss_key = MLX5_ADDR_OF(tirc, tirc, rx_hash_toeplitz_key); size_t len = MLX5_FLD_SZ_BYTES(tirc, rx_hash_toeplitz_key); if (len != ucmd.rx_key_len) { err = -EINVAL; goto err; } printf("setting MLX5_RX_HASH_FUNC_TOEPLITZ\n"); MLX5_SET(tirc, tirc, rx_hash_fn, MLX5_RX_HASH_FUNC_TOEPLITZ); MLX5_SET(tirc, tirc, rx_hash_symmetric, 1); memcpy(rss_key, ucmd.rx_hash_key, len); printf("rx_hash_toeplitz_key(%ld):\n", len); for(int ix = 0; ix < len; ++ix) { printf("%02X ", ((u_char *)rss_key)[ix]); if (((ix + 1) % 10) == 0) printf("\n"); } break; } default: err = -EOPNOTSUPP; goto err; } Which resulted in: Mar 7 09:25:56 fb100g1 kernel: setting MLX5_RX_HASH_FUNC_TOEPLITZ Mar 7 09:25:56 fb100g1 kernel: rx_hash_toeplitz_key(40): Mar 7 09:25:56 fb100g1 kernel: 2C C6 81 D1 5B DB F4 F7 FC A2 Mar 7 09:25:56 fb100g1 kernel: 83 19 DB 1A 3E 94 6B 9E 38 D9 Mar 7 09:25:56 fb100g1 kernel: 2C 9C 03 D1 AD 99 44 A7 D9 56 Mar 7 09:25:56 fb100g1 kernel: 3D 59 06 3C 25 F3 FC 1F DC 2A So, to me, everything looks good? Thanks again! Mit Matelske > > All this is "stock" FB 12.0 and DPDK 18.08 code, FWIW. > > Any and all help is appreciated. I'm a little out of my depths with this. > > Thanks! > > Mit Matelske > > > > > Thanks! > > > > Mit Matelske > > > > ----- Original Message ----- > > From: "shahafs" > > To: "Mit Matelske" , "stephen" > > > > Cc: "Thomas Monjalon" , "dev" > , > > "Yongseok Koh" > > Sent: Sunday, March 3, 2019 2:43:41 AM > > Subject: RE: [dpdk-dev] mlx5 under FreeBSD > > > > Friday, March 1, 2019 5:58 PM, Mit Matelske: > > > Subject: Re: [dpdk-dev] mlx5 under FreeBSD > > > > > > I was wondering if I could lean on the experts to help me out with an > issue? > > > I'm finally testing this port but don't understand why the following > > > error is > > > happening: > > > > > > net_mlx5: mlx5_rxq.c:2151: mlx5_hrxq_drop_new(): port 0 cannot > > > allocate QP for drop queue > > > > > > The driver is giving the following error corresponding with that > > > DPDK > > > message: > > > > > > Feb 28 15:00:44 fb100g1 kernel: mlx5_core0: ERR: > > > mlx5_cmd_check:701:(pid > > > 6819): CREATE_TIR(0x900) op_mod(0x0) failed, status bad > > > parameter(0x3), syndrome (0x569ee6) > > > > Hard to say why. I tried to look for the syndrome above on FW but > > couldn't find. Are you sure this is correct one? > > Which FW version you use? > > > > > > > > The mlx5_core_create_tir function completes successfully for the en > > > driver, just not the ib driver when trying to build the hash tables. > > > Any > > ideas? > > > > > > I've instrumented libmlx5.so.1, mlx5.ko and mlx5ib.ko to debug this > > > and everything looks fine according to my limited knowledge. Not > > > sure why the card is balking on this action. > > > > > > Thanks! > > > > > > Mit Matelske > > > > > > ----- Original Message ----- > > > From: "stephen" > > > To: "Mit Matelske" > > > Cc: "shahafs" , "Thomas Monjalon" > > > , "dev" , "Yongseok Koh" > > > > > > Sent: Wednesday, November 21, 2018 2:12:04 PM > > > Subject: Re: [dpdk-dev] mlx5 under FreeBSD > > > > > > On Wed, 21 Nov 2018 11:11:51 -0600 (CST) Mit Matelske > > > wrote: > > > > > > > > > > > > > Rather than building a complex shim, why not just have an OS > > > > > dependent section of the driver. The bigger issue is that the > > > > > kernel side functionality in BSD is different or missing. For > > > > > example TAP > > > API is different and not multi-queue. > > > > > > > > Once I started working on it, I had to go this route. I haven't > > > > been able to but much time into it yet, but it is coming along: > > > > > > > > EAL: PCI device 0000:b3:00.0 on NUMA socket 0 > > > > EAL: probe driver: 15b3:1013 net_mlx5 > > > > net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_0" > > > > uverbs0 /sys/class/infiniband_verbs/uverbs0 > > > > /sys/class/infiniband/mlx5_0 > > > > net_mlx5: mlx5.c:1325: mlx5_pci_probe(): PCI information matches > > > > for > > > device "mlx5_0" > > > > net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_1" > > > > uverbs1 /sys/class/infiniband_verbs/uverbs1 > > > > /sys/class/infiniband/mlx5_1 > > > > net_mlx5: mlx5.c:1382: mlx5_pci_probe(): no switch support > > > > detected > > > > net_mlx5: mlx5.c:796: mlx5_dev_spawn(): MPW isn't supported > > > > net_mlx5: mlx5.c:852: mlx5_dev_spawn(): tunnel offloading disabled > > > > due to old OFED/rdma-core version > > > > net_mlx5: mlx5.c:864: mlx5_dev_spawn(): MPLS over GRE/UDP tunnel > > > > offloading disabled due to old OFED/rdma-core version or firmware > > > > configuration > > > > net_mlx5: mlx5.c:877: mlx5_dev_spawn(): naming Ethernet device > > > "0000:b3:00.0" > > > > net_mlx5: mlx5.c:1007: mlx5_dev_spawn(): checksum offloading is > > > > supported > > > > net_mlx5: mlx5.c:1024: mlx5_dev_spawn(): maximum Rx indirection > > > > table size is 512 > > > > net_mlx5: mlx5.c:1028: mlx5_dev_spawn(): VLAN stripping is not > > > > supported > > > > net_mlx5: mlx5.c:1032: mlx5_dev_spawn(): FCS stripping > > > > configuration is not supported > > > > net_mlx5: mlx5.c:1037: mlx5_dev_spawn(): hardware Rx end alignment > > > > padding is not supported > > > > net_mlx5: mlx5.c:1052: mlx5_dev_spawn(): MPS is disabled > > > > net_mlx5: mlx5.c:628: mlx5_uar_init_primary(): port 0 reserved UAR > > > > address space: 0x880000000 > > > > net_mlx5: mlx5.c:1107: mlx5_dev_spawn(): port 0 MAC address is > > > > 00:90:0b:78:2a:ea > > > > net_mlx5: mlx5.c:1114: mlx5_dev_spawn(): port 0 ifname is "mce0" > > > > net_mlx5: mlx5.c:1127: mlx5_dev_spawn(): port 0 MTU is 1500 > > > > net_mlx5: mlx5.c:1142: mlx5_dev_spawn(): flow rules relying on > > > > switch offloads will not be supported: cannot open libmnl socket: > > > > Address family not supported by protocol family > > > > net_mlx5: mlx5.c:1176: mlx5_dev_spawn(): port 0 forcing Ethernet > > > > interface up > > > > net_mlx5: mlx5_ethdev.c:735: mlx5_link_update_unlocked_gs(): port > > > > 0 > > > > ioctl(SIOCGIFFLAGS) failed: No such file or directory > > > > net_mlx5: mlx5_ethdev.c:664: mlx5_link_update_unlocked_gset(): > > > > port > > > > 0 > > > > ioctl(SIOCGIFFLAGS) failed: No such file or directory > > > > net_mlx5: mlx5_rxq.c:2150: mlx5_hrxq_drop_new(): port 0 cannot > > > > allocate QP for drop queue Segmentation fault (core dumped) > > > > > > > > Working through mlx5_ethdev now... > > > > > > > > Mit > > > > > > Also if you are targeting Azure or Hyper-V, then don't bother > > > porting TAP/VDEV_NETVSC/Failsafe model because that is being > > > replaced by netvsc native PMD. It would be easier to just get that > working. > > > It would mean writing a uio equivalent driver for FreeBSD.