From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from smtp-gw.pt.net (smtp-gw.pt.net [206.210.194.15]) by dpdk.org (Postfix) with ESMTP id 1405B4C8E for ; Mon, 11 Mar 2019 15:29:40 +0100 (CET) X-ASG-Debug-ID: 1552314579-09411a0e9b1aea0001-TfluYd Received: from mail.pt.net (mail.pt.net [206.210.194.11]) by smtp-gw.pt.net with ESMTP id c8Gofe6Zl5ihqJp8 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NO); Mon, 11 Mar 2019 09:29:39 -0500 (CDT) X-Barracuda-Envelope-From: mit@perftech.com X-Barracuda-Effective-Source-IP: mail.pt.net[206.210.194.11] X-Barracuda-Apparent-Source-IP: 206.210.194.11 Received: from localhost (localhost [IPv6:::1]) by mail.pt.net (Postfix) with ESMTP id 7EE0C841D85; Mon, 11 Mar 2019 09:29:39 -0500 (CDT) Received: from mail.pt.net ([IPv6:::1]) by localhost (mail.pt.net [IPv6:::1]) (amavisd-new, port 10032) with ESMTP id 9t-QmPRFvVWk; Mon, 11 Mar 2019 09:29:37 -0500 (CDT) Received: from localhost (localhost [IPv6:::1]) by mail.pt.net (Postfix) with ESMTP id D4C0784020D; Mon, 11 Mar 2019 09:29:37 -0500 (CDT) X-Virus-Scanned: amavisd-new at pt.net Received: from mail.pt.net ([IPv6:::1]) by localhost (mail.pt.net [IPv6:::1]) (amavisd-new, port 10026) with ESMTP id 5TFIvGzfmK8w; Mon, 11 Mar 2019 09:29:37 -0500 (CDT) Received: from mail.pt.net (mail.pt.net [206.210.194.11]) by mail.pt.net (Postfix) with ESMTP id B95F1841D85; Mon, 11 Mar 2019 09:29:37 -0500 (CDT) Date: Mon, 11 Mar 2019 09:29:37 -0500 (CDT) From: Mit Matelske To: shahafs Cc: stephen , Thomas Monjalon , dev , Yongseok Koh Message-ID: <223735957.115693.1552314577676.JavaMail.zimbra@perftech.com> In-Reply-To: References: <1203238912.103536.1542300483728.JavaMail.zimbra@perftech.com> <873778686.3682315.1551717896130.JavaMail.zimbra@perftech.com> <1452194944.4333375.1551906099316.JavaMail.zimbra@perftech.com> <1991170839.4488451.1551972473124.JavaMail.zimbra@perftech.com> MIME-Version: 1.0 X-ASG-Orig-Subj: Re: mlx5 under FreeBSD Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 7bit X-Mailer: Zimbra 8.8.11_GA_3780 (ZimbraWebClient - GC72 (Linux)/8.8.11_GA_3787) Thread-Topic: mlx5 under FreeBSD Thread-Index: AQHU1F/SgD54hViq1kmIc/XC6UC7qKX/s/5g1OXRg7mrHt1zkEqONnDf X-Barracuda-Connect: mail.pt.net[206.210.194.11] X-Barracuda-Start-Time: 1552314579 X-Barracuda-Encrypted: ECDHE-RSA-AES256-GCM-SHA384 X-Barracuda-URL: https://smtp-gw.pt.net:443/cgi-mod/mark.cgi X-Barracuda-BRTS-Status: 1 X-Virus-Scanned: by bsmtpd at pt.net X-Barracuda-Scan-Msg-Size: 9084 X-Barracuda-Spam-Score: 0.00 X-Barracuda-Spam-Status: No, SCORE=0.00 using global scores of TAG_LEVEL=1000.0 QUARANTINE_LEVEL=1000.0 KILL_LEVEL=9.0 tests= X-Barracuda-Spam-Report: Code version 3.2, rules version 3.2.3.68508 Rule breakdown below pts rule name description ---- ---------------------- -------------------------------------------------- Subject: Re: [dpdk-dev] mlx5 under FreeBSD X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 11 Mar 2019 14:29:41 -0000 I started spinning up a Linux box to do that exact test on Friday. I'll post the results later. Thanks! Mit Matelske ----- Original Message ----- From: "shahafs" To: "Mit Matelske" Cc: "stephen" , "Thomas Monjalon" , "dev" , "Yongseok Koh" Sent: Saturday, March 9, 2019 11:57:10 PM Subject: RE: mlx5 under FreeBSD Thursday, March 7, 2019 5:28 PM, Mit Matelske: > Subject: Re: [dpdk-dev] mlx5 under FreeBSD > > > > > > ----- Original Message ----- > > From: "shahafs" > > To: "Mit Matelske" > > > > What you print here is the verbs layer, and looks OK. I am more interested > on the values before the command reach the device. > > Can you print the values of: rx_hash_fn and rx_hash_toeplitz_key from the > CREATE_TIR input mailbox command? > > > > Relevant function on mlx5_ib kernel module are: mlx5_core_create_tir or > create_rss_raw_qp_tir. > > In create_rss_raw_qp_tir, starting at line 1374 I added the following printfs: > > switch (ucmd.rx_hash_function) { > case MLX5_RX_HASH_FUNC_TOEPLITZ: > { > void *rss_key = MLX5_ADDR_OF(tirc, tirc, rx_hash_toeplitz_key); > size_t len = MLX5_FLD_SZ_BYTES(tirc, rx_hash_toeplitz_key); > > if (len != ucmd.rx_key_len) { > err = -EINVAL; > goto err; > } > > printf("setting MLX5_RX_HASH_FUNC_TOEPLITZ\n"); > MLX5_SET(tirc, tirc, rx_hash_fn, MLX5_RX_HASH_FUNC_TOEPLITZ); > MLX5_SET(tirc, tirc, rx_hash_symmetric, 1); > memcpy(rss_key, ucmd.rx_hash_key, len); > printf("rx_hash_toeplitz_key(%ld):\n", len); for(int ix = 0; ix < len; ++ix) { > printf("%02X ", ((u_char *)rss_key)[ix]); > if (((ix + 1) % 10) == 0) printf("\n"); } > break; > } > default: > err = -EOPNOTSUPP; > goto err; > } > > > Which resulted in: > > Mar 7 09:25:56 fb100g1 kernel: setting MLX5_RX_HASH_FUNC_TOEPLITZ > Mar 7 09:25:56 fb100g1 kernel: rx_hash_toeplitz_key(40): > Mar 7 09:25:56 fb100g1 kernel: 2C C6 81 D1 5B DB F4 F7 FC A2 Mar 7 09:25:56 > fb100g1 kernel: 83 19 DB 1A 3E 94 6B 9E 38 D9 Mar 7 09:25:56 fb100g1 kernel: > 2C 9C 03 D1 AD 99 44 A7 D9 56 Mar 7 09:25:56 fb100g1 kernel: 3D 59 06 3C 25 > F3 FC 1F DC 2A > > So, to me, everything looks good? Yes it looks good. Wired, because it doesn't match the syndrome you got from the device. I don't have any more good ideas here. Have you tried to compare the input parameters for this command between a working version of Mellanox PMD over Linux and this version? > > Thanks again! > > Mit Matelske > > > > > > > All this is "stock" FB 12.0 and DPDK 18.08 code, FWIW. > > > > Any and all help is appreciated. I'm a little out of my depths with this. > > > > Thanks! > > > > Mit Matelske > > > > > > > > Thanks! > > > > > > Mit Matelske > > > > > > ----- Original Message ----- > > > From: "shahafs" > > > To: "Mit Matelske" , "stephen" > > > > > > Cc: "Thomas Monjalon" , "dev" > > , > > > "Yongseok Koh" > > > Sent: Sunday, March 3, 2019 2:43:41 AM > > > Subject: RE: [dpdk-dev] mlx5 under FreeBSD > > > > > > Friday, March 1, 2019 5:58 PM, Mit Matelske: > > > > Subject: Re: [dpdk-dev] mlx5 under FreeBSD > > > > > > > > I was wondering if I could lean on the experts to help me out with > > > > an > > issue? > > > > I'm finally testing this port but don't understand why the > > > > following error is > > > > happening: > > > > > > > > net_mlx5: mlx5_rxq.c:2151: mlx5_hrxq_drop_new(): port 0 cannot > > > > allocate QP for drop queue > > > > > > > > The driver is giving the following error corresponding with that > > > > DPDK > > > > message: > > > > > > > > Feb 28 15:00:44 fb100g1 kernel: mlx5_core0: ERR: > > > > mlx5_cmd_check:701:(pid > > > > 6819): CREATE_TIR(0x900) op_mod(0x0) failed, status bad > > > > parameter(0x3), syndrome (0x569ee6) > > > > > > Hard to say why. I tried to look for the syndrome above on FW but > > > couldn't find. Are you sure this is correct one? > > > Which FW version you use? > > > > > > > > > > > The mlx5_core_create_tir function completes successfully for the > > > > en driver, just not the ib driver when trying to build the hash tables. > > > > Any > > > ideas? > > > > > > > > I've instrumented libmlx5.so.1, mlx5.ko and mlx5ib.ko to debug > > > > this and everything looks fine according to my limited knowledge. > > > > Not sure why the card is balking on this action. > > > > > > > > Thanks! > > > > > > > > Mit Matelske > > > > > > > > ----- Original Message ----- > > > > From: "stephen" > > > > To: "Mit Matelske" > > > > Cc: "shahafs" , "Thomas Monjalon" > > > > , "dev" , "Yongseok Koh" > > > > > > > > Sent: Wednesday, November 21, 2018 2:12:04 PM > > > > Subject: Re: [dpdk-dev] mlx5 under FreeBSD > > > > > > > > On Wed, 21 Nov 2018 11:11:51 -0600 (CST) Mit Matelske > > > > wrote: > > > > > > > > > > > > > > > > Rather than building a complex shim, why not just have an OS > > > > > > dependent section of the driver. The bigger issue is that the > > > > > > kernel side functionality in BSD is different or missing. For > > > > > > example TAP > > > > API is different and not multi-queue. > > > > > > > > > > Once I started working on it, I had to go this route. I haven't > > > > > been able to but much time into it yet, but it is coming along: > > > > > > > > > > EAL: PCI device 0000:b3:00.0 on NUMA socket 0 > > > > > EAL: probe driver: 15b3:1013 net_mlx5 > > > > > net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_0" > > > > > uverbs0 /sys/class/infiniband_verbs/uverbs0 > > > > > /sys/class/infiniband/mlx5_0 > > > > > net_mlx5: mlx5.c:1325: mlx5_pci_probe(): PCI information matches > > > > > for > > > > device "mlx5_0" > > > > > net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_1" > > > > > uverbs1 /sys/class/infiniband_verbs/uverbs1 > > > > > /sys/class/infiniband/mlx5_1 > > > > > net_mlx5: mlx5.c:1382: mlx5_pci_probe(): no switch support > > > > > detected > > > > > net_mlx5: mlx5.c:796: mlx5_dev_spawn(): MPW isn't supported > > > > > net_mlx5: mlx5.c:852: mlx5_dev_spawn(): tunnel offloading > > > > > disabled due to old OFED/rdma-core version > > > > > net_mlx5: mlx5.c:864: mlx5_dev_spawn(): MPLS over GRE/UDP > tunnel > > > > > offloading disabled due to old OFED/rdma-core version or > > > > > firmware configuration > > > > > net_mlx5: mlx5.c:877: mlx5_dev_spawn(): naming Ethernet device > > > > "0000:b3:00.0" > > > > > net_mlx5: mlx5.c:1007: mlx5_dev_spawn(): checksum offloading is > > > > > supported > > > > > net_mlx5: mlx5.c:1024: mlx5_dev_spawn(): maximum Rx indirection > > > > > table size is 512 > > > > > net_mlx5: mlx5.c:1028: mlx5_dev_spawn(): VLAN stripping is not > > > > > supported > > > > > net_mlx5: mlx5.c:1032: mlx5_dev_spawn(): FCS stripping > > > > > configuration is not supported > > > > > net_mlx5: mlx5.c:1037: mlx5_dev_spawn(): hardware Rx end > > > > > alignment padding is not supported > > > > > net_mlx5: mlx5.c:1052: mlx5_dev_spawn(): MPS is disabled > > > > > net_mlx5: mlx5.c:628: mlx5_uar_init_primary(): port 0 reserved > > > > > UAR address space: 0x880000000 > > > > > net_mlx5: mlx5.c:1107: mlx5_dev_spawn(): port 0 MAC address is > > > > > 00:90:0b:78:2a:ea > > > > > net_mlx5: mlx5.c:1114: mlx5_dev_spawn(): port 0 ifname is "mce0" > > > > > net_mlx5: mlx5.c:1127: mlx5_dev_spawn(): port 0 MTU is 1500 > > > > > net_mlx5: mlx5.c:1142: mlx5_dev_spawn(): flow rules relying on > > > > > switch offloads will not be supported: cannot open libmnl socket: > > > > > Address family not supported by protocol family > > > > > net_mlx5: mlx5.c:1176: mlx5_dev_spawn(): port 0 forcing Ethernet > > > > > interface up > > > > > net_mlx5: mlx5_ethdev.c:735: mlx5_link_update_unlocked_gs(): > > > > > port > > > > > 0 > > > > > ioctl(SIOCGIFFLAGS) failed: No such file or directory > > > > > net_mlx5: mlx5_ethdev.c:664: mlx5_link_update_unlocked_gset(): > > > > > port > > > > > 0 > > > > > ioctl(SIOCGIFFLAGS) failed: No such file or directory > > > > > net_mlx5: mlx5_rxq.c:2150: mlx5_hrxq_drop_new(): port 0 cannot > > > > > allocate QP for drop queue Segmentation fault (core dumped) > > > > > > > > > > Working through mlx5_ethdev now... > > > > > > > > > > Mit > > > > > > > > Also if you are targeting Azure or Hyper-V, then don't bother > > > > porting TAP/VDEV_NETVSC/Failsafe model because that is being > > > > replaced by netvsc native PMD. It would be easier to just get > > > > that > > working. > > > > It would mean writing a uio equivalent driver for FreeBSD.