Re: [dpdk-dev] mlx5 under FreeBSD

[Mit Matelske <mit@pt.net>]

I started spinning up a Linux box to do that exact test on Friday.  I'll post the results later.

Thanks!

Mit Matelske

----- Original Message -----
From: "shahafs" <shahafs@mellanox.com>
To: "Mit Matelske" <mit@pt.net>
Cc: "stephen" <stephen@networkplumber.org>, "Thomas Monjalon" <thomas@monjalon.net>, "dev" <dev@dpdk.org>, "Yongseok Koh" <yskoh@mellanox.com>
Sent: Saturday, March 9, 2019 11:57:10 PM
Subject: RE: mlx5 under FreeBSD

Thursday, March 7, 2019 5:28 PM, Mit Matelske:
> Subject: Re: [dpdk-dev] mlx5 under FreeBSD
> 
> 
> >
> > ----- Original Message -----
> > From: "shahafs" <shahafs@mellanox.com>
> > To: "Mit Matelske" <mit@pt.net>
> >
> > What you print here is the verbs layer, and looks OK. I am more interested
> on the values before the command reach the device.
> > Can you print the values of: rx_hash_fn and rx_hash_toeplitz_key from the
> CREATE_TIR input mailbox command?
> >
> > Relevant function on mlx5_ib kernel module are:  mlx5_core_create_tir or
> create_rss_raw_qp_tir.
> 
> In create_rss_raw_qp_tir, starting at line 1374 I added the following printfs:
> 
>    switch (ucmd.rx_hash_function) {
>    case MLX5_RX_HASH_FUNC_TOEPLITZ:
>    {
>       void *rss_key = MLX5_ADDR_OF(tirc, tirc, rx_hash_toeplitz_key);
>       size_t len = MLX5_FLD_SZ_BYTES(tirc, rx_hash_toeplitz_key);
> 
>       if (len != ucmd.rx_key_len) {
>          err = -EINVAL;
>          goto err;
>       }
> 
> printf("setting MLX5_RX_HASH_FUNC_TOEPLITZ\n");
>       MLX5_SET(tirc, tirc, rx_hash_fn, MLX5_RX_HASH_FUNC_TOEPLITZ);
>       MLX5_SET(tirc, tirc, rx_hash_symmetric, 1);
>       memcpy(rss_key, ucmd.rx_hash_key, len);
> printf("rx_hash_toeplitz_key(%ld):\n", len); for(int ix = 0; ix < len; ++ix) {
>    printf("%02X ", ((u_char *)rss_key)[ix]);
>    if (((ix + 1) % 10) == 0) printf("\n"); }
>       break;
>    }
>    default:
>       err = -EOPNOTSUPP;
>       goto err;
>    }
> 
> 
> Which resulted in:
> 
> Mar  7 09:25:56 fb100g1 kernel: setting MLX5_RX_HASH_FUNC_TOEPLITZ
> Mar  7 09:25:56 fb100g1 kernel: rx_hash_toeplitz_key(40):
> Mar  7 09:25:56 fb100g1 kernel: 2C C6 81 D1 5B DB F4 F7 FC A2 Mar  7 09:25:56
> fb100g1 kernel: 83 19 DB 1A 3E 94 6B 9E 38 D9 Mar  7 09:25:56 fb100g1 kernel:
> 2C 9C 03 D1 AD 99 44 A7 D9 56 Mar  7 09:25:56 fb100g1 kernel: 3D 59 06 3C 25
> F3 FC 1F DC 2A
> 
> So, to me, everything looks good?

Yes it looks good. Wired, because it doesn't match the syndrome you got from the device.
I don't have any more good ideas here. 
Have you tried to compare the input parameters for this command between a working version of Mellanox PMD over Linux and this version? 

> 
> Thanks again!
> 
> Mit Matelske
> 
> 
> 
> >
> > All this is "stock" FB 12.0 and DPDK 18.08 code, FWIW.
> >
> > Any and all help is appreciated.  I'm a little out of my depths with this.
> >
> > Thanks!
> >
> > Mit Matelske
> >
> > >
> > > Thanks!
> > >
> > > Mit Matelske
> > >
> > > ----- Original Message -----
> > > From: "shahafs" <shahafs@mellanox.com>
> > > To: "Mit Matelske" <mit@pt.net>, "stephen"
> > > <stephen@networkplumber.org>
> > > Cc: "Thomas Monjalon" <thomas@monjalon.net>, "dev"
> > <dev@dpdk.org>,
> > > "Yongseok Koh" <yskoh@mellanox.com>
> > > Sent: Sunday, March 3, 2019 2:43:41 AM
> > > Subject: RE: [dpdk-dev] mlx5 under FreeBSD
> > >
> > > Friday, March 1, 2019 5:58 PM, Mit Matelske:
> > > > Subject: Re: [dpdk-dev] mlx5 under FreeBSD
> > > >
> > > > I was wondering if I could lean on the experts to help me out with
> > > > an
> > issue?
> > > > I'm finally testing this port but don't understand why the
> > > > following error is
> > > > happening:
> > > >
> > > > net_mlx5: mlx5_rxq.c:2151: mlx5_hrxq_drop_new(): port 0 cannot
> > > > allocate QP for drop queue
> > > >
> > > > The driver is giving the following error corresponding with that
> > > > DPDK
> > > > message:
> > > >
> > > > Feb 28 15:00:44 fb100g1 kernel: mlx5_core0: ERR:
> > > > mlx5_cmd_check:701:(pid
> > > > 6819): CREATE_TIR(0x900) op_mod(0x0) failed, status bad
> > > > parameter(0x3), syndrome (0x569ee6)
> > >
> > > Hard to say why. I tried to look for the syndrome above on FW but
> > > couldn't find. Are you sure this is correct one?
> > > Which FW version you use?
> > >
> > > >
> > > > The mlx5_core_create_tir function completes successfully for the
> > > > en driver, just not the ib driver when trying to build the hash tables.
> > > > Any
> > > ideas?
> > > >
> > > > I've instrumented libmlx5.so.1, mlx5.ko and mlx5ib.ko to debug
> > > > this and everything looks fine according to my limited knowledge.
> > > > Not sure why the card is balking on this action.
> > > >
> > > > Thanks!
> > > >
> > > > Mit Matelske
> > > >
> > > > ----- Original Message -----
> > > > From: "stephen" <stephen@networkplumber.org>
> > > > To: "Mit Matelske" <mit@pt.net>
> > > > Cc: "shahafs" <shahafs@mellanox.com>, "Thomas Monjalon"
> > > > <thomas@monjalon.net>, "dev" <dev@dpdk.org>, "Yongseok Koh"
> > > > <yskoh@mellanox.com>
> > > > Sent: Wednesday, November 21, 2018 2:12:04 PM
> > > > Subject: Re: [dpdk-dev] mlx5 under FreeBSD
> > > >
> > > > On Wed, 21 Nov 2018 11:11:51 -0600 (CST) Mit Matelske <mit@pt.net>
> > > > wrote:
> > > >
> > > > > >
> > > > > > Rather than building a complex shim, why not just have an OS
> > > > > > dependent section of the driver.  The bigger issue is that the
> > > > > > kernel side functionality in BSD is different or missing.  For
> > > > > > example TAP
> > > > API is different and not multi-queue.
> > > > >
> > > > > Once I started working on it, I had to go this route.  I haven't
> > > > > been able to but much time into it yet, but it is coming along:
> > > > >
> > > > > EAL: PCI device 0000:b3:00.0 on NUMA socket 0
> > > > > EAL:   probe driver: 15b3:1013 net_mlx5
> > > > > net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_0"
> > > > > uverbs0 /sys/class/infiniband_verbs/uverbs0
> > > > > /sys/class/infiniband/mlx5_0
> > > > > net_mlx5: mlx5.c:1325: mlx5_pci_probe(): PCI information matches
> > > > > for
> > > > device "mlx5_0"
> > > > > net_mlx5: mlx5.c:1315: mlx5_pci_probe(): checking device "mlx5_1"
> > > > > uverbs1 /sys/class/infiniband_verbs/uverbs1
> > > > > /sys/class/infiniband/mlx5_1
> > > > > net_mlx5: mlx5.c:1382: mlx5_pci_probe(): no switch support
> > > > > detected
> > > > > net_mlx5: mlx5.c:796: mlx5_dev_spawn(): MPW isn't supported
> > > > > net_mlx5: mlx5.c:852: mlx5_dev_spawn(): tunnel offloading
> > > > > disabled due to old OFED/rdma-core version
> > > > > net_mlx5: mlx5.c:864: mlx5_dev_spawn(): MPLS over GRE/UDP
> tunnel
> > > > > offloading disabled due to old OFED/rdma-core version or
> > > > > firmware configuration
> > > > > net_mlx5: mlx5.c:877: mlx5_dev_spawn(): naming Ethernet device
> > > > "0000:b3:00.0"
> > > > > net_mlx5: mlx5.c:1007: mlx5_dev_spawn(): checksum offloading is
> > > > > supported
> > > > > net_mlx5: mlx5.c:1024: mlx5_dev_spawn(): maximum Rx indirection
> > > > > table size is 512
> > > > > net_mlx5: mlx5.c:1028: mlx5_dev_spawn(): VLAN stripping is not
> > > > > supported
> > > > > net_mlx5: mlx5.c:1032: mlx5_dev_spawn(): FCS stripping
> > > > > configuration is not supported
> > > > > net_mlx5: mlx5.c:1037: mlx5_dev_spawn(): hardware Rx end
> > > > > alignment padding is not supported
> > > > > net_mlx5: mlx5.c:1052: mlx5_dev_spawn(): MPS is disabled
> > > > > net_mlx5: mlx5.c:628: mlx5_uar_init_primary(): port 0 reserved
> > > > > UAR address space: 0x880000000
> > > > > net_mlx5: mlx5.c:1107: mlx5_dev_spawn(): port 0 MAC address is
> > > > > 00:90:0b:78:2a:ea
> > > > > net_mlx5: mlx5.c:1114: mlx5_dev_spawn(): port 0 ifname is "mce0"
> > > > > net_mlx5: mlx5.c:1127: mlx5_dev_spawn(): port 0 MTU is 1500
> > > > > net_mlx5: mlx5.c:1142: mlx5_dev_spawn(): flow rules relying on
> > > > > switch offloads will not be supported: cannot open libmnl socket:
> > > > > Address family not supported by protocol family
> > > > > net_mlx5: mlx5.c:1176: mlx5_dev_spawn(): port 0 forcing Ethernet
> > > > > interface up
> > > > > net_mlx5: mlx5_ethdev.c:735: mlx5_link_update_unlocked_gs():
> > > > > port
> > > > > 0
> > > > > ioctl(SIOCGIFFLAGS) failed: No such file or directory
> > > > > net_mlx5: mlx5_ethdev.c:664: mlx5_link_update_unlocked_gset():
> > > > > port
> > > > > 0
> > > > > ioctl(SIOCGIFFLAGS) failed: No such file or directory
> > > > > net_mlx5: mlx5_rxq.c:2150: mlx5_hrxq_drop_new(): port 0 cannot
> > > > > allocate QP for drop queue Segmentation fault (core dumped)
> > > > >
> > > > > Working through mlx5_ethdev now...
> > > > >
> > > > > Mit
> > > >
> > > > Also if you are targeting Azure or Hyper-V, then don't bother
> > > > porting TAP/VDEV_NETVSC/Failsafe model because that is being
> > > > replaced by netvsc native PMD.  It would be easier to just get
> > > > that
> > working.
> > > > It would mean writing a  uio equivalent driver for FreeBSD.