Re: [dpdk-dev] [PATCH] net/ixgbe: initialize max_rx_pkt_len if rlpml_set_vf fails

[Tudor Cornea <tudor.cornea@gmail.com>]

Hi Ferruh,

I have tested with the dpdk-next-net branch.
It includes the commit 'ethdev: fix max Rx packet length'

Setup:
Hypervisor: VMware ESXi 6.0.0
PF ixgbe Driver: 3.7.13.7 (default PF driver installed with ESXi 6.0 and
6.5)
NIC: Intel 82599
Guest OS : Ubuntu 20.04

It seems that with the recent changes testpmd still can't initialize the
ports.

EAL: Detected CPU lcores: 8
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available 1048576 kB hugepages reported
EAL: VFIO support initialized
EAL: Probe PCI driver: net_ixgbe_vf (8086:10ed) device: 0000:0b:00.0
(socket 0)
EAL: Probe PCI driver: net_ixgbe_vf (8086:10ed) device: 0000:13:00.0
(socket 0)
TELEMETRY: No legacy callbacks, legacy socket not created
Interactive-mode selected
previous number of forwarding ports 2 - changed to number of configured
ports 1
Error picking flow transfer proxy for port 0: Function not implemented -
ignore
Error picking flow transfer proxy for port 1: Function not implemented -
ignore
testpmd: create a new mbuf pool <mb_pool_0>: n=171456, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc

Warning! port-topology=paired and odd forward ports number, the last port
will pair with itself.

Configuring Port 0 (socket 0)
ixgbevf_dev_rx_init(): Set max packet length to 1518 failed.
ixgbevf_dev_start(): Unable to initialize RX hardware (-22)
Fail to start port 0: Invalid argument
Configuring Port 1 (socket 0)
ixgbevf_dev_rx_init(): Set max packet length to 1518 failed.
ixgbevf_dev_start(): Unable to initialize RX hardware (-22)
Fail to start port 1: Invalid argument
Please stop the ports first
Done
Error during enabling promiscuous mode for port 0: Operation not supported
- ignore
Error during enabling promiscuous mode for port 1: Operation not supported
- ignore
testpmd> start tx_first
Not all ports were started

I think failing to set set 'max_rx_pkt_len' after ixgbevf_rlpml_set_vf()
call failed in ixgbevf_dev_set_mtu(), might have been one half of the
problem.
The other one is in ixgbevf_dev_rx_init(). The init function seems to
return prematurely, and not initialize the ports.

With the below patch (on top of net-next branch), I seem to be able to
initialize the ports correctly, and send packets using testpmd.

diff --git a/drivers/net/ixgbe/ixgbe_rxtx.c b/drivers/net/ixgbe/ixgbe_rxtx.c
index b263dfe1d5..fdd057c789 100644
--- a/drivers/net/ixgbe/ixgbe_rxtx.c
+++ b/drivers/net/ixgbe/ixgbe_rxtx.c
@@ -5676,7 +5676,6 @@ ixgbevf_dev_rx_init(struct rte_eth_dev *dev)
        if (ixgbevf_rlpml_set_vf(hw, frame_size) != 0) {
                PMD_INIT_LOG(ERR, "Set max packet length to %d failed.",
                             frame_size);
-               return -EINVAL;
        }

        /*

Alvin, would it be acceptable to not return -EINVAL in this case while
still printing an error, so that we can still debug mtu issues on 82599 ?

If I recall correctly, the NIC has issues when the MTU of VFs is bigger
than the MTU of the PFs. On my setup, however the MTUs have default values
(1500).
Should I rebase the patch on top of net-next ?

Thanks,
Tudor

On Wed, 20 Oct 2021 at 06:08, Zhang, AlvinX <alvinx.zhang@intel.com> wrote:

> > -----Original Message-----
> > From: Yigit, Ferruh <ferruh.yigit@intel.com>
> > Sent: Tuesday, October 19, 2021 8:58 PM
> > To: Tudor Cornea <tudor.cornea@gmail.com>; Zhang, Qi Z
> > <qi.z.zhang@intel.com>
> > Cc: Zhang, AlvinX <alvinx.zhang@intel.com>; Wang, Haiyue
> > <haiyue.wang@intel.com>; dev@dpdk.org
> > Subject: Re: [dpdk-dev] [PATCH] net/ixgbe: initialize max_rx_pkt_len if
> > rlpml_set_vf fails
> >
> > On 10/15/2021 3:20 PM, Tudor Cornea wrote:
> > > Some of our customers use ESXi 6.0 or 6.5 servers, which could have
> > > older versions of the PF ixgbe driver.
> > > It seems that with a more recent version of the PMD driver, we are not
> > > able to initialize 82599EB ports correctly.
> > > This scenario seems to have worked with DPDK 19.11.
> > >
> > > Would it be possible to print a warning, while still allowing the
> > > driver to initialize the ports ?
>
> There is a scenario that we initialize the port successful, with no any
> error, but we cannot get any packets.
> So keep printing error or warning and still allowing the driver to
> initialize the port may be the best solution.
>
> > >
> > > I was also thinking about the return code of ixgbevf_dev_set_mtu.
> > > Do you think it would be more appropriate to return ENOTSUP or ENOSYS
> > > instead of EINVAL ?
>
> If ixgbevf_dev_set_mtu fails, we have no way to know it because of invalid
> value or not supported?
> ixgbevf_dev_set_mtu returns one the these values: 0 or IXGBE_ERR_MBX
>
> > >
> > > As a user, calling 'rte_eth_dev_mtu_set', I would expect an error like
> > > EINVAL to suggest to me that the mtu value which I provided is
> > > incorrect [1]. The 82599 NIC, however has some particularities related
> > > to mtu, which could cause the operation to fail. I was thinking that
> > > EINVAL might not be most descriptive error.
> > >
> > > [1] https://doc.dpdk.org/api/rte__ethdev_8h.html
> > >
> > > Thanks,
> > > Tudor
> >
> > Hi Tudor,
> >
> > Can you please check if the patch is still after 'max_rx_pkt_len'
> related changes
> > in next-net?
> >
> > >
> > > On Fri, 15 Oct 2021 at 17:06, Tudor Cornea <tudor.cornea@gmail.com>
> > wrote:
> > >
> > >> It seems that if the call to ixgbevf_rlpml_set_vf fails, we will not
> > >> initialize dev_conf.rxmode.max_rx_pkt_len correctly anymore.
> > >>
> > >> This happens with a 82599EB NIC and a VMware ESXI 6.0 setup, and is
> > >> causing VF the ports to fail to initialize
> > >>
> > >> We see the following error:
> > >> ixgbevf_dev_rx_init(): Set max packet length to 1518 failed.
> > >>
> > >> Investigating over DPDK 19.11, it seems that the call still fails,
> > >> but it doesn't exit prematurely, and max_rx_pkt_len is initialized in
> > >> the respective case.
> > >>
> > >> On the ESXi server, we seem to have the following driver
> > >> net-ixgbe: 3.7.13.7.14iov-20vmw.600.0.0.2494585
> > >>
> > >> It seems that the behavior related to MTU setting has changed since
> > >> the following commit:
> > >>
> > >> commit c77866a16904 ("net/ixgbe: detect failed VF MTU set")
> > >>
> > >> We would like to still be able to support older setups if possible,
> > >> as we might have customers running ESXi 6.0 or 6.5, and these seem to
> > >> have an older version of the PF driver as default.
> > >>
> > >> Signed-off-by: Tudor Cornea <tudor.cornea@gmail.com>
> > >> ---
> > >>   drivers/net/ixgbe/ixgbe_ethdev.c | 5 +++--
> > >>   drivers/net/ixgbe/ixgbe_rxtx.c   | 1 -
> > >>   2 files changed, 3 insertions(+), 3 deletions(-)
> > >>
> > >> diff --git a/drivers/net/ixgbe/ixgbe_ethdev.c
> > >> b/drivers/net/ixgbe/ixgbe_ethdev.c
> > >> index 4dbe049..4301dfd 100644
> > >> --- a/drivers/net/ixgbe/ixgbe_ethdev.c
> > >> +++ b/drivers/net/ixgbe/ixgbe_ethdev.c
> > >> @@ -6369,6 +6369,9 @@ ixgbevf_dev_set_mtu(struct rte_eth_dev *dev,
> > >> uint16_t mtu)
> > >>                  return -EINVAL;
> > >>          }
> > >>
> > >> +       /* update max frame size */
> > >> +       dev->data->dev_conf.rxmode.max_rx_pkt_len = max_frame;
> > >> +
> > >>          /*
> > >>           * When supported by the underlying PF driver, use the
> > >> IXGBE_VF_SET_MTU
> > >>           * request of the version 2.0 of the mailbox API.
> > >> @@ -6381,8 +6384,6 @@ ixgbevf_dev_set_mtu(struct rte_eth_dev *dev,
> > >> uint16_t mtu)
> > >>          if (ixgbevf_rlpml_set_vf(hw, max_frame))
> > >>                  return -EINVAL;
> > >>
> > >> -       /* update max frame size */
> > >> -       dev->data->dev_conf.rxmode.max_rx_pkt_len = max_frame;
> > >>          return 0;
> > >>   }
> > >>
> > >> diff --git a/drivers/net/ixgbe/ixgbe_rxtx.c
> > >> b/drivers/net/ixgbe/ixgbe_rxtx.c index 0ac89cb..02d9809 100644
> > >> --- a/drivers/net/ixgbe/ixgbe_rxtx.c
> > >> +++ b/drivers/net/ixgbe/ixgbe_rxtx.c
> > >> @@ -5677,7 +5677,6 @@ ixgbevf_dev_rx_init(struct rte_eth_dev *dev)
> > >>              (uint16_t)dev->data->dev_conf.rxmode.max_rx_pkt_len)) {
> > >>                  PMD_INIT_LOG(ERR, "Set max packet length to %d
> > failed.",
> > >>
> > dev->data->dev_conf.rxmode.max_rx_pkt_len);
> > >> -               return -EINVAL;
> > >>          }
> > >>
> > >>          /*
> > >> --
> > >> 2.7.4
> > >>
> > >>
>
>