RE: vmxnet3 no longer functional on DPDK 21.11

["Ananyev, Konstantin" <konstantin.ananyev@intel.com>]

> -----Original Message-----
> From: Richardson, Bruce <bruce.richardson@intel.com>
> Sent: Monday, December 6, 2021 9:17 AM
> To: Lewis Donzis <lew@perftech.com>
> Cc: dev <dev@dpdk.org>; Wang, Yong <yongwang@vmware.com>; Ananyev, Konstantin <konstantin.ananyev@intel.com>
> Subject: Re: vmxnet3 no longer functional on DPDK 21.11
> 
> On Sun, Dec 05, 2021 at 07:52:33PM -0600, Lewis Donzis wrote:
> >
> >
> > ----- On Nov 30, 2021, at 7:42 AM, Bruce Richardson bruce.richardson@intel.com wrote:
> >
> > > On Mon, Nov 29, 2021 at 02:45:15PM -0600, Lewis Donzis wrote:
> > >>    Hello.
> > >>    We just upgraded from 21.08 to 21.11 and it's rather astounding the
> > >>    number of incompatible changes in three months.  Not a big deal, just
> > >>    kind of a surprise, that's all.
> > >>    Anyway, the problem is that the vmxnet3 driver is no longer functional
> > >>    on FreeBSD.
> > >>    In drivers/net/vmxnet3/vmxnet3_ethdev.c, vmxnet3_dev_start() gets an
> > >>    error calling rte_intr_enable().  So it logs "interrupt enable failed"
> > >>    and returns an error.
> > >>    In lib/eal/freebsd/eal_interrupts.c, rte_intr_enable() is returning an
> > >>    error because rte_intr_dev_fd_get(intr_handle) is returning -1.
> > >>    I don't see how that could ever return anything other than -1 since it
> > >>    appears that there is no code that ever calls rte_intr_dev_fd_set()
> > >>    with a value other than -1 on FreeBSD.  Also weird to me is that even
> > >>    if it didn't get an error, the switch statement that follows looks like
> > >>    it will return an error in every case.
> > >>    Nonetheless, it worked in 21.08, and I can't quite see why the
> > >>    difference, so I must be missing something.
> > >>    For the moment, I just commented the "return -EIO" in vmxnet3_ethdev.c,
> > >>    and it's now working again, but that's obviously not the correct
> > >>    solution.
> > >>    Can someone who's knowledgable about this mechanism perhaps explain a
> > >>    little bit about what's going on?  I'll be happy to help troubleshoot.
> > >>    It seems like it must be something simple, but I just don't see it yet.
> > >
> > > Hi
> > >
> > > if you have the chance, it would be useful if you could use "git bisect" to
> > > identify the commit in 21.11 that broke this driver. Looking through the
> > > logs for 21.11 I can't identify any particular likely-looking commit, so
> > > bisect is likely a good way to start looking into this.
> > >
> > > Regards,
> > > /Bruce
> >
> > Hi, Bruce.  git bisect is very time-consuming and very cool!
> >
> > I went back to 21.08, about 1100 commits, and worked through the process, but then I realized that I had forgotten to run ninja on one of
> the steps, so I did it again.
> >
> > I also re-checked it after the bisect, just to make sure that c87d435a4d79739c0cec2ed280b94b41cb908af7 is good, and
> 7a0935239b9eb817c65c03554a9954ddb8ea5044 is bad.
> >
> > Thanks,
> > lew
> >
> 
> Many thanks for taking the time to do this. Adding Konstantin to thread as
> author of the commit you identified. Konstantin, any thoughts on this
> issue?

Hmm, that's looks really strange to me.
So to clarify, it fails at:
static int
vmxnet3_dev_start(struct rte_eth_dev *dev)
{
	...
line 1695:
	if (rte_intr_enable(dev->intr_handle) < 0) {
                PMD_INIT_LOG(ERR, "interrupt enable failed");
                return -EIO;
        }	

Right?

The strange thing here is that 7a0935239b9e
doesn't change dev_start or rte_intr code in any way.
All it does - change rte_eth_rx_burst/rte_eth_tx_burst and other fast-path
functions.
Anyway, if git blames that commit, let's try to figure out what is going on.
Unfortunately, I don't have freebsd with vmxnet3, so will need to rely on your help here.
As the first thing can you try to run testpmd build with last good commit (c87d435a4d79)
and then testpmd build with bad commit applied and collect for both cases:
- contents of 'struct rte_eth_dev' and ' rte_eth_dev->intr_handle' for
  your vmxnet3 port
- debug log output (--log-level=eal,debug --log-level=pmd,debug) 

Konstantin