vmxnet3 no longer functional on DPDK 21.11

vmxnet3 no longer functional on DPDK 21.11

[Lewis Donzis]

[-- Attachment #1: Type: text/plain, Size: 1372 bytes --]

Hello. 

We just upgraded from 21.08 to 21.11 and it's rather astounding the number of incompatible changes in three months. Not a big deal, just kind of a surprise, that's all. 

Anyway, the problem is that the vmxnet3 driver is no longer functional on FreeBSD. 

In drivers/net/vmxnet3/vmxnet3_ethdev.c, vmxnet3_dev_start() gets an error calling rte_intr_enable(). So it logs "interrupt enable failed" and returns an error. 

In lib/eal/freebsd/eal_interrupts.c, rte_intr_enable() is returning an error because rte_intr_dev_fd_get(intr_handle) is returning -1. 

I don't see how that could ever return anything other than -1 since it appears that there is no code that ever calls rte_intr_dev_fd_set() with a value other than -1 on FreeBSD. Also weird to me is that even if it didn't get an error, the switch statement that follows looks like it will return an error in every case. 

Nonetheless, it worked in 21.08, and I can't quite see why the difference, so I must be missing something. 

For the moment, I just commented the "return -EIO" in vmxnet3_ethdev.c, and it's now working again, but that's obviously not the correct solution. 

Can someone who's knowledgable about this mechanism perhaps explain a little bit about what's going on? I'll be happy to help troubleshoot. It seems like it must be something simple, but I just don't see it yet. 

Thanks, 
lew 

[-- Attachment #2: Type: text/html, Size: 4052 bytes --]

Re: vmxnet3 no longer functional on DPDK 21.11

[Ferruh Yigit]

On 11/29/2021 8:45 PM, Lewis Donzis wrote:
> Hello.
> 
> We just upgraded from 21.08 to 21.11 and it's rather astounding the number of incompatible changes in three months.  Not a big deal, just kind of a surprise, that's all.
> 
> Anyway, the problem is that the vmxnet3 driver is no longer functional on FreeBSD.
> 
> In drivers/net/vmxnet3/vmxnet3_ethdev.c, vmxnet3_dev_start() gets an error calling rte_intr_enable().  So it logs "interrupt enable failed" and returns an error.
> 
> In lib/eal/freebsd/eal_interrupts.c, rte_intr_enable() is returning an error because rte_intr_dev_fd_get(intr_handle) is returning -1.
> 
> I don't see how that could ever return anything other than -1 since it appears that there is no code that ever calls rte_intr_dev_fd_set() with a value other than -1 on FreeBSD.  Also weird to me is that even if it didn't get an error, the switch statement that follows looks like it will return an error in every case.
> 
> Nonetheless, it worked in 21.08, and I can't quite see why the difference, so I must be missing something.
> 
> For the moment, I just commented the "return -EIO" in vmxnet3_ethdev.c, and it's now working again, but that's obviously not the correct solution.
> 
> Can someone who's knowledgable about this mechanism perhaps explain a little bit about what's going on?  I'll be happy to help troubleshoot.  It seems like it must be something simple, but I just don't see it yet.
> 
> Thanks,
> lew


+Yong

Re: vmxnet3 no longer functional on DPDK 21.11

[Bruce Richardson]

On Mon, Nov 29, 2021 at 02:45:15PM -0600, Lewis Donzis wrote:
>    Hello.
>    We just upgraded from 21.08 to 21.11 and it's rather astounding the
>    number of incompatible changes in three months.  Not a big deal, just
>    kind of a surprise, that's all.
>    Anyway, the problem is that the vmxnet3 driver is no longer functional
>    on FreeBSD.
>    In drivers/net/vmxnet3/vmxnet3_ethdev.c, vmxnet3_dev_start() gets an
>    error calling rte_intr_enable().  So it logs "interrupt enable failed"
>    and returns an error.
>    In lib/eal/freebsd/eal_interrupts.c, rte_intr_enable() is returning an
>    error because rte_intr_dev_fd_get(intr_handle) is returning -1.
>    I don't see how that could ever return anything other than -1 since it
>    appears that there is no code that ever calls rte_intr_dev_fd_set()
>    with a value other than -1 on FreeBSD.  Also weird to me is that even
>    if it didn't get an error, the switch statement that follows looks like
>    it will return an error in every case.
>    Nonetheless, it worked in 21.08, and I can't quite see why the
>    difference, so I must be missing something.
>    For the moment, I just commented the "return -EIO" in vmxnet3_ethdev.c,
>    and it's now working again, but that's obviously not the correct
>    solution.
>    Can someone who's knowledgable about this mechanism perhaps explain a
>    little bit about what's going on?  I'll be happy to help troubleshoot.
>    It seems like it must be something simple, but I just don't see it yet.

Hi

if you have the chance, it would be useful if you could use "git bisect" to
identify the commit in 21.11 that broke this driver. Looking through the
logs for 21.11 I can't identify any particular likely-looking commit, so
bisect is likely a good way to start looking into this.

Regards,
/Bruce

Re: vmxnet3 no longer functional on DPDK 21.11

[Lewis Donzis]

----- On Nov 30, 2021, at 7:42 AM, Bruce Richardson bruce.richardson@intel.com wrote:

> On Mon, Nov 29, 2021 at 02:45:15PM -0600, Lewis Donzis wrote:
>>    Hello.
>>    We just upgraded from 21.08 to 21.11 and it's rather astounding the
>>    number of incompatible changes in three months.  Not a big deal, just
>>    kind of a surprise, that's all.
>>    Anyway, the problem is that the vmxnet3 driver is no longer functional
>>    on FreeBSD.
>>    In drivers/net/vmxnet3/vmxnet3_ethdev.c, vmxnet3_dev_start() gets an
>>    error calling rte_intr_enable().  So it logs "interrupt enable failed"
>>    and returns an error.
>>    In lib/eal/freebsd/eal_interrupts.c, rte_intr_enable() is returning an
>>    error because rte_intr_dev_fd_get(intr_handle) is returning -1.
>>    I don't see how that could ever return anything other than -1 since it
>>    appears that there is no code that ever calls rte_intr_dev_fd_set()
>>    with a value other than -1 on FreeBSD.  Also weird to me is that even
>>    if it didn't get an error, the switch statement that follows looks like
>>    it will return an error in every case.
>>    Nonetheless, it worked in 21.08, and I can't quite see why the
>>    difference, so I must be missing something.
>>    For the moment, I just commented the "return -EIO" in vmxnet3_ethdev.c,
>>    and it's now working again, but that's obviously not the correct
>>    solution.
>>    Can someone who's knowledgable about this mechanism perhaps explain a
>>    little bit about what's going on?  I'll be happy to help troubleshoot.
>>    It seems like it must be something simple, but I just don't see it yet.
> 
> Hi
> 
> if you have the chance, it would be useful if you could use "git bisect" to
> identify the commit in 21.11 that broke this driver. Looking through the
> logs for 21.11 I can't identify any particular likely-looking commit, so
> bisect is likely a good way to start looking into this.
> 
> Regards,
> /Bruce

Hi, Bruce.  git bisect is very time-consuming and very cool!

I went back to 21.08, about 1100 commits, and worked through the process, but then I realized that I had forgotten to run ninja on one of the steps, so I did it again.

I also re-checked it after the bisect, just to make sure that c87d435a4d79739c0cec2ed280b94b41cb908af7 is good, and 7a0935239b9eb817c65c03554a9954ddb8ea5044 is bad.

Thanks,
lew


Here's the result:

root@fbdev:/usr/local/share/dpdk-git # git bisect start
root@fbdev:/usr/local/share/dpdk-git # git bisect bad
root@fbdev:/usr/local/share/dpdk-git # git bisect good 74bd4072996e64b0051d24d8d641554d225db196
Bisecting: 556 revisions left to test after this (roughly 9 steps)
[e2a289a788c0a128a15bc0f1099af7c031201ac5] net/ngbe: add mailbox process operations
root@fbdev:/usr/local/share/dpdk-git # git bisect bad
Bisecting: 277 revisions left to test after this (roughly 8 steps)
[5906be5af6570db8b70b307c96aace0b096d1a2c] ethdev: fix ID spelling in comments and log messages
root@fbdev:/usr/local/share/dpdk-git # git bisect bad
Bisecting: 138 revisions left to test after this (roughly 7 steps)
[a7c236b894a848c7bb9afb773a7e3c13615abaa8] net/cnxk: support meter ops get
root@fbdev:/usr/local/share/dpdk-git # git bisect bad
Bisecting: 69 revisions left to test after this (roughly 6 steps)
[14fc81aed73842d976dd19a93ca47e22d61c1759] ethdev: update modify field flow action
root@fbdev:/usr/local/share/dpdk-git # git bisect bad
Bisecting: 34 revisions left to test after this (roughly 5 steps)
[cdea571becb4dabf9962455f671af0c99594e380] common/sfc_efx/base: add flag to use Rx prefix user flag
root@fbdev:/usr/local/share/dpdk-git # git bisect good
Bisecting: 17 revisions left to test after this (roughly 4 steps)
[7a0935239b9eb817c65c03554a9954ddb8ea5044] ethdev: make fast-path functions to use new flat array
root@fbdev:/usr/local/share/dpdk-git # git bisect bad
Bisecting: 8 revisions left to test after this (roughly 3 steps)
[012bf708c20f4b23d055717e28f8de74887113d8] net/sfc: support group flows in tunnel offload
root@fbdev:/usr/local/share/dpdk-git # git bisect good
Bisecting: 4 revisions left to test after this (roughly 2 steps)
[9df2d8f5cc9653d6413cb2240c067ea455ab7c3c] net/sfc: support counters in tunnel offload jump rules
root@fbdev:/usr/local/share/dpdk-git # git bisect good
Bisecting: 2 revisions left to test after this (roughly 1 step)
[c024496ae8c8c075b0d0a3b43119475787b24b45] ethdev: allocate max space for internal queue array
root@fbdev:/usr/local/share/dpdk-git # git bisect good
Bisecting: 0 revisions left to test after this (roughly 1 step)
[c87d435a4d79739c0cec2ed280b94b41cb908af7] ethdev: copy fast-path API into separate structure
root@fbdev:/usr/local/share/dpdk-git # git bisect good
7a0935239b9eb817c65c03554a9954ddb8ea5044 is the first bad commit
commit 7a0935239b9eb817c65c03554a9954ddb8ea5044
Author: Konstantin Ananyev <konstantin.ananyev@intel.com>
Date:   Wed Oct 13 14:37:02 2021 +0100

    ethdev: make fast-path functions to use new flat array
    
    Rework fast-path ethdev functions to use rte_eth_fp_ops[].
    While it is an API/ABI breakage, this change is intended to be
    transparent for both users (no changes in user app is required) and
    PMD developers (no changes in PMD is required).
    One extra thing to note - RX/TX callback invocation will cause extra
    function call with these changes. That might cause some insignificant
    slowdown for code-path where RX/TX callbacks are heavily involved.
    
    Signed-off-by: Konstantin Ananyev <konstantin.ananyev@intel.com>
    Reviewed-by: Ferruh Yigit <ferruh.yigit@intel.com>
    Tested-by: Feifei Wang <feifei.wang2@arm.com>

 lib/ethdev/ethdev_private.c |  31 +++++
 lib/ethdev/rte_ethdev.h     | 270 +++++++++++++++++++++++++++++++-------------
 lib/ethdev/version.map      |   3 +
 3 files changed, 226 insertions(+), 78 deletions(-)

Re: vmxnet3 no longer functional on DPDK 21.11

[Bruce Richardson]

On Sun, Dec 05, 2021 at 07:52:33PM -0600, Lewis Donzis wrote:
> 
> 
> ----- On Nov 30, 2021, at 7:42 AM, Bruce Richardson bruce.richardson@intel.com wrote:
> 
> > On Mon, Nov 29, 2021 at 02:45:15PM -0600, Lewis Donzis wrote:
> >>    Hello.
> >>    We just upgraded from 21.08 to 21.11 and it's rather astounding the
> >>    number of incompatible changes in three months.  Not a big deal, just
> >>    kind of a surprise, that's all.
> >>    Anyway, the problem is that the vmxnet3 driver is no longer functional
> >>    on FreeBSD.
> >>    In drivers/net/vmxnet3/vmxnet3_ethdev.c, vmxnet3_dev_start() gets an
> >>    error calling rte_intr_enable().  So it logs "interrupt enable failed"
> >>    and returns an error.
> >>    In lib/eal/freebsd/eal_interrupts.c, rte_intr_enable() is returning an
> >>    error because rte_intr_dev_fd_get(intr_handle) is returning -1.
> >>    I don't see how that could ever return anything other than -1 since it
> >>    appears that there is no code that ever calls rte_intr_dev_fd_set()
> >>    with a value other than -1 on FreeBSD.  Also weird to me is that even
> >>    if it didn't get an error, the switch statement that follows looks like
> >>    it will return an error in every case.
> >>    Nonetheless, it worked in 21.08, and I can't quite see why the
> >>    difference, so I must be missing something.
> >>    For the moment, I just commented the "return -EIO" in vmxnet3_ethdev.c,
> >>    and it's now working again, but that's obviously not the correct
> >>    solution.
> >>    Can someone who's knowledgable about this mechanism perhaps explain a
> >>    little bit about what's going on?  I'll be happy to help troubleshoot.
> >>    It seems like it must be something simple, but I just don't see it yet.
> > 
> > Hi
> > 
> > if you have the chance, it would be useful if you could use "git bisect" to
> > identify the commit in 21.11 that broke this driver. Looking through the
> > logs for 21.11 I can't identify any particular likely-looking commit, so
> > bisect is likely a good way to start looking into this.
> > 
> > Regards,
> > /Bruce
> 
> Hi, Bruce.  git bisect is very time-consuming and very cool!
> 
> I went back to 21.08, about 1100 commits, and worked through the process, but then I realized that I had forgotten to run ninja on one of the steps, so I did it again.
> 
> I also re-checked it after the bisect, just to make sure that c87d435a4d79739c0cec2ed280b94b41cb908af7 is good, and 7a0935239b9eb817c65c03554a9954ddb8ea5044 is bad.
> 
> Thanks,
> lew
>

Many thanks for taking the time to do this. Adding Konstantin to thread as
author of the commit you identified. Konstantin, any thoughts on this
issue?

/Bruce 

RE: vmxnet3 no longer functional on DPDK 21.11

[Ananyev, Konstantin]

> -----Original Message-----
> From: Richardson, Bruce <bruce.richardson@intel.com>
> Sent: Monday, December 6, 2021 9:17 AM
> To: Lewis Donzis <lew@perftech.com>
> Cc: dev <dev@dpdk.org>; Wang, Yong <yongwang@vmware.com>; Ananyev, Konstantin <konstantin.ananyev@intel.com>
> Subject: Re: vmxnet3 no longer functional on DPDK 21.11
> 
> On Sun, Dec 05, 2021 at 07:52:33PM -0600, Lewis Donzis wrote:
> >
> >
> > ----- On Nov 30, 2021, at 7:42 AM, Bruce Richardson bruce.richardson@intel.com wrote:
> >
> > > On Mon, Nov 29, 2021 at 02:45:15PM -0600, Lewis Donzis wrote:
> > >>    Hello.
> > >>    We just upgraded from 21.08 to 21.11 and it's rather astounding the
> > >>    number of incompatible changes in three months.  Not a big deal, just
> > >>    kind of a surprise, that's all.
> > >>    Anyway, the problem is that the vmxnet3 driver is no longer functional
> > >>    on FreeBSD.
> > >>    In drivers/net/vmxnet3/vmxnet3_ethdev.c, vmxnet3_dev_start() gets an
> > >>    error calling rte_intr_enable().  So it logs "interrupt enable failed"
> > >>    and returns an error.
> > >>    In lib/eal/freebsd/eal_interrupts.c, rte_intr_enable() is returning an
> > >>    error because rte_intr_dev_fd_get(intr_handle) is returning -1.
> > >>    I don't see how that could ever return anything other than -1 since it
> > >>    appears that there is no code that ever calls rte_intr_dev_fd_set()
> > >>    with a value other than -1 on FreeBSD.  Also weird to me is that even
> > >>    if it didn't get an error, the switch statement that follows looks like
> > >>    it will return an error in every case.
> > >>    Nonetheless, it worked in 21.08, and I can't quite see why the
> > >>    difference, so I must be missing something.
> > >>    For the moment, I just commented the "return -EIO" in vmxnet3_ethdev.c,
> > >>    and it's now working again, but that's obviously not the correct
> > >>    solution.
> > >>    Can someone who's knowledgable about this mechanism perhaps explain a
> > >>    little bit about what's going on?  I'll be happy to help troubleshoot.
> > >>    It seems like it must be something simple, but I just don't see it yet.
> > >
> > > Hi
> > >
> > > if you have the chance, it would be useful if you could use "git bisect" to
> > > identify the commit in 21.11 that broke this driver. Looking through the
> > > logs for 21.11 I can't identify any particular likely-looking commit, so
> > > bisect is likely a good way to start looking into this.
> > >
> > > Regards,
> > > /Bruce
> >
> > Hi, Bruce.  git bisect is very time-consuming and very cool!
> >
> > I went back to 21.08, about 1100 commits, and worked through the process, but then I realized that I had forgotten to run ninja on one of
> the steps, so I did it again.
> >
> > I also re-checked it after the bisect, just to make sure that c87d435a4d79739c0cec2ed280b94b41cb908af7 is good, and
> 7a0935239b9eb817c65c03554a9954ddb8ea5044 is bad.
> >
> > Thanks,
> > lew
> >
> 
> Many thanks for taking the time to do this. Adding Konstantin to thread as
> author of the commit you identified. Konstantin, any thoughts on this
> issue?

Hmm, that's looks really strange to me.
So to clarify, it fails at:
static int
vmxnet3_dev_start(struct rte_eth_dev *dev)
{
	...
line 1695:
	if (rte_intr_enable(dev->intr_handle) < 0) {
                PMD_INIT_LOG(ERR, "interrupt enable failed");
                return -EIO;
        }	

Right?

The strange thing here is that 7a0935239b9e
doesn't change dev_start or rte_intr code in any way.
All it does - change rte_eth_rx_burst/rte_eth_tx_burst and other fast-path
functions.
Anyway, if git blames that commit, let's try to figure out what is going on.
Unfortunately, I don't have freebsd with vmxnet3, so will need to rely on your help here.
As the first thing can you try to run testpmd build with last good commit (c87d435a4d79)
and then testpmd build with bad commit applied and collect for both cases:
- contents of 'struct rte_eth_dev' and ' rte_eth_dev->intr_handle' for
  your vmxnet3 port
- debug log output (--log-level=eal,debug --log-level=pmd,debug) 

Konstantin

Re: vmxnet3 no longer functional on DPDK 21.11

[Lewis Donzis]

----- On Dec 6, 2021, at 6:08 AM, Ananyev, Konstantin konstantin.ananyev@intel.com wrote:

> So to clarify, it fails at:
> static int
> vmxnet3_dev_start(struct rte_eth_dev *dev)
> {
>	...
> line 1695:
>	if (rte_intr_enable(dev->intr_handle) < 0) {
>                PMD_INIT_LOG(ERR, "interrupt enable failed");
>                return -EIO;
>        }
> 
> Right?

That's right.  And further, the failure inside of rte_intr_enable() is the test on rte_intr_dev_fd_get():

int
rte_intr_enable(const struct rte_intr_handle *intr_handle)
{
        ...
        if (rte_intr_fd_get(intr_handle) < 0 ||
                        rte_intr_dev_fd_get(intr_handle) < 0) {
                rc = -1;
                goto out;
        }

        switch (rte_intr_type_get(intr_handle)) {
        /* not used at this moment */
        case RTE_INTR_HANDLE_ALARM:
                rc = -1;
                break;
        /* not used at this moment */
        case RTE_INTR_HANDLE_DEV_EVENT:
                rc = -1;
                break;
        /* unknown handle type */
        default:
                RTE_LOG(ERR, EAL, "Unknown handle type of fd %d\n",
                        rte_intr_fd_get(intr_handle));
                rc = -1;
                break;
        }

out:
        rte_eal_trace_intr_enable(intr_handle, rc);
        return rc;
}


Two things about this code that confuse me:

1. rte_intr_dev_fd_get(intr_handle) just returns the value of intr_handle->dev_fd, which is never set to anything other than -1 in any code I can find.

2. Even if it made it past that "if" statement, I don't see how the switch that follows ever *doesn't* return an error, i.e., every single case results in an error!  Even if it got past rte_intr_dev_fd_get(), the interrupt type is neither of the first two cases, so it would presumably execute the default case.


> The strange thing here is that 7a0935239b9e
> doesn't change dev_start or rte_intr code in any way.

I agree, and also don't see any way the diffs between those two commits could cause this.


> Anyway, if git blames that commit, let's try to figure out what is going on.
> Unfortunately, I don't have freebsd with vmxnet3, so will need to rely on your
> help here.

Sure no problem.  If it helps, we could arrange for remote access to a FreeBSD VM, but I suspect this is going to be something simple, so let's see if we can get it figured out.


> As the first thing can you try to run testpmd build with last good commit
> (c87d435a4d79)
> and then testpmd build with bad commit applied and collect for both cases:
> - contents of 'struct rte_eth_dev' and ' rte_eth_dev->intr_handle' for
>  your vmxnet3 port
> - debug log output (--log-level=eal,debug --log-level=pmd,debug)

Ok, we'll check it out.

Thanks,
lew

Re: vmxnet3 no longer functional on DPDK 21.11

[Lewis Donzis]

Hi, all.

Resurrecting this thread from six months ago, I apologize for not having more time to dig into it, but in light of recent findings, I see numerous other drivers and other parts of the code that have comments to the effect that "FreeBSD doesn't support interrupts" and they effectively #ifdef out the attempt.

Could this be as simple as needing to do the same in vmxnet3?  Empirically, ignoring the error from rte_intr_enable() allows the driver to work normally, for what that's worth.

Thanks,
lew

----- On Dec 6, 2021, at 6:08 AM, Konstantin Ananyev konstantin.ananyev@intel.com wrote:

>> -----Original Message-----
>> From: Richardson, Bruce <bruce.richardson@intel.com>
>> Sent: Monday, December 6, 2021 9:17 AM
>> To: Lewis Donzis <lew@perftech.com>
>> Cc: dev <dev@dpdk.org>; Wang, Yong <yongwang@vmware.com>; Ananyev, Konstantin
>> <konstantin.ananyev@intel.com>
>> Subject: Re: vmxnet3 no longer functional on DPDK 21.11
>> 
>> On Sun, Dec 05, 2021 at 07:52:33PM -0600, Lewis Donzis wrote:
>> >
>> >
>> > ----- On Nov 30, 2021, at 7:42 AM, Bruce Richardson bruce.richardson@intel.com
>> > wrote:
>> >
>> > > On Mon, Nov 29, 2021 at 02:45:15PM -0600, Lewis Donzis wrote:
>> > >>    Hello.
>> > >>    We just upgraded from 21.08 to 21.11 and it's rather astounding the
>> > >>    number of incompatible changes in three months.  Not a big deal, just
>> > >>    kind of a surprise, that's all.
>> > >>    Anyway, the problem is that the vmxnet3 driver is no longer functional
>> > >>    on FreeBSD.
>> > >>    In drivers/net/vmxnet3/vmxnet3_ethdev.c, vmxnet3_dev_start() gets an
>> > >>    error calling rte_intr_enable().  So it logs "interrupt enable failed"
>> > >>    and returns an error.
>> > >>    In lib/eal/freebsd/eal_interrupts.c, rte_intr_enable() is returning an
>> > >>    error because rte_intr_dev_fd_get(intr_handle) is returning -1.
>> > >>    I don't see how that could ever return anything other than -1 since it
>> > >>    appears that there is no code that ever calls rte_intr_dev_fd_set()
>> > >>    with a value other than -1 on FreeBSD.  Also weird to me is that even
>> > >>    if it didn't get an error, the switch statement that follows looks like
>> > >>    it will return an error in every case.
>> > >>    Nonetheless, it worked in 21.08, and I can't quite see why the
>> > >>    difference, so I must be missing something.
>> > >>    For the moment, I just commented the "return -EIO" in vmxnet3_ethdev.c,
>> > >>    and it's now working again, but that's obviously not the correct
>> > >>    solution.
>> > >>    Can someone who's knowledgable about this mechanism perhaps explain a
>> > >>    little bit about what's going on?  I'll be happy to help troubleshoot.
>> > >>    It seems like it must be something simple, but I just don't see it yet.
>> > >
>> > > Hi
>> > >
>> > > if you have the chance, it would be useful if you could use "git bisect" to
>> > > identify the commit in 21.11 that broke this driver. Looking through the
>> > > logs for 21.11 I can't identify any particular likely-looking commit, so
>> > > bisect is likely a good way to start looking into this.
>> > >
>> > > Regards,
>> > > /Bruce
>> >
>> > Hi, Bruce.  git bisect is very time-consuming and very cool!
>> >
>> > I went back to 21.08, about 1100 commits, and worked through the process, but
>> > then I realized that I had forgotten to run ninja on one of
>> the steps, so I did it again.
>> >
>> > I also re-checked it after the bisect, just to make sure that
>> > c87d435a4d79739c0cec2ed280b94b41cb908af7 is good, and
>> 7a0935239b9eb817c65c03554a9954ddb8ea5044 is bad.
>> >
>> > Thanks,
>> > lew
>> >
>> 
>> Many thanks for taking the time to do this. Adding Konstantin to thread as
>> author of the commit you identified. Konstantin, any thoughts on this
>> issue?
> 
> Hmm, that's looks really strange to me.
> So to clarify, it fails at:
> static int
> vmxnet3_dev_start(struct rte_eth_dev *dev)
> {
>	...
> line 1695:
>	if (rte_intr_enable(dev->intr_handle) < 0) {
>                PMD_INIT_LOG(ERR, "interrupt enable failed");
>                return -EIO;
>        }
> 
> Right?
> 
> The strange thing here is that 7a0935239b9e
> doesn't change dev_start or rte_intr code in any way.
> All it does - change rte_eth_rx_burst/rte_eth_tx_burst and other fast-path
> functions.
> Anyway, if git blames that commit, let's try to figure out what is going on.
> Unfortunately, I don't have freebsd with vmxnet3, so will need to rely on your
> help here.
> As the first thing can you try to run testpmd build with last good commit
> (c87d435a4d79)
> and then testpmd build with bad commit applied and collect for both cases:
> - contents of 'struct rte_eth_dev' and ' rte_eth_dev->intr_handle' for
>  your vmxnet3 port
> - debug log output (--log-level=eal,debug --log-level=pmd,debug)
> 
> Konstantin

Re: vmxnet3 no longer functional on DPDK 21.11

[Ferruh Yigit]

On 6/3/2022 2:19 PM, Lewis Donzis wrote:
> Hi, all.
> 
> Resurrecting this thread from six months ago, I apologize for not having more time to dig into it, but in light of recent findings, I see numerous other drivers and other parts of the code that have comments to the effect that "FreeBSD doesn't support interrupts" and they effectively #ifdef out the attempt.
> 
> Could this be as simple as needing to do the same in vmxnet3?  Empirically, ignoring the error from rte_intr_enable() allows the driver to work normally, for what that's worth.
> 
> Thanks,
> lew

cc'ed new maintainer, Jochen.

> 
> ----- On Dec 6, 2021, at 6:08 AM, Konstantin Ananyev konstantin.ananyev@intel.com wrote:
> 
>>> -----Original Message-----
>>> From: Richardson, Bruce <bruce.richardson@intel.com>
>>> Sent: Monday, December 6, 2021 9:17 AM
>>> To: Lewis Donzis <lew@perftech.com>
>>> Cc: dev <dev@dpdk.org>; Wang, Yong <yongwang@vmware.com>; Ananyev, Konstantin
>>> <konstantin.ananyev@intel.com>
>>> Subject: Re: vmxnet3 no longer functional on DPDK 21.11
>>>
>>> On Sun, Dec 05, 2021 at 07:52:33PM -0600, Lewis Donzis wrote:
>>>>
>>>>
>>>> ----- On Nov 30, 2021, at 7:42 AM, Bruce Richardson bruce.richardson@intel.com
>>>> wrote:
>>>>
>>>>> On Mon, Nov 29, 2021 at 02:45:15PM -0600, Lewis Donzis wrote:
>>>>>>     Hello.
>>>>>>     We just upgraded from 21.08 to 21.11 and it's rather astounding the
>>>>>>     number of incompatible changes in three months.  Not a big deal, just
>>>>>>     kind of a surprise, that's all.
>>>>>>     Anyway, the problem is that the vmxnet3 driver is no longer functional
>>>>>>     on FreeBSD.
>>>>>>     In drivers/net/vmxnet3/vmxnet3_ethdev.c, vmxnet3_dev_start() gets an
>>>>>>     error calling rte_intr_enable().  So it logs "interrupt enable failed"
>>>>>>     and returns an error.
>>>>>>     In lib/eal/freebsd/eal_interrupts.c, rte_intr_enable() is returning an
>>>>>>     error because rte_intr_dev_fd_get(intr_handle) is returning -1.
>>>>>>     I don't see how that could ever return anything other than -1 since it
>>>>>>     appears that there is no code that ever calls rte_intr_dev_fd_set()
>>>>>>     with a value other than -1 on FreeBSD.  Also weird to me is that even
>>>>>>     if it didn't get an error, the switch statement that follows looks like
>>>>>>     it will return an error in every case.
>>>>>>     Nonetheless, it worked in 21.08, and I can't quite see why the
>>>>>>     difference, so I must be missing something.
>>>>>>     For the moment, I just commented the "return -EIO" in vmxnet3_ethdev.c,
>>>>>>     and it's now working again, but that's obviously not the correct
>>>>>>     solution.
>>>>>>     Can someone who's knowledgable about this mechanism perhaps explain a
>>>>>>     little bit about what's going on?  I'll be happy to help troubleshoot.
>>>>>>     It seems like it must be something simple, but I just don't see it yet.
>>>>>
>>>>> Hi
>>>>>
>>>>> if you have the chance, it would be useful if you could use "git bisect" to
>>>>> identify the commit in 21.11 that broke this driver. Looking through the
>>>>> logs for 21.11 I can't identify any particular likely-looking commit, so
>>>>> bisect is likely a good way to start looking into this.
>>>>>
>>>>> Regards,
>>>>> /Bruce
>>>>
>>>> Hi, Bruce.  git bisect is very time-consuming and very cool!
>>>>
>>>> I went back to 21.08, about 1100 commits, and worked through the process, but
>>>> then I realized that I had forgotten to run ninja on one of
>>> the steps, so I did it again.
>>>>
>>>> I also re-checked it after the bisect, just to make sure that
>>>> c87d435a4d79739c0cec2ed280b94b41cb908af7 is good, and
>>> 7a0935239b9eb817c65c03554a9954ddb8ea5044 is bad.
>>>>
>>>> Thanks,
>>>> lew
>>>>
>>>
>>> Many thanks for taking the time to do this. Adding Konstantin to thread as
>>> author of the commit you identified. Konstantin, any thoughts on this
>>> issue?
>>
>> Hmm, that's looks really strange to me.
>> So to clarify, it fails at:
>> static int
>> vmxnet3_dev_start(struct rte_eth_dev *dev)
>> {
>> 	...
>> line 1695:
>> 	if (rte_intr_enable(dev->intr_handle) < 0) {
>>                 PMD_INIT_LOG(ERR, "interrupt enable failed");
>>                 return -EIO;
>>         }
>>
>> Right?
>>
>> The strange thing here is that 7a0935239b9e
>> doesn't change dev_start or rte_intr code in any way.
>> All it does - change rte_eth_rx_burst/rte_eth_tx_burst and other fast-path
>> functions.
>> Anyway, if git blames that commit, let's try to figure out what is going on.
>> Unfortunately, I don't have freebsd with vmxnet3, so will need to rely on your
>> help here.
>> As the first thing can you try to run testpmd build with last good commit
>> (c87d435a4d79)
>> and then testpmd build with bad commit applied and collect for both cases:
>> - contents of 'struct rte_eth_dev' and ' rte_eth_dev->intr_handle' for
>>   your vmxnet3 port
>> - debug log output (--log-level=eal,debug --log-level=pmd,debug)
>>
>> Konstantin

Re: vmxnet3 no longer functional on DPDK 21.11

[Lewis Donzis]

Good morning.

I just wanted to mention that this problem still persists in 22.11.3, and we still have to patch the vmxnet3 driver every time we upgrade.

As mentioned before, ixgbe_ethdev.c is an example of a driver that ifdef's out the attempt to use interrupts on FreeBSD.

Thanks,
lew


----- On Jun 3, 2022, at 8:19 AM, Lewis Donzis lew@perftech.com wrote:

> Hi, all.
> 
> Resurrecting this thread from six months ago, I apologize for not having more
> time to dig into it, but in light of recent findings, I see numerous other
> drivers and other parts of the code that have comments to the effect that
> "FreeBSD doesn't support interrupts" and they effectively #ifdef out the
> attempt.
> 
> Could this be as simple as needing to do the same in vmxnet3?  Empirically,
> ignoring the error from rte_intr_enable() allows the driver to work normally,
> for what that's worth.
> 
> Thanks,
> lew
> 
> ----- On Dec 6, 2021, at 6:08 AM, Konstantin Ananyev
> konstantin.ananyev@intel.com wrote:
> 
>>> -----Original Message-----
>>> From: Richardson, Bruce <bruce.richardson@intel.com>
>>> Sent: Monday, December 6, 2021 9:17 AM
>>> To: Lewis Donzis <lew@perftech.com>
>>> Cc: dev <dev@dpdk.org>; Wang, Yong <yongwang@vmware.com>; Ananyev, Konstantin
>>> <konstantin.ananyev@intel.com>
>>> Subject: Re: vmxnet3 no longer functional on DPDK 21.11
>>> 
>>> On Sun, Dec 05, 2021 at 07:52:33PM -0600, Lewis Donzis wrote:
>>> >
>>> >
>>> > ----- On Nov 30, 2021, at 7:42 AM, Bruce Richardson bruce.richardson@intel.com
>>> > wrote:
>>> >
>>> > > On Mon, Nov 29, 2021 at 02:45:15PM -0600, Lewis Donzis wrote:
>>> > >>    Hello.
>>> > >>    We just upgraded from 21.08 to 21.11 and it's rather astounding the
>>> > >>    number of incompatible changes in three months.  Not a big deal, just
>>> > >>    kind of a surprise, that's all.
>>> > >>    Anyway, the problem is that the vmxnet3 driver is no longer functional
>>> > >>    on FreeBSD.
>>> > >>    In drivers/net/vmxnet3/vmxnet3_ethdev.c, vmxnet3_dev_start() gets an
>>> > >>    error calling rte_intr_enable().  So it logs "interrupt enable failed"
>>> > >>    and returns an error.
>>> > >>    In lib/eal/freebsd/eal_interrupts.c, rte_intr_enable() is returning an
>>> > >>    error because rte_intr_dev_fd_get(intr_handle) is returning -1.
>>> > >>    I don't see how that could ever return anything other than -1 since it
>>> > >>    appears that there is no code that ever calls rte_intr_dev_fd_set()
>>> > >>    with a value other than -1 on FreeBSD.  Also weird to me is that even
>>> > >>    if it didn't get an error, the switch statement that follows looks like
>>> > >>    it will return an error in every case.
>>> > >>    Nonetheless, it worked in 21.08, and I can't quite see why the
>>> > >>    difference, so I must be missing something.
>>> > >>    For the moment, I just commented the "return -EIO" in vmxnet3_ethdev.c,
>>> > >>    and it's now working again, but that's obviously not the correct
>>> > >>    solution.
>>> > >>    Can someone who's knowledgable about this mechanism perhaps explain a
>>> > >>    little bit about what's going on?  I'll be happy to help troubleshoot.
>>> > >>    It seems like it must be something simple, but I just don't see it yet.
>>> > >
>>> > > Hi
>>> > >
>>> > > if you have the chance, it would be useful if you could use "git bisect" to
>>> > > identify the commit in 21.11 that broke this driver. Looking through the
>>> > > logs for 21.11 I can't identify any particular likely-looking commit, so
>>> > > bisect is likely a good way to start looking into this.
>>> > >
>>> > > Regards,
>>> > > /Bruce
>>> >
>>> > Hi, Bruce.  git bisect is very time-consuming and very cool!
>>> >
>>> > I went back to 21.08, about 1100 commits, and worked through the process, but
>>> > then I realized that I had forgotten to run ninja on one of
>>> the steps, so I did it again.
>>> >
>>> > I also re-checked it after the bisect, just to make sure that
>>> > c87d435a4d79739c0cec2ed280b94b41cb908af7 is good, and
>>> 7a0935239b9eb817c65c03554a9954ddb8ea5044 is bad.
>>> >
>>> > Thanks,
>>> > lew
>>> >
>>> 
>>> Many thanks for taking the time to do this. Adding Konstantin to thread as
>>> author of the commit you identified. Konstantin, any thoughts on this
>>> issue?
>> 
>> Hmm, that's looks really strange to me.
>> So to clarify, it fails at:
>> static int
>> vmxnet3_dev_start(struct rte_eth_dev *dev)
>> {
>>	...
>> line 1695:
>>	if (rte_intr_enable(dev->intr_handle) < 0) {
>>                PMD_INIT_LOG(ERR, "interrupt enable failed");
>>                return -EIO;
>>        }
>> 
>> Right?
>> 
>> The strange thing here is that 7a0935239b9e
>> doesn't change dev_start or rte_intr code in any way.
>> All it does - change rte_eth_rx_burst/rte_eth_tx_burst and other fast-path
>> functions.
>> Anyway, if git blames that commit, let's try to figure out what is going on.
>> Unfortunately, I don't have freebsd with vmxnet3, so will need to rely on your
>> help here.
>> As the first thing can you try to run testpmd build with last good commit
>> (c87d435a4d79)
>> and then testpmd build with bad commit applied and collect for both cases:
>> - contents of 'struct rte_eth_dev' and ' rte_eth_dev->intr_handle' for
>>  your vmxnet3 port
>> - debug log output (--log-level=eal,debug --log-level=pmd,debug)
>> 
> > Konstantin

Re: vmxnet3 no longer functional on DPDK 21.11

[Bruce Richardson]

On Sat, Jan 06, 2024 at 08:50:03AM -0600, Lewis Donzis wrote:
> Good morning.
> 
> I just wanted to mention that this problem still persists in 22.11.3, and we still have to patch the vmxnet3 driver every time we upgrade.
> 
> As mentioned before, ixgbe_ethdev.c is an example of a driver that ifdef's out the attempt to use interrupts on FreeBSD.
> 
> Thanks,
> lew
> 
> 
> ----- On Jun 3, 2022, at 8:19 AM, Lewis Donzis lew@perftech.com wrote:
> 
> > Hi, all.
> > 
> > Resurrecting this thread from six months ago, I apologize for not having more
> > time to dig into it, but in light of recent findings, I see numerous other
> > drivers and other parts of the code that have comments to the effect that
> > "FreeBSD doesn't support interrupts" and they effectively #ifdef out the
> > attempt.
> > 
> > Could this be as simple as needing to do the same in vmxnet3?  Empirically,
> > ignoring the error from rte_intr_enable() allows the driver to work normally,
> > for what that's worth.
> > 

I'm not at all familiar with the vmxnet3 driver, so apologies for the lack
of response up till now. Does something like the below simple fix work for
you? If so, I'm happy enough to submit as a patch for upstream merge and
then backport.

/Bruce

diff --git a/drivers/net/vmxnet3/vmxnet3_ethdev.c b/drivers/net/vmxnet3/vmxnet3_ethdev.c
index e49191718a..d088b42d35 100644
--- a/drivers/net/vmxnet3/vmxnet3_ethdev.c
+++ b/drivers/net/vmxnet3/vmxnet3_ethdev.c
@@ -1129,6 +1129,7 @@ vmxnet3_dev_start(struct rte_eth_dev *dev)
        /* Setting proper Rx Mode and issue Rx Mode Update command */
        vmxnet3_dev_set_rxmode(hw, VMXNET3_RXM_UCAST | VMXNET3_RXM_BCAST, 1);
 
+#ifndef RTE_EXEC_ENV_FREEBSD
        /* Setup interrupt callback  */
        rte_intr_callback_register(dev->intr_handle,
                                   vmxnet3_interrupt_handler, dev);
@@ -1140,6 +1141,7 @@ vmxnet3_dev_start(struct rte_eth_dev *dev)
 
        /* enable all intrs */
        vmxnet3_enable_all_intrs(hw);
+#endif
 
        vmxnet3_process_events(dev);
 

Re: vmxnet3 no longer functional on DPDK 21.11

[Lewis Donzis]

Hi, Bruce.

I'm even less familiar with it, but we do quite a lot of testing using VMs, so it's been quite handy.

Your patch seems very reasonable, however it also produces a warning:

../drivers/net/vmxnet3/vmxnet3_ethdev.c:264:1: warning: unused function 'vmxnet3_enable_all_intrs' [-Wunused-function]

Adding an #ifndef around vmxnet3_enable_all_intrs() eliminates that warning.

Please pardon the uninformed view, but we've been using FreeBSD + DPDK for nearly a decade, and I thought the whole point was to avoid using interrupts.  We have no need or desire for them in our applications, so we just hope the sprinkling of interrupt support code throughout the drivers doesn't cause any harm.  But I also realize we're probably in the minority on this.

Thanks for the help,
lew



> I'm not at all familiar with the vmxnet3 driver, so apologies for the lack
> of response up till now. Does something like the below simple fix work for
> you? If so, I'm happy enough to submit as a patch for upstream merge and
> then backport.
> 
> /Bruce
> 
> diff --git a/drivers/net/vmxnet3/vmxnet3_ethdev.c
> b/drivers/net/vmxnet3/vmxnet3_ethdev.c
> index e49191718a..d088b42d35 100644
> --- a/drivers/net/vmxnet3/vmxnet3_ethdev.c
> +++ b/drivers/net/vmxnet3/vmxnet3_ethdev.c
> @@ -1129,6 +1129,7 @@ vmxnet3_dev_start(struct rte_eth_dev *dev)
>        /* Setting proper Rx Mode and issue Rx Mode Update command */
>        vmxnet3_dev_set_rxmode(hw, VMXNET3_RXM_UCAST | VMXNET3_RXM_BCAST, 1);
> 
> +#ifndef RTE_EXEC_ENV_FREEBSD
>        /* Setup interrupt callback  */
>        rte_intr_callback_register(dev->intr_handle,
>                                   vmxnet3_interrupt_handler, dev);
> @@ -1140,6 +1141,7 @@ vmxnet3_dev_start(struct rte_eth_dev *dev)
> 
>        /* enable all intrs */
>        vmxnet3_enable_all_intrs(hw);
> +#endif
> 
>         vmxnet3_process_events(dev);

[PATCH] net/vmxnet3: fix use of interrupts on FreeBSD

[Bruce Richardson]

DPDK does not support interrupts on FreeBSD, so the vmxnet3 driver
returns error when enabling interrupts as it initializes. We can fix
this by #ifdef'ing out the interrupt calls when building for FreeBSD,
allowing the driver to initialize correctly.

Fixes: 046f11619567 ("net/vmxnet3: support MSI-X interrupt")

Reported-by: Lewis Donzis <lew@perftech.com>
Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
---
 drivers/net/vmxnet3/vmxnet3_ethdev.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/net/vmxnet3/vmxnet3_ethdev.c b/drivers/net/vmxnet3/vmxnet3_ethdev.c
index e49191718a..7032f0e324 100644
--- a/drivers/net/vmxnet3/vmxnet3_ethdev.c
+++ b/drivers/net/vmxnet3/vmxnet3_ethdev.c
@@ -257,6 +257,7 @@ vmxnet3_disable_all_intrs(struct vmxnet3_hw *hw)
 		vmxnet3_disable_intr(hw, i);
 }
 
+#ifndef RTE_EXEC_ENV_FREEBSD
 /*
  * Enable all intrs used by the device
  */
@@ -280,6 +281,7 @@ vmxnet3_enable_all_intrs(struct vmxnet3_hw *hw)
 			vmxnet3_enable_intr(hw, i);
 	}
 }
+#endif
 
 /*
  * Gets tx data ring descriptor size.
@@ -1129,6 +1131,7 @@ vmxnet3_dev_start(struct rte_eth_dev *dev)
 	/* Setting proper Rx Mode and issue Rx Mode Update command */
 	vmxnet3_dev_set_rxmode(hw, VMXNET3_RXM_UCAST | VMXNET3_RXM_BCAST, 1);
 
+#ifndef RTE_EXEC_ENV_FREEBSD
 	/* Setup interrupt callback  */
 	rte_intr_callback_register(dev->intr_handle,
 				   vmxnet3_interrupt_handler, dev);
@@ -1140,6 +1143,7 @@ vmxnet3_dev_start(struct rte_eth_dev *dev)
 
 	/* enable all intrs */
 	vmxnet3_enable_all_intrs(hw);
+#endif
 
 	vmxnet3_process_events(dev);
 
-- 
2.42.0

Re: vmxnet3 no longer functional on DPDK 21.11

[Bruce Richardson]

On Tue, Jan 09, 2024 at 07:46:47AM -0600, Lewis Donzis wrote:
> Hi, Bruce.
> 
> I'm even less familiar with it, but we do quite a lot of testing using VMs, so it's been quite handy.
> 
> Your patch seems very reasonable, however it also produces a warning:
> 
> ../drivers/net/vmxnet3/vmxnet3_ethdev.c:264:1: warning: unused function 'vmxnet3_enable_all_intrs' [-Wunused-function]
> 
> Adding an #ifndef around vmxnet3_enable_all_intrs() eliminates that warning.

Right, I should have compile-tested on FreeBSD myself, before sending the
suggestion. Patch has now been submitted. Please test and ack if the fix
works for your use-cases, thanks.

> 
> Please pardon the uninformed view, but we've been using FreeBSD + DPDK for nearly a decade, and I thought the whole point was to avoid using interrupts.  We have no need or desire for them in our applications, so we just hope the sprinkling of interrupt support code throughout the drivers doesn't cause any harm.  But I also realize we're probably in the minority on this.
> 
In general, yes we try and avoid interrupts on the data-path or fast-path
and use polling. However, for some use-cases where traffic levels are low,
interrupts may make sense to save power for fast-path. Even if not,
interrupts are useful for things like error conditions or for monitoring
link-status changes (LSC). Unfortunately, we don't have any interrupt
support on BSD, so fixes like this are necessary.

/Bruce

Re: vmxnet3 no longer functional on DPDK 21.11

[Lewis Donzis]

----- On Jan 9, 2024, at 8:28 AM, Bruce Richardson bruce.richardson@intel.com wrote:

> On Tue, Jan 09, 2024 at 07:46:47AM -0600, Lewis Donzis wrote:
>> Hi, Bruce.
>> 
>> I'm even less familiar with it, but we do quite a lot of testing using VMs, so
>> it's been quite handy.
>> 
>> Your patch seems very reasonable, however it also produces a warning:
>> 
>> ../drivers/net/vmxnet3/vmxnet3_ethdev.c:264:1: warning: unused function
>> 'vmxnet3_enable_all_intrs' [-Wunused-function]
>> 
>> Adding an #ifndef around vmxnet3_enable_all_intrs() eliminates that warning.
> 
> Right, I should have compile-tested on FreeBSD myself, before sending the
> suggestion. Patch has now been submitted. Please test and ack if the fix
> works for your use-cases, thanks.

I compiled it and ran it just now and it appears to work just fine.  Thanks very much for submitting.

> In general, yes we try and avoid interrupts on the data-path or fast-path
> and use polling. However, for some use-cases where traffic levels are low,
> interrupts may make sense to save power for fast-path. Even if not,
> interrupts are useful for things like error conditions or for monitoring
> link-status changes (LSC). Unfortunately, we don't have any interrupt
> support on BSD, so fixes like this are necessary.

That makes sense.  Makes me wonder why there's no interrupt support on BSD, i.e., maybe it's better to fix that than to have to fix "avoiding it" in the drivers?

I kind of feel like we're a bit orphaned in the FreeBSD world.  I don't know how many others are using BSD, but it seems like we're in a relatively less-supported environment.

Thanks again,
lew

Re: vmxnet3 no longer functional on DPDK 21.11

[Bruce Richardson]

On Tue, Jan 09, 2024 at 09:21:47AM -0600, Lewis Donzis wrote:
> 
> 
> ----- On Jan 9, 2024, at 8:28 AM, Bruce Richardson bruce.richardson@intel.com wrote:
> 
> > On Tue, Jan 09, 2024 at 07:46:47AM -0600, Lewis Donzis wrote:
> >> Hi, Bruce.
> >> 
> >> I'm even less familiar with it, but we do quite a lot of testing using VMs, so
> >> it's been quite handy.
> >> 
> >> Your patch seems very reasonable, however it also produces a warning:
> >> 
> >> ../drivers/net/vmxnet3/vmxnet3_ethdev.c:264:1: warning: unused function
> >> 'vmxnet3_enable_all_intrs' [-Wunused-function]
> >> 
> >> Adding an #ifndef around vmxnet3_enable_all_intrs() eliminates that warning.
> > 
> > Right, I should have compile-tested on FreeBSD myself, before sending the
> > suggestion. Patch has now been submitted. Please test and ack if the fix
> > works for your use-cases, thanks.
> 
> I compiled it and ran it just now and it appears to work just fine.  Thanks very much for submitting.
> 
> > In general, yes we try and avoid interrupts on the data-path or fast-path
> > and use polling. However, for some use-cases where traffic levels are low,
> > interrupts may make sense to save power for fast-path. Even if not,
> > interrupts are useful for things like error conditions or for monitoring
> > link-status changes (LSC). Unfortunately, we don't have any interrupt
> > support on BSD, so fixes like this are necessary.
> 
> That makes sense.  Makes me wonder why there's no interrupt support on BSD, i.e., maybe it's better to fix that than to have to fix "avoiding it" in the drivers?

Sadly, interrupt support was never implemented for BSD, which is why it is
missing. If someone has the time and interest to do up the patches to add
it, it would be a welcome addition to DPDK.

> 
> I kind of feel like we're a bit orphaned in the FreeBSD world.  I don't know how many others are using BSD, but it seems like we're in a relatively less-supported environment.
> 

Linux is the dominant environment for DPDK, which does mean it gets most of
the attention. However, we do want to keep supporting BSD, so please
continue to flag issues to the community as you encounter them. Smaller
bugs we can certainly endeavour to fix, but adding things like interrupt
support probably requires someone to explicitly step up and dedicate time
to implementing it properly.

Regards,
/Bruce

Re: [PATCH] net/vmxnet3: fix use of interrupts on FreeBSD

[Lewis Donzis]

----- On Jan 9, 2024, at 8:23 AM, Bruce Richardson bruce.richardson@intel.com wrote:

> DPDK does not support interrupts on FreeBSD, so the vmxnet3 driver
> returns error when enabling interrupts as it initializes. We can fix
> this by #ifdef'ing out the interrupt calls when building for FreeBSD,
> allowing the driver to initialize correctly.
> 
> Fixes: 046f11619567 ("net/vmxnet3: support MSI-X interrupt")
> 
> Reported-by: Lewis Donzis <lew@perftech.com>
Tested-by: Lewis Donzis <lew@perftech.com>
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> ---
> drivers/net/vmxnet3/vmxnet3_ethdev.c | 4 ++++
> 1 file changed, 4 insertions(+)
> 
> diff --git a/drivers/net/vmxnet3/vmxnet3_ethdev.c
> b/drivers/net/vmxnet3/vmxnet3_ethdev.c
> index e49191718a..7032f0e324 100644
> --- a/drivers/net/vmxnet3/vmxnet3_ethdev.c
> +++ b/drivers/net/vmxnet3/vmxnet3_ethdev.c
> @@ -257,6 +257,7 @@ vmxnet3_disable_all_intrs(struct vmxnet3_hw *hw)
> 		vmxnet3_disable_intr(hw, i);
> }
> 
> +#ifndef RTE_EXEC_ENV_FREEBSD
> /*
>  * Enable all intrs used by the device
>  */
> @@ -280,6 +281,7 @@ vmxnet3_enable_all_intrs(struct vmxnet3_hw *hw)
> 			vmxnet3_enable_intr(hw, i);
> 	}
> }
> +#endif
> 
> /*
>  * Gets tx data ring descriptor size.
> @@ -1129,6 +1131,7 @@ vmxnet3_dev_start(struct rte_eth_dev *dev)
> 	/* Setting proper Rx Mode and issue Rx Mode Update command */
> 	vmxnet3_dev_set_rxmode(hw, VMXNET3_RXM_UCAST | VMXNET3_RXM_BCAST, 1);
> 
> +#ifndef RTE_EXEC_ENV_FREEBSD
> 	/* Setup interrupt callback  */
> 	rte_intr_callback_register(dev->intr_handle,
> 				   vmxnet3_interrupt_handler, dev);
> @@ -1140,6 +1143,7 @@ vmxnet3_dev_start(struct rte_eth_dev *dev)
> 
> 	/* enable all intrs */
> 	vmxnet3_enable_all_intrs(hw);
> +#endif
> 
> 	vmxnet3_process_events(dev);
> 
> --
> 2.42.0

Re: vmxnet3 no longer functional on DPDK 21.11

[Stephen Hemminger]

On Tue, 9 Jan 2024 10:21:04 +0000
Bruce Richardson <bruce.richardson@intel.com> wrote:

> diff --git a/drivers/net/vmxnet3/vmxnet3_ethdev.c b/drivers/net/vmxnet3/vmxnet3_ethdev.c
> index e49191718a..d088b42d35 100644
> --- a/drivers/net/vmxnet3/vmxnet3_ethdev.c
> +++ b/drivers/net/vmxnet3/vmxnet3_ethdev.c
> @@ -1129,6 +1129,7 @@ vmxnet3_dev_start(struct rte_eth_dev *dev)
>         /* Setting proper Rx Mode and issue Rx Mode Update command */
>         vmxnet3_dev_set_rxmode(hw, VMXNET3_RXM_UCAST | VMXNET3_RXM_BCAST, 1);
>  
> +#ifndef RTE_EXEC_ENV_FREEBSD
>         /* Setup interrupt callback  */
>         rte_intr_callback_register(dev->intr_handle,
>                                    vmxnet3_interrupt_handler, dev);
> @@ -1140,6 +1141,7 @@ vmxnet3_dev_start(struct rte_eth_dev *dev)
>  
>         /* enable all intrs */
>         vmxnet3_enable_all_intrs(hw);
> +#endif
>  
>         vmxnet3_process_events(dev);
>  

Probably need to go further with this.
 - what about unreigster in vmxnet3_dev_stop
 - vmxnet3_interrupt_handler is then dead code, should it be #ifdef guarded?
 - and vmxnet3_dev_rx_queue_intr_enable/disable
 - and vmxnet3_enable_intr
 - and vmxnet3_configure_msix
 - and checks for rte_eth_intr_conf bits? in configure

Re: vmxnet3 no longer functional on DPDK 21.11

[Lewis Donzis]

----- On Jan 9, 2024, at 5:55 PM, Stephen Hemminger stephen@networkplumber.org wrote:
> Probably need to go further with this.
> - what about unreigster in vmxnet3_dev_stop
> - vmxnet3_interrupt_handler is then dead code, should it be #ifdef guarded?
> - and vmxnet3_dev_rx_queue_intr_enable/disable
> - and vmxnet3_enable_intr
> - and vmxnet3_configure_msix
>  - and checks for rte_eth_intr_conf bits? in configure

I wondered the same thing, but checking other drivers, there appears to be little provision for this.  Just as an example, ixgbe has a FREEBSD ifdef, but it doesn't bother with trying to avoid all interrupt code.  And hardly any other network drivers have FREEBSD ifdefs at all and just ignore errors from registering interrupt callbacks.

In addition, the FreeBSD EAL interrupt code appears to have stubs that return "false" for calls that question whether interrupts are enabled, e.g., rte_intr_dp_is_en(), or an error for calls that attempt to use interrupts.

It's interesting that it works as well as it does because most drivers appear to be enabling interrupts in the NIC hardware.  But if the interrupt remains masked in the APIC due to lack of support in freebsd/eal_interrupts.c, then perhaps it doesn't matter?

So while your suggestions seem quite well-founded, I don't see any equivalent provision in any other driver.  Apparently there is no harm in attempting to use interrupts on FreeBSD since they will never trigger anyway?

Re: [PATCH] net/vmxnet3: fix use of interrupts on FreeBSD

[Ferruh Yigit]

On 1/9/2024 4:00 PM, Lewis Donzis wrote:
> 
> 
> ----- On Jan 9, 2024, at 8:23 AM, Bruce Richardson bruce.richardson@intel.com wrote:
> 
>> DPDK does not support interrupts on FreeBSD, so the vmxnet3 driver
>> returns error when enabling interrupts as it initializes. We can fix
>> this by #ifdef'ing out the interrupt calls when building for FreeBSD,
>> allowing the driver to initialize correctly.
>>
>> Fixes: 046f11619567 ("net/vmxnet3: support MSI-X interrupt")
>>
>> Reported-by: Lewis Donzis <lew@perftech.com>
>> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
>> >
> Tested-by: Lewis Donzis <lew@perftech.com>
>

Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>

Applied to dpdk-next-net/main, thanks.

Re: [PATCH] net/vmxnet3: fix use of interrupts on FreeBSD

[Lewis Donzis]

Did this get checked in?

Just curious because I don't see it in 22.11.4 that was released yesterday, so wasn't sure when it would show up.

Thanks,
lew

----- On Jan 11, 2024, at 6:03 AM, Ferruh Yigit ferruh.yigit@amd.com wrote:

> On 1/9/2024 4:00 PM, Lewis Donzis wrote:
>> 
>> 
>> ----- On Jan 9, 2024, at 8:23 AM, Bruce Richardson bruce.richardson@intel.com
>> wrote:
>> 
>>> DPDK does not support interrupts on FreeBSD, so the vmxnet3 driver
>>> returns error when enabling interrupts as it initializes. We can fix
>>> this by #ifdef'ing out the interrupt calls when building for FreeBSD,
>>> allowing the driver to initialize correctly.
>>>
>>> Fixes: 046f11619567 ("net/vmxnet3: support MSI-X interrupt")
>>>
>>> Reported-by: Lewis Donzis <lew@perftech.com>
>>> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
>>> >
>> Tested-by: Lewis Donzis <lew@perftech.com>
>>
> 
> Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>
> 
> Applied to dpdk-next-net/main, thanks.

Re: [PATCH] net/vmxnet3: fix use of interrupts on FreeBSD

[Ferruh Yigit]

On 1/24/2024 12:34 PM, Lewis Donzis wrote:

> ----- On Jan 11, 2024, at 6:03 AM, Ferruh Yigit ferruh.yigit@amd.com wrote:
> 
>> On 1/9/2024 4:00 PM, Lewis Donzis wrote:
>>>
>>>
>>> ----- On Jan 9, 2024, at 8:23 AM, Bruce Richardson bruce.richardson@intel.com
>>> wrote:
>>>
>>>> DPDK does not support interrupts on FreeBSD, so the vmxnet3 driver
>>>> returns error when enabling interrupts as it initializes. We can fix
>>>> this by #ifdef'ing out the interrupt calls when building for FreeBSD,
>>>> allowing the driver to initialize correctly.
>>>>
>>>> Fixes: 046f11619567 ("net/vmxnet3: support MSI-X interrupt")
>>>>
>>>> Reported-by: Lewis Donzis <lew@perftech.com>
>>>> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
>>>>>
>>> Tested-by: Lewis Donzis <lew@perftech.com>
>>>
>>
>> Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>
>>
>> Applied to dpdk-next-net/main, thanks.
>
> Did this get checked in?
>
> Just curious because I don't see it in 22.11.4 that was released
yesterday, so wasn't sure when it would show up.
>
>

Hi Lew,

It has been merged into the 'next-net' subtree and will be subsequently
pulled into the 'main' tree. Eventually, it will be included in the
24.03 release.

LTS releases will backport the fix after the release of v24.03.

If you are interested in 22.11.# LTS, this fix is expected to be
included in v22.11.5 scheduled for release in April."

Re: [PATCH] net/vmxnet3: fix use of interrupts on FreeBSD

[Lewis Donzis]

----- On Jan 24, 2024, at 7:58 AM, Ferruh Yigit ferruh.yigit@amd.com wrote:

> On 1/24/2024 12:34 PM, Lewis Donzis wrote:
> 
>> ----- On Jan 11, 2024, at 6:03 AM, Ferruh Yigit ferruh.yigit@amd.com wrote:
>> 
>>> On 1/9/2024 4:00 PM, Lewis Donzis wrote:
>>>>
>>>>
>>>> ----- On Jan 9, 2024, at 8:23 AM, Bruce Richardson bruce.richardson@intel.com
>>>> wrote:
>>>>
>>>>> DPDK does not support interrupts on FreeBSD, so the vmxnet3 driver
>>>>> returns error when enabling interrupts as it initializes. We can fix
>>>>> this by #ifdef'ing out the interrupt calls when building for FreeBSD,
>>>>> allowing the driver to initialize correctly.
>>>>>
>>>>> Fixes: 046f11619567 ("net/vmxnet3: support MSI-X interrupt")
>>>>>
>>>>> Reported-by: Lewis Donzis <lew@perftech.com>
>>>>> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
>>>>>>
>>>> Tested-by: Lewis Donzis <lew@perftech.com>
>>>>
>>>
>>> Acked-by: Ferruh Yigit <ferruh.yigit@amd.com>
>>>
>>> Applied to dpdk-next-net/main, thanks.
>>
>> Did this get checked in?
>>
>> Just curious because I don't see it in 22.11.4 that was released
> yesterday, so wasn't sure when it would show up.
>>
>>
> 
> Hi Lew,
> 
> It has been merged into the 'next-net' subtree and will be subsequently
> pulled into the 'main' tree. Eventually, it will be included in the
> 24.03 release.
> 
> LTS releases will backport the fix after the release of v24.03.
> 
> If you are interested in 22.11.# LTS, this fix is expected to be
> included in v22.11.5 scheduled for release in April."

Perfect, thanks very much for the explanation.

lew