[dpdk-users] CX4-Lx VF link status in Azure

[dpdk-users] CX4-Lx VF link status in Azure

[Benoit Ganne (bganne)]

Hi all,

We noticed a problematic behavior when using DPDK in Azure with Mellanox CX4-Lx VFs on Ubuntu 18.04: the link status (link up or down) is never updated - and so for an application trusting the link state reported by DPDK, the link looks unavailable.
The root cause is mlx5_link_update_unlocked_gs() failing to get the link speed from the Linux driver and always returning EAGAIN because of this test at the end of the function:

    if (((dev_link.link_speed && !dev_link.link_status) ||
         (!dev_link.link_speed && dev_link.link_status)))

Here, dev_link.link_status is correctly set but not dev_link.link_speed.
The Linux kernel driver does not seem to be able to get the link speed because ethtool also fails to retrieve it from the kernel so the DPDK cannot do any miracle here, but I wonder whether this should prevent to update the other link status information to be correctly updated.

Any thoughts?

Thx
ben

Re: [dpdk-users] CX4-Lx VF link status in Azure

[Stephen Hemminger]

On Wed, 25 Mar 2020 19:07:09 +0000
"Benoit Ganne (bganne)" <bganne@cisco.com> wrote:

> Hi all,
> 
> We noticed a problematic behavior when using DPDK in Azure with Mellanox CX4-Lx VFs on Ubuntu 18.04: the link status (link up or down) is never updated - and so for an application trusting the link state reported by DPDK, the link looks unavailable.
> The root cause is mlx5_link_update_unlocked_gs() failing to get the link speed from the Linux driver and always returning EAGAIN because of this test at the end of the function:
> 
>     if (((dev_link.link_speed && !dev_link.link_status) ||
>          (!dev_link.link_speed && dev_link.link_status)))
> 
> Here, dev_link.link_status is correctly set but not dev_link.link_speed.
> The Linux kernel driver does not seem to be able to get the link speed because ethtool also fails to retrieve it from the kernel so the DPDK cannot do any miracle here, but I wonder whether this should prevent to update the other link status information to be correctly updated.
> 
> Any thoughts?
> 
> Thx
> ben

Is this with netvsc PMD or failsafe PMD?

Re: [dpdk-users] CX4-Lx VF link status in Azure

[Stephen Hemminger]

On Wed, 25 Mar 2020 19:07:09 +0000
"Benoit Ganne (bganne)" <bganne@cisco.com> wrote:

> Hi all,
> 
> We noticed a problematic behavior when using DPDK in Azure with Mellanox CX4-Lx VFs on Ubuntu 18.04: the link status (link up or down) is never updated - and so for an application trusting the link state reported by DPDK, the link looks unavailable.
> The root cause is mlx5_link_update_unlocked_gs() failing to get the link speed from the Linux driver and always returning EAGAIN because of this test at the end of the function:
> 
>     if (((dev_link.link_speed && !dev_link.link_status) ||
>          (!dev_link.link_speed && dev_link.link_status)))
> 
> Here, dev_link.link_status is correctly set but not dev_link.link_speed.
> The Linux kernel driver does not seem to be able to get the link speed because ethtool also fails to retrieve it from the kernel so the DPDK cannot do any miracle here, but I wonder whether this should prevent to update the other link status information to be correctly updated.
> 
> Any thoughts?
> 
> Thx
> ben

You maybe missing this patch, which is only in current development branch.
Since it is tagged for stable, it should end up in later LTS versions as well 18.11.X and 19.11.X.

commit 047ad3787a2f5d99277e0d8d756580a1d5ea2891
Author: Stephen Hemminger <stephen@networkplumber.org>
Date:   Fri Feb 7 10:08:16 2020 -0800

    net/netvsc: initialize link state
    
    If application is using link state interrupt, the correct link state
    needs to be filled in when device is started. This is similar to
    how virtio updates link information.
    
    Fixes: 4e9c73e96e83 ("net/netvsc: add Hyper-V network device")
    Cc: stable@dpdk.org
    
    Reported-by: Mohammed Gamal <mgamal@redhat.com>
    Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
    Tested-by: Mohammed Gamal <mgamal@redhat.com>

Re: [dpdk-users] CX4-Lx VF link status in Azure

[Benoit Ganne (bganne)]

Hi Stephen,

> Is this with netvsc PMD or failsafe PMD?

I am using failsafe PMD using the string "--vdev net_vdev_netvsc0,iface=eth1" etc. as mentioned here: https://docs.microsoft.com/en-us/azure/virtual-network/setup-dpdk
I checked with gdb what are the underlying devices and there is 1 mlx5 and 1 tap instance as expected.
To get the link state, the call stack is rte_eth_link_get_nowait() -> fs_link_update() -> mlx5_link_update() -> mlx5_link_update_unlocked_gs() which looks good to me.
The link state update fails in mlx5_link_update_unlocked_gs() because the link speed retrieved from the Linux kernel driver is '0' (unknown). Note that ethtool and '/sys/class/net/<iface>/speed' also fails to report the link speed (but not the link status).
At the end of the day, maybe the Linux kernel driver should report a link speed, however I think it should not prevent DPDK to update the link state (up/down). 
Just removing the offending test makes everything working again but I do not think it is the correct solution: I do not know why this test was added for.

> You maybe missing this patch, which is only in current development branch.
> Since it is tagged for stable, it should end up in later LTS versions as
> well 18.11.X and 19.11.X.

I tried the patch but it did not solve the issue. I think it is expected as I am using failsafe with tap and not netvsc, correct?

Best
ben

Re: [dpdk-users] CX4-Lx VF link status in Azure

[Stephen Hemminger]

On Thu, 26 Mar 2020 14:26:56 +0000
"Benoit Ganne (bganne)" <bganne@cisco.com> wrote:

> Hi Stephen,
> 
> > Is this with netvsc PMD or failsafe PMD?  
> 
> I am using failsafe PMD using the string "--vdev net_vdev_netvsc0,iface=eth1" etc. as mentioned here: https://docs.microsoft.com/en-us/azure/virtual-network/setup-dpdk
> I checked with gdb what are the underlying devices and there is 1 mlx5 and 1 tap instance as expected.
> To get the link state, the call stack is rte_eth_link_get_nowait() -> fs_link_update() -> mlx5_link_update() -> mlx5_link_update_unlocked_gs() which looks good to me.
> The link state update fails in mlx5_link_update_unlocked_gs() because the link speed retrieved from the Linux kernel driver is '0' (unknown). Note that ethtool and '/sys/class/net/<iface>/speed' also fails to report the link speed (but not the link status).
> At the end of the day, maybe the Linux kernel driver should report a link speed, however I think it should not prevent DPDK to update the link state (up/down). 
> Just removing the offending test makes everything working again but I do not think it is the correct solution: I do not know why this test was added for.
> 
> > You maybe missing this patch, which is only in current development branch.
> > Since it is tagged for stable, it should end up in later LTS versions as
> > well 18.11.X and 19.11.X.  
> 
> I tried the patch but it did not solve the issue. I think it is expected as I am using failsafe with tap and not netvsc, correct?
> 
> Best
> ben


Is the Mellanox device being brought up by the base kernel setup?
I find that for Mellanox the device has to be started from kernel (like ip)
and DPDK doesn't do itself.

Re: [dpdk-users] CX4-Lx VF link status in Azure

[Benoit Ganne (bganne)]

> Is the Mellanox device being brought up by the base kernel setup?
> I find that for Mellanox the device has to be started from kernel (like
> ip) and DPDK doesn't do itself.

Yes everything is initialized correctly. The netdev itself is configured and usable from Linux (ping etc.). Just removing the over-strict check in mlx5 PMD is enough for everything to work fine: https://gerrit.fd.io/r/c/vpp/+/26152/1/build/external/patches/dpdk_20.02/0002-mlx5-azure-workaround.patch
The link speed is unknown but this is not issue, and link state and other link info are correctly reported.
Thomas, any input regarding this behavior in mlx5 PMD?

Thx
ben

Re: [dpdk-users] CX4-Lx VF link status in Azure

[Thomas Monjalon]

Pasting back this important info:
"
Note that ethtool and '/sys/class/net/<iface>/speed' also fails
to report the link speed (but not the link status).
"

26/03/2020 19:27, Benoit Ganne (bganne):
> Yes everything is initialized correctly. The netdev itself is configured and usable from Linux (ping etc.). Just removing the over-strict check in mlx5 PMD is enough for everything to work fine: https://gerrit.fd.io/r/c/vpp/+/26152/1/build/external/patches/dpdk_20.02/0002-mlx5-azure-workaround.patch
> The link speed is unknown but this is not issue, and link state and other link info are correctly reported.
> Thomas, any input regarding this behavior in mlx5 PMD?

I am not aware about the lack of link speed info.
It is probably not specific to ConnectX-4 Lx.
I guess it happens only with Hyper-V?

Cc mlx5 maintainers

Re: [dpdk-users] CX4-Lx VF link status in Azure

[Benoit Ganne (bganne)]

> Pasting back this important info:
> "
> Note that ethtool and '/sys/class/net/<iface>/speed' also fails
> to report the link speed (but not the link status).
> "
> 
> 26/03/2020 19:27, Benoit Ganne (bganne):
> > Yes everything is initialized correctly. The netdev itself is configured
> and usable from Linux (ping etc.). Just removing the over-strict check in
> mlx5 PMD is enough for everything to work fine:
> https://gerrit.fd.io/r/c/vpp/+/26152/1/build/external/patches/dpdk_20.02/0
> 002-mlx5-azure-workaround.patch
> > The link speed is unknown but this is not issue, and link state and
> other link info are correctly reported.
> > Thomas, any input regarding this behavior in mlx5 PMD?
> 
> I am not aware about the lack of link speed info.
> It is probably not specific to ConnectX-4 Lx.
> I guess it happens only with Hyper-V?

For me there are 2 separate issues:
 1) Linux kernel driver does not report link speed in Azure for CX4-Lx in Ubuntu 18.04
 2) mlx5 PMD enforce that both link speed is defined and link is up to update interface state

If (1) is fixed, (2) should work, but to me (2) is too strict for no good reason: we do not really care about reported link speed, esp in a virtual environment it usually does not mean much, but we do care about link state.

ben

Re: [dpdk-users] CX4-Lx VF link status in Azure

[Mark Bloch]

On 3/26/2020 12:00, Benoit Ganne (bganne) wrote:
>> Pasting back this important info:
>> "
>> Note that ethtool and '/sys/class/net/<iface>/speed' also fails
>> to report the link speed (but not the link status).
>> "
>>
>> 26/03/2020 19:27, Benoit Ganne (bganne):
>>> Yes everything is initialized correctly. The netdev itself is configured
>> and usable from Linux (ping etc.). Just removing the over-strict check in
>> mlx5 PMD is enough for everything to work fine:
>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgerrit.fd.io%2Fr%2Fc%2Fvpp%2F%2B%2F26152%2F1%2Fbuild%2Fexternal%2Fpatches%2Fdpdk_20.02%2F0&amp;data=02%7C01%7Cmarkb%40mellanox.com%7Ccf7438ae7d924c03b03308d7d1b7fd83%7Ca652971c7d2e4d9ba6a4d149256f461b%7C0%7C0%7C637208460478847341&amp;sdata=fV4wrAvzmHNM88mfCr6lxFXG0ANVOP0rBjfovZii84c%3D&amp;reserved=0
>> 002-mlx5-azure-workaround.patch
>>> The link speed is unknown but this is not issue, and link state and
>> other link info are correctly reported.
>>> Thomas, any input regarding this behavior in mlx5 PMD?
>>
>> I am not aware about the lack of link speed info.
>> It is probably not specific to ConnectX-4 Lx.
>> I guess it happens only with Hyper-V?

Should be fixed by those 3 commits (last 1 one is just cosmetic):

https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=dc392fc56f39a00a46d6db2d150571ccafe99734
https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=c268ca6087f553bfc0e16ffec412b983ffe32fd4
https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=2f5438ca0ee01a1b3a9c37e3f33d47c8122afe74

Mark

> 
> For me there are 2 separate issues:
>  1) Linux kernel driver does not report link speed in Azure for CX4-Lx in Ubuntu 18.04
>  2) mlx5 PMD enforce that both link speed is defined and link is up to update interface state
> 
> If (1) is fixed, (2) should work, but to me (2) is too strict for no good reason: we do not really care about reported link speed, esp in a virtual environment it usually does not mean much, but we do care about link state.
> 
> ben
> 

Re: [dpdk-users] CX4-Lx VF link status in Azure

[Thomas Monjalon]

26/03/2020 21:09, Mark Bloch:
> 
> On 3/26/2020 12:00, Benoit Ganne (bganne) wrote:
> >> Pasting back this important info:
> >> "
> >> Note that ethtool and '/sys/class/net/<iface>/speed' also fails
> >> to report the link speed (but not the link status).
> >> "
> >>
> >> 26/03/2020 19:27, Benoit Ganne (bganne):
> >>> Yes everything is initialized correctly. The netdev itself is configured
> >> and usable from Linux (ping etc.). Just removing the over-strict check in
> >> mlx5 PMD is enough for everything to work fine:
> >> https://gerrit.fd.io/r/c/vpp/+/26152/1/build/external/patches/dpdk_20.02/0002-mlx5-azure-workaround.patch
> >>> The link speed is unknown but this is not issue, and link state and
> >> other link info are correctly reported.
> >>> Thomas, any input regarding this behavior in mlx5 PMD?
> >>
> >> I am not aware about the lack of link speed info.
> >> It is probably not specific to ConnectX-4 Lx.
> >> I guess it happens only with Hyper-V?
> 
> Should be fixed by those 3 commits (last 1 one is just cosmetic):
> 
> https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=dc392fc56f39a00a46d6db2d150571ccafe99734
> https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=c268ca6087f553bfc0e16ffec412b983ffe32fd4
> https://git.kernel.org/pub/scm/linux/kernel/git/netdev/net-next.git/commit/?id=2f5438ca0ee01a1b3a9c37e3f33d47c8122afe74

Thanks for the patches Mark.

> > For me there are 2 separate issues:
> >  1) Linux kernel driver does not report link speed in Azure for CX4-Lx in Ubuntu 18.04

1 looks to be addressed with patches above.

> >  2) mlx5 PMD enforce that both link speed is defined and link is up to update interface state

Yes we can look at this issue.

> > If (1) is fixed, (2) should work, but to me (2) is too strict
> > for no good reason: we do not really care about reported link speed,

I agree that link speed is less important than link status.

> > esp in a virtual environment it usually does not mean much,

Yes the link speed is shared between all VFs.

> > but we do care about link state.

Re: [dpdk-users] CX4-Lx VF link status in Azure

[Thomas Monjalon]

On 3/26/2020 12:00, Benoit Ganne (bganne) wrote:
> Just removing the over-strict check in mlx5 PMD is enough for everything to work fine:
> https://gerrit.fd.io/r/c/vpp/+/26152/1/build/external/patches/dpdk_20.02/0002-mlx5-azure-workaround.patch
[...]
>  2) mlx5 PMD enforce that both link speed is defined and link is up to update interface state

The original commit introducing this logic is:
http://git.dpdk.org/dpdk/commit/?id=cfee94752b8f8f

I would say that the first issue is a lack of comment in this code.

Second, as Benoit said, we should relax this requirement.
If the link speed is unknown, a second request can be tried, no more.

Benoit, feel free to submit a patch showing how you think it should behave.
Otherwise, I guess a maintainer of mlx5 will try to arrange it later.
Note: a patch (even not perfect) is usually speeding up resolution.

Thanks

Re: [dpdk-users] CX4-Lx VF link status in Azure

[Benoit Ganne (bganne)]

> Second, as Benoit said, we should relax this requirement.
> If the link speed is unknown, a second request can be tried, no more.
> Benoit, feel free to submit a patch showing how you think it should
> behave.
> Otherwise, I guess a maintainer of mlx5 will try to arrange it later.
> Note: a patch (even not perfect) is usually speeding up resolution.

I can do that, but I am not sure I understand the logic of this test to begin with: looking into other PMD (mlx4, i40e), it seems to be the only one worrying about updating link state only when "ready" for some not clear (to me) definition of "ready".
I'll tend to agree with other PMD here: if the syscalls did not failed we should just update with what we know.
Why was this test introduced and what did it fixed?

ben

Re: [dpdk-users] CX4-Lx VF link status in Azure

[Thomas Monjalon]

27/03/2020 11:02, Benoit Ganne (bganne):
> > Second, as Benoit said, we should relax this requirement.
> > If the link speed is unknown, a second request can be tried, no more.
> > Benoit, feel free to submit a patch showing how you think it should
> > behave.
> > Otherwise, I guess a maintainer of mlx5 will try to arrange it later.
> > Note: a patch (even not perfect) is usually speeding up resolution.
> 
> I can do that, but I am not sure I understand the logic of this test to begin with: looking into other PMD (mlx4, i40e), it seems to be the only one worrying about updating link state only when "ready" for some not clear (to me) definition of "ready".
> I'll tend to agree with other PMD here: if the syscalls did not failed we should just update with what we know.
> Why was this test introduced and what did it fixed?

Unfortunately the reason was not documented.
I suggest we go with a patch from your understanding
and we'll test it in multiple conditions to validate nothing is broken.

Re: [dpdk-users] CX4-Lx VF link status in Azure

[Benoit Ganne (bganne)]

> Unfortunately the reason was not documented.
> I suggest we go with a patch from your understanding
> and we'll test it in multiple conditions to validate nothing is broken.

Done: http://mails.dpdk.org/archives/dev/2020-March/161096.html

Thx
ben

Re: [dpdk-users] CX4-Lx VF link status in Azure

[Thomas Monjalon]

27/03/2020 18:26, Benoit Ganne (bganne):
> > Unfortunately the reason was not documented.
> > I suggest we go with a patch from your understanding
> > and we'll test it in multiple conditions to validate nothing is broken.
> 
> Done: http://mails.dpdk.org/archives/dev/2020-March/161096.html

Very good, thanks.