[dpdk-dev] reply: [disscussion] A problem about dpdk backup-mode bond switching with mlx4 VF devices

[dpdk-dev] reply: [disscussion] A problem about dpdk backup-mode bond switching with mlx4 VF devices

[chenchanghu]

Hi, Adrien Mazarguil,

Thanks for your reply very much.

It means in our  tests,  when ‘ifconfig eth7 down’,  the expect result is the bond primary netdevice will switch to eth8.

However,  we find the bond primary is not changed to eth8 in 19 times of 20 time tests., and it means the dpdk bond doesn't receive an LSC interrupt signal.

In our test, the netdevice is mlx4 VF in Virtual Machine, which netdevice is  direct through by SR-IOV.



We also test mlx4 PF in Physical Machine, when ‘ifconfig eth7 down’,  the test esult is the bond primary netdevice  switched to eth8.



Quesetion:

Is it related with SR-IOV direct through ? For example, the VF netdevice status changed, but it will not send an LSC interruput signal.

Looking forward to your any reply.









-----邮件原件-----
发件人: Adrien Mazarguil [mailto:adrien.mazarguil@6wind.com]
发送时间: 2018年4月17日 18:08
收件人: chenchanghu <chenchanghu@huawei.com>
抄送: dev@dpdk.org; nelio.laranjeiro@6wind.com; Zhoujingbin (Robin, Cloud Networking) <zhoujingbin@huawei.com>; Zhoulei (G) <stone.zhou@huawei.com>; yangleyuan <yangleyuan@huawei.com>
主题: Re: [disscussion] A problem about dpdk backup-mode bond switching with mlx4 VF devices



On Tue, Apr 17, 2018 at 06:40:20AM +0000, chenchanghu wrote:

>

> Hi,

>          When I used the mlx4 pmd, I meet a problem about mlx4 VF bond switching which bond mod is backup-mode . The detail test is descripted below.

> 1.Test environmemt infomation:

>   a. Linux distribution: CentOS

>   b. dpdk version: dpdk-16.04

>   c. Ethernet device : mlx4 VF

>   d. pmd info: mlx4 poll-mode-driver

>

> 2.Test step:

>   a. we bond the mlx4 VF Ethernet device eth7,eth8 into backup-mode by dpdk application. Eth7 and eth8 are both active, and  eth7 is the primary device.

>   b. As we know,  the device eth7 , eth8 are also  visible by kernel driver mlx4_en.

>   c. Then we config the Ethernet device eth7 down by the command ' ifconfig eth7 down', the expect result is the bond primary device will  not switch.

>   d. However we find the dpdk bond primary device switch to eth8 by dpdk maintenance interface one time in all 20 test times.

>

> 3.Question:

>    Is the VF up or down State of kernel interface has some relations to user-space state? For example, when ifconfig eth7 down, and the user-space will change to down state too.



Yes, this is expected. Netdevices and the mlx4 DPDK PMD share a common link status. Bringing a netdevice down causes link status to be down for all its users. This behavior is documented [1].



[1] http://dpdk.org/doc/guides/nics/mlx4.html#run-time-configuration



--

Adrien Mazarguil

6WIND

Re: [dpdk-dev] reply: [disscussion] A problem about dpdk backup-mode bond switching with mlx4 VF devices

[Adrien Mazarguil]

On Wed, Apr 18, 2018 at 01:55:41AM +0000, chenchanghu wrote:
> Hi, Adrien Mazarguil,
> 
> Thanks for your reply very much.
> 
> It means in our  tests,  when ‘ifconfig eth7 down’,  the expect result is the bond primary netdevice will switch to eth8.
> 
> However,  we find the bond primary is not changed to eth8 in 19 times of 20 time tests., and it means the dpdk bond doesn't receive an LSC interrupt signal.
> 
> In our test, the netdevice is mlx4 VF in Virtual Machine, which netdevice is  direct through by SR-IOV.

I overlooked this part, I'm not sure VFs impact link status on the DPDK side
since they can't really bring it down. Regardless, you should have
encountered an identical behavior 20 out of 20 times otherwise there could
be a bug in the DPDK version you're using.

> We also test mlx4 PF in Physical Machine, when ‘ifconfig eth7 down’,  the test esult is the bond primary netdevice  switched to eth8.

Right, PF has authority over link status, this is expected as documented.

> Quesetion:
> 
> Is it related with SR-IOV direct through ? For example, the VF netdevice status changed, but it will not send an LSC interruput signal.
> 
> Looking forward to your any reply.

To summarize, updating the link status of a netdevice associated with a VF
shouldn't impact a DPDK application. On the other hand updating it on a
netdevice associated with PF will impact all VFs and their applications.

Please check again with a more recent DPDK version (e.g. 18.02). If you
manage to get a consistent behavior every time, it means a bug is present in
in 16.04.

> -----邮件原件-----
> 发件人: Adrien Mazarguil [mailto:adrien.mazarguil@6wind.com]
> 发送时间: 2018年4月17日 18:08
> 收件人: chenchanghu <chenchanghu@huawei.com>
> 抄送: dev@dpdk.org; nelio.laranjeiro@6wind.com; Zhoujingbin (Robin, Cloud Networking) <zhoujingbin@huawei.com>; Zhoulei (G) <stone.zhou@huawei.com>; yangleyuan <yangleyuan@huawei.com>
> 主题: Re: [disscussion] A problem about dpdk backup-mode bond switching with mlx4 VF devices
> 
> 
> 
> On Tue, Apr 17, 2018 at 06:40:20AM +0000, chenchanghu wrote:
> 
> >
> 
> > Hi,
> 
> >          When I used the mlx4 pmd, I meet a problem about mlx4 VF bond switching which bond mod is backup-mode . The detail test is descripted below.
> 
> > 1.Test environmemt infomation:
> 
> >   a. Linux distribution: CentOS
> 
> >   b. dpdk version: dpdk-16.04
> 
> >   c. Ethernet device : mlx4 VF
> 
> >   d. pmd info: mlx4 poll-mode-driver
> 
> >
> 
> > 2.Test step:
> 
> >   a. we bond the mlx4 VF Ethernet device eth7,eth8 into backup-mode by dpdk application. Eth7 and eth8 are both active, and  eth7 is the primary device.
> 
> >   b. As we know,  the device eth7 , eth8 are also  visible by kernel driver mlx4_en.
> 
> >   c. Then we config the Ethernet device eth7 down by the command ' ifconfig eth7 down', the expect result is the bond primary device will  not switch.
> 
> >   d. However we find the dpdk bond primary device switch to eth8 by dpdk maintenance interface one time in all 20 test times.
> 
> >
> 
> > 3.Question:
> 
> >    Is the VF up or down State of kernel interface has some relations to user-space state? For example, when ifconfig eth7 down, and the user-space will change to down state too.
> 
> 
> 
> Yes, this is expected. Netdevices and the mlx4 DPDK PMD share a common link status. Bringing a netdevice down causes link status to be down for all its users. This behavior is documented [1].
> 
> 
> 
> [1] http://dpdk.org/doc/guides/nics/mlx4.html#run-time-configuration
> 
> 
> 
> --
> 
> Adrien Mazarguil
> 
> 6WIND

-- 
Adrien Mazarguil
6WIND

[dpdk-dev] 答复: reply: [disscussion] A problem about dpdk backup-mode bond switching with mlx4 VF devices

[chenchanghu]

Thanks very much, OK, we will test again with recent dpdk version.

-----邮件原件-----
发件人: Adrien Mazarguil [mailto:adrien.mazarguil@6wind.com] 
发送时间: 2018年4月18日 17:05
收件人: chenchanghu <chenchanghu@huawei.com>
抄送: dev@dpdk.org; nelio.laranjeiro@6wind.com; Zhoujingbin (Robin, Cloud Networking) <zhoujingbin@huawei.com>; Zhoulei (G) <stone.zhou@huawei.com>; yangleyuan <yangleyuan@huawei.com>
主题: Re: reply: [disscussion] A problem about dpdk backup-mode bond switching with mlx4 VF devices

On Wed, Apr 18, 2018 at 01:55:41AM +0000, chenchanghu wrote:
> Hi, Adrien Mazarguil,
> 
> Thanks for your reply very much.
> 
> It means in our  tests,  when ‘ifconfig eth7 down’,  the expect result is the bond primary netdevice will switch to eth8.
> 
> However,  we find the bond primary is not changed to eth8 in 19 times of 20 time tests., and it means the dpdk bond doesn't receive an LSC interrupt signal.
> 
> In our test, the netdevice is mlx4 VF in Virtual Machine, which netdevice is  direct through by SR-IOV.

I overlooked this part, I'm not sure VFs impact link status on the DPDK side since they can't really bring it down. Regardless, you should have encountered an identical behavior 20 out of 20 times otherwise there could be a bug in the DPDK version you're using.

> We also test mlx4 PF in Physical Machine, when ‘ifconfig eth7 down’,  the test esult is the bond primary netdevice  switched to eth8.

Right, PF has authority over link status, this is expected as documented.

> Quesetion:
> 
> Is it related with SR-IOV direct through ? For example, the VF netdevice status changed, but it will not send an LSC interruput signal.
> 
> Looking forward to your any reply.

To summarize, updating the link status of a netdevice associated with a VF shouldn't impact a DPDK application. On the other hand updating it on a netdevice associated with PF will impact all VFs and their applications.

Please check again with a more recent DPDK version (e.g. 18.02). If you manage to get a consistent behavior every time, it means a bug is present in in 16.04.

> -----邮件原件-----
> 发件人: Adrien Mazarguil [mailto:adrien.mazarguil@6wind.com]
> 发送时间: 2018年4月17日 18:08
> 收件人: chenchanghu <chenchanghu@huawei.com>
> 抄送: dev@dpdk.org; nelio.laranjeiro@6wind.com; Zhoujingbin (Robin, 
> Cloud Networking) <zhoujingbin@huawei.com>; Zhoulei (G) 
> <stone.zhou@huawei.com>; yangleyuan <yangleyuan@huawei.com>
> 主题: Re: [disscussion] A problem about dpdk backup-mode bond switching 
> with mlx4 VF devices
> 
> 
> 
> On Tue, Apr 17, 2018 at 06:40:20AM +0000, chenchanghu wrote:
> 
> >
> 
> > Hi,
> 
> >          When I used the mlx4 pmd, I meet a problem about mlx4 VF bond switching which bond mod is backup-mode . The detail test is descripted below.
> 
> > 1.Test environmemt infomation:
> 
> >   a. Linux distribution: CentOS
> 
> >   b. dpdk version: dpdk-16.04
> 
> >   c. Ethernet device : mlx4 VF
> 
> >   d. pmd info: mlx4 poll-mode-driver
> 
> >
> 
> > 2.Test step:
> 
> >   a. we bond the mlx4 VF Ethernet device eth7,eth8 into backup-mode by dpdk application. Eth7 and eth8 are both active, and  eth7 is the primary device.
> 
> >   b. As we know,  the device eth7 , eth8 are also  visible by kernel driver mlx4_en.
> 
> >   c. Then we config the Ethernet device eth7 down by the command ' ifconfig eth7 down', the expect result is the bond primary device will  not switch.
> 
> >   d. However we find the dpdk bond primary device switch to eth8 by dpdk maintenance interface one time in all 20 test times.
> 
> >
> 
> > 3.Question:
> 
> >    Is the VF up or down State of kernel interface has some relations to user-space state? For example, when ifconfig eth7 down, and the user-space will change to down state too.
> 
> 
> 
> Yes, this is expected. Netdevices and the mlx4 DPDK PMD share a common link status. Bringing a netdevice down causes link status to be down for all its users. This behavior is documented [1].
> 
> 
> 
> [1] http://dpdk.org/doc/guides/nics/mlx4.html#run-time-configuration
> 
> 
> 
> --
> 
> Adrien Mazarguil
> 
> 6WIND

--
Adrien Mazarguil
6WIND