[dpdk-dev] [sriov] [ixgbevf] We have to down PF and VF port to reconfig a port?

[dpdk-dev] [sriov] [ixgbevf] We have to down PF and VF port to reconfig a port?

[Sam]

Hi all,

I'm running OVS-DPDK on a two netdevice server(eth0 and eth1), and I use
sriov to make one VF and one PF on each netdevice, as bellow. Please omit
eth2 and eth3. Then two VF port(enp1s16 and enp1s16f1) combine into linux
bond1 port as management port, two PF port(0000:01:00.0 and 0000:01:00.1)
combine into OVS-DPDK bond port(mode 4) as dataplane port.

[root@yf-mos-test-net14 ~]#
> /usr/local/share/openvswitch/scripts/dpdk_nic_bind --status
> Network devices using DPDK-compatible driver
> ============================================
> 0000:01:00.0 'Ethernet Controller 10-Gigabit X540-AT2' drv=igb_uio
> unused=ixgbe
> 0000:01:00.1 'Ethernet Controller 10-Gigabit X540-AT2' drv=igb_uio
> unused=ixgbe
> Network devices using kernel driver
> ===================================
> 0000:01:10.0 'X540 Ethernet Controller Virtual Function' if=enp1s16
> drv=ixgbevf unused=igb_uio
> 0000:01:10.1 'X540 Ethernet Controller Virtual Function' if=enp1s16f1
> drv=ixgbevf unused=igb_uio
> 0000:08:00.0 'I350 Gigabit Network Connection' if=eth2 drv=igb
> unused=igb_uio
> 0000:08:00.1 'I350 Gigabit Network Connection' if=eth3 drv=igb
> unused=igb_uio
> Other network devices
> =====================
> <none>


I change OVS-DPDK code, and use command to stop  OVS-DPDK bond port and its
slaves, and then change its txconf and rxconf, and then start up them. But
the kernel report "Detected Tx Unit Hang ..." and no packets could go
through OVS-DPDK bond port and linux bond port, but use command to show
OVS-DPDK port and linux bond port, all status is OK.

But when I stop linux bond port first, and then do the same process as
above, no error report and packets could go through OVS-DPDK bond port and
linux bond port.

So what's this problem and how to fix it ? Thank you~

Re: [dpdk-dev] [sriov] [ixgbevf] We have to down PF and VF port to reconfig a port?

[Sam]

2017-12-27 10:42 GMT+08:00 Sam <batmanustc@gmail.com>:

> Hi all,
>
> I'm running OVS-DPDK on a two netdevice server(eth0 and eth1), and I use
> sriov to make one VF and one PF on each netdevice, as bellow. Please omit
> eth2 and eth3. Then two VF port(enp1s16 and enp1s16f1) combine into linux
> bond1 port as management port, two PF port(0000:01:00.0 and 0000:01:00.1)
> combine into OVS-DPDK bond port(mode 4) as dataplane port.
>
> [root@yf-mos-test-net14 ~]# /usr/local/share/openvswitch/scripts/dpdk_nic_bind
>> --status
>> Network devices using DPDK-compatible driver
>> ============================================
>> 0000:01:00.0 'Ethernet Controller 10-Gigabit X540-AT2' drv=igb_uio
>> unused=ixgbe
>> 0000:01:00.1 'Ethernet Controller 10-Gigabit X540-AT2' drv=igb_uio
>> unused=ixgbe
>> Network devices using kernel driver
>> ===================================
>> 0000:01:10.0 'X540 Ethernet Controller Virtual Function' if=enp1s16
>> drv=ixgbevf unused=igb_uio
>> 0000:01:10.1 'X540 Ethernet Controller Virtual Function' if=enp1s16f1
>> drv=ixgbevf unused=igb_uio
>> 0000:08:00.0 'I350 Gigabit Network Connection' if=eth2 drv=igb
>> unused=igb_uio
>> 0000:08:00.1 'I350 Gigabit Network Connection' if=eth3 drv=igb
>> unused=igb_uio
>> Other network devices
>> =====================
>> <none>
>
>
> I change OVS-DPDK code, and use command to stop  OVS-DPDK bond port and
> its slaves, and then change its txconf and rxconf, and then start up them.
> But the kernel report "Detected Tx Unit Hang ..." and no packets could go
> through OVS-DPDK bond port and linux bond port, but use command to show
> OVS-DPDK port and linux bond port,
>
all status is NOT OK(LACP status not OK, I don't if it's because link error
or LACP packet process error).

>
> But when I stop linux bond port first, and then do the same process as
> above, no error report and packets could go through OVS-DPDK bond port and
> linux bond port.
>
> So what's this problem and how to fix it ? Thank you~
>
I want to make OVS-DPDK bond port could be stop/start/reconfig while
running. If others use some other way to do this, could you share it, thank
you~

Re: [dpdk-dev] [sriov] [ixgbevf] We have to down PF and VF port to reconfig a port?

[Sam]

This is kernel ixgbevf error log:

Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865542] ------------[ cut
here ]------------
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865556] WARNING: at
net/sched/sch_generic.c:297 dev_watchdog+0x276/0x280()
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865558] NETDEV WATCHDOG:
enp1s16f1 (ixgbevf): transmit queue 0 timed out
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865559] Modules linked
in: nbd iptable_mangle iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4
nf_nat_ipv4 nf_nat nf_con
ntrack iptable_filter vhost_net vhost macvtap macvlan ixgbevf(OE)
igb_uio(OE) udp_diag unix_diag af_packet_diag netlink_diag tun uio tcp_diag
inet_diag bonding ext4 m
bcache jbd2 intel_powerclamp coretemp kvm_intel kvm irqbypass crc32_pclmul
ghash_clmulni_intel aesni_intel lrw gf128mul glue_helper ablk_helper cryptd
ipmi_devintf sg
 iTCO_wdt iTCO_vendor_support dcdbas pcspkr mei_me mei lpc_ich sb_edac
edac_core ipmi_si wmi shpchp ipmi_msghandler acpi_pad acpi_power_meter
ip_tables xfs libcrc32c
sd_mod crc_t10dif mgag200 drm_kms_helper syscopyarea crct10dif_pclmul
sysfillrect crct10dif_common sysimgblt crc32c_intel fb_sys_fops ttm ahci
drm ixgbe libahci igb l
ibata i2c_algo_bit mdio ptp i2c_core megaraid_sas pps_core dca fjes [last
unloaded: nbd]
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865619] CPU: 0 PID: 0
Comm: swapper/0 Tainted: G           OE  ------------
 3.10.0-514.10.2.el7.mt20170307.x86_64
#1
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865622] Hardware name:
Dell Inc. PowerEdge R720/068CDY, BIOS 2.5.2 01/28/2015
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865624]  ffff880fff803d88
d4910673413043a2 ffff880fff803d40 ffffffff8168648f
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865626]  ffff880fff803d78
ffffffff81085940 0000000000000000 ffff880ffdfd0000
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865629]  ffff880ffdfd1940
0000000000000008 0000000000000000 ffff880fff803de0
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865631] Call Trace:
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865632]  <IRQ>
[<ffffffff8168648f>] dump_stack+0x19/0x1b
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865639]
[<ffffffff81085940>] warn_slowpath_common+0x70/0xb0
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865641]
[<ffffffff810859dc>] warn_slowpath_fmt+0x5c/0x80
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865644]
[<ffffffff81596616>] dev_watchdog+0x276/0x280
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865647]
[<ffffffff815963a0>] ? dev_graft_qdisc+0x80/0x80
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865650]
[<ffffffff81095b46>] call_timer_fn+0x36/0x110
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865652]
[<ffffffff815963a0>] ? dev_graft_qdisc+0x80/0x80
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865655]
[<ffffffff81098837>] run_timer_softirq+0x237/0x340
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865658]
[<ffffffff8108f2cf>] __do_softirq+0xef/0x280
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865661]
[<ffffffff8169851c>] call_softirq+0x1c/0x30
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865664]
[<ffffffff8102d365>] do_softirq+0x65/0xa0
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865667]
[<ffffffff8108f665>] irq_exit+0x115/0x120
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865670]
[<ffffffff81699195>] smp_apic_timer_interrupt+0x45/0x60
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865672]
[<ffffffff816976dd>] apic_timer_interrupt+0x6d/0x80
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865673]  <EOI>
[<ffffffff810b4776>] ? hrtimer_start_range_ns+0x1c6/0x390
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865679]
[<ffffffff81514392>] ? cpuidle_enter_state+0x52/0xc0
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865681]
[<ffffffff81514388>] ? cpuidle_enter_state+0x48/0xc0
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865683]
[<ffffffff815144d9>] cpuidle_idle_call+0xd9/0x210
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865687]
[<ffffffff810350ee>] arch_cpu_idle+0xe/0x30
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865690]
[<ffffffff810e7e65>] cpu_startup_entry+0x245/0x290
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865694]
[<ffffffff816748e7>] rest_init+0x77/0x80
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865698]
[<ffffffff81b0a05a>] start_kernel+0x42c/0x44d
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865700]
[<ffffffff81b09a37>] ? repair_env_string+0x5c/0x5c
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865703]
[<ffffffff81b09120>] ? early_idt_handlers+0x120/0x120
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865705]
[<ffffffff81b095ee>] x86_64_start_reservations+0x2a/0x2c
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865708]
[<ffffffff81b09742>] x86_64_start_kernel+0x152/0x175
Dec 27 20:23:22 yf-mos-test-net14 kernel: [103406.865709] ---[ end trace
45a157a1f564e086 ]---



2017-12-27 10:54 GMT+08:00 Sam <batmanustc@gmail.com>:

>
> 2017-12-27 10:42 GMT+08:00 Sam <batmanustc@gmail.com>:
>
>> Hi all,
>>
>> I'm running OVS-DPDK on a two netdevice server(eth0 and eth1), and I use
>> sriov to make one VF and one PF on each netdevice, as bellow. Please omit
>> eth2 and eth3. Then two VF port(enp1s16 and enp1s16f1) combine into linux
>> bond1 port as management port, two PF port(0000:01:00.0 and 0000:01:00.1)
>> combine into OVS-DPDK bond port(mode 4) as dataplane port.
>>
>> [root@yf-mos-test-net14 ~]# /usr/local/share/openvswitch/scripts/dpdk_nic_bind
>>> --status
>>> Network devices using DPDK-compatible driver
>>> ============================================
>>> 0000:01:00.0 'Ethernet Controller 10-Gigabit X540-AT2' drv=igb_uio
>>> unused=ixgbe
>>> 0000:01:00.1 'Ethernet Controller 10-Gigabit X540-AT2' drv=igb_uio
>>> unused=ixgbe
>>> Network devices using kernel driver
>>> ===================================
>>> 0000:01:10.0 'X540 Ethernet Controller Virtual Function' if=enp1s16
>>> drv=ixgbevf unused=igb_uio
>>> 0000:01:10.1 'X540 Ethernet Controller Virtual Function' if=enp1s16f1
>>> drv=ixgbevf unused=igb_uio
>>> 0000:08:00.0 'I350 Gigabit Network Connection' if=eth2 drv=igb
>>> unused=igb_uio
>>> 0000:08:00.1 'I350 Gigabit Network Connection' if=eth3 drv=igb
>>> unused=igb_uio
>>> Other network devices
>>> =====================
>>> <none>
>>
>>
>> I change OVS-DPDK code, and use command to stop  OVS-DPDK bond port and
>> its slaves, and then change its txconf and rxconf, and then start up them.
>> But the kernel report "Detected Tx Unit Hang ..." and no packets could go
>> through OVS-DPDK bond port and linux bond port, but use command to show
>> OVS-DPDK port and linux bond port,
>>
> all status is NOT OK(LACP status not OK, I don't if it's because link
> error or LACP packet process error).
>
>>
>> But when I stop linux bond port first, and then do the same process as
>> above, no error report and packets could go through OVS-DPDK bond port and
>> linux bond port.
>>
>> So what's this problem and how to fix it ? Thank you~
>>
> I want to make OVS-DPDK bond port could be stop/start/reconfig while
> running. If others use some other way to do this, could you share it, thank
> you~
>