net/netvsc: problem with configuring netvsc port after first port start

[Edwin Brossette <edwin.brossette@6wind.com>]

[-- Attachment #1: Type: text/plain, Size: 7473 bytes --]

Hello,

I have run into an issue since I switched from the failsafe/vdev-netvsc pmd
to the netvsc pmd.
I have noticed that once I have first started my port, if I try to stop and
reconfigure it, the call to rte_eth_dev_configure() fails with a couple of
error logged from the netvsc pmd. It can be reproduced quite easily with
testpmd.

I tried it on my azure set-up:

root@dut-azure:~# ip -d l
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode
DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 promiscuity 0
minmtu 0 maxmtu 0 addrgenmode eui64 numtxqueues 1 numrxqueues 1
gso_max_size 65536 gso_max_segs 65535
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode
DEFAULT group default qlen 1000
    link/ether 00:22:48:39:09:b7 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu
68 maxmtu 65521 addrgenmode eui64 numtxqueues 64 numrxqueues 64
gso_max_size 62780 gso_max_segs 65535 parentbus vmbus parentdev
00224839-09b7-0022-4839-09b700224839
6: enP27622s2: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq
master eth1 state UP mode DEFAULT group default qlen 1000
    link/ether 00:22:48:39:02:ca brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu
68 maxmtu 9978 addrgenmode eui64 numtxqueues 64 numrxqueues 8 gso_max_size
65536 gso_max_segs 65535 parentbus pci parentdev 6be6:00:02.0
    altname enP27622p0s2
7: enP25210s4: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq
master eth3 state UP mode DEFAULT group default qlen 1000
    link/ether 00:22:48:39:0f:cd brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu
68 maxmtu 9978 addrgenmode eui64 numtxqueues 64 numrxqueues 8 gso_max_size
65536 gso_max_segs 65535 parentbus pci parentdev 627a:00:02.0
    altname enP25210p0s2
8: enP16113s3: <BROADCAST,MULTICAST,SLAVE,UP,LOWER_UP> mtu 1500 qdisc mq
master eth2 state UP mode DEFAULT group default qlen 1000
    link/ether 00:22:48:39:09:36 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu
68 maxmtu 9978 addrgenmode eui64 numtxqueues 64 numrxqueues 8 gso_max_size
65536 gso_max_segs 65535 parentbus pci parentdev 3ef1:00:02.0
    altname enP16113p0s2
9: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode
DEFAULT group default qlen 1000
    link/ether 00:22:48:39:02:ca brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu
68 maxmtu 65521 addrgenmode eui64 numtxqueues 64 numrxqueues 64
gso_max_size 62780 gso_max_segs 65535 parentbus vmbus parentdev
00224839-02ca-0022-4839-02ca00224839
10: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode
DEFAULT group default qlen 1000
    link/ether 00:22:48:39:09:36 brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu
68 maxmtu 65521 addrgenmode eui64 numtxqueues 64 numrxqueues 64
gso_max_size 62780 gso_max_segs 65535 parentbus vmbus parentdev
00224839-0936-0022-4839-093600224839
11: eth3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode
DEFAULT group default qlen 1000
    link/ether 00:22:48:39:0f:cd brd ff:ff:ff:ff:ff:ff promiscuity 0 minmtu
68 maxmtu 65521 addrgenmode eui64 numtxqueues 64 numrxqueues 64
gso_max_size 62780 gso_max_segs 65535 parentbus vmbus parentdev
00224839-0fcd-0022-4839-0fcd00224839

As you can see here, I have 3 netvsc interfaces, eth1, eth2 and eth3 with
their 3 nic-accelerated counterparts.
I rebind them to uio_hv_generic and start testpmd:

root@dut-azure:~# dpdk-testpmd -- -i --rxq=2 --txq=2 --coremask=0x0c
--total-num-mbufs=25000
EAL: Detected CPU lcores: 8
EAL: Detected NUMA nodes: 1
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: VFIO support initialized
mlx5_net: No available register for sampler.
mlx5_net: No available register for sampler.
mlx5_net: No available register for sampler.
hn_vf_attach(): found matching VF port 2
hn_vf_attach(): found matching VF port 0
hn_vf_attach(): found matching VF port 1
Interactive-mode selected
previous number of forwarding cores 1 - changed to number of configured
cores 2
testpmd: create a new mbuf pool <mb_pool_0>: n=25000, size=2176, socket=0
testpmd: preferred mempool ops selected: ring_mp_mc

Warning! port-topology=paired and odd forward ports number, the last port
will pair with itself.

Configuring Port 3 (socket 0)
Port 3: 00:22:48:39:02:CA
Configuring Port 4 (socket 0)
Port 4: 00:22:48:39:09:36
Configuring Port 5 (socket 0)
Port 5: 00:22:48:39:0F:CD
Checking link statuses...
Done
testpmd>


The 3 ports are initialized and started correctly. For example, here is the
port info for the first port:

testpmd> show port info 3

********************* Infos for port 3  *********************
MAC address: 00:22:48:39:02:CA
Device name: 00224839-02ca-0022-4839-02ca00224839
Driver name: net_netvsc
Firmware-version: not available
Connect to socket: 0
memory allocation on the socket: 0
Link status: up
Link speed: 50 Gbps
Link duplex: full-duplex
Autoneg status: On
MTU: 1500
Promiscuous mode: enabled
Allmulticast mode: disabled
Maximum number of MAC addresses: 1
Maximum number of MAC addresses of hash filtering: 0
VLAN offload:
  strip off, filter off, extend off, qinq strip off
Hash key size in bytes: 40
Redirection table size: 128
Supported RSS offload flow types:
  ipv4  ipv4-tcp  ipv4-udp  ipv6  ipv6-tcp
Minimum size of RX buffer: 1024
Maximum configurable length of RX packet: 65536
Maximum configurable size of LRO aggregated packet: 0
Current number of RX queues: 2
Max possible RX queues: 64
Max possible number of RXDs per queue: 65535
Min possible number of RXDs per queue: 0
RXDs number alignment: 1
Current number of TX queues: 2
Max possible TX queues: 64
Max possible number of TXDs per queue: 4096
Min possible number of TXDs per queue: 1
TXDs number alignment: 1
Max segment number per packet: 40
Max segment number per MTU/TSO: 40
Device capabilities: 0x0( )
Device error handling mode: none
Device private info:
  none

 -> First, stop port:

testpmd> port stop 3
Stopping ports...
Done

 -> Then, change something in the port config. This will trigger a call
to rte_eth_dev_configure() on the next port start. Here I change the link
speed/duplex:
testpmd> port config 3 speed 10000 duplex full
testpmd>

 -> Finally, try to start the port:

testpmd> port start 3
Configuring Port 3 (socket 0)
hn_nvs_alloc_subchans(): nvs subch alloc failed: 0x2
hn_dev_configure(): subchannel configuration failed
ETHDEV: Port3 dev_configure = -5
Fail to configure port 3  <------


As you can see, the port configuration fails.
The error happens in hn_nvs_alloc_subchans(). Maybe the previous ressources
were not properly deallocated on port stop?

When I looked around in the pmd's code, I noticed the function hn_reinit()
with the following commentary:

/*
 * Connects EXISTING rx/tx queues to NEW vmbus channel(s), and
 * re-initializes NDIS and RNDIS, including re-sending initial
 * NDIS/RNDIS configuration. To be used after the underlying vmbus
 * has been un- and re-mapped, e.g. as must happen when the device
 * MTU is changed.
 */

This function shows that it is possible to call rte_eth_dev_configure()
without failing, as this funtion hn_reinit() calls hn_dev_configure(), and
is called when the mtu is changed.
I suspect the operations described in comment above might also be needed
when running rte_eth_dev_configure().

Is this a known issue ?
In my application case, this bug causes issues when I need to restart and
configure the port.
Thank you for considering this issue.

Regards,
Edwin Brossette.

[-- Attachment #2: Type: text/html, Size: 8244 bytes --]