From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id 2B0F0455DD; Tue, 9 Jul 2024 21:07:13 +0200 (CEST) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 03D37402A9; Tue, 9 Jul 2024 21:07:13 +0200 (CEST) Received: from inbox.dpdk.org (inbox.dpdk.org [95.142.172.178]) by mails.dpdk.org (Postfix) with ESMTP id 0C0354025C for ; Tue, 9 Jul 2024 21:07:11 +0200 (CEST) Received: by inbox.dpdk.org (Postfix, from userid 33) id 0133D455DE; Tue, 9 Jul 2024 21:07:10 +0200 (CEST) From: bugzilla@dpdk.org To: dev@dpdk.org Subject: [DPDK/ethdev Bug 1484] net/netvsc deadlock during VF hot add when setting MTU Date: Tue, 09 Jul 2024 19:07:10 +0000 X-Bugzilla-Reason: AssignedTo X-Bugzilla-Type: new X-Bugzilla-Watch-Reason: None X-Bugzilla-Product: DPDK X-Bugzilla-Component: ethdev X-Bugzilla-Version: 23.11 X-Bugzilla-Keywords: X-Bugzilla-Severity: normal X-Bugzilla-Who: joshua.knight@netscout.com X-Bugzilla-Status: UNCONFIRMED X-Bugzilla-Resolution: X-Bugzilla-Priority: Normal X-Bugzilla-Assigned-To: dev@dpdk.org X-Bugzilla-Target-Milestone: --- X-Bugzilla-Flags: X-Bugzilla-Changed-Fields: bug_id short_desc product version rep_platform op_sys bug_status bug_severity priority component assigned_to reporter target_milestone attachments.created Message-ID: Content-Type: multipart/alternative; boundary=17205520300.f2BFE4.3174408 Content-Transfer-Encoding: 7bit X-Bugzilla-URL: http://bugs.dpdk.org/ Auto-Submitted: auto-generated X-Auto-Response-Suppress: All MIME-Version: 1.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org --17205520300.f2BFE4.3174408 Date: Tue, 9 Jul 2024 21:07:10 +0200 MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: http://bugs.dpdk.org/ Auto-Submitted: auto-generated X-Auto-Response-Suppress: All https://bugs.dpdk.org/show_bug.cgi?id=3D1484 Bug ID: 1484 Summary: net/netvsc deadlock during VF hot add when setting MTU Product: DPDK Version: 23.11 Hardware: x86 OS: Linux Status: UNCONFIRMED Severity: normal Priority: Normal Component: ethdev Assignee: dev@dpdk.org Reporter: joshua.knight@netscout.com Target Milestone: --- Created attachment 284 --> https://bugs.dpdk.org/attachment.cgi?id=3D284&action=3Dedit netvsc deadlock patch This issue is seen on DPDK 23.11.1 running on Ubuntu 22.04 in Micrsoft Azur= e. During VF hot add, there is a deadlock that occurs when the MTU is being se= t. The hn_vf_add function takes the VF write lock, then proceeds to call hn_vf_mtu_set which attempts to take the read lock, causing deadlock. We originally found this while cherrypicking this commit "0479315ce9636e43affc6d24d39c63bbd1f44d5a net/netvsc: support MTU set" to 22.11, but we are able to reproduce it in 23.11.1 I've attached a patch we are using to work around this issue, by not additionally locking while setting MTU. You can reproduce this easily with the testpmd on Ubuntu. In 22.11 when the hn_vf_attach function is called it eventually ends in the= VF port being started hn_vf_attach(): Couldn't find port for VF hn_vf_add(): RNDIS reports VF but device not found, retrying hn_vf_attach(): Couldn't find port for VF hn_vf_add(): RNDIS reports VF but device not found, retrying netvsc_hotplug_retry(): Found matching MAC address, adding device 997f:00:0= 2.0 network name enP39295s2 EAL: Probe PCI driver: mlx5_pci (15b3:1016) device: 997f:00:02.0 (socket 0) mlx5_net: No available register for sampler. Port 0: link state change event hn_vf_attach(): found matching VF port 0 hn_vf_add(): configuring VF port 0 hn_vf_add(): Starting VF port 0 In 23.11.1 however, it never gets past configuring the port stage. Attempti= ng to exit testpmd with 'quit' hangs as well, waiting for lcores to finish whi= ch never happens. hn_vf_attach(): Couldn't find port for VF hn_vf_add(): RNDIS reports VF but device not found, retrying hn_vf_attach(): Couldn't find port for VF hn_vf_add(): RNDIS reports VF but device not found, retrying netvsc_hotplug_retry(): Found matching MAC address, adding device 997f:00:0= 2.0 network name enP39295s2 EAL: Probe PCI driver: mlx5_pci (15b3:1016) device: 997f:00:02.0 (socket 0) mlx5_net: No available register for sampler. Port 0: link state change event hn_vf_attach(): found matching VF port 0 hn_vf_add(): configuring VF port 0 --=20 You are receiving this mail because: You are the assignee for the bug.= --17205520300.f2BFE4.3174408 Date: Tue, 9 Jul 2024 21:07:10 +0200 MIME-Version: 1.0 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Bugzilla-URL: http://bugs.dpdk.org/ Auto-Submitted: auto-generated X-Auto-Response-Suppress: All
Bug ID 1484
Summary net/netvsc deadlock during VF hot add when setting MTU
Product DPDK
Version 23.11
Hardware x86
OS Linux
Status UNCONFIRMED
Severity normal
Priority Normal
Component ethdev
Assignee dev@dpdk.org
Reporter joshua.knight@netscout.com
Target Milestone --- 

Created attachment 284 [details]
netvsc deadlock patch

This issue is seen on DPDK 23.11.1 running on Ubuntu 22.04 in Micrsoft Azur=
e.
During VF hot add, there is a deadlock that occurs when the MTU is being se=
t.

The hn_vf_add function takes the VF write lock, then proceeds to call
hn_vf_mtu_set which attempts to take the read lock, causing deadlock.


We originally found this while cherrypicking this commit
"0479315ce9636e43affc6d24d39c63bbd1f44d5a net/netvsc: support MTU set&=
quot; to
22.11, but we are able to reproduce it in 23.11.1


I've attached a patch we are using to work around this issue, by not
additionally locking while setting MTU.

You can reproduce this easily with the testpmd on Ubuntu.

In 22.11 when the hn_vf_attach function is called it eventually ends in the=
 VF
port being started

hn_vf_attach(): Couldn't find port for VF
hn_vf_add(): RNDIS reports VF but device not found, retrying
hn_vf_attach(): Couldn't find port for VF
hn_vf_add(): RNDIS reports VF but device not found, retrying
netvsc_hotplug_retry(): Found matching MAC address, adding device 997f:00:0=
2.0
network name enP39295s2
EAL: Probe PCI driver: mlx5_pci (15b3:1016) device: 997f:00:02.0 (socket 0)
mlx5_net: No available register for sampler.

Port 0: link state change event
hn_vf_attach(): found matching VF port 0
hn_vf_add(): configuring VF port 0
hn_vf_add(): Starting VF port 0




In 23.11.1 however, it never gets past configuring the port stage. Attempti=
ng
to exit testpmd with 'quit' hangs as well, waiting for lcores to finish whi=
ch
never happens.

hn_vf_attach(): Couldn't find port for VF
hn_vf_add(): RNDIS reports VF but device not found, retrying
hn_vf_attach(): Couldn't find port for VF
hn_vf_add(): RNDIS reports VF but device not found, retrying
netvsc_hotplug_retry(): Found matching MAC address, adding device 997f:00:0=
2.0
network name enP39295s2
EAL: Probe PCI driver: mlx5_pci (15b3:1016) device: 997f:00:02.0 (socket 0)
mlx5_net: No available register for sampler.

Port 0: link state change event
hn_vf_attach(): found matching VF port 0
hn_vf_add(): configuring VF port 0

You are receiving this mail because:
  • You are the assignee for the bug.
=20=20=20=20=20=20=20=20=20=20
= --17205520300.f2BFE4.3174408--