From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga09.intel.com (mga09.intel.com [134.134.136.24]) by dpdk.org (Postfix) with ESMTP id 1C2F0239 for ; Thu, 8 Nov 2018 09:31:00 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga006.jf.intel.com ([10.7.209.51]) by orsmga102.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 08 Nov 2018 00:30:59 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.54,478,1534834800"; d="scan'208";a="89516472" Received: from jguo15x-mobl.ccr.corp.intel.com (HELO [10.67.68.69]) ([10.67.68.69]) by orsmga006.jf.intel.com with ESMTP; 08 Nov 2018 00:30:58 -0800 To: Matan Azrad , "konstantin.ananyev@intel.com" , "anatoly.burakov@intel.com" , Thomas Monjalon , "bernard.iremonger@intel.com" , "jingjing.wu@intel.com" , "wenzhuo.lu@intel.com" Cc: "ferruh.yigit@intel.com" , "dev@dpdk.org" , "helin.zhang@intel.com" , "shaopeng.he@intel.com" References: <1541484436-91320-1-git-send-email-jia.guo@intel.com> <1541484436-91320-3-git-send-email-jia.guo@intel.com> From: Jeff Guo Message-ID: Date: Thu, 8 Nov 2018 16:30:57 +0800 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:60.0) Gecko/20100101 Thunderbird/60.3.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 8bit Content-Language: en-US Subject: Re: [dpdk-dev] [PATCH 2/3] vfio: fix to add handler lock for hot-unplug X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 08 Nov 2018 08:31:01 -0000 On 11/8/2018 3:20 PM, Matan Azrad wrote: > > From: Jeff Guo >> On 11/6/2018 2:23 PM, Matan Azrad wrote: >>> Hi Jeff >>> >>> Can you detail more in the commit log that we can understand the >> synchronization problematic scenario. And how does this commit fix it? >> Please check my reply in the 1/3 mail. And explain more here is that, when >> device be hot-unplugged in vfio, the req notifier will invoked, then user >> space could release device resource in user space side, >> >> then vfio check that the device be released out from the device group, it will >> take the device control again and trigger the device kernel release >> processing, at the mean time it will sent remove uevent to >> >> user space. Here although the req handler seems will always process before >> uevent handler, but even for fast path and slow path protection of device >> accessing when device is removing , it should also be need. >> >> what do you think about that? > Just don't understanf hoe the fast\control-path can access to pci_vfio_req_handler... The pci_vfio_req_handler is register in eal interrupt thread when pci probe driver, so it should be access in the control path. Since the sigbus handler will be enable when enable hotplug , if origin sigbus occur the sigbus handler be invoked,  in the handler it will process the bus and device.  So i think protection of the bus and device so that it will not cause race condition that should be need. >> >>>> -----Original Message----- >>>> From: Jeff Guo >>>> Sent: Tuesday, November 6, 2018 8:07 AM >>>> To: konstantin.ananyev@intel.com; anatoly.burakov@intel.com; Thomas >>>> Monjalon ; bernard.iremonger@intel.com; >>>> jingjing.wu@intel.com; wenzhuo.lu@intel.com >>>> Cc: ferruh.yigit@intel.com; dev@dpdk.org; jia.guo@intel.com; >>>> helin.zhang@intel.com; Matan Azrad ; >>>> shaopeng.he@intel.com >>>> Subject: [PATCH 2/3] vfio: fix to add handler lock for hot-unplug >>>> >>>> This patch add hot-unplug handler lock and unlock in device request >>>> handler when process bus and device resource, in order to avoid the >>>> synchronization issue when device be hot-unplugged. >>>> >>>> Fixes: c115fd000c32 ("vfio: handle hotplug request notifier") >>>> Signed-off-by: Jeff Guo >>>> --- >>>> drivers/bus/pci/linux/pci_vfio.c | 14 +++++++++++++- >>>> 1 file changed, 13 insertions(+), 1 deletion(-) >>>> >>>> diff --git a/drivers/bus/pci/linux/pci_vfio.c >>>> b/drivers/bus/pci/linux/pci_vfio.c >>>> index 305cc06..d2c8410 100644 >>>> --- a/drivers/bus/pci/linux/pci_vfio.c >>>> +++ b/drivers/bus/pci/linux/pci_vfio.c >>>> @@ -19,6 +19,7 @@ >>>> #include >>>> #include >>>> #include >>>> +#include >>>> >>>> #include "eal_filesystem.h" >>>> >>>> @@ -35,6 +36,14 @@ >>>> * This file is only compiled if CONFIG_RTE_EAL_VFIO is set to "y". >>>> */ >>>> >>>> +/* >>>> + * spinlock for device hot-unplug failure handling. If it try to >>>> +access bus or >>>> + * device, such as handle sigbus on bus or handle memory failure for >>>> +device >>>> + * just need to use this lock. It could protect the bus and the >>>> +device to avoid >>>> + * race condition. >>>> + */ >>>> +static rte_spinlock_t failure_handle_lock = >>>> +RTE_SPINLOCK_INITIALIZER; >>>> + >>>> #ifdef VFIO_PRESENT >>>> >>>> #ifndef PAGE_SIZE >>>> @@ -289,11 +298,12 @@ pci_vfio_req_handler(void *param) >>>> int ret; >>>> struct rte_device *device = (struct rte_device *)param; >>>> >>>> + rte_spinlock_lock(&failure_handle_lock); >>>> bus = rte_bus_find_by_device(device); >>>> if (bus == NULL) { >>>> RTE_LOG(ERR, EAL, "Cannot find bus for device (%s)\n", >>>> device->name); >>>> - return; >>>> + goto handle_end; >>>> } >>>> >>>> /* >>>> @@ -306,6 +316,8 @@ pci_vfio_req_handler(void *param) >>>> RTE_LOG(ERR, EAL, >>>> "Can not handle hot-unplug for device (%s)\n", >>>> device->name); >>>> +handle_end: >>>> + rte_spinlock_unlock(&failure_handle_lock); >>>> } >>>> >>>> /* enable notifier (only enable req now) */ >>>> -- >>>> 2.7.4