From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga14.intel.com (mga14.intel.com [192.55.52.115]) by dpdk.org (Postfix) with ESMTP id 0157BF72 for ; Tue, 29 May 2018 13:20:18 +0200 (CEST) X-Amp-Result: UNKNOWN X-Amp-Original-Verdict: FILE UNKNOWN X-Amp-File-Uploaded: False Received: from orsmga007.jf.intel.com ([10.7.209.58]) by fmsmga103.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 29 May 2018 04:20:17 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.49,456,1520924400"; d="scan'208";a="44760928" Received: from bricha3-mobl.ger.corp.intel.com ([10.237.221.55]) by orsmga007.jf.intel.com with SMTP; 29 May 2018 04:20:12 -0700 Received: by (sSMTP sendmail emulation); Tue, 29 May 2018 12:20:12 +0100 Date: Tue, 29 May 2018 12:20:11 +0100 From: Bruce Richardson To: "Guo, Jia" Cc: "dev@dpdk.org" , "Ananyev, Konstantin" , "stephen@networkplumber.org" , "Yigit, Ferruh" , "gaetan.rivet@6wind.com" , "Wu, Jingjing" , "thomas@monjalon.net" , "motih@mellanox.com" , "matan@mellanox.com" , "Van Haaren, Harry" , "Zhang, Qi Z" , "Zhang, Helin" , "jblunck@infradead.org" , "shreyansh.jain@nxp.com" Message-ID: <20180529112011.GA22740@bricha3-MOBL.ger.corp.intel.com> References: <01BA8470C017D6468C8290E4B9C5E1E83B379B43@shsmsx102.ccr.corp.intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: <01BA8470C017D6468C8290E4B9C5E1E83B379B43@shsmsx102.ccr.corp.intel.com> Organization: Intel Research and Development Ireland Ltd. User-Agent: Mutt/1.10.0 (2018-05-17) Subject: Re: [dpdk-dev] [RFC] hot plug failure handle mechanism X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 29 May 2018 11:20:19 -0000 On Thu, May 24, 2018 at 07:55:43AM +0100, Guo, Jia wrote: > The hot plug failure handle mechanism should be come across as bellow: > > 1. Add a new bus ops “handle_hot-unplug”in bus to handle bus > read/write error, it is bus-specific and each > > kind of bus can implement its own logic. > > 2. Implement pci bus specific ops“pci_handle_hot_unplug”, in the > function, base on the > > failure address to remap memory which belong to the corresponding > device that unplugged. > > 3. Implement a new sigbus handler, and register it when start > device event monitoring, > > once the MMIO sigbus error exposure, it will trigger the above hot plug > failure handle mechanism, > > that will keep app, that working on packet processing, would not be > broken and crash, then could > > keep going clean, fail-safe or other working task. > > 4. Also also will introduce the solution by use testpmd to show > the example of the whole procedure like that: > > device unplug ->failure handle->stop forwarding->stop port->close > port->detach port. > Hi Jeff, so if I understand this correctly the proposal is that we need two parallel solutions to handle safe removal of a device. 1. We need a solution to support unpluging of the device at the bus level, ie. remove the device from the list of devices and to make access to that device invalid. 2. Since the removal of the device from the software lists is not going to be instantaneous, we need a mechanism to handle any accesses to the device from the data path until such time as the removal is complete. To support that, you propose to add a sigbus handler which will automatically replace any mmio bar mappings with some other memory that is ok to access - presumable zero memory or similar. Is this understanding correct? Point #2 seems reasonably clear to me, but for #1, presumably the trigger to the bus needs to come from the kernel. What is planned to be used there? You also talk about using testpmd as a reference for this, but you don't explain how an application can be notified of a device removal, or even why that is necessary. Since all applications should now be using the proper macros when iterating device lists, and not just assuming devices 0-N are valid, what changes would you see a normal app having to make to be hotplug-safe? Regards, /Bruce