From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga07.intel.com (mga07.intel.com [134.134.136.100]) by dpdk.org (Postfix) with ESMTP id 5363ADE3 for ; Fri, 25 May 2018 09:49:33 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga003.jf.intel.com ([10.7.209.27]) by orsmga105.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 25 May 2018 00:49:32 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.49,439,1520924400"; d="scan'208";a="53747942" Received: from jguo15x-mobl3.ccr.corp.intel.com (HELO [10.67.68.59]) ([10.67.68.59]) by orsmga003.jf.intel.com with ESMTP; 25 May 2018 00:49:29 -0700 To: Matan Azrad , "dev@dpdk.org" References: <01BA8470C017D6468C8290E4B9C5E1E83B379B43@shsmsx102.ccr.corp.intel.com> Cc: "Ananyev, Konstantin" , "stephen@networkplumber.org" , "Richardson, Bruce" , "Yigit, Ferruh" , "gaetan.rivet@6wind.com" , "Wu, Jingjing" , Thomas Monjalon , Mordechay Haimovsky , "Van Haaren, Harry" , "Zhang, Qi Z" , "Zhang, Helin" , "jblunck@infradead.org" , "shreyansh.jain@nxp.com" From: "Guo, Jia" Message-ID: Date: Fri, 25 May 2018 15:49:28 +0800 User-Agent: Mozilla/5.0 (Windows NT 6.3; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.4.0 MIME-Version: 1.0 In-Reply-To: Content-Type: text/plain; charset=windows-1252; format=flowed Content-Transfer-Encoding: 7bit Subject: Re: [dpdk-dev] [RFC] hot plug failure handle mechanism X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 25 May 2018 07:49:34 -0000 hi,matan On 5/24/2018 10:57 PM, Matan Azrad wrote: > Hi Guo > > Some questions. > > From: Guo Jia >> As we know, hot plug is an importance feature whenever it use for the >> datacenter device's fail-safe and consumption management , or use for the >> dynamic deployment and SRIOV Live Migration in SDN/NFV, it could be bring >> the higher flexibility and continuality of the networking services in multiple use >> case in industry. >> >> So let we see, dpdk as an importance networking combine framework with >> packet control path/fast path lib and multiple diversity PMD drivers, what can it >> do to help if application want to achieve their hot plug solution when they are >> working in packet processing by dpdk. >> >> We already have a general device event mechanism, failsafe driver, bonding >> driver and hot plug/unplug api in framework, app could use these api to >> develop functional, but for the case of hot plug failure handle, that is removing >> a device at run-time will cause app trigger MMIO error and crash out, it is lack >> of a mechanism to handle the failure when hot unplug device. At present, >> kernel only guantiy the hotplug handle safer on the kernel side, but for the user >> mode side, no more specific 3rd tools such as udev/driverctl have especially >> cover about these part of mechanism, and considerate feasibility of the >> implementation, runtime performance and the general for almost user mode >> PMD driver, here a general hot plug failure handle mechanism in dpdk >> framework would be proposed. >> >> The hot plug failure handle mechanism should be come across as bellow: >> 1. Add a new bus ops "handle_hot-unplug"in bus to handle bus read/write >> error, it is bus-specific and each kind of bus can implement its own logic. >> 2. Implement pci bus specific ops"pci_handle_hot_unplug", in the function, >> base on the failure address to remap memory which belong to the >> corresponding device that unplugged. >> 3. Implement a new sigbus handler, and register it when start device event >> monitoring, once the MMIO sigbus error exposure, it will trigger the above hot >> plug failure handle mechanism, that will keep app, that working on packet >> processing, would not be broken and crash, then could keep going clean, fail- >> safe or other working task. > Can you explain more what's happened with all the threads? Master thread, host thread, data-path threads, > The signal may happened only in a datapath thread or even from a control thread? i will explain it here for you at first, sigbus handler is register per process, cause of the signal event mechanism, control thread and data-path thread will random receive the sigbus error, but will go to the common sigbus handler, in the handler find the device according the failure address, then remap the memory for the device. > What's about resource leak? (mainly relevant for control threads): > If you jump from the signal address to the restart address, how can you clean the process which was started and got the signal? it will not use long jump to turn back the restart address, just capture the sigbus event and then do failure handle, then let the thread keep going at current position. > Matan. >> 4. Also also will introduce the solution by use testpmd to show the example of >> the whole procedure like that: >> device unplug ->failure handle->stop forwarding->stop port->close port->detach >> port. >> >> Best regards, >> >> Jeff Guo