From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dispatch1-us1.ppe-hosted.com (dispatch1-us1.ppe-hosted.com [148.163.129.52]) by dpdk.org (Postfix) with ESMTP id B16B21BA9A for ; Fri, 22 Jun 2018 15:54:36 +0200 (CEST) X-Virus-Scanned: Proofpoint Essentials engine Received: from webmail.solarflare.com (uk.solarflare.com [193.34.186.16]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mx1-us3.ppe-hosted.com (Proofpoint Essentials ESMTP Server) with ESMTPS id 49819600057; Fri, 22 Jun 2018 13:54:33 +0000 (UTC) Received: from [192.168.38.17] (91.220.146.112) by ukex01.SolarFlarecom.com (10.17.10.4) with Microsoft SMTP Server (TLS) id 15.0.1044.25; Fri, 22 Jun 2018 14:54:25 +0100 To: Qi Zhang , , CC: , , , , , References: <20180607123849.14439-1-qi.z.zhang@intel.com> <20180621020059.1198-1-qi.z.zhang@intel.com> <20180621020059.1198-5-qi.z.zhang@intel.com> From: Andrew Rybchenko Message-ID: <7754a18a-8647-1f05-294f-9b7400387e15@solarflare.com> Date: Fri, 22 Jun 2018 16:54:21 +0300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:52.0) Gecko/20100101 Thunderbird/52.8.0 MIME-Version: 1.0 In-Reply-To: <20180621020059.1198-5-qi.z.zhang@intel.com> Content-Language: en-GB X-Originating-IP: [91.220.146.112] X-ClientProxiedBy: ocex03.SolarFlarecom.com (10.20.40.36) To ukex01.SolarFlarecom.com (10.17.10.4) X-TM-AS-Product-Ver: SMEX-11.0.0.1191-8.100.1062-23922.003 X-TM-AS-Result: No--26.573300-0.000000-31 X-TM-AS-User-Approved-Sender: Yes X-TM-AS-User-Blocked-Sender: No X-MDID: 1529675674-xbXCFshXVZhO Content-Type: text/plain; charset="utf-8"; format=flowed Content-Transfer-Encoding: 7bit X-Content-Filtered-By: Mailman/MimeDel 2.1.15 Subject: Re: [dpdk-dev] [PATCH v2 04/22] ethdev: enable hotplug on multi-process X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 22 Jun 2018 13:54:37 -0000 On 06/21/2018 05:00 AM, Qi Zhang wrote: > We are going to introduce the solution to handle different hotplug > cases in multi-process situation, it include below scenario: > > 1. Attach a share device from primary > 2. Detach a share device from primary > 3. Attach a share device from secondary > 4. Detach a share device from secondary > 5. Attach a private device from secondary > 6. Detach a private device from secondary > 7. Detach a share device from secondary privately > 8. Attach a share device from secondary privately > > In primary-secondary process model, we assume device is shared by default. > that means attach or detach a device on any process will broadcast to > all other processes through mp channel then device information will be > synchronized on all processes. > > Any failure during attaching process will cause inconsistent status > between processes, so proper rollback action should be considered. > Also it is not safe to detach a share device when other process still use > it, so a handshake mechanism is introduced. > > This patch covers the implementation of case 1,2,5,6,7,8. > Case 3,4 will be implemented on separate patch as well as handshake > mechanism. > > Scenario for Case 1, 2: > > attach device > a) primary attach the new device if failed goto h). > b) primary send attach sync request to all secondary. > c) secondary receive request and attach device and send reply. > d) primary check the reply if all success go to i). > e) primary send attach rollback sync request to all secondary. > f) secondary receive the request and detach device and send reply. > g) primary receive the reply and detach device as rollback action. > h) attach fail > i) attach success > > detach device > a) primary perform pre-detach check, if device is locked, goto i). > b) primary send pre-detach sync request to all secondary. > c) secondary perform pre-detach check and send reply. > d) primary check the reply if any fail goto i). > e) primary send detach sync request to all secondary > f) secondary detach the device and send reply (assume no fail) > g) primary detach the device. > h) detach success > i) detach failed > > Case 5, 6: > Secondary process can attach private device which only visible to itself, > in this case no IPC is involved, primary process is not allowed to have > private device so far. > > Case 7, 8: > Secondary process can also temporally to detach a share device "privately" > then attach it back later, this action also not impact other processes. > > APIs changes: > > rte_eth_dev_attach and rte_eth_dev_attach are extended to support > share device attach/detach in primary-secondary process model, it will > be called in case 1,2,3,4. > > New API rte_eth_dev_attach_private and rte_eth_dev_detach_private are > introduced to cover case 5,6,7,8, this API can only be invoked in secondary > process. > > Signed-off-by: Qi Zhang > --- > > v2: > - rename rte_ethdev_mp.* to ethdev_mp.* > - add experimental tag for rte_eth_dev_attach_private and > rte_ethdev_detach_private. > - move do_eth_dev_attach and do_eth_dev_detach to ethdev_private.h > - move rte_eth_dev_mp_init before rte_eal_mcfg_complete. > - fix meson.build. > - improve commit log. > > lib/librte_eal/common/eal_private.h | 8 ++ > lib/librte_eal/linuxapp/eal/eal.c | 7 ++ > lib/librte_ethdev/Makefile | 1 + > lib/librte_ethdev/ethdev_mp.c | 198 ++++++++++++++++++++++++++++++++++++ > lib/librte_ethdev/ethdev_mp.h | 44 ++++++++ > lib/librte_ethdev/ethdev_private.h | 39 +++++++ > lib/librte_ethdev/meson.build | 1 + > lib/librte_ethdev/rte_ethdev.c | 184 +++++++++++++++++++++++++++++---- > lib/librte_ethdev/rte_ethdev.h | 45 ++++++++ > lib/librte_ethdev/rte_ethdev_core.h | 5 + > 10 files changed, 515 insertions(+), 17 deletions(-) > create mode 100644 lib/librte_ethdev/ethdev_mp.c > create mode 100644 lib/librte_ethdev/ethdev_mp.h > create mode 100644 lib/librte_ethdev/ethdev_private.h > > diff --git a/lib/librte_eal/common/eal_private.h b/lib/librte_eal/common/eal_private.h > index bdadc4d50..92fa59bed 100644 > --- a/lib/librte_eal/common/eal_private.h > +++ b/lib/librte_eal/common/eal_private.h > @@ -258,4 +258,12 @@ int rte_mp_channel_init(void); > */ > void dev_callback_process(char *device_name, enum rte_dev_event_type event); > > +/** > + * Register mp channel callback functions of ethdev layer. > + * > + * @return > + * 0 on success. > + * (<0) on failure. > + */ > +int rte_eth_dev_mp_init(void); It looks like it makes cross-dependency between EAL and ethdev. As far as I can see EAL does not have references to rte_eth_dev functions yet. It looks really suspicious. The function is declared in EAL, but implemented in ethdev. Moreover, it is declared once again in ethdev_mp.h. <...> > diff --git a/lib/librte_ethdev/rte_ethdev_core.h b/lib/librte_ethdev/rte_ethdev_core.h > index 33d12b3a2..2cb6de745 100644 > --- a/lib/librte_ethdev/rte_ethdev_core.h > +++ b/lib/librte_ethdev/rte_ethdev_core.h > @@ -622,4 +622,9 @@ struct rte_eth_dev_data { > */ > extern struct rte_eth_dev rte_eth_devices[]; > > +extern int ethdev_logtype; > +#define ethdev_log(level, fmt, ...) \ > + rte_log(RTE_LOG_ ## level, ethdev_logtype, fmt "\n", ## __VA_ARGS__) It looks like it clashes with ethdev logging changes submitted by Ferruh. Andrew.