From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga03.intel.com (mga03.intel.com [134.134.136.65]) by dpdk.org (Postfix) with ESMTP id 828867F18 for ; Tue, 17 Apr 2018 13:00:29 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from orsmga007.jf.intel.com ([10.7.209.58]) by orsmga103.jf.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 17 Apr 2018 04:00:28 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.48,463,1517904000"; d="scan'208";a="33989807" Received: from irsmsx104.ger.corp.intel.com ([163.33.3.159]) by orsmga007.jf.intel.com with ESMTP; 17 Apr 2018 04:00:26 -0700 Received: from irsmsx156.ger.corp.intel.com (10.108.20.68) by IRSMSX104.ger.corp.intel.com (163.33.3.159) with Microsoft SMTP Server (TLS) id 14.3.319.2; Tue, 17 Apr 2018 12:00:25 +0100 Received: from irsmsx102.ger.corp.intel.com ([169.254.2.164]) by IRSMSX156.ger.corp.intel.com ([169.254.3.229]) with mapi id 14.03.0319.002; Tue, 17 Apr 2018 12:00:25 +0100 From: "Ananyev, Konstantin" To: Matan Azrad , "Richardson, Bruce" CC: "Burakov, Anatoly" , Thomas Monjalon , "dev@dpdk.org" , "pmatilai@redhat.com" , "david.marchand@6wind.com" , "Guo, Jia" , "stephen@networkplumber.org" , "fbl@redhat.com" Thread-Topic: kernel binding of devices + hotplug Thread-Index: AQHT00TmmE7o5Zr910mkyHe0bqbSjKP+1QoAgAAQwwCAAbwpAIACYXuAgACAVACAAS5iMIAACAMAgAARV6A= Date: Tue, 17 Apr 2018 11:00:24 +0000 Message-ID: <2601191342CEEE43887BDE71AB977258AE916AF5@IRSMSX102.ger.corp.intel.com> References: <2407757.yEAnF6RcS7@xps> <20180413164046.GD37024@bricha3-MOBL.ger.corp.intel.com> <20180416083153.GA50020@bricha3-MOBL.ger.corp.intel.com> <2601191342CEEE43887BDE71AB977258AE916938@IRSMSX102.ger.corp.intel.com> In-Reply-To: Accept-Language: en-IE, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiMzQ0OWViYzgtMmI4Zi00ZmZlLTg5ZmItYjRmYmQzYzQ0ZDkzIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE2LjUuOS4zIiwiVHJ1c3RlZExhYmVsSGFzaCI6IlE2VVNCbWRcL0pzc252eEZcL25SR1luck1IN1Bkak41Q25Qck96ZWNBWEd6Zz0ifQ== x-ctpclassification: CTP_NT dlp-product: dlpe-windows dlp-version: 11.0.200.100 dlp-reaction: no-action x-originating-ip: [163.33.239.181] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] kernel binding of devices + hotplug X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Apr 2018 11:00:30 -0000 Hi Matan, >=20 >=20 > Hi Konstantin >=20 > From: Ananyev, Konstantin, Tuesday, April 17, 2018 12:23 PM > > To: Matan Azrad ; Richardson, Bruce > > > > Cc: Burakov, Anatoly ; Thomas Monjalon > > ; dev@dpdk.org; pmatilai@redhat.com; > > david.marchand@6wind.com; Guo, Jia ; > > stephen@networkplumber.org; fbl@redhat.com > > Subject: RE: kernel binding of devices + hotplug > > > > > > Hi everyone, > > Few comments from me below. > > In summary I am also in favor to keep binding decision at system level > > (outside DPDK). > > Konstantin > > > > > -----Original Message----- > > > From: Matan Azrad [mailto:matan@mellanox.com] > > > Sent: Monday, April 16, 2018 5:11 PM > > > To: Richardson, Bruce > > > Cc: Burakov, Anatoly ; Thomas Monjalon > > > ; dev@dpdk.org; pmatilai@redhat.com; > > > david.marchand@6wind.com; Guo, Jia ; Ananyev, > > > Konstantin ; > > stephen@networkplumber.org; > > > fbl@redhat.com > > > Subject: RE: kernel binding of devices + hotplug > > > > > > Hi Bruce > > > > > > From: Bruce Richardson, Monday, April 16, 2018 11:32 AM > > > > On Sat, Apr 14, 2018 at 08:10:28PM +0000, Matan Azrad wrote: > > > > > Hi all > > > > > > > > > > From: Burakov, Anatoly, Friday, April 13, 2018 8:41 PM > > > > > > To: Bruce Richardson ; Thomas > > > > > > Monjalon > > > > > > Cc: dev@dpdk.org; pmatilai@redhat.com; > > david.marchand@6wind.com; > > > > > > jia.guo@intel.com; Matan Azrad ; > > > > > > konstantin.ananyev@intel.com; stephen@networkplumber.org; > > > > > > fbl@redhat.com > > > > > > Subject: Re: kernel binding of devices + hotplug > > > > > > > > > > > > On 13-Apr-18 5:40 PM, Bruce Richardson wrote: > > > > > > > On Fri, Apr 13, 2018 at 06:31:21PM +0200, Thomas Monjalon wro= te: > > > > > > >> It's time to think (again) how we bind devices with kernel m= odules. > > > > > > >> We need to decide how we want to manage hotplugged devices > > > > > > >> with > > > > > > DPDK. > > > > > > >> > > > > > > >> A bit of history first. > > > > > > >> There was some code in DPDK for bind/unbind, but it has been > > > > > > >> removed in DPDK 1.7 - > > > > > > >> > > > > > > > > > > > > https://emea01.safelinks.protection.outlook.com/?url=3Dhttp%3A%2F%2Fdp > > > > > > d > > > > > > >> > > > > > > > > > > > > k.org%2Fcommit%2F5d8751b83&data=3D02%7C01%7Cmatan%40mellanox.com > > > > > > %7C6ea5 > > > > > > >> > > > > > > > > > > > > 5ce994ff4bb0d65208d5a165b417%7Ca652971c7d2e4d9ba6a4d149256f461b%7 > > > > > > C0%7 > > > > > > >> > > > > > > > > > > > > C0%7C636592380565078675&sdata=3DuLRDAk65hYtJYxjIvY20de377yayCN5DrjCZ > > > > > > x8H > > > > > > >> p61o%3D&reserved=3D0 Copy of the commit message (in 2014): > > > > > > >> " > > > > > > >> The bind/unbind operations should not be handled by the > > eal. > > > > > > >> These operations should be either done outside of dpdk or > > > > > > >> inside the PMDs themselves as these are their problems. > > > > > > >> " > > > > > > >> > > > > > > >> The question raised at this time (4 years ago) is still unde= r > > discussion. > > > > > > >> Should we manage binding inside or outside DPDK? > > > > > > >> Should it be controlled in the application or in the OS base= ? > > > > > > >> > > > > > > >> As you know, we use dpdk-devbind.py. > > > > > > >> This tool lacks two major features: > > > > > > >> - persistent configuration > > > > > > >> - hotplug > > > > > > >> > > > > > > >> If we consider that the DPDK applications should be able to > > > > > > >> apply its own policy to choose the devices to bind, then we > > > > > > >> need to implement binding in the PMD (with EAL helpers). > > > > > > >> > > > > > > >> On the other hand, if we consider that it is the system > > > > > > >> responsibility, then we could choose systemd/udev and driver= ctl. > > > > > > >> > > > > > > >> The debate is launched! > > > > > > >> > > > > > > > > > > > > > > Allow me to nail my colours to the mast early! :-) > > > > > > > > > > > > > > I believe it's system not application responsibility. > > > > > > > I also believe I have previously explained my reasons for tha= t > > > > > > > choice in some of the previous email threads. > > > > > > > > > > > > For what it's worth, I tend to agree, if only because writing > > > > > > code for what is essentially a bunch of read/write/filesystem > > > > > > enumeration in C is extremely fiddly and error prone :) IMO > > > > > > things like this are better handled either by scripts, or by > > > > > > tools whose sole purpose is doing > > > > exactly that (or both). > > > > > > > > > > > > I like having scripts like devbind in DPDK because we can tailo= r > > > > > > them to our use cases better, and having them is amenable to > > > > > > automation, but while I wouldn't be opposed to removing them > > > > > > altogether in favor of some external tool > > > > > > (systemd/udev/driverctl/whatever), in my humble opinion moving > > > > > > them > > > > back into EAL or even PMD's would be a mistake. > > > > > > > > > > > > > > > > Since the application runs in the system by a command of the > > > > > system user I > > > > think the responsibility is for the user. > > > > > The DPDK user forwards the control of some devices to the DPDK > > > > > application using the EAL whitelist\blacklist mode to specify the > > > > > devices, Any DPDK PMD should know which binding it needs to > > > >> probe\control the device and can apply it, So, if the user asks to > > > >> control on a device by DPDK application it makes sense that the > > > >> application will do the correct binding to the device since the > > > >> user wants to use it(no need to ask more operation of pre binding = from > > the user). > > > > > > > > Completely agree that it is ultimately up to the user. However, wha= t > > > > I don't want to see is the case where the user always has to specif= y > > > > a big long list of device whitelist and blacklist options to each > > > > run of an application. Instead, if device management is done at the > > > > system level via udev (for example) configured via devicectl, then > > > > the application commandline can be vastly simplified. > > > > > > Actually you say that the whitelist\blacklist mechanism is not good e= nough > > and the binding workarounds it. > > > The user need to specify somehow the devices it want to run, I think > > > that specifying the device you want by -w option (no need to specify > > > what you don't want in -w case) is really simpler and more descripti= ve than > > binding each device you want by prior process to its correct driver. > > > > But what if user changes his mind and decides to give particular device= back > > to the kernel? > > Should he restart the dpdk application? In some cases it might be not > > desirable. >=20 > And what is the behavior now in this case? Now we don't have hotplug support at all, so nothing to worry about, I gues= s :) > Looks like if we want to solve it we need to add mechanism to stop these = particular devices DPDK management in any case. I am not talking about managing a device itself (start/stop/attach/detach). Let say you start dpdk with '-w dev -w dev2'. dev1 is active, traffic is going through it, etc., while dev2 is unplugged. Now user is about to plug dev2, though just before that he changed his mind= and decided to give that device to kernel (or different dpdk app), ideally without re-starting = the app. With just command-line support there is no option to do that. Same for opposite case - you started dpdk app with '-w dev1' and then decid= ed that you=20 also want that app to manage hot-pluggable dev2 too.=20 >=20 > > > > It also allows better usability across systems, since the same > > > > commandline can be used on multiple systems with different hardware= , > > > > with the actual device management rules having been already > > > > configured at system install/setup time in udev. > > > > > > But the user still needs to configure the udev per device for each sy= stem, I > > think that command line is better. > > > > > > > > Regarding the conflict of system rules for a device, it is again > > > > > the user > > > > responsibility, whatever we will decide for the binding procedure o= f > > > > DPDK application the user needs to take it into account and to solv= e > > > > such like conflicts. > > > > > One option is to remove any binding rules of a DPDK device in the > > > > > DPDK > > > > application initialization and adjust the new rules by the PMDs, > > > > then any conflict should not disturb the user. > > > > > > > > If the device management is only managed in one place, i.e. not in > > > > DPDK, then there is no conflict to manage. > > > > > > I can't agree with this statement, > > > The essence of DPDK is to give a good alternative to managing network > > > devices, DPDK actually takes a lot of management area to manage by > > > itself to do the user life better :) > > > > From my point - it is not about managing particular device. > > It is about making decision who (kenel/dpdk) will manage that device. >=20 > Doesn't the w\b mechanism comes to solve it? It does till some extent, but I don't think it is the best possible way. >=20 > > From usability perspective it seems to me that better to keep it in one= place > > for all devices. > > So udev (or any other sysadmin tool) seems like a right choice here. >=20 > Please explain why? > And if you are talking about 1 place: > Why not to do the binding in the same place where we define which device = to run? Not all devices are always managed by dpdk apps. Some of them will still be managed by kernel. Again there could be several independent dpdk apps running on the same syst= em. Obviously dpdk app itself can't be a centric place to handle all that varie= ties. Again - if there exists a system tool that provides that functionality - wh= y not to use it? > Why not to ask from the user to specify the device and that's it? >=20 > > > Moreover, > > > Instead of bind script usage and dpdk running , just run dpdk (do all= the job > > in one place). > > > > > > > > In current hot-plug case the application will need to do a lot of > > > > > work to > > > > bind\remap devices in plug-in\plug-out events while the PMD could > > > > have all the knowledge to do it. > > > > > > > > At the cost of duplicating a lot of code between PMDs. > > > > > > Why a lot of code? 1 helper in EAL to be used for each relevant PMD. > > > Each PMD just call to the EAL helper with the bind driver type. > > > Really simple and immediate. > > > > > > I think it is better than duplication of user operations. > > > >=20 > And what's about hot-plug? Who should do the bind in plug-in event? >=20 > > > > > One more issue with the script is that the user should do > > > > > different bind per > > > > device, in case of PMD responsibility the user can forget it: > > > > > Think about that, any time the user wants to switch\add new > > > > > supported nic > > > > it should update the script usage and to do per nic operation > > > > contrary to the DPDK principles. > > > > > > > > > > > > > The udev rules syntax should provide adequate capabilities here for > > > > us to match the correct binding behaviour. No need to have it in DP= DK > > too. > > > > > > We can use it from DPDK. > > > > > > > /Bruce