From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga17.intel.com (mga17.intel.com [192.55.52.151]) by dpdk.org (Postfix) with ESMTP id 2CDBEA495 for ; Tue, 17 Apr 2018 11:23:31 +0200 (CEST) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga003.fm.intel.com ([10.253.24.29]) by fmsmga107.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 17 Apr 2018 02:23:29 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.48,463,1517904000"; d="scan'208";a="42549234" Received: from irsmsx153.ger.corp.intel.com ([163.33.192.75]) by FMSMGA003.fm.intel.com with ESMTP; 17 Apr 2018 02:23:27 -0700 Received: from irsmsx112.ger.corp.intel.com (10.108.20.5) by IRSMSX153.ger.corp.intel.com (163.33.192.75) with Microsoft SMTP Server (TLS) id 14.3.319.2; Tue, 17 Apr 2018 10:23:26 +0100 Received: from irsmsx102.ger.corp.intel.com ([169.254.2.164]) by irsmsx112.ger.corp.intel.com ([169.254.1.226]) with mapi id 14.03.0319.002; Tue, 17 Apr 2018 10:23:26 +0100 From: "Ananyev, Konstantin" To: Matan Azrad , "Richardson, Bruce" CC: "Burakov, Anatoly" , Thomas Monjalon , "dev@dpdk.org" , "pmatilai@redhat.com" , "david.marchand@6wind.com" , "Guo, Jia" , "stephen@networkplumber.org" , "fbl@redhat.com" Thread-Topic: kernel binding of devices + hotplug Thread-Index: AQHT00TmmE7o5Zr910mkyHe0bqbSjKP+1QoAgAAQwwCAAbwpAIACYXuAgACAVACAAS5iMA== Date: Tue, 17 Apr 2018 09:23:25 +0000 Message-ID: <2601191342CEEE43887BDE71AB977258AE916938@IRSMSX102.ger.corp.intel.com> References: <2407757.yEAnF6RcS7@xps> <20180413164046.GD37024@bricha3-MOBL.ger.corp.intel.com> <20180416083153.GA50020@bricha3-MOBL.ger.corp.intel.com> In-Reply-To: Accept-Language: en-IE, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiNDVlY2UxYzktMjk4NS00OTQ2LWI3YmItODVkMWFlYmQyZTFjIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE2LjUuOS4zIiwiVHJ1c3RlZExhYmVsSGFzaCI6IlorTlRYV1BTM3NpYzJZRHJDT01uQ0lzdGxvZ3h3ekZiNUlhbGhxOVliNms9In0= x-ctpclassification: CTP_NT dlp-product: dlpe-windows dlp-version: 11.0.200.100 dlp-reaction: no-action x-originating-ip: [163.33.239.181] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] kernel binding of devices + hotplug X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Apr 2018 09:23:32 -0000 Hi everyone, Few comments from me below. In summary I am also in favor to keep binding decision at system level (out= side DPDK). Konstantin > -----Original Message----- > From: Matan Azrad [mailto:matan@mellanox.com] > Sent: Monday, April 16, 2018 5:11 PM > To: Richardson, Bruce > Cc: Burakov, Anatoly ; Thomas Monjalon ; dev@dpdk.org; pmatilai@redhat.com; > david.marchand@6wind.com; Guo, Jia ; Ananyev, Konstant= in ; > stephen@networkplumber.org; fbl@redhat.com > Subject: RE: kernel binding of devices + hotplug >=20 > Hi Bruce >=20 > From: Bruce Richardson, Monday, April 16, 2018 11:32 AM > > On Sat, Apr 14, 2018 at 08:10:28PM +0000, Matan Azrad wrote: > > > Hi all > > > > > > From: Burakov, Anatoly, Friday, April 13, 2018 8:41 PM > > > > To: Bruce Richardson ; Thomas Monjalon > > > > > > > > Cc: dev@dpdk.org; pmatilai@redhat.com; david.marchand@6wind.com; > > > > jia.guo@intel.com; Matan Azrad ; > > > > konstantin.ananyev@intel.com; stephen@networkplumber.org; > > > > fbl@redhat.com > > > > Subject: Re: kernel binding of devices + hotplug > > > > > > > > On 13-Apr-18 5:40 PM, Bruce Richardson wrote: > > > > > On Fri, Apr 13, 2018 at 06:31:21PM +0200, Thomas Monjalon wrote: > > > > >> It's time to think (again) how we bind devices with kernel modul= es. > > > > >> We need to decide how we want to manage hotplugged devices with > > > > DPDK. > > > > >> > > > > >> A bit of history first. > > > > >> There was some code in DPDK for bind/unbind, but it has been > > > > >> removed in DPDK 1.7 - > > > > >> > > > > > > https://emea01.safelinks.protection.outlook.com/?url=3Dhttp%3A%2F%2Fdp > > > > d > > > > >> > > > > > > k.org%2Fcommit%2F5d8751b83&data=3D02%7C01%7Cmatan%40mellanox.com > > > > %7C6ea5 > > > > >> > > > > > > 5ce994ff4bb0d65208d5a165b417%7Ca652971c7d2e4d9ba6a4d149256f461b%7 > > > > C0%7 > > > > >> > > > > > > C0%7C636592380565078675&sdata=3DuLRDAk65hYtJYxjIvY20de377yayCN5DrjCZ > > > > x8H > > > > >> p61o%3D&reserved=3D0 Copy of the commit message (in 2014): > > > > >> " > > > > >> The bind/unbind operations should not be handled by the eal. > > > > >> These operations should be either done outside of dpdk or > > > > >> inside the PMDs themselves as these are their problems. > > > > >> " > > > > >> > > > > >> The question raised at this time (4 years ago) is still under di= scussion. > > > > >> Should we manage binding inside or outside DPDK? > > > > >> Should it be controlled in the application or in the OS base? > > > > >> > > > > >> As you know, we use dpdk-devbind.py. > > > > >> This tool lacks two major features: > > > > >> - persistent configuration > > > > >> - hotplug > > > > >> > > > > >> If we consider that the DPDK applications should be able to appl= y > > > > >> its own policy to choose the devices to bind, then we need to > > > > >> implement binding in the PMD (with EAL helpers). > > > > >> > > > > >> On the other hand, if we consider that it is the system > > > > >> responsibility, then we could choose systemd/udev and driverctl. > > > > >> > > > > >> The debate is launched! > > > > >> > > > > > > > > > > Allow me to nail my colours to the mast early! :-) > > > > > > > > > > I believe it's system not application responsibility. > > > > > I also believe I have previously explained my reasons for that > > > > > choice in some of the previous email threads. > > > > > > > > For what it's worth, I tend to agree, if only because writing code > > > > for what is essentially a bunch of read/write/filesystem enumeratio= n > > > > in C is extremely fiddly and error prone :) IMO things like this ar= e > > > > better handled either by scripts, or by tools whose sole purpose is= doing > > exactly that (or both). > > > > > > > > I like having scripts like devbind in DPDK because we can tailor > > > > them to our use cases better, and having them is amenable to > > > > automation, but while I wouldn't be opposed to removing them > > > > altogether in favor of some external tool > > > > (systemd/udev/driverctl/whatever), in my humble opinion moving them > > back into EAL or even PMD's would be a mistake. > > > > > > > > > > Since the application runs in the system by a command of the system u= ser I > > think the responsibility is for the user. > > > The DPDK user forwards the control of some devices to the DPDK > > > application using the EAL whitelist\blacklist mode to specify the > > > devices, Any DPDK PMD should know which binding it needs to > >> probe\control the device and can apply it, So, if the user asks to con= trol on a > >> device by DPDK application it makes sense that the application will do= the > >> correct binding to the device since the user wants to use it(no need t= o ask > >> more operation of pre binding from the user). > > > > Completely agree that it is ultimately up to the user. However, what I = don't > > want to see is the case where the user always has to specify a big long= list of > > device whitelist and blacklist options to each run of an application. I= nstead, if > > device management is done at the system level via udev (for example) > > configured via devicectl, then the application commandline can be vastl= y > > simplified. >=20 > Actually you say that the whitelist\blacklist mechanism is not good enoug= h and the binding workarounds it. > The user need to specify somehow the devices it want to run, > I think that specifying the device you want by -w option (no need to spec= ify what you don't want in -w case) is really simpler and more > descriptive than binding each device you want by prior process to its cor= rect driver. But what if user changes his mind and decides to give particular device bac= k to the kernel? Should he restart the dpdk application? In some cases it might be not desir= able. >=20 > > It also allows better usability across systems, since the same > > commandline can be used on multiple systems with different hardware, wi= th > > the actual device management rules having been already configured at > > system install/setup time in udev. >=20 > But the user still needs to configure the udev per device for each system= , I think that command line is better. >=20 > > > Regarding the conflict of system rules for a device, it is again the = user > > responsibility, whatever we will decide for the binding procedure of DP= DK > > application the user needs to take it into account and to solve such li= ke > > conflicts. > > > One option is to remove any binding rules of a DPDK device in the DPD= K > > application initialization and adjust the new rules by the PMDs, then a= ny > > conflict should not disturb the user. > > > > If the device management is only managed in one place, i.e. not in DPDK= , > > then there is no conflict to manage. >=20 > I can't agree with this statement, > The essence of DPDK is to give a good alternative to managing network dev= ices, > DPDK actually takes a lot of management area to manage by itself to do th= e user life better :) >>From my point - it is not about managing particular device. It is about making decision who (kenel/dpdk) will manage that device. >>From usability perspective it seems to me that better to keep it in one pla= ce for all devices. So udev (or any other sysadmin tool) seems like a right choice here.=20 >=20 > Moreover, > Instead of bind script usage and dpdk running , just run dpdk (do all the= job in one place). >=20 > > > In current hot-plug case the application will need to do a lot of wor= k to > > bind\remap devices in plug-in\plug-out events while the PMD could have = all > > the knowledge to do it. > > > > At the cost of duplicating a lot of code between PMDs. >=20 > Why a lot of code? 1 helper in EAL to be used for each relevant PMD. > Each PMD just call to the EAL helper with the bind driver type. > Really simple and immediate. >=20 > I think it is better than duplication of user operations. >=20 > > > One more issue with the script is that the user should do different b= ind per > > device, in case of PMD responsibility the user can forget it: > > > Think about that, any time the user wants to switch\add new supported= nic > > it should update the script usage and to do per nic operation contrary = to the > > DPDK principles. > > > > > > > The udev rules syntax should provide adequate capabilities here for us = to > > match the correct binding behaviour. No need to have it in DPDK too. >=20 > We can use it from DPDK. >=20 > > /Bruce