From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR01-DB5-obe.outbound.protection.outlook.com (mail-db5eur01on0088.outbound.protection.outlook.com [104.47.2.88]) by dpdk.org (Postfix) with ESMTP id 469472C2F for ; Tue, 17 Apr 2018 12:42:11 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=11hk6aZzH5RPnPbl91UCLaa73R1M42ai3USIar7yhq0=; b=X+Vxuhe2tB4kjODssBSbrr+oHDZpaIzSay9qEKgizGmGnSc6Kn8QBJqWxdOPWit0N6Yg61omdpgolAEYovNubRxkutKVMKRxAe0/6wHfMyjPVkCMX4tjIciWeijWe9qX7MW10gZAwzVHr4GwAXV3H0+eMGhKga9HS5AUhF2SxBY= Received: from AM4PR0501MB2657.eurprd05.prod.outlook.com (10.172.215.19) by AM4PR0501MB2642.eurprd05.prod.outlook.com (10.172.215.16) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.696.12; Tue, 17 Apr 2018 10:42:09 +0000 Received: from AM4PR0501MB2657.eurprd05.prod.outlook.com ([fe80::6885:c169:afcb:37e6]) by AM4PR0501MB2657.eurprd05.prod.outlook.com ([fe80::6885:c169:afcb:37e6%9]) with mapi id 15.20.0696.011; Tue, 17 Apr 2018 10:42:09 +0000 From: Matan Azrad To: "Ananyev, Konstantin" , "Richardson, Bruce" CC: "Burakov, Anatoly" , Thomas Monjalon , "dev@dpdk.org" , "pmatilai@redhat.com" , "david.marchand@6wind.com" , "Guo, Jia" , "stephen@networkplumber.org" , "fbl@redhat.com" Thread-Topic: kernel binding of devices + hotplug Thread-Index: AQHT00ThIlNWXyeugUC+Eesfo2Tt3qP+5c0AgAAQxACAAasfUIACcoSAgAAcrMCAAYQPgIAADrow Date: Tue, 17 Apr 2018 10:42:08 +0000 Message-ID: References: <2407757.yEAnF6RcS7@xps> <20180413164046.GD37024@bricha3-MOBL.ger.corp.intel.com> <20180416083153.GA50020@bricha3-MOBL.ger.corp.intel.com> <2601191342CEEE43887BDE71AB977258AE916938@IRSMSX102.ger.corp.intel.com> In-Reply-To: <2601191342CEEE43887BDE71AB977258AE916938@IRSMSX102.ger.corp.intel.com> Accept-Language: en-US, he-IL Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=matan@mellanox.com; x-originating-ip: [193.47.165.251] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; AM4PR0501MB2642; 7:/tMMwNM8R0dc8ymviD0DXH1dhY4hfxMpuCXNL2mf1fyc5SQ4KFc/jAIwHXTME3lnvIIuNoqLeKphL1gKG+QFHAVN+xkUhJm7xc/Zvg5xP8CP0BlWBZ2c6VCR5W47R8zdzdZ1KENGj0XrRQ6yWuxuh1JHLkpgXWCri4MWltaVYwaSejGHxGEDTOlhz0JE4AIkHjqz38I/12R0KZjEwStz+JaVMRHTo3/ggOiehhNotLGIYpee1Gy6ik/MPSekttAQ x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(5600026)(4534165)(4627221)(201703031133081)(201702281549075)(48565401081)(2017052603328)(7153060)(7193020); SRVR:AM4PR0501MB2642; x-ms-traffictypediagnostic: AM4PR0501MB2642: x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(278428928389397)(189930954265078)(45079756050767)(228905959029699)(17755550239193); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(8121501046)(5005006)(10201501046)(93006095)(93001095)(3002001)(3231232)(944501327)(52105095)(6055026)(6041310)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123564045)(20161123560045)(20161123558120)(20161123562045)(6072148)(201708071742011); SRVR:AM4PR0501MB2642; BCL:0; PCL:0; RULEID:; SRVR:AM4PR0501MB2642; x-forefront-prvs: 0645BEB7AA x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(39860400002)(376002)(39380400002)(366004)(346002)(396003)(53754006)(189003)(199004)(13464003)(81156014)(106356001)(81166006)(14454004)(3660700001)(110136005)(93886005)(3280700002)(6116002)(54906003)(966005)(3846002)(2900100001)(6246003)(7416002)(68736007)(86362001)(7736002)(5250100002)(4326008)(305945005)(478600001)(45080400002)(25786009)(53936002)(486006)(575784001)(5660300001)(74316002)(446003)(26005)(33656002)(476003)(11346002)(102836004)(105586002)(6506007)(53546011)(76176011)(99286004)(59450400001)(7696005)(229853002)(186003)(55016002)(316002)(6436002)(9686003)(97736004)(2906002)(66066001)(8936002)(6306002)(8676002); DIR:OUT; SFP:1101; SCL:1; SRVR:AM4PR0501MB2642; H:AM4PR0501MB2657.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: Z+3IRdE/GOuM7/RRAL+895V9aTxkI879vNJElyH3lEwsc4P6BiyHdaLlXErO/03WeAElNQuhBYEURNlID/1cSByw2equlIPFAIBhIxqpGCtbTicnwXEIa6Ye0iw5FjPwWEWlESYS7R8SsA80FneSaTgVjqQuZxK2Z37y80ClfZ0myMEoYPdaqJVN4LYRv6Zp spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: 191ec6d0-442a-4c81-8f89-08d5a44fdef2 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: 191ec6d0-442a-4c81-8f89-08d5a44fdef2 X-MS-Exchange-CrossTenant-originalarrivaltime: 17 Apr 2018 10:42:09.0671 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM4PR0501MB2642 Subject: Re: [dpdk-dev] kernel binding of devices + hotplug X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Tue, 17 Apr 2018 10:42:11 -0000 Hi Konstantin From: Ananyev, Konstantin, Tuesday, April 17, 2018 12:23 PM > To: Matan Azrad ; Richardson, Bruce > > Cc: Burakov, Anatoly ; Thomas Monjalon > ; dev@dpdk.org; pmatilai@redhat.com; > david.marchand@6wind.com; Guo, Jia ; > stephen@networkplumber.org; fbl@redhat.com > Subject: RE: kernel binding of devices + hotplug >=20 >=20 > Hi everyone, > Few comments from me below. > In summary I am also in favor to keep binding decision at system level > (outside DPDK). > Konstantin >=20 > > -----Original Message----- > > From: Matan Azrad [mailto:matan@mellanox.com] > > Sent: Monday, April 16, 2018 5:11 PM > > To: Richardson, Bruce > > Cc: Burakov, Anatoly ; Thomas Monjalon > > ; dev@dpdk.org; pmatilai@redhat.com; > > david.marchand@6wind.com; Guo, Jia ; Ananyev, > > Konstantin ; > stephen@networkplumber.org; > > fbl@redhat.com > > Subject: RE: kernel binding of devices + hotplug > > > > Hi Bruce > > > > From: Bruce Richardson, Monday, April 16, 2018 11:32 AM > > > On Sat, Apr 14, 2018 at 08:10:28PM +0000, Matan Azrad wrote: > > > > Hi all > > > > > > > > From: Burakov, Anatoly, Friday, April 13, 2018 8:41 PM > > > > > To: Bruce Richardson ; Thomas > > > > > Monjalon > > > > > Cc: dev@dpdk.org; pmatilai@redhat.com; > david.marchand@6wind.com; > > > > > jia.guo@intel.com; Matan Azrad ; > > > > > konstantin.ananyev@intel.com; stephen@networkplumber.org; > > > > > fbl@redhat.com > > > > > Subject: Re: kernel binding of devices + hotplug > > > > > > > > > > On 13-Apr-18 5:40 PM, Bruce Richardson wrote: > > > > > > On Fri, Apr 13, 2018 at 06:31:21PM +0200, Thomas Monjalon wrote= : > > > > > >> It's time to think (again) how we bind devices with kernel mod= ules. > > > > > >> We need to decide how we want to manage hotplugged devices > > > > > >> with > > > > > DPDK. > > > > > >> > > > > > >> A bit of history first. > > > > > >> There was some code in DPDK for bind/unbind, but it has been > > > > > >> removed in DPDK 1.7 - > > > > > >> > > > > > > > > > https://emea01.safelinks.protection.outlook.com/?url=3Dhttp%3A%2F%2Fdp > > > > > d > > > > > >> > > > > > > > > > k.org%2Fcommit%2F5d8751b83&data=3D02%7C01%7Cmatan%40mellanox.com > > > > > %7C6ea5 > > > > > >> > > > > > > > > > 5ce994ff4bb0d65208d5a165b417%7Ca652971c7d2e4d9ba6a4d149256f461b%7 > > > > > C0%7 > > > > > >> > > > > > > > > > C0%7C636592380565078675&sdata=3DuLRDAk65hYtJYxjIvY20de377yayCN5DrjCZ > > > > > x8H > > > > > >> p61o%3D&reserved=3D0 Copy of the commit message (in 2014): > > > > > >> " > > > > > >> The bind/unbind operations should not be handled by the > eal. > > > > > >> These operations should be either done outside of dpdk or > > > > > >> inside the PMDs themselves as these are their problems. > > > > > >> " > > > > > >> > > > > > >> The question raised at this time (4 years ago) is still under > discussion. > > > > > >> Should we manage binding inside or outside DPDK? > > > > > >> Should it be controlled in the application or in the OS base? > > > > > >> > > > > > >> As you know, we use dpdk-devbind.py. > > > > > >> This tool lacks two major features: > > > > > >> - persistent configuration > > > > > >> - hotplug > > > > > >> > > > > > >> If we consider that the DPDK applications should be able to > > > > > >> apply its own policy to choose the devices to bind, then we > > > > > >> need to implement binding in the PMD (with EAL helpers). > > > > > >> > > > > > >> On the other hand, if we consider that it is the system > > > > > >> responsibility, then we could choose systemd/udev and driverct= l. > > > > > >> > > > > > >> The debate is launched! > > > > > >> > > > > > > > > > > > > Allow me to nail my colours to the mast early! :-) > > > > > > > > > > > > I believe it's system not application responsibility. > > > > > > I also believe I have previously explained my reasons for that > > > > > > choice in some of the previous email threads. > > > > > > > > > > For what it's worth, I tend to agree, if only because writing > > > > > code for what is essentially a bunch of read/write/filesystem > > > > > enumeration in C is extremely fiddly and error prone :) IMO > > > > > things like this are better handled either by scripts, or by > > > > > tools whose sole purpose is doing > > > exactly that (or both). > > > > > > > > > > I like having scripts like devbind in DPDK because we can tailor > > > > > them to our use cases better, and having them is amenable to > > > > > automation, but while I wouldn't be opposed to removing them > > > > > altogether in favor of some external tool > > > > > (systemd/udev/driverctl/whatever), in my humble opinion moving > > > > > them > > > back into EAL or even PMD's would be a mistake. > > > > > > > > > > > > > Since the application runs in the system by a command of the > > > > system user I > > > think the responsibility is for the user. > > > > The DPDK user forwards the control of some devices to the DPDK > > > > application using the EAL whitelist\blacklist mode to specify the > > > > devices, Any DPDK PMD should know which binding it needs to > > >> probe\control the device and can apply it, So, if the user asks to > > >> control on a device by DPDK application it makes sense that the > > >> application will do the correct binding to the device since the > > >> user wants to use it(no need to ask more operation of pre binding fr= om > the user). > > > > > > Completely agree that it is ultimately up to the user. However, what > > > I don't want to see is the case where the user always has to specify > > > a big long list of device whitelist and blacklist options to each > > > run of an application. Instead, if device management is done at the > > > system level via udev (for example) configured via devicectl, then > > > the application commandline can be vastly simplified. > > > > Actually you say that the whitelist\blacklist mechanism is not good eno= ugh > and the binding workarounds it. > > The user need to specify somehow the devices it want to run, I think > > that specifying the device you want by -w option (no need to specify > > what you don't want in -w case) is really simpler and more descriptive= than > binding each device you want by prior process to its correct driver. >=20 > But what if user changes his mind and decides to give particular device b= ack > to the kernel? > Should he restart the dpdk application? In some cases it might be not > desirable. And what is the behavior now in this case? Looks like if we want to solve it we need to add mechanism to stop these pa= rticular devices DPDK management in any case. =20 > > > It also allows better usability across systems, since the same > > > commandline can be used on multiple systems with different hardware, > > > with the actual device management rules having been already > > > configured at system install/setup time in udev. > > > > But the user still needs to configure the udev per device for each syst= em, I > think that command line is better. > > > > > > Regarding the conflict of system rules for a device, it is again > > > > the user > > > responsibility, whatever we will decide for the binding procedure of > > > DPDK application the user needs to take it into account and to solve > > > such like conflicts. > > > > One option is to remove any binding rules of a DPDK device in the > > > > DPDK > > > application initialization and adjust the new rules by the PMDs, > > > then any conflict should not disturb the user. > > > > > > If the device management is only managed in one place, i.e. not in > > > DPDK, then there is no conflict to manage. > > > > I can't agree with this statement, > > The essence of DPDK is to give a good alternative to managing network > > devices, DPDK actually takes a lot of management area to manage by > > itself to do the user life better :) >=20 > From my point - it is not about managing particular device. > It is about making decision who (kenel/dpdk) will manage that device. Doesn't the w\b mechanism comes to solve it? > From usability perspective it seems to me that better to keep it in one p= lace > for all devices. > So udev (or any other sysadmin tool) seems like a right choice here. Please explain why? And if you are talking about 1 place: Why not to do the binding in the same place where we define which device to= run? Why not to ask from the user to specify the device and that's it? > > Moreover, > > Instead of bind script usage and dpdk running , just run dpdk (do all t= he job > in one place). > > > > > > In current hot-plug case the application will need to do a lot of > > > > work to > > > bind\remap devices in plug-in\plug-out events while the PMD could > > > have all the knowledge to do it. > > > > > > At the cost of duplicating a lot of code between PMDs. > > > > Why a lot of code? 1 helper in EAL to be used for each relevant PMD. > > Each PMD just call to the EAL helper with the bind driver type. > > Really simple and immediate. > > > > I think it is better than duplication of user operations. > > And what's about hot-plug? Who should do the bind in plug-in event? =20 > > > > One more issue with the script is that the user should do > > > > different bind per > > > device, in case of PMD responsibility the user can forget it: > > > > Think about that, any time the user wants to switch\add new > > > > supported nic > > > it should update the script usage and to do per nic operation > > > contrary to the DPDK principles. > > > > > > > > > > The udev rules syntax should provide adequate capabilities here for > > > us to match the correct binding behaviour. No need to have it in DPDK > too. > > > > We can use it from DPDK. > > > > > /Bruce