From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from EUR01-VE1-obe.outbound.protection.outlook.com (mail-ve1eur01on0089.outbound.protection.outlook.com [104.47.1.89]) by dpdk.org (Postfix) with ESMTP id 2D630AAD4 for ; Mon, 16 Apr 2018 18:11:14 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector1; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=wf3Vt6oASd0VKNo8OfFnU2MbAezV+dT49dd120jpW5I=; b=l/+DavWsCXazsgPe+pylMWXjsgMDgR0jRAO77mCvfbqI2Px0PPon7Q0VrJ2vUfdzfAnCL6aRvGgDVHTYCviGTnPOrFSlYCfTHK7YGrB9/czULQA1vPRBXQywn9Tt11C+jtYA2UZWob/Dx67tCwdhFzzm9xQ4CFjGHqnbfGmRZP8= Received: from AM4PR0501MB2657.eurprd05.prod.outlook.com (10.172.215.19) by AM4PR0501MB2691.eurprd05.prod.outlook.com (10.172.215.140) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.675.10; Mon, 16 Apr 2018 16:11:12 +0000 Received: from AM4PR0501MB2657.eurprd05.prod.outlook.com ([fe80::6885:c169:afcb:37e6]) by AM4PR0501MB2657.eurprd05.prod.outlook.com ([fe80::6885:c169:afcb:37e6%9]) with mapi id 15.20.0675.015; Mon, 16 Apr 2018 16:11:12 +0000 From: Matan Azrad To: Bruce Richardson CC: "Burakov, Anatoly" , Thomas Monjalon , "dev@dpdk.org" , "pmatilai@redhat.com" , "david.marchand@6wind.com" , "jia.guo@intel.com" , "konstantin.ananyev@intel.com" , "stephen@networkplumber.org" , "fbl@redhat.com" Thread-Topic: kernel binding of devices + hotplug Thread-Index: AQHT00ThIlNWXyeugUC+Eesfo2Tt3qP+5c0AgAAQxACAAasfUIACcoSAgAAcrMA= Date: Mon, 16 Apr 2018 16:11:12 +0000 Message-ID: References: <2407757.yEAnF6RcS7@xps> <20180413164046.GD37024@bricha3-MOBL.ger.corp.intel.com> <20180416083153.GA50020@bricha3-MOBL.ger.corp.intel.com> In-Reply-To: <20180416083153.GA50020@bricha3-MOBL.ger.corp.intel.com> Accept-Language: en-US, he-IL Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=matan@mellanox.com; x-originating-ip: [193.47.165.251] x-ms-publictraffictype: Email x-microsoft-exchange-diagnostics: 1; AM4PR0501MB2691; 7:AbVIazmA4WOJmq+alGjbbKw5xN269jubOyHUi8P9vkWN/4+YuayJiCXOMuElas8B2BICTWEdGXJ+wuLfTcB/jobg7pV+696ACdeXt2onyAt7HDwkkEegaHmS0bkNUUJSPpSnjS4/jh3GR5PB8h9qf5oLtPsPOjjt5GaW20gUBkyhciPatMbkbd6BHthx2a972WUeCFrXfX6xZp0sPsg2bawzISG2+3jx9UAHvhm1U1lgHqapZMaGBwQuvqey8D7Q x-ms-exchange-antispam-srfa-diagnostics: SOS; x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: UriScan:; BCL:0; PCL:0; RULEID:(7020095)(4652020)(5600026)(4534165)(4627221)(201703031133081)(201702281549075)(48565401081)(2017052603328)(7153060)(7193020); SRVR:AM4PR0501MB2691; x-ms-traffictypediagnostic: AM4PR0501MB2691: x-ld-processed: a652971c-7d2e-4d9b-a6a4-d149256f461b,ExtAddr x-microsoft-antispam-prvs: x-exchange-antispam-report-test: UriScan:(278428928389397)(189930954265078)(45079756050767)(228905959029699)(17755550239193); x-exchange-antispam-report-cfa-test: BCL:0; PCL:0; RULEID:(8211001083)(6040522)(2401047)(8121501046)(5005006)(93006095)(93001095)(3231232)(944501327)(52105095)(3002001)(10201501046)(6055026)(6041310)(20161123564045)(201703131423095)(201702281528075)(20161123555045)(201703061421075)(201703061406153)(20161123560045)(20161123558120)(20161123562045)(6072148)(201708071742011); SRVR:AM4PR0501MB2691; BCL:0; PCL:0; RULEID:; SRVR:AM4PR0501MB2691; x-forefront-prvs: 0644578634 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(346002)(39380400002)(376002)(39860400002)(366004)(396003)(53754006)(199004)(189003)(51444003)(6506007)(3846002)(53546011)(45080400002)(54906003)(6116002)(26005)(5660300001)(106356001)(102836004)(4326008)(59450400001)(478600001)(186003)(76176011)(7696005)(99286004)(316002)(8936002)(7416002)(446003)(25786009)(11346002)(305945005)(7736002)(6246003)(74316002)(86362001)(575784001)(81166006)(81156014)(93886005)(33656002)(6436002)(2900100001)(2906002)(9686003)(55016002)(229853002)(6306002)(105586002)(6916009)(3660700001)(8676002)(97736004)(3280700002)(966005)(66066001)(5250100002)(476003)(14454004)(68736007)(53936002)(486006); DIR:OUT; SFP:1101; SCL:1; SRVR:AM4PR0501MB2691; H:AM4PR0501MB2657.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; A:1; MX:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-microsoft-antispam-message-info: 8buq85qmneTQY1IEtSwIkdE/xPJdr6eLT1qecKX2d5G6jHMk8fKfOr2TW5JqzEIq0tQQhIUvMcHmM6W03fsQ/zkIKvfqH5R0eaP7hLRZ15pnp9luQP3OAwAzqYfO1V8Ap9/5BhRcqqSS3iXQyzrXnwLRQCMBUF+tdnQrhTfBSfxOWAjz9ldM1QnVl/8iNf1c spamdiagnosticoutput: 1:99 spamdiagnosticmetadata: NSPM Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-MS-Office365-Filtering-Correlation-Id: d4421406-dd04-41b1-660b-08d5a3b4ac70 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: d4421406-dd04-41b1-660b-08d5a3b4ac70 X-MS-Exchange-CrossTenant-originalarrivaltime: 16 Apr 2018 16:11:12.4270 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM4PR0501MB2691 Subject: Re: [dpdk-dev] kernel binding of devices + hotplug X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 16 Apr 2018 16:11:14 -0000 Hi Bruce From: Bruce Richardson, Monday, April 16, 2018 11:32 AM > On Sat, Apr 14, 2018 at 08:10:28PM +0000, Matan Azrad wrote: > > Hi all > > > > From: Burakov, Anatoly, Friday, April 13, 2018 8:41 PM > > > To: Bruce Richardson ; Thomas Monjalon > > > > > > Cc: dev@dpdk.org; pmatilai@redhat.com; david.marchand@6wind.com; > > > jia.guo@intel.com; Matan Azrad ; > > > konstantin.ananyev@intel.com; stephen@networkplumber.org; > > > fbl@redhat.com > > > Subject: Re: kernel binding of devices + hotplug > > > > > > On 13-Apr-18 5:40 PM, Bruce Richardson wrote: > > > > On Fri, Apr 13, 2018 at 06:31:21PM +0200, Thomas Monjalon wrote: > > > >> It's time to think (again) how we bind devices with kernel modules= . > > > >> We need to decide how we want to manage hotplugged devices with > > > DPDK. > > > >> > > > >> A bit of history first. > > > >> There was some code in DPDK for bind/unbind, but it has been > > > >> removed in DPDK 1.7 - > > > >> > > > > https://emea01.safelinks.protection.outlook.com/?url=3Dhttp%3A%2F%2Fdp > > > d > > > >> > > > > k.org%2Fcommit%2F5d8751b83&data=3D02%7C01%7Cmatan%40mellanox.com > > > %7C6ea5 > > > >> > > > > 5ce994ff4bb0d65208d5a165b417%7Ca652971c7d2e4d9ba6a4d149256f461b%7 > > > C0%7 > > > >> > > > > C0%7C636592380565078675&sdata=3DuLRDAk65hYtJYxjIvY20de377yayCN5DrjCZ > > > x8H > > > >> p61o%3D&reserved=3D0 Copy of the commit message (in 2014): > > > >> " > > > >> The bind/unbind operations should not be handled by the eal. > > > >> These operations should be either done outside of dpdk or > > > >> inside the PMDs themselves as these are their problems. > > > >> " > > > >> > > > >> The question raised at this time (4 years ago) is still under disc= ussion. > > > >> Should we manage binding inside or outside DPDK? > > > >> Should it be controlled in the application or in the OS base? > > > >> > > > >> As you know, we use dpdk-devbind.py. > > > >> This tool lacks two major features: > > > >> - persistent configuration > > > >> - hotplug > > > >> > > > >> If we consider that the DPDK applications should be able to apply > > > >> its own policy to choose the devices to bind, then we need to > > > >> implement binding in the PMD (with EAL helpers). > > > >> > > > >> On the other hand, if we consider that it is the system > > > >> responsibility, then we could choose systemd/udev and driverctl. > > > >> > > > >> The debate is launched! > > > >> > > > > > > > > Allow me to nail my colours to the mast early! :-) > > > > > > > > I believe it's system not application responsibility. > > > > I also believe I have previously explained my reasons for that > > > > choice in some of the previous email threads. > > > > > > For what it's worth, I tend to agree, if only because writing code > > > for what is essentially a bunch of read/write/filesystem enumeration > > > in C is extremely fiddly and error prone :) IMO things like this are > > > better handled either by scripts, or by tools whose sole purpose is d= oing > exactly that (or both). > > > > > > I like having scripts like devbind in DPDK because we can tailor > > > them to our use cases better, and having them is amenable to > > > automation, but while I wouldn't be opposed to removing them > > > altogether in favor of some external tool > > > (systemd/udev/driverctl/whatever), in my humble opinion moving them > back into EAL or even PMD's would be a mistake. > > > > > > > Since the application runs in the system by a command of the system use= r I > think the responsibility is for the user. > > The DPDK user forwards the control of some devices to the DPDK > > application using the EAL whitelist\blacklist mode to specify the > > devices, Any DPDK PMD should know which binding it needs to >> probe\control the device and can apply it, So, if the user asks to contr= ol on a >> device by DPDK application it makes sense that the application will do t= he >> correct binding to the device since the user wants to use it(no need to = ask >> more operation of pre binding from the user). >=20 > Completely agree that it is ultimately up to the user. However, what I do= n't > want to see is the case where the user always has to specify a big long l= ist of > device whitelist and blacklist options to each run of an application. Ins= tead, if > device management is done at the system level via udev (for example) > configured via devicectl, then the application commandline can be vastly > simplified. Actually you say that the whitelist\blacklist mechanism is not good enough = and the binding workarounds it. The user need to specify somehow the devices it want to run,=20 I think that specifying the device you want by -w option (no need to specif= y what you don't want in -w case) is really simpler and more descriptive t= han binding each device you want by prior process to its correct driver. =20 > It also allows better usability across systems, since the same > commandline can be used on multiple systems with different hardware, with > the actual device management rules having been already configured at > system install/setup time in udev. But the user still needs to configure the udev per device for each system, = I think that command line is better. > > Regarding the conflict of system rules for a device, it is again the us= er > responsibility, whatever we will decide for the binding procedure of DPDK > application the user needs to take it into account and to solve such like > conflicts. > > One option is to remove any binding rules of a DPDK device in the DPDK > application initialization and adjust the new rules by the PMDs, then any > conflict should not disturb the user. >=20 > If the device management is only managed in one place, i.e. not in DPDK, > then there is no conflict to manage. I can't agree with this statement, The essence of DPDK is to give a good alternative to managing network devic= es, DPDK actually takes a lot of management area to manage by itself to do the = user life better :) Moreover, Instead of bind script usage and dpdk running , just run dpdk (do all the j= ob in one place). > > In current hot-plug case the application will need to do a lot of work = to > bind\remap devices in plug-in\plug-out events while the PMD could have al= l > the knowledge to do it. >=20 > At the cost of duplicating a lot of code between PMDs. Why a lot of code? 1 helper in EAL to be used for each relevant PMD. Each PMD just call to the EAL helper with the bind driver type. Really simple and immediate. I think it is better than duplication of user operations. > > One more issue with the script is that the user should do different bin= d per > device, in case of PMD responsibility the user can forget it: > > Think about that, any time the user wants to switch\add new supported n= ic > it should update the script usage and to do per nic operation contrary to= the > DPDK principles. > > >=20 > The udev rules syntax should provide adequate capabilities here for us to > match the correct binding behaviour. No need to have it in DPDK too. We can use it from DPDK. =20 > /Bruce