From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM01-BY2-obe.outbound.protection.outlook.com (mail-by2nam01on0062.outbound.protection.outlook.com [104.47.34.62]) by dpdk.org (Postfix) with ESMTP id 3E12D234 for ; Wed, 10 May 2017 14:18:38 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=gHkTJnraKo2V0IGzjh1AaNQgr66MBhp3Ou0f9F9rym4=; b=faokNAOlogj1SHnjJm6QTovKOFRkixeUa5Ervi83L3GLGAdehyOSxPpA7ud2p1n4NKD4UFq+FWmPQPSF/hZ4KcDlaPdvaYxBJ1+n+QlPG3M4P0GhU7hc3SqA4U62qViugx3GGqKSGqcC/DmhhUG8wiB1ld9P91NBembLjMBVBAo= Authentication-Results: intel.com; dkim=none (message not signed) header.d=none;intel.com; dmarc=none action=none header.from=caviumnetworks.com; Received: from jerin (171.76.125.152) by BY1PR0701MB1722.namprd07.prod.outlook.com (10.162.111.141) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256_P256) id 15.1.1075.11; Wed, 10 May 2017 12:18:33 +0000 Date: Wed, 10 May 2017 17:48:10 +0530 From: Jerin Jacob To: Declan Doherty Cc: Thomas Monjalon , Radu Nicolau , dev@dpdk.org Message-ID: <20170510121809.GA25209@jerin> References: <1494341879-18718-1-git-send-email-radu.nicolau@intel.com> <1571308.M4Vgp8n44D@xps> <18252037.5gudFzihFo@xps> <1c20ee02-530f-a39a-47c5-15cc5ce1a973@intel.com> <20170510110843.GA22712@jerin> <73b94d12-f1cf-295f-284f-0ee1bd5f0d15@intel.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <73b94d12-f1cf-295f-284f-0ee1bd5f0d15@intel.com> User-Agent: Mutt/1.8.2 (2017-04-18) X-Originating-IP: [171.76.125.152] X-ClientProxiedBy: BM1PR01CA0061.INDPRD01.PROD.OUTLOOK.COM (10.163.199.33) To BY1PR0701MB1722.namprd07.prod.outlook.com (10.162.111.141) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 77cdecc4-8694-448d-a0b3-08d4979eae74 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(22001)(201703131423075)(201703031133081); SRVR:BY1PR0701MB1722; X-Microsoft-Exchange-Diagnostics: 1; BY1PR0701MB1722; 3:Cug09mqJ85ZddfWrra/kcXanbkvx9ZkFup6MJ7pgENGNqq+LqCdlbr2qmIJBf82v3yIRS5vmxYN6RbDEZEfJK/sbUwufxMeFsFp3MP3y+NCINng86f+Ca/n+tIilkkBKYY2o7zIHIYrxYErwuZF0K/wunsMn7IIqG0sTVjJVAeJd342yIhfkHxkQfiSJOu2rdpbyhl/fGI12pdIKAY0XQZarZT7Itkl7d/WdZ7ILUssoDncigKxWIm+FHEY8uxBrYRfFLHXfLDG93O1Nwp2WI2pIBGkH4MCXMS9ui4WP7xJ4fPSXWUA9lKTp62uJ+wySrVTrt7sAbxOTL78DnYZnuQ==; 25:SLWop3kMyZXGmwrvD2NqpmBu8vnlJWlav6o2m2rKc6N/nZZfpngygBN9BnH1Mh8tVJlp4ww3BRc2T3ucbsAu/ZXGILS8RAStMdS3YPsUH6f1fBBIoqI8CujL6+ljAEH+ltGCUloJi/dnkAQHWBsf9zCDJkCpP3eWETrwAcSN5hqvbd1a6MMaSYnaYD6uE3Ty7CwGMZlIguxsvahZ7TCYBV9fKLDZCl0usTUwen6SmyCMzQZe9U/nPJxLy/+BjbuaJxh9O4iVsufOxGQrgcnnBPPZL8G/HmmZk86jlibjLEHfmriOA+45i5WpUmCUrrHQTCCGL/yGkwVK3queg82qhgRnjlWXY4VaQFW5M6scjrj+Cr4QaCMeBa3k7LsjOzXaWurVReO0pRgn7jZA375ujKvdD6Ayxfw7hU+miHDvsbIvkqmCf9nq2Afdy0Ke4aF0/0H0a8s1Pq/IGVH5mi03aZTtvI73m+2I36NK1aGHuOs= X-Microsoft-Exchange-Diagnostics: 1; BY1PR0701MB1722; 31:skobj6BEyVekLLz3boq/Uhj7KARzfXVo7WLuIcgi36Io1tN2plhmttbF2nDwiBsLjYR5KzWGF/H9yBEeBD7fF0jWmbvnrmtKHs+4KDpM6jPnVV3ehEMPqPx7CyIYm8N+OXTHXhRrM/6nmZaXoxsPoy2tMA874baDr28Ie/agCBX8UIXVikWL5+90EHe+f1Rlhd0h8FXcOEx4qNAVUlyb0AOKBT3EPGhrI6iMPeEV5DVOkLi8Jj6gS9xAx7xAk4EnFM1efIQhbmTkUj+jTr8Zy7KCQU/7z318rI5BXrZe7kY=; 20:fjGARupZGQluaDeYI8apNsW8gEb0ZHNhifvb0W1u1/Fna7I/fYUfoxBFLjWO7kbvFJW0PWQA5YJEs15JJ48YIWc43Mvg5+orKle7XaHgqPmxSfOk0dAHeboi3OpzRXHC98WCS+2LgRvbpUxGbgw27i1GylMBwC7ZKtWkCp65FJqeOjb1JdHbHIpbFATprgQFmpUXliHr68EjRM+VcJbbJwx3hmsQDDXfnMOC0MWyXgBwKw5B3JQ8LdXpN2dl0Z6GWnIFCdOG9sF+4G6hNDC6LH7m9t83oercNAM9S+mAoZ9neVvLzQD+QHcGXvsLmq7BDsj5ukZjKJkUGwEhtKM53ey2kWcAc93gD/NSkr/S9asvR8NVZ8ES/o+D+e02B9mkaF2mC7mIbZPC9NJJdx567eRU3gfyxjKuQcokZlWY6aNcEDMFiGZy5DTM7mt32g75MEbLYwK6SgnXMq7f2BPMgEXynGHISsE5c4DvBSDSu78L687yXvvMPJgoscLRSh9SXJcKqnjAbKhOR2z6w9j7URQrdW6F8QaEtp+fXdH7tlxx3swyRugjBClU2SfkHnrDYKOKyRXmPvM2bQkyIYVOJOFkHXYT6lc7qxy5nGgxgEM= X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-Test: UriScan:(228905959029699); X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(6040450)(601004)(2401047)(5005006)(8121501046)(93006095)(3002001)(10201501046)(6041248)(20161123562025)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123555025)(20161123564025)(20161123558100)(20161123560025)(6072148); SRVR:BY1PR0701MB1722; BCL:0; PCL:0; RULEID:; SRVR:BY1PR0701MB1722; X-Microsoft-Exchange-Diagnostics: 1; BY1PR0701MB1722; 4:okrFYies6HeDq/vzg4VAZRJiBb2BNKx6q7YjRHpltnmD8IM40eJIqYgawRR1SCIEF4HGHKg3Doh+4Sxxx2QAIc0D/YVAZQgqrpSy5d2CzML2tN87dgEvwB6nZ5WftbOxQRllSP+6od3nwZAlwIcvaC42iiHj8vV8FCh+UY/PySt7cBtPVP7YtwzkZAOM8YTHqbnfN9r4AkZu/T0wXgPHHSdU2Ofmw93JCObQmjKhlciNH8ekZEGRihU512h83x6yNvtK5K4LVNeF58c3cbmrQdFZczJFpIrfkpgykSM3VBbtRQZxzKW8jaUilTuZBpv/gPWbRV/n8nMQoFsYYkv43s9o6HQnegY9bgQbGHohsqaW6qqA2TGW702MecKwR/JUHoqA6GEva4LBVTV/vpBhBNQtmeVAXaqpythwzCAgS2QYn15q8noihjd/yT5SrDP8+qDN+RLYwYfKaZcnyPCqLpyANoi1ZTFTKIO8cOtmFm0UvK3ZgXs9c8mSAmdDONWMt9UDZDDhW6DNpVoM6HBDB9zmV+izV9raTb36vNkOoJVuD+n7zbx01Ob42v28MWuVEpJ5hCKDuY79wnr9TODfkrDpy3uwG4sVMFpC7YKQL/ZkPwefg+NSJ/AOrMSQny9ro2iyiqBHSbB0qnitZOltSKbuI4CbVgGesieHOW4Ohn4qxVAqmVivSsKVBnrsxq/p0DgL/ypUD5WbyZuhGhtrBLGd/fUkyCaKEQT/r1qkUZSxMRtpFvly4b5rJf9fGdSnjKU0umzvzFqRYzwCDBbVXcWDyhPPoLqSwMrBNR1uoEiUMjf6UYcmwxi43PxO+KZG X-Forefront-PRVS: 03030B9493 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(4630300001)(6009001)(39400400002)(39410400002)(39850400002)(39450400003)(39840400002)(24454002)(13464003)(377454003)(4001350100001)(42882006)(2950100002)(6916009)(6666003)(93886004)(5660300001)(23726003)(189998001)(38730400002)(966004)(478600001)(110136004)(54356999)(1076002)(33716001)(76176999)(6246003)(6496005)(50986999)(83506001)(50466002)(9686003)(3846002)(54906002)(6306002)(53546009)(25786009)(55016002)(33656002)(2906002)(6116002)(305945005)(4326008)(229853002)(81166006)(42186005)(72206003)(47776003)(8676002)(66066001)(53936002)(18370500001); DIR:OUT; SFP:1101; SCL:1; SRVR:BY1PR0701MB1722; H:jerin; FPR:; SPF:None; MLV:sfv; LANG:en; X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; BY1PR0701MB1722; 23:Bbk/t9FnPex07bc+F0pNde9ULC/DjH2kBdsZTyw?= =?us-ascii?Q?nIDGBmhIWBCo6jxqc6qhWItAqwgTZ/3rC2LWZyGMBw7tIezdcKqb43eH11zL?= =?us-ascii?Q?O6W2l5UG6HF0WvpNKAsusdxi/1XqWBfzOlnogGF0krRr+yVdtu2VZSj4H/8d?= =?us-ascii?Q?J2CPl07hRvSpMIdDFN5Vy9CAvdhJBPQs9Cl/olvB1I37s0SwBl92JiqKiRoC?= =?us-ascii?Q?3Q2clEt2H+OJBWmokfRwToWq6IrY+DH3jvQ+0KraN25JQny1v3A7/iOfd7up?= =?us-ascii?Q?6PTyyFEiQxpzeBq45RD9qtQOqMxfSHGXtidKtGDpFyakd6G9Xbl6RPZjcZ7p?= =?us-ascii?Q?OxioLevPJ0JV09eZkBFmOVnsrIurrAIqGHwFqjz6dd7S+NiK63vHwA/wWBUI?= =?us-ascii?Q?Uzswce2ZQ7jeE7JP121Ag/2Uv28cqpSVLsVjUER6WcN1uTb531d1IZNJPmkG?= =?us-ascii?Q?1B0TxqqrhWywjztJLHIoHaowYfgg5jpFfbVbYhlqj+hakseuvMsRveZ50G13?= =?us-ascii?Q?3E3WYJaL8h5MXxQrXu4Flc2aFeO1ShzCcojn49wCCPrZzM0awEEI38uvEOyb?= =?us-ascii?Q?teB+movX3Nc1q2DX3HTkl5fU8nUupM6zTU/LOxlno7TnMveODKKl3ENSf9eO?= =?us-ascii?Q?lTNlparnC1bg0E8xu7/B75vkGiQRFnZFg78mw+O+WirgEpMo5JopIHOR/aet?= =?us-ascii?Q?6dKjRF70tFpLPeFf+dyV4a4Zwyglfj//N0qs0yxMfioZ0uokeTSKUXfJ+SxH?= =?us-ascii?Q?KqnlJEe4XU5XiGVWDN8pLzVtC67gDZdawgQFln1nD3UU9ok6f4KUACblw3tb?= =?us-ascii?Q?2mhlWtl1EpEVJylfnfLlGoM7fck+cykK1cRhSjS+ToaE9wdKJy3Z/NB/+kv9?= =?us-ascii?Q?E//BmxdZasam5brjXRr9bmRoRvF19J5sHjN3sj5eRjDs0xcxcgNfvD2O62KA?= =?us-ascii?Q?X5Sx8GC3gg7ZiidyEtgjTNhf9cmoqNO9rXqmcgjaO47R8w1YMFOi1MFRq958?= =?us-ascii?Q?e9Aop1P0afPoxop2eNmPfU4rhnUQSjS9If0gOwtvXguULL2mmhjsTxhWUamH?= =?us-ascii?Q?CptDFA4IL7RgGebhxdiPrBbBpZg1WGHnDp34ZaVeVx0W4hMhiHC2Bazg+By2?= =?us-ascii?Q?nCZ/ZVN48m8bdnINfvqlc0HV7jXz8VuGTsHj5SBzUc80wqAJrFewU5JzbH8p?= =?us-ascii?Q?bxW8aFNwx6FYX7pAZsDOSTiSeCRfCIGc9V+lokyX1Gy4vpjqFIB0FJWZEWTg?= =?us-ascii?Q?aQcW3xup3OVQ1DOM63FaSPJekwnzlR8Re7W1RUSjGw1buqU+DZvFgWcXFzlW?= =?us-ascii?Q?Mb9D2V6kbD8AZPyKKBBnr6LQ=3D?= X-Microsoft-Exchange-Diagnostics: 1; BY1PR0701MB1722; 6:3hVZMB8hp0pLPg4dkZ7V8UTajfEaB8d205FpUOrWRH8+fGyBFGt1UQ+HhxyWdUXi3aszoeZW3vEa33ZYhwSHeNav65NJqqPdfb14FRUtuxcsUIS5H57o+ehN2bkqfnVwN3NkBOdtDPmYR9WzV2iwhol1bgYtO3otaSg6icdFm4lFKC/DO1jKt/j2s0XZ5OiEdc04Lcd4oDGWxheJ3j6OaK+3RcFZyEQsGoiy/Eg4rk4tPNk9YRo6YLn1NKxwPkkRIzpnZvk291Z93KupUS69BTBtq8nhDBbX/KaXUyzwh+nrE1l0ZDP0uIMWReWyqzIRqJp8yvgpTbn9JFowSRRSbEz6g5FOZkCBgmyUFyDHKIQ9kymKZRHyIRe9YTkCMUJyaCGWzx8idkQC0+ZYIY/qwVPi64Xyd4w1++TY99BRidHxiaeqTGIdBDSR21+q7OdsV1cvlR3suXpWAgYlZen7Z01L/h42BwwUMTlO+UUusgsjoDRUVl6hwFuNrH12ponH2z46fNX5+m4YrKogKu/Mew==; 5:Dz2LY6FqAZhS8NtCstrLcyoSLwxlxgl2AOmQC++egwOm6Ox20VTcXpjjTYJQOfMqt4CcNuE4njTPPG8B73wyT+shcFX2ClF/PbXC7hFXhMEiyLjPgTbqtaNg5vlEllRa82UwRLQV/uNo6VXc9UwGqA==; 24:50zCzwAjtScHhFyphzPVoyXTvSYf+4Lb9Qo+oubJPzOpVVB0+XfEAtXv4sBixtnlBmrL8/VkRcFV8pevyc4CF78QC6hsB9ZKXKVHwpF4BSA= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-Microsoft-Exchange-Diagnostics: 1; BY1PR0701MB1722; 7:7eN8/wZN7PgcI9B0i6A1oyMC+AU8vzJRgI++92R+tMyxoZa/PMB8scDcYz0wuTLr2ojQmo+D9VG7ZblEFSDkcAJ0tjprhiDZtwC4VS9RcbBRRgxjPYe34mZzk/YprQw3fZzMARhsPdLj6F8SmHPLO9+oyZjZ+zAYbDXQRdnXBuH/4B1A5avWP368YBtSHe4Da8Bv8yPVim80vIzqrM2Z0+E2CI4iTwtkdqUpiQVVOOCvwHjC1JhwZB4L3toG2MwUgbXFXfb9zRsG6nVRxAtvd1gOuNUayObu/rKpBjnIBy+OgRXqwqzdrjCrRaS29qwQ5RQFYukPpWesBD8kNOomFA== X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 10 May 2017 12:18:33.3962 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-Transport-CrossTenantHeadersStamped: BY1PR0701MB1722 Subject: Re: [dpdk-dev] [RFC][PATCH 2/5] pci: allow shared device instances. X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 10 May 2017 12:18:38 -0000 -----Original Message----- > Date: Wed, 10 May 2017 12:31:48 +0100 > From: Declan Doherty > To: Jerin Jacob > CC: Thomas Monjalon , Radu Nicolau > , dev@dpdk.org > Subject: Re: [dpdk-dev] [RFC][PATCH 2/5] pci: allow shared device instances. > User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 > Thunderbird/45.8.0 > > On 10/05/2017 12:08 PM, Jerin Jacob wrote: > > -----Original Message----- > > > Date: Wed, 10 May 2017 11:52:45 +0100 > > > From: Declan Doherty > > > To: Thomas Monjalon , Radu Nicolau > > > > > > CC: dev@dpdk.org > > > Subject: Re: [dpdk-dev] [RFC][PATCH 2/5] pci: allow shared device instances. > > > User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 > > > Thunderbird/45.8.0 > > > > > > Hey Thomas, I've been working on this with Radu, so see my take below > > > > > > On 10/05/2017 11:28 AM, Thomas Monjalon wrote: > > > > 10/05/2017 12:11, Radu Nicolau: > > > > > Hi > > > > > > > > > > > > > > > On 5/10/2017 10:09 AM, Thomas Monjalon wrote: > > > > > > Hi, > > > > > > > > > > > > 09/05/2017 16:57, Radu Nicolau: > > > > > > > Updated PCI initialization code to allow devices to be shared across multiple PMDs. > > > > > > > > > > > > > > Signed-off-by: Radu Nicolau > > > > > > I was waiting the day when we have a device shared > > > > > > by two different interfaces. > > > > > > Note that some Mellanox and Chelsio devices already instantiate > > > > > > two ethdev ports per PCI device. > > > > > > > > > > > > Please explain your idea behind this "shared" flag. > > > > > > What is your exact need? > > > > > > > > > > Currently for each pci device a look-up into a list of PMDs is > > > > > performed, and when a match is found the system moves to the next > > > > > device. Having this flag will allow a PMD to inform the system that > > > > > there may be more matches, more PMDs that can be used for this > > > > > particular device. > > > > > There is a difference when comparing to the devices you mentioned above, > > > > > in this case the PMDs are totally different types, one network and one > > > > > cryptodev PMD for each IXGBE network card. > > > > > > > > Yes I know it is a lack in DPDK. > > > > Linux introduced MultiFunction Device in 2005: > > > > http://events.linuxfoundation.org/sites/events/files/slides/belloni-mfd-regmap-syscon_0.pdf > > > > > > > > > > So at the most basic level the intention is to allow more than one device of > > > different types, in our case a net PMD and a crypto PMD, to be instantiated > > > on a single PCI bar, in essence to share the bar. I'm not familiar with the > > > approaches taken in the Mellanox and Chelsio devices but I assume they are > > > handled with the driver probe/create functions independently from the EAL > > > infrastructure? > > > > > > For the initial proto-typing of this RFC we only implemented the > > > multi-device creation but I envisage that there will be a requirement for > > > sharing state between drivers, or at a minimum implementing locking around > > > shared resources, registers etc. And I would like to see this done in a > > > generic fashion that can me leverage by any driver and not have each driver > > > having to solve this independently. > > > > Cavium's next generation PCI based NW devices has similar scheme where we > > need to share the same BAR with multiple DPDK subsystems(ethdev, > > eventdev etc) unlike current generation(OcteonTX). > > > > Have you done investigation into how you would like to support this, and are > you trending to any particular approach. The rte_bus approach as you outline > below does sound like it would suit this multi-function device. Not much investigation has been done as its for next generation. It is no PCIe multi function device. There will be a lot shared functions between these shared DPDK devices and there should be place holder for this in code.I thought driver/bus/foo may a option. In additional to this, If we expose new function pointer based interfaces in bus for the shared device register access and other shared resource alloc/free between these two DPDK devices, it can be centralized to one place(driver/bus/foo) and generalized. Just 2c. We haven't done any prototype. > > > I think, Another possible way to handle this in generic way is to: > > Register a new rte_bus for the shared PCI access which sits on top PCIe bus. > > With new bus's scan and probe scheme, it can probe the two devices. > > > > > > Yes, this would work and I think it makes a lot of sense in the case where > you have logically independent hardware functional blocks on a shared bus. > In our particular case, we only have a single physical device, which we are > presenting as 2 logical devices purely to improve the sw model through DPDK > existing infrastructure. We may also need to implement some shared context > for protecting access to shared resources such as register and to > synchronized exposure of capabilities. In the case of the IXBGE family of > devices they can support MACsec or IPsec functionality but not both at the > same time, so some mechanism of passing this state between the net and > crypto PMDs will be required. I guess it should be possible to do this > through the bus model as well but we'll need to have another look, although > my initial feeling is they are slightly different problems.