From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from NAM01-BY2-obe.outbound.protection.outlook.com (mail-by2nam01on0067.outbound.protection.outlook.com [104.47.34.67]) by dpdk.org (Postfix) with ESMTP id D0BE47CCD for ; Thu, 14 Sep 2017 09:15:39 +0200 (CEST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=CAVIUMNETWORKS.onmicrosoft.com; s=selector1-cavium-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version; bh=4uI1nFY0bc0nmgFqMUBPTAJ2kPRDbDQo7wQS9js3HGU=; b=HdINzZNhtB0S1PO6FEV0Bq2suzoFSUdtyMDKPkarTvo8eaV/MOTkbCIQ3IuNWgTMFYPrhYREb80wJYBnTFjVUhBz2xgCsByRnoX+RiOjBPhWbvYQCAd8A/T2XsNtNBF5j2HcjKi8nqpyrtPyHYWLsotle/XU0RPO6hp/PAhspi4= Authentication-Results: spf=none (sender IP is ) smtp.mailfrom=Shijith.Thotton@cavium.com; Received: from localhost.localdomain (14.140.2.178) by CY4PR07MB3062.namprd07.prod.outlook.com (10.172.116.143) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384_P256) id 15.20.35.12; Thu, 14 Sep 2017 07:15:34 +0000 Date: Thu, 14 Sep 2017 12:45:19 +0530 From: Shijith Thotton To: "Yang, Qiming" Cc: "dev@dpdk.org" , "Tan, Jianfeng" , "Hu, Xuekun" , Gregory Etelson , "Yigit, Ferruh" , "Patil, Harish" Message-ID: <20170914071518.GA24657@localhost.localdomain> References: <20170913104840.GA14229@localhost.localdomain> <95faf7af-367a-ff42-ed16-b8bf3fdea466@intel.com> <88A92D351643BA4CB23E303155170626630E6054@SHSMSX103.ccr.corp.intel.com> <50bb4358-cd0c-003f-1ae7-a953ef569807@intel.com> <20170914070053.GA23787@localhost.localdomain> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20170914070053.GA23787@localhost.localdomain> User-Agent: Mutt/1.5.21 (2010-09-15) X-Originating-IP: [14.140.2.178] X-ClientProxiedBy: PN1PR01CA0090.INDPRD01.PROD.OUTLOOK.COM (10.174.144.158) To CY4PR07MB3062.namprd07.prod.outlook.com (10.172.116.143) X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 6b8f820b-9604-49f8-b188-08d4fb406653 X-Microsoft-Antispam: UriScan:; BCL:0; PCL:0; RULEID:(300000500095)(300135000095)(300000501095)(300135300095)(22001)(300000502095)(300135100095)(2017030254152)(300000503095)(300135400095)(2017052603199)(201703131423075)(201703031133081)(201702281549075)(300000504095)(300135200095)(300000505095)(300135600095)(300000506095)(300135500095); SRVR:CY4PR07MB3062; X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3062; 3:sQQ4rouRH1RvSZfhj+mZfmyJJMBY17NpxAarWOT9QsDqN/fQBdax7iyIRXXG2nIabeo1JJfrdp3kjOeGXdNWBXZyGi/DoyntoJxIbUxwfLdlmU+2M5h9caNJ3gIWk6JrpbGmvRbDoKXK6xMiYnRcX4+h0p0lQOVT+HOC+C+Tpw3cwoEb/9iX8lNjifa9abQ0nAAecgJV4JGC+V+vZLoa6zjiZDXPhxJdmffOKLRW0+CG8lF5ab+bbDUVU3icHrVp; 25:ZW/QpdMDFdbbHwJ36ORQ67Th6wpTiPVMWbaIj0eKhUUFWuO2XTSuvMOk3xeDp4dtd2Tir0doO3F0koRN/gRs3XpSX23azYZgP3AaTq2ZJeHnWAh13LPJHEmCzM/f2RsuWOhz5TvPyULiv0va9KVilDiBNRETOl1mqiYgU2DQoq8mXkLCjjx9S1IYRZGP4rWJN0RaVuXtzN6zNhit+MQFeVXvMSOnwL8jJmJx8fUwNtVPAPe9zUIdmAo5lJKRRQ4zxje1xFLhoZtxyR6n/y8ox2h42pu/f4EVH5rj63/xwaQacPMixHtHBFRXO0LcK5vU/vEv6s4U4eKdeGREimcvWg==; 31:n/NpXuue1PimxoqBgOoyVoApWaJdiu9leHxJRItZNA8jAvG4eBHeLjsh3ZgwSActxHl3AHFt94iaOUinUmOXqsYTJHLqrUXOSheH50tzLxiIJ/43v1n7q6UwCEVsRcEREo99Qzan5oaDlt+i5++l9+G503s0jqZY5n69IrYix7pBwgWf6uPesMNSPFhNOxg7gf7fP39WnBDoE4AUTIYr6PMpRqtBYJ07KESn1clI4gg= X-MS-TrafficTypeDiagnostic: CY4PR07MB3062: X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3062; 20:Q1z5zMALzrMCagr3F/s0rmcJgQTeG5NZTuPUQJKCcGXyJ3FtrxCqsBMwPUyJZ5jkljDvQpkojtT7H+ab7KTljtiuK0Ecaq0XUJqvE2gn0oHAYtjlkt7qqa/pklCLi+61HTB3RYrteBsNeDsmVPHm0xnkhgkWJxIGYn1VlKjMhnjLyyOvUppDlTbqPeKswXZXYL9KeZ2+KQR7EHTO1oTTvwWYq3FglrR4NFf7D24j24UVxl28dwXQHP/M8CBLnQbs5lDQMBtIfpa09Bl/3ZRAWVGACEAgLU2goHsK2hXnXq16eRocwYxb7HCBjlz31RNF6Ay0Wj7l/PpJ4TJF0aGIlrhxux9AeNk/SgrV8JGvqsQ91RE4vwpE9s4Ov7/kTu8MdsUVIkHQ0OZ6x8BtbfZ/qK/5ls3xkgfoDZaOfkgWTBVHjPdQXa7LsZAUbmlnuko5HFiiU4n8jqsz6w6Ye44wQhPkEfF8nQAAXcI7sio+3yVQ7/t6VmJccVJlwp+2+rY1O4IaGMMo4YE5ZwIP6gjafS/i4tkFBRMwaF67HShIvBo+TIgCtjhYFqU5dItUJaadvyuRaEiKrtpnH48+nABYL/yKZkHbYSUACrSAspn1C4E= X-Exchange-Antispam-Report-Test: UriScan:(265634631926514)(228905959029699)(17755550239193); X-Microsoft-Antispam-PRVS: X-Exchange-Antispam-Report-CFA-Test: BCL:0; PCL:0; RULEID:(100000700101)(100105000095)(100000701101)(100105300095)(100000702101)(100105100095)(6040450)(2401047)(5005006)(8121501046)(10201501046)(100000703101)(100105400095)(93006095)(3002001)(6041248)(201703131423075)(201702281528075)(201703061421075)(201703061406153)(20161123555025)(20161123558100)(20161123564025)(20161123562025)(20161123560025)(6072148)(201708071742011)(100000704101)(100105200095)(100000705101)(100105500095); SRVR:CY4PR07MB3062; BCL:0; PCL:0; RULEID:(100000800101)(100110000095)(100000801101)(100110300095)(100000802101)(100110100095)(100000803101)(100110400095)(100000804101)(100110200095)(100000805101)(100110500095); SRVR:CY4PR07MB3062; X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3062; 4:4DcSgDRbLZ7qcC9K55FhFkXoOjejg0EdkHttrC31jQOqZcI2rh8i5wgWvZkeRCDt3zzEUjGuvTNUVF8rCltBFq9Z4/1rwtEqRbf1XjYV/syIkfQ5YZVsTXUQLmH0v779lVl7uWjxvK5VM4M3ah+WHc16wT9kA5KPZb02ZYbO79UVewytPg6S8cEJbxV4ZL8YtX5VhSeSx4xPk45x1u3uPGsYtaGt4UU63j5H+/fKu4DnIbJLU0J+cNTuK26n3mD0QrGhG2gxlI6MEIeswk2JjiEnxaK4ZHR7MNB08R/T5LFwOAUzn5YynY12AFBW5qjntmACa0ce0hS+XaQVDVj6rVLg9wqNjcLeb0B8ryUK/+gPT0B6miviZLX81KNKmif2 X-Forefront-PRVS: 0430FA5CB7 X-Forefront-Antispam-Report: SFV:NSPM; SFS:(10009020)(6069001)(6009001)(346002)(376002)(43544003)(13464003)(189002)(199003)(24454002)(377454003)(83506001)(50986999)(53546010)(76176999)(7736002)(93886005)(305945005)(478600001)(72206003)(81166006)(97736004)(16526017)(4326008)(8676002)(81156014)(54356999)(4001350100001)(1076002)(68736007)(25786009)(8936002)(50466002)(101416001)(316002)(66066001)(54906002)(5009440100003)(9686003)(110136004)(107886003)(53936002)(6116002)(5660300001)(33656002)(106356001)(105586002)(3846002)(23726003)(6506006)(42882006)(61506002)(229853002)(6916009)(2950100002)(2906002)(6246003)(55016002)(6666003)(16586007)(189998001)(47776003)(110426004)(18370500001); DIR:OUT; SFP:1101; SCL:1; SRVR:CY4PR07MB3062; H:localhost.localdomain; FPR:; SPF:None; PTR:InfoNoRecords; MX:1; A:1; LANG:en; Received-SPF: None (protection.outlook.com: cavium.com does not designate permitted sender hosts) X-Microsoft-Exchange-Diagnostics: =?us-ascii?Q?1; CY4PR07MB3062; 23:cB+MBRwuGPUPQgB3Xb8lePet8dEdNOQQVFmBjMNdt?= =?us-ascii?Q?xD01S1eZLeHuneMyrU/BGKCcubQX4NaDe7yULBiITdFWM1DQKEhJiQW4cpwa?= =?us-ascii?Q?ajmbKCbcjCZHlYciuK/QUkUO8AabjAm1sqmxhp7N3gQaRFcjz6NuZhApNDg+?= =?us-ascii?Q?7dP78cQNNv91OhlPJEl50l9qy5G03ao9KatOPMJrAvmfMkzfsbu6pjNRfkv6?= =?us-ascii?Q?fxQZkhIRKFXUj4Lu6EqVbOoWUCxDeyMOlgvzS2SYCWdeIpNeM/MtPBI2gkUi?= =?us-ascii?Q?jJsexpOnUWi3MMgSsh0EAN1dqmE8/aYnJZ09zD/xU4G2gJDyY/BYJI5Ygdl2?= =?us-ascii?Q?kYHJR3ggjfATyaBDWiBZEpvud1lMYg6VBacH6mBiUCsfjR0A6ZvOK20/km7p?= =?us-ascii?Q?gLd9kWOMJzpq75F2QpDqGYX+PU6kni+W+p4NUrj/Qgd79LHefML6l8NGy18z?= =?us-ascii?Q?s72YMAQ65VjClZemwjpbrbbdsWuT+3DK52wrSfbrRgGLGt5mh9GMN9Ewx3aD?= =?us-ascii?Q?xwW4N8d2GTQeK91VU6qRq1Nl02c+3Hihb2ZmHL+DO5sb8Gq3uxX8Y89iR0A3?= =?us-ascii?Q?dGcUB7h5epGqnKI88gnCup8GLM0J/B57EG/6tnpGMG31rOQ9xfvQe6kYBedq?= =?us-ascii?Q?FbTLk6qesati/7x4CoZnoG6W5c8qmTAj2jhhWHdHR7Z9RZwxqnArvN0Eu78D?= =?us-ascii?Q?yv6MQkxVaOqDT/UXh0D+fWN5GiecAId9NWa5tV5Lrd0X8RvQqCYY89B0qLt4?= =?us-ascii?Q?zy8a/LZjk115uXp4aPNoG+c5PF6iI/mcnjgSotkSKbjN9xyN9USRHYzPsyiV?= =?us-ascii?Q?fudulWEKupcGmkWf2o4KMEAbaaFsBbaab3jGusy6IRGwer1W2Mpw3JZ3hHEj?= =?us-ascii?Q?eLWvj5alMG8PZhz7XYqVC4XZejwrXiTYf/2dYjQxpvi/XQXIQvYD8OZ6v+vn?= =?us-ascii?Q?K/6ENfgnEhYQoJIQUZx2v5dQuST7SmcD2jZzk7roJqNsk/1KLjzsjBvARipk?= =?us-ascii?Q?Gn5Cksr97njI8datSkStdtLmMQfNhGiiyA5k4Rg9jJu/j//rLVskrsaewfI3?= =?us-ascii?Q?GnvLeY9Db8OCZ1DoCLmXaB6LZuYCa044w5OrsF1zC/WQ4D8R9tnF5WiCNGk3?= =?us-ascii?Q?q6pOVBuDQl7YIU5QUZbunuBAVFRn9b6a63AYvRWSMMYw+oK3cKwslFtjXVXt?= =?us-ascii?Q?NB+MTUkRkxMmS0UKLT5OrXj/lDixfJNdQWmM+pZSz2umJyh6gzLeHd5ihIRP?= =?us-ascii?Q?Va8lcOxr9kIfKNteKdZg4od2YHAYtlfRCTL5Kg2owMDyYgfVVgulbf9STK/0?= =?us-ascii?Q?OZkHlYVUtuX+Naimvr5UmUWWn36ooEdxg6axZBcuOxUhTbaOTZE3DvD2eA6m?= =?us-ascii?Q?cQSbGbqsU9p0548FmNd4q5j492OmpQO/2JsL+Ic8IG1Zr3uFcVCpVt9Zx6hF?= =?us-ascii?Q?WaWOrUwiGLppF4GmiTV++Mx0PN1AosKU7tNnV19C0dWkZWdJEsm?= X-Microsoft-Exchange-Diagnostics: 1; CY4PR07MB3062; 6:/p/L6sQ11EJxry4uCdQ2aTq3rI1nTEI/uLyTWbmmMAAibAv1V4KAHjritVKyWxRZU1ybGrw5BGinr6Dgpf3oafPRogE4xMDvDYAmrbw6uwWCyWxAuwrsOf9itFK8M6lZBqSFLLjsoMRUFgERerBHuday+jVbRpKfqz2fi99urUfcsZ3waTnEqv0TibdNt41kWx7ohyjQ/IRjqmJIWTnPn6zbL4MVABTUs1ZWeYWZ0px0EYmIQXDgLaV1OFMlfDDS/RmYQoQxC/iq177EwkmTtLaFw+MXFhEzATbK0GEEu6bvGvUU0tX7oQHQMFn0kkzdXFm5hASphIKJKp1vltbKsA==; 5:lfXJi24s72rmANRWk5+OLgN4arCwpiQsiGYU0vXoYGOawaQ6QsgTXkp6QMdBJIUbpHjcxJDs2x5ptIiT76oh0zUdPCnkvoG36urptcFOEKjykK3U5pZEgUUp9zXH5QwaQkIEYdt9dJk+oiQKfM19+g==; 24:5ckCJpgsvd6uVcxOyEv16ohPuYRASouHKvKRa7e2hMX7ElAxlzi8TwuIAIs4eoEzZe2f5CG6X2n2iI+fW3LBkJVnHwNze0ZLXiBWtvkf4TQ=; 7:UmYwgAX5Kwh1OVa3DXLWnWiTLGnGKJeRUTrA1aeTEbC9nfddgeNzIW2HqR4mZcvZ2l4AyBLCFByzYBjYFhRSNzaO9+hMUXsHDmy5zFkslc2q2yhtGqkVvzD0J6xei9+mFDNLHVtfFJijpJv/Q1CSgUzz4Gj0WHZii/L/GFuxUMlDGArrVJBkGHsKkm5mxXsC6vzrzF965tmMKI8FaU4i/GZBnkYM5JjYczF83l1UozY= SpamDiagnosticOutput: 1:99 SpamDiagnosticMetadata: NSPM X-OriginatorOrg: caviumnetworks.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 14 Sep 2017 07:15:34.9321 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 711e4ccf-2e9b-4bcf-a551-4094005b6194 X-MS-Exchange-Transport-CrossTenantHeadersStamped: CY4PR07MB3062 Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open and release of device file X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Thu, 14 Sep 2017 07:15:40 -0000 On Thu, Sep 14, 2017 at 12:30:55PM +0530, Shijith Thotton wrote: > On Thu, Sep 14, 2017 at 03:16:05AM +0000, Yang, Qiming wrote: > > > -----Original Message----- > > > From: Yigit, Ferruh > > > Sent: Thursday, September 14, 2017 1:06 AM > > > To: Hu, Xuekun ; Shijith Thotton > > > ; Yang, Qiming > > > ; Gregory Etelson > > > Cc: dev@dpdk.org; Tan, Jianfeng > > > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR during open > > > and release of device file > > > > > > On 9/13/2017 3:25 PM, Hu, Xuekun wrote: > > > > I met the same issue too, only with i40e 2.1.26 PF kernel driver. > > > > > > > > -----Original Message----- > > > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Ferruh Yigit > > > > Sent: Wednesday, September 13, 2017 7:04 PM > > > > To: Shijith Thotton ; Yang, Qiming > > > > ; Gregory Etelson > > > > Cc: dev@dpdk.org; Tan, Jianfeng > > > > Subject: Re: [dpdk-dev] vf init issue with patch igb_uio: issue FLR > > > > during open and release of device file > > > > > > > > On 9/13/2017 11:48 AM, Shijith Thotton wrote: > > > >> On Wed, Sep 13, 2017 at 07:51:30AM +0000, Yang, Qiming wrote: > > > >>> Hi, Shijith > > > >>> > > > >>> > > > >>> > > > >>> VF init error will happen after apply your patch, error log as below. If > > > >>> revert your commit, all things work well. And this issue is not only occur > > > >>> in i40 VF but also ixgbe. > > > > > > Hi Qiming, > > > > > > I can reproduce the issue for the case: > > > > > > 1- i40e 2.1.26 PF kernel driver (NOT reproduced with 1.6.27-k) > > > 2- DPDK app run on host, so both PF and VF are in host. Linux driver for PF and > > > DPDK for VF. (When VF used in a VM, Linux PF on host and DPDK VF on guest, > > > this works fine) > > > > > > And I confirm the function igbuio_pci_open() added with below patch is causing > > > it. > > > > > Hi, Shijith > > Could you tell why this adding is necessary? And confirm why this function will cause the vf init issue. As Harish said, it seems not only reproduced in intel NICs. > > Reset was added in open to make the device ready for init. I'm not sure about > the exact cause of this issue and was unable to reproduce the same with LiquidIO > VFs. > > Reset happens during open and if VFs are not ready by the time init happens in > PMD, it can fail. Issue could be in firmware as we don't see it for all versions. > If there are no other way to fix this, I think, it is best to remove reset > during open as it breaks PMD from multiple vendors. > > Please try removing igbuio_pci_open. Sorry. Please try removing pci_reset_function from igbuio_pci_open. Thanks, Shijith > > > igbuio_pci_open() does: > > > pci_reset_function(dev); > > > pci_set_master(dev); > > > > > > Don't know yet root cause of the error, will dig more. > > > > > > >>> > > > >>> Could you help to check it soon? > > > >>> > > > >>> > > > >>> > > > >>> [root@localhost app]# ./testpmd -c 7 -n 4 -- -i > > > >>> > > > >>> EAL: Detected 10 lcore(s) > > > >>> > > > >>> EAL: No free hugepages reported in hugepages-1048576kB > > > >>> > > > >>> EAL: Probing VFIO support... > > > >>> > > > >>> EAL: WARNING: cpu flags constant_tsc=yes nonstop_tsc=no -> using > > > >>> unreliable clock cycles ! > > > >>> > > > >>> EAL: PCI device 0000:00:03.0 on NUMA socket -1 > > > >>> > > > >>> EAL: Invalid NUMA socket, default to 0 > > > >>> > > > >>> EAL: probe driver: 8086:154c net_i40e_vf > > > >>> > > > >>> i40evf_init_vf(): init_adminq failed: -53 > > > >>> > > > >>> i40evf_dev_init(): Init vf failed > > > >>> > > > >>> EAL: Requested device 0000:00:03.0 cannot be used > > > >>> > > > >>> > > > >>> > > > >>> commit b58eedfc7dd57eef6d12e2c654a52c834f36084a > > > >>> Author: Shijith Thotton > > > >>> Date: Fri Jul 7 16:43:51 2017 +0530 > > > >>> > > > >>> igb_uio: issue FLR during open and release of device file > > > >>> > > > >>> Set UIO info device file operations open and release. Call pci reset > > > >>> function inside open and release to clear device state at start and end. > > > >>> Copied this behaviour from vfio_pci kernel module code. With this patch, > > > >>> it is not mandatory to issue FLR by PMD's during init and close. > > > >>> > > > >>> Bus master enable and disable are added in open and release respectively > > > >>> to take care of device DMA. > > > >>> > > > >>> Signed-off-by: Shijith Thotton > > > >>> Reviewed-by: Jianfeng Tan > > > >>> Acked-by: Ferruh Yigit > > > >>> Acked-by: Gregory Etelson > > > >>> > > > >>> > > > >>> > > > >>> Best Regard, > > > >>> > > > >>> Yang Qiming > > > >>> > > > >>> > > > >> > > > >> Hi Yang Qiming, > > > >> > > > >> We didn't face this issue while preparing patch. I think, Gregory > > > >> tested it for Intel NICs. Adding more people to the loop for help. > > > >> > > > > > > > > I will try to reproduce the issue. > > > > > > > >> Shijith > > > >> > > > > > >