From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: <3chas3@gmail.com> Received: from mail-qg0-f44.google.com (mail-qg0-f44.google.com [209.85.192.44]) by dpdk.org (Postfix) with ESMTP id 59DF695DA for ; Wed, 27 Jan 2016 16:58:38 +0100 (CET) Received: by mail-qg0-f44.google.com with SMTP id 6so10001999qgy.1 for ; Wed, 27 Jan 2016 07:58:38 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=message-id:subject:from:to:date:in-reply-to:references:content-type :mime-version:content-transfer-encoding; bh=MdKJUHgZIEbIJ4+u+nhQkcf4HPHGXkqYXGa8lwWTWwA=; b=esa8toxvb6ZtU4hXaCl2sEDHzpwRWuxmdDRQ3TBSw7whsSoPAWPxWqioR/v9gAy1H7 fkNxeY2hrstRs0UYl2Z+SU0TTKVyEtUj1Jl8pNBqVr9Bkx8nr1fidMeBssYa+vWzanLs t7bMlIp8uItHs4lfl7nWWunDbJC+/+WPS2r6mopSv5QPSYslJ+fMh1DsbD30BJ6rQBp6 eeD4lIg0YiGaREi62OXRZgcP4AfHnERthSDBKjSJfjmVAO+f8aNMuz4Uc3yjYBwI5i8A 41pwXGGJBnF4y6XOkGDMpbUaJ4WmzMr6nxy3MHdbrX+BBNfSrKM4FyuitbbJOCIrn+E/ jc0g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:message-id:subject:from:to:date:in-reply-to :references:content-type:mime-version:content-transfer-encoding; bh=MdKJUHgZIEbIJ4+u+nhQkcf4HPHGXkqYXGa8lwWTWwA=; b=OE7QfsOLb6AeioIS5nCWD5qWFwxz3nd7G/DYNfP50+nzh01chaibSzrqXs613yz4zT 01e98VIrQHgVCCdDQXDfx1M+lZ9LCoCFWMLizm8zGMB1YAv/ToujZqdKFzhQFW8vzzKT HbH+0ZPrMhMj48wQxsOwbt/3Esbcxp8a/grPkeOKWYYnEFoHL0Z6jFwVD/aBTbefPxsB uwSeVHeq28ENy4jGlrtvBk2MzUBlWVragjhCJIp4h2/zECX8Si4PWMBtU8QLD/1RiAsP vIt/TP5OZnVcwnc9tj17LXgsM0kvXb3sSZV7bWWgvEpcm6GYfT0jVga7Cwj3ZtWCEOcE wAPg== X-Gm-Message-State: AG10YOSy4yASSxonMvJMEooTWXCJ/Zcn1712Gv02+vsAUOrxqUduK7TrtF4AkY7DdKjQhQ== X-Received: by 10.140.29.74 with SMTP id a68mr36378434qga.24.1453910317876; Wed, 27 Jan 2016 07:58:37 -0800 (PST) Received: from monolith.home (pool-173-66-34-64.washdc.fios.verizon.net. [173.66.34.64]) by smtp.googlemail.com with ESMTPSA id w75sm1878958qka.49.2016.01.27.07.58.36 (version=TLSv1/SSLv3 cipher=OTHER); Wed, 27 Jan 2016 07:58:37 -0800 (PST) Message-ID: <1453910316.30152.1.camel@gmail.com> From: Chas Williams <3chas3@gmail.com> To: Harish Patil , "dev@dpdk.org" Date: Wed, 27 Jan 2016 10:58:36 -0500 In-Reply-To: References: <1453853598.28524.23.camel@gmail.com> Content-Type: text/plain; charset="UTF-8" X-Mailer: Evolution 3.18.3 (3.18.3-1.fc23) Mime-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: Re: [dpdk-dev] bnx2x driver and 57800 versus 57810 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Wed, 27 Jan 2016 15:58:38 -0000 On Wed, 2016-01-27 at 07:32 +0000, Harish Patil wrote: > > > >I have to practically identical systems, same hypervisor on each > (Centos > >7.x).  In one, I have a 57800 card which works fine with DPDK with > >SRIOV.  In the other, I have a 57810 card which doesn't work with > SRIOV. > > > >For the 57810 I have tracked this down to the status block in the VF > >failing to be updated.  The linux driver works fine but it appears to > >use a slightly different scheme -- writing some sort of fastpath > status > >block generation per interrupt. > > > >Does anyone have any suggestions or a programming guide for this > device? > > > > >  > What is not working with 57810? Is it link related or traffic? Please > provide the details. > Attached is the SW programming guide for 577xx/578xx. I’m not sure if > it has details pertaining to the specific issue that you have. The DPDK PMD driver seems to be able to transmit packets on the 57810. But since the status block isn't getting updated, you can't reclaim the sent buffers.  I modified the driver to use the marker based receive detection (similar to the method used in the Linux driver) and I can see packets getting received (certainly broadcast is received -- possibly not unicast packets though which seems to indicate that part of the RX path is possibly still broken). I have tried a couple things.  The status page in the DPDK PMD driver isn't getting page aligned (as well as a bunch of other structures that should probably be page aligned). The Linux driver happens to do this as a side effect of the DMA allocator.  Fixing this didn't seem to improve matters though.  The status block doesn't seem to get updated. I verified that the correct DMA address is getting passed to the PF. And since it works on the 57800, I thought perhaps something changed. Also, the DPDK driver probably gets the RX/TX queue indices wrong during initial setup.  The final values coming out of the allocation loop are probably bigger than they should be.  Should they point to the end of the queue or just past the end?  Also, the tail of the queue needs to be corrected for the double entry at the end of the pages.  Again, fixing this didn't seem to help either. The VF-PF interaction seems to be ok as well.  Other than not supporting SGE, the DPDK PMD driver seems to send reasonably correct messages to the PF. I don't see the DPDK PMD driver doing anything to 'reset' the PCI apsect of the VF.  If there is any left over configuration for interrupts, like leaving the IGU enabled that maybe not be cleared, I am not sure what the interaction might be.  I do know the Linux driver does seem to use MSI-X interrupts. > Thanks, > Harish Thanks for looking at this and thanks for the programming guide.  It will take me a bit to digest it.