[dpdk-dev] bnx2x driver and 57800 versus 57810

[dpdk-dev] bnx2x driver and 57800 versus 57810

[Chas Williams]

I have to practically identical systems, same hypervisor on each (Centos
7.x).  In one, I have a 57800 card which works fine with DPDK with
SRIOV.  In the other, I have a 57810 card which doesn't work with SRIOV.

For the 57810 I have tracked this down to the status block in the VF
failing to be updated.  The linux driver works fine but it appears to
use a slightly different scheme -- writing some sort of fastpath status
block generation per interrupt.

Does anyone have any suggestions or a programming guide for this device?

Re: [dpdk-dev] bnx2x driver and 57800 versus 57810

[Harish Patil]

>
>I have to practically identical systems, same hypervisor on each (Centos
>7.x).  In one, I have a 57800 card which works fine with DPDK with
>SRIOV.  In the other, I have a 57810 card which doesn't work with SRIOV.
>
>For the 57810 I have tracked this down to the status block in the VF
>failing to be updated.  The linux driver works fine but it appears to
>use a slightly different scheme -- writing some sort of fastpath status
>block generation per interrupt.
>
>Does anyone have any suggestions or a programming guide for this device?
>
>

What is not working with 57810? Is it link related or traffic? Please
provide the details.
Attached is the SW programming guide for 577xx/578xx. I’m not sure if it
has details pertaining to the specific issue that you have.

Thanks,
Harish


FYI- I had replied to your email earlier with the doc attached but it did
not go thru’ yet due to size restrictions.

Your mail to 'dev' with the subject

    Re: [dpdk-dev] bnx2x driver and 57800 versus 57810

Is being held until the list moderator can review it for approval.

The reason it is being held:

    Message body is too big: 1350322 bytes with a limit of 300 KB


________________________________

This message and any attached documents contain information from the sending company or its parent company(s), subsidiaries, divisions or branch offices that may be confidential. If you are not the intended recipient, you may not read, copy, distribute, or use this information. If you have received this transmission in error, please notify the sender immediately by reply e-mail and then delete this message.

Re: [dpdk-dev] bnx2x driver and 57800 versus 57810

[Chas Williams]

On Wed, 2016-01-27 at 07:32 +0000, Harish Patil wrote:
> >
> >I have to practically identical systems, same hypervisor on each
> (Centos
> >7.x).  In one, I have a 57800 card which works fine with DPDK with
> >SRIOV.  In the other, I have a 57810 card which doesn't work with
> SRIOV.
> >
> >For the 57810 I have tracked this down to the status block in the VF
> >failing to be updated.  The linux driver works fine but it appears to
> >use a slightly different scheme -- writing some sort of fastpath
> status
> >block generation per interrupt.
> >
> >Does anyone have any suggestions or a programming guide for this
> device?
> >
> >
> 
> What is not working with 57810? Is it link related or traffic? Please
> provide the details.
> Attached is the SW programming guide for 577xx/578xx. I’m not sure if
> it has details pertaining to the specific issue that you have.

The DPDK PMD driver seems to be able to transmit packets on the 57810.
But since the status block isn't getting updated, you can't reclaim the
sent buffers.  I modified the driver to use the marker based receive
detection (similar to the method used in the Linux driver) and I can see
packets getting received (certainly broadcast is received -- possibly
not unicast packets though which seems to indicate that part of the
RX path is possibly still broken).

I have tried a couple things.  The status page in the DPDK PMD driver
isn't getting page aligned (as well as a bunch of other structures
that should probably be page aligned). The Linux driver happens to do
this as a side effect of the DMA allocator.  Fixing this didn't seem to
improve matters though.  The status block doesn't seem to get updated.
I verified that the correct DMA address is getting passed to the PF.
And since it works on the 57800, I thought perhaps something changed.

Also, the DPDK driver probably gets the RX/TX queue indices wrong during
initial setup.  The final values coming out of the allocation loop
are probably bigger than they should be.  Should they point to the end
of the queue or just past the end?  Also, the tail of the queue needs
to be corrected for the double entry at the end of the pages.  Again,
fixing this didn't seem to help either.

The VF-PF interaction seems to be ok as well.  Other than not supporting
SGE, the DPDK PMD driver seems to send reasonably correct messages to
the PF.

I don't see the DPDK PMD driver doing anything to 'reset' the PCI apsect
of the VF.  If there is any left over configuration for interrupts,
like leaving the IGU enabled that maybe not be cleared, I am not sure
what the interaction might be.  I do know the Linux driver does seem
to use MSI-X interrupts.

> Thanks,
> Harish

Thanks for looking at this and thanks for the programming guide.  It
will
take me a bit to digest it.

Re: [dpdk-dev] bnx2x driver and 57800 versus 57810

[Chas Williams]

I finally tracked this down.  On guests with MSI-X enabled on the VF
devices, the
bnx2x was determining the vector count incorrectly.  Apparently, the PF
needs this
count set correctly during ACQUIRE to initialize the Interrupt Generation
Units even
though the DPDK driver does polling.

http://www.dpdk.org/dev/patchwork/patch/13459



On Wed, Jan 27, 2016 at 10:58 AM, Chas Williams <3chas3@gmail.com> wrote:

> On Wed, 2016-01-27 at 07:32 +0000, Harish Patil wrote:
> > >
> > >I have to practically identical systems, same hypervisor on each
> > (Centos
> > >7.x).  In one, I have a 57800 card which works fine with DPDK with
> > >SRIOV.  In the other, I have a 57810 card which doesn't work with
> > SRIOV.
> > >
> > >For the 57810 I have tracked this down to the status block in the VF
> > >failing to be updated.  The linux driver works fine but it appears to
> > >use a slightly different scheme -- writing some sort of fastpath
> > status
> > >block generation per interrupt.
> > >
> > >Does anyone have any suggestions or a programming guide for this
> > device?
> > >
> > >
> >
> > What is not working with 57810? Is it link related or traffic? Please
> > provide the details.
> > Attached is the SW programming guide for 577xx/578xx. I’m not sure if
> > it has details pertaining to the specific issue that you have.
>
> The DPDK PMD driver seems to be able to transmit packets on the 57810.
> But since the status block isn't getting updated, you can't reclaim the
> sent buffers.  I modified the driver to use the marker based receive
> detection (similar to the method used in the Linux driver) and I can see
> packets getting received (certainly broadcast is received -- possibly
> not unicast packets though which seems to indicate that part of the
> RX path is possibly still broken).
>
> I have tried a couple things.  The status page in the DPDK PMD driver
> isn't getting page aligned (as well as a bunch of other structures
> that should probably be page aligned). The Linux driver happens to do
> this as a side effect of the DMA allocator.  Fixing this didn't seem to
> improve matters though.  The status block doesn't seem to get updated.
> I verified that the correct DMA address is getting passed to the PF.
> And since it works on the 57800, I thought perhaps something changed.
>
> Also, the DPDK driver probably gets the RX/TX queue indices wrong during
> initial setup.  The final values coming out of the allocation loop
> are probably bigger than they should be.  Should they point to the end
> of the queue or just past the end?  Also, the tail of the queue needs
> to be corrected for the double entry at the end of the pages.  Again,
> fixing this didn't seem to help either.
>
> The VF-PF interaction seems to be ok as well.  Other than not supporting
> SGE, the DPDK PMD driver seems to send reasonably correct messages to
> the PF.
>
> I don't see the DPDK PMD driver doing anything to 'reset' the PCI apsect
> of the VF.  If there is any left over configuration for interrupts,
> like leaving the IGU enabled that maybe not be cleared, I am not sure
> what the interaction might be.  I do know the Linux driver does seem
> to use MSI-X interrupts.
>
> > Thanks,
> > Harish
>
> Thanks for looking at this and thanks for the programming guide.  It
> will
> take me a bit to digest it.
>