[DPDK/bnxt] Issues starting bnxt on an Arm server

[DPDK/bnxt] Issues starting bnxt on an Arm server

[Thierry Herbelot]

Hello,

I would like to start testpmd with bnxt ports, when running on an Arm 
server.

This is the log at startup (with testpmd from DPDK v20.11.4):

dpdk-devbind.py -b vfio-pci 03:00.0
dpdk-devbind.py -b vfio-pci 03:00.1

dpdk-testpmd -l 1,3,5 --main-lcore 1
EAL: Detected 48 lcore(s)
EAL: Detected 2 NUMA nodes
EAL: Detected static linkage of DPDK
EAL: Multi-process socket /var/run/dpdk/rte/mp_socket
EAL: Selected IOVA mode 'PA'
EAL: No available hugepages reported in hugepages-16777216kB
EAL: No available hugepages reported in hugepages-2048kB
EAL: Probing VFIO support...
EAL: VFIO support initialized
EAL:   using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_bnxt (14e4:1750) device: 0000:03:00.0 (socket 0)
bnxt_hwrm_send_message(): Error(timeout) sending msg 0x0000, seq_id 0
bnxt_hwrm_ver_get(): failed rc:-110
ethdev initialisation failed
EAL: Releasing pci mapped resource for 0000:03:00.0
EAL: Calling pci_unmap_resource for 0000:03:00.0 at 0x4180000000
EAL: Calling pci_unmap_resource for 0000:03:00.0 at 0x4180010000
EAL: Requested device 0000:03:00.0 cannot be used
EAL:   using IOMMU type 8 (No-IOMMU)
EAL: Probe PCI driver: net_bnxt (14e4:1750) device: 0000:03:00.1 (socket 0)
bnxt_hwrm_send_message(): Error(timeout) sending msg 0x0000, seq_id 0
bnxt_hwrm_ver_get(): failed rc:-110
ethdev initialisation failed
EAL: Releasing pci mapped resource for 0000:03:00.1
EAL: Calling pci_unmap_resource for 0000:03:00.1 at 0x4180110000
EAL: Calling pci_unmap_resource for 0000:03:00.1 at 0x4180120000
EAL: Requested device 0000:03:00.1 cannot be used

The server is running Redhat-8 with the following kernel:
4.18.0-348.7.1.el8_5.aarch64 #1 SMP Wed Dec 8 21:35:15 EST 2021

As the IOMMU on this server is not working, the kernel is started with 
the following option:
vfio.enable_unsafe_noiommu_mode=1

A similar testpmd log is obtained with DPDK v21.11.0.


Is this a know issue ?

	Thanks

	Thierry Herbelot

PS: a similar error is seen for the Ethernet ports, when handled by the 
kernel:
dmesg|grep bnxt
[   57.823746] bnxt_en 0000:03:00.0 (unnamed net_device) 
(uninitialized): Error (timeout: 500015) msg {0x0 0x0} len:0
[   59.202255] bnxt_en: probe of 0000:03:00.0 failed with error -16
[   60.641519] bnxt_en 0000:03:00.1 (unnamed net_device) 
(uninitialized): Error (timeout: 500015) msg {0x0 0x0} len:0
[   61.502237] bnxt_en: probe of 0000:03:00.1 failed with error -16


-- 
Thierry Herbelot
Senior Software Engineer
Tel: +33 1 39 30 92 61
http://www.6wind.com/

Follow us:
https://www.linkedin.com/company/6wind/
https://twitter.com/6WINDsoftware
https://www.youtube.com/user/6windsoftware

Re: [DPDK/bnxt] Issues starting bnxt on an Arm server

[Ajit Khaparde]

[-- Attachment #1: Type: text/plain, Size: 953 bytes --]

>
> Is this a know issue ?
No.
We have tried the Broadcom PMD on ARM server(s) and have been
able to use it without any issues.
But we can try it again and get back to you.


>
>         Thanks
>
>         Thierry Herbelot
>
> PS: a similar error is seen for the Ethernet ports, when handled by the
> kernel:
> dmesg|grep bnxt
> [   57.823746] bnxt_en 0000:03:00.0 (unnamed net_device)
> (uninitialized): Error (timeout: 500015) msg {0x0 0x0} len:0
> [   59.202255] bnxt_en: probe of 0000:03:00.0 failed with error -16
> [   60.641519] bnxt_en 0000:03:00.1 (unnamed net_device)
> (uninitialized): Error (timeout: 500015) msg {0x0 0x0} len:0
> [   61.502237] bnxt_en: probe of 0000:03:00.1 failed with error -16
>
>
> --
> Thierry Herbelot
> Senior Software Engineer
> Tel: +33 1 39 30 92 61
> http://www.6wind.com/
>
> Follow us:
> https://www.linkedin.com/company/6wind/
> https://twitter.com/6WINDsoftware
> https://www.youtube.com/user/6windsoftware

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4218 bytes --]

Re: [DPDK/bnxt] Issues starting bnxt on an Arm server

[Thierry Herbelot]

Hello Ajit

Thanks for the quick response.

Please keep me posted when you check on your side.

	Best regards

	Thierry

On 3/2/22 16:08, Ajit Khaparde wrote:
>>
>> Is this a know issue ?
> No.
> We have tried the Broadcom PMD on ARM server(s) and have been
> able to use it without any issues.
> But we can try it again and get back to you.
> 
> 
>>
>>          Thanks
>>
>>          Thierry Herbelot
>>
>> PS: a similar error is seen for the Ethernet ports, when handled by the
>> kernel:
>> dmesg|grep bnxt
>> [   57.823746] bnxt_en 0000:03:00.0 (unnamed net_device)
>> (uninitialized): Error (timeout: 500015) msg {0x0 0x0} len:0
>> [   59.202255] bnxt_en: probe of 0000:03:00.0 failed with error -16
>> [   60.641519] bnxt_en 0000:03:00.1 (unnamed net_device)
>> (uninitialized): Error (timeout: 500015) msg {0x0 0x0} len:0
>> [   61.502237] bnxt_en: probe of 0000:03:00.1 failed with error -16
>>
>>
>> --
>> Thierry Herbelot
>> Senior Software Engineer
>> Tel: +33 1 39 30 92 61
>> http://www.6wind.com/
>>
>> Follow us:
>> https://www.linkedin.com/company/6wind/
>> https://twitter.com/6WINDsoftware
>> https://www.youtube.com/user/6windsoftware

-- 
Thierry Herbelot
Senior Software Engineer
Tel: +33 1 39 30 92 61
http://www.6wind.com/

Follow us:
https://www.linkedin.com/company/6wind/
https://twitter.com/6WINDsoftware
https://www.youtube.com/user/6windsoftware

Re: [DPDK/bnxt] Issues starting bnxt on an Arm server

[Ajit Khaparde]

[-- Attachment #1: Type: text/plain, Size: 1633 bytes --]

On Wed, Mar 2, 2022 at 7:13 AM Thierry Herbelot
<thierry.herbelot@6wind.com> wrote:
>
> Hello Ajit
>
> Thanks for the quick response.
>
> Please keep me posted when you check on your side.
Can you tell what is the firmware version on the card?

>
>         Best regards
>
>         Thierry
>
> On 3/2/22 16:08, Ajit Khaparde wrote:
> >>
> >> Is this a know issue ?
> > No.
> > We have tried the Broadcom PMD on ARM server(s) and have been
> > able to use it without any issues.
> > But we can try it again and get back to you.
> >
> >
> >>
> >>          Thanks
> >>
> >>          Thierry Herbelot
> >>
> >> PS: a similar error is seen for the Ethernet ports, when handled by the
> >> kernel:
> >> dmesg|grep bnxt
> >> [   57.823746] bnxt_en 0000:03:00.0 (unnamed net_device)
> >> (uninitialized): Error (timeout: 500015) msg {0x0 0x0} len:0
> >> [   59.202255] bnxt_en: probe of 0000:03:00.0 failed with error -16
> >> [   60.641519] bnxt_en 0000:03:00.1 (unnamed net_device)
> >> (uninitialized): Error (timeout: 500015) msg {0x0 0x0} len:0
> >> [   61.502237] bnxt_en: probe of 0000:03:00.1 failed with error -16
> >>
> >>
> >> --
> >> Thierry Herbelot
> >> Senior Software Engineer
> >> Tel: +33 1 39 30 92 61
> >> http://www.6wind.com/
> >>
> >> Follow us:
> >> https://www.linkedin.com/company/6wind/
> >> https://twitter.com/6WINDsoftware
> >> https://www.youtube.com/user/6windsoftware
>
> --
> Thierry Herbelot
> Senior Software Engineer
> Tel: +33 1 39 30 92 61
> http://www.6wind.com/
>
> Follow us:
> https://www.linkedin.com/company/6wind/
> https://twitter.com/6WINDsoftware
> https://www.youtube.com/user/6windsoftware

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4218 bytes --]

Re: [DPDK/bnxt] Issues starting bnxt on an Arm server

[Thierry Herbelot]

Hello Ajit

The card is not correctly probed by the kernel, and so the FW version is 
not visible.

I see this in the kernel dmesg:
bnxt_en 0000:03:00.0 (unnamed net_device) (uninitialized): Error 
(timeout: 500015) msg {0x0 0x0} len:0
bnxt_en 0000:03:00.0 (unnamed net_device) (uninitialized): Firmware not 
responding, status: 0x8ffff

And then there are no bnxt_en netdevices.

	Best regards

	Thierry


On 3/2/22 18:13, Ajit Khaparde wrote:
> On Wed, Mar 2, 2022 at 7:13 AM Thierry Herbelot
> <thierry.herbelot@6wind.com> wrote:
>>
>> Hello Ajit
>>
>> Thanks for the quick response.
>>
>> Please keep me posted when you check on your side.
> Can you tell what is the firmware version on the card?
> 
>>
>>          Best regards
>>
>>          Thierry
>>
>> On 3/2/22 16:08, Ajit Khaparde wrote:
>>>>
>>>> Is this a know issue ?
>>> No.
>>> We have tried the Broadcom PMD on ARM server(s) and have been
>>> able to use it without any issues.
>>> But we can try it again and get back to you.
>>>
>>>
>>>>
>>>>           Thanks
>>>>
>>>>           Thierry Herbelot
>>>>
>>>> PS: a similar error is seen for the Ethernet ports, when handled by the
>>>> kernel:
>>>> dmesg|grep bnxt
>>>> [   57.823746] bnxt_en 0000:03:00.0 (unnamed net_device)
>>>> (uninitialized): Error (timeout: 500015) msg {0x0 0x0} len:0
>>>> [   59.202255] bnxt_en: probe of 0000:03:00.0 failed with error -16
>>>> [   60.641519] bnxt_en 0000:03:00.1 (unnamed net_device)
>>>> (uninitialized): Error (timeout: 500015) msg {0x0 0x0} len:0
>>>> [   61.502237] bnxt_en: probe of 0000:03:00.1 failed with error -16
>>>>
>>>>
>>>> --
>>>> Thierry Herbelot
>>>> Senior Software Engineer
>>>> Tel: +33 1 39 30 92 61
>>>> http://www.6wind.com/
>>>>
>>>> Follow us:
>>>> https://www.linkedin.com/company/6wind/
>>>> https://twitter.com/6WINDsoftware
>>>> https://www.youtube.com/user/6windsoftware
>>
>> --
>> Thierry Herbelot
>> Senior Software Engineer
>> Tel: +33 1 39 30 92 61
>> http://www.6wind.com/
>>
>> Follow us:
>> https://www.linkedin.com/company/6wind/
>> https://twitter.com/6WINDsoftware
>> https://www.youtube.com/user/6windsoftware

-- 
Thierry Herbelot
Senior Software Engineer
Tel: +33 1 39 30 92 61
http://www.6wind.com/

Follow us:
https://www.linkedin.com/company/6wind/
https://twitter.com/6WINDsoftware
https://www.youtube.com/user/6windsoftware

Re: [DPDK/bnxt] Issues starting bnxt on an Arm server

[Ajit Khaparde]

[-- Attachment #1: Type: text/plain, Size: 2734 bytes --]

On Wed, Mar 2, 2022 at 9:16 AM Thierry Herbelot
<thierry.herbelot@6wind.com> wrote:
>
> Hello Ajit
>
> The card is not correctly probed by the kernel, and so the FW version is
> not visible.
Thanks Thierry.
I thought it was just the PMD running into trouble.

Can you give details of the ARM server?
Make, Vendor, etc..?


>
> I see this in the kernel dmesg:
> bnxt_en 0000:03:00.0 (unnamed net_device) (uninitialized): Error
> (timeout: 500015) msg {0x0 0x0} len:0
> bnxt_en 0000:03:00.0 (unnamed net_device) (uninitialized): Firmware not
> responding, status: 0x8ffff
>
> And then there are no bnxt_en netdevices.
>
>         Best regards
>
>         Thierry
>
>
> On 3/2/22 18:13, Ajit Khaparde wrote:
> > On Wed, Mar 2, 2022 at 7:13 AM Thierry Herbelot
> > <thierry.herbelot@6wind.com> wrote:
> >>
> >> Hello Ajit
> >>
> >> Thanks for the quick response.
> >>
> >> Please keep me posted when you check on your side.
> > Can you tell what is the firmware version on the card?
> >
> >>
> >>          Best regards
> >>
> >>          Thierry
> >>
> >> On 3/2/22 16:08, Ajit Khaparde wrote:
> >>>>
> >>>> Is this a know issue ?
> >>> No.
> >>> We have tried the Broadcom PMD on ARM server(s) and have been
> >>> able to use it without any issues.
> >>> But we can try it again and get back to you.
> >>>
> >>>
> >>>>
> >>>>           Thanks
> >>>>
> >>>>           Thierry Herbelot
> >>>>
> >>>> PS: a similar error is seen for the Ethernet ports, when handled by the
> >>>> kernel:
> >>>> dmesg|grep bnxt
> >>>> [   57.823746] bnxt_en 0000:03:00.0 (unnamed net_device)
> >>>> (uninitialized): Error (timeout: 500015) msg {0x0 0x0} len:0
> >>>> [   59.202255] bnxt_en: probe of 0000:03:00.0 failed with error -16
> >>>> [   60.641519] bnxt_en 0000:03:00.1 (unnamed net_device)
> >>>> (uninitialized): Error (timeout: 500015) msg {0x0 0x0} len:0
> >>>> [   61.502237] bnxt_en: probe of 0000:03:00.1 failed with error -16
> >>>>
> >>>>
> >>>> --
> >>>> Thierry Herbelot
> >>>> Senior Software Engineer
> >>>> Tel: +33 1 39 30 92 61
> >>>> http://www.6wind.com/
> >>>>
> >>>> Follow us:
> >>>> https://www.linkedin.com/company/6wind/
> >>>> https://twitter.com/6WINDsoftware
> >>>> https://www.youtube.com/user/6windsoftware
> >>
> >> --
> >> Thierry Herbelot
> >> Senior Software Engineer
> >> Tel: +33 1 39 30 92 61
> >> http://www.6wind.com/
> >>
> >> Follow us:
> >> https://www.linkedin.com/company/6wind/
> >> https://twitter.com/6WINDsoftware
> >> https://www.youtube.com/user/6windsoftware
>
> --
> Thierry Herbelot
> Senior Software Engineer
> Tel: +33 1 39 30 92 61
> http://www.6wind.com/
>
> Follow us:
> https://www.linkedin.com/company/6wind/
> https://twitter.com/6WINDsoftware
> https://www.youtube.com/user/6windsoftware

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4218 bytes --]

Re: [DPDK/bnxt] Issues starting bnxt on an Arm server

[Thierry Herbelot]

On 3/2/22 18:26, Ajit Khaparde wrote:
> On Wed, Mar 2, 2022 at 9:16 AM Thierry Herbelot
> <thierry.herbelot@6wind.com> wrote:
>>
>> Hello Ajit
>>
>> The card is not correctly probed by the kernel, and so the FW version is
>> not visible.
> Thanks Thierry.
> I thought it was just the PMD running into trouble.
> 
> Can you give details of the ARM server?
> Make, Vendor, etc..?

This a Huawei server:
TaiShan 200 Server (Model 2180) aarch64, 48 cores, 2.6GHz, 64GB RAM

	Thierry

> 
> 
>>
>> I see this in the kernel dmesg:
>> bnxt_en 0000:03:00.0 (unnamed net_device) (uninitialized): Error
>> (timeout: 500015) msg {0x0 0x0} len:0
>> bnxt_en 0000:03:00.0 (unnamed net_device) (uninitialized): Firmware not
>> responding, status: 0x8ffff
>>
>> And then there are no bnxt_en netdevices.
>>
>>          Best regards
>>
>>          Thierry
>>
>>
>> On 3/2/22 18:13, Ajit Khaparde wrote:
>>> On Wed, Mar 2, 2022 at 7:13 AM Thierry Herbelot
>>> <thierry.herbelot@6wind.com> wrote:
>>>>
>>>> Hello Ajit
>>>>
>>>> Thanks for the quick response.
>>>>
>>>> Please keep me posted when you check on your side.
>>> Can you tell what is the firmware version on the card?
>>>
>>>>
>>>>           Best regards
>>>>
>>>>           Thierry
>>>>
>>>> On 3/2/22 16:08, Ajit Khaparde wrote:
>>>>>>
>>>>>> Is this a know issue ?
>>>>> No.
>>>>> We have tried the Broadcom PMD on ARM server(s) and have been
>>>>> able to use it without any issues.
>>>>> But we can try it again and get back to you.
>>>>>
>>>>>
>>>>>>
>>>>>>            Thanks
>>>>>>
>>>>>>            Thierry Herbelot
>>>>>>
>>>>>> PS: a similar error is seen for the Ethernet ports, when handled by the
>>>>>> kernel:
>>>>>> dmesg|grep bnxt
>>>>>> [   57.823746] bnxt_en 0000:03:00.0 (unnamed net_device)
>>>>>> (uninitialized): Error (timeout: 500015) msg {0x0 0x0} len:0
>>>>>> [   59.202255] bnxt_en: probe of 0000:03:00.0 failed with error -16
>>>>>> [   60.641519] bnxt_en 0000:03:00.1 (unnamed net_device)
>>>>>> (uninitialized): Error (timeout: 500015) msg {0x0 0x0} len:0
>>>>>> [   61.502237] bnxt_en: probe of 0000:03:00.1 failed with error -16
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Thierry Herbelot
>>>>>> Senior Software Engineer
>>>>>> Tel: +33 1 39 30 92 61
>>>>>> http://www.6wind.com/
>>>>>>
>>>>>> Follow us:
>>>>>> https://www.linkedin.com/company/6wind/
>>>>>> https://twitter.com/6WINDsoftware
>>>>>> https://www.youtube.com/user/6windsoftware
>>>>
>>>> --
>>>> Thierry Herbelot
>>>> Senior Software Engineer
>>>> Tel: +33 1 39 30 92 61
>>>> http://www.6wind.com/
>>>>
>>>> Follow us:
>>>> https://www.linkedin.com/company/6wind/
>>>> https://twitter.com/6WINDsoftware
>>>> https://www.youtube.com/user/6windsoftware
>>
>> --
>> Thierry Herbelot
>> Senior Software Engineer
>> Tel: +33 1 39 30 92 61
>> http://www.6wind.com/
>>
>> Follow us:
>> https://www.linkedin.com/company/6wind/
>> https://twitter.com/6WINDsoftware
>> https://www.youtube.com/user/6windsoftware

-- 
Thierry Herbelot
Senior Software Engineer
Tel: +33 1 39 30 92 61
http://www.6wind.com/

Follow us:
https://www.linkedin.com/company/6wind/
https://twitter.com/6WINDsoftware
https://www.youtube.com/user/6windsoftware

Re: [DPDK/bnxt] Issues starting bnxt on an Arm server

[Ajit Khaparde]

[-- Attachment #1: Type: text/plain, Size: 3793 bytes --]

On Wed, Mar 2, 2022 at 9:28 AM Thierry Herbelot
<thierry.herbelot@6wind.com> wrote:
>
> On 3/2/22 18:26, Ajit Khaparde wrote:
> > On Wed, Mar 2, 2022 at 9:16 AM Thierry Herbelot
> > <thierry.herbelot@6wind.com> wrote:
> >>
> >> Hello Ajit
> >>
> >> The card is not correctly probed by the kernel, and so the FW version is
> >> not visible.
> > Thanks Thierry.
> > I thought it was just the PMD running into trouble.
> >
> > Can you give details of the ARM server?
> > Make, Vendor, etc..?
>
> This a Huawei server:
> TaiShan 200 Server (Model 2180) aarch64, 48 cores, 2.6GHz, 64GB RAM

Ok. I tried this on a HUAWEI Kunpeng 920 running 4.18.0-348.2.1.el8_5.aarch64.
I can load the bnxt_en kernel driver and the DPDK PMD without issues.
And I did load vfio with enable_unsafe_noiommu_mode=1 .

While we are pursuing this further internally,
I have added Honnappa to see if he can provide any insight.

Ajit

>
>         Thierry
>
> >
> >
> >>
> >> I see this in the kernel dmesg:
> >> bnxt_en 0000:03:00.0 (unnamed net_device) (uninitialized): Error
> >> (timeout: 500015) msg {0x0 0x0} len:0
> >> bnxt_en 0000:03:00.0 (unnamed net_device) (uninitialized): Firmware not
> >> responding, status: 0x8ffff
> >>
> >> And then there are no bnxt_en netdevices.
> >>
> >>          Best regards
> >>
> >>          Thierry
> >>
> >>
> >> On 3/2/22 18:13, Ajit Khaparde wrote:
> >>> On Wed, Mar 2, 2022 at 7:13 AM Thierry Herbelot
> >>> <thierry.herbelot@6wind.com> wrote:
> >>>>
> >>>> Hello Ajit
> >>>>
> >>>> Thanks for the quick response.
> >>>>
> >>>> Please keep me posted when you check on your side.
> >>> Can you tell what is the firmware version on the card?
> >>>
> >>>>
> >>>>           Best regards
> >>>>
> >>>>           Thierry
> >>>>
> >>>> On 3/2/22 16:08, Ajit Khaparde wrote:
> >>>>>>
> >>>>>> Is this a know issue ?
> >>>>> No.
> >>>>> We have tried the Broadcom PMD on ARM server(s) and have been
> >>>>> able to use it without any issues.
> >>>>> But we can try it again and get back to you.
> >>>>>
> >>>>>
> >>>>>>
> >>>>>>            Thanks
> >>>>>>
> >>>>>>            Thierry Herbelot
> >>>>>>
> >>>>>> PS: a similar error is seen for the Ethernet ports, when handled by the
> >>>>>> kernel:
> >>>>>> dmesg|grep bnxt
> >>>>>> [   57.823746] bnxt_en 0000:03:00.0 (unnamed net_device)
> >>>>>> (uninitialized): Error (timeout: 500015) msg {0x0 0x0} len:0
> >>>>>> [   59.202255] bnxt_en: probe of 0000:03:00.0 failed with error -16
> >>>>>> [   60.641519] bnxt_en 0000:03:00.1 (unnamed net_device)
> >>>>>> (uninitialized): Error (timeout: 500015) msg {0x0 0x0} len:0
> >>>>>> [   61.502237] bnxt_en: probe of 0000:03:00.1 failed with error -16
> >>>>>>
> >>>>>>
> >>>>>> --
> >>>>>> Thierry Herbelot
> >>>>>> Senior Software Engineer
> >>>>>> Tel: +33 1 39 30 92 61
> >>>>>> http://www.6wind.com/
> >>>>>>
> >>>>>> Follow us:
> >>>>>> https://www.linkedin.com/company/6wind/
> >>>>>> https://twitter.com/6WINDsoftware
> >>>>>> https://www.youtube.com/user/6windsoftware
> >>>>
> >>>> --
> >>>> Thierry Herbelot
> >>>> Senior Software Engineer
> >>>> Tel: +33 1 39 30 92 61
> >>>> http://www.6wind.com/
> >>>>
> >>>> Follow us:
> >>>> https://www.linkedin.com/company/6wind/
> >>>> https://twitter.com/6WINDsoftware
> >>>> https://www.youtube.com/user/6windsoftware
> >>
> >> --
> >> Thierry Herbelot
> >> Senior Software Engineer
> >> Tel: +33 1 39 30 92 61
> >> http://www.6wind.com/
> >>
> >> Follow us:
> >> https://www.linkedin.com/company/6wind/
> >> https://twitter.com/6WINDsoftware
> >> https://www.youtube.com/user/6windsoftware
>
> --
> Thierry Herbelot
> Senior Software Engineer
> Tel: +33 1 39 30 92 61
> http://www.6wind.com/
>
> Follow us:
> https://www.linkedin.com/company/6wind/
> https://twitter.com/6WINDsoftware
> https://www.youtube.com/user/6windsoftware

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4218 bytes --]

RE: [DPDK/bnxt] Issues starting bnxt on an Arm server

[Honnappa Nagarahalli]

+ Ruifeng 

<snip>

> 
> On Wed, Mar 2, 2022 at 9:28 AM Thierry Herbelot
> <thierry.herbelot@6wind.com> wrote:
> >
> > On 3/2/22 18:26, Ajit Khaparde wrote:
> > > On Wed, Mar 2, 2022 at 9:16 AM Thierry Herbelot
> > > <thierry.herbelot@6wind.com> wrote:
> > >>
> > >> Hello Ajit
> > >>
> > >> The card is not correctly probed by the kernel, and so the FW version is
> > >> not visible.
> > > Thanks Thierry.
> > > I thought it was just the PMD running into trouble.
> > >
> > > Can you give details of the ARM server?
> > > Make, Vendor, etc..?
> >
> > This a Huawei server:
> > TaiShan 200 Server (Model 2180) aarch64, 48 cores, 2.6GHz, 64GB RAM
> 
> Ok. I tried this on a HUAWEI Kunpeng 920 running 4.18.0-
> 348.2.1.el8_5.aarch64.
> I can load the bnxt_en kernel driver and the DPDK PMD without issues.
> And I did load vfio with enable_unsafe_noiommu_mode=1 .
> 
> While we are pursuing this further internally,
> I have added Honnappa to see if he can provide any insight.
There is not much information on the error.
Just wondering if it is possible to move the card to x86 machine and check the firmware version and upgrade the firmware version there.

> 
> Ajit
> 
> >
> >         Thierry
> >
> > >
> > >
> > >>
> > >> I see this in the kernel dmesg:
> > >> bnxt_en 0000:03:00.0 (unnamed net_device) (uninitialized): Error
> > >> (timeout: 500015) msg {0x0 0x0} len:0
> > >> bnxt_en 0000:03:00.0 (unnamed net_device) (uninitialized): Firmware not
> > >> responding, status: 0x8ffff
> > >>
> > >> And then there are no bnxt_en netdevices.
> > >>
> > >>          Best regards
> > >>
> > >>          Thierry
> > >>
> > >>
> > >> On 3/2/22 18:13, Ajit Khaparde wrote:
> > >>> On Wed, Mar 2, 2022 at 7:13 AM Thierry Herbelot
> > >>> <thierry.herbelot@6wind.com> wrote:
> > >>>>
> > >>>> Hello Ajit
> > >>>>
> > >>>> Thanks for the quick response.
> > >>>>
> > >>>> Please keep me posted when you check on your side.
> > >>> Can you tell what is the firmware version on the card?
> > >>>
> > >>>>
> > >>>>           Best regards
> > >>>>
> > >>>>           Thierry
> > >>>>
> > >>>> On 3/2/22 16:08, Ajit Khaparde wrote:
> > >>>>>>
> > >>>>>> Is this a know issue ?
> > >>>>> No.
> > >>>>> We have tried the Broadcom PMD on ARM server(s) and have been
> > >>>>> able to use it without any issues.
> > >>>>> But we can try it again and get back to you.
> > >>>>>
> > >>>>>
> > >>>>>>
> > >>>>>>            Thanks
> > >>>>>>
> > >>>>>>            Thierry Herbelot
> > >>>>>>
> > >>>>>> PS: a similar error is seen for the Ethernet ports, when handled by the
> > >>>>>> kernel:
> > >>>>>> dmesg|grep bnxt
> > >>>>>> [   57.823746] bnxt_en 0000:03:00.0 (unnamed net_device)
> > >>>>>> (uninitialized): Error (timeout: 500015) msg {0x0 0x0} len:0
> > >>>>>> [   59.202255] bnxt_en: probe of 0000:03:00.0 failed with error -16
> > >>>>>> [   60.641519] bnxt_en 0000:03:00.1 (unnamed net_device)
> > >>>>>> (uninitialized): Error (timeout: 500015) msg {0x0 0x0} len:0
> > >>>>>> [   61.502237] bnxt_en: probe of 0000:03:00.1 failed with error -16
> > >>>>>>
> > >>>>>>
> > >>>>>> --
> > >>>>>> Thierry Herbelot
> > >>>>>> Senior Software Engineer
> > >>>>>> Tel: +33 1 39 30 92 61
> > >>>>>> http://www.6wind.com/
> > >>>>>>
> > >>>>>> Follow us:
> > >>>>>> https://www.linkedin.com/company/6wind/
> > >>>>>> https://twitter.com/6WINDsoftware
> > >>>>>> https://www.youtube.com/user/6windsoftware
> > >>>>
> > >>>> --
> > >>>> Thierry Herbelot
> > >>>> Senior Software Engineer
> > >>>> Tel: +33 1 39 30 92 61
> > >>>> http://www.6wind.com/
> > >>>>
> > >>>> Follow us:
> > >>>> https://www.linkedin.com/company/6wind/
> > >>>> https://twitter.com/6WINDsoftware
> > >>>> https://www.youtube.com/user/6windsoftware
> > >>
> > >> --
> > >> Thierry Herbelot
> > >> Senior Software Engineer
> > >> Tel: +33 1 39 30 92 61
> > >> http://www.6wind.com/
> > >>
> > >> Follow us:
> > >> https://www.linkedin.com/company/6wind/
> > >> https://twitter.com/6WINDsoftware
> > >> https://www.youtube.com/user/6windsoftware
> >
> > --
> > Thierry Herbelot
> > Senior Software Engineer
> > Tel: +33 1 39 30 92 61
> > http://www.6wind.com/
> >
> > Follow us:
> > https://www.linkedin.com/company/6wind/
> > https://twitter.com/6WINDsoftware
> > https://www.youtube.com/user/6windsoftware

Re: [DPDK/bnxt] Issues starting bnxt on an Arm server

[Thierry Herbelot]

Hello,

After upgrading the firmware on the bnxt NIC, the NIC is correctly 
detected by a DPDK application with both DPDK-v20.11 and DPDK-v21.11.

	Regards

	Thierry

On 3/4/22 01:56, Honnappa Nagarahalli wrote:
> + Ruifeng
> 
> <snip>
> 
>>
>> On Wed, Mar 2, 2022 at 9:28 AM Thierry Herbelot
>> <thierry.herbelot@6wind.com> wrote:
>>>
>>> On 3/2/22 18:26, Ajit Khaparde wrote:
>>>> On Wed, Mar 2, 2022 at 9:16 AM Thierry Herbelot
>>>> <thierry.herbelot@6wind.com> wrote:
>>>>>
>>>>> Hello Ajit
>>>>>
>>>>> The card is not correctly probed by the kernel, and so the FW version is
>>>>> not visible.
>>>> Thanks Thierry.
>>>> I thought it was just the PMD running into trouble.
>>>>
>>>> Can you give details of the ARM server?
>>>> Make, Vendor, etc..?
>>>
>>> This a Huawei server:
>>> TaiShan 200 Server (Model 2180) aarch64, 48 cores, 2.6GHz, 64GB RAM
>>
>> Ok. I tried this on a HUAWEI Kunpeng 920 running 4.18.0-
>> 348.2.1.el8_5.aarch64.
>> I can load the bnxt_en kernel driver and the DPDK PMD without issues.
>> And I did load vfio with enable_unsafe_noiommu_mode=1 .
>>
>> While we are pursuing this further internally,
>> I have added Honnappa to see if he can provide any insight.
> There is not much information on the error.
> Just wondering if it is possible to move the card to x86 machine and check the firmware version and upgrade the firmware version there.
> 
>>
>> Ajit
>>
>>>
>>>          Thierry
>>>
>>>>
>>>>
>>>>>
>>>>> I see this in the kernel dmesg:
>>>>> bnxt_en 0000:03:00.0 (unnamed net_device) (uninitialized): Error
>>>>> (timeout: 500015) msg {0x0 0x0} len:0
>>>>> bnxt_en 0000:03:00.0 (unnamed net_device) (uninitialized): Firmware not
>>>>> responding, status: 0x8ffff
>>>>>
>>>>> And then there are no bnxt_en netdevices.
>>>>>
>>>>>           Best regards
>>>>>
>>>>>           Thierry
>>>>>
>>>>>
>>>>> On 3/2/22 18:13, Ajit Khaparde wrote:
>>>>>> On Wed, Mar 2, 2022 at 7:13 AM Thierry Herbelot
>>>>>> <thierry.herbelot@6wind.com> wrote:
>>>>>>>
>>>>>>> Hello Ajit
>>>>>>>
>>>>>>> Thanks for the quick response.
>>>>>>>
>>>>>>> Please keep me posted when you check on your side.
>>>>>> Can you tell what is the firmware version on the card?
>>>>>>
>>>>>>>
>>>>>>>            Best regards
>>>>>>>
>>>>>>>            Thierry
>>>>>>>
>>>>>>> On 3/2/22 16:08, Ajit Khaparde wrote:
>>>>>>>>>
>>>>>>>>> Is this a know issue ?
>>>>>>>> No.
>>>>>>>> We have tried the Broadcom PMD on ARM server(s) and have been
>>>>>>>> able to use it without any issues.
>>>>>>>> But we can try it again and get back to you.
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>>             Thanks
>>>>>>>>>
>>>>>>>>>             Thierry Herbelot
>>>>>>>>>
>>>>>>>>> PS: a similar error is seen for the Ethernet ports, when handled by the
>>>>>>>>> kernel:
>>>>>>>>> dmesg|grep bnxt
>>>>>>>>> [   57.823746] bnxt_en 0000:03:00.0 (unnamed net_device)
>>>>>>>>> (uninitialized): Error (timeout: 500015) msg {0x0 0x0} len:0
>>>>>>>>> [   59.202255] bnxt_en: probe of 0000:03:00.0 failed with error -16
>>>>>>>>> [   60.641519] bnxt_en 0000:03:00.1 (unnamed net_device)
>>>>>>>>> (uninitialized): Error (timeout: 500015) msg {0x0 0x0} len:0
>>>>>>>>> [   61.502237] bnxt_en: probe of 0000:03:00.1 failed with error -16
>>>>>>>>>
>>>>>>>>>
-- 
Thierry Herbelot
Senior Software Engineer
Tel: +33 1 39 30 92 61
http://www.6wind.com/

Follow us:
https://www.linkedin.com/company/6wind/
https://twitter.com/6WINDsoftware
https://www.youtube.com/user/6windsoftware

Re: [DPDK/bnxt] Issues starting bnxt on an Arm server

[Ajit Khaparde]

[-- Attachment #1: Type: text/plain, Size: 3820 bytes --]

On Tue, Mar 8, 2022 at 12:19 AM Thierry Herbelot
<thierry.herbelot@6wind.com> wrote:
>
> Hello,
>
> After upgrading the firmware on the bnxt NIC, the NIC is correctly
> detected by a DPDK application with both DPDK-v20.11 and DPDK-v21.11.
Great. Thanks for the update Thierry.

>
>         Regards
>
>         Thierry
>
> On 3/4/22 01:56, Honnappa Nagarahalli wrote:
> > + Ruifeng
> >
> > <snip>
> >
> >>
> >> On Wed, Mar 2, 2022 at 9:28 AM Thierry Herbelot
> >> <thierry.herbelot@6wind.com> wrote:
> >>>
> >>> On 3/2/22 18:26, Ajit Khaparde wrote:
> >>>> On Wed, Mar 2, 2022 at 9:16 AM Thierry Herbelot
> >>>> <thierry.herbelot@6wind.com> wrote:
> >>>>>
> >>>>> Hello Ajit
> >>>>>
> >>>>> The card is not correctly probed by the kernel, and so the FW version is
> >>>>> not visible.
> >>>> Thanks Thierry.
> >>>> I thought it was just the PMD running into trouble.
> >>>>
> >>>> Can you give details of the ARM server?
> >>>> Make, Vendor, etc..?
> >>>
> >>> This a Huawei server:
> >>> TaiShan 200 Server (Model 2180) aarch64, 48 cores, 2.6GHz, 64GB RAM
> >>
> >> Ok. I tried this on a HUAWEI Kunpeng 920 running 4.18.0-
> >> 348.2.1.el8_5.aarch64.
> >> I can load the bnxt_en kernel driver and the DPDK PMD without issues.
> >> And I did load vfio with enable_unsafe_noiommu_mode=1 .
> >>
> >> While we are pursuing this further internally,
> >> I have added Honnappa to see if he can provide any insight.
> > There is not much information on the error.
> > Just wondering if it is possible to move the card to x86 machine and check the firmware version and upgrade the firmware version there.
> >
> >>
> >> Ajit
> >>
> >>>
> >>>          Thierry
> >>>
> >>>>
> >>>>
> >>>>>
> >>>>> I see this in the kernel dmesg:
> >>>>> bnxt_en 0000:03:00.0 (unnamed net_device) (uninitialized): Error
> >>>>> (timeout: 500015) msg {0x0 0x0} len:0
> >>>>> bnxt_en 0000:03:00.0 (unnamed net_device) (uninitialized): Firmware not
> >>>>> responding, status: 0x8ffff
> >>>>>
> >>>>> And then there are no bnxt_en netdevices.
> >>>>>
> >>>>>           Best regards
> >>>>>
> >>>>>           Thierry
> >>>>>
> >>>>>
> >>>>> On 3/2/22 18:13, Ajit Khaparde wrote:
> >>>>>> On Wed, Mar 2, 2022 at 7:13 AM Thierry Herbelot
> >>>>>> <thierry.herbelot@6wind.com> wrote:
> >>>>>>>
> >>>>>>> Hello Ajit
> >>>>>>>
> >>>>>>> Thanks for the quick response.
> >>>>>>>
> >>>>>>> Please keep me posted when you check on your side.
> >>>>>> Can you tell what is the firmware version on the card?
> >>>>>>
> >>>>>>>
> >>>>>>>            Best regards
> >>>>>>>
> >>>>>>>            Thierry
> >>>>>>>
> >>>>>>> On 3/2/22 16:08, Ajit Khaparde wrote:
> >>>>>>>>>
> >>>>>>>>> Is this a know issue ?
> >>>>>>>> No.
> >>>>>>>> We have tried the Broadcom PMD on ARM server(s) and have been
> >>>>>>>> able to use it without any issues.
> >>>>>>>> But we can try it again and get back to you.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>>
> >>>>>>>>>             Thanks
> >>>>>>>>>
> >>>>>>>>>             Thierry Herbelot
> >>>>>>>>>
> >>>>>>>>> PS: a similar error is seen for the Ethernet ports, when handled by the
> >>>>>>>>> kernel:
> >>>>>>>>> dmesg|grep bnxt
> >>>>>>>>> [   57.823746] bnxt_en 0000:03:00.0 (unnamed net_device)
> >>>>>>>>> (uninitialized): Error (timeout: 500015) msg {0x0 0x0} len:0
> >>>>>>>>> [   59.202255] bnxt_en: probe of 0000:03:00.0 failed with error -16
> >>>>>>>>> [   60.641519] bnxt_en 0000:03:00.1 (unnamed net_device)
> >>>>>>>>> (uninitialized): Error (timeout: 500015) msg {0x0 0x0} len:0
> >>>>>>>>> [   61.502237] bnxt_en: probe of 0000:03:00.1 failed with error -16
> >>>>>>>>>
> >>>>>>>>>
> --
> Thierry Herbelot
> Senior Software Engineer
> Tel: +33 1 39 30 92 61
> http://www.6wind.com/
>
> Follow us:
> https://www.linkedin.com/company/6wind/
> https://twitter.com/6WINDsoftware
> https://www.youtube.com/user/6windsoftware

[-- Attachment #2: S/MIME Cryptographic Signature --]
[-- Type: application/pkcs7-signature, Size: 4218 bytes --]

Re: [DPDK/bnxt] Issues starting bnxt on an Arm server

[David Marchand]

On Wed, Mar 2, 2022 at 3:51 PM Thierry Herbelot
<thierry.herbelot@6wind.com> wrote:
> EAL: Probe PCI driver: net_bnxt (14e4:1750) device: 0000:03:00.0 (socket 0)
> bnxt_hwrm_send_message(): Error(timeout) sending msg 0x0000, seq_id 0
> bnxt_hwrm_ver_get(): failed rc:-110
> ethdev initialisation failed
> EAL: Releasing pci mapped resource for 0000:03:00.0
> EAL: Calling pci_unmap_resource for 0000:03:00.0 at 0x4180000000
> EAL: Calling pci_unmap_resource for 0000:03:00.0 at 0x4180010000
> EAL: Requested device 0000:03:00.0 cannot be used
> EAL:   using IOMMU type 8 (No-IOMMU)
> EAL: Probe PCI driver: net_bnxt (14e4:1750) device: 0000:03:00.1 (socket 0)
> bnxt_hwrm_send_message(): Error(timeout) sending msg 0x0000, seq_id 0
> bnxt_hwrm_ver_get(): failed rc:-110
> ethdev initialisation failed
> EAL: Releasing pci mapped resource for 0000:03:00.1
> EAL: Calling pci_unmap_resource for 0000:03:00.1 at 0x4180110000
> EAL: Calling pci_unmap_resource for 0000:03:00.1 at 0x4180120000
> EAL: Requested device 0000:03:00.1 cannot be used
>

Maybe the same as:
http://inbox.dpdk.org/dev/CAJFAV8yezCJ==az6h0k9Wztiia-azs19Ydr9cGjnd+5TYjGOXQ@mail.gmail.com/T/#u


-- 
David Marchand