From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mails.dpdk.org (mails.dpdk.org [217.70.189.124]) by inbox.dpdk.org (Postfix) with ESMTP id F2F6742437; Fri, 20 Jan 2023 16:47:45 +0100 (CET) Received: from mails.dpdk.org (localhost [127.0.0.1]) by mails.dpdk.org (Postfix) with ESMTP id 915F240150; Fri, 20 Jan 2023 16:47:45 +0100 (CET) Received: from mail-yw1-f171.google.com (mail-yw1-f171.google.com [209.85.128.171]) by mails.dpdk.org (Postfix) with ESMTP id F1C8F400D5 for ; Fri, 20 Jan 2023 16:47:43 +0100 (CET) Received: by mail-yw1-f171.google.com with SMTP id 00721157ae682-50112511ba7so18735617b3.3 for ; Fri, 20 Jan 2023 07:47:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=gbflDu3Ogc+sEI3tN0AjGetDf9AoOnu7Eu+pTd7ZyDU=; b=fd5Gc7+YhNKF8QcN238HGDdawTL/K4eXIczkmiQI7ipK2w8/ayO9BW3R4jHyrGShiu +yMhVvN5L7rlR73akR03YHimRqvzgX4rWJy9FRhaxBMQHWSdbWG/7j4DgRqoCmGZxVPt lJWhYOz+KqcZp6NJ/8zv9jk1I6wHzDjLnB4jpPx3xTUIBl2rSY/pDLAeRxOeRyPuhmXr 6gm+FL9V24Q05MzF+/LlKTkLs78GErsauaZ+wQ9ASS/w6c+K3LZw1D/DRhIjm5K4iFrs GJ8sypo7pVeA+B4QsPhedACWewul2BS8BN2Qfktg3MQdzlwpZx57m5oVLTzyBokEVM8+ Jt3w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=gbflDu3Ogc+sEI3tN0AjGetDf9AoOnu7Eu+pTd7ZyDU=; b=V0LteIPQbyb1Y3DX7EzBDk4NSNWh4HYv267MnywkhXlCJHKBIwO/Fx0dYbqINacK9e k9/79ZOctDYMQ1NFpxAgRf4BmXWiAVJl5jYRBYjYGN+VCPpiPirXu+1n63YNFn/Rp9js m1TU3pegRDBDMeTdLRRn1b0bIbfLlSElES9wUILqE0q4GokW1yiGTxss70w0vxdTpF8M X5D8Rnyn3kp5swXOpY3HtJGhIEwn6umkPFg2d0oz6pvPAeQ8/ngokCBggRDGbhqo+cUO O9EsYSqcQgyQxc+o3hLajZjN5p50aNPJ0OHjcV+l0G+eWpYLoysggPOqiszUu084P8iR ykSg== X-Gm-Message-State: AFqh2krAGPN3L76cv4XF2HUFIBhd8AvOBBsvTsNn0Tbo3GvdnPsGUB+x 2FJ8eVF7YF/Qm303+gmyy9nu8iUhBAv/DFPqtM96D/b7Vrw= X-Google-Smtp-Source: AMrXdXvqL1DbM9xP9Ed3no7xvBs1Out9RrmfW6gCsauXcZaCypZo62ryO41akG4tSSXY7YDL0oqO01nXi3V8KobSVsM= X-Received: by 2002:a81:7309:0:b0:4f9:d46b:9eff with SMTP id o9-20020a817309000000b004f9d46b9effmr729643ywc.92.1674229662905; Fri, 20 Jan 2023 07:47:42 -0800 (PST) MIME-Version: 1.0 References: In-Reply-To: From: Ben Magistro Date: Fri, 20 Jan 2023 10:47:32 -0500 Message-ID: Subject: Re: ICE DDP Load assistance To: dev@dpdk.org Cc: ben.magistro@trinitycyber.com, Stefan Baranoff Content-Type: multipart/alternative; boundary="000000000000547a2805f2b3f898" X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.29 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org --000000000000547a2805f2b3f898 Content-Type: text/plain; charset="UTF-8" Okay, believe we have tracked down and can explain most of the current behavior. The question I have now is, is this really what is desired or should it be possible for DPDK be passed a flag and load a DDPK file anyways, or should it always try to update the DDP if a newer one is found. In our case we are unlikely to utilize the ICE based driver in the OS and would only be using it with DPDK. To me, in this scenario, it makes little sense to maintain/think about the OS driver and related items. With some increased logging from testpmd and looking at the ICE PMD source it was identified that we are hitting `ICE_ERR_AQ_NO_WORK` in `ice_ddp.c -> ice_acquire_global_cfg_lock:2232` which indicates that another driver has already written the package or has found that no update was necessary. My uneducated guess is this is happening as the OS starts up and then transitions the device over to vfio-pci which would also explain why the initial firmware load was resolved after running dracut. In the log scenario below, an update should definitely be necessary still as 1.3.30 is found but 1.3.26 is loaded. ---- 1822 pkg: /lib/firmware/updates/intel/ice/ddp/ice-40a6b7ffff8437f8.pkg 1829 pkg: /lib/firmware/intel/ice/ddp/ice-40a6b7ffff8437f8.pkg 1835 pkg: /lib/firmware/updates/intel/ice/ddp/ice.pkg 1847 pkg: /lib/firmware/updates/intel/ice/ddp/ice.pkg ice_load_pkg(): DDP package name: /lib/firmware/updates/intel/ice/ddp/ice.pkg init enter ice_find_seg_in_pkg(): ice 00.0 Package format version: 1.0.0.0 ice_init_pkg_info(): ice 00.0 Pkg using segment id: 0x00000010 ice_find_seg_in_pkg(): ice 00.0 Package format version: 1.0.0.0 ice_init_pkg_info(): ice 00.0 Pkg: 1.3.30.0, ICE OS Default Package ice_init_pkg_info(): ice 00.0 Ice Seg: 1.0.1.0, ICE Configuration Data ice_find_seg_in_pkg(): ice 00.0 Package format version: 1.0.0.0 ice_clean_sq(): ice 00.0 ntc 10 head 11. ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer: ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback: dl w sig ice_download_pkg_with_sig_seg(): ice 00.0 Segment ID 16 ice_download_pkg_with_sig_seg(): ice 00.0 Signature type 1 ice_acquire_res(): ice 00.0 ice_acquire_res ice_aq_req_res(): ice 00.0 ice_aq_req_res ice_clean_sq(): ice 00.0 ntc 11 head 12. ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer: ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback: ice_acquire_res(): ice 00.0 resource indicates no work to do. ice_acquire_global_cfg_lock(): ice 00.0 Global config lock: No work to do ice_clean_sq(): ice 00.0 ntc 12 head 13. ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer: ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback: ice_clean_sq(): ice 00.0 ntc 13 head 14. ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer: ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback: ice_clean_sq(): ice 00.0 ntc 14 head 15. ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer: ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback: ice_init_pkg(): ice 00.0 package previously loaded - no work. ice_clean_sq(): ice 00.0 ntc 15 head 16. ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer: ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback: ice_get_pkg_info(): ice 00.0 Pkg[0]: 1.0.0.17,ICE NVM Package,BMN ice_get_pkg_info(): ice 00.0 Pkg[1]: 1.3.26.0,ICE OS Default Package,A init end ice_load_pkg_type(): Active package is: 1.3.26.0, ICE OS Default Package (single VLAN mode) On Thu, Jan 19, 2023 at 9:14 AM Ben Magistro wrote: > Hello, > > We are still trying to track this down but want to reach out to the > community and see if there is something obvious we are missing. This > happens to be a CentOS 7 based host with DPDK 22.11.1 in use. We have > obtained the latest compatible Intel drivers and DDP from > https://sourceforge.net/projects/e1000/files/ice%20stable/ , at this time > that is version 1.6.7 with DDP 1.3.26.0. As best as we can tell, testpmd > is still loading the OS default DDP 1.3.4.0. The updated DDP package is > located in /usr/lib/firmware/updates/intel/ice/ddp/ice.pkg. > > I've added some additional log lines to ice_ethdev.c and ice_ddp.c to try > and start debugging. I've attached what I believe is the relevant > information below. Quickly jumping out at me is that the pkg_name going > into `load_fw` in ice_ethdev.c appears to be the desired path however in > `ice_download_pkg` says it does not include a signature. The hex dump from > the desired file appears to show a signature which makes us think this file > isn't actually being loaded/utilized. > > We will continue trying to debug this too but any additional assistance > would be greatly appreciated. > > ---- > > EAL: Probe PCI driver: net_ice (8086:1593) device: 0000:ca:00.2 (socket 1) > 1822 pkg: /lib/firmware/updates/intel/ice/ddp/ice-40a6b7ffff843810.pkg > 1829 pkg: /lib/firmware/intel/ice/ddp/ice-40a6b7ffff843810.pkg > 1835 pkg: /lib/firmware/updates/intel/ice/ddp/ice.pkg > 1846 pkg: /lib/firmware/updates/intel/ice/ddp/ice.pkg > dl wo sig > ice_load_pkg_type(): Active package is: 1.3.4.0, ICE OS Default Package > (single VLAN mode) > > > ls -l /usr/lib/firmware/updates/intel/ice/ddp/ > total 624 > -rwxr-xr-x. 1 root root 635256 Jan 18 15:53 ice-1.3.26.0.pkg > lrwxrwxrwx. 1 root root 56 Jan 18 15:55 ice.pkg -> > /usr/lib/firmware/updates/intel/ice/ddp/ice-1.3.26.0.pkg > > > xxd /usr/lib/firmware/updates/intel/ice/ddp/ice.pkg | head -n 17 > 0000000: 0100 0000 0400 0000 1800 0000 6c00 0000 ............l... > 0000010: d000 0000 4031 0000 0100 0000 0100 0000 ....@1.......... > 0000020: 5400 0000 476c 6f62 616c 204d 6574 6164 T...Global Metad > 0000030: 6174 6100 0000 0000 0000 0000 0000 0000 ata............. > 0000040: 0000 0000 0103 1a00 ffff ffff 4943 4520 ............ICE > 0000050: 4f53 2044 6566 6175 6c74 2050 6163 6b61 OS Default Packa > 0000060: 6765 0000 0000 0000 0000 0000 0200 0000 ge.............. > 0000070: 0100 0000 6400 0000 4e6f 7465 7300 0000 ....d...Notes... > 0000080: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > 0000090: 0000 0000 0000 0000 496e 7465 6c28 5229 ........Intel(R) > 00000a0: 2043 6f6e 6669 6775 7261 626c 6520 5069 Configurable Pi > 00000b0: 7065 6c69 6e65 2054 6f6f 6c20 5665 7273 peline Tool Vers > 00000c0: 696f 6e20 322e 312e 302e 3330 0a00 ffff ion 2.1.0.30.... > 00000d0: 0010 0000 0100 0000 7030 0000 5253 412d ........p0..RSA- > 00000e0: 324b 2053 6967 6e61 7475 7265 2053 6567 2K Signature Seg > 00000f0: 6d65 6e74 0000 0000 0000 0000 1000 0000 ment............ > 0000100: 0100 0000 0300 0000 0300 0000 0000 0000 ................ > --000000000000547a2805f2b3f898 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Okay, believe we have tracked down and can explain most of= the current behavior. The question I have now is, is this really what is d= esired or should it be possible for DPDK be passed a flag and load a DDPK f= ile anyways, or should it always try to update the DDP if a newer=C2=A0one = is found.=C2=A0 In our case we are unlikely to utilize the ICE based driver= in the OS and would only be using it with DPDK.=C2=A0 To me, in this scena= rio, it makes little sense to maintain/think about the OS driver and relate= d items.

With some increased logging from testpmd and lo= oking at the ICE PMD source it was identified that we are hitting `ICE_ERR_= AQ_NO_WORK` in `ice_ddp.c -> ice_acquire_global_cfg_lock:2232` which ind= icates that another driver has already written the package or has found tha= t no update was necessary.=C2=A0 My uneducated guess is this is happening a= s the OS starts up and then transitions the=C2=A0device over to vfio-pci wh= ich would also=C2=A0explain=C2=A0why the initial firmware load=C2=A0was res= olved after running dracut.=C2=A0 In the log scenario below, an update shou= ld definitely be necessary still as 1.3.30 is found but 1.3.26 is loaded.

----

1822 pkg: /lib/firmwa= re/updates/intel/ice/ddp/ice-40a6b7ffff8437f8.pkg
1829 pkg: /lib/firmwar= e/intel/ice/ddp/ice-40a6b7ffff8437f8.pkg
1835 pkg: /lib/firmware/updates= /intel/ice/ddp/ice.pkg
1847 pkg: /lib/firmware/updates/intel/ice/ddp/ice= .pkg
ice_load_pkg(): DDP package name: /lib/firmware/updates/intel/ice/d= dp/ice.pkg
init enter
ice_find_seg_in_pkg(): ice 00.0 Package format = version: 1.0.0.0
ice_init_pkg_info(): ice 00.0 Pkg using segment id: 0x0= 0000010
ice_find_seg_in_pkg(): ice 00.0 Package format version: 1.0.0.0<= br>ice_init_pkg_info(): ice 00.0 Pkg: 1.3.30.0, ICE OS Default Package
i= ce_init_pkg_info(): ice 00.0 Ice Seg: 1.0.1.0, ICE Configuration Data
ic= e_find_seg_in_pkg(): ice 00.0 Package format version: 1.0.0.0
ice_clean_= sq(): ice 00.0 ntc 10 head 11.
ice_sq_send_cmd_nolock(): ice 00.0 ATQ: C= ontrol Send queue desc and buffer:
ice_sq_send_cmd_nolock(): ice 00.0 AT= Q: desc and buffer writeback:
dl w sig
ice_download_pkg_with_sig_seg(= ): ice 00.0 Segment ID 16
ice_download_pkg_with_sig_seg(): ice 00.0 Sign= ature type 1
ice_acquire_res(): ice 00.0 ice_acquire_res
ice_aq_req_r= es(): ice 00.0 ice_aq_req_res
ice_clean_sq(): ice 00.0 ntc 11 head 12.ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffe= r:
ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback:ice_acquire_res(): ice 00.0 resource indicates no work to do.
ice_acqui= re_global_cfg_lock(): ice 00.0 Global config lock: No work to do
ice_cle= an_sq(): ice 00.0 ntc 12 head 13.
ice_sq_send_cmd_nolock(): ice 00.0 ATQ= : Control Send queue desc and buffer:
ice_sq_send_cmd_nolock(): ice 00.0= ATQ: desc and buffer writeback:
ice_clean_sq(): ice 00.0 ntc 13 head 14= .
ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and bu= ffer:
ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback:=
ice_clean_sq(): ice 00.0 ntc 14 head 15.
ice_sq_send_cmd_nolock(): i= ce 00.0 ATQ: Control Send queue desc and buffer:
ice_sq_send_cmd_nolock(= ): ice 00.0 ATQ: desc and buffer writeback:
ice_init_pkg(): ice 00.0 pac= kage previously loaded - no work.
ice_clean_sq(): ice 00.0 ntc 15 head 1= 6.
ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and b= uffer:
ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback= :
ice_get_pkg_info(): ice 00.0 Pkg[0]: 1.0.0.17,ICE NVM Package,BMN
i= ce_get_pkg_info(): ice 00.0 Pkg[1]: 1.3.26.0,ICE OS Default Package,A
in= it end
ice_load_pkg_type(): Active package is: 1.3.26.0, ICE OS Default = Package (single VLAN mode)

On Thu, Jan 19, 2023 at 9:14 AM Ben Mag= istro <koncept1@gmail.com> = wrote:
Hello,

We are still trying to track this down b= ut want to reach out to the community and see if there is something obvious= we are missing.=C2=A0 This happens to be a CentOS 7 based host with DPDK 2= 2.11.1 in use.=C2=A0 We have obtained=C2=A0the latest compatible Intel driv= ers and DDP from=C2=A0https://sourceforge.net/projects/e1000/= files/ice%20stable/ , at this time that is version 1.6.7 with DDP 1.3.2= 6.0.=C2=A0 As best as we can tell, testpmd is still loading the OS=C2=A0def= ault DDP 1.3.4.0.=C2=A0 The updated DDP package is located in /usr/lib/firm= ware/updates/intel/ice/ddp/ice.pkg.

I've added= some additional log lines to ice_ethdev.c and ice_ddp.c to try and start d= ebugging.=C2=A0 I've attached what I believe is the relevant informatio= n below.=C2=A0 Quickly jumping out at me is that the pkg_name going into `l= oad_fw` in ice_ethdev.c appears to be the desired path however in `ice_down= load_pkg` says it does not include a signature.=C2=A0 The hex dump from the= desired file appears to show a signature which makes us think this file is= n't actually being loaded/utilized.

We will co= ntinue trying to debug this too but any additional assistance would be grea= tly appreciated.

----

EAL= : Probe PCI driver: net_ice (8086:1593) device: 0000:ca:00.2 (socket 1)
= 1822 pkg: /lib/firmware/updates/intel/ice/ddp/ice-40a6b7ffff843810.pkg
1= 829 pkg: /lib/firmware/intel/ice/ddp/ice-40a6b7ffff843810.pkg
1835 pkg: = /lib/firmware/updates/intel/ice/ddp/ice.pkg
1846 pkg: /lib/firmware/upda= tes/intel/ice/ddp/ice.pkg
dl wo sig
ice_load_pkg_type(): Active packa= ge is: 1.3.4.0, ICE OS Default Package (single VLAN mode)

> ls -l= /usr/lib/firmware/updates/intel/ice/ddp/
total 624
-rwxr-xr-x. 1 roo= t root 635256 Jan 18 15:53 ice-1.3.26.0.pkg
lrwxrwxrwx. 1 root root =C2= =A0 =C2=A0 56 Jan 18 15:55 ice.pkg -> /usr/lib/firmware/updates/intel/ic= e/ddp/ice-1.3.26.0.pkg

> xxd /usr/lib/firmware/updates/intel/ice/= ddp/ice.pkg | head -n 17
0000000: 0100 0000 0400 0000 1800 0000 6c00 000= 0 =C2=A0............l...
0000010: d000 0000 4031 0000 0100 0000 0100 000= 0 =C2=A0....@1..........
0000020: 5400 0000 476c 6f62 616c 204d 6574 616= 4 =C2=A0T...Global Metad
0000030: 6174 6100 0000 0000 0000 0000 0000 000= 0 =C2=A0ata.............
0000040: 0000 0000 0103 1a00 ffff ffff 4943 452= 0 =C2=A0............ICE
0000050: 4f53 2044 6566 6175 6c74 2050 6163 6b6= 1 =C2=A0OS Default Packa
0000060: 6765 0000 0000 0000 0000 0000 0200 000= 0 =C2=A0ge..............
0000070: 0100 0000 6400 0000 4e6f 7465 7300 000= 0 =C2=A0....d...Notes...
0000080: 0000 0000 0000 0000 0000 0000 0000 000= 0 =C2=A0................
0000090: 0000 0000 0000 0000 496e 7465 6c28 522= 9 =C2=A0........Intel(R)
00000a0: 2043 6f6e 6669 6775 7261 626c 6520 506= 9 =C2=A0 Configurable Pi
00000b0: 7065 6c69 6e65 2054 6f6f 6c20 5665 727= 3 =C2=A0peline Tool Vers
00000c0: 696f 6e20 322e 312e 302e 3330 0a00 fff= f =C2=A0ion 2.1.0.30....
00000d0: 0010 0000 0100 0000 7030 0000 5253 412= d =C2=A0........p0..RSA-
00000e0: 324b 2053 6967 6e61 7475 7265 2053 656= 7 =C2=A02K Signature Seg
00000f0: 6d65 6e74 0000 0000 0000 0000 1000 000= 0 =C2=A0ment............
0000100: 0100 0000 0300 0000 0300 0000 0000 000= 0 =C2=A0................
--000000000000547a2805f2b3f898--