* ICE DDP Load assistance @ 2023-01-19 14:14 Ben Magistro 2023-01-20 15:47 ` Ben Magistro 0 siblings, 1 reply; 4+ messages in thread From: Ben Magistro @ 2023-01-19 14:14 UTC (permalink / raw) To: dev; +Cc: ben.magistro, Stefan Baranoff [-- Attachment #1: Type: text/plain, Size: 3001 bytes --] Hello, We are still trying to track this down but want to reach out to the community and see if there is something obvious we are missing. This happens to be a CentOS 7 based host with DPDK 22.11.1 in use. We have obtained the latest compatible Intel drivers and DDP from https://sourceforge.net/projects/e1000/files/ice%20stable/ , at this time that is version 1.6.7 with DDP 1.3.26.0. As best as we can tell, testpmd is still loading the OS default DDP 1.3.4.0. The updated DDP package is located in /usr/lib/firmware/updates/intel/ice/ddp/ice.pkg. I've added some additional log lines to ice_ethdev.c and ice_ddp.c to try and start debugging. I've attached what I believe is the relevant information below. Quickly jumping out at me is that the pkg_name going into `load_fw` in ice_ethdev.c appears to be the desired path however in `ice_download_pkg` says it does not include a signature. The hex dump from the desired file appears to show a signature which makes us think this file isn't actually being loaded/utilized. We will continue trying to debug this too but any additional assistance would be greatly appreciated. ---- EAL: Probe PCI driver: net_ice (8086:1593) device: 0000:ca:00.2 (socket 1) 1822 pkg: /lib/firmware/updates/intel/ice/ddp/ice-40a6b7ffff843810.pkg 1829 pkg: /lib/firmware/intel/ice/ddp/ice-40a6b7ffff843810.pkg 1835 pkg: /lib/firmware/updates/intel/ice/ddp/ice.pkg 1846 pkg: /lib/firmware/updates/intel/ice/ddp/ice.pkg dl wo sig ice_load_pkg_type(): Active package is: 1.3.4.0, ICE OS Default Package (single VLAN mode) > ls -l /usr/lib/firmware/updates/intel/ice/ddp/ total 624 -rwxr-xr-x. 1 root root 635256 Jan 18 15:53 ice-1.3.26.0.pkg lrwxrwxrwx. 1 root root 56 Jan 18 15:55 ice.pkg -> /usr/lib/firmware/updates/intel/ice/ddp/ice-1.3.26.0.pkg > xxd /usr/lib/firmware/updates/intel/ice/ddp/ice.pkg | head -n 17 0000000: 0100 0000 0400 0000 1800 0000 6c00 0000 ............l... 0000010: d000 0000 4031 0000 0100 0000 0100 0000 ....@1.......... 0000020: 5400 0000 476c 6f62 616c 204d 6574 6164 T...Global Metad 0000030: 6174 6100 0000 0000 0000 0000 0000 0000 ata............. 0000040: 0000 0000 0103 1a00 ffff ffff 4943 4520 ............ICE 0000050: 4f53 2044 6566 6175 6c74 2050 6163 6b61 OS Default Packa 0000060: 6765 0000 0000 0000 0000 0000 0200 0000 ge.............. 0000070: 0100 0000 6400 0000 4e6f 7465 7300 0000 ....d...Notes... 0000080: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000090: 0000 0000 0000 0000 496e 7465 6c28 5229 ........Intel(R) 00000a0: 2043 6f6e 6669 6775 7261 626c 6520 5069 Configurable Pi 00000b0: 7065 6c69 6e65 2054 6f6f 6c20 5665 7273 peline Tool Vers 00000c0: 696f 6e20 322e 312e 302e 3330 0a00 ffff ion 2.1.0.30.... 00000d0: 0010 0000 0100 0000 7030 0000 5253 412d ........p0..RSA- 00000e0: 324b 2053 6967 6e61 7475 7265 2053 6567 2K Signature Seg 00000f0: 6d65 6e74 0000 0000 0000 0000 1000 0000 ment............ 0000100: 0100 0000 0300 0000 0300 0000 0000 0000 ................ [-- Attachment #2: Type: text/html, Size: 3361 bytes --] ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: ICE DDP Load assistance 2023-01-19 14:14 ICE DDP Load assistance Ben Magistro @ 2023-01-20 15:47 ` Ben Magistro 2023-01-24 1:52 ` Zhang, Qi Z 0 siblings, 1 reply; 4+ messages in thread From: Ben Magistro @ 2023-01-20 15:47 UTC (permalink / raw) To: dev; +Cc: ben.magistro, Stefan Baranoff [-- Attachment #1: Type: text/plain, Size: 6783 bytes --] Okay, believe we have tracked down and can explain most of the current behavior. The question I have now is, is this really what is desired or should it be possible for DPDK be passed a flag and load a DDPK file anyways, or should it always try to update the DDP if a newer one is found. In our case we are unlikely to utilize the ICE based driver in the OS and would only be using it with DPDK. To me, in this scenario, it makes little sense to maintain/think about the OS driver and related items. With some increased logging from testpmd and looking at the ICE PMD source it was identified that we are hitting `ICE_ERR_AQ_NO_WORK` in `ice_ddp.c -> ice_acquire_global_cfg_lock:2232` which indicates that another driver has already written the package or has found that no update was necessary. My uneducated guess is this is happening as the OS starts up and then transitions the device over to vfio-pci which would also explain why the initial firmware load was resolved after running dracut. In the log scenario below, an update should definitely be necessary still as 1.3.30 is found but 1.3.26 is loaded. ---- 1822 pkg: /lib/firmware/updates/intel/ice/ddp/ice-40a6b7ffff8437f8.pkg 1829 pkg: /lib/firmware/intel/ice/ddp/ice-40a6b7ffff8437f8.pkg 1835 pkg: /lib/firmware/updates/intel/ice/ddp/ice.pkg 1847 pkg: /lib/firmware/updates/intel/ice/ddp/ice.pkg ice_load_pkg(): DDP package name: /lib/firmware/updates/intel/ice/ddp/ice.pkg init enter ice_find_seg_in_pkg(): ice 00.0 Package format version: 1.0.0.0 ice_init_pkg_info(): ice 00.0 Pkg using segment id: 0x00000010 ice_find_seg_in_pkg(): ice 00.0 Package format version: 1.0.0.0 ice_init_pkg_info(): ice 00.0 Pkg: 1.3.30.0, ICE OS Default Package ice_init_pkg_info(): ice 00.0 Ice Seg: 1.0.1.0, ICE Configuration Data ice_find_seg_in_pkg(): ice 00.0 Package format version: 1.0.0.0 ice_clean_sq(): ice 00.0 ntc 10 head 11. ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer: ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback: dl w sig ice_download_pkg_with_sig_seg(): ice 00.0 Segment ID 16 ice_download_pkg_with_sig_seg(): ice 00.0 Signature type 1 ice_acquire_res(): ice 00.0 ice_acquire_res ice_aq_req_res(): ice 00.0 ice_aq_req_res ice_clean_sq(): ice 00.0 ntc 11 head 12. ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer: ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback: ice_acquire_res(): ice 00.0 resource indicates no work to do. ice_acquire_global_cfg_lock(): ice 00.0 Global config lock: No work to do ice_clean_sq(): ice 00.0 ntc 12 head 13. ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer: ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback: ice_clean_sq(): ice 00.0 ntc 13 head 14. ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer: ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback: ice_clean_sq(): ice 00.0 ntc 14 head 15. ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer: ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback: ice_init_pkg(): ice 00.0 package previously loaded - no work. ice_clean_sq(): ice 00.0 ntc 15 head 16. ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer: ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback: ice_get_pkg_info(): ice 00.0 Pkg[0]: 1.0.0.17,ICE NVM Package,BMN ice_get_pkg_info(): ice 00.0 Pkg[1]: 1.3.26.0,ICE OS Default Package,A init end ice_load_pkg_type(): Active package is: 1.3.26.0, ICE OS Default Package (single VLAN mode) On Thu, Jan 19, 2023 at 9:14 AM Ben Magistro <koncept1@gmail.com> wrote: > Hello, > > We are still trying to track this down but want to reach out to the > community and see if there is something obvious we are missing. This > happens to be a CentOS 7 based host with DPDK 22.11.1 in use. We have > obtained the latest compatible Intel drivers and DDP from > https://sourceforge.net/projects/e1000/files/ice%20stable/ , at this time > that is version 1.6.7 with DDP 1.3.26.0. As best as we can tell, testpmd > is still loading the OS default DDP 1.3.4.0. The updated DDP package is > located in /usr/lib/firmware/updates/intel/ice/ddp/ice.pkg. > > I've added some additional log lines to ice_ethdev.c and ice_ddp.c to try > and start debugging. I've attached what I believe is the relevant > information below. Quickly jumping out at me is that the pkg_name going > into `load_fw` in ice_ethdev.c appears to be the desired path however in > `ice_download_pkg` says it does not include a signature. The hex dump from > the desired file appears to show a signature which makes us think this file > isn't actually being loaded/utilized. > > We will continue trying to debug this too but any additional assistance > would be greatly appreciated. > > ---- > > EAL: Probe PCI driver: net_ice (8086:1593) device: 0000:ca:00.2 (socket 1) > 1822 pkg: /lib/firmware/updates/intel/ice/ddp/ice-40a6b7ffff843810.pkg > 1829 pkg: /lib/firmware/intel/ice/ddp/ice-40a6b7ffff843810.pkg > 1835 pkg: /lib/firmware/updates/intel/ice/ddp/ice.pkg > 1846 pkg: /lib/firmware/updates/intel/ice/ddp/ice.pkg > dl wo sig > ice_load_pkg_type(): Active package is: 1.3.4.0, ICE OS Default Package > (single VLAN mode) > > > ls -l /usr/lib/firmware/updates/intel/ice/ddp/ > total 624 > -rwxr-xr-x. 1 root root 635256 Jan 18 15:53 ice-1.3.26.0.pkg > lrwxrwxrwx. 1 root root 56 Jan 18 15:55 ice.pkg -> > /usr/lib/firmware/updates/intel/ice/ddp/ice-1.3.26.0.pkg > > > xxd /usr/lib/firmware/updates/intel/ice/ddp/ice.pkg | head -n 17 > 0000000: 0100 0000 0400 0000 1800 0000 6c00 0000 ............l... > 0000010: d000 0000 4031 0000 0100 0000 0100 0000 ....@1.......... > 0000020: 5400 0000 476c 6f62 616c 204d 6574 6164 T...Global Metad > 0000030: 6174 6100 0000 0000 0000 0000 0000 0000 ata............. > 0000040: 0000 0000 0103 1a00 ffff ffff 4943 4520 ............ICE > 0000050: 4f53 2044 6566 6175 6c74 2050 6163 6b61 OS Default Packa > 0000060: 6765 0000 0000 0000 0000 0000 0200 0000 ge.............. > 0000070: 0100 0000 6400 0000 4e6f 7465 7300 0000 ....d...Notes... > 0000080: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > 0000090: 0000 0000 0000 0000 496e 7465 6c28 5229 ........Intel(R) > 00000a0: 2043 6f6e 6669 6775 7261 626c 6520 5069 Configurable Pi > 00000b0: 7065 6c69 6e65 2054 6f6f 6c20 5665 7273 peline Tool Vers > 00000c0: 696f 6e20 322e 312e 302e 3330 0a00 ffff ion 2.1.0.30.... > 00000d0: 0010 0000 0100 0000 7030 0000 5253 412d ........p0..RSA- > 00000e0: 324b 2053 6967 6e61 7475 7265 2053 6567 2K Signature Seg > 00000f0: 6d65 6e74 0000 0000 0000 0000 1000 0000 ment............ > 0000100: 0100 0000 0300 0000 0300 0000 0000 0000 ................ > [-- Attachment #2: Type: text/html, Size: 7539 bytes --] ^ permalink raw reply [flat|nested] 4+ messages in thread
* RE: ICE DDP Load assistance 2023-01-20 15:47 ` Ben Magistro @ 2023-01-24 1:52 ` Zhang, Qi Z 2023-02-03 14:43 ` Ben Magistro 0 siblings, 1 reply; 4+ messages in thread From: Zhang, Qi Z @ 2023-01-24 1:52 UTC (permalink / raw) To: Ben Magistro, dev; +Cc: Magistro, Benjamin, Stefan Baranoff [-- Attachment #1: Type: text/plain, Size: 7648 bytes --] Both kernel ice and DPDK ice driver will try to load DDP package from below folder #define ICE_PKG_FILE_DEFAULT "/lib/firmware/intel/ice/ddp/ice.pkg" #define ICE_PKG_FILE_UPDATES "/lib/firmware/updates/intel/ice/ddp/ice.pkg" And only the first PF driver be probed will load DDP package. If the driver found a DDP package already be loaded, it just check if it is compatible with current required version and will not overwrite it. So to upgrade DDP package, we need to make sure no active driver is stilling using the old package, so we may need “rmmod ice” first Regard Qi From: Ben Magistro <koncept1@gmail.com> Sent: Friday, January 20, 2023 11:48 PM To: dev@dpdk.org Cc: Magistro, Benjamin <ben.magistro@trinitycyber.com>; Stefan Baranoff <stefan.baranoff@trinitycyber.com> Subject: Re: ICE DDP Load assistance Okay, believe we have tracked down and can explain most of the current behavior. The question I have now is, is this really what is desired or should it be possible for DPDK be passed a flag and load a DDPK file anyways, or should it always try to update the DDP if a newer one is found. In our case we are unlikely to utilize the ICE based driver in the OS and would only be using it with DPDK. To me, in this scenario, it makes little sense to maintain/think about the OS driver and related items. With some increased logging from testpmd and looking at the ICE PMD source it was identified that we are hitting `ICE_ERR_AQ_NO_WORK` in `ice_ddp.c -> ice_acquire_global_cfg_lock:2232` which indicates that another driver has already written the package or has found that no update was necessary. My uneducated guess is this is happening as the OS starts up and then transitions the device over to vfio-pci which would also explain why the initial firmware load was resolved after running dracut. In the log scenario below, an update should definitely be necessary still as 1.3.30 is found but 1.3.26 is loaded. ---- 1822 pkg: /lib/firmware/updates/intel/ice/ddp/ice-40a6b7ffff8437f8.pkg 1829 pkg: /lib/firmware/intel/ice/ddp/ice-40a6b7ffff8437f8.pkg 1835 pkg: /lib/firmware/updates/intel/ice/ddp/ice.pkg 1847 pkg: /lib/firmware/updates/intel/ice/ddp/ice.pkg ice_load_pkg(): DDP package name: /lib/firmware/updates/intel/ice/ddp/ice.pkg init enter ice_find_seg_in_pkg(): ice 00.0 Package format version: 1.0.0.0 ice_init_pkg_info(): ice 00.0 Pkg using segment id: 0x00000010 ice_find_seg_in_pkg(): ice 00.0 Package format version: 1.0.0.0 ice_init_pkg_info(): ice 00.0 Pkg: 1.3.30.0, ICE OS Default Package ice_init_pkg_info(): ice 00.0 Ice Seg: 1.0.1.0, ICE Configuration Data ice_find_seg_in_pkg(): ice 00.0 Package format version: 1.0.0.0 ice_clean_sq(): ice 00.0 ntc 10 head 11. ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer: ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback: dl w sig ice_download_pkg_with_sig_seg(): ice 00.0 Segment ID 16 ice_download_pkg_with_sig_seg(): ice 00.0 Signature type 1 ice_acquire_res(): ice 00.0 ice_acquire_res ice_aq_req_res(): ice 00.0 ice_aq_req_res ice_clean_sq(): ice 00.0 ntc 11 head 12. ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer: ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback: ice_acquire_res(): ice 00.0 resource indicates no work to do. ice_acquire_global_cfg_lock(): ice 00.0 Global config lock: No work to do ice_clean_sq(): ice 00.0 ntc 12 head 13. ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer: ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback: ice_clean_sq(): ice 00.0 ntc 13 head 14. ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer: ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback: ice_clean_sq(): ice 00.0 ntc 14 head 15. ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer: ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback: ice_init_pkg(): ice 00.0 package previously loaded - no work. ice_clean_sq(): ice 00.0 ntc 15 head 16. ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer: ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback: ice_get_pkg_info(): ice 00.0 Pkg[0]: 1.0.0.17,ICE NVM Package,BMN ice_get_pkg_info(): ice 00.0 Pkg[1]: 1.3.26.0,ICE OS Default Package,A init end ice_load_pkg_type(): Active package is: 1.3.26.0, ICE OS Default Package (single VLAN mode) On Thu, Jan 19, 2023 at 9:14 AM Ben Magistro <koncept1@gmail.com<mailto:koncept1@gmail.com>> wrote: Hello, We are still trying to track this down but want to reach out to the community and see if there is something obvious we are missing. This happens to be a CentOS 7 based host with DPDK 22.11.1 in use. We have obtained the latest compatible Intel drivers and DDP from https://sourceforge.net/projects/e1000/files/ice%20stable/ , at this time that is version 1.6.7 with DDP 1.3.26.0. As best as we can tell, testpmd is still loading the OS default DDP 1.3.4.0. The updated DDP package is located in /usr/lib/firmware/updates/intel/ice/ddp/ice.pkg. I've added some additional log lines to ice_ethdev.c and ice_ddp.c to try and start debugging. I've attached what I believe is the relevant information below. Quickly jumping out at me is that the pkg_name going into `load_fw` in ice_ethdev.c appears to be the desired path however in `ice_download_pkg` says it does not include a signature. The hex dump from the desired file appears to show a signature which makes us think this file isn't actually being loaded/utilized. We will continue trying to debug this too but any additional assistance would be greatly appreciated. ---- EAL: Probe PCI driver: net_ice (8086:1593) device: 0000:ca:00.2 (socket 1) 1822 pkg: /lib/firmware/updates/intel/ice/ddp/ice-40a6b7ffff843810.pkg 1829 pkg: /lib/firmware/intel/ice/ddp/ice-40a6b7ffff843810.pkg 1835 pkg: /lib/firmware/updates/intel/ice/ddp/ice.pkg 1846 pkg: /lib/firmware/updates/intel/ice/ddp/ice.pkg dl wo sig ice_load_pkg_type(): Active package is: 1.3.4.0, ICE OS Default Package (single VLAN mode) > ls -l /usr/lib/firmware/updates/intel/ice/ddp/ total 624 -rwxr-xr-x. 1 root root 635256 Jan 18 15:53 ice-1.3.26.0.pkg lrwxrwxrwx. 1 root root 56 Jan 18 15:55 ice.pkg -> /usr/lib/firmware/updates/intel/ice/ddp/ice-1.3.26.0.pkg > xxd /usr/lib/firmware/updates/intel/ice/ddp/ice.pkg | head -n 17 0000000: 0100 0000 0400 0000 1800 0000 6c00 0000 ............l... 0000010: d000 0000 4031 0000 0100 0000 0100 0000 ....@1<mailto:....@1>.......... 0000020: 5400 0000 476c 6f62 616c 204d 6574 6164 T...Global Metad 0000030: 6174 6100 0000 0000 0000 0000 0000 0000 ata............. 0000040: 0000 0000 0103 1a00 ffff ffff 4943 4520 ............ICE 0000050: 4f53 2044 6566 6175 6c74 2050 6163 6b61 OS Default Packa 0000060: 6765 0000 0000 0000 0000 0000 0200 0000 ge.............. 0000070: 0100 0000 6400 0000 4e6f 7465 7300 0000 ....d...Notes... 0000080: 0000 0000 0000 0000 0000 0000 0000 0000 ................ 0000090: 0000 0000 0000 0000 496e 7465 6c28 5229 ........Intel(R) 00000a0: 2043 6f6e 6669 6775 7261 626c 6520 5069 Configurable Pi 00000b0: 7065 6c69 6e65 2054 6f6f 6c20 5665 7273 peline Tool Vers 00000c0: 696f 6e20 322e 312e 302e 3330 0a00 ffff ion 2.1.0.30.... 00000d0: 0010 0000 0100 0000 7030 0000 5253 412d ........p0..RSA- 00000e0: 324b 2053 6967 6e61 7475 7265 2053 6567 2K Signature Seg 00000f0: 6d65 6e74 0000 0000 0000 0000 1000 0000 ment............ 0000100: 0100 0000 0300 0000 0300 0000 0000 0000 ................ [-- Attachment #2: Type: text/html, Size: 13358 bytes --] ^ permalink raw reply [flat|nested] 4+ messages in thread
* Re: ICE DDP Load assistance 2023-01-24 1:52 ` Zhang, Qi Z @ 2023-02-03 14:43 ` Ben Magistro 0 siblings, 0 replies; 4+ messages in thread From: Ben Magistro @ 2023-02-03 14:43 UTC (permalink / raw) To: Zhang, Qi Z; +Cc: dev, Magistro, Benjamin, Stefan Baranoff [-- Attachment #1: Type: text/plain, Size: 8417 bytes --] Just to close the loop on this, most of our nics have been dual port. When looking at this I forgot/didn't realize this one was a quad port nic so some ports were still controlled by the kernel which explains the observed behavior. Also thanks for the explanation around an existing DDP being found. On Mon, Jan 23, 2023 at 8:52 PM Zhang, Qi Z <qi.z.zhang@intel.com> wrote: > Both kernel ice and DPDK ice driver will try to load DDP package from > below folder > > #define ICE_PKG_FILE_DEFAULT "/lib/firmware/intel/ice/ddp/ice.pkg" > > #define ICE_PKG_FILE_UPDATES "/lib/firmware/updates/intel/ice/ddp/ice.pkg" > > > > And only the first PF driver be probed will load DDP package. > > If the driver found a DDP package already be loaded, it just check if it > is compatible with current required version and will not overwrite it. > > > > So to upgrade DDP package, we need to make sure no active driver is > stilling using the old package, so we may need “rmmod ice” first > > > > Regard > > Qi > > > > *From:* Ben Magistro <koncept1@gmail.com> > *Sent:* Friday, January 20, 2023 11:48 PM > *To:* dev@dpdk.org > *Cc:* Magistro, Benjamin <ben.magistro@trinitycyber.com>; Stefan Baranoff > <stefan.baranoff@trinitycyber.com> > *Subject:* Re: ICE DDP Load assistance > > > > Okay, believe we have tracked down and can explain most of the current > behavior. The question I have now is, is this really what is desired or > should it be possible for DPDK be passed a flag and load a DDPK file > anyways, or should it always try to update the DDP if a newer one is > found. In our case we are unlikely to utilize the ICE based driver in the > OS and would only be using it with DPDK. To me, in this scenario, it makes > little sense to maintain/think about the OS driver and related items. > > > > With some increased logging from testpmd and looking at the ICE PMD source > it was identified that we are hitting `ICE_ERR_AQ_NO_WORK` in `ice_ddp.c -> > ice_acquire_global_cfg_lock:2232` which indicates that another driver has > already written the package or has found that no update was necessary. My > uneducated guess is this is happening as the OS starts up and then > transitions the device over to vfio-pci which would also explain why the > initial firmware load was resolved after running dracut. In the log > scenario below, an update should definitely be necessary still as 1.3.30 is > found but 1.3.26 is loaded. > > > > ---- > > > > 1822 pkg: /lib/firmware/updates/intel/ice/ddp/ice-40a6b7ffff8437f8.pkg > 1829 pkg: /lib/firmware/intel/ice/ddp/ice-40a6b7ffff8437f8.pkg > 1835 pkg: /lib/firmware/updates/intel/ice/ddp/ice.pkg > 1847 pkg: /lib/firmware/updates/intel/ice/ddp/ice.pkg > ice_load_pkg(): DDP package name: > /lib/firmware/updates/intel/ice/ddp/ice.pkg > init enter > ice_find_seg_in_pkg(): ice 00.0 Package format version: 1.0.0.0 > ice_init_pkg_info(): ice 00.0 Pkg using segment id: 0x00000010 > ice_find_seg_in_pkg(): ice 00.0 Package format version: 1.0.0.0 > ice_init_pkg_info(): ice 00.0 Pkg: 1.3.30.0, ICE OS Default Package > ice_init_pkg_info(): ice 00.0 Ice Seg: 1.0.1.0, ICE Configuration Data > ice_find_seg_in_pkg(): ice 00.0 Package format version: 1.0.0.0 > ice_clean_sq(): ice 00.0 ntc 10 head 11. > ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer: > ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback: > dl w sig > ice_download_pkg_with_sig_seg(): ice 00.0 Segment ID 16 > ice_download_pkg_with_sig_seg(): ice 00.0 Signature type 1 > ice_acquire_res(): ice 00.0 ice_acquire_res > ice_aq_req_res(): ice 00.0 ice_aq_req_res > ice_clean_sq(): ice 00.0 ntc 11 head 12. > ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer: > ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback: > ice_acquire_res(): ice 00.0 resource indicates no work to do. > ice_acquire_global_cfg_lock(): ice 00.0 Global config lock: No work to do > ice_clean_sq(): ice 00.0 ntc 12 head 13. > ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer: > ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback: > ice_clean_sq(): ice 00.0 ntc 13 head 14. > ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer: > ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback: > ice_clean_sq(): ice 00.0 ntc 14 head 15. > ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer: > ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback: > ice_init_pkg(): ice 00.0 package previously loaded - no work. > ice_clean_sq(): ice 00.0 ntc 15 head 16. > ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer: > ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback: > ice_get_pkg_info(): ice 00.0 Pkg[0]: 1.0.0.17,ICE NVM Package,BMN > ice_get_pkg_info(): ice 00.0 Pkg[1]: 1.3.26.0,ICE OS Default Package,A > init end > ice_load_pkg_type(): Active package is: 1.3.26.0, ICE OS Default Package > (single VLAN mode) > > > > On Thu, Jan 19, 2023 at 9:14 AM Ben Magistro <koncept1@gmail.com> wrote: > > Hello, > > > > We are still trying to track this down but want to reach out to the > community and see if there is something obvious we are missing. This > happens to be a CentOS 7 based host with DPDK 22.11.1 in use. We have > obtained the latest compatible Intel drivers and DDP from > https://sourceforge.net/projects/e1000/files/ice%20stable/ , at this time > that is version 1.6.7 with DDP 1.3.26.0. As best as we can tell, testpmd > is still loading the OS default DDP 1.3.4.0. The updated DDP package is > located in /usr/lib/firmware/updates/intel/ice/ddp/ice.pkg. > > > > I've added some additional log lines to ice_ethdev.c and ice_ddp.c to try > and start debugging. I've attached what I believe is the relevant > information below. Quickly jumping out at me is that the pkg_name going > into `load_fw` in ice_ethdev.c appears to be the desired path however in > `ice_download_pkg` says it does not include a signature. The hex dump from > the desired file appears to show a signature which makes us think this file > isn't actually being loaded/utilized. > > > > We will continue trying to debug this too but any additional assistance > would be greatly appreciated. > > > > ---- > > > > EAL: Probe PCI driver: net_ice (8086:1593) device: 0000:ca:00.2 (socket 1) > 1822 pkg: /lib/firmware/updates/intel/ice/ddp/ice-40a6b7ffff843810.pkg > 1829 pkg: /lib/firmware/intel/ice/ddp/ice-40a6b7ffff843810.pkg > 1835 pkg: /lib/firmware/updates/intel/ice/ddp/ice.pkg > 1846 pkg: /lib/firmware/updates/intel/ice/ddp/ice.pkg > dl wo sig > ice_load_pkg_type(): Active package is: 1.3.4.0, ICE OS Default Package > (single VLAN mode) > > > ls -l /usr/lib/firmware/updates/intel/ice/ddp/ > total 624 > -rwxr-xr-x. 1 root root 635256 Jan 18 15:53 ice-1.3.26.0.pkg > lrwxrwxrwx. 1 root root 56 Jan 18 15:55 ice.pkg -> > /usr/lib/firmware/updates/intel/ice/ddp/ice-1.3.26.0.pkg > > > xxd /usr/lib/firmware/updates/intel/ice/ddp/ice.pkg | head -n 17 > 0000000: 0100 0000 0400 0000 1800 0000 6c00 0000 ............l... > 0000010: d000 0000 4031 0000 0100 0000 0100 0000 ....@1.......... > 0000020: 5400 0000 476c 6f62 616c 204d 6574 6164 T...Global Metad > 0000030: 6174 6100 0000 0000 0000 0000 0000 0000 ata............. > 0000040: 0000 0000 0103 1a00 ffff ffff 4943 4520 ............ICE > 0000050: 4f53 2044 6566 6175 6c74 2050 6163 6b61 OS Default Packa > 0000060: 6765 0000 0000 0000 0000 0000 0200 0000 ge.............. > 0000070: 0100 0000 6400 0000 4e6f 7465 7300 0000 ....d...Notes... > 0000080: 0000 0000 0000 0000 0000 0000 0000 0000 ................ > 0000090: 0000 0000 0000 0000 496e 7465 6c28 5229 ........Intel(R) > 00000a0: 2043 6f6e 6669 6775 7261 626c 6520 5069 Configurable Pi > 00000b0: 7065 6c69 6e65 2054 6f6f 6c20 5665 7273 peline Tool Vers > 00000c0: 696f 6e20 322e 312e 302e 3330 0a00 ffff ion 2.1.0.30.... > 00000d0: 0010 0000 0100 0000 7030 0000 5253 412d ........p0..RSA- > 00000e0: 324b 2053 6967 6e61 7475 7265 2053 6567 2K Signature Seg > 00000f0: 6d65 6e74 0000 0000 0000 0000 1000 0000 ment............ > 0000100: 0100 0000 0300 0000 0300 0000 0000 0000 ................ > > [-- Attachment #2: Type: text/html, Size: 11459 bytes --] ^ permalink raw reply [flat|nested] 4+ messages in thread
end of thread, other threads:[~2023-02-03 14:43 UTC | newest] Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2023-01-19 14:14 ICE DDP Load assistance Ben Magistro 2023-01-20 15:47 ` Ben Magistro 2023-01-24 1:52 ` Zhang, Qi Z 2023-02-03 14:43 ` Ben Magistro
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).