Re: ICE DDP Load assistance

[Ben Magistro <koncept1@gmail.com>]

[-- Attachment #1: Type: text/plain, Size: 8417 bytes --]

Just to close the loop on this, most of our nics have been dual port.  When
looking at this I forgot/didn't realize this one was a quad port nic so
some ports were still controlled by the kernel which explains the observed
behavior.  Also thanks for the explanation around an existing DDP being
found.

On Mon, Jan 23, 2023 at 8:52 PM Zhang, Qi Z <qi.z.zhang@intel.com> wrote:

> Both kernel ice and DPDK ice driver will try to load DDP package from
> below folder
>
> #define ICE_PKG_FILE_DEFAULT "/lib/firmware/intel/ice/ddp/ice.pkg"
>
> #define ICE_PKG_FILE_UPDATES "/lib/firmware/updates/intel/ice/ddp/ice.pkg"
>
>
>
> And only the first PF driver be probed will load DDP package.
>
> If the driver found a DDP package already be loaded, it just check if it
> is compatible with current required version and will not overwrite it.
>
>
>
> So to upgrade DDP package, we need to make sure no active driver is
> stilling using the old package, so we may need “rmmod ice” first
>
>
>
> Regard
>
> Qi
>
>
>
> *From:* Ben Magistro <koncept1@gmail.com>
> *Sent:* Friday, January 20, 2023 11:48 PM
> *To:* dev@dpdk.org
> *Cc:* Magistro, Benjamin <ben.magistro@trinitycyber.com>; Stefan Baranoff
> <stefan.baranoff@trinitycyber.com>
> *Subject:* Re: ICE DDP Load assistance
>
>
>
> Okay, believe we have tracked down and can explain most of the current
> behavior. The question I have now is, is this really what is desired or
> should it be possible for DPDK be passed a flag and load a DDPK file
> anyways, or should it always try to update the DDP if a newer one is
> found.  In our case we are unlikely to utilize the ICE based driver in the
> OS and would only be using it with DPDK.  To me, in this scenario, it makes
> little sense to maintain/think about the OS driver and related items.
>
>
>
> With some increased logging from testpmd and looking at the ICE PMD source
> it was identified that we are hitting `ICE_ERR_AQ_NO_WORK` in `ice_ddp.c ->
> ice_acquire_global_cfg_lock:2232` which indicates that another driver has
> already written the package or has found that no update was necessary.  My
> uneducated guess is this is happening as the OS starts up and then
> transitions the device over to vfio-pci which would also explain why the
> initial firmware load was resolved after running dracut.  In the log
> scenario below, an update should definitely be necessary still as 1.3.30 is
> found but 1.3.26 is loaded.
>
>
>
> ----
>
>
>
> 1822 pkg: /lib/firmware/updates/intel/ice/ddp/ice-40a6b7ffff8437f8.pkg
> 1829 pkg: /lib/firmware/intel/ice/ddp/ice-40a6b7ffff8437f8.pkg
> 1835 pkg: /lib/firmware/updates/intel/ice/ddp/ice.pkg
> 1847 pkg: /lib/firmware/updates/intel/ice/ddp/ice.pkg
> ice_load_pkg(): DDP package name:
> /lib/firmware/updates/intel/ice/ddp/ice.pkg
> init enter
> ice_find_seg_in_pkg(): ice 00.0 Package format version: 1.0.0.0
> ice_init_pkg_info(): ice 00.0 Pkg using segment id: 0x00000010
> ice_find_seg_in_pkg(): ice 00.0 Package format version: 1.0.0.0
> ice_init_pkg_info(): ice 00.0 Pkg: 1.3.30.0, ICE OS Default Package
> ice_init_pkg_info(): ice 00.0 Ice Seg: 1.0.1.0, ICE Configuration Data
> ice_find_seg_in_pkg(): ice 00.0 Package format version: 1.0.0.0
> ice_clean_sq(): ice 00.0 ntc 10 head 11.
> ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer:
> ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback:
> dl w sig
> ice_download_pkg_with_sig_seg(): ice 00.0 Segment ID 16
> ice_download_pkg_with_sig_seg(): ice 00.0 Signature type 1
> ice_acquire_res(): ice 00.0 ice_acquire_res
> ice_aq_req_res(): ice 00.0 ice_aq_req_res
> ice_clean_sq(): ice 00.0 ntc 11 head 12.
> ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer:
> ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback:
> ice_acquire_res(): ice 00.0 resource indicates no work to do.
> ice_acquire_global_cfg_lock(): ice 00.0 Global config lock: No work to do
> ice_clean_sq(): ice 00.0 ntc 12 head 13.
> ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer:
> ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback:
> ice_clean_sq(): ice 00.0 ntc 13 head 14.
> ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer:
> ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback:
> ice_clean_sq(): ice 00.0 ntc 14 head 15.
> ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer:
> ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback:
> ice_init_pkg(): ice 00.0 package previously loaded - no work.
> ice_clean_sq(): ice 00.0 ntc 15 head 16.
> ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer:
> ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback:
> ice_get_pkg_info(): ice 00.0 Pkg[0]: 1.0.0.17,ICE NVM Package,BMN
> ice_get_pkg_info(): ice 00.0 Pkg[1]: 1.3.26.0,ICE OS Default Package,A
> init end
> ice_load_pkg_type(): Active package is: 1.3.26.0, ICE OS Default Package
> (single VLAN mode)
>
>
>
> On Thu, Jan 19, 2023 at 9:14 AM Ben Magistro <koncept1@gmail.com> wrote:
>
> Hello,
>
>
>
> We are still trying to track this down but want to reach out to the
> community and see if there is something obvious we are missing.  This
> happens to be a CentOS 7 based host with DPDK 22.11.1 in use.  We have
> obtained the latest compatible Intel drivers and DDP from
> https://sourceforge.net/projects/e1000/files/ice%20stable/ , at this time
> that is version 1.6.7 with DDP 1.3.26.0.  As best as we can tell, testpmd
> is still loading the OS default DDP 1.3.4.0.  The updated DDP package is
> located in /usr/lib/firmware/updates/intel/ice/ddp/ice.pkg.
>
>
>
> I've added some additional log lines to ice_ethdev.c and ice_ddp.c to try
> and start debugging.  I've attached what I believe is the relevant
> information below.  Quickly jumping out at me is that the pkg_name going
> into `load_fw` in ice_ethdev.c appears to be the desired path however in
> `ice_download_pkg` says it does not include a signature.  The hex dump from
> the desired file appears to show a signature which makes us think this file
> isn't actually being loaded/utilized.
>
>
>
> We will continue trying to debug this too but any additional assistance
> would be greatly appreciated.
>
>
>
> ----
>
>
>
> EAL: Probe PCI driver: net_ice (8086:1593) device: 0000:ca:00.2 (socket 1)
> 1822 pkg: /lib/firmware/updates/intel/ice/ddp/ice-40a6b7ffff843810.pkg
> 1829 pkg: /lib/firmware/intel/ice/ddp/ice-40a6b7ffff843810.pkg
> 1835 pkg: /lib/firmware/updates/intel/ice/ddp/ice.pkg
> 1846 pkg: /lib/firmware/updates/intel/ice/ddp/ice.pkg
> dl wo sig
> ice_load_pkg_type(): Active package is: 1.3.4.0, ICE OS Default Package
> (single VLAN mode)
>
> > ls -l /usr/lib/firmware/updates/intel/ice/ddp/
> total 624
> -rwxr-xr-x. 1 root root 635256 Jan 18 15:53 ice-1.3.26.0.pkg
> lrwxrwxrwx. 1 root root     56 Jan 18 15:55 ice.pkg ->
> /usr/lib/firmware/updates/intel/ice/ddp/ice-1.3.26.0.pkg
>
> > xxd /usr/lib/firmware/updates/intel/ice/ddp/ice.pkg | head -n 17
> 0000000: 0100 0000 0400 0000 1800 0000 6c00 0000  ............l...
> 0000010: d000 0000 4031 0000 0100 0000 0100 0000  ....@1..........
> 0000020: 5400 0000 476c 6f62 616c 204d 6574 6164  T...Global Metad
> 0000030: 6174 6100 0000 0000 0000 0000 0000 0000  ata.............
> 0000040: 0000 0000 0103 1a00 ffff ffff 4943 4520  ............ICE
> 0000050: 4f53 2044 6566 6175 6c74 2050 6163 6b61  OS Default Packa
> 0000060: 6765 0000 0000 0000 0000 0000 0200 0000  ge..............
> 0000070: 0100 0000 6400 0000 4e6f 7465 7300 0000  ....d...Notes...
> 0000080: 0000 0000 0000 0000 0000 0000 0000 0000  ................
> 0000090: 0000 0000 0000 0000 496e 7465 6c28 5229  ........Intel(R)
> 00000a0: 2043 6f6e 6669 6775 7261 626c 6520 5069   Configurable Pi
> 00000b0: 7065 6c69 6e65 2054 6f6f 6c20 5665 7273  peline Tool Vers
> 00000c0: 696f 6e20 322e 312e 302e 3330 0a00 ffff  ion 2.1.0.30....
> 00000d0: 0010 0000 0100 0000 7030 0000 5253 412d  ........p0..RSA-
> 00000e0: 324b 2053 6967 6e61 7475 7265 2053 6567  2K Signature Seg
> 00000f0: 6d65 6e74 0000 0000 0000 0000 1000 0000  ment............
> 0000100: 0100 0000 0300 0000 0300 0000 0000 0000  ................
>
>

[-- Attachment #2: Type: text/html, Size: 11459 bytes --]