Okay, believe we have tracked down and can explain most of the current behavior. The question I have now is, is this really what is desired or should it be possible for DPDK be passed a flag and load a DDPK file anyways, or should it always try to update the DDP if a newer one is found. In our case we are unlikely to utilize the ICE based driver in the OS and would only be using it with DPDK. To me, in this scenario, it makes little sense to maintain/think about the OS driver and related items.
With some increased logging from testpmd and looking at the ICE PMD source it was identified that we are hitting `ICE_ERR_AQ_NO_WORK` in `ice_ddp.c -> ice_acquire_global_cfg_lock:2232` which indicates that another driver has already written the package or has found that no update was necessary. My uneducated guess is this is happening as the OS starts up and then transitions the device over to vfio-pci which would also explain why the initial firmware load was resolved after running dracut. In the log scenario below, an update should definitely be necessary still as 1.3.30 is found but 1.3.26 is loaded.
----
1822 pkg: /lib/firmware/updates/intel/ice/ddp/ice-40a6b7ffff8437f8.pkg
1829 pkg: /lib/firmware/intel/ice/ddp/ice-40a6b7ffff8437f8.pkg
1835 pkg: /lib/firmware/updates/intel/ice/ddp/ice.pkg
1847 pkg: /lib/firmware/updates/intel/ice/ddp/ice.pkg
ice_load_pkg(): DDP package name: /lib/firmware/updates/intel/ice/ddp/ice.pkg
init enter
ice_find_seg_in_pkg(): ice 00.0 Package format version: 1.0.0.0
ice_init_pkg_info(): ice 00.0 Pkg using segment id: 0x00000010
ice_find_seg_in_pkg(): ice 00.0 Package format version: 1.0.0.0
ice_init_pkg_info(): ice 00.0 Pkg: 1.3.30.0, ICE OS Default Package
ice_init_pkg_info(): ice 00.0 Ice Seg: 1.0.1.0, ICE Configuration Data
ice_find_seg_in_pkg(): ice 00.0 Package format version: 1.0.0.0
ice_clean_sq(): ice 00.0 ntc 10 head 11.
ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer:
ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback:
dl w sig
ice_download_pkg_with_sig_seg(): ice 00.0 Segment ID 16
ice_download_pkg_with_sig_seg(): ice 00.0 Signature type 1
ice_acquire_res(): ice 00.0 ice_acquire_res
ice_aq_req_res(): ice 00.0 ice_aq_req_res
ice_clean_sq(): ice 00.0 ntc 11 head 12.
ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer:
ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback:
ice_acquire_res(): ice 00.0 resource indicates no work to do.
ice_acquire_global_cfg_lock(): ice 00.0 Global config lock: No work to do
ice_clean_sq(): ice 00.0 ntc 12 head 13.
ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer:
ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback:
ice_clean_sq(): ice 00.0 ntc 13 head 14.
ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer:
ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback:
ice_clean_sq(): ice 00.0 ntc 14 head 15.
ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer:
ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback:
ice_init_pkg(): ice 00.0 package previously loaded - no work.
ice_clean_sq(): ice 00.0 ntc 15 head 16.
ice_sq_send_cmd_nolock(): ice 00.0 ATQ: Control Send queue desc and buffer:
ice_sq_send_cmd_nolock(): ice 00.0 ATQ: desc and buffer writeback:
ice_get_pkg_info(): ice 00.0 Pkg[0]: 1.0.0.17,ICE NVM Package,BMN
ice_get_pkg_info(): ice 00.0 Pkg[1]: 1.3.26.0,ICE OS Default Package,A
init end
ice_load_pkg_type(): Active package is: 1.3.26.0, ICE OS Default Package (single VLAN mode)