[PATCH v2] eal/unix: support ZSTD compression for firmware

[PATCH v2] eal/unix: support ZSTD compression for firmware

[David Marchand]

Ubuntu 24.04 started to compress firmware files with ZSTD compression.

Bugzilla ID: 1437
Cc: stable@dpdk.org

Signed-off-by: David Marchand <david.marchand@redhat.com>
---
Chances since v1:
- fixed link issue when libarchive is not available,
- Cc'd stable@dpdk.org,

---
 lib/eal/unix/eal_firmware.c | 24 ++++++++++++++++++------
 1 file changed, 18 insertions(+), 6 deletions(-)

diff --git a/lib/eal/unix/eal_firmware.c b/lib/eal/unix/eal_firmware.c
index 1d47e879c8..31de027598 100644
--- a/lib/eal/unix/eal_firmware.c
+++ b/lib/eal/unix/eal_firmware.c
@@ -16,6 +16,8 @@
 #include "eal_firmware.h"
 #include "eal_private.h"
 
+static const char * const compression_algorithms[] = { "xz", "zst" };
+
 #ifdef RTE_HAS_LIBARCHIVE
 
 struct firmware_read_ctx {
@@ -37,7 +39,11 @@ firmware_open(struct firmware_read_ctx *ctx, const char *name, size_t blocksize)
 
 	err = archive_read_support_filter_xz(ctx->a);
 	if (err != ARCHIVE_OK && err != ARCHIVE_WARN)
-		goto error;
+		EAL_LOG(DEBUG, "could not initialise libarchive for xz compression");
+
+	err = archive_read_support_filter_zstd(ctx->a);
+	if (err != ARCHIVE_OK && err != ARCHIVE_WARN)
+		EAL_LOG(DEBUG, "could not initialise libarchive for zstd compression");
 
 	if (archive_read_open_filename(ctx->a, name, blocksize) != ARCHIVE_OK)
 		goto error;
@@ -148,16 +154,22 @@ rte_firmware_read(const char *name, void **buf, size_t *bufsz)
 
 	ret = firmware_read(name, buf, bufsz);
 	if (ret < 0) {
-		snprintf(path, sizeof(path), "%s.xz", name);
-		path[PATH_MAX - 1] = '\0';
+		unsigned int i;
+
+		for (i = 0; i < RTE_DIM(compression_algorithms); i++) {
+			snprintf(path, sizeof(path), "%s.%s", name,
+				compression_algorithms[i]);
+			path[PATH_MAX - 1] = '\0';
+			if (access(path, F_OK) != 0)
+				continue;
 #ifndef RTE_HAS_LIBARCHIVE
-		if (access(path, F_OK) == 0) {
 			EAL_LOG(WARNING, "libarchive not linked, %s cannot be decompressed",
 				path);
-		}
 #else
-		ret = firmware_read(path, buf, bufsz);
+			ret = firmware_read(path, buf, bufsz);
 #endif
+			break;
+		}
 	}
 	return ret;
 }
-- 
2.44.0

Re: [PATCH v2] eal/unix: support ZSTD compression for firmware

[Bruce Richardson]

On Mon, May 13, 2024 at 01:12:04PM +0200, David Marchand wrote:
> Ubuntu 24.04 started to compress firmware files with ZSTD compression.
> 
> Bugzilla ID: 1437
> Cc: stable@dpdk.org
> 
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> Chances since v1:
> - fixed link issue when libarchive is not available,
> - Cc'd stable@dpdk.org,
> 
> ---
>  lib/eal/unix/eal_firmware.c | 24 ++++++++++++++++++------
>  1 file changed, 18 insertions(+), 6 deletions(-)
> 

Acked-by: Bruce Richardson <bruce.richardson@intel.com>

Verified that port initializes ok with zstd compressed firmware, and we get
suitable error if libarchive is not installed.

Tested-by: Bruce Richardson <bruce.richardson@intel.com>

One minor suggestion below.

> diff --git a/lib/eal/unix/eal_firmware.c b/lib/eal/unix/eal_firmware.c
> index 1d47e879c8..31de027598 100644
> --- a/lib/eal/unix/eal_firmware.c
> +++ b/lib/eal/unix/eal_firmware.c
> @@ -16,6 +16,8 @@
>  #include "eal_firmware.h"
>  #include "eal_private.h"
>  
> +static const char * const compression_algorithms[] = { "xz", "zst" };
Very minor nit: these are the suffixes used to find the files, more than
they are the compression algorithms. There I'd suggest renaming to
"compression_suffixes".
[Perhaps just fix on apply if taking this suggestion]

> +
>  #ifdef RTE_HAS_LIBARCHIVE
>  
>  struct firmware_read_ctx {
<snip>

Re: [PATCH v2] eal/unix: support ZSTD compression for firmware

[David Marchand]

Hello Bruce,

On Mon, May 13, 2024 at 1:37 PM Bruce Richardson
<bruce.richardson@intel.com> wrote:
> > @@ -16,6 +16,8 @@
> >  #include "eal_firmware.h"
> >  #include "eal_private.h"
> >
> > +static const char * const compression_algorithms[] = { "xz", "zst" };
> Very minor nit: these are the suffixes used to find the files, more than
> they are the compression algorithms. There I'd suggest renaming to
> "compression_suffixes".
> [Perhaps just fix on apply if taking this suggestion]

I'll fix when applying.
Thanks for the review and tests.


-- 
David Marchand

Re: [PATCH v2] eal/unix: support ZSTD compression for firmware

[David Marchand]

On Mon, May 13, 2024 at 1:37 PM Bruce Richardson
<bruce.richardson@intel.com> wrote:
> On Mon, May 13, 2024 at 01:12:04PM +0200, David Marchand wrote:
> > Ubuntu 24.04 started to compress firmware files with ZSTD compression.
> >
> > Bugzilla ID: 1437
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> Acked-by: Bruce Richardson <bruce.richardson@intel.com>

Applied with additional nit from Bruce on variable name.
Thanks.


-- 
David Marchand

Re: [PATCH v2] eal/unix: support ZSTD compression for firmware

[Xueming Li]

[-- Attachment #1: Type: text/plain, Size: 3081 bytes --]

Hi David,

I get unix compilation failure when backporting this patch to 23.11 LTS, the EAL_LOG macro is not backported to 23.11 LTS, any suggestion?

Regards,
Xueming

________________________________
From: David Marchand <david.marchand@redhat.com>
Sent: Monday, May 13, 2024 7:12 PM
To: dev@dpdk.org <dev@dpdk.org>
Cc: Yu Jiang <yux.jiang@intel.com>; Bruce Richardson <bruce.richardson@intel.com>; stable@dpdk.org <stable@dpdk.org>
Subject: [PATCH v2] eal/unix: support ZSTD compression for firmware

Ubuntu 24.04 started to compress firmware files with ZSTD compression.

Bugzilla ID: 1437
Cc: stable@dpdk.org

Signed-off-by: David Marchand <david.marchand@redhat.com>
---
Chances since v1:
- fixed link issue when libarchive is not available,
- Cc'd stable@dpdk.org,

---
 lib/eal/unix/eal_firmware.c | 24 ++++++++++++++++++------
 1 file changed, 18 insertions(+), 6 deletions(-)

diff --git a/lib/eal/unix/eal_firmware.c b/lib/eal/unix/eal_firmware.c
index 1d47e879c8..31de027598 100644
--- a/lib/eal/unix/eal_firmware.c
+++ b/lib/eal/unix/eal_firmware.c
@@ -16,6 +16,8 @@
 #include "eal_firmware.h"
 #include "eal_private.h"

+static const char * const compression_algorithms[] = { "xz", "zst" };
+
 #ifdef RTE_HAS_LIBARCHIVE

 struct firmware_read_ctx {
@@ -37,7 +39,11 @@ firmware_open(struct firmware_read_ctx *ctx, const char *name, size_t blocksize)

         err = archive_read_support_filter_xz(ctx->a);
         if (err != ARCHIVE_OK && err != ARCHIVE_WARN)
-               goto error;
+               EAL_LOG(DEBUG, "could not initialise libarchive for xz compression");
+
+       err = archive_read_support_filter_zstd(ctx->a);
+       if (err != ARCHIVE_OK && err != ARCHIVE_WARN)
+               EAL_LOG(DEBUG, "could not initialise libarchive for zstd compression");

         if (archive_read_open_filename(ctx->a, name, blocksize) != ARCHIVE_OK)
                 goto error;
@@ -148,16 +154,22 @@ rte_firmware_read(const char *name, void **buf, size_t *bufsz)

         ret = firmware_read(name, buf, bufsz);
         if (ret < 0) {
-               snprintf(path, sizeof(path), "%s.xz", name);
-               path[PATH_MAX - 1] = '\0';
+               unsigned int i;
+
+               for (i = 0; i < RTE_DIM(compression_algorithms); i++) {
+                       snprintf(path, sizeof(path), "%s.%s", name,
+                               compression_algorithms[i]);
+                       path[PATH_MAX - 1] = '\0';
+                       if (access(path, F_OK) != 0)
+                               continue;
 #ifndef RTE_HAS_LIBARCHIVE
-               if (access(path, F_OK) == 0) {
                         EAL_LOG(WARNING, "libarchive not linked, %s cannot be decompressed",
                                 path);
-               }
 #else
-               ret = firmware_read(path, buf, bufsz);
+                       ret = firmware_read(path, buf, bufsz);
 #endif
+                       break;
+               }
         }
         return ret;
 }
--
2.44.0


[-- Attachment #2: Type: text/html, Size: 7374 bytes --]

Re: [PATCH v2] eal/unix: support ZSTD compression for firmware

[David Marchand]

On Wed, Jul 3, 2024 at 9:12 AM Xueming Li <xuemingl@nvidia.com> wrote:
>
> Hi David,
>
> I get unix compilation failure when backporting this patch to 23.11 LTS, the EAL_LOG macro is not backported to 23.11 LTS, any suggestion?

Like other logging calls in this same file, you can replace
EAL_LOG(level, fmt, args); with a simple RTE_LOG(level, EAL, fmt "\n",
args);
Am I missing something?


-- 
David Marchand

Re: [PATCH v2] eal/unix: support ZSTD compression for firmware

[Xueming Li]

[-- Attachment #1: Type: text/plain, Size: 871 bytes --]

Thanks, I tried but missed the "EAL" part :-)

________________________________
From: David Marchand <david.marchand@redhat.com>
Sent: Wednesday, July 3, 2024 3:27 PM
To: Xueming Li <xuemingl@nvidia.com>
Cc: dev@dpdk.org <dev@dpdk.org>; Yu Jiang <yux.jiang@intel.com>; Bruce Richardson <bruce.richardson@intel.com>; stable@dpdk.org <stable@dpdk.org>
Subject: Re: [PATCH v2] eal/unix: support ZSTD compression for firmware

On Wed, Jul 3, 2024 at 9:12 AM Xueming Li <xuemingl@nvidia.com> wrote:
>
> Hi David,
>
> I get unix compilation failure when backporting this patch to 23.11 LTS, the EAL_LOG macro is not backported to 23.11 LTS, any suggestion?

Like other logging calls in this same file, you can replace
EAL_LOG(level, fmt, args); with a simple RTE_LOG(level, EAL, fmt "\n",
args);
Am I missing something?
--------


--
David Marchand


[-- Attachment #2: Type: text/html, Size: 2158 bytes --]