[PATCH] eal/unix: support ZSTD compression for firmwares

[PATCH] eal/unix: support ZSTD compression for firmwares

[David Marchand]

Ubuntu 24.04 started to compress firmwares with ZSTD compression.

Bugzilla ID: 1437

Signed-off-by: David Marchand <david.marchand@redhat.com>
---
 lib/eal/unix/eal_firmware.c | 42 +++++++++++++++++++++++++++++--------
 1 file changed, 33 insertions(+), 9 deletions(-)

diff --git a/lib/eal/unix/eal_firmware.c b/lib/eal/unix/eal_firmware.c
index 1d47e879c8..065e251f9d 100644
--- a/lib/eal/unix/eal_firmware.c
+++ b/lib/eal/unix/eal_firmware.c
@@ -16,6 +16,21 @@
 #include "eal_firmware.h"
 #include "eal_private.h"
 
+#ifndef RTE_HAS_LIBARCHIVE
+/* Fake definitions for the compression_algorithms array below. */
+struct archive;
+extern int archive_read_support_filter_xz(struct archive *a);
+extern int archive_read_support_filter_zstd(struct archive *a);
+#endif
+
+static struct {
+	const char *suffix;
+	int (*support_callback)(struct archive *a);
+} compression_algorithms[] = {
+	{ "xz", archive_read_support_filter_xz, },
+	{ "zst", archive_read_support_filter_zstd, },
+};
+
 #ifdef RTE_HAS_LIBARCHIVE
 
 struct firmware_read_ctx {
@@ -26,7 +41,7 @@ static int
 firmware_open(struct firmware_read_ctx *ctx, const char *name, size_t blocksize)
 {
 	struct archive_entry *e;
-	int err;
+	unsigned int i;
 
 	ctx->a = archive_read_new();
 	if (ctx->a == NULL)
@@ -35,9 +50,12 @@ firmware_open(struct firmware_read_ctx *ctx, const char *name, size_t blocksize)
 	if (archive_read_support_format_raw(ctx->a) != ARCHIVE_OK)
 		goto error;
 
-	err = archive_read_support_filter_xz(ctx->a);
-	if (err != ARCHIVE_OK && err != ARCHIVE_WARN)
-		goto error;
+	for (i = 0; i < RTE_DIM(compression_algorithms); i++) {
+		int err = compression_algorithms[i].support_callback(ctx->a);
+		if (err != ARCHIVE_OK && err != ARCHIVE_WARN)
+			EAL_LOG(WARNING, "could not initialise libarchive for %s compression",
+				compression_algorithms[i].suffix);
+	}
 
 	if (archive_read_open_filename(ctx->a, name, blocksize) != ARCHIVE_OK)
 		goto error;
@@ -148,16 +166,22 @@ rte_firmware_read(const char *name, void **buf, size_t *bufsz)
 
 	ret = firmware_read(name, buf, bufsz);
 	if (ret < 0) {
-		snprintf(path, sizeof(path), "%s.xz", name);
-		path[PATH_MAX - 1] = '\0';
+		unsigned int i;
+
+		for (i = 0; i < RTE_DIM(compression_algorithms); i++) {
+			snprintf(path, sizeof(path), "%s.%s", name,
+				compression_algorithms[i].suffix);
+			path[PATH_MAX - 1] = '\0';
+			if (access(path, F_OK) != 0)
+				continue;
 #ifndef RTE_HAS_LIBARCHIVE
-		if (access(path, F_OK) == 0) {
 			EAL_LOG(WARNING, "libarchive not linked, %s cannot be decompressed",
 				path);
-		}
 #else
-		ret = firmware_read(path, buf, bufsz);
+			ret = firmware_read(path, buf, bufsz);
 #endif
+			break;
+		}
 	}
 	return ret;
 }
-- 
2.44.0

Re: [PATCH] eal/unix: support ZSTD compression for firmwares

[Bruce Richardson]

On Wed, May 08, 2024 at 11:52:14AM +0200, David Marchand wrote:
> Ubuntu 24.04 started to compress firmwares with ZSTD compression.
> 
> Bugzilla ID: 1437
> 
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
>  lib/eal/unix/eal_firmware.c | 42 +++++++++++++++++++++++++++++--------
>  1 file changed, 33 insertions(+), 9 deletions(-)
> 
> diff --git a/lib/eal/unix/eal_firmware.c b/lib/eal/unix/eal_firmware.c
> index 1d47e879c8..065e251f9d 100644
> --- a/lib/eal/unix/eal_firmware.c
> +++ b/lib/eal/unix/eal_firmware.c
> @@ -16,6 +16,21 @@
>  #include "eal_firmware.h"
>  #include "eal_private.h"
>  
> +#ifndef RTE_HAS_LIBARCHIVE
> +/* Fake definitions for the compression_algorithms array below. */
> +struct archive;
> +extern int archive_read_support_filter_xz(struct archive *a);
> +extern int archive_read_support_filter_zstd(struct archive *a);
> +#endif
> +

Do these not lead to unresolved symbols on link?

> +static struct {
> +	const char *suffix;
> +	int (*support_callback)(struct archive *a);
> +} compression_algorithms[] = {
> +	{ "xz", archive_read_support_filter_xz, },
> +	{ "zst", archive_read_support_filter_zstd, },
> +};
> +

Rather than defining stubs for these functions from libarchive, can you
just have an empty list if no libarchive?

  struct archive;  /* may need to be #ifdef'ed perhaps? */
  static struct {
  	const char *suffix;
  	int (*support_callback)(struct archive *a);
  } compression_algorithms[] = {
  #ifdef RTE_HAS_LIBARCHIVE
  	{ "xz", archive_read_support_filter_xz, },
  	{ "zst", archive_read_support_filter_zstd, },
  #endif
  };


>  #ifdef RTE_HAS_LIBARCHIVE
>  

<snip>

Re: [PATCH] eal/unix: support ZSTD compression for firmwares

[Bruce Richardson]

On Wed, May 08, 2024 at 11:50:02AM +0100, Bruce Richardson wrote:
> On Wed, May 08, 2024 at 11:52:14AM +0200, David Marchand wrote:
> > Ubuntu 24.04 started to compress firmwares with ZSTD compression.
> > 

Minor nit, "firmware" doesn't really have a plural in English. I'd suggest
using "firmware files" here rather than "firmwares".

> > Bugzilla ID: 1437
> > 
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > ---
> >  lib/eal/unix/eal_firmware.c | 42 +++++++++++++++++++++++++++++--------
> >  1 file changed, 33 insertions(+), 9 deletions(-)
> > 
> > diff --git a/lib/eal/unix/eal_firmware.c b/lib/eal/unix/eal_firmware.c
> > index 1d47e879c8..065e251f9d 100644
> > --- a/lib/eal/unix/eal_firmware.c
> > +++ b/lib/eal/unix/eal_firmware.c
> > @@ -16,6 +16,21 @@
> >  #include "eal_firmware.h"
> >  #include "eal_private.h"
> >  
> > +#ifndef RTE_HAS_LIBARCHIVE
> > +/* Fake definitions for the compression_algorithms array below. */
> > +struct archive;
> > +extern int archive_read_support_filter_xz(struct archive *a);
> > +extern int archive_read_support_filter_zstd(struct archive *a);
> > +#endif
> > +
> 
> Do these not lead to unresolved symbols on link?
> 
Confirmed; in my tests, I do get unresolved symbols on linking without
libarchive.

On the plus side, this does fix the issues when libarchive-dev package is
installed.

/Bruce

Re: [PATCH] eal/unix: support ZSTD compression for firmwares

[David Marchand]

On Wed, May 8, 2024 at 12:50 PM Bruce Richardson
<bruce.richardson@intel.com> wrote:
>
> On Wed, May 08, 2024 at 11:52:14AM +0200, David Marchand wrote:
> > Ubuntu 24.04 started to compress firmwares with ZSTD compression.
> >
> > Bugzilla ID: 1437
> >
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > ---
> >  lib/eal/unix/eal_firmware.c | 42 +++++++++++++++++++++++++++++--------
> >  1 file changed, 33 insertions(+), 9 deletions(-)
> >
> > diff --git a/lib/eal/unix/eal_firmware.c b/lib/eal/unix/eal_firmware.c
> > index 1d47e879c8..065e251f9d 100644
> > --- a/lib/eal/unix/eal_firmware.c
> > +++ b/lib/eal/unix/eal_firmware.c
> > @@ -16,6 +16,21 @@
> >  #include "eal_firmware.h"
> >  #include "eal_private.h"
> >
> > +#ifndef RTE_HAS_LIBARCHIVE
> > +/* Fake definitions for the compression_algorithms array below. */
> > +struct archive;
> > +extern int archive_read_support_filter_xz(struct archive *a);
> > +extern int archive_read_support_filter_zstd(struct archive *a);
> > +#endif
> > +
>
> Do these not lead to unresolved symbols on link?

Obviously yes... sorry, I was not thinking right and did really
limited testing before sending (only compiled librte_eal.a).
I'll fix this.


>
> > +static struct {
> > +     const char *suffix;
> > +     int (*support_callback)(struct archive *a);
> > +} compression_algorithms[] = {
> > +     { "xz", archive_read_support_filter_xz, },
> > +     { "zst", archive_read_support_filter_zstd, },
> > +};
> > +
>
> Rather than defining stubs for these functions from libarchive, can you
> just have an empty list if no libarchive?
>
>   struct archive;  /* may need to be #ifdef'ed perhaps? */
>   static struct {
>         const char *suffix;
>         int (*support_callback)(struct archive *a);
>   } compression_algorithms[] = {
>   #ifdef RTE_HAS_LIBARCHIVE
>         { "xz", archive_read_support_filter_xz, },
>         { "zst", archive_read_support_filter_zstd, },
>   #endif
>   };

That's an option, but then we lose a warning message to the user
telling that some compressed files are on the system, and linking with
libarchive could help.

I think I'll keep an array of supported extensions and put explicit
calls to archive_read_support_filter_xz/archive_read_support_filter_zstd
under #ifdef.


-- 
David Marchand

[PATCH v2] eal/unix: support ZSTD compression for firmware

[David Marchand]

Ubuntu 24.04 started to compress firmware files with ZSTD compression.

Bugzilla ID: 1437
Cc: stable@dpdk.org

Signed-off-by: David Marchand <david.marchand@redhat.com>
---
Chances since v1:
- fixed link issue when libarchive is not available,
- Cc'd stable@dpdk.org,

---
 lib/eal/unix/eal_firmware.c | 24 ++++++++++++++++++------
 1 file changed, 18 insertions(+), 6 deletions(-)

diff --git a/lib/eal/unix/eal_firmware.c b/lib/eal/unix/eal_firmware.c
index 1d47e879c8..31de027598 100644
--- a/lib/eal/unix/eal_firmware.c
+++ b/lib/eal/unix/eal_firmware.c
@@ -16,6 +16,8 @@
 #include "eal_firmware.h"
 #include "eal_private.h"
 
+static const char * const compression_algorithms[] = { "xz", "zst" };
+
 #ifdef RTE_HAS_LIBARCHIVE
 
 struct firmware_read_ctx {
@@ -37,7 +39,11 @@ firmware_open(struct firmware_read_ctx *ctx, const char *name, size_t blocksize)
 
 	err = archive_read_support_filter_xz(ctx->a);
 	if (err != ARCHIVE_OK && err != ARCHIVE_WARN)
-		goto error;
+		EAL_LOG(DEBUG, "could not initialise libarchive for xz compression");
+
+	err = archive_read_support_filter_zstd(ctx->a);
+	if (err != ARCHIVE_OK && err != ARCHIVE_WARN)
+		EAL_LOG(DEBUG, "could not initialise libarchive for zstd compression");
 
 	if (archive_read_open_filename(ctx->a, name, blocksize) != ARCHIVE_OK)
 		goto error;
@@ -148,16 +154,22 @@ rte_firmware_read(const char *name, void **buf, size_t *bufsz)
 
 	ret = firmware_read(name, buf, bufsz);
 	if (ret < 0) {
-		snprintf(path, sizeof(path), "%s.xz", name);
-		path[PATH_MAX - 1] = '\0';
+		unsigned int i;
+
+		for (i = 0; i < RTE_DIM(compression_algorithms); i++) {
+			snprintf(path, sizeof(path), "%s.%s", name,
+				compression_algorithms[i]);
+			path[PATH_MAX - 1] = '\0';
+			if (access(path, F_OK) != 0)
+				continue;
 #ifndef RTE_HAS_LIBARCHIVE
-		if (access(path, F_OK) == 0) {
 			EAL_LOG(WARNING, "libarchive not linked, %s cannot be decompressed",
 				path);
-		}
 #else
-		ret = firmware_read(path, buf, bufsz);
+			ret = firmware_read(path, buf, bufsz);
 #endif
+			break;
+		}
 	}
 	return ret;
 }
-- 
2.44.0

Re: [PATCH v2] eal/unix: support ZSTD compression for firmware

[Bruce Richardson]

On Mon, May 13, 2024 at 01:12:04PM +0200, David Marchand wrote:
> Ubuntu 24.04 started to compress firmware files with ZSTD compression.
> 
> Bugzilla ID: 1437
> Cc: stable@dpdk.org
> 
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
> Chances since v1:
> - fixed link issue when libarchive is not available,
> - Cc'd stable@dpdk.org,
> 
> ---
>  lib/eal/unix/eal_firmware.c | 24 ++++++++++++++++++------
>  1 file changed, 18 insertions(+), 6 deletions(-)
> 

Acked-by: Bruce Richardson <bruce.richardson@intel.com>

Verified that port initializes ok with zstd compressed firmware, and we get
suitable error if libarchive is not installed.

Tested-by: Bruce Richardson <bruce.richardson@intel.com>

One minor suggestion below.

> diff --git a/lib/eal/unix/eal_firmware.c b/lib/eal/unix/eal_firmware.c
> index 1d47e879c8..31de027598 100644
> --- a/lib/eal/unix/eal_firmware.c
> +++ b/lib/eal/unix/eal_firmware.c
> @@ -16,6 +16,8 @@
>  #include "eal_firmware.h"
>  #include "eal_private.h"
>  
> +static const char * const compression_algorithms[] = { "xz", "zst" };
Very minor nit: these are the suffixes used to find the files, more than
they are the compression algorithms. There I'd suggest renaming to
"compression_suffixes".
[Perhaps just fix on apply if taking this suggestion]

> +
>  #ifdef RTE_HAS_LIBARCHIVE
>  
>  struct firmware_read_ctx {
<snip>

Re: [PATCH v2] eal/unix: support ZSTD compression for firmware

[David Marchand]

Hello Bruce,

On Mon, May 13, 2024 at 1:37 PM Bruce Richardson
<bruce.richardson@intel.com> wrote:
> > @@ -16,6 +16,8 @@
> >  #include "eal_firmware.h"
> >  #include "eal_private.h"
> >
> > +static const char * const compression_algorithms[] = { "xz", "zst" };
> Very minor nit: these are the suffixes used to find the files, more than
> they are the compression algorithms. There I'd suggest renaming to
> "compression_suffixes".
> [Perhaps just fix on apply if taking this suggestion]

I'll fix when applying.
Thanks for the review and tests.


-- 
David Marchand

Re: [PATCH v2] eal/unix: support ZSTD compression for firmware

[David Marchand]

On Mon, May 13, 2024 at 1:37 PM Bruce Richardson
<bruce.richardson@intel.com> wrote:
> On Mon, May 13, 2024 at 01:12:04PM +0200, David Marchand wrote:
> > Ubuntu 24.04 started to compress firmware files with ZSTD compression.
> >
> > Bugzilla ID: 1437
> > Cc: stable@dpdk.org
> >
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> Acked-by: Bruce Richardson <bruce.richardson@intel.com>

Applied with additional nit from Bruce on variable name.
Thanks.


-- 
David Marchand

Re: [PATCH v2] eal/unix: support ZSTD compression for firmware

[Xueming Li]

[-- Attachment #1: Type: text/plain, Size: 3081 bytes --]

Hi David,

I get unix compilation failure when backporting this patch to 23.11 LTS, the EAL_LOG macro is not backported to 23.11 LTS, any suggestion?

Regards,
Xueming

________________________________
From: David Marchand <david.marchand@redhat.com>
Sent: Monday, May 13, 2024 7:12 PM
To: dev@dpdk.org <dev@dpdk.org>
Cc: Yu Jiang <yux.jiang@intel.com>; Bruce Richardson <bruce.richardson@intel.com>; stable@dpdk.org <stable@dpdk.org>
Subject: [PATCH v2] eal/unix: support ZSTD compression for firmware

Ubuntu 24.04 started to compress firmware files with ZSTD compression.

Bugzilla ID: 1437
Cc: stable@dpdk.org

Signed-off-by: David Marchand <david.marchand@redhat.com>
---
Chances since v1:
- fixed link issue when libarchive is not available,
- Cc'd stable@dpdk.org,

---
 lib/eal/unix/eal_firmware.c | 24 ++++++++++++++++++------
 1 file changed, 18 insertions(+), 6 deletions(-)

diff --git a/lib/eal/unix/eal_firmware.c b/lib/eal/unix/eal_firmware.c
index 1d47e879c8..31de027598 100644
--- a/lib/eal/unix/eal_firmware.c
+++ b/lib/eal/unix/eal_firmware.c
@@ -16,6 +16,8 @@
 #include "eal_firmware.h"
 #include "eal_private.h"

+static const char * const compression_algorithms[] = { "xz", "zst" };
+
 #ifdef RTE_HAS_LIBARCHIVE

 struct firmware_read_ctx {
@@ -37,7 +39,11 @@ firmware_open(struct firmware_read_ctx *ctx, const char *name, size_t blocksize)

         err = archive_read_support_filter_xz(ctx->a);
         if (err != ARCHIVE_OK && err != ARCHIVE_WARN)
-               goto error;
+               EAL_LOG(DEBUG, "could not initialise libarchive for xz compression");
+
+       err = archive_read_support_filter_zstd(ctx->a);
+       if (err != ARCHIVE_OK && err != ARCHIVE_WARN)
+               EAL_LOG(DEBUG, "could not initialise libarchive for zstd compression");

         if (archive_read_open_filename(ctx->a, name, blocksize) != ARCHIVE_OK)
                 goto error;
@@ -148,16 +154,22 @@ rte_firmware_read(const char *name, void **buf, size_t *bufsz)

         ret = firmware_read(name, buf, bufsz);
         if (ret < 0) {
-               snprintf(path, sizeof(path), "%s.xz", name);
-               path[PATH_MAX - 1] = '\0';
+               unsigned int i;
+
+               for (i = 0; i < RTE_DIM(compression_algorithms); i++) {
+                       snprintf(path, sizeof(path), "%s.%s", name,
+                               compression_algorithms[i]);
+                       path[PATH_MAX - 1] = '\0';
+                       if (access(path, F_OK) != 0)
+                               continue;
 #ifndef RTE_HAS_LIBARCHIVE
-               if (access(path, F_OK) == 0) {
                         EAL_LOG(WARNING, "libarchive not linked, %s cannot be decompressed",
                                 path);
-               }
 #else
-               ret = firmware_read(path, buf, bufsz);
+                       ret = firmware_read(path, buf, bufsz);
 #endif
+                       break;
+               }
         }
         return ret;
 }
--
2.44.0


[-- Attachment #2: Type: text/html, Size: 7374 bytes --]

Re: [PATCH v2] eal/unix: support ZSTD compression for firmware

[David Marchand]

On Wed, Jul 3, 2024 at 9:12 AM Xueming Li <xuemingl@nvidia.com> wrote:
>
> Hi David,
>
> I get unix compilation failure when backporting this patch to 23.11 LTS, the EAL_LOG macro is not backported to 23.11 LTS, any suggestion?

Like other logging calls in this same file, you can replace
EAL_LOG(level, fmt, args); with a simple RTE_LOG(level, EAL, fmt "\n",
args);
Am I missing something?


-- 
David Marchand

Re: [PATCH v2] eal/unix: support ZSTD compression for firmware

[Xueming Li]

[-- Attachment #1: Type: text/plain, Size: 871 bytes --]

Thanks, I tried but missed the "EAL" part :-)

________________________________
From: David Marchand <david.marchand@redhat.com>
Sent: Wednesday, July 3, 2024 3:27 PM
To: Xueming Li <xuemingl@nvidia.com>
Cc: dev@dpdk.org <dev@dpdk.org>; Yu Jiang <yux.jiang@intel.com>; Bruce Richardson <bruce.richardson@intel.com>; stable@dpdk.org <stable@dpdk.org>
Subject: Re: [PATCH v2] eal/unix: support ZSTD compression for firmware

On Wed, Jul 3, 2024 at 9:12 AM Xueming Li <xuemingl@nvidia.com> wrote:
>
> Hi David,
>
> I get unix compilation failure when backporting this patch to 23.11 LTS, the EAL_LOG macro is not backported to 23.11 LTS, any suggestion?

Like other logging calls in this same file, you can replace
EAL_LOG(level, fmt, args); with a simple RTE_LOG(level, EAL, fmt "\n",
args);
Am I missing something?
--------


--
David Marchand


[-- Attachment #2: Type: text/html, Size: 2158 bytes --]