Meson 0.60 switched the format of uninstalled static libraries to thin archives, that is, they contain only paths to object files, not the files themselves. Files cannot be extracted in this case, resulting in build errors: ar: `x' cannot be used on thin archives. Handle thin archives when invoking pmdinfogen by directly using the files referenced in the archive. Bugzilla ID: 836 Fixes: e6e9730c7066 ("buildtools: support object file extraction for Windows") Cc: stable@dpdk.org Reported-by: Michal Berger <michallinuxstuff@gmail.com> Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> --- buildtools/gen-pmdinfo-cfile.py | 23 ++++++++++++++--------- 1 file changed, 14 insertions(+), 9 deletions(-) diff --git a/buildtools/gen-pmdinfo-cfile.py b/buildtools/gen-pmdinfo-cfile.py index 58fe3ad152..3453e5b4b9 100644 --- a/buildtools/gen-pmdinfo-cfile.py +++ b/buildtools/gen-pmdinfo-cfile.py @@ -8,13 +8,18 @@ import tempfile _, tmp_root, ar, archive, output, *pmdinfogen = sys.argv -with tempfile.TemporaryDirectory(dir=tmp_root) as temp: - run_ar = lambda command: subprocess.run( - [ar, command, os.path.abspath(archive)], - stdout=subprocess.PIPE, check=True, cwd=temp - ) - # Don't use "ar p", because its output is corrupted on Windows. - run_ar("x") - names = run_ar("t").stdout.decode().splitlines() - paths = [os.path.join(temp, name) for name in names] +archive = os.path.abspath(archive) +names = subprocess.run([ar, "t", archive], + stdout=subprocess.PIPE, check=True).stdout.decode().splitlines() +with open(archive, "rb") as f: + is_thin = f.read(7) == b"!<thin>" +if is_thin: + # Thin archive needs no unpacking, just use the paths within. + paths = [os.path.join(archive, name) for name in names] subprocess.run(pmdinfogen + paths + [output], check=True) +else: + with tempfile.TemporaryDirectory(dir=tmp_root) as temp: + # Don't use "ar p", because its output is corrupted on Windows. + paths = [os.path.join(temp, name) for name in names] + subprocess.run([ar, "x", archive], check=True, cwd=temp) + subprocess.run(pmdinfogen + paths + [output], check=True) -- 2.29.3
On Tue, Oct 26, 2021 at 10:32:39PM +0300, Dmitry Kozlyuk wrote: > Meson 0.60 switched the format of uninstalled static libraries > to thin archives, that is, they contain only paths to object files, > not the files themselves. Files cannot be extracted in this case, > resulting in build errors: > > ar: `x' cannot be used on thin archives. > > Handle thin archives when invoking pmdinfogen > by directly using the files referenced in the archive. > > Bugzilla ID: 836 > Fixes: e6e9730c7066 ("buildtools: support object file extraction for Windows") > Cc: stable@dpdk.org > > Reported-by: Michal Berger <michallinuxstuff@gmail.com> > Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> Thanks for looking into this and proposing a fix. Some thoughts inline below. /Bruce > --- > buildtools/gen-pmdinfo-cfile.py | 23 ++++++++++++++--------- > 1 file changed, 14 insertions(+), 9 deletions(-) > > diff --git a/buildtools/gen-pmdinfo-cfile.py b/buildtools/gen-pmdinfo-cfile.py > index 58fe3ad152..3453e5b4b9 100644 > --- a/buildtools/gen-pmdinfo-cfile.py > +++ b/buildtools/gen-pmdinfo-cfile.py > @@ -8,13 +8,18 @@ > import tempfile > > _, tmp_root, ar, archive, output, *pmdinfogen = sys.argv > -with tempfile.TemporaryDirectory(dir=tmp_root) as temp: > - run_ar = lambda command: subprocess.run( > - [ar, command, os.path.abspath(archive)], > - stdout=subprocess.PIPE, check=True, cwd=temp > - ) > - # Don't use "ar p", because its output is corrupted on Windows. > - run_ar("x") > - names = run_ar("t").stdout.decode().splitlines() > - paths = [os.path.join(temp, name) for name in names] > +archive = os.path.abspath(archive) > +names = subprocess.run([ar, "t", archive], > + stdout=subprocess.PIPE, check=True).stdout.decode().splitlines() > +with open(archive, "rb") as f: > + is_thin = f.read(7) == b"!<thin>" This part seems overly low-level, and a bit nasty to me, compared to the rest of the script. Since the thin archive files already exist, I wonder if we could work on a file by file basis here, and ignore generally the type of the ".a" file. For example, something like: o_files = [] for n in names: if os.path.exists(n): o_files += n else <extract file> o_files += <extracted_path> WDYT?
On Tue, Oct 26, 2021 at 10:32:39PM +0300, Dmitry Kozlyuk wrote: > Meson 0.60 switched the format of uninstalled static libraries > to thin archives, that is, they contain only paths to object files, > not the files themselves. Files cannot be extracted in this case, > resulting in build errors: > > ar: `x' cannot be used on thin archives. > > Handle thin archives when invoking pmdinfogen > by directly using the files referenced in the archive. > > Bugzilla ID: 836 > Fixes: e6e9730c7066 ("buildtools: support object file extraction for Windows") > Cc: stable@dpdk.org > > Reported-by: Michal Berger <michallinuxstuff@gmail.com> > Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> > --- > buildtools/gen-pmdinfo-cfile.py | 23 ++++++++++++++--------- > 1 file changed, 14 insertions(+), 9 deletions(-) > Here is an alternative fix that works in my testing, based on my earlier suggestion: Regards, /Bruce diff --git a/buildtools/gen-pmdinfo-cfile.py b/buildtools/gen-pmdinfo-cfile.py index 58fe3ad152..5fbd51658a 100644 --- a/buildtools/gen-pmdinfo-cfile.py +++ b/buildtools/gen-pmdinfo-cfile.py @@ -9,12 +9,13 @@ _, tmp_root, ar, archive, output, *pmdinfogen = sys.argv with tempfile.TemporaryDirectory(dir=tmp_root) as temp: - run_ar = lambda command: subprocess.run( - [ar, command, os.path.abspath(archive)], - stdout=subprocess.PIPE, check=True, cwd=temp - ) - # Don't use "ar p", because its output is corrupted on Windows. - run_ar("x") - names = run_ar("t").stdout.decode().splitlines() - paths = [os.path.join(temp, name) for name in names] + paths = [] + for name in subprocess.run([ar, "t", archive], stdout=subprocess.PIPE, + check=True).stdout.decode().splitlines(): + if os.path.exists(name): + paths.append(name) + else: + subprocess.run([ar, "x", os.path.abspath(archive), name], + check=True, cwd=temp) + paths.append(os.path.join(temp, name)) subprocess.run(pmdinfogen + paths + [output], check=True)
2021-10-27 17:16 (UTC+0100), Bruce Richardson:
> On Tue, Oct 26, 2021 at 10:32:39PM +0300, Dmitry Kozlyuk wrote:
> > Meson 0.60 switched the format of uninstalled static libraries
> > to thin archives, that is, they contain only paths to object files,
> > not the files themselves. Files cannot be extracted in this case,
> > resulting in build errors:
> >
> > ar: `x' cannot be used on thin archives.
> >
> > Handle thin archives when invoking pmdinfogen
> > by directly using the files referenced in the archive.
> >
> > Bugzilla ID: 836
> > Fixes: e6e9730c7066 ("buildtools: support object file extraction for Windows")
> > Cc: stable@dpdk.org
> >
> > Reported-by: Michal Berger <michallinuxstuff@gmail.com>
> > Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> > ---
> > buildtools/gen-pmdinfo-cfile.py | 23 ++++++++++++++---------
> > 1 file changed, 14 insertions(+), 9 deletions(-)
> >
>
> Here is an alternative fix that works in my testing, based on my earlier
> suggestion:
>
> Regards,
> /Bruce
>
> diff --git a/buildtools/gen-pmdinfo-cfile.py b/buildtools/gen-pmdinfo-cfile.py
> index 58fe3ad152..5fbd51658a 100644
> --- a/buildtools/gen-pmdinfo-cfile.py
> +++ b/buildtools/gen-pmdinfo-cfile.py
> @@ -9,12 +9,13 @@
>
> _, tmp_root, ar, archive, output, *pmdinfogen = sys.argv
> with tempfile.TemporaryDirectory(dir=tmp_root) as temp:
> - run_ar = lambda command: subprocess.run(
> - [ar, command, os.path.abspath(archive)],
> - stdout=subprocess.PIPE, check=True, cwd=temp
> - )
> - # Don't use "ar p", because its output is corrupted on Windows.
> - run_ar("x")
> - names = run_ar("t").stdout.decode().splitlines()
> - paths = [os.path.join(temp, name) for name in names]
> + paths = []
> + for name in subprocess.run([ar, "t", archive], stdout=subprocess.PIPE,
> + check=True).stdout.decode().splitlines():
> + if os.path.exists(name):
> + paths.append(name)
> + else:
> + subprocess.run([ar, "x", os.path.abspath(archive), name],
> + check=True, cwd=temp)
> + paths.append(os.path.join(temp, name))
> subprocess.run(pmdinfogen + paths + [output], check=True)
>
It buys with simplicity.
I hoped to avoid creating temporary directories in vain
when all archives are thin, but maybe it's not that important.
On Wed, Oct 27, 2021 at 07:45:28PM +0300, Dmitry Kozlyuk wrote:
> 2021-10-27 17:16 (UTC+0100), Bruce Richardson:
> > On Tue, Oct 26, 2021 at 10:32:39PM +0300, Dmitry Kozlyuk wrote:
> > > Meson 0.60 switched the format of uninstalled static libraries
> > > to thin archives, that is, they contain only paths to object files,
> > > not the files themselves. Files cannot be extracted in this case,
> > > resulting in build errors:
> > >
> > > ar: `x' cannot be used on thin archives.
> > >
> > > Handle thin archives when invoking pmdinfogen
> > > by directly using the files referenced in the archive.
> > >
> > > Bugzilla ID: 836
> > > Fixes: e6e9730c7066 ("buildtools: support object file extraction for Windows")
> > > Cc: stable@dpdk.org
> > >
> > > Reported-by: Michal Berger <michallinuxstuff@gmail.com>
> > > Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> > > ---
> > > buildtools/gen-pmdinfo-cfile.py | 23 ++++++++++++++---------
> > > 1 file changed, 14 insertions(+), 9 deletions(-)
> > >
> >
> > Here is an alternative fix that works in my testing, based on my earlier
> > suggestion:
> >
> > Regards,
> > /Bruce
> >
> > diff --git a/buildtools/gen-pmdinfo-cfile.py b/buildtools/gen-pmdinfo-cfile.py
> > index 58fe3ad152..5fbd51658a 100644
> > --- a/buildtools/gen-pmdinfo-cfile.py
> > +++ b/buildtools/gen-pmdinfo-cfile.py
> > @@ -9,12 +9,13 @@
> >
> > _, tmp_root, ar, archive, output, *pmdinfogen = sys.argv
> > with tempfile.TemporaryDirectory(dir=tmp_root) as temp:
> > - run_ar = lambda command: subprocess.run(
> > - [ar, command, os.path.abspath(archive)],
> > - stdout=subprocess.PIPE, check=True, cwd=temp
> > - )
> > - # Don't use "ar p", because its output is corrupted on Windows.
> > - run_ar("x")
> > - names = run_ar("t").stdout.decode().splitlines()
> > - paths = [os.path.join(temp, name) for name in names]
> > + paths = []
> > + for name in subprocess.run([ar, "t", archive], stdout=subprocess.PIPE,
> > + check=True).stdout.decode().splitlines():
> > + if os.path.exists(name):
> > + paths.append(name)
> > + else:
> > + subprocess.run([ar, "x", os.path.abspath(archive), name],
> > + check=True, cwd=temp)
> > + paths.append(os.path.join(temp, name))
> > subprocess.run(pmdinfogen + paths + [output], check=True)
> >
>
> It buys with simplicity.
> I hoped to avoid creating temporary directories in vain
> when all archives are thin, but maybe it's not that important.
Since the temp directory is contained within the build directory, I don't
think it matters. If you don't mind then, I'll do up a v2 patch with my
alternative fix and submit that to the list.
Regards,
/Bruce
Meson 0.60 switched the format of uninstalled static libraries to thin archives, that is, they contain only paths to object files, not the files themselves. Files cannot be extracted in this case, resulting in build errors: ar: `x' cannot be used on thin archives. Handle thin archives when invoking pmdinfogen by directly using the files referenced in the archive, when they already exist, and extracting them if not. Bugzilla ID: 836 Fixes: e6e9730c7066 ("buildtools: support object file extraction for Windows") Cc: stable@dpdk.org Reported-by: Michal Berger <michallinuxstuff@gmail.com> Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com> --- v2: Reworked to not check explicitly for thin archives by checking for specific bytes in the file, but instead by checking file by file if an object path from the .a file exists already. [Kept Dmitry's signoff since v2 re-uses some snippets and commit log messages from v1] --- buildtools/gen-pmdinfo-cfile.py | 17 +++++++++-------- 1 file changed, 9 insertions(+), 8 deletions(-) diff --git a/buildtools/gen-pmdinfo-cfile.py b/buildtools/gen-pmdinfo-cfile.py index 58fe3ad152..5fbd51658a 100644 --- a/buildtools/gen-pmdinfo-cfile.py +++ b/buildtools/gen-pmdinfo-cfile.py @@ -9,12 +9,13 @@ _, tmp_root, ar, archive, output, *pmdinfogen = sys.argv with tempfile.TemporaryDirectory(dir=tmp_root) as temp: - run_ar = lambda command: subprocess.run( - [ar, command, os.path.abspath(archive)], - stdout=subprocess.PIPE, check=True, cwd=temp - ) - # Don't use "ar p", because its output is corrupted on Windows. - run_ar("x") - names = run_ar("t").stdout.decode().splitlines() - paths = [os.path.join(temp, name) for name in names] + paths = [] + for name in subprocess.run([ar, "t", archive], stdout=subprocess.PIPE, + check=True).stdout.decode().splitlines(): + if os.path.exists(name): + paths.append(name) + else: + subprocess.run([ar, "x", os.path.abspath(archive), name], + check=True, cwd=temp) + paths.append(os.path.join(temp, name)) subprocess.run(pmdinfogen + paths + [output], check=True) -- 2.32.0
2021-11-01 16:54 (UTC+0000), Bruce Richardson:
> On Wed, Oct 27, 2021 at 07:45:28PM +0300, Dmitry Kozlyuk wrote:
> > 2021-10-27 17:16 (UTC+0100), Bruce Richardson:
> > > On Tue, Oct 26, 2021 at 10:32:39PM +0300, Dmitry Kozlyuk wrote:
> > > > Meson 0.60 switched the format of uninstalled static libraries
> > > > to thin archives, that is, they contain only paths to object files,
> > > > not the files themselves. Files cannot be extracted in this case,
> > > > resulting in build errors:
> > > >
> > > > ar: `x' cannot be used on thin archives.
> > > >
> > > > Handle thin archives when invoking pmdinfogen
> > > > by directly using the files referenced in the archive.
> > > >
> > > > Bugzilla ID: 836
> > > > Fixes: e6e9730c7066 ("buildtools: support object file extraction for Windows")
> > > > Cc: stable@dpdk.org
> > > >
> > > > Reported-by: Michal Berger <michallinuxstuff@gmail.com>
> > > > Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> > > > ---
> > > > buildtools/gen-pmdinfo-cfile.py | 23 ++++++++++++++---------
> > > > 1 file changed, 14 insertions(+), 9 deletions(-)
> > > >
> > >
> > > Here is an alternative fix that works in my testing, based on my earlier
> > > suggestion:
> > >
> > > Regards,
> > > /Bruce
> > >
> > > diff --git a/buildtools/gen-pmdinfo-cfile.py b/buildtools/gen-pmdinfo-cfile.py
> > > index 58fe3ad152..5fbd51658a 100644
> > > --- a/buildtools/gen-pmdinfo-cfile.py
> > > +++ b/buildtools/gen-pmdinfo-cfile.py
> > > @@ -9,12 +9,13 @@
> > >
> > > _, tmp_root, ar, archive, output, *pmdinfogen = sys.argv
> > > with tempfile.TemporaryDirectory(dir=tmp_root) as temp:
> > > - run_ar = lambda command: subprocess.run(
> > > - [ar, command, os.path.abspath(archive)],
> > > - stdout=subprocess.PIPE, check=True, cwd=temp
> > > - )
> > > - # Don't use "ar p", because its output is corrupted on Windows.
> > > - run_ar("x")
> > > - names = run_ar("t").stdout.decode().splitlines()
> > > - paths = [os.path.join(temp, name) for name in names]
> > > + paths = []
> > > + for name in subprocess.run([ar, "t", archive], stdout=subprocess.PIPE,
> > > + check=True).stdout.decode().splitlines():
> > > + if os.path.exists(name):
> > > + paths.append(name)
> > > + else:
> > > + subprocess.run([ar, "x", os.path.abspath(archive), name],
> > > + check=True, cwd=temp)
> > > + paths.append(os.path.join(temp, name))
> > > subprocess.run(pmdinfogen + paths + [output], check=True)
> > >
> >
> > It buys with simplicity.
> > I hoped to avoid creating temporary directories in vain
> > when all archives are thin, but maybe it's not that important.
>
> Since the temp directory is contained within the build directory, I don't
> think it matters. If you don't mind then, I'll do up a v2 patch with my
> alternative fix and submit that to the list.
OK, thank you!
I meant the time spent on creating those dirs rather than polluting any place.
But it won't be any worse then it is now.
2021-11-01 17:03 (UTC+0000), Bruce Richardson:
> Meson 0.60 switched the format of uninstalled static libraries
> to thin archives, that is, they contain only paths to object files,
> not the files themselves. Files cannot be extracted in this case,
> resulting in build errors:
>
> ar: `x' cannot be used on thin archives.
>
> Handle thin archives when invoking pmdinfogen by directly using the
> files referenced in the archive, when they already exist, and extracting
> them if not.
>
> Bugzilla ID: 836
> Fixes: e6e9730c7066 ("buildtools: support object file extraction for Windows")
> Cc: stable@dpdk.org
>
> Reported-by: Michal Berger <michallinuxstuff@gmail.com>
> Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
LGTM.
Note: meson 0.60 can't build DPDK on Windows due to a meson bug,
but *.pmd.c generation stage passes OK.
02/11/2021 19:08, Dmitry Kozlyuk:
> 2021-11-01 17:03 (UTC+0000), Bruce Richardson:
> > Meson 0.60 switched the format of uninstalled static libraries
> > to thin archives, that is, they contain only paths to object files,
> > not the files themselves. Files cannot be extracted in this case,
> > resulting in build errors:
> >
> > ar: `x' cannot be used on thin archives.
> >
> > Handle thin archives when invoking pmdinfogen by directly using the
> > files referenced in the archive, when they already exist, and extracting
> > them if not.
> >
> > Bugzilla ID: 836
> > Fixes: e6e9730c7066 ("buildtools: support object file extraction for Windows")
> > Cc: stable@dpdk.org
> >
> > Reported-by: Michal Berger <michallinuxstuff@gmail.com>
> > Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
>
> LGTM.
> Note: meson 0.60 can't build DPDK on Windows due to a meson bug,
> but *.pmd.c generation stage passes OK.
Should we add this comment in the commit log?
Knowing that Windows build is broken looks to be an interesting info.
Should we add a known issue in the release notes?
03/11/2021 14:20, Thomas Monjalon:
> 02/11/2021 19:08, Dmitry Kozlyuk:
> > 2021-11-01 17:03 (UTC+0000), Bruce Richardson:
> > > Meson 0.60 switched the format of uninstalled static libraries
> > > to thin archives, that is, they contain only paths to object files,
> > > not the files themselves. Files cannot be extracted in this case,
> > > resulting in build errors:
> > >
> > > ar: `x' cannot be used on thin archives.
> > >
> > > Handle thin archives when invoking pmdinfogen by directly using the
> > > files referenced in the archive, when they already exist, and extracting
> > > them if not.
> > >
> > > Bugzilla ID: 836
> > > Fixes: e6e9730c7066 ("buildtools: support object file extraction for Windows")
> > > Cc: stable@dpdk.org
> > >
> > > Reported-by: Michal Berger <michallinuxstuff@gmail.com>
> > > Signed-off-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>
> > > Signed-off-by: Bruce Richardson <bruce.richardson@intel.com>
> >
> > LGTM.
> > Note: meson 0.60 can't build DPDK on Windows due to a meson bug,
> > but *.pmd.c generation stage passes OK.
>
> Should we add this comment in the commit log?
> Knowing that Windows build is broken looks to be an interesting info.
> Should we add a known issue in the release notes?
It seems the issue on Windows is older than Meson 0.60,
so no additional info is needed in this commit.
A separate update can be done for this Windows-specific issue.
Applied, thanks.