[PATCH 0/1] avoid break due to failure to cleanup temporary directory

[PATCH 0/1] avoid break due to failure to cleanup temporary directory

[Andre Muezerie]

When compiling drivers on Windows, instances have been seen where a
temporary directory fails to get cleaned up due to
ERROR_SHARING_VIOLATION (32).

Code inspection did not reveal problems with the DPDK code and scripts,
and this issue was only seen on Windows.

The proposed fix adds a mitigation for this problem, and also sets a flag
to ignore cleanup errors.

For completeness, the error message is included below.

--- stderr ---
Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 631, in _rmtree_unsafe
    os.rmdir(path)
    ~~~~~~~~^^^^^^
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'V:\\github\\dpdk15\\dpdk\\build\\buildtools\\tmpnrqvi9nc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 909, in onexc
    _os.unlink(path)
    ~~~~~~~~~~^^^^^^
PermissionError: [WinError 5] Access is denied: 'V:\\github\\dpdk15\\dpdk\\build\\buildtools\\tmpnrqvi9nc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "V:\github\dpdk15\dpdk\buildtools\gen-pmdinfo-cfile.py", line 13, in <module>
    with tempfile.TemporaryDirectory(dir=tmp_root) as temp:
         ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 950, in __exit__
    self.cleanup()
    ~~~~~~~~~~~~^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 954, in cleanup
    self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 934, in _rmtree
    _shutil.rmtree(name, onexc=onexc)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 790, in rmtree
    return _rmtree_unsafe(path, onexc)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 635, in _rmtree_unsafe
    onexc(os.rmdir, path, err)
    ~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 924, in onexc
    cls._rmtree(path, ignore_errors=ignore_errors,
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                repeated=(path == name))
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 934, in _rmtree
    _shutil.rmtree(name, onexc=onexc)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 790, in rmtree
    return _rmtree_unsafe(path, onexc)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 635, in _rmtree_unsafe
    onexc(os.rmdir, path, err)
    ~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 631, in _rmtree_unsafe
    os.rmdir(path)
    ~~~~~~~~^^^^^^
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'V:\\github\\dpdk15\\dpdk\\build\\buildtools\\tmpnrqvi9nc'

[53/450] Compiling C object lib/librte_pcapng.a.p/pcapng_rte_pcapng.c.obj
ninja: build stopped: subcommand failed.

Andre Muezerie (1):
  buildtools: avoid break due to failure to cleanup temporary directory

 buildtools/gen-pmdinfo-cfile.py | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

--
2.49.0.vfs.0.2

[PATCH 1/1] buildtools: avoid break due to failure to cleanup temporary directory

[Andre Muezerie]

When compiling drivers on Windows, instances have been seen where a
temporary directory fails to get cleaned up due to
ERROR_SHARING_VIOLATION (32).

Code inspection did not reveal problems with the DPDK code and scripts,
and this issue was only seen on Windows. Adding a 1 second sleep before
cleaning up the temporary directory seems to be effective, but to
guarantee that this break does not happen anymore, flag
"ignore_cleanup_errors" is set to "True".

Signed-off-by: Andre Muezerie <andremue@linux.microsoft.com>
---
 buildtools/gen-pmdinfo-cfile.py | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/buildtools/gen-pmdinfo-cfile.py b/buildtools/gen-pmdinfo-cfile.py
index 5fbd51658a..2ddbdb9502 100644
--- a/buildtools/gen-pmdinfo-cfile.py
+++ b/buildtools/gen-pmdinfo-cfile.py
@@ -6,9 +6,10 @@
 import subprocess
 import sys
 import tempfile
+import time
 
 _, tmp_root, ar, archive, output, *pmdinfogen = sys.argv
-with tempfile.TemporaryDirectory(dir=tmp_root) as temp:
+with tempfile.TemporaryDirectory(dir=tmp_root, ignore_cleanup_errors=True) as temp:
     paths = []
     for name in subprocess.run([ar, "t", archive], stdout=subprocess.PIPE,
                                check=True).stdout.decode().splitlines():
@@ -19,3 +20,10 @@
                            check=True, cwd=temp)
             paths.append(os.path.join(temp, name))
     subprocess.run(pmdinfogen + paths + [output], check=True)
+
+    if os.name == "nt":
+        # Instances have been seen on Windows where the temporary directory fails to get cleaned
+        # up due to ERROR_SHARING_VIOLATION (32).
+        # The sleep below is a mitigation for that issue, while ignore_cleanup_errors=True avoids
+        # failures when the issue is hit despite the mitigation.
+        time.sleep(1)
-- 
2.49.0.vfs.0.2

Re: [PATCH 1/1] buildtools: avoid break due to failure to cleanup temporary directory

[Andre Muezerie]

On Mon, May 05, 2025 at 12:46:45PM -0700, Andre Muezerie wrote:
> When compiling drivers on Windows, instances have been seen where a
> temporary directory fails to get cleaned up due to
> ERROR_SHARING_VIOLATION (32).
> 
> Code inspection did not reveal problems with the DPDK code and scripts,
> and this issue was only seen on Windows. Adding a 1 second sleep before
> cleaning up the temporary directory seems to be effective, but to
> guarantee that this break does not happen anymore, flag
> "ignore_cleanup_errors" is set to "True".
> 
> Signed-off-by: Andre Muezerie <andremue@linux.microsoft.com>
> ---
>  buildtools/gen-pmdinfo-cfile.py | 10 +++++++++-
>  1 file changed, 9 insertions(+), 1 deletion(-)
> 
> diff --git a/buildtools/gen-pmdinfo-cfile.py b/buildtools/gen-pmdinfo-cfile.py
> index 5fbd51658a..2ddbdb9502 100644
> --- a/buildtools/gen-pmdinfo-cfile.py
> +++ b/buildtools/gen-pmdinfo-cfile.py
> @@ -6,9 +6,10 @@
>  import subprocess
>  import sys
>  import tempfile
> +import time
>  
>  _, tmp_root, ar, archive, output, *pmdinfogen = sys.argv
> -with tempfile.TemporaryDirectory(dir=tmp_root) as temp:
> +with tempfile.TemporaryDirectory(dir=tmp_root, ignore_cleanup_errors=True) as temp:

This built fine on my machine but failed on the CI.

According to doc\guides\linux_gsg\sys_reqs.rst, currently the minimum
required Python version is 3.6.
However, parameter ignore_cleanup_errors parameter was added in 3.10
(https://docs.python.org/3/library/tempfile.html)

Is there something that blocks us from requiring 3.10 at minimum? Is this
something that could be done for the next release?

>      paths = []
>      for name in subprocess.run([ar, "t", archive], stdout=subprocess.PIPE,
>                                 check=True).stdout.decode().splitlines():
> @@ -19,3 +20,10 @@
>                             check=True, cwd=temp)
>              paths.append(os.path.join(temp, name))
>      subprocess.run(pmdinfogen + paths + [output], check=True)
> +
> +    if os.name == "nt":
> +        # Instances have been seen on Windows where the temporary directory fails to get cleaned
> +        # up due to ERROR_SHARING_VIOLATION (32).
> +        # The sleep below is a mitigation for that issue, while ignore_cleanup_errors=True avoids
> +        # failures when the issue is hit despite the mitigation.
> +        time.sleep(1)
> -- 
> 2.49.0.vfs.0.2

Re: [PATCH 1/1] buildtools: avoid break due to failure to cleanup temporary directory

[Dmitry Kozlyuk]

Hi Andre,

On 05.05.2025 22:46, Andre Muezerie wrote:
> When compiling drivers on Windows, instances have been seen where a
> temporary directory fails to get cleaned up due to
> ERROR_SHARING_VIOLATION (32).
>
> Code inspection did not reveal problems with the DPDK code and scripts,
> and this issue was only seen on Windows. Adding a 1 second sleep before
> cleaning up the temporary directory seems to be effective, but to
> guarantee that this break does not happen anymore, flag
> "ignore_cleanup_errors" is set to "True".
>
> Signed-off-by: Andre Muezerie <andremue@linux.microsoft.com>
> ---
>   buildtools/gen-pmdinfo-cfile.py | 10 +++++++++-
>   1 file changed, 9 insertions(+), 1 deletion(-)
>
> diff --git a/buildtools/gen-pmdinfo-cfile.py b/buildtools/gen-pmdinfo-cfile.py
> index 5fbd51658a..2ddbdb9502 100644
> --- a/buildtools/gen-pmdinfo-cfile.py
> +++ b/buildtools/gen-pmdinfo-cfile.py
> @@ -6,9 +6,10 @@
>   import subprocess
>   import sys
>   import tempfile
> +import time
>   
>   _, tmp_root, ar, archive, output, *pmdinfogen = sys.argv
> -with tempfile.TemporaryDirectory(dir=tmp_root) as temp:
> +with tempfile.TemporaryDirectory(dir=tmp_root, ignore_cleanup_errors=True) as temp:
>       paths = []
>       for name in subprocess.run([ar, "t", archive], stdout=subprocess.PIPE,
>                                  check=True).stdout.decode().splitlines():
> @@ -19,3 +20,10 @@
>                              check=True, cwd=temp)
>               paths.append(os.path.join(temp, name))
>       subprocess.run(pmdinfogen + paths + [output], check=True)
> +
> +    if os.name == "nt":
> +        # Instances have been seen on Windows where the temporary directory fails to get cleaned
> +        # up due to ERROR_SHARING_VIOLATION (32).
> +        # The sleep below is a mitigation for that issue, while ignore_cleanup_errors=True avoids
> +        # failures when the issue is hit despite the mitigation.
> +        time.sleep(1)

Not fond of this timing hack at all.
Meson 0.51 (minimum requirement is 0.57) introduced |@PRIVATE_DIR@ for 
custom_target() [1].
Can we use it to get rid of tempfile.TemporaryDirectory?

drivers/meson.build:
     custom_target(command: [pmdinfo, '@PRIVATE_DIR@', ...], ...)

buildtools/gen-pmdinfo-cfile.py:
     _, temp, ..., *pmdinfogen = sys.argv
     for name in subprocess.run([ar, "t", ...):
        ...
        subprocess.run([ar, "x", ...], cwd=temp)
        ...
|

|[1]: https://mesonbuild.com/Reference-manual_functions.html#custom_target
|

Re: [PATCH 1/1] buildtools: avoid break due to failure to cleanup temporary directory

[Andre Muezerie]

On Tue, May 06, 2025 at 01:47:03AM +0300, Dmitry Kozlyuk wrote:
> Hi Andre,
> 
> On 05.05.2025 22:46, Andre Muezerie wrote:
> >When compiling drivers on Windows, instances have been seen where a
> >temporary directory fails to get cleaned up due to
> >ERROR_SHARING_VIOLATION (32).
> >
> >Code inspection did not reveal problems with the DPDK code and scripts,
> >and this issue was only seen on Windows. Adding a 1 second sleep before
> >cleaning up the temporary directory seems to be effective, but to
> >guarantee that this break does not happen anymore, flag
> >"ignore_cleanup_errors" is set to "True".
> >
> >Signed-off-by: Andre Muezerie <andremue@linux.microsoft.com>
> >---
> >  buildtools/gen-pmdinfo-cfile.py | 10 +++++++++-
> >  1 file changed, 9 insertions(+), 1 deletion(-)
> >
> >diff --git a/buildtools/gen-pmdinfo-cfile.py b/buildtools/gen-pmdinfo-cfile.py
> >index 5fbd51658a..2ddbdb9502 100644
> >--- a/buildtools/gen-pmdinfo-cfile.py
> >+++ b/buildtools/gen-pmdinfo-cfile.py
> >@@ -6,9 +6,10 @@
> >  import subprocess
> >  import sys
> >  import tempfile
> >+import time
> >  _, tmp_root, ar, archive, output, *pmdinfogen = sys.argv
> >-with tempfile.TemporaryDirectory(dir=tmp_root) as temp:
> >+with tempfile.TemporaryDirectory(dir=tmp_root, ignore_cleanup_errors=True) as temp:
> >      paths = []
> >      for name in subprocess.run([ar, "t", archive], stdout=subprocess.PIPE,
> >                                 check=True).stdout.decode().splitlines():
> >@@ -19,3 +20,10 @@
> >                             check=True, cwd=temp)
> >              paths.append(os.path.join(temp, name))
> >      subprocess.run(pmdinfogen + paths + [output], check=True)
> >+
> >+    if os.name == "nt":
> >+        # Instances have been seen on Windows where the temporary directory fails to get cleaned
> >+        # up due to ERROR_SHARING_VIOLATION (32).
> >+        # The sleep below is a mitigation for that issue, while ignore_cleanup_errors=True avoids
> >+        # failures when the issue is hit despite the mitigation.
> >+        time.sleep(1)
> 
> Not fond of this timing hack at all.
> Meson 0.51 (minimum requirement is 0.57) introduced |@PRIVATE_DIR@
> for custom_target() [1].
> Can we use it to get rid of tempfile.TemporaryDirectory?
> 
> drivers/meson.build:
>     custom_target(command: [pmdinfo, '@PRIVATE_DIR@', ...], ...)
> 
> buildtools/gen-pmdinfo-cfile.py:
>     _, temp, ..., *pmdinfogen = sys.argv
>     for name in subprocess.run([ar, "t", ...):
>        ...
>        subprocess.run([ar, "x", ...], cwd=temp)
>        ...
> |
> 
> |[1]: https://mesonbuild.com/Reference-manual_functions.html#custom_target
> |

Yes, I like your suggestion. I'll send out a V2 with the change you suggested.
Thanks for providing all the details about custom_target() and @PRIVATE_DIR@.

[PATCH v2 0/1] avoid break due to failure to cleanup temporary directory

[Andre Muezerie]

When compiling drivers on Windows, instances have been seen where a
temporary directory fails to get cleaned up due to
ERROR_SHARING_VIOLATION (32).

Code inspection did not reveal problems with the DPDK code and scripts,
and this issue was only seen on Windows.

The proposed fix adds a mitigation for this problem, and also sets a flag
to ignore cleanup errors.

For completeness, the error message is included below.

--- stderr ---
Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 631, in _rmtree_unsafe
    os.rmdir(path)
    ~~~~~~~~^^^^^^
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'V:\\github\\dpdk15\\dpdk\\build\\buildtools\\tmpnrqvi9nc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 909, in onexc
    _os.unlink(path)
    ~~~~~~~~~~^^^^^^
PermissionError: [WinError 5] Access is denied: 'V:\\github\\dpdk15\\dpdk\\build\\buildtools\\tmpnrqvi9nc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "V:\github\dpdk15\dpdk\buildtools\gen-pmdinfo-cfile.py", line 13, in <module>
    with tempfile.TemporaryDirectory(dir=tmp_root) as temp:
         ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 950, in __exit__
    self.cleanup()
    ~~~~~~~~~~~~^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 954, in cleanup
    self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 934, in _rmtree
    _shutil.rmtree(name, onexc=onexc)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 790, in rmtree
    return _rmtree_unsafe(path, onexc)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 635, in _rmtree_unsafe
    onexc(os.rmdir, path, err)
    ~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 924, in onexc
    cls._rmtree(path, ignore_errors=ignore_errors,
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                repeated=(path == name))
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 934, in _rmtree
    _shutil.rmtree(name, onexc=onexc)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 790, in rmtree
    return _rmtree_unsafe(path, onexc)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 635, in _rmtree_unsafe
    onexc(os.rmdir, path, err)
    ~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 631, in _rmtree_unsafe
    os.rmdir(path)
    ~~~~~~~~^^^^^^
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'V:\\github\\dpdk15\\dpdk\\build\\buildtools\\tmpnrqvi9nc'

[53/450] Compiling C object lib/librte_pcapng.a.p/pcapng_rte_pcapng.c.obj
ninja: build stopped: subcommand failed.

v2:
- removed use of Python's tempfile.TemporaryDirectory and used
  '@PRIVATE_DIR@' in meson's custom_target command instead.

Andre Muezerie (1):
  buildtools: avoid break due to failure to cleanup temporary directory

 buildtools/gen-pmdinfo-cfile.py | 24 +++++++++++-------------
 drivers/meson.build             |  2 +-
 2 files changed, 12 insertions(+), 14 deletions(-)

--
2.49.0.vfs.0.2

[PATCH v2 1/1] buildtools: avoid break due to failure to cleanup temporary directory

[Andre Muezerie]

When compiling drivers on Windows, instances have been seen where a
temporary directory fails to get cleaned up due to
ERROR_SHARING_VIOLATION (32).

Code inspection did not reveal problems with the DPDK code and scripts,
and this issue was only seen on Windows. Adding a 1 second sleep before
cleaning up the temporary directory seems to be effective, but to
guarantee that this break does not happen anymore, flag
"ignore_cleanup_errors" is set to "True".

Signed-off-by: Andre Muezerie <andremue@linux.microsoft.com>
---
 buildtools/gen-pmdinfo-cfile.py | 24 +++++++++++-------------
 drivers/meson.build             |  2 +-
 2 files changed, 12 insertions(+), 14 deletions(-)

diff --git a/buildtools/gen-pmdinfo-cfile.py b/buildtools/gen-pmdinfo-cfile.py
index 5fbd51658a..cf8881297f 100644
--- a/buildtools/gen-pmdinfo-cfile.py
+++ b/buildtools/gen-pmdinfo-cfile.py
@@ -5,17 +5,15 @@
 import os
 import subprocess
 import sys
-import tempfile
 
-_, tmp_root, ar, archive, output, *pmdinfogen = sys.argv
-with tempfile.TemporaryDirectory(dir=tmp_root) as temp:
-    paths = []
-    for name in subprocess.run([ar, "t", archive], stdout=subprocess.PIPE,
-                               check=True).stdout.decode().splitlines():
-        if os.path.exists(name):
-            paths.append(name)
-        else:
-            subprocess.run([ar, "x", os.path.abspath(archive), name],
-                           check=True, cwd=temp)
-            paths.append(os.path.join(temp, name))
-    subprocess.run(pmdinfogen + paths + [output], check=True)
+_, tmp_root, ar, tmp_dir, archive, output, *pmdinfogen = sys.argv
+paths = []
+for name in subprocess.run([ar, "t", archive], stdout=subprocess.PIPE,
+                            check=True).stdout.decode().splitlines():
+    if os.path.exists(name):
+        paths.append(name)
+    else:
+        subprocess.run([ar, "x", os.path.abspath(archive), name],
+                        check=True, cwd=tmp_dir)
+        paths.append(os.path.join(tmp_dir, name))
+subprocess.run(pmdinfogen + paths + [output], check=True)
diff --git a/drivers/meson.build b/drivers/meson.build
index caee1e3b79..8aad099c40 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -274,7 +274,7 @@ foreach subpath:subdirs
                 c_args: cflags)
         objs += tmp_lib.extract_all_objects(recursive: true)
         sources_pmd_info = custom_target(out_filename,
-                command: [pmdinfo, tmp_lib.full_path(), '@OUTPUT@', pmdinfogen],
+                command: [pmdinfo, '@PRIVATE_DIR@', tmp_lib.full_path(), '@OUTPUT@', pmdinfogen],
                 output: out_filename,
                 depends: [tmp_lib])
 
-- 
2.49.0.vfs.0.2

Re: [PATCH v2 1/1] buildtools: avoid break due to failure to cleanup temporary directory

[Dmitry Kozlyuk]

On 07.05.2025 01:14, Andre Muezerie wrote:
> When compiling drivers on Windows, instances have been seen where a
> temporary directory fails to get cleaned up due to
> ERROR_SHARING_VIOLATION (32).
>
> Code inspection did not reveal problems with the DPDK code and scripts,
> and this issue was only seen on Windows. Adding a 1 second sleep before
> cleaning up the temporary directory seems to be effective, but to
> guarantee that this break does not happen anymore, flag
> "ignore_cleanup_errors" is set to "True".

This paragraph needs an update.

> --- a/buildtools/gen-pmdinfo-cfile.py
> +++ b/buildtools/gen-pmdinfo-cfile.py
> @@ -5,17 +5,15 @@
>   import os
>   import subprocess
>   import sys
> -import tempfile
>   
> -_, tmp_root, ar, archive, output, *pmdinfogen = sys.argv
> -with tempfile.TemporaryDirectory(dir=tmp_root) as temp:
> -    paths = []
> -    for name in subprocess.run([ar, "t", archive], stdout=subprocess.PIPE,
> -                               check=True).stdout.decode().splitlines():
> -        if os.path.exists(name):
> -            paths.append(name)
> -        else:
> -            subprocess.run([ar, "x", os.path.abspath(archive), name],
> -                           check=True, cwd=temp)
> -            paths.append(os.path.join(temp, name))
> -    subprocess.run(pmdinfogen + paths + [output], check=True)
> +_, tmp_root, ar, tmp_dir, archive, output, *pmdinfogen = sys.argv
No need for `tmp_root` anymore: it was required to put temporary files 
inside the build directory, but `@PRIVATE_DIR@` is already there.

[PATCH v3 0/1] avoid break due to failure to cleanup temporary directory

[Andre Muezerie]

When compiling drivers on Windows, instances have been seen where a
temporary directory fails to get cleaned up due to
ERROR_SHARING_VIOLATION (32).

This issue was not seen on operating systems other than Windows.

This patch eliminates the use of tempfile.TemporaryDirectory, which
was triggering the issue.

For completeness, the error message is included below.

--- stderr ---
Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 631, in _rmtree_unsafe
    os.rmdir(path)
    ~~~~~~~~^^^^^^
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'V:\\github\\dpdk15\\dpdk\\build\\buildtools\\tmpnrqvi9nc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 909, in onexc
    _os.unlink(path)
    ~~~~~~~~~~^^^^^^
PermissionError: [WinError 5] Access is denied: 'V:\\github\\dpdk15\\dpdk\\build\\buildtools\\tmpnrqvi9nc'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "V:\github\dpdk15\dpdk\buildtools\gen-pmdinfo-cfile.py", line 13, in <module>
    with tempfile.TemporaryDirectory(dir=tmp_root) as temp:
         ~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 950, in __exit__
    self.cleanup()
    ~~~~~~~~~~~~^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 954, in cleanup
    self._rmtree(self.name, ignore_errors=self._ignore_cleanup_errors)
    ~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 934, in _rmtree
    _shutil.rmtree(name, onexc=onexc)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 790, in rmtree
    return _rmtree_unsafe(path, onexc)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 635, in _rmtree_unsafe
    onexc(os.rmdir, path, err)
    ~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 924, in onexc
    cls._rmtree(path, ignore_errors=ignore_errors,
    ~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                repeated=(path == name))
                ^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\tempfile.py", line 934, in _rmtree
    _shutil.rmtree(name, onexc=onexc)
    ~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 790, in rmtree
    return _rmtree_unsafe(path, onexc)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 635, in _rmtree_unsafe
    onexc(os.rmdir, path, err)
    ~~~~~^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.13_3.13.1008.0_x64__qbz5n2kfra8p0\Lib\shutil.py", line 631, in _rmtree_unsafe
    os.rmdir(path)
    ~~~~~~~~^^^^^^
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'V:\\github\\dpdk15\\dpdk\\build\\buildtools\\tmpnrqvi9nc'

[53/450] Compiling C object lib/librte_pcapng.a.p/pcapng_rte_pcapng.c.obj
ninja: build stopped: subcommand failed.

v3:
- removed parameter tmp_root from gen-pmdinfo-cfile.py
- modified gen-pmdinfo-cfile.py to create '@PRIVATE_DIR@' when it does not exist.

v2:
- removed use of Python's tempfile.TemporaryDirectory and used
  '@PRIVATE_DIR@' in meson's custom_target command instead.

Andre Muezerie (1):
  buildtools: avoid break due to failure to cleanup temporary directory

 buildtools/gen-pmdinfo-cfile.py | 26 +++++++++++++-------------
 buildtools/meson.build          |  2 +-
 drivers/meson.build             |  2 +-
 3 files changed, 15 insertions(+), 15 deletions(-)

--
2.49.0.vfs.0.2

[PATCH v3 1/1] buildtools: avoid break due to failure to cleanup temporary directory

[Andre Muezerie]

When compiling drivers on Windows, instances have been seen where a
temporary directory fails to get cleaned up due to
ERROR_SHARING_VIOLATION (32).

This issue was not seen on operating systems other than Windows.

This patch eliminates the use of tempfile.TemporaryDirectory, which
was triggering the issue.

Signed-off-by: Andre Muezerie <andremue@linux.microsoft.com>
---
 buildtools/gen-pmdinfo-cfile.py | 26 +++++++++++++-------------
 buildtools/meson.build          |  2 +-
 drivers/meson.build             |  2 +-
 3 files changed, 15 insertions(+), 15 deletions(-)

diff --git a/buildtools/gen-pmdinfo-cfile.py b/buildtools/gen-pmdinfo-cfile.py
index 5fbd51658a..8ffc192c40 100644
--- a/buildtools/gen-pmdinfo-cfile.py
+++ b/buildtools/gen-pmdinfo-cfile.py
@@ -5,17 +5,17 @@
 import os
 import subprocess
 import sys
-import tempfile
 
-_, tmp_root, ar, archive, output, *pmdinfogen = sys.argv
-with tempfile.TemporaryDirectory(dir=tmp_root) as temp:
-    paths = []
-    for name in subprocess.run([ar, "t", archive], stdout=subprocess.PIPE,
-                               check=True).stdout.decode().splitlines():
-        if os.path.exists(name):
-            paths.append(name)
-        else:
-            subprocess.run([ar, "x", os.path.abspath(archive), name],
-                           check=True, cwd=temp)
-            paths.append(os.path.join(temp, name))
-    subprocess.run(pmdinfogen + paths + [output], check=True)
+_, ar, tmp_dir, archive, output, *pmdinfogen = sys.argv
+paths = []
+for name in subprocess.run([ar, "t", archive], stdout=subprocess.PIPE,
+                            check=True).stdout.decode().splitlines():
+    if os.path.exists(name):
+        paths.append(name)
+    else:
+        if not os.path.exists(tmp_dir):
+            os.makedirs(tmp_dir)
+        subprocess.run([ar, "x", os.path.abspath(archive), name],
+                        check=True, cwd=tmp_dir)
+        paths.append(os.path.join(tmp_dir, name))
+subprocess.run(pmdinfogen + paths + [output], check=True)
diff --git a/buildtools/meson.build b/buildtools/meson.build
index 1cd1ce02fd..e7c53c9a84 100644
--- a/buildtools/meson.build
+++ b/buildtools/meson.build
@@ -34,7 +34,7 @@ install_data([
         install_dir: 'bin')
 
 # select library and object file format
-pmdinfo = py3 + files('gen-pmdinfo-cfile.py') + [meson.current_build_dir()]
+pmdinfo = py3 + files('gen-pmdinfo-cfile.py')
 pmdinfogen = py3 + files('pmdinfogen.py')
 if host_machine.system() == 'windows'
     if cc.get_id() == 'gcc'
diff --git a/drivers/meson.build b/drivers/meson.build
index caee1e3b79..8aad099c40 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -274,7 +274,7 @@ foreach subpath:subdirs
                 c_args: cflags)
         objs += tmp_lib.extract_all_objects(recursive: true)
         sources_pmd_info = custom_target(out_filename,
-                command: [pmdinfo, tmp_lib.full_path(), '@OUTPUT@', pmdinfogen],
+                command: [pmdinfo, '@PRIVATE_DIR@', tmp_lib.full_path(), '@OUTPUT@', pmdinfogen],
                 output: out_filename,
                 depends: [tmp_lib])
 
-- 
2.49.0.vfs.0.2

Re: [PATCH v3 1/1] buildtools: avoid break due to failure to cleanup temporary directory

[Dmitry Kozlyuk]

Acked-by: Dmitry Kozlyuk <dmitry.kozliuk@gmail.com>