[PATCH] build: check drivers class dependencies early

[PATCH] build: check drivers class dependencies early

[David Marchand]

Drivers implementing a class of devices (for example, drivers/event)
depend on the associated abstraction library (lib/eventdev).
This dependency is expressed in the top level meson.build for this class
(drivers/event/meson.build).

As we are making more libraries optional, custom constructs referencing
the class dependencies in some drivers meson.build (event/dlb2) may break.

It would be possible to add more checks in those drivers meson.build but
it is more straightforward to not even consider a driver meson.build when
the class dependencies are not met.

Signed-off-by: David Marchand <david.marchand@redhat.com>
---
 drivers/meson.build | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/drivers/meson.build b/drivers/meson.build
index 74ae8cb96b..c375352e77 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -70,6 +70,17 @@ foreach subpath:subdirs
     else
         class = subpath
         subdir(class)
+        skip_class = false
+        foreach d:std_deps
+            if not is_variable('shared_rte_' + d)
+                skip_class = true
+                message('Disabling all @1@ drivers: missing internal dependency "@0@"'
+                        .format(d, class))
+            endif
+        endforeach
+        if skip_class
+            continue
+        endif
     endif
 
     # save class name on first occurrence
-- 
2.41.0

Re: [PATCH] build: check drivers class dependencies early

[Bruce Richardson]

On Tue, Aug 01, 2023 at 10:52:53AM +0200, David Marchand wrote:
> Drivers implementing a class of devices (for example, drivers/event)
> depend on the associated abstraction library (lib/eventdev).
> This dependency is expressed in the top level meson.build for this class
> (drivers/event/meson.build).
> 
> As we are making more libraries optional, custom constructs referencing
> the class dependencies in some drivers meson.build (event/dlb2) may break.
> 
> It would be possible to add more checks in those drivers meson.build but
> it is more straightforward to not even consider a driver meson.build when
> the class dependencies are not met.
> 
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> ---
>  drivers/meson.build | 11 +++++++++++
>  1 file changed, 11 insertions(+)
> 
> diff --git a/drivers/meson.build b/drivers/meson.build
> index 74ae8cb96b..c375352e77 100644
> --- a/drivers/meson.build
> +++ b/drivers/meson.build
> @@ -70,6 +70,17 @@ foreach subpath:subdirs
>      else
>          class = subpath
>          subdir(class)
> +        skip_class = false
> +        foreach d:std_deps
> +            if not is_variable('shared_rte_' + d)
> +                skip_class = true
> +                message('Disabling all @1@ drivers: missing internal dependency "@0@"'
> +                        .format(d, class))
> +            endif
> +        endforeach
> +        if skip_class
> +            continue
> +        endif
>      endif
>  
>      # save class name on first occurrence

I like this approach. However, we do need something in the summary at the
end of the build too. Either:

* Single message stating all drivers of a given class are skipped
* (as now), message for each driver stating that it has been disabled

The former is nice as it gives us a shorter summary. The latter is nice
because it's consistent with what we have now.  Authors choice, which to go
for! :-)

Thanks.
/Bruce

Re: [PATCH] build: check drivers class dependencies early

[David Marchand]

On Tue, Aug 1, 2023 at 11:25 AM Bruce Richardson
<bruce.richardson@intel.com> wrote:
>
> On Tue, Aug 01, 2023 at 10:52:53AM +0200, David Marchand wrote:
> > Drivers implementing a class of devices (for example, drivers/event)
> > depend on the associated abstraction library (lib/eventdev).
> > This dependency is expressed in the top level meson.build for this class
> > (drivers/event/meson.build).
> >
> > As we are making more libraries optional, custom constructs referencing
> > the class dependencies in some drivers meson.build (event/dlb2) may break.
> >
> > It would be possible to add more checks in those drivers meson.build but
> > it is more straightforward to not even consider a driver meson.build when
> > the class dependencies are not met.
> >
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > ---
> >  drivers/meson.build | 11 +++++++++++
> >  1 file changed, 11 insertions(+)
> >
> > diff --git a/drivers/meson.build b/drivers/meson.build
> > index 74ae8cb96b..c375352e77 100644
> > --- a/drivers/meson.build
> > +++ b/drivers/meson.build
> > @@ -70,6 +70,17 @@ foreach subpath:subdirs
> >      else
> >          class = subpath
> >          subdir(class)
> > +        skip_class = false
> > +        foreach d:std_deps
> > +            if not is_variable('shared_rte_' + d)
> > +                skip_class = true
> > +                message('Disabling all @1@ drivers: missing internal dependency "@0@"'
> > +                        .format(d, class))
> > +            endif
> > +        endforeach
> > +        if skip_class
> > +            continue
> > +        endif
> >      endif
> >
> >      # save class name on first occurrence
>
> I like this approach. However, we do need something in the summary at the
> end of the build too. Either:
>
> * Single message stating all drivers of a given class are skipped
> * (as now), message for each driver stating that it has been disabled
>
> The former is nice as it gives us a shorter summary. The latter is nice
> because it's consistent with what we have now.  Authors choice, which to go
> for! :-)

I like the shorter summary.
WDYT of:

diff --git a/drivers/meson.build b/drivers/meson.build
index c375352e77..02268918e4 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -74,11 +74,18 @@ foreach subpath:subdirs
         foreach d:std_deps
             if not is_variable('shared_rte_' + d)
                 skip_class = true
-                message('Disabling all @1@ drivers: missing internal
dependency "@0@"'
+                reason = 'missing internal dependency, "@0@"'.format(d)
+                if dpdk_libs_deprecated.contains(d)
+                    reason += ' (deprecated lib)'
+                endif
+                message('Disabling @1@/* drivers: missing internal
dependency "@0@"'
                         .format(d, class))
             endif
         endforeach
         if skip_class
+            drv_path = join_paths(class, '*')
+            dpdk_drvs_disabled += drv_path
+            set_variable(drv_path.underscorify() + '_disable_reason', reason)
             continue
         endif
     endif


This gives the following output (testing on top of your series
extending optional libs and a fix on the new reasm perf test).
https://github.com/david-marchand/dpdk/actions/runs/5725429526/job/15513923381#step:19:346

...
=================
Content Skipped
=================
...
drivers:
...
    mempool/dpaa2: not in enabled drivers build config
    mempool/octeontx: not in enabled drivers build config
    mempool/stack: not in enabled drivers build config
    dma/*: missing internal dependency, "dmadev"
    net/af_packet: not in enabled drivers build config
    net/af_xdp: not in enabled drivers build config
    net/ark: not in enabled drivers build config
...


-- 
David Marchand

Re: [PATCH] build: check drivers class dependencies early

[Bruce Richardson]

On Tue, Aug 01, 2023 at 12:20:47PM +0200, David Marchand wrote:
> On Tue, Aug 1, 2023 at 11:25 AM Bruce Richardson
> <bruce.richardson@intel.com> wrote:
> >
> > On Tue, Aug 01, 2023 at 10:52:53AM +0200, David Marchand wrote:
> > > Drivers implementing a class of devices (for example, drivers/event)
> > > depend on the associated abstraction library (lib/eventdev).
> > > This dependency is expressed in the top level meson.build for this class
> > > (drivers/event/meson.build).
> > >
> > > As we are making more libraries optional, custom constructs referencing
> > > the class dependencies in some drivers meson.build (event/dlb2) may break.
> > >
> > > It would be possible to add more checks in those drivers meson.build but
> > > it is more straightforward to not even consider a driver meson.build when
> > > the class dependencies are not met.
> > >
> > > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > > ---
> > >  drivers/meson.build | 11 +++++++++++
> > >  1 file changed, 11 insertions(+)
> > >
> > > diff --git a/drivers/meson.build b/drivers/meson.build
> > > index 74ae8cb96b..c375352e77 100644
> > > --- a/drivers/meson.build
> > > +++ b/drivers/meson.build
> > > @@ -70,6 +70,17 @@ foreach subpath:subdirs
> > >      else
> > >          class = subpath
> > >          subdir(class)
> > > +        skip_class = false
> > > +        foreach d:std_deps
> > > +            if not is_variable('shared_rte_' + d)
> > > +                skip_class = true
> > > +                message('Disabling all @1@ drivers: missing internal dependency "@0@"'
> > > +                        .format(d, class))
> > > +            endif
> > > +        endforeach
> > > +        if skip_class
> > > +            continue
> > > +        endif
> > >      endif
> > >
> > >      # save class name on first occurrence
> >
> > I like this approach. However, we do need something in the summary at the
> > end of the build too. Either:
> >
> > * Single message stating all drivers of a given class are skipped
> > * (as now), message for each driver stating that it has been disabled
> >
> > The former is nice as it gives us a shorter summary. The latter is nice
> > because it's consistent with what we have now.  Authors choice, which to go
> > for! :-)
> 
> I like the shorter summary.
> WDYT of:
> 
> diff --git a/drivers/meson.build b/drivers/meson.build
> index c375352e77..02268918e4 100644
> --- a/drivers/meson.build
> +++ b/drivers/meson.build
> @@ -74,11 +74,18 @@ foreach subpath:subdirs
>          foreach d:std_deps
>              if not is_variable('shared_rte_' + d)
>                  skip_class = true
> -                message('Disabling all @1@ drivers: missing internal
> dependency "@0@"'
> +                reason = 'missing internal dependency, "@0@"'.format(d)
> +                if dpdk_libs_deprecated.contains(d)
> +                    reason += ' (deprecated lib)'
> +                endif
> +                message('Disabling @1@/* drivers: missing internal
> dependency "@0@"'
>                          .format(d, class))

Minor nit: You probably want a break here, as we only have room for one
missing dependency, so no point in checking more after the first missing.

>              endif
>          endforeach
>          if skip_class
> +            drv_path = join_paths(class, '*')
> +            dpdk_drvs_disabled += drv_path
> +            set_variable(drv_path.underscorify() + '_disable_reason', reason)
>              continue
>          endif
>      endif
> 
> 
> This gives the following output (testing on top of your series
> extending optional libs and a fix on the new reasm perf test).
> https://github.com/david-marchand/dpdk/actions/runs/5725429526/job/15513923381#step:19:346
> 
> ...
> =================
> Content Skipped
> =================
> ...
> drivers:
> ...
>     mempool/dpaa2: not in enabled drivers build config
>     mempool/octeontx: not in enabled drivers build config
>     mempool/stack: not in enabled drivers build config
>     dma/*: missing internal dependency, "dmadev"
>     net/af_packet: not in enabled drivers build config
>     net/af_xdp: not in enabled drivers build config
>     net/ark: not in enabled drivers build config
> ...
> 

+1 to this. Works for me.

RE: [PATCH] build: check drivers class dependencies early

[Morten Brørup]

> From: David Marchand [mailto:david.marchand@redhat.com]
> Sent: Tuesday, 1 August 2023 12.21
> 
> On Tue, Aug 1, 2023 at 11:25 AM Bruce Richardson
> <bruce.richardson@intel.com> wrote:
> >
> > On Tue, Aug 01, 2023 at 10:52:53AM +0200, David Marchand wrote:
> > > Drivers implementing a class of devices (for example, drivers/event)
> > > depend on the associated abstraction library (lib/eventdev).
> > > This dependency is expressed in the top level meson.build for this class
> > > (drivers/event/meson.build).
> > >
> > > As we are making more libraries optional, custom constructs referencing
> > > the class dependencies in some drivers meson.build (event/dlb2) may break.
> > >
> > > It would be possible to add more checks in those drivers meson.build but
> > > it is more straightforward to not even consider a driver meson.build when
> > > the class dependencies are not met.
> > >
> > > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > > ---
> 
> This gives the following output (testing on top of your series
> extending optional libs and a fix on the new reasm perf test).
> https://github.com/david-
> marchand/dpdk/actions/runs/5725429526/job/15513923381#step:19:346
> 
> ...
> =================
> Content Skipped
> =================
> ...
> drivers:
> ...
>     mempool/dpaa2: not in enabled drivers build config
>     mempool/octeontx: not in enabled drivers build config
>     mempool/stack: not in enabled drivers build config
>     dma/*: missing internal dependency, "dmadev"
>     net/af_packet: not in enabled drivers build config
>     net/af_xdp: not in enabled drivers build config
>     net/ark: not in enabled drivers build config
> ...
> 

For the concept suggested by David (and preferably with Bruce's output):

Acked-by: Morten Brørup <mb@smartsharesystems.com>

RE: [PATCH] build: check drivers class dependencies early

[Morten Brørup]

> From: Morten Brørup [mailto:mb@smartsharesystems.com]
> Sent: Tuesday, 1 August 2023 12.49
> 
> > From: David Marchand [mailto:david.marchand@redhat.com]
> > Sent: Tuesday, 1 August 2023 12.21
> >
> > On Tue, Aug 1, 2023 at 11:25 AM Bruce Richardson
> > <bruce.richardson@intel.com> wrote:
> > >
> > > On Tue, Aug 01, 2023 at 10:52:53AM +0200, David Marchand wrote:
> > > > Drivers implementing a class of devices (for example, drivers/event)
> > > > depend on the associated abstraction library (lib/eventdev).
> > > > This dependency is expressed in the top level meson.build for this class
> > > > (drivers/event/meson.build).
> > > >
> > > > As we are making more libraries optional, custom constructs referencing
> > > > the class dependencies in some drivers meson.build (event/dlb2) may
> break.
> > > >
> > > > It would be possible to add more checks in those drivers meson.build but
> > > > it is more straightforward to not even consider a driver meson.build
> when
> > > > the class dependencies are not met.
> > > >
> > > > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > > > ---
> >
> > This gives the following output (testing on top of your series
> > extending optional libs and a fix on the new reasm perf test).
> > https://github.com/david-
> > marchand/dpdk/actions/runs/5725429526/job/15513923381#step:19:346
> >
> > ...
> > =================
> > Content Skipped
> > =================
> > ...
> > drivers:
> > ...
> >     mempool/dpaa2: not in enabled drivers build config
> >     mempool/octeontx: not in enabled drivers build config
> >     mempool/stack: not in enabled drivers build config
> >     dma/*: missing internal dependency, "dmadev"
> >     net/af_packet: not in enabled drivers build config
> >     net/af_xdp: not in enabled drivers build config
> >     net/ark: not in enabled drivers build config
> > ...
> >
> 
> For the concept suggested by David (and preferably with Bruce's output):

Sorry... This was David's suggestion, which I prefer. I thought I was replying to Bruce's reply.

> 
> Acked-by: Morten Brørup <mb@smartsharesystems.com>

My ACK still valid.

[PATCH v2] build: check drivers class dependencies early

[David Marchand]

Drivers implementing a class of devices (for example, drivers/event)
depend on the associated abstraction library (lib/eventdev).
This dependency is expressed in the top level meson.build for this class
(drivers/event/meson.build).

As we are making more libraries optional, custom constructs referencing
the class dependencies in some drivers meson.build (event/dlb2) may break.

It would be possible to add more checks in those drivers meson.build but
it is more straightforward to not even consider a driver meson.build when
the class dependencies are not met.

Signed-off-by: David Marchand <david.marchand@redhat.com>
Acked-by: Morten Brørup <mb@smartsharesystems.com>
---
Changes since v1:
- added break on the first missing dependency,
- added logs for disabled drivers,

---
 drivers/meson.build | 19 +++++++++++++++++++
 1 file changed, 19 insertions(+)

diff --git a/drivers/meson.build b/drivers/meson.build
index 74ae8cb96b..c909070c30 100644
--- a/drivers/meson.build
+++ b/drivers/meson.build
@@ -70,6 +70,25 @@ foreach subpath:subdirs
     else
         class = subpath
         subdir(class)
+        skip_class = false
+        foreach d:std_deps
+            if not is_variable('shared_rte_' + d)
+                skip_class = true
+                reason = 'missing internal dependency, "@0@"'.format(d)
+                if dpdk_libs_deprecated.contains(d)
+                    reason += ' (deprecated lib)'
+                endif
+                message('Disabling @1@/* drivers: missing internal dependency "@0@"'
+                        .format(d, class))
+                break
+            endif
+        endforeach
+        if skip_class
+            drv_path = join_paths(class, '*')
+            dpdk_drvs_disabled += drv_path
+            set_variable(drv_path.underscorify() + '_disable_reason', reason)
+            continue
+        endif
     endif
 
     # save class name on first occurrence
-- 
2.41.0

Re: [PATCH v2] build: check drivers class dependencies early

[Bruce Richardson]

On Tue, Aug 01, 2023 at 03:41:33PM +0200, David Marchand wrote:
> Drivers implementing a class of devices (for example, drivers/event)
> depend on the associated abstraction library (lib/eventdev).
> This dependency is expressed in the top level meson.build for this class
> (drivers/event/meson.build).
> 
> As we are making more libraries optional, custom constructs referencing
> the class dependencies in some drivers meson.build (event/dlb2) may break.
> 
> It would be possible to add more checks in those drivers meson.build but
> it is more straightforward to not even consider a driver meson.build when
> the class dependencies are not met.
> 
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> Acked-by: Morten Brørup <mb@smartsharesystems.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>

Re: [PATCH v2] build: check drivers class dependencies early

[Tyler Retzlaff]

On Tue, Aug 01, 2023 at 03:01:31PM +0100, Bruce Richardson wrote:
> On Tue, Aug 01, 2023 at 03:41:33PM +0200, David Marchand wrote:
> > Drivers implementing a class of devices (for example, drivers/event)
> > depend on the associated abstraction library (lib/eventdev).
> > This dependency is expressed in the top level meson.build for this class
> > (drivers/event/meson.build).
> > 
> > As we are making more libraries optional, custom constructs referencing
> > the class dependencies in some drivers meson.build (event/dlb2) may break.
> > 
> > It would be possible to add more checks in those drivers meson.build but
> > it is more straightforward to not even consider a driver meson.build when
> > the class dependencies are not met.
> > 
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > Acked-by: Morten Brørup <mb@smartsharesystems.com>
> Acked-by: Bruce Richardson <bruce.richardson@intel.com>

Acked-by: Tyler Retzlaff <roretzla@linux.microsoft.com>

Re: [PATCH v2] build: check drivers class dependencies early

[David Marchand]

On Tue, Aug 1, 2023 at 3:41 PM David Marchand <david.marchand@redhat.com> wrote:
>
> Drivers implementing a class of devices (for example, drivers/event)
> depend on the associated abstraction library (lib/eventdev).
> This dependency is expressed in the top level meson.build for this class
> (drivers/event/meson.build).
>
> As we are making more libraries optional, custom constructs referencing
> the class dependencies in some drivers meson.build (event/dlb2) may break.
>
> It would be possible to add more checks in those drivers meson.build but
> it is more straightforward to not even consider a driver meson.build when
> the class dependencies are not met.
>
> Signed-off-by: David Marchand <david.marchand@redhat.com>
> Acked-by: Morten Brørup <mb@smartsharesystems.com>
Acked-by: Bruce Richardson <bruce.richardson@intel.com>
Acked-by: Tyler Retzlaff <roretzla@linux.microsoft.com>

Applied, thanks.

Bruce, could you rebase your series marking more libraries optional?
There is some conflicts after removal of kni/flow_classify, and adding
a unit test (perf reasm).


-- 
David Marchand

Re: [PATCH v2] build: check drivers class dependencies early

[Bruce Richardson]

On Tue, Aug 08, 2023 at 09:43:25AM +0200, David Marchand wrote:
> On Tue, Aug 1, 2023 at 3:41 PM David Marchand <david.marchand@redhat.com> wrote:
> >
> > Drivers implementing a class of devices (for example, drivers/event)
> > depend on the associated abstraction library (lib/eventdev).
> > This dependency is expressed in the top level meson.build for this class
> > (drivers/event/meson.build).
> >
> > As we are making more libraries optional, custom constructs referencing
> > the class dependencies in some drivers meson.build (event/dlb2) may break.
> >
> > It would be possible to add more checks in those drivers meson.build but
> > it is more straightforward to not even consider a driver meson.build when
> > the class dependencies are not met.
> >
> > Signed-off-by: David Marchand <david.marchand@redhat.com>
> > Acked-by: Morten Brørup <mb@smartsharesystems.com>
> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
> Acked-by: Tyler Retzlaff <roretzla@linux.microsoft.com>
> 
> Applied, thanks.
> 
> Bruce, could you rebase your series marking more libraries optional?
> There is some conflicts after removal of kni/flow_classify, and adding
> a unit test (perf reasm).
>
Yes, I plan to do so. However, I think I'd rather defer that set until
after the set reworking the build of the tests completely. It makes
everything a lot simpler in that regard. In fact, I may merge the two sets
into one.

/Bruce