[dpdk-dev] Heads up: Fedora packaging plans

[dpdk-dev] Heads up: Fedora packaging plans

[Neil Horman]

Hey all-
	This isn't really germaine to dpdk development, but Thomas and Vincent,
you expressed interest in my progress regarding packaging of dpdk for Fedora, so
I figured I would post here in case others were interested.

Please find here:
http://people.redhat.com/nhorman/dpdk-1.7.0-0.1.gitb20539d68.src.rpm

My current effort to do so.  I've made some changes from the stock spec file
included in dpdk:

* Modified the version and release values to be separate from the name.  I did
some reading on requirements for packaging and it seems we can be a bit more lax
with ABI version on a pre-release I think, so I setup the N-V-R to use
pre-release conventions, which makes sense, give that this is a 1.7.0
pre-release.  The git tag on the relase value will get bumped as we move forward
in the patch series.

* Added config files to match desired configs for Fedora (i.e. disabled PMD's
that require out of tree kernel modules

* Removed Packager tag (Fedora doesn't use those)

* Moved the package target directories to include N-V of the package in the path
names.  This allows for multiple versions of the dpdk to be installed in
parallel (I.e. dpdk-1.7.0 files are in /lib/dpdk-1.7.0, /usr/include/dpdk-1.7.0,
etc).  This is how java packages allow for multiple version installs, and makes
sense given ABI instability in dpdk.  It will require that developers add some
-I / -L paths to their makefiles to pull the proper version, but I think thats a
fair tradeoff.

My plan is to go through the review process with this package, and update to
tagged 1.7.0 as soon as its ready.  

Neil
 

Re: [dpdk-dev] Heads up: Fedora packaging plans

[Vincent JARDIN]

Neil,

 > Please find here:
 >   http://people.redhat.com/nhorman/dpdk-1.7.0-0.1.gitb20539d68.src.rpm

Your spec file is broken:

A simple patch:
--- dpdk.spec	2014-05-13 22:43:15.892000000 +0000
+++ ../dpdk.spec.vj	2014-05-13 22:42:40.221000000 +0000
@@ -75,7 +75,7 @@

  %build
  make O=%{target} T=%{target} config
-sed -ri 's,(RTE_MACHINE=).*,\1%{machine},  %{target}/.config
+sed -ri 's,(RTE_MACHINE=).*,\1%{machine},' %{target}/.config
  sed -ri 's,(RTE_APP_TEST=).*,\1n,'         %{target}/.config
  sed -ri 's,(RTE_BUILD_SHARED_LIB=).*,\1y,' %{target}/.config
  make O=%{target} %{?_smp_mflags}

Best regards,
   Vincent

Re: [dpdk-dev] Heads up: Fedora packaging plans

[Neil Horman]

On Wed, May 14, 2014 at 12:46:38AM +0200, Vincent JARDIN wrote:
> Neil,
> 
> > Please find here:
> >   http://people.redhat.com/nhorman/dpdk-1.7.0-0.1.gitb20539d68.src.rpm
> 
> Your spec file is broken:
> 
> A simple patch:
> --- dpdk.spec	2014-05-13 22:43:15.892000000 +0000
> +++ ../dpdk.spec.vj	2014-05-13 22:42:40.221000000 +0000
> @@ -75,7 +75,7 @@
> 
>  %build
>  make O=%{target} T=%{target} config
> -sed -ri 's,(RTE_MACHINE=).*,\1%{machine},  %{target}/.config
> +sed -ri 's,(RTE_MACHINE=).*,\1%{machine},' %{target}/.config
>  sed -ri 's,(RTE_APP_TEST=).*,\1n,'         %{target}/.config
>  sed -ri 's,(RTE_BUILD_SHARED_LIB=).*,\1y,' %{target}/.config
>  make O=%{target} %{?_smp_mflags}
> 
> Best regards,
>   Vincent
> 
Thank you, that should actually be entirely removed though.  We don't need to be
altering the configuration at all since I dropped in complete new config files
as sources .

Re: [dpdk-dev] Heads up: Fedora packaging plans

[Thomas Monjalon]

Hi Neil,

Thanks for sharing your progress.

My main concerns are about naming and extensions.
We must keep "dpdk-core" naming in order to distinguish it from PMD 
extensions. And then, packaging of memnic and non-uio paravirtualization PMDs 
(virtio/vmxnet3) are missing.

2014-05-13 15:08, Neil Horman:
> My current effort to do so.  I've made some changes from the stock spec file
> included in dpdk:

We should try to get .spec for Fedora and in-tree .spec as common as possible.
There are probably some things to push.

> * Modified the version and release values to be separate from the name.  I
> did some reading on requirements for packaging and it seems we can be a bit
> more lax with ABI version on a pre-release I think, so I setup the N-V-R to
> use pre-release conventions, which makes sense, give that this is a 1.7.0
> pre-release.  The git tag on the relase value will get bumped as we move
> forward in the patch series.

I thought that we should put version in the name, in order to be able to 
install many versions together. How is it handled by yum?

> * Added config files to match desired configs for Fedora (i.e. disabled
> PMD's that require out of tree kernel modules

It would be clearer to make your configuration changes with "sed -i".
In a near future we would probably need a "configure" script to do it.

So you don't package igb_uio but you build it because there is no option to 
disable it currently. We should add such option.

> * Moved the package target directories to include N-V of the package in the
> path names.  This allows for multiple versions of the dpdk to be installed
> in parallel (I.e. dpdk-1.7.0 files are in /lib/dpdk-1.7.0,
> /usr/include/dpdk-1.7.0, etc).  This is how java packages allow for
> multiple version installs, and makes sense given ABI instability in dpdk. 
> It will require that developers add some -I / -L paths to their makefiles
> to pull the proper version, but I think thats a fair tradeoff.

I don't see version for include directory and bin directory (testpmd).

Thanks
-- 
Thomas

Re: [dpdk-dev] Heads up: Fedora packaging plans

[Neil Horman]

On Mon, May 19, 2014 at 12:11:35PM +0200, Thomas Monjalon wrote:
> Hi Neil,
> 
> Thanks for sharing your progress.
> 
No worries.

> My main concerns are about naming and extensions.
> We must keep "dpdk-core" naming in order to distinguish it from PMD 
> extensions.
I don't see why.  We can name packages whatever we want, as long as the spec and
srpm share the same name. It seems to me that the core should be the base name
of the package while the extensions should have some extension on their name.

> And then, packaging of memnic and non-uio paravirtualization PMDs 
> (virtio/vmxnet3) are missing.
> 
They're in separate repositories, I was planning on packaging them at a later
time separately, since their versioning and development is handled separately.

> 2014-05-13 15:08, Neil Horman:
> > My current effort to do so.  I've made some changes from the stock spec file
> > included in dpdk:
> 
> We should try to get .spec for Fedora and in-tree .spec as common as possible.
> There are probably some things to push.
> 
Ok, sure, just keep in mind that different distributions have different
packaging requirements that may affect the contents of the spec file, and so
attaining parity may not be possible (or even worthwhile).

> > * Modified the version and release values to be separate from the name.  I
> > did some reading on requirements for packaging and it seems we can be a bit
> > more lax with ABI version on a pre-release I think, so I setup the N-V-R to
> > use pre-release conventions, which makes sense, give that this is a 1.7.0
> > pre-release.  The git tag on the relase value will get bumped as we move
> > forward in the patch series.
> 
> I thought that we should put version in the name, in order to be able to 
> install many versions together. How is it handled by yum?
> 
So, I spent some time thinking about this, and I _really_ want to avoid the
inclusion of a version with the package name.  Doing so, while it allows yum to
install multiple versions side-by-side, is a real overhead for me, as it
requires that I go through a new pacakge review process for each released
version that we want to package.  I do not have time to do that.  If someone
from 6wind or intel wants to get involved in the Packaging process we can look
at that as a solution, but while I'm doing it, its really just too much
overhead.  This method will allow multiple version to be installed side by side
as well.  The tradeoff is that yum doesn't directly allow that, as it will just
preform an upgrade.  The multiple version solution will require that you
download older versions and install them directly using rpm commands.  I think
thats a fair tradeoff.

> > * Added config files to match desired configs for Fedora (i.e. disabled
> > PMD's that require out of tree kernel modules
> 
> It would be clearer to make your configuration changes with "sed -i".
> In a near future we would probably need a "configure" script to do it.
> 
I really disagree.  Its not clearer in my mind at all - in that the final config
file is a product of two pieces of information (the base config file, and the
sed scripts that you run on it), as opposed to one piece (the canonical modified
config specified in the source line).  Using sed also implies that you need to
list sed as a BuildRequires (minimal buildroots may not include sed when they
are spun up).

> So you don't package igb_uio but you build it because there is no option to 
> disable it currently. We should add such option.
> 
Not sure what you mean here.  The only uio code I see in the package is the uio
unbind script for igb, which should still work just fine (save for the fact that
we don't have a user space PMD to attach the hardware to).  I can certainly
remove the script though so it doesn't appear in the package until such time as
the LAD group integrates the uio code in the upstream driver.

> > * Moved the package target directories to include N-V of the package in the
> > path names.  This allows for multiple versions of the dpdk to be installed
> > in parallel (I.e. dpdk-1.7.0 files are in /lib/dpdk-1.7.0,
> > /usr/include/dpdk-1.7.0, etc).  This is how java packages allow for
> > multiple version installs, and makes sense given ABI instability in dpdk. 
> > It will require that developers add some -I / -L paths to their makefiles
> > to pull the proper version, but I think thats a fair tradeoff.
> 
> I don't see version for include directory and bin directory (testpmd).
> 
Yup, need to fix that.  Thank you!
Neil

> Thanks
> -- 
> Thomas
> 

Re: [dpdk-dev] Heads up: Fedora packaging plans

[Thomas Monjalon]

2014-05-19 09:18, Neil Horman:
> On Mon, May 19, 2014 at 12:11:35PM +0200, Thomas Monjalon wrote:
> > My main concerns are about naming and extensions.
> > We must keep "dpdk-core" naming in order to distinguish it from PMD
> > extensions.
> 
> I don't see why.  We can name packages whatever we want, as long as the spec
> and srpm share the same name. It seems to me that the core should be the
> base name of the package while the extensions should have some extension on
> their name.

OK

> > And then, packaging of memnic and non-uio paravirtualization PMDs
> > (virtio/vmxnet3) are missing.
> 
> They're in separate repositories, I was planning on packaging them at a
> later time separately, since their versioning and development is handled
> separately.

OK

> > We should try to get .spec for Fedora and in-tree .spec as common as
> > possible. There are probably some things to push.
> 
> Ok, sure, just keep in mind that different distributions have different
> packaging requirements that may affect the contents of the spec file, and so
> attaining parity may not be possible (or even worthwhile).

Yes, the in-tree file should be a common base.

> > I thought that we should put version in the name, in order to be able to
> > install many versions together. How is it handled by yum?
> 
> So, I spent some time thinking about this, and I _really_ want to avoid the
> inclusion of a version with the package name.  Doing so, while it allows yum
> to install multiple versions side-by-side, is a real overhead for me, as it
> requires that I go through a new pacakge review process for each released
> version that we want to package.  I do not have time to do that.  If
> someone from 6wind or intel wants to get involved in the Packaging process
> we can look at that as a solution, but while I'm doing it, its really just
> too much overhead.  This method will allow multiple version to be installed
> side by side as well.  The tradeoff is that yum doesn't directly allow
> that, as it will just preform an upgrade.  The multiple version solution
> will require that you download older versions and install them directly
> using rpm commands.  I think thats a fair tradeoff.

OK

> > > * Added config files to match desired configs for Fedora (i.e. disabled
> > > PMD's that require out of tree kernel modules
> > 
> > It would be clearer to make your configuration changes with "sed -i".
> > In a near future we would probably need a "configure" script to do it.
> 
> I really disagree.  Its not clearer in my mind at all - in that the final
> config file is a product of two pieces of information (the base config
> file, and the sed scripts that you run on it), as opposed to one piece (the
> canonical modified config specified in the source line).  Using sed also
> implies that you need to list sed as a BuildRequires (minimal buildroots
> may not include sed when they are spun up).

I don't understand the logic.
When packaging a library using autotools, you are setting some options to 
"configure" script which are not saved elsewhere than in the spec file, right?
Why it wouldn't be the same here (options in spec file)? 

> > So you don't package igb_uio but you build it because there is no option
> > to disable it currently. We should add such option.
> 
> Not sure what you mean here.  The only uio code I see in the package is the
> uio unbind script for igb, which should still work just fine (save for the
> fact that we don't have a user space PMD to attach the hardware to).  I can
> certainly remove the script though so it doesn't appear in the package
> until such time as the LAD group integrates the uio code in the upstream
> driver.

I'm speaking about lib/librte_eal/linuxapp/igb_uio.
I think you are building it and it would fail if you no kernel header 
installed.

-- 
Thomas

Re: [dpdk-dev] Heads up: Fedora packaging plans

[Neil Horman]

On Mon, May 19, 2014 at 06:28:47PM +0200, Thomas Monjalon wrote:
> 2014-05-19 09:18, Neil Horman:
> > On Mon, May 19, 2014 at 12:11:35PM +0200, Thomas Monjalon wrote:
> > > My main concerns are about naming and extensions.
> > > We must keep "dpdk-core" naming in order to distinguish it from PMD
> > > extensions.
> > 
> > I don't see why.  We can name packages whatever we want, as long as the spec
> > and srpm share the same name. It seems to me that the core should be the
> > base name of the package while the extensions should have some extension on
> > their name.
> 
> OK
> 
> > > And then, packaging of memnic and non-uio paravirtualization PMDs
> > > (virtio/vmxnet3) are missing.
> > 
> > They're in separate repositories, I was planning on packaging them at a
> > later time separately, since their versioning and development is handled
> > separately.
> 
> OK
> 
> > > We should try to get .spec for Fedora and in-tree .spec as common as
> > > possible. There are probably some things to push.
> > 
> > Ok, sure, just keep in mind that different distributions have different
> > packaging requirements that may affect the contents of the spec file, and so
> > attaining parity may not be possible (or even worthwhile).
> 
> Yes, the in-tree file should be a common base.
> 
> > > I thought that we should put version in the name, in order to be able to
> > > install many versions together. How is it handled by yum?
> > 
> > So, I spent some time thinking about this, and I _really_ want to avoid the
> > inclusion of a version with the package name.  Doing so, while it allows yum
> > to install multiple versions side-by-side, is a real overhead for me, as it
> > requires that I go through a new pacakge review process for each released
> > version that we want to package.  I do not have time to do that.  If
> > someone from 6wind or intel wants to get involved in the Packaging process
> > we can look at that as a solution, but while I'm doing it, its really just
> > too much overhead.  This method will allow multiple version to be installed
> > side by side as well.  The tradeoff is that yum doesn't directly allow
> > that, as it will just preform an upgrade.  The multiple version solution
> > will require that you download older versions and install them directly
> > using rpm commands.  I think thats a fair tradeoff.
> 
> OK
> 
> > > > * Added config files to match desired configs for Fedora (i.e. disabled
> > > > PMD's that require out of tree kernel modules
> > > 
> > > It would be clearer to make your configuration changes with "sed -i".
> > > In a near future we would probably need a "configure" script to do it.
> > 
> > I really disagree.  Its not clearer in my mind at all - in that the final
> > config file is a product of two pieces of information (the base config
> > file, and the sed scripts that you run on it), as opposed to one piece (the
> > canonical modified config specified in the source line).  Using sed also
> > implies that you need to list sed as a BuildRequires (minimal buildroots
> > may not include sed when they are spun up).
> 
> I don't understand the logic.
> When packaging a library using autotools, you are setting some options to 
> "configure" script which are not saved elsewhere than in the spec file, right?
> Why it wouldn't be the same here (options in spec file)? 
> 
When you need to override options that can't be auto-detected, then yes.  In
that case, you can override options that can't be automatically determined from
the build environment.  Thats a common use case and its well understood by
everyone, which is fine.

But thats not what you have here.  DPDK has a build environment that more akin
to the linux kernel, in that, from an rpm build perspective a flat file is
inserted as the configuration.  the spec file just drops it in the proper place
and uses it, unmodified.  Theres no expectation that, when a configuration is
completely supplied, that it will be modified prior to build time.

Its the difference between static and dynamic generation of configuration data.
If you're using autoconf, then you know that you don't have a set configuration
until you run ./configure, which means you know to look to the spec file to
determine any options for building, and you know that your configuration will be
generated after the spec file runs configure, whatever it is.  If conversely you
have a static configuration file (as DPDK does), the assumption is that the
selected configuration is included in the source package and is unaltered at
build time.

> > > So you don't package igb_uio but you build it because there is no option
> > > to disable it currently. We should add such option.
> > 
> > Not sure what you mean here.  The only uio code I see in the package is the
> > uio unbind script for igb, which should still work just fine (save for the
> > fact that we don't have a user space PMD to attach the hardware to).  I can
> > certainly remove the script though so it doesn't appear in the package
> > until such time as the LAD group integrates the uio code in the upstream
> > driver.
> 
> I'm speaking about lib/librte_eal/linuxapp/igb_uio.
> I think you are building it and it would fail if you no kernel header 
> installed.
> 
Thats true, but you'll always have those header files because, at least for
fedora uio_driver.h is packaged as part of the kernel-devel package, which is a
BuildRequires: depedency for dpdk.  Note that may vary from distro to distro, so
my assertion isn't always true (see previous note about pushing changes to
upstream spec files).  It is safe for Fedora however.


That all said, I noted a patch series to convert DPDK to VFIO, which I am highly
supportive of, which I think would also affect this in that it would modify this
script (or arguably obviate it), as I think there is a common vfio script that
can detach any pci device.

Neil

> -- 
> Thomas
>