[PATCH] doc: update minimum Linux kernel version

[PATCH] doc: update minimum Linux kernel version

[Stephen Hemminger]

The last version of 4.14 kernel was just released (4.14.336),
and it is now end of life. Update the DPDK kernel minimum version
to the next LTS kernel version.

Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
---
 doc/guides/linux_gsg/sys_reqs.rst | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/doc/guides/linux_gsg/sys_reqs.rst b/doc/guides/linux_gsg/sys_reqs.rst
index 13be715933f4..9c5282573e4a 100644
--- a/doc/guides/linux_gsg/sys_reqs.rst
+++ b/doc/guides/linux_gsg/sys_reqs.rst
@@ -106,7 +106,7 @@ System Software
 
 **Required:**
 
-*   Kernel version >= 4.14
+*   Kernel version >= 4.19
 
     The kernel version required is based on the oldest long term stable kernel available
     at kernel.org when the DPDK version is in development.
-- 
2.43.0

RE: [PATCH] doc: update minimum Linux kernel version

[Morten Brørup]

> From: Stephen Hemminger [mailto:stephen@networkplumber.org]
> Sent: Wednesday, 10 January 2024 17.58
> 
> The last version of 4.14 kernel was just released (4.14.336),
> and it is now end of life. Update the DPDK kernel minimum version
> to the next LTS kernel version.
> 
> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
> ---
>  doc/guides/linux_gsg/sys_reqs.rst | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/doc/guides/linux_gsg/sys_reqs.rst
> b/doc/guides/linux_gsg/sys_reqs.rst
> index 13be715933f4..9c5282573e4a 100644
> --- a/doc/guides/linux_gsg/sys_reqs.rst
> +++ b/doc/guides/linux_gsg/sys_reqs.rst
> @@ -106,7 +106,7 @@ System Software
> 
>  **Required:**
> 
> -*   Kernel version >= 4.14
> +*   Kernel version >= 4.19
> 
>      The kernel version required is based on the oldest long term
> stable kernel available
>      at kernel.org when the DPDK version is in development.
> --
> 2.43.0

I wonder if any automated tests are using this specific kernel version, or if we are only testing using the distros' native kernels. @Aaron?

Regardless, the documented version requirement needs updating, so...

Acked-by: Morten Brørup <mb@smartsharesystems.com>

Re: [PATCH] doc: update minimum Linux kernel version

[Aaron Conole]

Morten Brørup <mb@smartsharesystems.com> writes:

>> From: Stephen Hemminger [mailto:stephen@networkplumber.org]
>> Sent: Wednesday, 10 January 2024 17.58
>> 
>> The last version of 4.14 kernel was just released (4.14.336),
>> and it is now end of life. Update the DPDK kernel minimum version
>> to the next LTS kernel version.
>> 
>> Signed-off-by: Stephen Hemminger <stephen@networkplumber.org>
>> ---
>>  doc/guides/linux_gsg/sys_reqs.rst | 2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>> 
>> diff --git a/doc/guides/linux_gsg/sys_reqs.rst
>> b/doc/guides/linux_gsg/sys_reqs.rst
>> index 13be715933f4..9c5282573e4a 100644
>> --- a/doc/guides/linux_gsg/sys_reqs.rst
>> +++ b/doc/guides/linux_gsg/sys_reqs.rst
>> @@ -106,7 +106,7 @@ System Software
>> 
>>  **Required:**
>> 
>> -*   Kernel version >= 4.14
>> +*   Kernel version >= 4.19
>> 
>>      The kernel version required is based on the oldest long term
>> stable kernel available
>>      at kernel.org when the DPDK version is in development.
>> --
>> 2.43.0
>
> I wonder if any automated tests are using this specific kernel
> version, or if we are only testing using the distros' native
> kernels. @Aaron?

I believe that we generally try to keep to the distro version.  There
are some exceptions for specific platforms (that is to support specific
pieces of hardware, but those are very recent).

> Regardless, the documented version requirement needs updating, so...
>
> Acked-by: Morten Brørup <mb@smartsharesystems.com>

Re: [PATCH] doc: update minimum Linux kernel version

[Patrick Robb]

[-- Attachment #1: Type: text/plain, Size: 622 bytes --]

On Thu, Jan 11, 2024 at 4:18 AM Morten Brørup <mb@smartsharesystems.com>
wrote:

>
> I wonder if any automated tests are using this specific kernel version, or
> if we are only testing using the distros' native kernels. @Aaron?
>

For UNH, we generally run from the distros' native kernels. Any exceptions
are not for testing older kernel versions, so we don't have anything
running from 4.14 in our testing right now.

If running some automated testing for, say, the minimum supported kernel
version at any point in time is a value to anyone, certainly they can speak
up and we can discuss adding that.

[-- Attachment #2: Type: text/html, Size: 994 bytes --]

RE: [PATCH] doc: update minimum Linux kernel version

[Morten Brørup]

[-- Attachment #1: Type: text/plain, Size: 1605 bytes --]

From: Patrick Robb [mailto:probb@iol.unh.edu] 
Sent: Thursday, 11 January 2024 20.02



 

On Thu, Jan 11, 2024 at 4:18 AM Morten Brørup <mb@smartsharesystems.com> wrote:

	
	I wonder if any automated tests are using this specific kernel version, or if we are only testing using the distros' native kernels. @Aaron?

 

For UNH, we generally run from the distros' native kernels. Any exceptions are not for testing older kernel versions, so we don't have anything running from 4.14 in our testing right now. 

 

If running some automated testing for, say, the minimum supported kernel version at any point in time is a value to anyone, certainly they can speak up and we can discuss adding that.  

 

 

When the documentation specifies a minimum required kernel version, it implicitly claims that DPDK works with that kernel version.

So we should either test with the specified kernel version (which requires significant effort to set up, so I’m not going to ask for it!), or add a big fat disclaimer/warning that DPDK is not tested with the mentioned kernel version, and list the kernel versions actually tested.

 

As an appliance vendor, we build our systems from scratch, including the bootloader, kernel and file systems. We don’t use any of the distro stuff.

Having information about well tested kernel versions would be helpful when choosing kernel version for our appliances. I suppose other appliance vendors also use their own hardened/purpose-built kernel versions, and would consider this information useful for their decision process too.

 


[-- Attachment #2: Type: text/html, Size: 4929 bytes --]

Re: [PATCH] doc: update minimum Linux kernel version

[Patrick Robb]

[-- Attachment #1: Type: text/plain, Size: 2600 bytes --]

On Thu, Jan 11, 2024 at 2:27 PM Morten Brørup <mb@smartsharesystems.com>
wrote:

> *From:* Patrick Robb [mailto:probb@iol.unh.edu]
> *Sent:* Thursday, 11 January 2024 20.02
>
>
>
> On Thu, Jan 11, 2024 at 4:18 AM Morten Brørup <mb@smartsharesystems.com>
> wrote:
>
>
> I wonder if any automated tests are using this specific kernel version, or
> if we are only testing using the distros' native kernels. @Aaron?
>
>
>
> For UNH, we generally run from the distros' native kernels. Any exceptions
> are not for testing older kernel versions, so we don't have anything
> running from 4.14 in our testing right now.
>
>
>
> If running some automated testing for, say, the minimum supported kernel
> version at any point in time is a value to anyone, certainly they can speak
> up and we can discuss adding that.
>
>
>
>
>
> When the documentation specifies a minimum required kernel version, it
> implicitly claims that DPDK works with that kernel version.
>
> So we should either test with the specified kernel version (which requires
> significant effort to set up, so I’m not going to ask for it!), or add a
> big fat disclaimer/warning that DPDK is not tested with the mentioned
> kernel version, and list the kernel versions actually tested.
>
Well, adding one system which moves with the minimum supported kernel
version probably wouldn't be too onerous, so I will add a ticket noting
this as a community request. On the other hand, one of the reasons we're
moving to running more and more of our testing from containers is so that
we don't have to do as much VM "babysitting" and our testing environment is
more cleanly defined. Obviously that doesn't apply in this case, so we'd
set up a custom VM for this testing job specifically. But, as an LTS kernel
version reaches EOL approximately 1x per year, it shouldn't be too bad.


>
>
> As an appliance vendor, we build our systems from scratch, including the
> bootloader, kernel and file systems. We don’t use any of the distro stuff.
>
> Having information about well tested kernel versions would be helpful when
> choosing kernel version for our appliances. I suppose other appliance
> vendors also use their own hardened/purpose-built kernel versions, and
> would consider this information useful for their decision process too.
>
>
>
Maybe the best thing we can do without a significant overhaul of our
process is to more explicitly display the kernel version for a system when
reporting results. If others chime in and there is big interest here, we
can go further.

[-- Attachment #2: Type: text/html, Size: 5561 bytes --]

Re: [PATCH] doc: update minimum Linux kernel version

[Stephen Hemminger]

On Thu, 11 Jan 2024 20:26:56 +0100
Morten Brørup <mb@smartsharesystems.com> wrote:

>  
> 
> When the documentation specifies a minimum required kernel version, it implicitly claims that DPDK works with that kernel version.
> 
> So we should either test with the specified kernel version (which requires significant effort to set up, so I’m not going to ask for it!), or add a big fat disclaimer/warning that DPDK is not tested with the mentioned kernel version, and list the kernel versions actually tested.

It is much less of an issue than it used to be since there should be no need for
DPDK specific kernel components. The kernel API/ABI is stable across releases 
(with the notable exception of BPF). Therefore the DPDK kernel version dependency
is much less than it used to be.

Poking around the source, still see some leftovers.

Like quick assist documentation wanting kernel version for sources?
Why does QAT depend on kernel source being present. Thats not right.

The other bad spot is the Intel GVE driver reading and passing
the OS version in. Looks like some host side validation.

Putting OS version anywhere in DPDK is a mistake.

RE: [PATCH] doc: update minimum Linux kernel version

[Morten Brørup]

> From: Stephen Hemminger [mailto:stephen@networkplumber.org]
> Sent: Thursday, 11 January 2024 20.55
> 
> On Thu, 11 Jan 2024 20:26:56 +0100
> Morten Brørup <mb@smartsharesystems.com> wrote:
> 
> >
> >
> > When the documentation specifies a minimum required kernel version,
> it implicitly claims that DPDK works with that kernel version.
> >
> > So we should either test with the specified kernel version (which
> requires significant effort to set up, so I’m not going to ask for
> it!), or add a big fat disclaimer/warning that DPDK is not tested with
> the mentioned kernel version, and list the kernel versions actually
> tested.
> 
> It is much less of an issue than it used to be since there should be no
> need for
> DPDK specific kernel components. The kernel API/ABI is stable across
> releases
> (with the notable exception of BPF). Therefore the DPDK kernel version
> dependency
> is much less than it used to be.

That is my experience too.
Your BPF exception is a great example of the Devil being in the details!
Last time we upgraded the kernel version in our appliances (because we needed a feature not available in the old kernel we were using), we ran into two major problems; the first was so bad that we eventually gave up on debugging it and moved on to yet a newer kernel version; the second was a well documented API change, and thus relatively easy to fix (by using the new API instead of the old) once it started causing problems.

I think many hardware appliances (like ours) are using very old kernel versions, mainly for historical reasons.
Jumping ahead to a modern kernel version seems risky, so taking small steps still leaves us on old kernel versions.
There are also build environment issues, i.e. building a certain kernel version only works with certain GCC versions; you cannot build an old kernel version using a new GCC version or vice versa.
And similar dependencies between libc versions and kernel versions.
And root file system dependencies, mainly expectations to what is in /dev and /proc.
And it's not only DPDK itself and DPDK applications that may run into issues when upgrading the kernel version. Any application or library on the system might run into similar or other problems when upgrading to a newer kernel version.

So the easy solution is to simply stick with whatever kernel version was used for the system initially, and never upgrade it. Or only upgrade to a slightly newer version, to keep the knock-on effect at a minimum.
(The distro companies must be putting significant effort into keeping up with new versions of kernels, libraries and applications, to offer new features and ensure system-wide interoperability. The resulting value of this should be appreciated!)

> 
> Poking around the source, still see some leftovers.
> 
> Like quick assist documentation wanting kernel version for sources?
> Why does QAT depend on kernel source being present. Thats not right.
> 
> The other bad spot is the Intel GVE driver reading and passing
> the OS version in. Looks like some host side validation.
> 
> Putting OS version anywhere in DPDK is a mistake.

Interesting observations. Again, I agree very much - it is likely to become problematic to maintain and/or use with other OS versions than its developer had originally foreseen.

Re: [PATCH] doc: update minimum Linux kernel version

[Stephen Hemminger]

On Thu, 11 Jan 2024 23:38:07 +0100
Morten Brørup <mb@smartsharesystems.com> wrote:

> > From: Stephen Hemminger [mailto:stephen@networkplumber.org]
> > Sent: Thursday, 11 January 2024 20.55
> > 
> > On Thu, 11 Jan 2024 20:26:56 +0100
> > Morten Brørup <mb@smartsharesystems.com> wrote:
> >   
> > >
> > >
> > > When the documentation specifies a minimum required kernel version,  
> > it implicitly claims that DPDK works with that kernel version.  
> > >
> > > So we should either test with the specified kernel version (which  
> > requires significant effort to set up, so I’m not going to ask for
> > it!), or add a big fat disclaimer/warning that DPDK is not tested with
> > the mentioned kernel version, and list the kernel versions actually
> > tested.
> > 
> > It is much less of an issue than it used to be since there should be no
> > need for
> > DPDK specific kernel components. The kernel API/ABI is stable across
> > releases
> > (with the notable exception of BPF). Therefore the DPDK kernel version
> > dependency
> > is much less than it used to be.  

There are three issues here:

1. Supporting out of date kernel also means supporting out of date build environments
   that maybe missing headers. The recent example was the TAP device requiring (or cloning
   which is worse) the headers to the FLOWER classifier.  If we move the kernel version
   to current LTS, then FLOWER is always present.
2. Supporting out of date kernel means more test infrastructure. Some CI needs to build
   test on older environments.
3. The place it impacts current CI is the building on CentOS7. CentOS7 is end of life
   do we have to keep it? The compiler also lack good C11 support so not sure how CI keeps working.

The way I view it, if you are on an old system, then stick to old DPDK LTS version.
We don't want to here about regressions on end of life systems.

RE: [PATCH] doc: update minimum Linux kernel version

[Morten Brørup]

> From: Stephen Hemminger [mailto:stephen@networkplumber.org]
> Sent: Friday, 16 February 2024 04.05
> 
> On Thu, 11 Jan 2024 23:38:07 +0100
> Morten Brørup <mb@smartsharesystems.com> wrote:
> 
> > > From: Stephen Hemminger [mailto:stephen@networkplumber.org]
> > > Sent: Thursday, 11 January 2024 20.55
> > >
> > > On Thu, 11 Jan 2024 20:26:56 +0100
> > > Morten Brørup <mb@smartsharesystems.com> wrote:
> > >
> > > >
> > > >
> > > > When the documentation specifies a minimum required kernel
> version,
> > > it implicitly claims that DPDK works with that kernel version.
> > > >
> > > > So we should either test with the specified kernel version (which
> > > requires significant effort to set up, so I’m not going to ask for
> > > it!), or add a big fat disclaimer/warning that DPDK is not tested
> with
> > > the mentioned kernel version, and list the kernel versions actually
> > > tested.
> > >
> > > It is much less of an issue than it used to be since there should
> be no
> > > need for
> > > DPDK specific kernel components. The kernel API/ABI is stable
> across
> > > releases
> > > (with the notable exception of BPF). Therefore the DPDK kernel
> version
> > > dependency
> > > is much less than it used to be.
> 
> There are three issues here:
> 
> 1. Supporting out of date kernel also means supporting out of date
> build environments
>    that maybe missing headers. The recent example was the TAP device
> requiring (or cloning
>    which is worse) the headers to the FLOWER classifier.  If we move
> the kernel version
>    to current LTS, then FLOWER is always present.
> 2. Supporting out of date kernel means more test infrastructure. Some
> CI needs to build
>    test on older environments.
> 3. The place it impacts current CI is the building on CentOS7. CentOS7
> is end of life
>    do we have to keep it? The compiler also lack good C11 support so
> not sure how CI keeps working.
> 
> The way I view it, if you are on an old system, then stick to old DPDK
> LTS version.
> We don't want to here about regressions on end of life systems.

The system requirements in the Getting Started Guide [1] says:

Kernel version >= 4.14
The kernel version required is based on the oldest long term stable kernel available at kernel.org when the DPDK version is in development.
Compatibility for recent distribution kernels will be kept, notably RHEL/CentOS 7.

[1]: https://doc.dpdk.org/guides/linux_gsg/sys_reqs.html#system-software

If we consider it API breakage to change that, we have to wait until the 24.11 release.
For future DPDK LTS releases, we should be more careful about what we claim to support. And again: If we claim to support something, people expect it to be tested in CI.

Disregarding the API breakage by stopping support for a system we claim to support... RHEL7 testing was changed to LTS only [2], that should probably have been applied to CentOS 7 too.

[2]: https://inbox.dpdk.org/dev/CAJvnSUBcq3gznQD4k=krQ+gu2OxTxA2YJBc=J=LtidFXqgg_hg@mail.gmail.com/

Re: [PATCH] doc: update minimum Linux kernel version

[Stephen Hemminger]

On Fri, 16 Feb 2024 09:29:47 +0100
Morten Brørup <mb@smartsharesystems.com> wrote:

> > From: Stephen Hemminger [mailto:stephen@networkplumber.org]
> > Sent: Friday, 16 February 2024 04.05
> > 
> > On Thu, 11 Jan 2024 23:38:07 +0100
> > Morten Brørup <mb@smartsharesystems.com> wrote:
> >   
> > > > From: Stephen Hemminger [mailto:stephen@networkplumber.org]
> > > > Sent: Thursday, 11 January 2024 20.55
> > > >
> > > > On Thu, 11 Jan 2024 20:26:56 +0100
> > > > Morten Brørup <mb@smartsharesystems.com> wrote:
> > > >  
> > > > >
> > > > >
> > > > > When the documentation specifies a minimum required kernel  
> > version,  
> > > > it implicitly claims that DPDK works with that kernel version.  
> > > > >
> > > > > So we should either test with the specified kernel version (which  
> > > > requires significant effort to set up, so I’m not going to ask for
> > > > it!), or add a big fat disclaimer/warning that DPDK is not tested  
> > with  
> > > > the mentioned kernel version, and list the kernel versions actually
> > > > tested.
> > > >
> > > > It is much less of an issue than it used to be since there should  
> > be no  
> > > > need for
> > > > DPDK specific kernel components. The kernel API/ABI is stable  
> > across  
> > > > releases
> > > > (with the notable exception of BPF). Therefore the DPDK kernel  
> > version  
> > > > dependency
> > > > is much less than it used to be.  
> > 
> > There are three issues here:
> > 
> > 1. Supporting out of date kernel also means supporting out of date
> > build environments
> >    that maybe missing headers. The recent example was the TAP device
> > requiring (or cloning
> >    which is worse) the headers to the FLOWER classifier.  If we move
> > the kernel version
> >    to current LTS, then FLOWER is always present.
> > 2. Supporting out of date kernel means more test infrastructure. Some
> > CI needs to build
> >    test on older environments.
> > 3. The place it impacts current CI is the building on CentOS7. CentOS7
> > is end of life
> >    do we have to keep it? The compiler also lack good C11 support so
> > not sure how CI keeps working.
> > 
> > The way I view it, if you are on an old system, then stick to old DPDK
> > LTS version.
> > We don't want to here about regressions on end of life systems.  
> 
> The system requirements in the Getting Started Guide [1] says:
> 
> Kernel version >= 4.14
> The kernel version required is based on the oldest long term stable kernel available at kernel.org when the DPDK version is in development.
> Compatibility for recent distribution kernels will be kept, notably RHEL/CentOS 7.

We need to drop CentOS 7 soon.

https://www.redhat.com/en/topics/linux/centos-linux-eol

	CentOS Linux 7 will reach end of life (EOL) on June 30, 2024. 

Re: [PATCH] doc: update minimum Linux kernel version

[Stephen Hemminger]

On Fri, 16 Feb 2024 09:29:47 +0100
Morten Brørup <mb@smartsharesystems.com> wrote:

> The system requirements in the Getting Started Guide [1] says:
> 
> Kernel version >= 4.14
> The kernel version required is based on the oldest long term stable kernel available at kernel.org when the DPDK version is in development.
> Compatibility for recent distribution kernels will be kept, notably RHEL/CentOS 7.
> 
> [1]: https://doc.dpdk.org/guides/linux_gsg/sys_reqs.html#system-software
> 
> If we consider it API breakage to change that, we have to wait until the 24.11 release.
> For future DPDK LTS releases, we should be more careful about what we claim to support. And again: If we claim to support something, people expect it to be tested in CI.
> 
> Disregarding the API breakage by stopping support for a system we claim to support... RHEL7 testing was changed to LTS only [2], that should probably have been applied to CentOS 7 too.
> 
> [2]: https://inbox.dpdk.org/dev/CAJvnSUBcq3gznQD4k=krQ+gu2OxTxA2YJBc=J=LtidFXqgg_hg@mail.gmail.com/
> 

This patch is too late for 24.03 release, by the time the next one happens,
we can drop CentOS 7 as well as the old kernel.

Re: [PATCH] doc: update minimum Linux kernel version

[Patrick Robb]

[-- Attachment #1: Type: text/plain, Size: 1861 bytes --]

+Gao, DaxueX <daxuex.gao@intel.com> +Mcnamara, John
<john.mcnamara@intel.com>

Hello,

As you say the Community Lab dropped main and next-* testing for RHEL7 when
the requirement for a C11 compliant compiler was added last year, so we
should already be good to go. We don't have any centos7 testing (centos8
was the first centos introduced for testing at the Community Lab).

Adding Daxue and John so they are aware the Intel Lab will want to
(probably) drop the centos7 testing by 24.07.

Thanks!

On Fri, Feb 16, 2024 at 12:42 PM Stephen Hemminger <
stephen@networkplumber.org> wrote:

> On Fri, 16 Feb 2024 09:29:47 +0100
> Morten Brørup <mb@smartsharesystems.com> wrote:
>
> > The system requirements in the Getting Started Guide [1] says:
> >
> > Kernel version >= 4.14
> > The kernel version required is based on the oldest long term stable
> kernel available at kernel.org when the DPDK version is in development.
> > Compatibility for recent distribution kernels will be kept, notably
> RHEL/CentOS 7.
> >
> > [1]: https://doc.dpdk.org/guides/linux_gsg/sys_reqs.html#system-software
> >
> > If we consider it API breakage to change that, we have to wait until the
> 24.11 release.
> > For future DPDK LTS releases, we should be more careful about what we
> claim to support. And again: If we claim to support something, people
> expect it to be tested in CI.
> >
> > Disregarding the API breakage by stopping support for a system we claim
> to support... RHEL7 testing was changed to LTS only [2], that should
> probably have been applied to CentOS 7 too.
> >
> > [2]:
> https://inbox.dpdk.org/dev/CAJvnSUBcq3gznQD4k=krQ+gu2OxTxA2YJBc=J=LtidFXqgg_hg@mail.gmail.com/
> >
>
> This patch is too late for 24.03 release, by the time the next one happens,
> we can drop CentOS 7 as well as the old kernel.
>

[-- Attachment #2: Type: text/html, Size: 2917 bytes --]