[dpdk-dev] [PATCH] doc: note KNI alternatives and deprecation plan

[dpdk-dev] [PATCH] doc: note KNI alternatives and deprecation plan

[Ferruh Yigit]

Add a note that KNI kernel module will be moved to dpdk-kmods git repo
and there is a long term plan to deprecate it.

Also add some more details on the alternatives to KNI and cons of the
KNI against these alternatives.

Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
---
 doc/guides/nics/tap.rst                       |  2 +
 .../prog_guide/kernel_nic_interface.rst       | 38 +++++++++++++++++--
 2 files changed, 37 insertions(+), 3 deletions(-)

diff --git a/doc/guides/nics/tap.rst b/doc/guides/nics/tap.rst
index 3ce696b605d1..07315fe32422 100644
--- a/doc/guides/nics/tap.rst
+++ b/doc/guides/nics/tap.rst
@@ -1,6 +1,8 @@
 ..  SPDX-License-Identifier: BSD-3-Clause
     Copyright(c) 2016 Intel Corporation.
 
+.. _TunTap_PMD:
+
 Tun|Tap Poll Mode Driver
 ========================
 
diff --git a/doc/guides/prog_guide/kernel_nic_interface.rst b/doc/guides/prog_guide/kernel_nic_interface.rst
index 1ce03ec1a374..29f8c92fd9d6 100644
--- a/doc/guides/prog_guide/kernel_nic_interface.rst
+++ b/doc/guides/prog_guide/kernel_nic_interface.rst
@@ -6,16 +6,48 @@
 Kernel NIC Interface
 ====================
 
+.. Note::
+
+   KNI kernel module will be removed from main git repository to `dpdk-kmods <https://git.dpdk.org/dpdk-kmods/>`_
+   repository by the `DPDK technical board decision <https://mails.dpdk.org/archives/dev/2021-January/197077.html>`_.
+   Also there is a `long term plan <https://mails.dpdk.org/archives/dev/2021-May/209026.html>`_ to deprecate the KNI.
+
+   :ref:`virtio_user_as_exceptional_path` alternative is preferred way for
+   interfacing with Linux network stack as it is being in-kernel solution and
+   similar performance expectations.
+
 The DPDK Kernel NIC Interface (KNI) allows userspace applications access to the Linux* control plane.
 
-The benefits of using the DPDK KNI are:
+KNI allows an interface with the kernel network stack and allows management of
+DPDK ports using standard Linux net tools such as ``ethtool``, ``ifconfig`` and
+``tcpdump``.
+
+Main use case of KNI is get/receive exception packets from/to Linux network
+stack while main datapath IO is done bypassing the networking stack.
+
+There are other alternatives to KNI, all are available in the upstream Linux:
+
+#. :ref:`TunTap_PMD` as wrapper to `Linux tun/tap
+   <https://www.kernel.org/doc/Documentation/networking/tuntap.txt>`_
+
+#. :ref:`virtio_user_as_exceptional_path`
+
+The benefits of using the DPDK KNI against alternatives are:
 
 *   Faster than existing Linux TUN/TAP interfaces
     (by eliminating system calls and copy_to_user()/copy_from_user() operations.
 
-*   Allows management of DPDK ports using standard Linux net tools such as ethtool, ifconfig and tcpdump.
+The cons of the DPDK KNI are:
+
+* It is out-of-tree Linux kernel module and it can't be distributed as binary as
+  part of OSV DPDK packages. This makes it harder to consume, although it is
+  always possible to compile it from the source code.
+
+* As it shares memory between userspace and kernelspace, and kernel part
+  directly uses input provided by userspace, it is not safe. This makes hard to
+  upstream the module.
 
-*   Allows an interface with the kernel network stack.
+* Only a subset of control commands are supported by KNI.
 
 The components of an application using the DPDK Kernel NIC Interface are shown in :numref:`figure_kernel_nic_intf`.
 
-- 
2.31.1

Re: [dpdk-dev] [PATCH] doc: note KNI alternatives and deprecation plan

[Stephen Hemminger]

On Wed, 23 Jun 2021 18:31:42 +0100
Ferruh Yigit <ferruh.yigit@intel.com> wrote:

> Add a note that KNI kernel module will be moved to dpdk-kmods git repo
> and there is a long term plan to deprecate it.
> 
> Also add some more details on the alternatives to KNI and cons of the
> KNI against these alternatives.
> 
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> ---
>  doc/guides/nics/tap.rst                       |  2 +
>  .../prog_guide/kernel_nic_interface.rst       | 38 +++++++++++++++++--
>  2 files changed, 37 insertions(+), 3 deletions(-)
> 
> diff --git a/doc/guides/nics/tap.rst b/doc/guides/nics/tap.rst
> index 3ce696b605d1..07315fe32422 100644
> --- a/doc/guides/nics/tap.rst
> +++ b/doc/guides/nics/tap.rst
> @@ -1,6 +1,8 @@
>  ..  SPDX-License-Identifier: BSD-3-Clause
>      Copyright(c) 2016 Intel Corporation.
>  
> +.. _TunTap_PMD:
> +
>  Tun|Tap Poll Mode Driver
>  ========================
>  
> diff --git a/doc/guides/prog_guide/kernel_nic_interface.rst b/doc/guides/prog_guide/kernel_nic_interface.rst
> index 1ce03ec1a374..29f8c92fd9d6 100644
> --- a/doc/guides/prog_guide/kernel_nic_interface.rst
> +++ b/doc/guides/prog_guide/kernel_nic_interface.rst
> @@ -6,16 +6,48 @@
>  Kernel NIC Interface
>  ====================
>  
> +.. Note::
> +
> +   KNI kernel module will be removed from main git repository to `dpdk-kmods <https://git.dpdk.org/dpdk-kmods/>`_
> +   repository by the `DPDK technical board decision <https://mails.dpdk.org/archives/dev/2021-January/197077.html>`_.
> +   Also there is a `long term plan <https://mails.dpdk.org/archives/dev/2021-May/209026.html>`_ to deprecate the KNI.
> +
> +   :ref:`virtio_user_as_exceptional_path` alternative is preferred way for
> +   interfacing with Linux network stack as it is being in-kernel solution and
> +   similar performance expectations.
> +
>  The DPDK Kernel NIC Interface (KNI) allows userspace applications access to the Linux* control plane.
>  
> -The benefits of using the DPDK KNI are:
> +KNI allows an interface with the kernel network stack and allows management of
> +DPDK ports using standard Linux net tools such as ``ethtool``, ``ifconfig`` and
> +``tcpdump``.
> +
> +Main use case of KNI is get/receive exception packets from/to Linux network
> +stack while main datapath IO is done bypassing the networking stack.
> +
> +There are other alternatives to KNI, all are available in the upstream Linux:
> +
> +#. :ref:`TunTap_PMD` as wrapper to `Linux tun/tap
> +   <https://www.kernel.org/doc/Documentation/networking/tuntap.txt>`_
> +
> +#. :ref:`virtio_user_as_exceptional_path`
> +
> +The benefits of using the DPDK KNI against alternatives are:
>  
>  *   Faster than existing Linux TUN/TAP interfaces
>      (by eliminating system calls and copy_to_user()/copy_from_user() operations.
>  
> -*   Allows management of DPDK ports using standard Linux net tools such as ethtool, ifconfig and tcpdump.
> +The cons of the DPDK KNI are:
> +
> +* It is out-of-tree Linux kernel module and it can't be distributed as binary as
> +  part of OSV DPDK packages. This makes it harder to consume, although it is
> +  always possible to compile it from the source code.
> +
> +* As it shares memory between userspace and kernelspace, and kernel part
> +  directly uses input provided by userspace, it is not safe. This makes hard to
> +  upstream the module.
>  
> -*   Allows an interface with the kernel network stack.
> +* Only a subset of control commands are supported by KNI.
>  
>  The components of an application using the DPDK Kernel NIC Interface are shown in :numref:`figure_kernel_nic_intf`.
>  

You should also link to virtio user as exception path. In my testing virtio is as fast as KNI and uses less CPU.

Re: [dpdk-dev] [PATCH] doc: note KNI alternatives and deprecation plan

[Ferruh Yigit]

On 6/23/2021 7:33 PM, Stephen Hemminger wrote:
> On Wed, 23 Jun 2021 18:31:42 +0100
> Ferruh Yigit <ferruh.yigit@intel.com> wrote:
> 
>> Add a note that KNI kernel module will be moved to dpdk-kmods git repo
>> and there is a long term plan to deprecate it.
>>
>> Also add some more details on the alternatives to KNI and cons of the
>> KNI against these alternatives.
>>
>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>> ---
>>  doc/guides/nics/tap.rst                       |  2 +
>>  .../prog_guide/kernel_nic_interface.rst       | 38 +++++++++++++++++--
>>  2 files changed, 37 insertions(+), 3 deletions(-)
>>
>> diff --git a/doc/guides/nics/tap.rst b/doc/guides/nics/tap.rst
>> index 3ce696b605d1..07315fe32422 100644
>> --- a/doc/guides/nics/tap.rst
>> +++ b/doc/guides/nics/tap.rst
>> @@ -1,6 +1,8 @@
>>  ..  SPDX-License-Identifier: BSD-3-Clause
>>      Copyright(c) 2016 Intel Corporation.
>>  
>> +.. _TunTap_PMD:
>> +
>>  Tun|Tap Poll Mode Driver
>>  ========================
>>  
>> diff --git a/doc/guides/prog_guide/kernel_nic_interface.rst b/doc/guides/prog_guide/kernel_nic_interface.rst
>> index 1ce03ec1a374..29f8c92fd9d6 100644
>> --- a/doc/guides/prog_guide/kernel_nic_interface.rst
>> +++ b/doc/guides/prog_guide/kernel_nic_interface.rst
>> @@ -6,16 +6,48 @@
>>  Kernel NIC Interface
>>  ====================
>>  
>> +.. Note::
>> +
>> +   KNI kernel module will be removed from main git repository to `dpdk-kmods <https://git.dpdk.org/dpdk-kmods/>`_
>> +   repository by the `DPDK technical board decision <https://mails.dpdk.org/archives/dev/2021-January/197077.html>`_.
>> +   Also there is a `long term plan <https://mails.dpdk.org/archives/dev/2021-May/209026.html>`_ to deprecate the KNI.
>> +
>> +   :ref:`virtio_user_as_exceptional_path` alternative is preferred way for
>> +   interfacing with Linux network stack as it is being in-kernel solution and
>> +   similar performance expectations.
>> +
>>  The DPDK Kernel NIC Interface (KNI) allows userspace applications access to the Linux* control plane.
>>  
>> -The benefits of using the DPDK KNI are:
>> +KNI allows an interface with the kernel network stack and allows management of
>> +DPDK ports using standard Linux net tools such as ``ethtool``, ``ifconfig`` and
>> +``tcpdump``.
>> +
>> +Main use case of KNI is get/receive exception packets from/to Linux network
>> +stack while main datapath IO is done bypassing the networking stack.
>> +
>> +There are other alternatives to KNI, all are available in the upstream Linux:
>> +
>> +#. :ref:`TunTap_PMD` as wrapper to `Linux tun/tap
>> +   <https://www.kernel.org/doc/Documentation/networking/tuntap.txt>`_
>> +
>> +#. :ref:`virtio_user_as_exceptional_path`
>> +
>> +The benefits of using the DPDK KNI against alternatives are:
>>  
>>  *   Faster than existing Linux TUN/TAP interfaces
>>      (by eliminating system calls and copy_to_user()/copy_from_user() operations.
>>  
>> -*   Allows management of DPDK ports using standard Linux net tools such as ethtool, ifconfig and tcpdump.
>> +The cons of the DPDK KNI are:
>> +
>> +* It is out-of-tree Linux kernel module and it can't be distributed as binary as
>> +  part of OSV DPDK packages. This makes it harder to consume, although it is
>> +  always possible to compile it from the source code.
>> +
>> +* As it shares memory between userspace and kernelspace, and kernel part
>> +  directly uses input provided by userspace, it is not safe. This makes hard to
>> +  upstream the module.
>>  
>> -*   Allows an interface with the kernel network stack.
>> +* Only a subset of control commands are supported by KNI.
>>  
>>  The components of an application using the DPDK Kernel NIC Interface are shown in :numref:`figure_kernel_nic_intf`.
>>  
> 
> You should also link to virtio user as exception path. In my testing virtio is as fast as KNI and uses less CPU.
> 

Patch mentions from virtio-user alternative, and in the notes it even documents
it as preferred way.

Re: [dpdk-dev] [PATCH] doc: note KNI alternatives and deprecation plan

[Olivier Matz]

Hi Ferruh,

Few minor comments below.

On Wed, Jun 23, 2021 at 06:31:42PM +0100, Ferruh Yigit wrote:
> Add a note that KNI kernel module will be moved to dpdk-kmods git repo
> and there is a long term plan to deprecate it.
> 
> Also add some more details on the alternatives to KNI and cons of the
> KNI against these alternatives.
> 
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> ---
>  doc/guides/nics/tap.rst                       |  2 +
>  .../prog_guide/kernel_nic_interface.rst       | 38 +++++++++++++++++--
>  2 files changed, 37 insertions(+), 3 deletions(-)
> 
> diff --git a/doc/guides/nics/tap.rst b/doc/guides/nics/tap.rst
> index 3ce696b605d1..07315fe32422 100644
> --- a/doc/guides/nics/tap.rst
> +++ b/doc/guides/nics/tap.rst
> @@ -1,6 +1,8 @@
>  ..  SPDX-License-Identifier: BSD-3-Clause
>      Copyright(c) 2016 Intel Corporation.
>  
> +.. _TunTap_PMD:
> +
>  Tun|Tap Poll Mode Driver
>  ========================
>  
> diff --git a/doc/guides/prog_guide/kernel_nic_interface.rst b/doc/guides/prog_guide/kernel_nic_interface.rst
> index 1ce03ec1a374..29f8c92fd9d6 100644
> --- a/doc/guides/prog_guide/kernel_nic_interface.rst
> +++ b/doc/guides/prog_guide/kernel_nic_interface.rst
> @@ -6,16 +6,48 @@
>  Kernel NIC Interface
>  ====================
>  
> +.. Note::
> +
> +   KNI kernel module will be removed from main git repository to `dpdk-kmods <https://git.dpdk.org/dpdk-kmods/>`_

removed -> moved?

Or "removed from main and moved to"

> +   repository by the `DPDK technical board decision <https://mails.dpdk.org/archives/dev/2021-January/197077.html>`_.
> +   Also there is a `long term plan <https://mails.dpdk.org/archives/dev/2021-May/209026.html>`_ to deprecate the KNI.
> +
> +   :ref:`virtio_user_as_exceptional_path` alternative is preferred way for
> +   interfacing with Linux network stack as it is being in-kernel solution and
> +   similar performance expectations.
> +
>  The DPDK Kernel NIC Interface (KNI) allows userspace applications access to the Linux* control plane.
>  
> -The benefits of using the DPDK KNI are:
> +KNI allows an interface with the kernel network stack and allows management of
> +DPDK ports using standard Linux net tools such as ``ethtool``, ``ifconfig`` and
> +``tcpdump``.
> +
> +Main use case of KNI is get/receive exception packets from/to Linux network
> +stack while main datapath IO is done bypassing the networking stack.
> +
> +There are other alternatives to KNI, all are available in the upstream Linux:
> +
> +#. :ref:`TunTap_PMD` as wrapper to `Linux tun/tap
> +   <https://www.kernel.org/doc/Documentation/networking/tuntap.txt>`_
> +
> +#. :ref:`virtio_user_as_exceptional_path`

Shouldn't virtio_user be the first item?

> +
> +The benefits of using the DPDK KNI against alternatives are:
>  
>  *   Faster than existing Linux TUN/TAP interfaces
>      (by eliminating system calls and copy_to_user()/copy_from_user() operations.
>  
> -*   Allows management of DPDK ports using standard Linux net tools such as ethtool, ifconfig and tcpdump.
> +The cons of the DPDK KNI are:
> +
> +* It is out-of-tree Linux kernel module and it can't be distributed as binary as
> +  part of OSV DPDK packages. This makes it harder to consume, although it is

OSV -> OVS

> +  always possible to compile it from the source code.
> +
> +* As it shares memory between userspace and kernelspace, and kernel part
> +  directly uses input provided by userspace, it is not safe. This makes hard to
> +  upstream the module.
>  
> -*   Allows an interface with the kernel network stack.
> +* Only a subset of control commands are supported by KNI.
>  
>  The components of an application using the DPDK Kernel NIC Interface are shown in :numref:`figure_kernel_nic_intf`.
>  
> -- 
> 2.31.1
> 

Re: [dpdk-dev] [PATCH] doc: note KNI alternatives and deprecation plan

[David Marchand]

On Wed, Jun 23, 2021 at 7:32 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>
> Add a note that KNI kernel module will be moved to dpdk-kmods git repo
> and there is a long term plan to deprecate it.
>
> Also add some more details on the alternatives to KNI and cons of the
> KNI against these alternatives.

Some suggestions on wording below, feel free to ignore if this sounds
like French ;-).


>
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> ---
>  doc/guides/nics/tap.rst                       |  2 +
>  .../prog_guide/kernel_nic_interface.rst       | 38 +++++++++++++++++--
>  2 files changed, 37 insertions(+), 3 deletions(-)
>
> diff --git a/doc/guides/nics/tap.rst b/doc/guides/nics/tap.rst
> index 3ce696b605d1..07315fe32422 100644
> --- a/doc/guides/nics/tap.rst
> +++ b/doc/guides/nics/tap.rst
> @@ -1,6 +1,8 @@
>  ..  SPDX-License-Identifier: BSD-3-Clause
>      Copyright(c) 2016 Intel Corporation.
>
> +.. _TunTap_PMD:
> +
>  Tun|Tap Poll Mode Driver
>  ========================
>
> diff --git a/doc/guides/prog_guide/kernel_nic_interface.rst b/doc/guides/prog_guide/kernel_nic_interface.rst
> index 1ce03ec1a374..29f8c92fd9d6 100644
> --- a/doc/guides/prog_guide/kernel_nic_interface.rst
> +++ b/doc/guides/prog_guide/kernel_nic_interface.rst
> @@ -6,16 +6,48 @@
>  Kernel NIC Interface
>  ====================
>
> +.. Note::
> +

> +   KNI kernel module will be removed from main git repository to `dpdk-kmods <https://git.dpdk.org/dpdk-kmods/>`_
The* KNI kernel module will be moved* (Olivier comment) to ...

> +   repository by the `DPDK technical board decision <https://mails.dpdk.org/archives/dev/2021-January/197077.html>`_.
repository following* the `...`_.

I'd move this first part in doc/guides/rel_notes/deprecation.rst
and/or release notes.
Then, reword this as:

> +   Also there is a `long term plan <https://mails.dpdk.org/archives/dev/2021-May/209026.html>`_ to deprecate the KNI.

The `long term plan...`_ is* to deprecate KNI, see <XXX link to
deprecation notice/release notes>.


> +
> +   :ref:`virtio_user_as_exceptional_path` alternative is preferred way for

is one/a* preferred way

> +   interfacing with Linux network stack as it is being in-kernel solution and

interfacing with the* Linux network stack as it is a* in-kernel and

> +   similar performance expectations.

has* similar performance expectations.


> +
>  The DPDK Kernel NIC Interface (KNI) allows userspace applications access to the Linux* control plane.
>
> -The benefits of using the DPDK KNI are:
> +KNI allows an interface with the kernel network stack and allows management of

KNI provides* an interface ...

> +DPDK ports using standard Linux net tools such as ``ethtool``, ``ifconfig`` and
> +``tcpdump``.
> +
> +Main use case of KNI is get/receive exception packets from/to Linux network

The* main use case of KNI is to* ...

> +stack while main datapath IO is done bypassing the networking stack.
> +
> +There are other alternatives to KNI, all are available in the upstream Linux:
> +
> +#. :ref:`TunTap_PMD` as wrapper to `Linux tun/tap
> +   <https://www.kernel.org/doc/Documentation/networking/tuntap.txt>`_
> +
> +#. :ref:`virtio_user_as_exceptional_path`
> +

+1 to Olivier comment, for making virtio-user first.


> +The benefits of using the DPDK KNI against alternatives are:
>
>  *   Faster than existing Linux TUN/TAP interfaces
>      (by eliminating system calls and copy_to_user()/copy_from_user() operations.
>
> -*   Allows management of DPDK ports using standard Linux net tools such as ethtool, ifconfig and tcpdump.
> +The cons of the DPDK KNI are:

Not sure why we need an extra DPDK, KNI alone is enough.


> +
> +* It is out-of-tree Linux kernel module and it can't be distributed as binary as
> +  part of OSV DPDK packages. This makes it harder to consume, although it is

OSV as in OS vendors, right? ok for me.


> +  always possible to compile it from the source code.
> +
> +* As it shares memory between userspace and kernelspace, and kernel part
> +  directly uses input provided by userspace, it is not safe. This makes hard to
> +  upstream the module.
>
> -*   Allows an interface with the kernel network stack.
> +* Only a subset of control commands are supported by KNI.

of net devices control commands*

>
>  The components of an application using the DPDK Kernel NIC Interface are shown in :numref:`figure_kernel_nic_intf`.
>
> --
> 2.31.1
>


-- 
David Marchand

Re: [dpdk-dev] [PATCH] doc: note KNI alternatives and deprecation plan

[Ferruh Yigit]

On 7/30/2021 1:48 PM, Olivier Matz wrote:
> Hi Ferruh,
> 
> Few minor comments below.
> 
> On Wed, Jun 23, 2021 at 06:31:42PM +0100, Ferruh Yigit wrote:
>> Add a note that KNI kernel module will be moved to dpdk-kmods git repo
>> and there is a long term plan to deprecate it.
>>
>> Also add some more details on the alternatives to KNI and cons of the
>> KNI against these alternatives.
>>
>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>> ---
>>   doc/guides/nics/tap.rst                       |  2 +
>>   .../prog_guide/kernel_nic_interface.rst       | 38 +++++++++++++++++--
>>   2 files changed, 37 insertions(+), 3 deletions(-)
>>
>> diff --git a/doc/guides/nics/tap.rst b/doc/guides/nics/tap.rst
>> index 3ce696b605d1..07315fe32422 100644
>> --- a/doc/guides/nics/tap.rst
>> +++ b/doc/guides/nics/tap.rst
>> @@ -1,6 +1,8 @@
>>   ..  SPDX-License-Identifier: BSD-3-Clause
>>       Copyright(c) 2016 Intel Corporation.
>>   
>> +.. _TunTap_PMD:
>> +
>>   Tun|Tap Poll Mode Driver
>>   ========================
>>   
>> diff --git a/doc/guides/prog_guide/kernel_nic_interface.rst b/doc/guides/prog_guide/kernel_nic_interface.rst
>> index 1ce03ec1a374..29f8c92fd9d6 100644
>> --- a/doc/guides/prog_guide/kernel_nic_interface.rst
>> +++ b/doc/guides/prog_guide/kernel_nic_interface.rst
>> @@ -6,16 +6,48 @@
>>   Kernel NIC Interface
>>   ====================
>>   
>> +.. Note::
>> +
>> +   KNI kernel module will be removed from main git repository to `dpdk-kmods <https://git.dpdk.org/dpdk-kmods/>`_
> 
> removed -> moved?
> 

ack

> Or "removed from main and moved to"
> 
>> +   repository by the `DPDK technical board decision <https://mails.dpdk.org/archives/dev/2021-January/197077.html>`_.
>> +   Also there is a `long term plan <https://mails.dpdk.org/archives/dev/2021-May/209026.html>`_ to deprecate the KNI.
>> +
>> +   :ref:`virtio_user_as_exceptional_path` alternative is preferred way for
>> +   interfacing with Linux network stack as it is being in-kernel solution and
>> +   similar performance expectations.
>> +
>>   The DPDK Kernel NIC Interface (KNI) allows userspace applications access to the Linux* control plane.
>>   
>> -The benefits of using the DPDK KNI are:
>> +KNI allows an interface with the kernel network stack and allows management of
>> +DPDK ports using standard Linux net tools such as ``ethtool``, ``ifconfig`` and
>> +``tcpdump``.
>> +
>> +Main use case of KNI is get/receive exception packets from/to Linux network
>> +stack while main datapath IO is done bypassing the networking stack.
>> +
>> +There are other alternatives to KNI, all are available in the upstream Linux:
>> +
>> +#. :ref:`TunTap_PMD` as wrapper to `Linux tun/tap
>> +   <https://www.kernel.org/doc/Documentation/networking/tuntap.txt>`_
>> +
>> +#. :ref:`virtio_user_as_exceptional_path`
> 
> Shouldn't virtio_user be the first item?
> 

ack

>> +
>> +The benefits of using the DPDK KNI against alternatives are:
>>   
>>   *   Faster than existing Linux TUN/TAP interfaces
>>       (by eliminating system calls and copy_to_user()/copy_from_user() operations.
>>   
>> -*   Allows management of DPDK ports using standard Linux net tools such as ethtool, ifconfig and tcpdump.
>> +The cons of the DPDK KNI are:
>> +
>> +* It is out-of-tree Linux kernel module and it can't be distributed as binary as
>> +  part of OSV DPDK packages. This makes it harder to consume, although it is
> 
> OSV -> OVS
> 

I mean 'Operating System Vendor', I will use long version to prevent confusion.

>> +  always possible to compile it from the source code.
>> +
>> +* As it shares memory between userspace and kernelspace, and kernel part
>> +  directly uses input provided by userspace, it is not safe. This makes hard to
>> +  upstream the module.
>>   
>> -*   Allows an interface with the kernel network stack.
>> +* Only a subset of control commands are supported by KNI.
>>   
>>   The components of an application using the DPDK Kernel NIC Interface are shown in :numref:`figure_kernel_nic_intf`.
>>   
>> -- 
>> 2.31.1
>>

Re: [dpdk-dev] [PATCH] doc: note KNI alternatives and deprecation plan

[Ferruh Yigit]

On 8/20/2021 1:58 PM, David Marchand wrote:
> On Wed, Jun 23, 2021 at 7:32 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>>
>> Add a note that KNI kernel module will be moved to dpdk-kmods git repo
>> and there is a long term plan to deprecate it.
>>
>> Also add some more details on the alternatives to KNI and cons of the
>> KNI against these alternatives.
> 
> Some suggestions on wording below, feel free to ignore if this sounds
> like French ;-).
> 
> 
>>
>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>> ---
>>   doc/guides/nics/tap.rst                       |  2 +
>>   .../prog_guide/kernel_nic_interface.rst       | 38 +++++++++++++++++--
>>   2 files changed, 37 insertions(+), 3 deletions(-)
>>
>> diff --git a/doc/guides/nics/tap.rst b/doc/guides/nics/tap.rst
>> index 3ce696b605d1..07315fe32422 100644
>> --- a/doc/guides/nics/tap.rst
>> +++ b/doc/guides/nics/tap.rst
>> @@ -1,6 +1,8 @@
>>   ..  SPDX-License-Identifier: BSD-3-Clause
>>       Copyright(c) 2016 Intel Corporation.
>>
>> +.. _TunTap_PMD:
>> +
>>   Tun|Tap Poll Mode Driver
>>   ========================
>>
>> diff --git a/doc/guides/prog_guide/kernel_nic_interface.rst b/doc/guides/prog_guide/kernel_nic_interface.rst
>> index 1ce03ec1a374..29f8c92fd9d6 100644
>> --- a/doc/guides/prog_guide/kernel_nic_interface.rst
>> +++ b/doc/guides/prog_guide/kernel_nic_interface.rst
>> @@ -6,16 +6,48 @@
>>   Kernel NIC Interface
>>   ====================
>>
>> +.. Note::
>> +
> 
>> +   KNI kernel module will be removed from main git repository to `dpdk-kmods <https://git.dpdk.org/dpdk-kmods/>`_
> The* KNI kernel module will be moved* (Olivier comment) to ...
> 
>> +   repository by the `DPDK technical board decision <https://mails.dpdk.org/archives/dev/2021-January/197077.html>`_.
> repository following* the `...`_.
> 
> I'd move this first part in doc/guides/rel_notes/deprecation.rst

Perhaps it is better to clarify the decisions first in the deprecation
notice, and document it later referencing it, instead of documenting it first.

I will try to separate the decision part from the KNI documentation.

> and/or release notes.
> Then, reword this as:
> 
>> +   Also there is a `long term plan <https://mails.dpdk.org/archives/dev/2021-May/209026.html>`_ to deprecate the KNI.
> 
> The `long term plan...`_ is* to deprecate KNI, see <XXX link to
> deprecation notice/release notes>.
> 
> 
>> +
>> +   :ref:`virtio_user_as_exceptional_path` alternative is preferred way for
> 
> is one/a* preferred way
> 

ack

>> +   interfacing with Linux network stack as it is being in-kernel solution and
> 
> interfacing with the* Linux network stack as it is a* in-kernel and
>

ack
  
>> +   similar performance expectations.
> 
> has* similar performance expectations.
> 

ack

> 
>> +
>>   The DPDK Kernel NIC Interface (KNI) allows userspace applications access to the Linux* control plane.
>>
>> -The benefits of using the DPDK KNI are:
>> +KNI allows an interface with the kernel network stack and allows management of
> 
> KNI provides* an interface ...
> 

ack

>> +DPDK ports using standard Linux net tools such as ``ethtool``, ``ifconfig`` and
>> +``tcpdump``.
>> +
>> +Main use case of KNI is get/receive exception packets from/to Linux network
> 
> The* main use case of KNI is to* ...
> 

ack

>> +stack while main datapath IO is done bypassing the networking stack.
>> +
>> +There are other alternatives to KNI, all are available in the upstream Linux:
>> +
>> +#. :ref:`TunTap_PMD` as wrapper to `Linux tun/tap
>> +   <https://www.kernel.org/doc/Documentation/networking/tuntap.txt>`_
>> +
>> +#. :ref:`virtio_user_as_exceptional_path`
>> +
> 
> +1 to Olivier comment, for making virtio-user first.
> 
> 
>> +The benefits of using the DPDK KNI against alternatives are:
>>
>>   *   Faster than existing Linux TUN/TAP interfaces
>>       (by eliminating system calls and copy_to_user()/copy_from_user() operations.
>>
>> -*   Allows management of DPDK ports using standard Linux net tools such as ethtool, ifconfig and tcpdump.
>> +The cons of the DPDK KNI are:
> 
> Not sure why we need an extra DPDK, KNI alone is enough.
> 

ack

> 
>> +
>> +* It is out-of-tree Linux kernel module and it can't be distributed as binary as
>> +  part of OSV DPDK packages. This makes it harder to consume, although it is
> 
> OSV as in OS vendors, right? ok for me.
> 
> 
>> +  always possible to compile it from the source code.
>> +
>> +* As it shares memory between userspace and kernelspace, and kernel part
>> +  directly uses input provided by userspace, it is not safe. This makes hard to
>> +  upstream the module.
>>
>> -*   Allows an interface with the kernel network stack.
>> +* Only a subset of control commands are supported by KNI.
> 
> of net devices control commands*
> 

ack

>>
>>   The components of an application using the DPDK Kernel NIC Interface are shown in :numref:`figure_kernel_nic_intf`.
>>
>> --
>> 2.31.1
>>
> 
> 

[PATCH v2 1/2] doc: note KNI alternatives

[Ferruh Yigit]

Add more information on alternatives of KNI and the cons of KNI against
these alternatives.

Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
---
Cc: Olivier Matz <olivier.matz@6wind.com>
Cc: David Marchand <david.marchand@redhat.com>
Cc: Stephen Hemminger <stephen@networkplumber.org>
Cc: Elad Nachman <eladv6@gmail.com>
Cc: Igor Ryzhov <iryzhov@nfware.com>
Cc: Dan Gora <dg@adax.com>
---
 doc/guides/nics/tap.rst                       |  2 ++
 .../prog_guide/kernel_nic_interface.rst       | 34 +++++++++++++++++--
 2 files changed, 33 insertions(+), 3 deletions(-)

diff --git a/doc/guides/nics/tap.rst b/doc/guides/nics/tap.rst
index 681010d9ed7d..2ba98893d564 100644
--- a/doc/guides/nics/tap.rst
+++ b/doc/guides/nics/tap.rst
@@ -1,6 +1,8 @@
 ..  SPDX-License-Identifier: BSD-3-Clause
     Copyright(c) 2016 Intel Corporation.
 
+.. _TunTap_PMD:
+
 Tun|Tap Poll Mode Driver
 ========================
 
diff --git a/doc/guides/prog_guide/kernel_nic_interface.rst b/doc/guides/prog_guide/kernel_nic_interface.rst
index 1ce03ec1a374..70e92687d711 100644
--- a/doc/guides/prog_guide/kernel_nic_interface.rst
+++ b/doc/guides/prog_guide/kernel_nic_interface.rst
@@ -6,16 +6,44 @@
 Kernel NIC Interface
 ====================
 
+.. Note::
+
+   :ref:`virtio_user_as_exceptional_path` alternative is the preferred way for
+   interfacing with the Linux network stack as it is an in-kernel solution and
+   has similar performance expectations.
+
 The DPDK Kernel NIC Interface (KNI) allows userspace applications access to the Linux* control plane.
 
-The benefits of using the DPDK KNI are:
+KNI provides an interface with the kernel network stack and allows management of
+DPDK ports using standard Linux net tools such as ``ethtool``, ``ifconfig`` and
+``tcpdump``.
+
+The main use case of KNI is to get/receive exception packets from/to Linux network
+stack while main datapath IO is done bypassing the networking stack.
+
+There are other alternatives to KNI, all are available in the upstream Linux:
+
+#. :ref:`virtio_user_as_exceptional_path`
+
+#. :ref:`TunTap_PMD` as wrapper to `Linux tun/tap
+   <https://www.kernel.org/doc/Documentation/networking/tuntap.txt>`_
+
+The benefits of using the KNI against alternatives are:
 
 *   Faster than existing Linux TUN/TAP interfaces
     (by eliminating system calls and copy_to_user()/copy_from_user() operations.
 
-*   Allows management of DPDK ports using standard Linux net tools such as ethtool, ifconfig and tcpdump.
+The cons of the KNI are:
+
+* It is out-of-tree Linux kernel module and it can't be distributed as binary as
+  part of operating system vendor DPDK packages. This makes it harder to
+  consume, although it is always possible to compile it from the source code.
+
+* As it shares memory between userspace and kernelspace, and kernel part
+  directly uses input provided by userspace, it is not safe. This makes hard to
+  upstream the module.
 
-*   Allows an interface with the kernel network stack.
+* Only a subset of net devices control commands are supported by KNI.
 
 The components of an application using the DPDK Kernel NIC Interface are shown in :numref:`figure_kernel_nic_intf`.
 
-- 
2.31.1

[PATCH v2 2/2] doc: announce KNI deprecation

[Ferruh Yigit]

Announce the KNI kernel module move to out of dpdk repo and announce
long term plan to deprecate the KNI.

Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
---
Cc: Olivier Matz <olivier.matz@6wind.com>
Cc: David Marchand <david.marchand@redhat.com>
Cc: Stephen Hemminger <stephen@networkplumber.org>
Cc: Elad Nachman <eladv6@gmail.com>
Cc: Igor Ryzhov <iryzhov@nfware.com>
Cc: Dan Gora <dg@adax.com>

Dates are not discussed before, the patch aims to trigger a discussion
for the dates.
---
 doc/guides/prog_guide/kernel_nic_interface.rst | 2 ++
 doc/guides/rel_notes/deprecation.rst           | 6 ++++++
 2 files changed, 8 insertions(+)

diff --git a/doc/guides/prog_guide/kernel_nic_interface.rst b/doc/guides/prog_guide/kernel_nic_interface.rst
index 70e92687d711..276014fe28bb 100644
--- a/doc/guides/prog_guide/kernel_nic_interface.rst
+++ b/doc/guides/prog_guide/kernel_nic_interface.rst
@@ -7,6 +7,8 @@ Kernel NIC Interface
 ====================
 
 .. Note::
+   KNI kernel module will be moved from main git repository to `dpdk-kmods <https://git.dpdk.org/dpdk-kmods/>`_ repository.
+   There is a long term plan to deprecate the KNI. See :doc:`../rel_notes/deprecation`
 
    :ref:`virtio_user_as_exceptional_path` alternative is the preferred way for
    interfacing with the Linux network stack as it is an in-kernel solution and
diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index 6d087c64ef28..62fd991e4eb4 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -48,6 +48,12 @@ Deprecation Notices
   in the header will not be considered as ABI anymore. This change is inspired
   by the RFC https://patchwork.dpdk.org/project/dpdk/list/?series=17176.
 
+* kni: KNI kernel module will be moved to `dpdk-kmods <https://git.dpdk.org/dpdk-kmods/>`_
+  repository by the `DPDK technical board decision
+  <https://mails.dpdk.org/archives/dev/2021-January/197077.html>`_, on v22.11.
+* kni: will be depreciated, will remove all kni lib, kernel module and example code
+  on v23.11.
+
 * lib: will fix extending some enum/define breaking the ABI. There are multiple
   samples in DPDK that enum/define terminated with a ``.*MAX.*`` value which is
   used by iterators, and arrays holding these values are sized with this
-- 
2.31.1

Re: [PATCH v2 1/2] doc: note KNI alternatives

[Thomas Monjalon]

23/11/2021 13:08, Ferruh Yigit:
> --- a/doc/guides/nics/tap.rst
> +++ b/doc/guides/nics/tap.rst
> @@ -1,6 +1,8 @@
>  ..  SPDX-License-Identifier: BSD-3-Clause
>      Copyright(c) 2016 Intel Corporation.
>  
> +.. _TunTap_PMD:

No need for such anchor at the beginning of a doc.
You can reference a document with :doc:

[...]
> +There are other alternatives to KNI, all are available in the upstream Linux:
> +
> +#. :ref:`virtio_user_as_exceptional_path`
> +
> +#. :ref:`TunTap_PMD` as wrapper to `Linux tun/tap
> +   <https://www.kernel.org/doc/Documentation/networking/tuntap.txt>`_

What about pcap, AF_PACKET and AF_XDP PMDs?

Re: [PATCH v2 2/2] doc: announce KNI deprecation

[Thomas Monjalon]

23/11/2021 13:08, Ferruh Yigit:
> --- a/doc/guides/rel_notes/deprecation.rst
> +++ b/doc/guides/rel_notes/deprecation.rst
> +* kni: KNI kernel module will be moved to `dpdk-kmods <https://git.dpdk.org/dpdk-kmods/>`_
> +  repository by the `DPDK technical board decision
> +  <https://mails.dpdk.org/archives/dev/2021-January/197077.html>`_, on v22.11.
> +* kni: will be depreciated, will remove all kni lib, kernel module and example code
> +  on v23.11.

+1 for this plan

Re: [PATCH v2 1/2] doc: note KNI alternatives

[Ferruh Yigit]

On 11/24/2021 1:58 PM, Thomas Monjalon wrote:
> 23/11/2021 13:08, Ferruh Yigit:
>> --- a/doc/guides/nics/tap.rst
>> +++ b/doc/guides/nics/tap.rst
>> @@ -1,6 +1,8 @@
>>   ..  SPDX-License-Identifier: BSD-3-Clause
>>       Copyright(c) 2016 Intel Corporation.
>>   
>> +.. _TunTap_PMD:
> 
> No need for such anchor at the beginning of a doc.
> You can reference a document with :doc:
> 

ack

> [...]
>> +There are other alternatives to KNI, all are available in the upstream Linux:
>> +
>> +#. :ref:`virtio_user_as_exceptional_path`
>> +
>> +#. :ref:`TunTap_PMD` as wrapper to `Linux tun/tap
>> +   <https://www.kernel.org/doc/Documentation/networking/tuntap.txt>`_
> 
> What about pcap, AF_PACKET and AF_XDP PMDs?
> 
> 

They are different, not exactly KNI alternative.

pcap, af_packet & af_xdp are on top of sockets for specific interface,
KNI & virtio-user enables injecting packets to Linux networking stack
without being tied to any existing interface.

[PATCH v3 1/2] doc: note KNI alternatives

[Ferruh Yigit]

Add more information on alternatives of KNI and the cons of KNI against
these alternatives.

Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
---
Cc: Olivier Matz Olivier Matz <olivier.matz@6wind.com>
Cc: David Marchand David Marchand <david.marchand@redhat.com>
Cc: Stephen Hemminger Stephen Hemminger <stephen@networkplumber.org>
Cc: Elad Nachman <eladv6@gmail.com>
Cc: Igor Ryzhov <iryzhov@nfware.com>
Cc: Dan Gora <dg@adax.com>

v3:
* reference tap document directly instead of adding label to it.
---
 .../prog_guide/kernel_nic_interface.rst       | 34 +++++++++++++++++--
 1 file changed, 31 insertions(+), 3 deletions(-)

diff --git a/doc/guides/prog_guide/kernel_nic_interface.rst b/doc/guides/prog_guide/kernel_nic_interface.rst
index 1ce03ec1a374..f5a8b7c0782c 100644
--- a/doc/guides/prog_guide/kernel_nic_interface.rst
+++ b/doc/guides/prog_guide/kernel_nic_interface.rst
@@ -6,16 +6,44 @@
 Kernel NIC Interface
 ====================
 
+.. Note::
+
+   :ref:`virtio_user_as_exceptional_path` alternative is the preferred way for
+   interfacing with the Linux network stack as it is an in-kernel solution and
+   has similar performance expectations.
+
 The DPDK Kernel NIC Interface (KNI) allows userspace applications access to the Linux* control plane.
 
-The benefits of using the DPDK KNI are:
+KNI provides an interface with the kernel network stack and allows management of
+DPDK ports using standard Linux net tools such as ``ethtool``, ``ifconfig`` and
+``tcpdump``.
+
+The main use case of KNI is to get/receive exception packets from/to Linux network
+stack while main datapath IO is done bypassing the networking stack.
+
+There are other alternatives to KNI, all are available in the upstream Linux:
+
+#. :ref:`virtio_user_as_exceptional_path`
+
+#. :doc:`../nics/tap` as wrapper to `Linux tun/tap
+   <https://www.kernel.org/doc/Documentation/networking/tuntap.txt>`_
+
+The benefits of using the KNI against alternatives are:
 
 *   Faster than existing Linux TUN/TAP interfaces
     (by eliminating system calls and copy_to_user()/copy_from_user() operations.
 
-*   Allows management of DPDK ports using standard Linux net tools such as ethtool, ifconfig and tcpdump.
+The cons of the KNI are:
+
+* It is out-of-tree Linux kernel module and it can't be distributed as binary as
+  part of operating system vendor DPDK packages. This makes it harder to
+  consume, although it is always possible to compile it from the source code.
+
+* As it shares memory between userspace and kernelspace, and kernel part
+  directly uses input provided by userspace, it is not safe. This makes hard to
+  upstream the module.
 
-*   Allows an interface with the kernel network stack.
+* Only a subset of net devices control commands are supported by KNI.
 
 The components of an application using the DPDK Kernel NIC Interface are shown in :numref:`figure_kernel_nic_intf`.
 
-- 
2.31.1

[PATCH v3 2/2] doc: announce KNI deprecation

[Ferruh Yigit]

Announce the KNI kernel module move to out of dpdk repo and announce
long term plan to deprecate the KNI.

Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
---
Cc: Olivier Matz Olivier Matz <olivier.matz@6wind.com>
Cc: David Marchand David Marchand <david.marchand@redhat.com>
Cc: Stephen Hemminger Stephen Hemminger <stephen@networkplumber.org>
Cc: Elad Nachman <eladv6@gmail.com>
Cc: Igor Ryzhov <iryzhov@nfware.com>
Cc: Dan Gora <dg@adax.com>

Dates are not discussed before, the patch aims to trigger a discussion
for the dates.
---
 doc/guides/prog_guide/kernel_nic_interface.rst | 2 ++
 doc/guides/rel_notes/deprecation.rst           | 6 ++++++
 2 files changed, 8 insertions(+)

diff --git a/doc/guides/prog_guide/kernel_nic_interface.rst b/doc/guides/prog_guide/kernel_nic_interface.rst
index f5a8b7c0782c..d1c5ccd0851d 100644
--- a/doc/guides/prog_guide/kernel_nic_interface.rst
+++ b/doc/guides/prog_guide/kernel_nic_interface.rst
@@ -7,6 +7,8 @@ Kernel NIC Interface
 ====================
 
 .. Note::
+   KNI kernel module will be moved from main git repository to `dpdk-kmods <https://git.dpdk.org/dpdk-kmods/>`_ repository.
+   There is a long term plan to deprecate the KNI. See :doc:`../rel_notes/deprecation`
 
    :ref:`virtio_user_as_exceptional_path` alternative is the preferred way for
    interfacing with the Linux network stack as it is an in-kernel solution and
diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index 2262b8de6093..f20852504319 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -48,6 +48,12 @@ Deprecation Notices
   in the header will not be considered as ABI anymore. This change is inspired
   by the RFC https://patchwork.dpdk.org/project/dpdk/list/?series=17176.
 
+* kni: KNI kernel module will be moved to `dpdk-kmods <https://git.dpdk.org/dpdk-kmods/>`_
+  repository by the `DPDK technical board decision
+  <https://mails.dpdk.org/archives/dev/2021-January/197077.html>`_, on v22.11.
+* kni: will be depreciated, will remove all kni lib, kernel module and example code
+  on v23.11.
+
 * lib: will fix extending some enum/define breaking the ABI. There are multiple
   samples in DPDK that enum/define terminated with a ``.*MAX.*`` value which is
   used by iterators, and arrays holding these values are sized with this
-- 
2.31.1

RE: [PATCH v3 1/2] doc: note KNI alternatives

[Morten Brørup]

> From: Ferruh Yigit [mailto:ferruh.yigit@intel.com]
> Sent: Wednesday, 24 November 2021 18.16
> 
> Add more information on alternatives of KNI and the cons of KNI against
> these alternatives.
> 
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> ---
> Cc: Olivier Matz Olivier Matz <olivier.matz@6wind.com>
> Cc: David Marchand David Marchand <david.marchand@redhat.com>
> Cc: Stephen Hemminger Stephen Hemminger <stephen@networkplumber.org>
> Cc: Elad Nachman <eladv6@gmail.com>
> Cc: Igor Ryzhov <iryzhov@nfware.com>
> Cc: Dan Gora <dg@adax.com>
> 
> v3:
> * reference tap document directly instead of adding label to it.
> ---
>  .../prog_guide/kernel_nic_interface.rst       | 34 +++++++++++++++++--
>  1 file changed, 31 insertions(+), 3 deletions(-)
> 
> diff --git a/doc/guides/prog_guide/kernel_nic_interface.rst
> b/doc/guides/prog_guide/kernel_nic_interface.rst
> index 1ce03ec1a374..f5a8b7c0782c 100644
> --- a/doc/guides/prog_guide/kernel_nic_interface.rst
> +++ b/doc/guides/prog_guide/kernel_nic_interface.rst
> @@ -6,16 +6,44 @@
>  Kernel NIC Interface
>  ====================
> 
> +.. Note::
> +
> +   :ref:`virtio_user_as_exceptional_path` alternative is the preferred
> way for
> +   interfacing with the Linux network stack as it is an in-kernel
> solution and
> +   has similar performance expectations.
> +
>  The DPDK Kernel NIC Interface (KNI) allows userspace applications
> access to the Linux* control plane.
> 
> -The benefits of using the DPDK KNI are:
> +KNI provides an interface with the kernel network stack and allows
> management of
> +DPDK ports using standard Linux net tools such as ``ethtool``,
> ``ifconfig`` and
> +``tcpdump``.
> +
> +The main use case of KNI is to get/receive exception packets from/to
> Linux network
> +stack while main datapath IO is done bypassing the networking stack.
> +
> +There are other alternatives to KNI, all are available in the upstream
> Linux:
> +
> +#. :ref:`virtio_user_as_exceptional_path`
> +
> +#. :doc:`../nics/tap` as wrapper to `Linux tun/tap
> +   <https://www.kernel.org/doc/Documentation/networking/tuntap.txt>`_
> +
> +The benefits of using the KNI against alternatives are:
> 
>  *   Faster than existing Linux TUN/TAP interfaces
>      (by eliminating system calls and copy_to_user()/copy_from_user()
> operations.
> 
> -*   Allows management of DPDK ports using standard Linux net tools
> such as ethtool, ifconfig and tcpdump.
> +The cons of the KNI are:

Consider if "disadvantages" is more appropriate than "cons". (I'm not a native English speaker.)

> +
> +* It is out-of-tree Linux kernel module and it can't be distributed as
> binary as
> +  part of operating system vendor DPDK packages. This makes it harder
> to
> +  consume, although it is always possible to compile it from the
> source code.
> +
> +* As it shares memory between userspace and kernelspace, and kernel
> part
> +  directly uses input provided by userspace, it is not safe. This
> makes hard to
> +  upstream the module.
> 
> -*   Allows an interface with the kernel network stack.
> +* Only a subset of net devices control commands are supported by KNI.

If it is still relevant, add something along the lines of:
* Requires dedicated kernel cores.

> 
>  The components of an application using the DPDK Kernel NIC Interface
> are shown in :numref:`figure_kernel_nic_intf`.
> 
> --
> 2.31.1
> 

Could you perhaps also promote the virtio-user documentation by moving it from the HowTo Guides to the Programmer's Guide?

-Morten

Re: [PATCH v3 2/2] doc: announce KNI deprecation

[Jerin Jacob]

On Wed, Nov 24, 2021 at 10:48 PM Ferruh Yigit <ferruh.yigit@intel.com> wrote:
>
> Announce the KNI kernel module move to out of dpdk repo and announce
> long term plan to deprecate the KNI.
>
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>

Acked-by: Jerin Jacob <jerinj@marvell.com>


> ---
> Cc: Olivier Matz Olivier Matz <olivier.matz@6wind.com>
> Cc: David Marchand David Marchand <david.marchand@redhat.com>
> Cc: Stephen Hemminger Stephen Hemminger <stephen@networkplumber.org>
> Cc: Elad Nachman <eladv6@gmail.com>
> Cc: Igor Ryzhov <iryzhov@nfware.com>
> Cc: Dan Gora <dg@adax.com>
>
> Dates are not discussed before, the patch aims to trigger a discussion
> for the dates.
> ---
>  doc/guides/prog_guide/kernel_nic_interface.rst | 2 ++
>  doc/guides/rel_notes/deprecation.rst           | 6 ++++++
>  2 files changed, 8 insertions(+)
>
> diff --git a/doc/guides/prog_guide/kernel_nic_interface.rst b/doc/guides/prog_guide/kernel_nic_interface.rst
> index f5a8b7c0782c..d1c5ccd0851d 100644
> --- a/doc/guides/prog_guide/kernel_nic_interface.rst
> +++ b/doc/guides/prog_guide/kernel_nic_interface.rst
> @@ -7,6 +7,8 @@ Kernel NIC Interface
>  ====================
>
>  .. Note::
> +   KNI kernel module will be moved from main git repository to `dpdk-kmods <https://git.dpdk.org/dpdk-kmods/>`_ repository.
> +   There is a long term plan to deprecate the KNI. See :doc:`../rel_notes/deprecation`
>
>     :ref:`virtio_user_as_exceptional_path` alternative is the preferred way for
>     interfacing with the Linux network stack as it is an in-kernel solution and
> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
> index 2262b8de6093..f20852504319 100644
> --- a/doc/guides/rel_notes/deprecation.rst
> +++ b/doc/guides/rel_notes/deprecation.rst
> @@ -48,6 +48,12 @@ Deprecation Notices
>    in the header will not be considered as ABI anymore. This change is inspired
>    by the RFC https://patchwork.dpdk.org/project/dpdk/list/?series=17176.
>
> +* kni: KNI kernel module will be moved to `dpdk-kmods <https://git.dpdk.org/dpdk-kmods/>`_
> +  repository by the `DPDK technical board decision
> +  <https://mails.dpdk.org/archives/dev/2021-January/197077.html>`_, on v22.11.
> +* kni: will be depreciated, will remove all kni lib, kernel module and example code
> +  on v23.11.
> +
>  * lib: will fix extending some enum/define breaking the ABI. There are multiple
>    samples in DPDK that enum/define terminated with a ``.*MAX.*`` value which is
>    used by iterators, and arrays holding these values are sized with this
> --
> 2.31.1
>

Re: [PATCH v3 2/2] doc: announce KNI deprecation

[Thomas Monjalon]

24/11/2021 18:16, Ferruh Yigit:
> Announce the KNI kernel module move to out of dpdk repo and announce
> long term plan to deprecate the KNI.
> 
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> --- a/doc/guides/rel_notes/deprecation.rst
> +++ b/doc/guides/rel_notes/deprecation.rst
> +* kni: KNI kernel module will be moved to `dpdk-kmods <https://git.dpdk.org/dpdk-kmods/>`_
> +  repository by the `DPDK technical board decision
> +  <https://mails.dpdk.org/archives/dev/2021-January/197077.html>`_, on v22.11.
> +* kni: will be depreciated, will remove all kni lib, kernel module and example code
> +  on v23.11.

Bruce sent a notice about the example code. The goal is to remove it in 22.11.

Bruce, should we merge this notice with yours for a clear plan?

Re: [PATCH v3 2/2] doc: announce KNI deprecation

[Bruce Richardson]

On Wed, Jul 13, 2022 at 12:01:28PM +0200, Thomas Monjalon wrote:
> 24/11/2021 18:16, Ferruh Yigit:
> > Announce the KNI kernel module move to out of dpdk repo and announce
> > long term plan to deprecate the KNI.
> > 
> > Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> > --- a/doc/guides/rel_notes/deprecation.rst
> > +++ b/doc/guides/rel_notes/deprecation.rst
> > +* kni: KNI kernel module will be moved to `dpdk-kmods <https://git.dpdk.org/dpdk-kmods/>`_
> > +  repository by the `DPDK technical board decision
> > +  <https://mails.dpdk.org/archives/dev/2021-January/197077.html>`_, on v22.11.
> > +* kni: will be depreciated, will remove all kni lib, kernel module and example code
> > +  on v23.11.
> 
> Bruce sent a notice about the example code. The goal is to remove it in 22.11.
> 
> Bruce, should we merge this notice with yours for a clear plan?
> 
Yes, it would be good to do so. I won't have time to do a patch for this
today, though.

[PATCH v4] doc: announce KNI deprecation

[Thomas Monjalon]

From: Ferruh Yigit <ferruh.yigit@intel.com>

Announce the deprecation plan for KNI kernel module, library and example.

Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
---
Squashed and updated 2 deprecations:
https://patches.dpdk.org/project/dpdk/patch/20211124171609.3101896-2-ferruh.yigit@intel.com/
https://patches.dpdk.org/project/dpdk/patch/20211124171609.3101896-2-ferruh.yigit@intel.com/
---
 doc/guides/prog_guide/kernel_nic_interface.rst | 3 +++
 doc/guides/rel_notes/deprecation.rst           | 9 +++++++++
 2 files changed, 12 insertions(+)

diff --git a/doc/guides/prog_guide/kernel_nic_interface.rst b/doc/guides/prog_guide/kernel_nic_interface.rst
index e021cc69b6..03b5bca958 100644
--- a/doc/guides/prog_guide/kernel_nic_interface.rst
+++ b/doc/guides/prog_guide/kernel_nic_interface.rst
@@ -8,6 +8,9 @@ Kernel NIC Interface
 
 .. note::
 
+   KNI is deprecated and will be removed in future.
+   See :doc:`../rel_notes/deprecation`.
+
    For an alternative to KNI, that does not require any out-of-tree Linux kernel modules,
    or a custom library, see :ref:`virtio_user_as_exception_path`.
 
diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
index 4e5b23c53d..e54597c591 100644
--- a/doc/guides/rel_notes/deprecation.rst
+++ b/doc/guides/rel_notes/deprecation.rst
@@ -48,6 +48,15 @@ Deprecation Notices
   in the header will not be considered as ABI anymore. This change is inspired
   by the RFC https://patchwork.dpdk.org/project/dpdk/list/?series=17176.
 
+* kni: The KNI kernel module and library are not recommended for use by new
+  applications - other technologies such as virtio-user are recommended instead.
+  The KNI kernel module and library will be removed from DPDK 23.11,
+  following the DPDK technical board
+  `decision <https://mails.dpdk.org/archives/dev/2021-January/197077.html>`_
+  and `refinement <http://mails.dpdk.org/archives/dev/2022-June/243596.html>`_.
+  The first steps are to add deprecation warnings
+  and to remove the example application from 22.11.
+
 * lib: will fix extending some enum/define breaking the ABI. There are multiple
   samples in DPDK that enum/define terminated with a ``.*MAX.*`` value which is
   used by iterators, and arrays holding these values are sized with this
-- 
2.36.1

Re: [PATCH v4] doc: announce KNI deprecation

[Thomas Monjalon]

13/07/2022 18:10, Thomas Monjalon:
> From: Ferruh Yigit <ferruh.yigit@intel.com>
> 
> Announce the deprecation plan for KNI kernel module, library and example.
> 
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> ---
> Squashed and updated 2 deprecations:
> https://patches.dpdk.org/project/dpdk/patch/20211124171609.3101896-2-ferruh.yigit@intel.com/
> https://patches.dpdk.org/project/dpdk/patch/20211124171609.3101896-2-ferruh.yigit@intel.com/

This is a mistake. The second deprecation squashed here is
https://patches.dpdk.org/project/dpdk/patch/20220630165031.79183-1-bruce.richardson@intel.com/

> ---
>  doc/guides/prog_guide/kernel_nic_interface.rst | 3 +++
>  doc/guides/rel_notes/deprecation.rst           | 9 +++++++++
>  2 files changed, 12 insertions(+)
> 
> diff --git a/doc/guides/prog_guide/kernel_nic_interface.rst b/doc/guides/prog_guide/kernel_nic_interface.rst
> index e021cc69b6..03b5bca958 100644
> --- a/doc/guides/prog_guide/kernel_nic_interface.rst
> +++ b/doc/guides/prog_guide/kernel_nic_interface.rst
> @@ -8,6 +8,9 @@ Kernel NIC Interface
>  
>  .. note::
>  
> +   KNI is deprecated and will be removed in future.
> +   See :doc:`../rel_notes/deprecation`.
> +
>     For an alternative to KNI, that does not require any out-of-tree Linux kernel modules,
>     or a custom library, see :ref:`virtio_user_as_exception_path`.
>  
> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
> index 4e5b23c53d..e54597c591 100644
> --- a/doc/guides/rel_notes/deprecation.rst
> +++ b/doc/guides/rel_notes/deprecation.rst
> @@ -48,6 +48,15 @@ Deprecation Notices
>    in the header will not be considered as ABI anymore. This change is inspired
>    by the RFC https://patchwork.dpdk.org/project/dpdk/list/?series=17176.
>  
> +* kni: The KNI kernel module and library are not recommended for use by new
> +  applications - other technologies such as virtio-user are recommended instead.
> +  The KNI kernel module and library will be removed from DPDK 23.11,
> +  following the DPDK technical board
> +  `decision <https://mails.dpdk.org/archives/dev/2021-January/197077.html>`_
> +  and `refinement <http://mails.dpdk.org/archives/dev/2022-June/243596.html>`_.
> +  The first steps are to add deprecation warnings
> +  and to remove the example application from 22.11.
> +
>  * lib: will fix extending some enum/define breaking the ABI. There are multiple
>    samples in DPDK that enum/define terminated with a ``.*MAX.*`` value which is
>    used by iterators, and arrays holding these values are sized with this
> 

Re: [PATCH v4] doc: announce KNI deprecation

[Bruce Richardson]

On Wed, Jul 13, 2022 at 06:10:47PM +0200, Thomas Monjalon wrote:
> From: Ferruh Yigit <ferruh.yigit@intel.com>
> 
> Announce the deprecation plan for KNI kernel module, library and example.
> 
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> ---
> Squashed and updated 2 deprecations:
> https://patches.dpdk.org/project/dpdk/patch/20211124171609.3101896-2-ferruh.yigit@intel.com/
> https://patches.dpdk.org/project/dpdk/patch/20211124171609.3101896-2-ferruh.yigit@intel.com/
> ---

One suggestion below.
With corrected link and with/without suggested change:

Acked-by: Bruce Richardson <bruce.richardson@intel.com>

>  doc/guides/prog_guide/kernel_nic_interface.rst | 3 +++
>  doc/guides/rel_notes/deprecation.rst           | 9 +++++++++
>  2 files changed, 12 insertions(+)
> 
> diff --git a/doc/guides/prog_guide/kernel_nic_interface.rst b/doc/guides/prog_guide/kernel_nic_interface.rst
> index e021cc69b6..03b5bca958 100644
> --- a/doc/guides/prog_guide/kernel_nic_interface.rst
> +++ b/doc/guides/prog_guide/kernel_nic_interface.rst
> @@ -8,6 +8,9 @@ Kernel NIC Interface
>  
>  .. note::
>  
> +   KNI is deprecated and will be removed in future.
> +   See :doc:`../rel_notes/deprecation`.
> +
>     For an alternative to KNI, that does not require any out-of-tree Linux kernel modules,
>     or a custom library, see :ref:`virtio_user_as_exception_path`.
>  
> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
> index 4e5b23c53d..e54597c591 100644
> --- a/doc/guides/rel_notes/deprecation.rst
> +++ b/doc/guides/rel_notes/deprecation.rst
> @@ -48,6 +48,15 @@ Deprecation Notices
>    in the header will not be considered as ABI anymore. This change is inspired
>    by the RFC https://patchwork.dpdk.org/project/dpdk/list/?series=17176.
>  
> +* kni: The KNI kernel module and library are not recommended for use by new
> +  applications - other technologies such as virtio-user are recommended instead.
> +  The KNI kernel module and library will be removed from DPDK 23.11,
> +  following the DPDK technical board
> +  `decision <https://mails.dpdk.org/archives/dev/2021-January/197077.html>`_
> +  and `refinement <http://mails.dpdk.org/archives/dev/2022-June/243596.html>`_.
> +  The first steps are to add deprecation warnings
> +  and to remove the example application from 22.11.
> +

I wonder whether having this done as bullet points in chronological order
might be clearer. Something like:

* kni: The KNI kernel module and library are not recommended for use by new
  applications - other technologies such as virtio-user are recommended instead.
  Following the DPDK technical board
  `decision <https://mails.dpdk.org/archives/dev/2021-January/197077.html>`_
  and `refinement <http://mails.dpdk.org/archives/dev/2022-June/243596.html>`_:
  * The KNI example application will be removed from DPDK 22.11 release
  * The KNI kernel module and library will be removed from the DPDK 23.11
    release

Re: [PATCH v4] doc: announce KNI deprecation

[David Marchand]

On Wed, Jul 13, 2022 at 6:10 PM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> From: Ferruh Yigit <ferruh.yigit@intel.com>
>
> Announce the deprecation plan for KNI kernel module, library and example.
>
> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> Signed-off-by: Thomas Monjalon <thomas@monjalon.net>

Acked-by: David Marchand <david.marchand@redhat.com>


-- 
David Marchand

Re: [PATCH v4] doc: announce KNI deprecation

[Thomas Monjalon]

14/07/2022 11:20, Bruce Richardson:
> On Wed, Jul 13, 2022 at 06:10:47PM +0200, Thomas Monjalon wrote:
> > From: Ferruh Yigit <ferruh.yigit@intel.com>
> > 
> > Announce the deprecation plan for KNI kernel module, library and example.
> > 
> > Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> > Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> > ---
> > Squashed and updated 2 deprecations:
> > https://patches.dpdk.org/project/dpdk/patch/20211124171609.3101896-2-ferruh.yigit@intel.com/
> > https://patches.dpdk.org/project/dpdk/patch/20211124171609.3101896-2-ferruh.yigit@intel.com/
> > ---
> 
> One suggestion below.
> With corrected link and with/without suggested change:
> 
> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
> 
> >  doc/guides/prog_guide/kernel_nic_interface.rst | 3 +++
> >  doc/guides/rel_notes/deprecation.rst           | 9 +++++++++
> >  2 files changed, 12 insertions(+)
> > 
> > diff --git a/doc/guides/prog_guide/kernel_nic_interface.rst b/doc/guides/prog_guide/kernel_nic_interface.rst
> > index e021cc69b6..03b5bca958 100644
> > --- a/doc/guides/prog_guide/kernel_nic_interface.rst
> > +++ b/doc/guides/prog_guide/kernel_nic_interface.rst
> > @@ -8,6 +8,9 @@ Kernel NIC Interface
> >  
> >  .. note::
> >  
> > +   KNI is deprecated and will be removed in future.
> > +   See :doc:`../rel_notes/deprecation`.
> > +
> >     For an alternative to KNI, that does not require any out-of-tree Linux kernel modules,
> >     or a custom library, see :ref:`virtio_user_as_exception_path`.
> >  
> > diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
> > index 4e5b23c53d..e54597c591 100644
> > --- a/doc/guides/rel_notes/deprecation.rst
> > +++ b/doc/guides/rel_notes/deprecation.rst
> > @@ -48,6 +48,15 @@ Deprecation Notices
> >    in the header will not be considered as ABI anymore. This change is inspired
> >    by the RFC https://patchwork.dpdk.org/project/dpdk/list/?series=17176.
> >  
> > +* kni: The KNI kernel module and library are not recommended for use by new
> > +  applications - other technologies such as virtio-user are recommended instead.
> > +  The KNI kernel module and library will be removed from DPDK 23.11,
> > +  following the DPDK technical board
> > +  `decision <https://mails.dpdk.org/archives/dev/2021-January/197077.html>`_
> > +  and `refinement <http://mails.dpdk.org/archives/dev/2022-June/243596.html>`_.
> > +  The first steps are to add deprecation warnings
> > +  and to remove the example application from 22.11.
> > +
> 
> I wonder whether having this done as bullet points in chronological order
> might be clearer. Something like:
> 
> * kni: The KNI kernel module and library are not recommended for use by new
>   applications - other technologies such as virtio-user are recommended instead.
>   Following the DPDK technical board
>   `decision <https://mails.dpdk.org/archives/dev/2021-January/197077.html>`_
>   and `refinement <http://mails.dpdk.org/archives/dev/2022-June/243596.html>`_:
>   * The KNI example application will be removed from DPDK 22.11 release
>   * The KNI kernel module and library will be removed from the DPDK 23.11
>     release

Yes I prefer your version. Bullets are always better :)

Re: [PATCH v4] doc: announce KNI deprecation

[Ferruh Yigit]

On 7/14/2022 10:20 AM, Bruce Richardson wrote:
> On Wed, Jul 13, 2022 at 06:10:47PM +0200, Thomas Monjalon wrote:
>> From: Ferruh Yigit <ferruh.yigit@intel.com>
>>
>> Announce the deprecation plan for KNI kernel module, library and example.
>>
>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>> Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
>> ---
>> Squashed and updated 2 deprecations:
>> https://patches.dpdk.org/project/dpdk/patch/20211124171609.3101896-2-ferruh.yigit@intel.com/
>> https://patches.dpdk.org/project/dpdk/patch/20211124171609.3101896-2-ferruh.yigit@intel.com/
>> ---
> 
> One suggestion below.
> With corrected link and with/without suggested change:
> 
> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
> 
>>   doc/guides/prog_guide/kernel_nic_interface.rst | 3 +++
>>   doc/guides/rel_notes/deprecation.rst           | 9 +++++++++
>>   2 files changed, 12 insertions(+)
>>
>> diff --git a/doc/guides/prog_guide/kernel_nic_interface.rst b/doc/guides/prog_guide/kernel_nic_interface.rst
>> index e021cc69b6..03b5bca958 100644
>> --- a/doc/guides/prog_guide/kernel_nic_interface.rst
>> +++ b/doc/guides/prog_guide/kernel_nic_interface.rst
>> @@ -8,6 +8,9 @@ Kernel NIC Interface
>>   
>>   .. note::
>>   
>> +   KNI is deprecated and will be removed in future.
>> +   See :doc:`../rel_notes/deprecation`.
>> +
>>      For an alternative to KNI, that does not require any out-of-tree Linux kernel modules,
>>      or a custom library, see :ref:`virtio_user_as_exception_path`.
>>   
>> diff --git a/doc/guides/rel_notes/deprecation.rst b/doc/guides/rel_notes/deprecation.rst
>> index 4e5b23c53d..e54597c591 100644
>> --- a/doc/guides/rel_notes/deprecation.rst
>> +++ b/doc/guides/rel_notes/deprecation.rst
>> @@ -48,6 +48,15 @@ Deprecation Notices
>>     in the header will not be considered as ABI anymore. This change is inspired
>>     by the RFC https://patchwork.dpdk.org/project/dpdk/list/?series=17176.
>>   
>> +* kni: The KNI kernel module and library are not recommended for use by new
>> +  applications - other technologies such as virtio-user are recommended instead.
>> +  The KNI kernel module and library will be removed from DPDK 23.11,
>> +  following the DPDK technical board
>> +  `decision <https://mails.dpdk.org/archives/dev/2021-January/197077.html>`_
>> +  and `refinement <http://mails.dpdk.org/archives/dev/2022-June/243596.html>`_.
>> +  The first steps are to add deprecation warnings
>> +  and to remove the example application from 22.11.
>> +
> 
> I wonder whether having this done as bullet points in chronological order
> might be clearer. Something like:
> 
> * kni: The KNI kernel module and library are not recommended for use by new
>    applications - other technologies such as virtio-user are recommended instead.
>    Following the DPDK technical board
>    `decision <https://mails.dpdk.org/archives/dev/2021-January/197077.html>`_
>    and `refinement <http://mails.dpdk.org/archives/dev/2022-June/243596.html>`_:
>    * The KNI example application will be removed from DPDK 22.11 release
>    * The KNI kernel module and library will be removed from the DPDK 23.11
>      release

+1 to list actions in chronological order

I have a concern with removing sample application without replacing one 
with alternate methods: virtio-user and tun/tap.

It is easy to create a virtio-user PMD by testpmd or any sample 
application, and as far as I understand Bruce already documented this. 
But for many KNI users they are not using KNI PMD, so replacing KNI with 
this new method may require some hand holding.
One option can be hotplug the virtio-user PMD and use the port_id for 
packet forwarding, and I don't know if the PMD has APIs that 
applications can use directly, as done in KNI, @Maxime & @Chenbo can 
answer this better.

It can be good to have a sample application for above before deprecating 
the KNI sample application.
And same sample can use tun/tap PMD with a runtime parameter, to show 
how other example can be used, again tap can be used as tap PMD or Linux 
tun/tap APIs.

Question is who can work on such a sample, but if we can find some 
resource I am for having a replacement exception path sample app before 
deprecating KNI.

Thanks,
ferruh

Re: [PATCH v4] doc: announce KNI deprecation

[Ferruh Yigit]

On 7/14/2022 3:50 PM, Ferruh Yigit wrote:
> On 7/14/2022 10:20 AM, Bruce Richardson wrote:
>> On Wed, Jul 13, 2022 at 06:10:47PM +0200, Thomas Monjalon wrote:
>>> From: Ferruh Yigit <ferruh.yigit@intel.com>
>>>
>>> Announce the deprecation plan for KNI kernel module, library and 
>>> example.
>>>
>>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>>> Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
>>> ---
>>> Squashed and updated 2 deprecations:
>>> https://patches.dpdk.org/project/dpdk/patch/20211124171609.3101896-2-ferruh.yigit@intel.com/
>>> https://patches.dpdk.org/project/dpdk/patch/20211124171609.3101896-2-ferruh.yigit@intel.com/
>>> ---
>>
>> One suggestion below.
>> With corrected link and with/without suggested change:
>>
>> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
>>
>>>   doc/guides/prog_guide/kernel_nic_interface.rst | 3 +++
>>>   doc/guides/rel_notes/deprecation.rst           | 9 +++++++++
>>>   2 files changed, 12 insertions(+)
>>>
>>> diff --git a/doc/guides/prog_guide/kernel_nic_interface.rst 
>>> b/doc/guides/prog_guide/kernel_nic_interface.rst
>>> index e021cc69b6..03b5bca958 100644
>>> --- a/doc/guides/prog_guide/kernel_nic_interface.rst
>>> +++ b/doc/guides/prog_guide/kernel_nic_interface.rst
>>> @@ -8,6 +8,9 @@ Kernel NIC Interface
>>>   .. note::
>>> +   KNI is deprecated and will be removed in future.
>>> +   See :doc:`../rel_notes/deprecation`.
>>> +
>>>      For an alternative to KNI, that does not require any out-of-tree 
>>> Linux kernel modules,
>>>      or a custom library, see :ref:`virtio_user_as_exception_path`.
>>> diff --git a/doc/guides/rel_notes/deprecation.rst 
>>> b/doc/guides/rel_notes/deprecation.rst
>>> index 4e5b23c53d..e54597c591 100644
>>> --- a/doc/guides/rel_notes/deprecation.rst
>>> +++ b/doc/guides/rel_notes/deprecation.rst
>>> @@ -48,6 +48,15 @@ Deprecation Notices
>>>     in the header will not be considered as ABI anymore. This change 
>>> is inspired
>>>     by the RFC 
>>> https://patchwork.dpdk.org/project/dpdk/list/?series=17176.
>>> +* kni: The KNI kernel module and library are not recommended for use 
>>> by new
>>> +  applications - other technologies such as virtio-user are 
>>> recommended instead.
>>> +  The KNI kernel module and library will be removed from DPDK 23.11,
>>> +  following the DPDK technical board
>>> +  `decision 
>>> <https://mails.dpdk.org/archives/dev/2021-January/197077.html>`_
>>> +  and `refinement 
>>> <http://mails.dpdk.org/archives/dev/2022-June/243596.html>`_.
>>> +  The first steps are to add deprecation warnings
>>> +  and to remove the example application from 22.11.
>>> +
>>
>> I wonder whether having this done as bullet points in chronological order
>> might be clearer. Something like:
>>
>> * kni: The KNI kernel module and library are not recommended for use 
>> by new
>>    applications - other technologies such as virtio-user are 
>> recommended instead.
>>    Following the DPDK technical board
>>    `decision 
>> <https://mails.dpdk.org/archives/dev/2021-January/197077.html>`_
>>    and `refinement 
>> <http://mails.dpdk.org/archives/dev/2022-June/243596.html>`_:
>>    * The KNI example application will be removed from DPDK 22.11 release
>>    * The KNI kernel module and library will be removed from the DPDK 
>> 23.11
>>      release
> 
> +1 to list actions in chronological order
> 
> I have a concern with removing sample application without replacing one 
> with alternate methods: virtio-user and tun/tap.
> 
> It is easy to create a virtio-user PMD by testpmd or any sample 
> application, and as far as I understand Bruce already documented this. 
> But for many KNI users they are not using KNI PMD, so replacing KNI with 
> this new method may require some hand holding.

As mentioned KNI PMD, it also should be removed when library and module 
are removed, may be good to list above to be explicit.

> One option can be hotplug the virtio-user PMD and use the port_id for 
> packet forwarding, and I don't know if the PMD has APIs that 
> applications can use directly, as done in KNI, @Maxime & @Chenbo can 
> answer this better.
> 
> It can be good to have a sample application for above before deprecating 
> the KNI sample application.
> And same sample can use tun/tap PMD with a runtime parameter, to show 
> how other example can be used, again tap can be used as tap PMD or Linux 
> tun/tap APIs.
> 
> Question is who can work on such a sample, but if we can find some 
> resource I am for having a replacement exception path sample app before 
> deprecating KNI.
> 
> Thanks,
> ferruh
> 
> 
> 

Re: [PATCH v4] doc: announce KNI deprecation

[Thomas Monjalon]

14/07/2022 17:03, Ferruh Yigit:
> On 7/14/2022 3:50 PM, Ferruh Yigit wrote:
> > On 7/14/2022 10:20 AM, Bruce Richardson wrote:
> >> On Wed, Jul 13, 2022 at 06:10:47PM +0200, Thomas Monjalon wrote:
> >>> From: Ferruh Yigit <ferruh.yigit@intel.com>
> >>>
> >>> Announce the deprecation plan for KNI kernel module, library and 
> >>> example.
> >>>
> >>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> >>> Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> >>> ---
> >>> Squashed and updated 2 deprecations:
> >>> https://patches.dpdk.org/project/dpdk/patch/20211124171609.3101896-2-ferruh.yigit@intel.com/
> >>> https://patches.dpdk.org/project/dpdk/patch/20211124171609.3101896-2-ferruh.yigit@intel.com/
> >>> ---
> >>
> >> One suggestion below.
> >> With corrected link and with/without suggested change:
> >>
> >> Acked-by: Bruce Richardson <bruce.richardson@intel.com>
> >>
> >>>   doc/guides/prog_guide/kernel_nic_interface.rst | 3 +++
> >>>   doc/guides/rel_notes/deprecation.rst           | 9 +++++++++
> >>>   2 files changed, 12 insertions(+)
> >>>
> >>> diff --git a/doc/guides/prog_guide/kernel_nic_interface.rst 
> >>> b/doc/guides/prog_guide/kernel_nic_interface.rst
> >>> index e021cc69b6..03b5bca958 100644
> >>> --- a/doc/guides/prog_guide/kernel_nic_interface.rst
> >>> +++ b/doc/guides/prog_guide/kernel_nic_interface.rst
> >>> @@ -8,6 +8,9 @@ Kernel NIC Interface
> >>>   .. note::
> >>> +   KNI is deprecated and will be removed in future.
> >>> +   See :doc:`../rel_notes/deprecation`.
> >>> +
> >>>      For an alternative to KNI, that does not require any out-of-tree 
> >>> Linux kernel modules,
> >>>      or a custom library, see :ref:`virtio_user_as_exception_path`.
> >>> diff --git a/doc/guides/rel_notes/deprecation.rst 
> >>> b/doc/guides/rel_notes/deprecation.rst
> >>> index 4e5b23c53d..e54597c591 100644
> >>> --- a/doc/guides/rel_notes/deprecation.rst
> >>> +++ b/doc/guides/rel_notes/deprecation.rst
> >>> @@ -48,6 +48,15 @@ Deprecation Notices
> >>>     in the header will not be considered as ABI anymore. This change 
> >>> is inspired
> >>>     by the RFC 
> >>> https://patchwork.dpdk.org/project/dpdk/list/?series=17176.
> >>> +* kni: The KNI kernel module and library are not recommended for use 
> >>> by new
> >>> +  applications - other technologies such as virtio-user are 
> >>> recommended instead.
> >>> +  The KNI kernel module and library will be removed from DPDK 23.11,
> >>> +  following the DPDK technical board
> >>> +  `decision 
> >>> <https://mails.dpdk.org/archives/dev/2021-January/197077.html>`_
> >>> +  and `refinement 
> >>> <http://mails.dpdk.org/archives/dev/2022-June/243596.html>`_.
> >>> +  The first steps are to add deprecation warnings
> >>> +  and to remove the example application from 22.11.
> >>> +
> >>
> >> I wonder whether having this done as bullet points in chronological order
> >> might be clearer. Something like:
> >>
> >> * kni: The KNI kernel module and library are not recommended for use 
> >> by new
> >>    applications - other technologies such as virtio-user are 
> >> recommended instead.
> >>    Following the DPDK technical board
> >>    `decision 
> >> <https://mails.dpdk.org/archives/dev/2021-January/197077.html>`_
> >>    and `refinement 
> >> <http://mails.dpdk.org/archives/dev/2022-June/243596.html>`_:
> >>    * The KNI example application will be removed from DPDK 22.11 release
> >>    * The KNI kernel module and library will be removed from the DPDK 
> >> 23.11
> >>      release
> > 
> > +1 to list actions in chronological order
> > 
> > I have a concern with removing sample application without replacing one 
> > with alternate methods: virtio-user and tun/tap.
> > 
> > It is easy to create a virtio-user PMD by testpmd or any sample 
> > application, and as far as I understand Bruce already documented this. 
> > But for many KNI users they are not using KNI PMD, so replacing KNI with 
> > this new method may require some hand holding.
> 
> As mentioned KNI PMD, it also should be removed when library and module 
> are removed, may be good to list above to be explicit.

Yes I'm adding the PMD in removal list.

> > One option can be hotplug the virtio-user PMD and use the port_id for 
> > packet forwarding, and I don't know if the PMD has APIs that 
> > applications can use directly, as done in KNI, @Maxime & @Chenbo can 
> > answer this better.
> > 
> > It can be good to have a sample application for above before deprecating 
> > the KNI sample application.
> > And same sample can use tun/tap PMD with a runtime parameter, to show 
> > how other example can be used, again tap can be used as tap PMD or Linux 
> > tun/tap APIs.
> > 
> > Question is who can work on such a sample, but if we can find some 
> > resource I am for having a replacement exception path sample app before 
> > deprecating KNI.

We are not sure we'll have time and resource,
and it has not been discussed earlier.
Thus I'll consider having a replacement example as a nice-to-have.
I have to be a bit conservative with techboard decisions.

Re: [PATCH v4] doc: announce KNI deprecation

[Thomas Monjalon]

13/07/2022 21:51, Thomas Monjalon:
> 13/07/2022 18:10, Thomas Monjalon:
> > From: Ferruh Yigit <ferruh.yigit@intel.com>
> > 
> > Announce the deprecation plan for KNI kernel module, library and example.
> > 
> > Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
> > Signed-off-by: Thomas Monjalon <thomas@monjalon.net>
> > ---
> > Squashed and updated 2 deprecations:
> > https://patches.dpdk.org/project/dpdk/patch/20211124171609.3101896-2-ferruh.yigit@intel.com/
> > https://patches.dpdk.org/project/dpdk/patch/20211124171609.3101896-2-ferruh.yigit@intel.com/
> 
> This is a mistake. The second deprecation squashed here is
> https://patches.dpdk.org/project/dpdk/patch/20220630165031.79183-1-bruce.richardson@intel.com/

Applied with minor improvements discussed in the thread:
	- ordered list of actions
	- PMD to be removed in 23.11

Re: [PATCH v3 1/2] doc: note KNI alternatives

[Ferruh Yigit]

On 12/1/2021 4:31 PM, Morten Brørup wrote:
>> From: Ferruh Yigit [mailto:ferruh.yigit@intel.com]
>> Sent: Wednesday, 24 November 2021 18.16
>>
>> Add more information on alternatives of KNI and the cons of KNI against
>> these alternatives.
>>
>> Signed-off-by: Ferruh Yigit <ferruh.yigit@intel.com>
>> ---
>> Cc: Olivier Matz Olivier Matz <olivier.matz@6wind.com>
>> Cc: David Marchand David Marchand <david.marchand@redhat.com>
>> Cc: Stephen Hemminger Stephen Hemminger <stephen@networkplumber.org>
>> Cc: Elad Nachman <eladv6@gmail.com>
>> Cc: Igor Ryzhov <iryzhov@nfware.com>
>> Cc: Dan Gora <dg@adax.com>
>>
>> v3:
>> * reference tap document directly instead of adding label to it.
>> ---
>>  .../prog_guide/kernel_nic_interface.rst       | 34 +++++++++++++++++--
>>  1 file changed, 31 insertions(+), 3 deletions(-)
>>
>> diff --git a/doc/guides/prog_guide/kernel_nic_interface.rst
>> b/doc/guides/prog_guide/kernel_nic_interface.rst
>> index 1ce03ec1a374..f5a8b7c0782c 100644
>> --- a/doc/guides/prog_guide/kernel_nic_interface.rst
>> +++ b/doc/guides/prog_guide/kernel_nic_interface.rst
>> @@ -6,16 +6,44 @@
>>  Kernel NIC Interface
>>  ====================
>>
>> +.. Note::
>> +
>> +   :ref:`virtio_user_as_exceptional_path` alternative is the preferred
>> way for
>> +   interfacing with the Linux network stack as it is an in-kernel
>> solution and
>> +   has similar performance expectations.
>> +
>>  The DPDK Kernel NIC Interface (KNI) allows userspace applications
>> access to the Linux* control plane.
>>
>> -The benefits of using the DPDK KNI are:
>> +KNI provides an interface with the kernel network stack and allows
>> management of
>> +DPDK ports using standard Linux net tools such as ``ethtool``,
>> ``ifconfig`` and
>> +``tcpdump``.
>> +
>> +The main use case of KNI is to get/receive exception packets from/to
>> Linux network
>> +stack while main datapath IO is done bypassing the networking stack.
>> +
>> +There are other alternatives to KNI, all are available in the upstream
>> Linux:
>> +
>> +#. :ref:`virtio_user_as_exceptional_path`
>> +
>> +#. :doc:`../nics/tap` as wrapper to `Linux tun/tap
>> +   <https://www.kernel.org/doc/Documentation/networking/tuntap.txt>`_
>> +
>> +The benefits of using the KNI against alternatives are:
>>
>>  *   Faster than existing Linux TUN/TAP interfaces
>>      (by eliminating system calls and copy_to_user()/copy_from_user()
>> operations.
>>
>> -*   Allows management of DPDK ports using standard Linux net tools
>> such as ethtool, ifconfig and tcpdump.
>> +The cons of the KNI are:
> 
> Consider if "disadvantages" is more appropriate than "cons". (I'm not a native English speaker.)
> 

Both fine for me, OK to use 'disadvantages'.

>> +
>> +* It is out-of-tree Linux kernel module and it can't be distributed as
>> binary as
>> +  part of operating system vendor DPDK packages. This makes it harder
>> to
>> +  consume, although it is always possible to compile it from the
>> source code.
>> +
>> +* As it shares memory between userspace and kernelspace, and kernel
>> part
>> +  directly uses input provided by userspace, it is not safe. This
>> makes hard to
>> +  upstream the module.
>>
>> -*   Allows an interface with the kernel network stack.
>> +* Only a subset of net devices control commands are supported by KNI.
> 
> If it is still relevant, add something along the lines of:
> * Requires dedicated kernel cores.
> 

ack

>>
>>  The components of an application using the DPDK Kernel NIC Interface
>> are shown in :numref:`figure_kernel_nic_intf`.
>>
>> --
>> 2.31.1
>>
> 
> Could you perhaps also promote the virtio-user documentation by moving it from the HowTo Guides to the Programmer's Guide?
> 

Hi Morten,

This is an old patch, I assumed merged v4 by Thomas superseded this
patch, but it seems this patch was not involved in that, so I will make
a new version.

For the virtio-user documentation, no objection to move it to
programmer's guide, I hope Maxime and Chenbo can help there.

[PATCH v5] doc: note KNI alternatives

[Ferruh Yigit]

Add more information on alternatives of KNI and the disadvantages of KNI
against these alternatives.

Signed-off-by: Ferruh Yigit <ferruh.yigit@amd.com>
---
Cc: Olivier Matz <olivier.matz@6wind.com>
Cc: David Marchand <david.marchand@redhat.com>
Cc: Stephen Hemminger <stephen@networkplumber.org>
Cc: Elad Nachman <eladv6@gmail.com>
Cc: Igor Ryzhov <iryzhov@nfware.com>
Cc: Dan Gora <dg@adax.com>
Cc: Morten Brørup <mb@smartsharesystems.com>

v5:
* Rebased
* Dedicated kernel core added as disadvantage
---
 .../prog_guide/kernel_nic_interface.rst       | 35 ++++++++++++++++---
 1 file changed, 30 insertions(+), 5 deletions(-)

diff --git a/doc/guides/prog_guide/kernel_nic_interface.rst b/doc/guides/prog_guide/kernel_nic_interface.rst
index 6a564f61cad8..cac0ab0cb278 100644
--- a/doc/guides/prog_guide/kernel_nic_interface.rst
+++ b/doc/guides/prog_guide/kernel_nic_interface.rst
@@ -11,8 +11,9 @@ Kernel NIC Interface
    KNI is deprecated and will be removed in future.
    See :doc:`../rel_notes/deprecation`.
 
-   For an alternative to KNI, that does not require any out-of-tree Linux kernel modules,
-   or a custom library, see :ref:`virtio_user_as_exception_path`.
+   :ref:`virtio_user_as_exception_path` alternative is the preferred way for
+   interfacing with the Linux network stack as it is an in-kernel solution and
+   has similar performance expectations.
 
 .. note::
 
@@ -21,14 +22,38 @@ Kernel NIC Interface
 
 The DPDK Kernel NIC Interface (KNI) allows userspace applications access to the Linux* control plane.
 
-The benefits of using the DPDK KNI are:
+KNI provides an interface with the kernel network stack and allows management of
+DPDK ports using standard Linux net tools such as ``ethtool``, ``ifconfig`` and
+``tcpdump``.
+
+The main use case of KNI is to get/receive exception packets from/to Linux network
+stack while main datapath IO is done bypassing the networking stack.
+
+There are other alternatives to KNI, all are available in the upstream Linux:
+
+#. :ref:`virtio_user_as_exception_path`
+
+#. :doc:`../nics/tap` as wrapper to `Linux tun/tap
+   <https://www.kernel.org/doc/Documentation/networking/tuntap.txt>`_
+
+The benefits of using the KNI against alternatives are:
 
 *   Faster than existing Linux TUN/TAP interfaces
     (by eliminating system calls and copy_to_user()/copy_from_user() operations.
 
-*   Allows management of DPDK ports using standard Linux net tools such as ethtool, ifconfig and tcpdump.
+The disadvantages of the KNI are:
+
+* It is out-of-tree Linux kernel module and it can't be distributed as binary as
+  part of operating system vendor DPDK packages. This makes it harder to
+  consume, although it is always possible to compile it from the source code.
+
+* As it shares memory between userspace and kernelspace, and kernel part
+  directly uses input provided by userspace, it is not safe. This makes hard to
+  upstream the module.
+
+* Requires dedicated kernel cores.
 
-*   Allows an interface with the kernel network stack.
+* Only a subset of net devices control commands are supported by KNI.
 
 The components of an application using the DPDK Kernel NIC Interface are shown in :numref:`figure_kernel_nic_intf`.
 
-- 
2.25.1

Re: [PATCH v5] doc: note KNI alternatives

[Stephen Hemminger]

On Thu, 24 Nov 2022 11:40:40 +0000
Ferruh Yigit <ferruh.yigit@amd.com> wrote:

> +* It is out-of-tree Linux kernel module and it can't be distributed as binary as
> +  part of operating system vendor DPDK packages. This makes it harder to
> +  consume, although it is always possible to compile it from the source code.

To be precise, there is no legal restriction here. Maybe something that describes
why the process is awkward.

* It is out-of-tree Linux kernel module which makes updating and distributing the
  driver more difficult. Most users end up building the KNI driver from source
  which requires the packages and tools to build kernel modules.

Re: [PATCH v5] doc: note KNI alternatives

[Ferruh Yigit]

On 11/24/2022 4:55 PM, Stephen Hemminger wrote:
> On Thu, 24 Nov 2022 11:40:40 +0000
> Ferruh Yigit <ferruh.yigit@amd.com> wrote:
> 
>> +* It is out-of-tree Linux kernel module and it can't be distributed as binary as
>> +  part of operating system vendor DPDK packages. This makes it harder to
>> +  consume, although it is always possible to compile it from the source code.
> 
> To be precise, there is no legal restriction here. Maybe something that describes
> why the process is awkward.
> 
> * It is out-of-tree Linux kernel module which makes updating and distributing the
>   driver more difficult. Most users end up building the KNI driver from source
>   which requires the packages and tools to build kernel modules.

ack, I will use above paragraph

[PATCH v6] doc: note KNI alternatives

[Ferruh Yigit]

Add more information on alternatives of KNI and the disadvantages of KNI
against these alternatives.

Signed-off-by: Ferruh Yigit <ferruh.yigit@amd.com>
---
Cc: Olivier Matz <olivier.matz@6wind.com>
Cc: David Marchand <david.marchand@redhat.com>
Cc: Stephen Hemminger <stephen@networkplumber.org>
Cc: Elad Nachman <eladv6@gmail.com>
Cc: Igor Ryzhov <iryzhov@nfware.com>
Cc: Dan Gora <dg@adax.com>
Cc: Morten Brørup <mb@smartsharesystems.com>

v6:
* s/ifconfig/iproute2/
* reword distributing binary module disadvantage

v5:
* Rebased
* Dedicated kernel core added as disadvantage
---
 .../prog_guide/kernel_nic_interface.rst       | 35 ++++++++++++++++---
 1 file changed, 30 insertions(+), 5 deletions(-)

diff --git a/doc/guides/prog_guide/kernel_nic_interface.rst b/doc/guides/prog_guide/kernel_nic_interface.rst
index 6a564f61cad8..fe088f0c4508 100644
--- a/doc/guides/prog_guide/kernel_nic_interface.rst
+++ b/doc/guides/prog_guide/kernel_nic_interface.rst
@@ -11,8 +11,9 @@ Kernel NIC Interface
    KNI is deprecated and will be removed in future.
    See :doc:`../rel_notes/deprecation`.
 
-   For an alternative to KNI, that does not require any out-of-tree Linux kernel modules,
-   or a custom library, see :ref:`virtio_user_as_exception_path`.
+   :ref:`virtio_user_as_exception_path` alternative is the preferred way for
+   interfacing with the Linux network stack as it is an in-kernel solution and
+   has similar performance expectations.
 
 .. note::
 
@@ -21,14 +22,38 @@ Kernel NIC Interface
 
 The DPDK Kernel NIC Interface (KNI) allows userspace applications access to the Linux* control plane.
 
-The benefits of using the DPDK KNI are:
+KNI provides an interface with the kernel network stack and allows management of
+DPDK ports using standard Linux net tools such as ``ethtool``, ``iproute2`` and
+``tcpdump``.
+
+The main use case of KNI is to get/receive exception packets from/to Linux network
+stack while main datapath IO is done bypassing the networking stack.
+
+There are other alternatives to KNI, all are available in the upstream Linux:
+
+#. :ref:`virtio_user_as_exception_path`
+
+#. :doc:`../nics/tap` as wrapper to `Linux tun/tap
+   <https://www.kernel.org/doc/Documentation/networking/tuntap.txt>`_
+
+The benefits of using the KNI against alternatives are:
 
 *   Faster than existing Linux TUN/TAP interfaces
     (by eliminating system calls and copy_to_user()/copy_from_user() operations.
 
-*   Allows management of DPDK ports using standard Linux net tools such as ethtool, ifconfig and tcpdump.
+The disadvantages of the KNI are:
+
+* It is out-of-tree Linux kernel module which makes updating and distributing the
+  driver more difficult. Most users end up building the KNI driver from source
+  which requires the packages and tools to build kernel modules.
+
+* As it shares memory between userspace and kernelspace, and kernel part
+  directly uses input provided by userspace, it is not safe. This makes hard to
+  upstream the module.
+
+* Requires dedicated kernel cores.
 
-*   Allows an interface with the kernel network stack.
+* Only a subset of net devices control commands are supported by KNI.
 
 The components of an application using the DPDK Kernel NIC Interface are shown in :numref:`figure_kernel_nic_intf`.
 
-- 
2.25.1

Re: [PATCH v6] doc: note KNI alternatives

[Thomas Monjalon]

24/11/2022 18:15, Ferruh Yigit:
> Add more information on alternatives of KNI and the disadvantages of KNI
> against these alternatives.
> 
> Signed-off-by: Ferruh Yigit <ferruh.yigit@amd.com>

Applied, thanks.