[dpdk-dev] KNI alternatives

[dpdk-dev] KNI alternatives

[Thomas Monjalon]

Hi,

As discussed today in the techboard meeting, KNI has probably
better alternatives today without using an out-of-tree module.
Virtio-user is a good candidate to replace KNI.
What is the performance of TAP?
Is there a way to leverage af_packet, af_xdp, or even pcap interfaces?

In order to avoid using the KNI out-of-tree module,
we should make the librte_kni compatible with an other interface.
The big plan is then to move the KNI module out of the main DPDK repo,
as we did for igb_uio.

The first step of this plan might be to document pros & cons
of the KNI alternatives inside the KNI documentation:
	https://doc.dpdk.org/guides/prog_guide/kernel_nic_interface.html

As a start, this doc should be better referenced:
	https://doc.dpdk.org/guides/howto/virtio_user_as_exceptional_path.html

Note: I won't do this update myself, so feel free to step up!
Thanks

Re: [dpdk-dev] KNI alternatives

[Stephen Hemminger]

Last time I tried.
Virtio user was as fast as KNI and consumed less cpu. Was seeing 10mpps
Tap was much slower. Like 1mpps.

Vpp uses virtio user.

Sorry for top post. Only have phone internet

On Wed, Jan 13, 2021, 9:06 AM Thomas Monjalon <thomas@monjalon.net> wrote:

> Hi,
>
> As discussed today in the techboard meeting, KNI has probably
> better alternatives today without using an out-of-tree module.
> Virtio-user is a good candidate to replace KNI.
> What is the performance of TAP?
> Is there a way to leverage af_packet, af_xdp, or even pcap interfaces?
>
> In order to avoid using the KNI out-of-tree module,
> we should make the librte_kni compatible with an other interface.
> The big plan is then to move the KNI module out of the main DPDK repo,
> as we did for igb_uio.
>
> The first step of this plan might be to document pros & cons
> of the KNI alternatives inside the KNI documentation:
>         https://doc.dpdk.org/guides/prog_guide/kernel_nic_interface.html
>
> As a start, this doc should be better referenced:
>
> https://doc.dpdk.org/guides/howto/virtio_user_as_exceptional_path.html
>
> Note: I won't do this update myself, so feel free to step up!
> Thanks
>
>
>

Re: [dpdk-dev] KNI alternatives

[Igor Ryzhov]

On Wed, Jan 13, 2021 at 8:10 PM Stephen Hemminger <
stephen@networkplumber.org> wrote:

> Last time I tried.
> Virtio user was as fast as KNI and consumed less cpu. Was seeing 10mpps
> Tap was much slower. Like 1mpps.
>
> Vpp uses virtio user.
>
> Sorry for top post. Only have phone internet
>
> On Wed, Jan 13, 2021, 9:06 AM Thomas Monjalon <thomas@monjalon.net> wrote:
>
> > Hi,
> >
> > As discussed today in the techboard meeting, KNI has probably
> > better alternatives today without using an out-of-tree module.
> > Virtio-user is a good candidate to replace KNI.
> > What is the performance of TAP?
> > Is there a way to leverage af_packet, af_xdp, or even pcap interfaces?


Performance is not the only question. The advantage of KNI we are currently
using is
the ability to control the DPDK interfaces by the kernel.
For example, to implement bonding in the DPDK application, it is possible
to create KNI
pair for each physical interface, create a bond interface in Linux over
those KNI interfaces
and just pass LACP packets between the app and the kernel. The kernel
itself will control
MACs, MTU, etc. of underlying interfaces. AFAIK it's not possible with
virtio-user or tap.
Am I wrong?


>

>
> > In order to avoid using the KNI out-of-tree module,
> > we should make the librte_kni compatible with an other interface.
> > The big plan is then to move the KNI module out of the main DPDK repo,
> > as we did for igb_uio.
> >
> > The first step of this plan might be to document pros & cons
> > of the KNI alternatives inside the KNI documentation:
> >         https://doc.dpdk.org/guides/prog_guide/kernel_nic_interface.html
> >
> > As a start, this doc should be better referenced:
> >
> > https://doc.dpdk.org/guides/howto/virtio_user_as_exceptional_path.html
> >
> > Note: I won't do this update myself, so feel free to step up!
> > Thanks
> >
> >
> >
>

Re: [dpdk-dev] KNI alternatives

[Thomas Monjalon]

13/01/2021 19:17, Igor Ryzhov:
> On Wed, Jan 13, 2021 at 8:10 PM Stephen Hemminger <
> stephen@networkplumber.org> wrote:
> > On Wed, Jan 13, 2021, 9:06 AM Thomas Monjalon <thomas@monjalon.net> wrote:
> >
> > > Hi,
> > >
> > > As discussed today in the techboard meeting, KNI has probably
> > > better alternatives today without using an out-of-tree module.
> > > Virtio-user is a good candidate to replace KNI.
> > > What is the performance of TAP?
> > > Is there a way to leverage af_packet, af_xdp, or even pcap interfaces?
> > 
> > Last time I tried.
> > Virtio user was as fast as KNI and consumed less cpu. Was seeing 10mpps
> > Tap was much slower. Like 1mpps.
> 
> Performance is not the only question. The advantage of KNI we are currently
> using is
> the ability to control the DPDK interfaces by the kernel.
> For example, to implement bonding in the DPDK application, it is possible
> to create KNI
> pair for each physical interface, create a bond interface in Linux over
> those KNI interfaces
> and just pass LACP packets between the app and the kernel. The kernel
> itself will control
> MACs, MTU, etc. of underlying interfaces. AFAIK it's not possible with
> virtio-user or tap.
> Am I wrong?

I see at least 2 alternatives for bonding with kernel management:
	- mlx5 bonding
	- af_xdp interface for most of NICs

Re: [dpdk-dev] KNI alternatives

[Igor Ryzhov]

On Wed, Jan 13, 2021 at 9:45 PM Thomas Monjalon <thomas@monjalon.net> wrote:

> 13/01/2021 19:17, Igor Ryzhov:
> > On Wed, Jan 13, 2021 at 8:10 PM Stephen Hemminger <
> > stephen@networkplumber.org> wrote:
> > > On Wed, Jan 13, 2021, 9:06 AM Thomas Monjalon <thomas@monjalon.net>
> wrote:
> > >
> > > > Hi,
> > > >
> > > > As discussed today in the techboard meeting, KNI has probably
> > > > better alternatives today without using an out-of-tree module.
> > > > Virtio-user is a good candidate to replace KNI.
> > > > What is the performance of TAP?
> > > > Is there a way to leverage af_packet, af_xdp, or even pcap
> interfaces?
> > >
> > > Last time I tried.
> > > Virtio user was as fast as KNI and consumed less cpu. Was seeing 10mpps
> > > Tap was much slower. Like 1mpps.
> >
> > Performance is not the only question. The advantage of KNI we are
> currently
> > using is
> > the ability to control the DPDK interfaces by the kernel.
> > For example, to implement bonding in the DPDK application, it is possible
> > to create KNI
> > pair for each physical interface, create a bond interface in Linux over
> > those KNI interfaces
> > and just pass LACP packets between the app and the kernel. The kernel
> > itself will control
> > MACs, MTU, etc. of underlying interfaces. AFAIK it's not possible with
> > virtio-user or tap.
> > Am I wrong?
>
> I see at least 2 alternatives for bonding with kernel management:
>         - mlx5 bonding
>         - af_xdp interface for most of NICs
>

Bonding is just an example. My point is that KNI allows changing MAC, MTU,
rx mode and admin status of a NIC used by DPDK application - all using
standard Linux utilities without any need for additional APIs provided by
the app.

Yes, Mellanox NICs also allow this but it's just one of many drivers
supported by DPDK.
I'll dig into af_xdp, thanks.

Re: [dpdk-dev] KNI alternatives

[Stephen Hemminger]

Controlling DPDK interfaces with KNI is known to be broken.  Kni calls
usepspace with network mutex held. The problem is not fixable.

The kernel does provide netlink and devlink API's but there is no generic
solution with DPDK.

On Wed, Jan 13, 2021, 11:54 AM Igor Ryzhov <iryzhov@nfware.com> wrote:

>
>
> On Wed, Jan 13, 2021 at 9:45 PM Thomas Monjalon <thomas@monjalon.net>
> wrote:
>
>> 13/01/2021 19:17, Igor Ryzhov:
>> > On Wed, Jan 13, 2021 at 8:10 PM Stephen Hemminger <
>> > stephen@networkplumber.org> wrote:
>> > > On Wed, Jan 13, 2021, 9:06 AM Thomas Monjalon <thomas@monjalon.net>
>> wrote:
>> > >
>> > > > Hi,
>> > > >
>> > > > As discussed today in the techboard meeting, KNI has probably
>> > > > better alternatives today without using an out-of-tree module.
>> > > > Virtio-user is a good candidate to replace KNI.
>> > > > What is the performance of TAP?
>> > > > Is there a way to leverage af_packet, af_xdp, or even pcap
>> interfaces?
>> > >
>> > > Last time I tried.
>> > > Virtio user was as fast as KNI and consumed less cpu. Was seeing
>> 10mpps
>> > > Tap was much slower. Like 1mpps.
>> >
>> > Performance is not the only question. The advantage of KNI we are
>> currently
>> > using is
>> > the ability to control the DPDK interfaces by the kernel.
>> > For example, to implement bonding in the DPDK application, it is
>> possible
>> > to create KNI
>> > pair for each physical interface, create a bond interface in Linux over
>> > those KNI interfaces
>> > and just pass LACP packets between the app and the kernel. The kernel
>> > itself will control
>> > MACs, MTU, etc. of underlying interfaces. AFAIK it's not possible with
>> > virtio-user or tap.
>> > Am I wrong?
>>
>> I see at least 2 alternatives for bonding with kernel management:
>>         - mlx5 bonding
>>         - af_xdp interface for most of NICs
>>
>
> Bonding is just an example. My point is that KNI allows changing MAC, MTU,
> rx mode and admin status of a NIC used by DPDK application - all using
> standard Linux utilities without any need for additional APIs provided by
> the app.
>
> Yes, Mellanox NICs also allow this but it's just one of many drivers
> supported by DPDK.
> I'll dig into af_xdp, thanks.
>