tap device speed

tap device speed

[Antonio Di Bacco]

I'm doing a test where we have a couple of tap devices, the two
devices are seen by testpmd that is setup in forward mode.

On the linux side, the two tap devices are confined in different
network namespaces and in one namespace we have an iperf server while
on the other namespace the iperf client sending either UDP or TCP.

I expected a bandwidth in the range of few gpbs while the actual
measured bandwidth is a few gigabits.

I suppose I need to configure the tap devices with optimized
parameters but I don't know where to look for advice.

If I try to use the loopback interface I can get something 40 gbps
with a command like this:

iperf -c 127.0.0.1 -u -i 1 -b 40g -t 10 -l 40000

.

Re: tap device speed

[Stephen Hemminger]

On Mon, 2 Oct 2023 21:13:03 +0200
Antonio Di Bacco <a.dibacco.ks@gmail.com> wrote:

> I'm doing a test where we have a couple of tap devices, the two
> devices are seen by testpmd that is setup in forward mode.
> 
> On the linux side, the two tap devices are confined in different
> network namespaces and in one namespace we have an iperf server while
> on the other namespace the iperf client sending either UDP or TCP.
> 
> I expected a bandwidth in the range of few gpbs while the actual
> measured bandwidth is a few gigabits.
> 
> I suppose I need to configure the tap devices with optimized
> parameters but I don't know where to look for advice.
> 
> If I try to use the loopback interface I can get something 40 gbps
> with a command like this:
> 
> iperf -c 127.0.0.1 -u -i 1 -b 40g -t 10 -l 40000
> 
> .

Sorry TAP device is inherently slow. It requires copies to/from Linux
kernel. You are doing well if you get 1 million packets per second.

One thing to check is that checksum is not being done twice.

Re: tap device speed

[Antonio Di Bacco]

I understand, could we use another solution ? Like a memif  interface
in DPDK and libmemif in Linux?

On Mon, Oct 2, 2023 at 11:21 PM Stephen Hemminger
<stephen@networkplumber.org> wrote:
>
> On Mon, 2 Oct 2023 21:13:03 +0200
> Antonio Di Bacco <a.dibacco.ks@gmail.com> wrote:
>
> > I'm doing a test where we have a couple of tap devices, the two
> > devices are seen by testpmd that is setup in forward mode.
> >
> > On the linux side, the two tap devices are confined in different
> > network namespaces and in one namespace we have an iperf server while
> > on the other namespace the iperf client sending either UDP or TCP.
> >
> > I expected a bandwidth in the range of few gpbs while the actual
> > measured bandwidth is a few gigabits.
> >
> > I suppose I need to configure the tap devices with optimized
> > parameters but I don't know where to look for advice.
> >
> > If I try to use the loopback interface I can get something 40 gbps
> > with a command like this:
> >
> > iperf -c 127.0.0.1 -u -i 1 -b 40g -t 10 -l 40000
> >
> > .
>
> Sorry TAP device is inherently slow. It requires copies to/from Linux
> kernel. You are doing well if you get 1 million packets per second.
>
> One thing to check is that checksum is not being done twice.

Re: tap device speed

[Stephen Hemminger]

On Tue, 3 Oct 2023 10:49:16 +0200
Antonio Di Bacco <a.dibacco.ks@gmail.com> wrote:

> I understand, could we use another solution ? Like a memif  interface
> in DPDK and libmemif in Linux?

The issue is accessing kernel networking devices. Both virtio user
and XDP are faster for that. Memif is for doing process to process networking.

Re: tap device speed

[David Marchand]

On Tue, Oct 3, 2023 at 6:01 PM Stephen Hemminger
<stephen@networkplumber.org> wrote:
>
> On Tue, 3 Oct 2023 10:49:16 +0200
> Antonio Di Bacco <a.dibacco.ks@gmail.com> wrote:
>
> > I understand, could we use another solution ? Like a memif  interface
> > in DPDK and libmemif in Linux?
>
> The issue is accessing kernel networking devices. Both virtio user
> and XDP are faster for that. Memif is for doing process to process networking.

For dpdk <-> kernel, as you are mentioning virtio-user/vhost, let me
add that there is some activity on this side, with VDUSE.

Maxime is working on the VDUSE kernel and dpdk bits.
He gave a talk about the current status during the summit and some
performance numbers:
https://dpdksummit2023.sched.com/event/1P9xA/vduse-performance-how-fast-is-it-maxime-coquelin-red-hat


-- 
David Marchand

Re: tap device speed

[Maxime Coquelin]

On 10/4/23 08:17, David Marchand wrote:
> On Tue, Oct 3, 2023 at 6:01 PM Stephen Hemminger
> <stephen@networkplumber.org> wrote:
>>
>> On Tue, 3 Oct 2023 10:49:16 +0200
>> Antonio Di Bacco <a.dibacco.ks@gmail.com> wrote:
>>
>>> I understand, could we use another solution ? Like a memif  interface
>>> in DPDK and libmemif in Linux?
>>
>> The issue is accessing kernel networking devices. Both virtio user
>> and XDP are faster for that. Memif is for doing process to process networking.
> 
> For dpdk <-> kernel, as you are mentioning virtio-user/vhost, let me
> add that there is some activity on this side, with VDUSE.
> 
> Maxime is working on the VDUSE kernel and dpdk bits.
> He gave a talk about the current status during the summit and some
> performance numbers:
> https://dpdksummit2023.sched.com/event/1P9xA/vduse-performance-how-fast-is-it-maxime-coquelin-red-hat
> 
> 

Thanks for sharing David.
I'd like just to add some more information on VDUSE if you want to 
experiment with VDUSE, which is still under development:
https://gitlab.com/mcoquelin/vduse-doc

Maxime

Re: tap device speed

[Antonio Di Bacco]

Thank you for your info that are giving me the right heads up
To experiment with VDUSE and share a virtual network interface (I
don't have a physical NIC) between the Linux kernel and DPDK using
VDUSE, I'm about to follow these steps:

Load Required Kernel Modules:
modprobe vduse
Create /dev/vdpa0 device with:
vdpa -d /dev/vdpa0 -n my_vdpa_driver -q queue_count

I wonder which vdpa_driver should I use, I don't have a real NIC
After having this vdpa0 interface up I can run my DPDK application:

./my_dpdk_app --vdev "net_vdpa0,iface=/dev/vdpa0"

Regards,
Antonio.

On Wed, Oct 4, 2023 at 9:08 AM Maxime Coquelin
<maxime.coquelin@redhat.com> wrote:
>
>
>
> On 10/4/23 08:17, David Marchand wrote:
> > On Tue, Oct 3, 2023 at 6:01 PM Stephen Hemminger
> > <stephen@networkplumber.org> wrote:
> >>
> >> On Tue, 3 Oct 2023 10:49:16 +0200
> >> Antonio Di Bacco <a.dibacco.ks@gmail.com> wrote:
> >>
> >>> I understand, could we use another solution ? Like a memif  interface
> >>> in DPDK and libmemif in Linux?
> >>
> >> The issue is accessing kernel networking devices. Both virtio user
> >> and XDP are faster for that. Memif is for doing process to process networking.
> >
> > For dpdk <-> kernel, as you are mentioning virtio-user/vhost, let me
> > add that there is some activity on this side, with VDUSE.
> >
> > Maxime is working on the VDUSE kernel and dpdk bits.
> > He gave a talk about the current status during the summit and some
> > performance numbers:
> > https://dpdksummit2023.sched.com/event/1P9xA/vduse-performance-how-fast-is-it-maxime-coquelin-red-hat
> >
> >
>
> Thanks for sharing David.
> I'd like just to add some more information on VDUSE if you want to
> experiment with VDUSE, which is still under development:
> https://gitlab.com/mcoquelin/vduse-doc
>
> Maxime
>

Re: tap device speed

[Maxime Coquelin]

On 10/4/23 09:42, Antonio Di Bacco wrote:
> Thank you for your info that are giving me the right heads up
> To experiment with VDUSE and share a virtual network interface (I
> don't have a physical NIC) between the Linux kernel and DPDK using
> VDUSE, I'm about to follow these steps:
> 
> Load Required Kernel Modules:
> modprobe vduse
> Create /dev/vdpa0 device with:
> vdpa -d /dev/vdpa0 -n my_vdpa_driver -q queue_count
> 
> I wonder which vdpa_driver should I use, I don't have a real NIC
> After having this vdpa0 interface up I can run my DPDK application:
> 
> ./my_dpdk_app --vdev "net_vdpa0,iface=/dev/vdpa0"

You will use the Vhost PMD in this case (alternative is to use the Vhost
API directly).

Maybe I missed to add steps with Vhost PMD in my doc repo, I'll improve
it but in the mean time you can refer to the steps provided in the DPDK
VDUSE series cover letter:
https://inbox.dpdk.org/dev/7f72500a-5317-c66d-3f36-2fd65c874b47@redhat.com/T/

Please read the vduse documentation on my gitlab repo anyways, it
provides pointers on the missing Kernel patches (being upstreamed).

Maxime


> Regards,
> Antonio.
> 
> On Wed, Oct 4, 2023 at 9:08 AM Maxime Coquelin
> <maxime.coquelin@redhat.com> wrote:
>>
>>
>>
>> On 10/4/23 08:17, David Marchand wrote:
>>> On Tue, Oct 3, 2023 at 6:01 PM Stephen Hemminger
>>> <stephen@networkplumber.org> wrote:
>>>>
>>>> On Tue, 3 Oct 2023 10:49:16 +0200
>>>> Antonio Di Bacco <a.dibacco.ks@gmail.com> wrote:
>>>>
>>>>> I understand, could we use another solution ? Like a memif  interface
>>>>> in DPDK and libmemif in Linux?
>>>>
>>>> The issue is accessing kernel networking devices. Both virtio user
>>>> and XDP are faster for that. Memif is for doing process to process networking.
>>>
>>> For dpdk <-> kernel, as you are mentioning virtio-user/vhost, let me
>>> add that there is some activity on this side, with VDUSE.
>>>
>>> Maxime is working on the VDUSE kernel and dpdk bits.
>>> He gave a talk about the current status during the summit and some
>>> performance numbers:
>>> https://dpdksummit2023.sched.com/event/1P9xA/vduse-performance-how-fast-is-it-maxime-coquelin-red-hat
>>>
>>>
>>
>> Thanks for sharing David.
>> I'd like just to add some more information on VDUSE if you want to
>> experiment with VDUSE, which is still under development:
>> https://gitlab.com/mcoquelin/vduse-doc
>>
>> Maxime
>>
> 

Re: tap device speed

[Antonio Di Bacco]

Hi Maxime,

trying to follow the steps at the link provided:


sage:
======

1. Probe required Kernel modules
# modprobe vdpa
# modprobe vduse
# modprobe virtio-vdpa

2. Build (require vduse kernel headers to be available)
# meson build
# ninja -C build

3. Create a VDUSE device (vduse0) using Vhost PMD with
testpmd (with 4 queue pairs in this example)
# ./build/app/dpdk-testpmd --no-pci
--vdev=net_vhost0,iface=/dev/vduse/vduse0,queues=4 --log-level=*:9  --
-i --txq=4 --rxq=4

4. Attach the VDUSE device to the vDPA bus
# vdpa dev add name vduse0 mgmtdev vduse

I'm stuck at step 4, with an error:

manderwo@fetgem01:~$ sudo vdpa dev add name vduse0 mgmtdev vduse
kernel answers: Invalid argument
manderwo@fetgem01:~$ vdpa mgmtdev show
vduse:
  supported_classes block
manderwo@fetgem01:~$

Do you know what could be wrong?

Thank you in advance,
Antonio.

On Wed, Oct 4, 2023 at 9:50 AM Maxime Coquelin
<maxime.coquelin@redhat.com> wrote:
>
>
>
> On 10/4/23 09:42, Antonio Di Bacco wrote:
> > Thank you for your info that are giving me the right heads up
> > To experiment with VDUSE and share a virtual network interface (I
> > don't have a physical NIC) between the Linux kernel and DPDK using
> > VDUSE, I'm about to follow these steps:
> >
> > Load Required Kernel Modules:
> > modprobe vduse
> > Create /dev/vdpa0 device with:
> > vdpa -d /dev/vdpa0 -n my_vdpa_driver -q queue_count
> >
> > I wonder which vdpa_driver should I use, I don't have a real NIC
> > After having this vdpa0 interface up I can run my DPDK application:
> >
> > ./my_dpdk_app --vdev "net_vdpa0,iface=/dev/vdpa0"
>
> You will use the Vhost PMD in this case (alternative is to use the Vhost
> API directly).
>
> Maybe I missed to add steps with Vhost PMD in my doc repo, I'll improve
> it but in the mean time you can refer to the steps provided in the DPDK
> VDUSE series cover letter:
> https://inbox.dpdk.org/dev/7f72500a-5317-c66d-3f36-2fd65c874b47@redhat.com/T/
>
> Please read the vduse documentation on my gitlab repo anyways, it
> provides pointers on the missing Kernel patches (being upstreamed).
>
> Maxime
>
>
> > Regards,
> > Antonio.
> >
> > On Wed, Oct 4, 2023 at 9:08 AM Maxime Coquelin
> > <maxime.coquelin@redhat.com> wrote:
> >>
> >>
> >>
> >> On 10/4/23 08:17, David Marchand wrote:
> >>> On Tue, Oct 3, 2023 at 6:01 PM Stephen Hemminger
> >>> <stephen@networkplumber.org> wrote:
> >>>>
> >>>> On Tue, 3 Oct 2023 10:49:16 +0200
> >>>> Antonio Di Bacco <a.dibacco.ks@gmail.com> wrote:
> >>>>
> >>>>> I understand, could we use another solution ? Like a memif  interface
> >>>>> in DPDK and libmemif in Linux?
> >>>>
> >>>> The issue is accessing kernel networking devices. Both virtio user
> >>>> and XDP are faster for that. Memif is for doing process to process networking.
> >>>
> >>> For dpdk <-> kernel, as you are mentioning virtio-user/vhost, let me
> >>> add that there is some activity on this side, with VDUSE.
> >>>
> >>> Maxime is working on the VDUSE kernel and dpdk bits.
> >>> He gave a talk about the current status during the summit and some
> >>> performance numbers:
> >>> https://dpdksummit2023.sched.com/event/1P9xA/vduse-performance-how-fast-is-it-maxime-coquelin-red-hat
> >>>
> >>>
> >>
> >> Thanks for sharing David.
> >> I'd like just to add some more information on VDUSE if you want to
> >> experiment with VDUSE, which is still under development:
> >> https://gitlab.com/mcoquelin/vduse-doc
> >>
> >> Maxime
> >>
> >
>

Re: tap device speed

[Maxime Coquelin]

Hi Antonio,

On 10/9/23 19:48, Antonio Di Bacco wrote:
> Hi Maxime,
> 
> trying to follow the steps at the link provided:
> 
> 
> sage:
> ======
> 
> 1. Probe required Kernel modules
> # modprobe vdpa
> # modprobe vduse
> # modprobe virtio-vdpa
> 
> 2. Build (require vduse kernel headers to be available)
> # meson build
> # ninja -C build
> 
> 3. Create a VDUSE device (vduse0) using Vhost PMD with
> testpmd (with 4 queue pairs in this example)
> # ./build/app/dpdk-testpmd --no-pci
> --vdev=net_vhost0,iface=/dev/vduse/vduse0,queues=4 --log-level=*:9  --
> -i --txq=4 --rxq=4
> 
> 4. Attach the VDUSE device to the vDPA bus
> # vdpa dev add name vduse0 mgmtdev vduse
> 
> I'm stuck at step 4, with an error:
> 
> manderwo@fetgem01:~$ sudo vdpa dev add name vduse0 mgmtdev vduse
> kernel answers: Invalid argument
> manderwo@fetgem01:~$ vdpa mgmtdev show
> vduse:
>    supported_classes block
> manderwo@fetgem01:~$
> 
> Do you know what could be wrong?

I guess you don't have the kernel patches applied, a recent branch  can 
be found on my gitlab aacount.
You can find instructions here:

https://gitlab.com/mcoquelin/vduse-doc/-/blob/master/SETUP.md?ref_type=heads#kernel

Maxime

> 
> Thank you in advance,
> Antonio.
> 
> On Wed, Oct 4, 2023 at 9:50 AM Maxime Coquelin
> <maxime.coquelin@redhat.com> wrote:
>>
>>
>>
>> On 10/4/23 09:42, Antonio Di Bacco wrote:
>>> Thank you for your info that are giving me the right heads up
>>> To experiment with VDUSE and share a virtual network interface (I
>>> don't have a physical NIC) between the Linux kernel and DPDK using
>>> VDUSE, I'm about to follow these steps:
>>>
>>> Load Required Kernel Modules:
>>> modprobe vduse
>>> Create /dev/vdpa0 device with:
>>> vdpa -d /dev/vdpa0 -n my_vdpa_driver -q queue_count
>>>
>>> I wonder which vdpa_driver should I use, I don't have a real NIC
>>> After having this vdpa0 interface up I can run my DPDK application:
>>>
>>> ./my_dpdk_app --vdev "net_vdpa0,iface=/dev/vdpa0"
>>
>> You will use the Vhost PMD in this case (alternative is to use the Vhost
>> API directly).
>>
>> Maybe I missed to add steps with Vhost PMD in my doc repo, I'll improve
>> it but in the mean time you can refer to the steps provided in the DPDK
>> VDUSE series cover letter:
>> https://inbox.dpdk.org/dev/7f72500a-5317-c66d-3f36-2fd65c874b47@redhat.com/T/
>>
>> Please read the vduse documentation on my gitlab repo anyways, it
>> provides pointers on the missing Kernel patches (being upstreamed).
>>
>> Maxime
>>
>>
>>> Regards,
>>> Antonio.
>>>
>>> On Wed, Oct 4, 2023 at 9:08 AM Maxime Coquelin
>>> <maxime.coquelin@redhat.com> wrote:
>>>>
>>>>
>>>>
>>>> On 10/4/23 08:17, David Marchand wrote:
>>>>> On Tue, Oct 3, 2023 at 6:01 PM Stephen Hemminger
>>>>> <stephen@networkplumber.org> wrote:
>>>>>>
>>>>>> On Tue, 3 Oct 2023 10:49:16 +0200
>>>>>> Antonio Di Bacco <a.dibacco.ks@gmail.com> wrote:
>>>>>>
>>>>>>> I understand, could we use another solution ? Like a memif  interface
>>>>>>> in DPDK and libmemif in Linux?
>>>>>>
>>>>>> The issue is accessing kernel networking devices. Both virtio user
>>>>>> and XDP are faster for that. Memif is for doing process to process networking.
>>>>>
>>>>> For dpdk <-> kernel, as you are mentioning virtio-user/vhost, let me
>>>>> add that there is some activity on this side, with VDUSE.
>>>>>
>>>>> Maxime is working on the VDUSE kernel and dpdk bits.
>>>>> He gave a talk about the current status during the summit and some
>>>>> performance numbers:
>>>>> https://dpdksummit2023.sched.com/event/1P9xA/vduse-performance-how-fast-is-it-maxime-coquelin-red-hat
>>>>>
>>>>>
>>>>
>>>> Thanks for sharing David.
>>>> I'd like just to add some more information on VDUSE if you want to
>>>> experiment with VDUSE, which is still under development:
>>>> https://gitlab.com/mcoquelin/vduse-doc
>>>>
>>>> Maxime
>>>>
>>>
>>
>