[dpdk-users] rte_eth_tx_burst: Can I insert timing gaps

[dpdk-users] rte_eth_tx_burst: Can I insert timing gaps

[Philipp B]

Hi all!

I am working on an RTP test traffic generator. The basic idea is
clock_nanosleep providing a 20ms clock cycle to start a (big) number
of rte_eth_tx_bursts, sending equally sized buffers. As long as the
timing within a 20ms cycle is dictated primarily by the line speed, I
can be sure that not just the first buffer of each cycle has a period
of 20ms, but also the n-th buffer. (I have sent n-1 buffers before
with the same size.)

Basically, I see one 20ms interval as a series of time slots, each
capable to store an active RTP stream. My question now is, what to to
with inactive time slots? As long as all active streams are located at
consecutive time slots from the start of the 20ms interval, everything
is fine. But I cannot guarantee this.

What I need is some kind of dummy buffer, which is not transmitted but
generates a tx timing gap as a buffer of X bytes would take to be
transferred.

Is such a functionality provided? As a workaround, I already thought
about sending invalid packets (bad IP Header checksum?). However, this
won't be optimal when multiple lines are aggregated.

Thanks!
Philipp Beyer

Re: [dpdk-users] rte_eth_tx_burst: Can I insert timing gaps

[Cliff Burdick]

Maybe I'm misunderstanding the problem, but do you need to transmit
anything? Can you just use the rte_cycles functions to sleep for the
remaining period in 20ms?

On Thu, Oct 11, 2018 at 2:04 AM Philipp B <philippb.ontour@gmail.com> wrote:

> Hi all!
>
> I am working on an RTP test traffic generator. The basic idea is
> clock_nanosleep providing a 20ms clock cycle to start a (big) number
> of rte_eth_tx_bursts, sending equally sized buffers. As long as the
> timing within a 20ms cycle is dictated primarily by the line speed, I
> can be sure that not just the first buffer of each cycle has a period
> of 20ms, but also the n-th buffer. (I have sent n-1 buffers before
> with the same size.)
>
> Basically, I see one 20ms interval as a series of time slots, each
> capable to store an active RTP stream. My question now is, what to to
> with inactive time slots? As long as all active streams are located at
> consecutive time slots from the start of the 20ms interval, everything
> is fine. But I cannot guarantee this.
>
> What I need is some kind of dummy buffer, which is not transmitted but
> generates a tx timing gap as a buffer of X bytes would take to be
> transferred.
>
> Is such a functionality provided? As a workaround, I already thought
> about sending invalid packets (bad IP Header checksum?). However, this
> won't be optimal when multiple lines are aggregated.
>
> Thanks!
> Philipp Beyer
>

Re: [dpdk-users] rte_eth_tx_burst: Can I insert timing gaps

[Philipp B]

Maybe I should explain with some ASCII art. It's not about sleeping
just the remaining period of 20ms. This is exactly what I am doing
already, and this works fine.

Think of 20ms intervals like this

|XXXXXXXXXXXXXXXXXX_____|XXXXXXXXXXXXXXXXXX_____

Where '|' is the start of the 20ms interval, X is a packet, and _ is
an idle gap of one packet. (Let's just pretend there are 1000s of Xs
per interval).

As I said, I let the CPU sleep upto the beginning of the interval
('|'). This beginning of interval is the only moment where CPU timing
controls adapter timing. Then I send out a few 1000 packets. In this
phase, I have a few 100 packets buffered by DPDK, so it will not help
to sleep on the CPU.

The pattern above is what I can easily produce just with an OS sleep,
a single buffer pool and rte_eth_tx_burst. What I am looking for, is a
way to e.g. remove every second packet from that pattern, while
keeping the other packet's timings unchanged:

|X_X_X_X_X_X_X_X_X______|X_X_X_X_X_X_X_X_X______

Basically, I do not need to transmit anything in the gaps. I just need
the delay. However, as my timing of CPU isn't coupled tightly to the
adapter, sleeping on the cpu will not help. This is intended by
design: I want to blow out a massive number of packets with exact
timing and virtually no CPU requirement.

What I look for is a sleep instruction executed by the adapter, which
is buffered in order with the packets issued by rte_eth_tx_burst.
(Plus some basic math rules how to convert packet sizes to durations,
based on line speeds).

Am Sa., 13. Okt. 2018 um 23:05 Uhr schrieb Cliff Burdick <shaklee3@gmail.com>:
>
> Maybe I'm misunderstanding the problem, but do you need to transmit anything? Can you just use the rte_cycles functions to sleep for the remaining period in 20ms?
>
> On Thu, Oct 11, 2018 at 2:04 AM Philipp B <philippb.ontour@gmail.com> wrote:
>>
>> Hi all!
>>
>> I am working on an RTP test traffic generator. The basic idea is
>> clock_nanosleep providing a 20ms clock cycle to start a (big) number
>> of rte_eth_tx_bursts, sending equally sized buffers. As long as the
>> timing within a 20ms cycle is dictated primarily by the line speed, I
>> can be sure that not just the first buffer of each cycle has a period
>> of 20ms, but also the n-th buffer. (I have sent n-1 buffers before
>> with the same size.)
>>
>> Basically, I see one 20ms interval as a series of time slots, each
>> capable to store an active RTP stream. My question now is, what to to
>> with inactive time slots? As long as all active streams are located at
>> consecutive time slots from the start of the 20ms interval, everything
>> is fine. But I cannot guarantee this.
>>
>> What I need is some kind of dummy buffer, which is not transmitted but
>> generates a tx timing gap as a buffer of X bytes would take to be
>> transferred.
>>
>> Is such a functionality provided? As a workaround, I already thought
>> about sending invalid packets (bad IP Header checksum?). However, this
>> won't be optimal when multiple lines are aggregated.
>>
>> Thanks!
>> Philipp Beyer

Re: [dpdk-users] rte_eth_tx_burst: Can I insert timing gaps

[Andrew Bainbridge]

Is the feature you are describing is called packet "pacing"? Here's a Mellanox document describing it: 
  https://community.mellanox.com/docs/DOC-2551 

I grep'ed the DPDK source for "rate_limit" and found rte_eth_set_queue_rate_limit(). Is that the function you need?

From my quick grep'ing, it looks to me like this feature isn't supported on Mellanox drivers but is in the ixgbe driver. However, this is all guess work. I'm not an expert. I'd like to know the real answer!

- Andrew

-----Original Message-----
From: users <users-bounces@dpdk.org> On Behalf Of Philipp B
Sent: 15 October 2018 11:45
To: shaklee3@gmail.com
Cc: users@dpdk.org
Subject: Re: [dpdk-users] rte_eth_tx_burst: Can I insert timing gaps

Maybe I should explain with some ASCII art. It's not about sleeping just the remaining period of 20ms. This is exactly what I am doing already, and this works fine.

Think of 20ms intervals like this

|XXXXXXXXXXXXXXXXXX_____|XXXXXXXXXXXXXXXXXX_____

Where '|' is the start of the 20ms interval, X is a packet, and _ is an idle gap of one packet. (Let's just pretend there are 1000s of Xs per interval).

As I said, I let the CPU sleep upto the beginning of the interval ('|'). This beginning of interval is the only moment where CPU timing controls adapter timing. Then I send out a few 1000 packets. In this phase, I have a few 100 packets buffered by DPDK, so it will not help to sleep on the CPU.

The pattern above is what I can easily produce just with an OS sleep, a single buffer pool and rte_eth_tx_burst. What I am looking for, is a way to e.g. remove every second packet from that pattern, while keeping the other packet's timings unchanged:

|X_X_X_X_X_X_X_X_X______|X_X_X_X_X_X_X_X_X______

Basically, I do not need to transmit anything in the gaps. I just need the delay. However, as my timing of CPU isn't coupled tightly to the adapter, sleeping on the cpu will not help. This is intended by
design: I want to blow out a massive number of packets with exact timing and virtually no CPU requirement.

What I look for is a sleep instruction executed by the adapter, which is buffered in order with the packets issued by rte_eth_tx_burst.
(Plus some basic math rules how to convert packet sizes to durations, based on line speeds).

Am Sa., 13. Okt. 2018 um 23:05 Uhr schrieb Cliff Burdick <shaklee3@gmail.com>:
>
> Maybe I'm misunderstanding the problem, but do you need to transmit anything? Can you just use the rte_cycles functions to sleep for the remaining period in 20ms?
>
> On Thu, Oct 11, 2018 at 2:04 AM Philipp B <philippb.ontour@gmail.com> wrote:
>>
>> Hi all!
>>
>> I am working on an RTP test traffic generator. The basic idea is 
>> clock_nanosleep providing a 20ms clock cycle to start a (big) number 
>> of rte_eth_tx_bursts, sending equally sized buffers. As long as the 
>> timing within a 20ms cycle is dictated primarily by the line speed, I 
>> can be sure that not just the first buffer of each cycle has a period 
>> of 20ms, but also the n-th buffer. (I have sent n-1 buffers before 
>> with the same size.)
>>
>> Basically, I see one 20ms interval as a series of time slots, each 
>> capable to store an active RTP stream. My question now is, what to to 
>> with inactive time slots? As long as all active streams are located 
>> at consecutive time slots from the start of the 20ms interval, 
>> everything is fine. But I cannot guarantee this.
>>
>> What I need is some kind of dummy buffer, which is not transmitted 
>> but generates a tx timing gap as a buffer of X bytes would take to be 
>> transferred.
>>
>> Is such a functionality provided? As a workaround, I already thought 
>> about sending invalid packets (bad IP Header checksum?). However, 
>> this won't be optimal when multiple lines are aggregated.
>>
>> Thanks!
>> Philipp Beyer

Re: [dpdk-users] rte_eth_tx_burst: Can I insert timing gaps

[Cliff Burdick]

I believe mellanox does support it, but it is only part of their other
library (rivermax) and not their DPDK driver. However, I don't know if that
means you would be able to do something similar to puncturing packets like
you are trying to do. My best guess would be that you can send the filler
packets to your own MAC address so that the switch will drop it.



On Mon, Oct 15, 2018, 04:21 Andrew Bainbridge <andbain@microsoft.com> wrote:

> Is the feature you are describing is called packet "pacing"? Here's a
> Mellanox document describing it:
>   https://community.mellanox.com/docs/DOC-2551
>
> I grep'ed the DPDK source for "rate_limit" and found
> rte_eth_set_queue_rate_limit(). Is that the function you need?
>
> From my quick grep'ing, it looks to me like this feature isn't supported
> on Mellanox drivers but is in the ixgbe driver. However, this is all guess
> work. I'm not an expert. I'd like to know the real answer!
>
> - Andrew
>
> -----Original Message-----
> From: users <users-bounces@dpdk.org> On Behalf Of Philipp B
> Sent: 15 October 2018 11:45
> To: shaklee3@gmail.com
> Cc: users@dpdk.org
> Subject: Re: [dpdk-users] rte_eth_tx_burst: Can I insert timing gaps
>
> Maybe I should explain with some ASCII art. It's not about sleeping just
> the remaining period of 20ms. This is exactly what I am doing already, and
> this works fine.
>
> Think of 20ms intervals like this
>
> |XXXXXXXXXXXXXXXXXX_____|XXXXXXXXXXXXXXXXXX_____
>
> Where '|' is the start of the 20ms interval, X is a packet, and _ is an
> idle gap of one packet. (Let's just pretend there are 1000s of Xs per
> interval).
>
> As I said, I let the CPU sleep upto the beginning of the interval ('|').
> This beginning of interval is the only moment where CPU timing controls
> adapter timing. Then I send out a few 1000 packets. In this phase, I have a
> few 100 packets buffered by DPDK, so it will not help to sleep on the CPU.
>
> The pattern above is what I can easily produce just with an OS sleep, a
> single buffer pool and rte_eth_tx_burst. What I am looking for, is a way to
> e.g. remove every second packet from that pattern, while keeping the other
> packet's timings unchanged:
>
> |X_X_X_X_X_X_X_X_X______|X_X_X_X_X_X_X_X_X______
>
> Basically, I do not need to transmit anything in the gaps. I just need the
> delay. However, as my timing of CPU isn't coupled tightly to the adapter,
> sleeping on the cpu will not help. This is intended by
> design: I want to blow out a massive number of packets with exact timing
> and virtually no CPU requirement.
>
> What I look for is a sleep instruction executed by the adapter, which is
> buffered in order with the packets issued by rte_eth_tx_burst.
> (Plus some basic math rules how to convert packet sizes to durations,
> based on line speeds).
>
> Am Sa., 13. Okt. 2018 um 23:05 Uhr schrieb Cliff Burdick <
> shaklee3@gmail.com>:
> >
> > Maybe I'm misunderstanding the problem, but do you need to transmit
> anything? Can you just use the rte_cycles functions to sleep for the
> remaining period in 20ms?
> >
> > On Thu, Oct 11, 2018 at 2:04 AM Philipp B <philippb.ontour@gmail.com>
> wrote:
> >>
> >> Hi all!
> >>
> >> I am working on an RTP test traffic generator. The basic idea is
> >> clock_nanosleep providing a 20ms clock cycle to start a (big) number
> >> of rte_eth_tx_bursts, sending equally sized buffers. As long as the
> >> timing within a 20ms cycle is dictated primarily by the line speed, I
> >> can be sure that not just the first buffer of each cycle has a period
> >> of 20ms, but also the n-th buffer. (I have sent n-1 buffers before
> >> with the same size.)
> >>
> >> Basically, I see one 20ms interval as a series of time slots, each
> >> capable to store an active RTP stream. My question now is, what to to
> >> with inactive time slots? As long as all active streams are located
> >> at consecutive time slots from the start of the 20ms interval,
> >> everything is fine. But I cannot guarantee this.
> >>
> >> What I need is some kind of dummy buffer, which is not transmitted
> >> but generates a tx timing gap as a buffer of X bytes would take to be
> >> transferred.
> >>
> >> Is such a functionality provided? As a workaround, I already thought
> >> about sending invalid packets (bad IP Header checksum?). However,
> >> this won't be optimal when multiple lines are aggregated.
> >>
> >> Thanks!
> >> Philipp Beyer
>

Re: [dpdk-users] rte_eth_tx_burst: Can I insert timing gaps

[Andrew Bainbridge]

> I believe mellanox does support it,

I think some cards do and others don’t. See the table at the end of this:
http://www.mellanox.com/related-docs/products/Ethernet_Adapter_Brochure.pdf
However they might be talking about pacing per-TCP-flow.

> My best guess would be that you can send the filler packets to your own MAC address so that the switch will drop it.

Yeah, nice.


From: Cliff Burdick <shaklee3@gmail.com>
Sent: 15 October 2018 15:30
To: Andrew Bainbridge <andbain@microsoft.com>
Cc: philippb.ontour@gmail.com; users <users@dpdk.org>
Subject: Re: [dpdk-users] rte_eth_tx_burst: Can I insert timing gaps

I believe mellanox does support it, but it is only part of their other library (rivermax) and not their DPDK driver. However, I don't know if that means you would be able to do something similar to puncturing packets like you are trying to do. My best guess would be that you can send the filler packets to your own MAC address so that the switch will drop it.



On Mon, Oct 15, 2018, 04:21 Andrew Bainbridge <andbain@microsoft.com<mailto:andbain@microsoft.com>> wrote:
Is the feature you are describing is called packet "pacing"? Here's a Mellanox document describing it:
  https://community.mellanox.com/docs/DOC-2551<https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcommunity.mellanox.com%2Fdocs%2FDOC-2551&data=02%7C01%7Candbain%40microsoft.com%7C70b0ea95e31a45c292e708d632aaaed7%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636752105994820376&sdata=Kd7MN7Hw3Ii7w3%2FUNeWMFzVQscmdbVMvvojGgfpNP9E%3D&reserved=0>

I grep'ed the DPDK source for "rate_limit" and found rte_eth_set_queue_rate_limit(). Is that the function you need?

From my quick grep'ing, it looks to me like this feature isn't supported on Mellanox drivers but is in the ixgbe driver. However, this is all guess work. I'm not an expert. I'd like to know the real answer!

- Andrew

-----Original Message-----
From: users <users-bounces@dpdk.org<mailto:users-bounces@dpdk.org>> On Behalf Of Philipp B
Sent: 15 October 2018 11:45
To: shaklee3@gmail.com<mailto:shaklee3@gmail.com>
Cc: users@dpdk.org<mailto:users@dpdk.org>
Subject: Re: [dpdk-users] rte_eth_tx_burst: Can I insert timing gaps

Maybe I should explain with some ASCII art. It's not about sleeping just the remaining period of 20ms. This is exactly what I am doing already, and this works fine.

Think of 20ms intervals like this

|XXXXXXXXXXXXXXXXXX_____|XXXXXXXXXXXXXXXXXX_____

Where '|' is the start of the 20ms interval, X is a packet, and _ is an idle gap of one packet. (Let's just pretend there are 1000s of Xs per interval).

As I said, I let the CPU sleep upto the beginning of the interval ('|'). This beginning of interval is the only moment where CPU timing controls adapter timing. Then I send out a few 1000 packets. In this phase, I have a few 100 packets buffered by DPDK, so it will not help to sleep on the CPU.

The pattern above is what I can easily produce just with an OS sleep, a single buffer pool and rte_eth_tx_burst. What I am looking for, is a way to e.g. remove every second packet from that pattern, while keeping the other packet's timings unchanged:

|X_X_X_X_X_X_X_X_X______|X_X_X_X_X_X_X_X_X______

Basically, I do not need to transmit anything in the gaps. I just need the delay. However, as my timing of CPU isn't coupled tightly to the adapter, sleeping on the cpu will not help. This is intended by
design: I want to blow out a massive number of packets with exact timing and virtually no CPU requirement.

What I look for is a sleep instruction executed by the adapter, which is buffered in order with the packets issued by rte_eth_tx_burst.
(Plus some basic math rules how to convert packet sizes to durations, based on line speeds).

Am Sa., 13. Okt. 2018 um 23:05 Uhr schrieb Cliff Burdick <shaklee3@gmail.com<mailto:shaklee3@gmail.com>>:
>
> Maybe I'm misunderstanding the problem, but do you need to transmit anything? Can you just use the rte_cycles functions to sleep for the remaining period in 20ms?
>
> On Thu, Oct 11, 2018 at 2:04 AM Philipp B <philippb.ontour@gmail.com<mailto:philippb.ontour@gmail.com>> wrote:
>>
>> Hi all!
>>
>> I am working on an RTP test traffic generator. The basic idea is
>> clock_nanosleep providing a 20ms clock cycle to start a (big) number
>> of rte_eth_tx_bursts, sending equally sized buffers. As long as the
>> timing within a 20ms cycle is dictated primarily by the line speed, I
>> can be sure that not just the first buffer of each cycle has a period
>> of 20ms, but also the n-th buffer. (I have sent n-1 buffers before
>> with the same size.)
>>
>> Basically, I see one 20ms interval as a series of time slots, each
>> capable to store an active RTP stream. My question now is, what to to
>> with inactive time slots? As long as all active streams are located
>> at consecutive time slots from the start of the 20ms interval,
>> everything is fine. But I cannot guarantee this.
>>
>> What I need is some kind of dummy buffer, which is not transmitted
>> but generates a tx timing gap as a buffer of X bytes would take to be
>> transferred.
>>
>> Is such a functionality provided? As a workaround, I already thought
>> about sending invalid packets (bad IP Header checksum?). However,
>> this won't be optimal when multiple lines are aggregated.
>>
>> Thanks!
>> Philipp Beyer

Re: [dpdk-users] rte_eth_tx_burst: Can I insert timing gaps

[Philipp B]

Thanks a lot guys.

This was really helpful and comprehensive!

Philipp
Am Mo., 15. Okt. 2018 um 21:08 Uhr schrieb Paul Emmerich
<emmericp@net.in.tum.de>:
>
>
> > Am 15.10.2018 um 16:29 schrieb Cliff Burdick <shaklee3@gmail.com>:
> >
> > My best guess would be that you can send the filler
> > packets to your own MAC address so that the switch will drop it.
>
> A variant of that (with broken CRC checksums; require patched drivers)
> of that yielded the best results in our evaluation linked above.
>
> Our implementation for this is here:
> https://github.com/emmericp/MoonGen/blob/master/src/crc-rate-limiter.c
>
> Implementation of a busy-wait based rate limiter that is often good enough is here:
> https://github.com/emmericp/MoonGen/blob/master/src/software-rate-limiter.cpp#L48
>
> It runs in a separate thread that is fed through an rte_ring, so there is only one tight loop
> on that core that sleeps and sends out packets.
>
>
> Paul
>
> >
> >
> >
> > On Mon, Oct 15, 2018, 04:21 Andrew Bainbridge <andbain@microsoft.com> wrote:
> >
> >> Is the feature you are describing is called packet "pacing"? Here's a
> >> Mellanox document describing it:
> >>  https://community.mellanox.com/docs/DOC-2551
> >>
> >> I grep'ed the DPDK source for "rate_limit" and found
> >> rte_eth_set_queue_rate_limit(). Is that the function you need?
> >>
> >> From my quick grep'ing, it looks to me like this feature isn't supported
> >> on Mellanox drivers but is in the ixgbe driver. However, this is all guess
> >> work. I'm not an expert. I'd like to know the real answer!
> >>
> >> - Andrew
> >>
> >> -----Original Message-----
> >> From: users <users-bounces@dpdk.org> On Behalf Of Philipp B
> >> Sent: 15 October 2018 11:45
> >> To: shaklee3@gmail.com
> >> Cc: users@dpdk.org
> >> Subject: Re: [dpdk-users] rte_eth_tx_burst: Can I insert timing gaps
> >>
> >> Maybe I should explain with some ASCII art. It's not about sleeping just
> >> the remaining period of 20ms. This is exactly what I am doing already, and
> >> this works fine.
> >>
> >> Think of 20ms intervals like this
> >>
> >> |XXXXXXXXXXXXXXXXXX_____|XXXXXXXXXXXXXXXXXX_____
> >>
> >> Where '|' is the start of the 20ms interval, X is a packet, and _ is an
> >> idle gap of one packet. (Let's just pretend there are 1000s of Xs per
> >> interval).
> >>
> >> As I said, I let the CPU sleep upto the beginning of the interval ('|').
> >> This beginning of interval is the only moment where CPU timing controls
> >> adapter timing. Then I send out a few 1000 packets. In this phase, I have a
> >> few 100 packets buffered by DPDK, so it will not help to sleep on the CPU.
> >>
> >> The pattern above is what I can easily produce just with an OS sleep, a
> >> single buffer pool and rte_eth_tx_burst. What I am looking for, is a way to
> >> e.g. remove every second packet from that pattern, while keeping the other
> >> packet's timings unchanged:
> >>
> >> |X_X_X_X_X_X_X_X_X______|X_X_X_X_X_X_X_X_X______
> >>
> >> Basically, I do not need to transmit anything in the gaps. I just need the
> >> delay. However, as my timing of CPU isn't coupled tightly to the adapter,
> >> sleeping on the cpu will not help. This is intended by
> >> design: I want to blow out a massive number of packets with exact timing
> >> and virtually no CPU requirement.
> >>
> >> What I look for is a sleep instruction executed by the adapter, which is
> >> buffered in order with the packets issued by rte_eth_tx_burst.
> >> (Plus some basic math rules how to convert packet sizes to durations,
> >> based on line speeds).
> >>
> >> Am Sa., 13. Okt. 2018 um 23:05 Uhr schrieb Cliff Burdick <
> >> shaklee3@gmail.com>:
> >>>
> >>> Maybe I'm misunderstanding the problem, but do you need to transmit
> >> anything? Can you just use the rte_cycles functions to sleep for the
> >> remaining period in 20ms?
> >>>
> >>> On Thu, Oct 11, 2018 at 2:04 AM Philipp B <philippb.ontour@gmail.com>
> >> wrote:
> >>>>
> >>>> Hi all!
> >>>>
> >>>> I am working on an RTP test traffic generator. The basic idea is
> >>>> clock_nanosleep providing a 20ms clock cycle to start a (big) number
> >>>> of rte_eth_tx_bursts, sending equally sized buffers. As long as the
> >>>> timing within a 20ms cycle is dictated primarily by the line speed, I
> >>>> can be sure that not just the first buffer of each cycle has a period
> >>>> of 20ms, but also the n-th buffer. (I have sent n-1 buffers before
> >>>> with the same size.)
> >>>>
> >>>> Basically, I see one 20ms interval as a series of time slots, each
> >>>> capable to store an active RTP stream. My question now is, what to to
> >>>> with inactive time slots? As long as all active streams are located
> >>>> at consecutive time slots from the start of the 20ms interval,
> >>>> everything is fine. But I cannot guarantee this.
> >>>>
> >>>> What I need is some kind of dummy buffer, which is not transmitted
> >>>> but generates a tx timing gap as a buffer of X bytes would take to be
> >>>> transferred.
> >>>>
> >>>> Is such a functionality provided? As a workaround, I already thought
> >>>> about sending invalid packets (bad IP Header checksum?). However,
> >>>> this won't be optimal when multiple lines are aggregated.
> >>>>
> >>>> Thanks!
> >>>> Philipp Beyer
> >>
>
> --
> Chair of Network Architectures and Services
> Department of Informatics
> Technical University of Munich
> Boltzmannstr. 3
> 85748 Garching bei München, Germany
>
>
>
>

Re: [dpdk-users] rte_eth_tx_burst: Can I insert timing gaps

[Paul Emmerich]

Hi,

you stumbled on a quite difficult problem here: controlling inter-packet gaps with high accuracy and precision.
Sounds simple but can be quite challenging.
We've faced this challenge when building our packet generator MoonGen: https://github.com/emmericp/MoonGen

We have written a longer evaluation on that problem before: https://www.h2020-endeavour.eu/sites/www.h2020-endeavour.eu/files/u125/mind_the_gap.pdf
(e.g., page 5 figure 2 shows an attempt to generate packets with a gap of exactly 1000ns).

For your problem it is probably good enough to just do a busy sleep (costs performance because you lose batching if you really want no back-to-back frames at all).
It might also help if your hardware has some support for it, but not all hardware features are suitable for this. For example, i40e NICs like to send batches of 128
packets at once when using the rate control feature.

Paul

> Am 15.10.2018 um 12:44 schrieb Philipp B <philippb.ontour@gmail.com>:
> 
> Maybe I should explain with some ASCII art. It's not about sleeping
> just the remaining period of 20ms. This is exactly what I am doing
> already, and this works fine.
> 
> Think of 20ms intervals like this
> 
> |XXXXXXXXXXXXXXXXXX_____|XXXXXXXXXXXXXXXXXX_____
> 
> Where '|' is the start of the 20ms interval, X is a packet, and _ is
> an idle gap of one packet. (Let's just pretend there are 1000s of Xs
> per interval).
> 
> As I said, I let the CPU sleep upto the beginning of the interval
> ('|'). This beginning of interval is the only moment where CPU timing
> controls adapter timing. Then I send out a few 1000 packets. In this
> phase, I have a few 100 packets buffered by DPDK, so it will not help
> to sleep on the CPU.
> 
> The pattern above is what I can easily produce just with an OS sleep,
> a single buffer pool and rte_eth_tx_burst. What I am looking for, is a
> way to e.g. remove every second packet from that pattern, while
> keeping the other packet's timings unchanged:
> 
> |X_X_X_X_X_X_X_X_X______|X_X_X_X_X_X_X_X_X______
> 
> Basically, I do not need to transmit anything in the gaps. I just need
> the delay. However, as my timing of CPU isn't coupled tightly to the
> adapter, sleeping on the cpu will not help. This is intended by
> design: I want to blow out a massive number of packets with exact
> timing and virtually no CPU requirement.
> 
> What I look for is a sleep instruction executed by the adapter, which
> is buffered in order with the packets issued by rte_eth_tx_burst.
> (Plus some basic math rules how to convert packet sizes to durations,
> based on line speeds).
> 
> Am Sa., 13. Okt. 2018 um 23:05 Uhr schrieb Cliff Burdick <shaklee3@gmail.com>:
>> 
>> Maybe I'm misunderstanding the problem, but do you need to transmit anything? Can you just use the rte_cycles functions to sleep for the remaining period in 20ms?
>> 
>> On Thu, Oct 11, 2018 at 2:04 AM Philipp B <philippb.ontour@gmail.com> wrote:
>>> 
>>> Hi all!
>>> 
>>> I am working on an RTP test traffic generator. The basic idea is
>>> clock_nanosleep providing a 20ms clock cycle to start a (big) number
>>> of rte_eth_tx_bursts, sending equally sized buffers. As long as the
>>> timing within a 20ms cycle is dictated primarily by the line speed, I
>>> can be sure that not just the first buffer of each cycle has a period
>>> of 20ms, but also the n-th buffer. (I have sent n-1 buffers before
>>> with the same size.)
>>> 
>>> Basically, I see one 20ms interval as a series of time slots, each
>>> capable to store an active RTP stream. My question now is, what to to
>>> with inactive time slots? As long as all active streams are located at
>>> consecutive time slots from the start of the 20ms interval, everything
>>> is fine. But I cannot guarantee this.
>>> 
>>> What I need is some kind of dummy buffer, which is not transmitted but
>>> generates a tx timing gap as a buffer of X bytes would take to be
>>> transferred.
>>> 
>>> Is such a functionality provided? As a workaround, I already thought
>>> about sending invalid packets (bad IP Header checksum?). However, this
>>> won't be optimal when multiple lines are aggregated.
>>> 
>>> Thanks!
>>> Philipp Beyer