[dpdk-dev] random pkt generator PMD

[dpdk-dev] random pkt generator PMD

[Yerden Zhumabekov]

Hello everybody,

DPDK already got a number of PMDs for various eth devices, it even has 
PMD emulations for backends such as pcap, sw rings etc.

I've been thinking about the idea of having PMD which would generate 
mbufs on the fly in some randomized fashion. This would serve goals 
like, for example:

1) running tests for applications with network processing capabilities 
without additional software packet generators;
2) making performance measurements with no hw inteference;
3) ability to run without root privileges, --no-pci, --no-huge, for CI 
build, so on.

Maybe there's no such need, and these goals may be achieved by other 
means and this idea is flawed? Any thoughts?

Re: [dpdk-dev] random pkt generator PMD

[Bruce Richardson]

On Wed, Jun 15, 2016 at 03:43:56PM +0600, Yerden Zhumabekov wrote:
> Hello everybody,
> 
> DPDK already got a number of PMDs for various eth devices, it even has PMD
> emulations for backends such as pcap, sw rings etc.
> 
> I've been thinking about the idea of having PMD which would generate mbufs
> on the fly in some randomized fashion. This would serve goals like, for
> example:
> 
> 1) running tests for applications with network processing capabilities
> without additional software packet generators;
> 2) making performance measurements with no hw inteference;
> 3) ability to run without root privileges, --no-pci, --no-huge, for CI
> build, so on.
> 
> Maybe there's no such need, and these goals may be achieved by other means
> and this idea is flawed? Any thoughts?

Isn't some of this already covered by the NULL PMD? Perhaps it could be extended
or enhanced to meet some more of your requirements?

/Bruce

Re: [dpdk-dev] random pkt generator PMD

[Yerden Zhumabekov]

On 15.06.2016 15:49, Bruce Richardson wrote:
> On Wed, Jun 15, 2016 at 03:43:56PM +0600, Yerden Zhumabekov wrote:
>> Hello everybody,
>>
>> DPDK already got a number of PMDs for various eth devices, it even has PMD
>> emulations for backends such as pcap, sw rings etc.
>>
>> I've been thinking about the idea of having PMD which would generate mbufs
>> on the fly in some randomized fashion. This would serve goals like, for
>> example:
>>
>> 1) running tests for applications with network processing capabilities
>> without additional software packet generators;
>> 2) making performance measurements with no hw inteference;
>> 3) ability to run without root privileges, --no-pci, --no-huge, for CI
>> build, so on.
>>
>> Maybe there's no such need, and these goals may be achieved by other means
>> and this idea is flawed? Any thoughts?
> Isn't some of this already covered by the NULL PMD? Perhaps it could be extended
> or enhanced to meet some more of your requirements?
>
> /Bruce
Right, but development of various features regarding L3/L4 etc requires 
more subtle approach, like live packets, different protocol versions, 
fields manipulation. In this case some packet mangling/randomizing 
capabilities would be quite useful. Something similar to what is done in 
Pktgen, but more lightweight approach, in a same app.

I've almost made my mind :) so the next question: is there any guide on 
PMD dev? I'm looking through rte_ether.h right now, but some doc would 
be very nice.

Re: [dpdk-dev] random pkt generator PMD

[Bruce Richardson]

On Wed, Jun 15, 2016 at 04:03:59PM +0600, Yerden Zhumabekov wrote:
> On 15.06.2016 15:49, Bruce Richardson wrote:
> >On Wed, Jun 15, 2016 at 03:43:56PM +0600, Yerden Zhumabekov wrote:
> >>Hello everybody,
> >>
> >>DPDK already got a number of PMDs for various eth devices, it even has PMD
> >>emulations for backends such as pcap, sw rings etc.
> >>
> >>I've been thinking about the idea of having PMD which would generate mbufs
> >>on the fly in some randomized fashion. This would serve goals like, for
> >>example:
> >>
> >>1) running tests for applications with network processing capabilities
> >>without additional software packet generators;
> >>2) making performance measurements with no hw inteference;
> >>3) ability to run without root privileges, --no-pci, --no-huge, for CI
> >>build, so on.
> >>
> >>Maybe there's no such need, and these goals may be achieved by other means
> >>and this idea is flawed? Any thoughts?
> >Isn't some of this already covered by the NULL PMD? Perhaps it could be extended
> >or enhanced to meet some more of your requirements?
> >
> >/Bruce
> Right, but development of various features regarding L3/L4 etc requires more
> subtle approach, like live packets, different protocol versions, fields
> manipulation. In this case some packet mangling/randomizing capabilities
> would be quite useful. Something similar to what is done in Pktgen, but more
> lightweight approach, in a same app.
> 
> I've almost made my mind :) so the next question: is there any guide on PMD
> dev? I'm looking through rte_ether.h right now, but some doc would be very
> nice.

Unfortunately not. My suggestion is to take one of the simple vdev's e.g. ring,
pcap, null, and work off a copy of it.

/Bruce

Re: [dpdk-dev] random pkt generator PMD

[Yerden Zhumabekov]

I've developed some preliminary version of the driver. The code is 
derived from Null PMD, but required a lot of rework.

It uses following devargs to generate packets:

1) edit=offset:size:[rnd|value]
     Edit a field within an mbuf packet data with given offset and size. 
Mark it as 'rnd' or assign it a hex value, for example:
    'edit=8:16:rnd' tags field with offset 8 bytes and with size of 16 
bytes random-generated,
    'edit=14:4:0xdeadbeef' assigns a specified sequence of bytes to the 
field (network byte order).

2) tmpl=name
     Use a template with name. Instead of editing data manually, specify 
a hard-coded template and then edit only intended fields. Implemented 
icmp4, tcp4, but needs to be expanded.

3) size=len
     Specify a size of packet. May not be less than size of template 
(checked on devinit).

I ran testpmd (start/stop), then l2fwd, looks like it works, but I'd be 
happy to hear about additional tests I need to run to ensure the PMD 
conformance.

With 64 bytes packet and one 8-byte random field it's about 6-7 Mpps 
now. I use rte_rand()/lrand48() as a source of random bytes, it impacts 
a performance, but I haven't come up with anything else.


On 15.06.2016 16:07, Bruce Richardson wrote:
> On Wed, Jun 15, 2016 at 04:03:59PM +0600, Yerden Zhumabekov wrote:
>>
>> Right, but development of various features regarding L3/L4 etc requires more
>> subtle approach, like live packets, different protocol versions, fields
>> manipulation. In this case some packet mangling/randomizing capabilities
>> would be quite useful. Something similar to what is done in Pktgen, but more
>> lightweight approach, in a same app.
>>
>> I've almost made my mind :) so the next question: is there any guide on PMD
>> dev? I'm looking through rte_ether.h right now, but some doc would be very
>> nice.
> Unfortunately not. My suggestion is to take one of the simple vdev's e.g. ring,
> pcap, null, and work off a copy of it.
>
> /Bruce

Re: [dpdk-dev] random pkt generator PMD

[Dumitrescu, Cristian]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Yerden
> Zhumabekov
> Sent: Wednesday, June 15, 2016 10:44 AM
> To: dev@dpdk.org
> Subject: [dpdk-dev] random pkt generator PMD
> 
> Hello everybody,
> 
> DPDK already got a number of PMDs for various eth devices, it even has
> PMD emulations for backends such as pcap, sw rings etc.
> 
> I've been thinking about the idea of having PMD which would generate
> mbufs on the fly in some randomized fashion. This would serve goals
> like, for example:
> 
> 1) running tests for applications with network processing capabilities
> without additional software packet generators;
> 2) making performance measurements with no hw inteference;
> 3) ability to run without root privileges, --no-pci, --no-huge, for CI
> build, so on.
> 
> Maybe there's no such need, and these goals may be achieved by other
> means and this idea is flawed? Any thoughts?

How about a Perl/Python script to generate a PCAP file with random packets and then feed the PCAP file to the PCAP PMD?

Random can mean different requirements for different users/application, I think it is difficult to fit this  under a simple generic API. Customizing the script for different requirements if a far better option in my opinion.

Regards,
Cristian

Re: [dpdk-dev] random pkt generator PMD

[Yerden Zhumabekov]

On 15.06.2016 16:43, Dumitrescu, Cristian wrote:
>
>> -----Original Message-----
>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Yerden
>> Zhumabekov
>> Sent: Wednesday, June 15, 2016 10:44 AM
>> To: dev@dpdk.org
>> Subject: [dpdk-dev] random pkt generator PMD
>>
>> Hello everybody,
>>
>> DPDK already got a number of PMDs for various eth devices, it even has
>> PMD emulations for backends such as pcap, sw rings etc.
>>
>> I've been thinking about the idea of having PMD which would generate
>> mbufs on the fly in some randomized fashion. This would serve goals
>> like, for example:
>>
>> 1) running tests for applications with network processing capabilities
>> without additional software packet generators;
>> 2) making performance measurements with no hw inteference;
>> 3) ability to run without root privileges, --no-pci, --no-huge, for CI
>> build, so on.
>>
>> Maybe there's no such need, and these goals may be achieved by other
>> means and this idea is flawed? Any thoughts?
> How about a Perl/Python script to generate a PCAP file with random packets and then feed the PCAP file to the PCAP PMD?
>
> Random can mean different requirements for different users/application, I think it is difficult to fit this  under a simple generic API. Customizing the script for different requirements if a far better option in my opinion.

AFAIK, the thing about pcap pmd is that one needs to rewind pcap file 
once pcap pmd reaches its end. It requires additional (non-generic) 
handling in app code.

Re: [dpdk-dev] random pkt generator PMD

[Panu Matilainen]

On 06/15/2016 02:10 PM, Yerden Zhumabekov wrote:
>
>
> On 15.06.2016 16:43, Dumitrescu, Cristian wrote:
>>
>>> -----Original Message-----
>>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Yerden
>>> Zhumabekov
>>> Sent: Wednesday, June 15, 2016 10:44 AM
>>> To: dev@dpdk.org
>>> Subject: [dpdk-dev] random pkt generator PMD
>>>
>>> Hello everybody,
>>>
>>> DPDK already got a number of PMDs for various eth devices, it even has
>>> PMD emulations for backends such as pcap, sw rings etc.
>>>
>>> I've been thinking about the idea of having PMD which would generate
>>> mbufs on the fly in some randomized fashion. This would serve goals
>>> like, for example:
>>>
>>> 1) running tests for applications with network processing capabilities
>>> without additional software packet generators;
>>> 2) making performance measurements with no hw inteference;
>>> 3) ability to run without root privileges, --no-pci, --no-huge, for CI
>>> build, so on.
>>>
>>> Maybe there's no such need, and these goals may be achieved by other
>>> means and this idea is flawed? Any thoughts?
>> How about a Perl/Python script to generate a PCAP file with random
>> packets and then feed the PCAP file to the PCAP PMD?
>>
>> Random can mean different requirements for different
>> users/application, I think it is difficult to fit this  under a simple
>> generic API. Customizing the script for different requirements if a
>> far better option in my opinion.
>
> AFAIK, the thing about pcap pmd is that one needs to rewind pcap file
> once pcap pmd reaches its end. It requires additional (non-generic)
> handling in app code.

So add a loop-mode to pcap pmd?

	- Panu -

Re: [dpdk-dev] random pkt generator PMD

[Yerden Zhumabekov]

On 15.06.2016 17:25, Panu Matilainen wrote:
> On 06/15/2016 02:10 PM, Yerden Zhumabekov wrote:
>>
>>
>> On 15.06.2016 16:43, Dumitrescu, Cristian wrote:
>>>
>>>>
>>>> Hello everybody,
>>>>
>>>> DPDK already got a number of PMDs for various eth devices, it even has
>>>> PMD emulations for backends such as pcap, sw rings etc.
>>>>
>>>> I've been thinking about the idea of having PMD which would generate
>>>> mbufs on the fly in some randomized fashion. This would serve goals
>>>> like, for example:
>>>>
>>>> 1) running tests for applications with network processing capabilities
>>>> without additional software packet generators;
>>>> 2) making performance measurements with no hw inteference;
>>>> 3) ability to run without root privileges, --no-pci, --no-huge, for CI
>>>> build, so on.
>>>>
>>>> Maybe there's no such need, and these goals may be achieved by other
>>>> means and this idea is flawed? Any thoughts?
>>> How about a Perl/Python script to generate a PCAP file with random
>>> packets and then feed the PCAP file to the PCAP PMD?
>>>
>>> Random can mean different requirements for different
>>> users/application, I think it is difficult to fit this  under a simple
>>> generic API. Customizing the script for different requirements if a
>>> far better option in my opinion.
>>
>> AFAIK, the thing about pcap pmd is that one needs to rewind pcap file
>> once pcap pmd reaches its end. It requires additional (non-generic)
>> handling in app code.
>
> So add a loop-mode to pcap pmd?

It would be nice to have an option like "...,rewind=1,...".

Re: [dpdk-dev] random pkt generator PMD

[Panu Matilainen]

On 06/15/2016 03:14 PM, Yerden Zhumabekov wrote:
>
>
> On 15.06.2016 17:25, Panu Matilainen wrote:
>> On 06/15/2016 02:10 PM, Yerden Zhumabekov wrote:
>>>
>>>
>>> On 15.06.2016 16:43, Dumitrescu, Cristian wrote:
>>>>
>>>>>
>>>>> Hello everybody,
>>>>>
>>>>> DPDK already got a number of PMDs for various eth devices, it even has
>>>>> PMD emulations for backends such as pcap, sw rings etc.
>>>>>
>>>>> I've been thinking about the idea of having PMD which would generate
>>>>> mbufs on the fly in some randomized fashion. This would serve goals
>>>>> like, for example:
>>>>>
>>>>> 1) running tests for applications with network processing capabilities
>>>>> without additional software packet generators;
>>>>> 2) making performance measurements with no hw inteference;
>>>>> 3) ability to run without root privileges, --no-pci, --no-huge, for CI
>>>>> build, so on.
>>>>>
>>>>> Maybe there's no such need, and these goals may be achieved by other
>>>>> means and this idea is flawed? Any thoughts?
>>>> How about a Perl/Python script to generate a PCAP file with random
>>>> packets and then feed the PCAP file to the PCAP PMD?
>>>>
>>>> Random can mean different requirements for different
>>>> users/application, I think it is difficult to fit this  under a simple
>>>> generic API. Customizing the script for different requirements if a
>>>> far better option in my opinion.
>>>
>>> AFAIK, the thing about pcap pmd is that one needs to rewind pcap file
>>> once pcap pmd reaches its end. It requires additional (non-generic)
>>> handling in app code.
>>
>> So add a loop-mode to pcap pmd?
>
> It would be nice to have an option like "...,rewind=1,...".

As Cristian points out in 
http://dpdk.org/ml/archives/dev/2016-June/041589.html, the current pmd 
behavior of stopping is the odd man out in the pmd crowd.

Rather than whether to rewind or not, I'd make the number of loops 
configurable, defaulting to forever and 1 being the equal to current 
behavior.

	- Panu -

Re: [dpdk-dev] random pkt generator PMD

[Dumitrescu, Cristian]

> -----Original Message-----
> From: Panu Matilainen [mailto:pmatilai@redhat.com]
> Sent: Wednesday, June 15, 2016 1:24 PM
> To: Yerden Zhumabekov <e_zhumabekov@sts.kz>; Dumitrescu, Cristian
> <cristian.dumitrescu@intel.com>; dev@dpdk.org
> Subject: Re: [dpdk-dev] random pkt generator PMD
> 
> On 06/15/2016 03:14 PM, Yerden Zhumabekov wrote:
> >
> >
> > On 15.06.2016 17:25, Panu Matilainen wrote:
> >> On 06/15/2016 02:10 PM, Yerden Zhumabekov wrote:
> >>>
> >>>
> >>> On 15.06.2016 16:43, Dumitrescu, Cristian wrote:
> >>>>
> >>>>>
> >>>>> Hello everybody,
> >>>>>
> >>>>> DPDK already got a number of PMDs for various eth devices, it even
> has
> >>>>> PMD emulations for backends such as pcap, sw rings etc.
> >>>>>
> >>>>> I've been thinking about the idea of having PMD which would
> generate
> >>>>> mbufs on the fly in some randomized fashion. This would serve goals
> >>>>> like, for example:
> >>>>>
> >>>>> 1) running tests for applications with network processing capabilities
> >>>>> without additional software packet generators;
> >>>>> 2) making performance measurements with no hw inteference;
> >>>>> 3) ability to run without root privileges, --no-pci, --no-huge, for CI
> >>>>> build, so on.
> >>>>>
> >>>>> Maybe there's no such need, and these goals may be achieved by
> other
> >>>>> means and this idea is flawed? Any thoughts?
> >>>> How about a Perl/Python script to generate a PCAP file with random
> >>>> packets and then feed the PCAP file to the PCAP PMD?
> >>>>
> >>>> Random can mean different requirements for different
> >>>> users/application, I think it is difficult to fit this  under a simple
> >>>> generic API. Customizing the script for different requirements if a
> >>>> far better option in my opinion.
> >>>
> >>> AFAIK, the thing about pcap pmd is that one needs to rewind pcap file
> >>> once pcap pmd reaches its end. It requires additional (non-generic)
> >>> handling in app code.
> >>
> >> So add a loop-mode to pcap pmd?
> >
> > It would be nice to have an option like "...,rewind=1,...".
> 
> As Cristian points out in
> http://dpdk.org/ml/archives/dev/2016-June/041589.html, the current pmd
> behavior of stopping is the odd man out in the pmd crowd.
> 
> Rather than whether to rewind or not, I'd make the number of loops
> configurable, defaulting to forever and 1 being the equal to current
> behavior.
> 
> 	- Panu -

+1

Re: [dpdk-dev] random pkt generator PMD

[Yerden Zhumabekov]

On 15.06.2016 18:25, Dumitrescu, Cristian wrote:

>>>> So add a loop-mode to pcap pmd?
>>> It would be nice to have an option like "...,rewind=1,...".
>> As Cristian points out in
>> http://dpdk.org/ml/archives/dev/2016-June/041589.html, the current pmd
>> behavior of stopping is the odd man out in the pmd crowd.
>>
>> Rather than whether to rewind or not, I'd make the number of loops
>> configurable, defaulting to forever and 1 being the equal to current
>> behavior.
>>
>> 	- Panu -
> +1

I'm afraid, all packets from pcap file would need to be preloaded to 
memory. Otherwise, each loop would infer pcap_open/pcap_close(), am I wrong?

Re: [dpdk-dev] random pkt generator PMD

[Dumitrescu, Cristian]

> -----Original Message-----
> From: Yerden Zhumabekov [mailto:e_zhumabekov@sts.kz]
> Sent: Wednesday, June 15, 2016 1:55 PM
> To: Dumitrescu, Cristian <cristian.dumitrescu@intel.com>; Panu Matilainen
> <pmatilai@redhat.com>; dev@dpdk.org
> Subject: Re: [dpdk-dev] random pkt generator PMD
> 
> 
> 
> On 15.06.2016 18:25, Dumitrescu, Cristian wrote:
> 
> >>>> So add a loop-mode to pcap pmd?
> >>> It would be nice to have an option like "...,rewind=1,...".
> >> As Cristian points out in
> >> http://dpdk.org/ml/archives/dev/2016-June/041589.html, the current
> pmd
> >> behavior of stopping is the odd man out in the pmd crowd.
> >>
> >> Rather than whether to rewind or not, I'd make the number of loops
> >> configurable, defaulting to forever and 1 being the equal to current
> >> behavior.
> >>
> >> 	- Panu -
> > +1
> 
> I'm afraid, all packets from pcap file would need to be preloaded to
> memory. Otherwise, each loop would infer pcap_open/pcap_close(), am I
> wrong?

This exactly what the code in source port is doing.

Basically, this is optimized for the case when number of packets in the PCAP file is relatively small, so the PCAP memory footprint when loaded into memory is small so it fits the L1/L2 cache. Provides traffic generation capability when performance measurements are not key: testing, code development on your laptop while on board of a plane, simulation environments, etc.

When the PCAP is large (e.g. capture of the traffic in your local cloud for 2 mins), then PCAP memory gets swapped to disk and performance obviously drops. Still better than opening PCAC for each packet. Useful for e.g. IDS/IPS testing. 

Re: [dpdk-dev] random pkt generator PMD

[Arnon Warshavsky]

On Wed, Jun 15, 2016 at 4:03 PM, Dumitrescu, Cristian <
cristian.dumitrescu@intel.com> wrote:

>
>
> > -----Original Message-----
> > From: Yerden Zhumabekov [mailto:e_zhumabekov@sts.kz]
> > Sent: Wednesday, June 15, 2016 1:55 PM
> > To: Dumitrescu, Cristian <cristian.dumitrescu@intel.com>; Panu
> Matilainen
> > <pmatilai@redhat.com>; dev@dpdk.org
> > Subject: Re: [dpdk-dev] random pkt generator PMD
> >
> >
> >
> > On 15.06.2016 18:25, Dumitrescu, Cristian wrote:
> >
> > >>>> So add a loop-mode to pcap pmd?
> > >>> It would be nice to have an option like "...,rewind=1,...".
> > >> As Cristian points out in
> > >> http://dpdk.org/ml/archives/dev/2016-June/041589.html, the current
> > pmd
> > >> behavior of stopping is the odd man out in the pmd crowd.
> > >>
> > >> Rather than whether to rewind or not, I'd make the number of loops
> > >> configurable, defaulting to forever and 1 being the equal to current
> > >> behavior.
> > >>
> > >>    - Panu -
> > > +1
> >
> > I'm afraid, all packets from pcap file would need to be preloaded to
> > memory. Otherwise, each loop would infer pcap_open/pcap_close(), am I
> > wrong?
>
> This exactly what the code in source port is doing.
>
> Basically, this is optimized for the case when number of packets in the
> PCAP file is relatively small, so the PCAP memory footprint when loaded
> into memory is small so it fits the L1/L2 cache. Provides traffic
> generation capability when performance measurements are not key: testing,
> code development on your laptop while on board of a plane, simulation
> environments, etc.
>
> When the PCAP is large (e.g. capture of the traffic in your local cloud
> for 2 mins), then PCAP memory gets swapped to disk and performance
> obviously drops. Still better than opening PCAC for each packet. Useful for
> e.g. IDS/IPS testing.
>


If you are after continuous traffic that varies all the time in high
performance rather than loop the same pcap over and over again,
check out  the T-Rex traffic generator which is an open source and dpdk
based.
https://trex-tgn.cisco.com/

/Arnon

Re: [dpdk-dev] random pkt generator PMD

[Dumitrescu, Cristian]

> -----Original Message-----
> From: Yerden Zhumabekov [mailto:e_zhumabekov@sts.kz]
> Sent: Wednesday, June 15, 2016 12:11 PM
> To: Dumitrescu, Cristian <cristian.dumitrescu@intel.com>; dev@dpdk.org
> Subject: Re: [dpdk-dev] random pkt generator PMD
> 
> 
> 
> On 15.06.2016 16:43, Dumitrescu, Cristian wrote:
> >
> >> -----Original Message-----
> >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Yerden
> >> Zhumabekov
> >> Sent: Wednesday, June 15, 2016 10:44 AM
> >> To: dev@dpdk.org
> >> Subject: [dpdk-dev] random pkt generator PMD
> >>
> >> Hello everybody,
> >>
> >> DPDK already got a number of PMDs for various eth devices, it even has
> >> PMD emulations for backends such as pcap, sw rings etc.
> >>
> >> I've been thinking about the idea of having PMD which would generate
> >> mbufs on the fly in some randomized fashion. This would serve goals
> >> like, for example:
> >>
> >> 1) running tests for applications with network processing capabilities
> >> without additional software packet generators;
> >> 2) making performance measurements with no hw inteference;
> >> 3) ability to run without root privileges, --no-pci, --no-huge, for CI
> >> build, so on.
> >>
> >> Maybe there's no such need, and these goals may be achieved by other
> >> means and this idea is flawed? Any thoughts?
> > How about a Perl/Python script to generate a PCAP file with random
> packets and then feed the PCAP file to the PCAP PMD?
> >
> > Random can mean different requirements for different users/application, I
> think it is difficult to fit this  under a simple generic API. Customizing the script
> for different requirements if a far better option in my opinion.
> 
> AFAIK, the thing about pcap pmd is that one needs to rewind pcap file
> once pcap pmd reaches its end. It requires additional (non-generic)
> handling in app code.

Yes, this is an obvious improvement that needs to happen to PCAP PMD.

Re: [dpdk-dev] random pkt generator PMD

[Dumitrescu, Cristian]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Dumitrescu,
> Cristian
> Sent: Wednesday, June 15, 2016 12:26 PM
> To: Yerden Zhumabekov <e_zhumabekov@sts.kz>; dev@dpdk.org
> Subject: Re: [dpdk-dev] random pkt generator PMD
> 
> 
> 
> > -----Original Message-----
> > From: Yerden Zhumabekov [mailto:e_zhumabekov@sts.kz]
> > Sent: Wednesday, June 15, 2016 12:11 PM
> > To: Dumitrescu, Cristian <cristian.dumitrescu@intel.com>; dev@dpdk.org
> > Subject: Re: [dpdk-dev] random pkt generator PMD
> >
> >
> >
> > On 15.06.2016 16:43, Dumitrescu, Cristian wrote:
> > >
> > >> -----Original Message-----
> > >> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Yerden
> > >> Zhumabekov
> > >> Sent: Wednesday, June 15, 2016 10:44 AM
> > >> To: dev@dpdk.org
> > >> Subject: [dpdk-dev] random pkt generator PMD
> > >>
> > >> Hello everybody,
> > >>
> > >> DPDK already got a number of PMDs for various eth devices, it even has
> > >> PMD emulations for backends such as pcap, sw rings etc.
> > >>
> > >> I've been thinking about the idea of having PMD which would generate
> > >> mbufs on the fly in some randomized fashion. This would serve goals
> > >> like, for example:
> > >>
> > >> 1) running tests for applications with network processing capabilities
> > >> without additional software packet generators;
> > >> 2) making performance measurements with no hw inteference;
> > >> 3) ability to run without root privileges, --no-pci, --no-huge, for CI
> > >> build, so on.
> > >>
> > >> Maybe there's no such need, and these goals may be achieved by other
> > >> means and this idea is flawed? Any thoughts?
> > > How about a Perl/Python script to generate a PCAP file with random
> > packets and then feed the PCAP file to the PCAP PMD?
> > >
> > > Random can mean different requirements for different
> users/application, I
> > think it is difficult to fit this  under a simple generic API. Customizing the
> script
> > for different requirements if a far better option in my opinion.
> >
> > AFAIK, the thing about pcap pmd is that one needs to rewind pcap file
> > once pcap pmd reaches its end. It requires additional (non-generic)
> > handling in app code.
> 
> Yes, this is an obvious improvement that needs to happen to PCAP PMD.

Please note the PCAP file rewind code is already available as part of the source port in librte_port, it might be just straightforward to fit this code into the PCAP PMD.

Since the NICs generate packets forever, my recommendation is to have the loop/rewind mode as the default behavior for the reworked PCAP PMD, with potentially an option to disable the loop/rewind passed through dev args.

Re: [dpdk-dev] random pkt generator PMD

[Thomas Monjalon]

2016-06-15 17:10, Yerden Zhumabekov:
> On 15.06.2016 16:43, Dumitrescu, Cristian wrote:
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Yerden Zhumabekov
> >> I've been thinking about the idea of having PMD which would generate
> >> mbufs on the fly in some randomized fashion. This would serve goals
> >> like, for example:
> >>
> >> 1) running tests for applications with network processing capabilities
> >> without additional software packet generators;
> >> 2) making performance measurements with no hw inteference;
> >> 3) ability to run without root privileges, --no-pci, --no-huge, for CI
> >> build, so on.
> >>
> >> Maybe there's no such need, and these goals may be achieved by other
> >> means and this idea is flawed? Any thoughts?
> > How about a Perl/Python script to generate a PCAP file with random packets and then feed the PCAP file to the PCAP PMD?
> >
> > Random can mean different requirements for different users/application, I think it is difficult to fit this  under a simple generic API. Customizing the script for different requirements if a far better option in my opinion.
> 
> AFAIK, the thing about pcap pmd is that one needs to rewind pcap file 
> once pcap pmd reaches its end. It requires additional (non-generic) 
> handling in app code.

This kind of specific configuration could be done with devargs.

Re: [dpdk-dev] random pkt generator PMD

[Mcnamara, John]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Yerden Zhumabekov
> Sent: Wednesday, June 15, 2016 12:11 PM
> To: Dumitrescu, Cristian <cristian.dumitrescu@intel.com>; dev@dpdk.org
> Subject: Re: [dpdk-dev] random pkt generator PMD
> 
> > How about a Perl/Python script to generate a PCAP file with random
> packets and then feed the PCAP file to the PCAP PMD?
> >
> > Random can mean different requirements for different users/application,
> I think it is difficult to fit this  under a simple generic API.
> Customizing the script for different requirements if a far better option
> in my opinion.
> 
> AFAIK, the thing about pcap pmd is that one needs to rewind pcap file once
> pcap pmd reaches its end. It requires additional (non-generic) handling in
> app code.

Hi,

Correct, the PCAP PMD currently stops generating packets once it reached the eof, for the file based interface. However, it could be extended to read a pcap into memory or mbufs and replay it/them. That might be a useful starting point before developing a new PMD.

John

Re: [dpdk-dev] random pkt generator PMD

[Jay Rolette]

On Wed, Jun 15, 2016 at 4:43 AM, Yerden Zhumabekov <e_zhumabekov@sts.kz>
wrote:

> Hello everybody,
>
> DPDK already got a number of PMDs for various eth devices, it even has PMD
> emulations for backends such as pcap, sw rings etc.
>
> I've been thinking about the idea of having PMD which would generate mbufs
> on the fly in some randomized fashion. This would serve goals like, for
> example:
>
> 1) running tests for applications with network processing capabilities
> without additional software packet generators;
> 2) making performance measurements with no hw inteference;
> 3) ability to run without root privileges, --no-pci, --no-huge, for CI
> build, so on.
>
> Maybe there's no such need, and these goals may be achieved by other means
> and this idea is flawed? Any thoughts?
>

Are you thinking of something along the lines of what BreakingPoint (now
part of Ixia) does, but as an open source software tool?

Re: [dpdk-dev] random pkt generator PMD

[Yerden Zhumabekov]

On 15.06.2016 17:50, Jay Rolette wrote:
> On Wed, Jun 15, 2016 at 4:43 AM, Yerden Zhumabekov <e_zhumabekov@sts.kz>
> wrote:
>
>> Hello everybody,
>>
>> DPDK already got a number of PMDs for various eth devices, it even has PMD
>> emulations for backends such as pcap, sw rings etc.
>>
>> I've been thinking about the idea of having PMD which would generate mbufs
>> on the fly in some randomized fashion. This would serve goals like, for
>> example:
>>
>> 1) running tests for applications with network processing capabilities
>> without additional software packet generators;
>> 2) making performance measurements with no hw inteference;
>> 3) ability to run without root privileges, --no-pci, --no-huge, for CI
>> build, so on.
>>
>> Maybe there's no such need, and these goals may be achieved by other means
>> and this idea is flawed? Any thoughts?
>>
> Are you thinking of something along the lines of what BreakingPoint (now
> part of Ixia) does, but as an open source software tool?
>

More dreaming than thinking though :) Live flows generation, malware, 
attacks simulation etc is way out of scope of PMD dev, I guess.

Re: [dpdk-dev] random pkt generator PMD

[Jay Rolette]

On Wed, Jun 15, 2016 at 7:11 AM, Yerden Zhumabekov <e_zhumabekov@sts.kz>
wrote:
>
>
> On 15.06.2016 17:50, Jay Rolette wrote:
>
>> On Wed, Jun 15, 2016 at 4:43 AM, Yerden Zhumabekov <e_zhumabekov@sts.kz>
>> wrote:
>>
>> Hello everybody,
>>>
>>> DPDK already got a number of PMDs for various eth devices, it even has
>>> PMD
>>> emulations for backends such as pcap, sw rings etc.
>>>
>>> I've been thinking about the idea of having PMD which would generate
>>> mbufs
>>> on the fly in some randomized fashion. This would serve goals like, for
>>> example:
>>>
>>> 1) running tests for applications with network processing capabilities
>>> without additional software packet generators;
>>> 2) making performance measurements with no hw inteference;
>>> 3) ability to run without root privileges, --no-pci, --no-huge, for CI
>>> build, so on.
>>>
>>> Maybe there's no such need, and these goals may be achieved by other
>>> means
>>> and this idea is flawed? Any thoughts?
>>>
>>> Are you thinking of something along the lines of what BreakingPoint (now
>> part of Ixia) does, but as an open source software tool?
>>
>>
> More dreaming than thinking though :) Live flows generation, malware,
> attacks simulation etc is way out of scope of PMD dev, I guess.
>

Having a DPDK-based open-source BreakingPoint app would be a _fantastic_
tool for the security community, but yes, it doesn't really make sense to
put any of that logic in the PMD itself.

Were you more after the capabilities from that sort of tool or the
experience of writing a PMD?

Re: [dpdk-dev] random pkt generator PMD

[Yerden Zhumabekov]

On 15.06.2016 18:33, Jay Rolette wrote:
> On Wed, Jun 15, 2016 at 7:11 AM, Yerden Zhumabekov <e_zhumabekov@sts.kz>
> wrote:
>>
>> On 15.06.2016 17:50, Jay Rolette wrote:
>>
>>> On Wed, Jun 15, 2016 at 4:43 AM, Yerden Zhumabekov <e_zhumabekov@sts.kz>
>>> wrote:
>>>
>>> Hello everybody,
>>>> DPDK already got a number of PMDs for various eth devices, it even has
>>>> PMD
>>>> emulations for backends such as pcap, sw rings etc.
>>>>
>>>> I've been thinking about the idea of having PMD which would generate
>>>> mbufs
>>>> on the fly in some randomized fashion. This would serve goals like, for
>>>> example:
>>>>
>>>> 1) running tests for applications with network processing capabilities
>>>> without additional software packet generators;
>>>> 2) making performance measurements with no hw inteference;
>>>> 3) ability to run without root privileges, --no-pci, --no-huge, for CI
>>>> build, so on.
>>>>
>>>> Maybe there's no such need, and these goals may be achieved by other
>>>> means
>>>> and this idea is flawed? Any thoughts?
>>>>
>>>> Are you thinking of something along the lines of what BreakingPoint (now
>>> part of Ixia) does, but as an open source software tool?
>>>
>>>
>> More dreaming than thinking though :) Live flows generation, malware,
>> attacks simulation etc is way out of scope of PMD dev, I guess.
>>
> Having a DPDK-based open-source BreakingPoint app would be a _fantastic_
> tool for the security community, but yes, it doesn't really make sense to
> put any of that logic in the PMD itself.
>
> Were you more after the capabilities from that sort of tool or the
> experience of writing a PMD?
>

We're developing packet processing applications for our company and, of 
course, having a testing tool with such capabilities would be great. As 
for experience in PMD development - sure, why not getting it.

Re: [dpdk-dev] random pkt generator PMD

[Neil Horman]

On Wed, Jun 15, 2016 at 03:43:56PM +0600, Yerden Zhumabekov wrote:
> Hello everybody,
> 
> DPDK already got a number of PMDs for various eth devices, it even has PMD
> emulations for backends such as pcap, sw rings etc.
> 
> I've been thinking about the idea of having PMD which would generate mbufs
> on the fly in some randomized fashion. This would serve goals like, for
> example:
> 
> 1) running tests for applications with network processing capabilities
> without additional software packet generators;
> 2) making performance measurements with no hw inteference;
> 3) ability to run without root privileges, --no-pci, --no-huge, for CI
> build, so on.
> 
> Maybe there's no such need, and these goals may be achieved by other means
> and this idea is flawed? Any thoughts?
> 

I think you already have a solution to this problem.  Linux/BSD have multiple
user space packet generators that can dump thier output to a pcap format file,
and dpdk has a pcap pmd that accepts a pcap file as input to send in packets.

Neil

Re: [dpdk-dev] random pkt generator PMD

[Yerden Zhumabekov]

On 15.06.2016 19:02, Neil Horman wrote:
> On Wed, Jun 15, 2016 at 03:43:56PM +0600, Yerden Zhumabekov wrote:
>> Hello everybody,
>>
>> DPDK already got a number of PMDs for various eth devices, it even has PMD
>> emulations for backends such as pcap, sw rings etc.
>>
>> I've been thinking about the idea of having PMD which would generate mbufs
>> on the fly in some randomized fashion. This would serve goals like, for
>> example:
>>
>> 1) running tests for applications with network processing capabilities
>> without additional software packet generators;
>> 2) making performance measurements with no hw inteference;
>> 3) ability to run without root privileges, --no-pci, --no-huge, for CI
>> build, so on.
>>
>> Maybe there's no such need, and these goals may be achieved by other means
>> and this idea is flawed? Any thoughts?
>>
> I think you already have a solution to this problem.  Linux/BSD have multiple
> user space packet generators that can dump thier output to a pcap format file,
> and dpdk has a pcap pmd that accepts a pcap file as input to send in packets.

Things that I don't like about the idea of using PCAP PMD:

1) the need to create additional files with additional scripts and keep 
those with your test suite;
2) the need to rewind pcap once you played it (fixable);
3) reading packets one-by-one, file operations which may lead to perf 
impact;
4) low variability among source packets.

Those are things which put me on idea of randomized packet generator 
PMD. Possible devargs could be:
1) id of a template, like "ipv4", "ipv6", "dot1q" etc;
2) size of mbuf payload;
3) array of tuples like (offset, size, value) with value being exact 
value or "rnd" keyword.