[dpdk-dev] DPDK User Space: Session onUseability and Ease of Use

[dpdk-dev] DPDK User Space: Session onUseability and Ease of Use

[Thomas Herbert]

[-- Attachment #1: Type: text/plain, Size: 186 bytes --]

All:

Captured notes from Jim McNamara's session on Useability and Ease of Use at DPDK User Space:

Jim will or someone will summarize these notes in more legible form.

Page 1


[-- Attachment #2: Type: text/plain, Size: 10 bytes --]



Page 2:

[-- Attachment #3: Type: text/plain, Size: 28 bytes --]



--Thomas F Herbert
Red Hat

[dpdk-dev] DPDK User Space: Session onUseability and Ease of Use

[Thomas F Herbert]

All:

Captured white board notes from Jim McNamara's session on Useability and 
Ease of Use at DPDK User Space today are here:

http://people.redhat.com/therbert/Useability_and_Ease_of_Use_DPDK_User_Space/

--Thomas F Herbert
Red Hat.

Re: [dpdk-dev] DPDK User Space: Session onUseability and Ease of Use

[Mcnamara, John]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Thomas F Herbert
> Sent: Thursday, October 8, 2015 7:30 PM
> To: dev@dpdk.org
> Subject: [dpdk-dev] DPDK User Space: Session onUseability and Ease of Use
> 
> All:
> 
> Captured white board notes from Jim McNamara's session on Useability and
> Ease of Use at DPDK User Space today are here:
> 
> http://people.redhat.com/therbert/Useability_and_Ease_of_Use_DPDK_User_Space/


Hi Tom,

Thanks for that. Here is my summary of the "Usability and Ease of
Use" session from memory and notes. Correction and additions welcome.

* Latest version of the DPDK docs

  - (From an earlier session). Add a doc/latest build of the docs
    to dpdk.org.

* PMD lite

  - Do we need a lighter PMD model? Perhaps based on the Mellanox
    model.

  - Vincent suggested be could remove 90% of the code. I'll leave
    Vincent explain this one.

* OvS issues with Usability

  - Discussion of the DPSK usability issues highlighted on the
    OvS mailing list.

  - http://openvswitch.org/pipermail/dev/2015-August/058814.html

* Distributed testing

  - There should be some form of distributed testing so patches
    can be tested on OSes and hardware that a dev doesn't have.

  - Suggestion to have an Open Lab/Pod similar to the OPNFV model
    that participants can use.

* Debuggability

  - User expectation for tools like tcpdump, ip, ipconfig to work
    with DPDK bound NICs.

  - Easier in a pipeline application where debug is added as a
    pipeline stage.

  - Maybe add debug hooks via rx/tx callbacks.

  - Add/extend a solution based on KNI.

  - Use systemd naming algorithm for KNI.

* Create a User Mailing List

  - An observation was made that the dev@dpdk.org list was very
    developer orientated and patch heavy.

  - Suggestion to add a user@dpdk.org mailing list for people
    with issues or subjects that aren't development related.

  - This seams easy to implement. It may not be well supported
    however resulting in users cross posting into dev@dpdk.org.

  - Probably worth trying anyway.

* Make the dpdk.org website patchable

  - There are already plans to host the dpdk.org code in a git
    repo.

* Add a Contributing Guide.

  - We are at the stage where we need one.

  - Suggestion to just use the Kernel guide.

  - Tailor it for DPDK.

  - Also explain the review process, acks, nacks, etc.

* Add a README .txt or .1st to the root dir.

  - This could just include links to the getting started guides
    and other docs. Either to the online docs or how to build the
    local html versions.

* EAL annoyances

  - Move the EAL to the kernel.

  - Have more/better/all default options. EAL figures out its own
    requirements.

  - Have a default for -n.

* Hugepage consumption

  - Do not allow DPDK applications to grab all available hugepages.

  - Issues with running DPDK apps in tandem with other hugepage
    hungry apps such as Java/Eclipse.

* rte_malloc()

  - Don't use rte_malloc() for non critical objects where
    malloc() would do.

  - Suggestion to allow the type of required memory to be
    specified by rte_malloc()-like function.

* The Build system

  - Make install needs to be improved. Doesn't so what the user expects.

  - Use autotools and configure. (There were some objections that
    this may not be an improvement.)

  - Use kconfig.

  - Keep going with what we have now until it gets too unwieldy
    and needs to be changed. Then use kconfig.

  - Add better support for cross compilation. Useful for arm target.

* Should DPDK applications be running as root

  - Clearly not a great option.

  - Currently required due to kernel.

* Mempool debugging

  - We need better tools to debug memory leaks in the mempools.

  - Suggestion to do this via a valgrind plugin.

* Kernel management of drivers

* Too much duplicated code in the PMDs

  - Duplicated code has crept in organically as PMDs have been
    added.

  - Should be moved up to moved up to the ethdev level

* Logging and debugging via a secondary process

  - Not a well known technique but very useful/powerful

* Run DPDK as a daemon.

* Issues with config files

  - Too many options turned off by default: code paths don't get
    compiled/tested.

* More sample apps

  - Some more examples of using secondary processes.



Of these we the following could be addressed in the near term:


* Latest version of the docs. - Needs support from 6Wind.

* Distributed testing. - Needs support from Intel initially.
  Some of this is already being rolled out on the
  test-report@dpdk.org list for Intel hardware:
  http://dpdk.org/ml/archives/test-report/. Other hardware
  vendors could use the same automated test framework to host
  something similar.

* Debuggability. - Need some volunteers or workable suggestions.

* Create a User Mailing List. - Needs support from 6Wind.

* Make the dpdk.org website patchable. - Needs support from
  6Wind.

* Add a Contributing Guide. - I will submit a doc patch.

* Add a README .txt or .1st to the root dir. - I will submit a
  doc patch.

* The Build system. - There are some patches on the list for an
  updated "make install". This could be a start point.

* Mempool debugging. - Any volunteers to add/improve valgrind
  support for DPDK? See also this patch from Brocade:
  https://bugs.kde.org/show_bug.cgi?id=350405

* Too much duplicated code in the PMDs. - Any volunteers to
  refactor common PMD code up into the ethdev layer?

* Logging and debugging via a secondary process. Any volunteers
  to add a sample app that demonstrates the technique?

John.
-- 

Re: [dpdk-dev] DPDK User Space: Session onUseability and Ease of Use

[Thomas Monjalon]

Thanks John for the summary.

2015-10-13 16:36, Mcnamara, John:
>   - Move the EAL to the kernel.

Please explain what you mean here.
It's difficult to imagine.

> * Latest version of the docs. - Needs support from 6Wind.

OK
It's as simple as a git hook.
Anybody to write and test one locally?
The script could be hosted in the future website git repo.

> * Distributed testing. - Needs support from Intel initially.
>   Some of this is already being rolled out on the
>   test-report@dpdk.org list for Intel hardware:
>   http://dpdk.org/ml/archives/test-report/. Other hardware
>   vendors could use the same automated test framework to host
>   something similar.

IBM has first started a daily compilation test.
Others are welcome.
We had a small session dedicated to this topic during the useability session.

> * Create a User Mailing List. - Needs support from 6Wind.

OK
Name: user@dpdk.org or other suggestion?

> * Make the dpdk.org website patchable. - Needs support from
>   6Wind.

OK
Needs a maintainer, a name for the git repo, the mailing list and the patchwork.

> * Add a Contributing Guide. - I will submit a doc patch.
> 
> * Add a README .txt or .1st to the root dir. - I will submit a
>   doc patch.

Thanks

> * Too much duplicated code in the PMDs. - Any volunteers to
>   refactor common PMD code up into the ethdev layer?

We need more details.
Please start a new thread or a RFC.

> * Logging and debugging via a secondary process. Any volunteers
>   to add a sample app that demonstrates the technique?

There is already one:
	http://dpdk.org/browse/dpdk/tree/app/proc_info/main.c#n289

Re: [dpdk-dev] DPDK User Space: Session onUseability and Ease of Use

[Dave Neary]

Hi,

On 10/13/2015 10:06 PM, Thomas Monjalon wrote:
>> * Create a User Mailing List. - Needs support from 6Wind.
> 
> OK
> Name: user@dpdk.org or other suggestion?

Other pages seem to have standardised on "users" - I don't think it
matters much, either is fine.

Thanks,
Dave.

-- 
Dave Neary - NFV/SDN Community Strategy
Open Source and Standards, Red Hat - http://community.redhat.com
Ph: +1-978-399-2182 / Cell: +1-978-799-3338

Re: [dpdk-dev] DPDK User Space: Session onUseability and Ease of Use

[Vincent JARDIN]

On 13/10/2015 22:06, Thomas Monjalon wrote:
> 2015-10-13 16:36, Mcnamara, John:
>> >   - Move the EAL to the kernel.
> Please explain what you mean here.
> It's difficult to imagine.
>
+Martin@ARM on it. We were rising this topic with him last week.

Re: [dpdk-dev] DPDK User Space: Session onUseability and Ease of Use

[Panu Matilainen]

On 10/13/2015 07:36 PM, Mcnamara, John wrote:
>> -----Original Message-----
>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Thomas F Herbert
>> Sent: Thursday, October 8, 2015 7:30 PM
>> To: dev@dpdk.org
>> Subject: [dpdk-dev] DPDK User Space: Session onUseability and Ease of Use
>>
>> All:
>>
>> Captured white board notes from Jim McNamara's session on Useability and
>> Ease of Use at DPDK User Space today are here:
>>
>> http://people.redhat.com/therbert/Useability_and_Ease_of_Use_DPDK_User_Space/
>
>
> Hi Tom,
>
> Thanks for that. Here is my summary of the "Usability and Ease of
> Use" session from memory and notes. Correction and additions welcome.
>
[...]
> * EAL annoyances
>
>    - Move the EAL to the kernel.
>
>    - Have more/better/all default options. EAL figures out its own
>      requirements.
>
>    - Have a default for -n.

Heh, somebody agrees with me. Patch sent:
http://dpdk.org/ml/archives/dev/2015-October/025045.html

	- Panu -

Re: [dpdk-dev] DPDK User Space: Session onUseability and Ease of Use

[Vincent JARDIN]

On 13/10/2015 18:36, Mcnamara, John wrote:
> * Should DPDK applications be running as root
>
>    - Clearly not a great option.
>
>    - Currently required due to kernel.

It is not 100% accurate. With some systems, it shall be possible to run 
DPDK application without running them as root.

It was commented that there is a solution when PMDs are designed 
differently (verbs API => memory registration of users). Moreover, it 
would ease the cleanup of resources when a DPDK application does a segfault.

Re: [dpdk-dev] DPDK User Space: Session onUseability and Ease of Use

[Vincent JARDIN]

Thomas, John,

thanks for your notes.

Enclosed the contents of our chats,

On 13/10/2015 18:36, Mcnamara, John wrote:
> * PMD lite
>
>    - Do we need a lighter PMD model? Perhaps based on the Mellanox
>      model.
>    - Vincent suggested be could remove 90% of the code. I'll leave
>      Vincent explain this one.

benefits of 100% userland drivers:
   - no dependency on getting kernel patch upstreams first
   - it is possible to run on a latest HW even when the Linux host (or 
host OS) does not support the new NIC/boards supported by the DPDK

drawbacks of 100% userland drivers:
   - redundancy of source code (duplication into the kernel and userland)
       => more maintenance
   - DPDK PMDs are in fact outdated while kernel drivers are supporting 
latest HW (port management - optics - , latest HW revisions, managing 
firmware updates) more frequently
   - do not run when not root applications
   - cannot reuse port management tools from the kernels (ethtool, etc.)

ixgbe pmd
=========
    100% lines of userland
    must be run as root
    ~ 40K lines
    ~ kernel overlaps with 
http://dpdk.org/browse/dpdk/tree/drivers/net/ixgbe/base
    ~ mostly only 
http://dpdk.org/browse/dpdk/tree/drivers/net/ixgbe/ixgbe_rxtx[_vec].c is 
needed, other files are not needed (6k lines of code)

mlx4 pmd
========
    10% lines userland, then using kernel's ones
    could be run as a non root process
    ~ 5.2K lines of code
    ~ the rxtx.c file is http://dpdk.org/browse/dpdk/tree/drivers/net/mlx4/
    BUT using infiniband :( => should be cleand up to use bare minimum like,
     + /* Allocate protection domain. */
       http://dpdk.org/browse/dpdk/tree/drivers/net/mlx4/mlx4.c#n4828

     + Register memory regions -> ibv_reg_mr()
       http://dpdk.org/browse/dpdk/tree/drivers/net/mlx4/mlx4.c#n792

Best regards,
   Vincent

Re: [dpdk-dev] DPDK User Space: Session onUseability and Ease of Use

[Bruce Richardson]

On Wed, Oct 14, 2015 at 04:36:27PM +0200, Vincent JARDIN wrote:
> Thomas, John,
> 
> thanks for your notes.
> 
> Enclosed the contents of our chats,
> 
> On 13/10/2015 18:36, Mcnamara, John wrote:
> >* PMD lite
> >
> >   - Do we need a lighter PMD model? Perhaps based on the Mellanox
> >     model.
> >   - Vincent suggested be could remove 90% of the code. I'll leave
> >     Vincent explain this one.
> 
> benefits of 100% userland drivers:
>   - no dependency on getting kernel patch upstreams first
>   - it is possible to run on a latest HW even when the Linux host (or host
> OS) does not support the new NIC/boards supported by the DPDK
> 
> drawbacks of 100% userland drivers:
>   - redundancy of source code (duplication into the kernel and userland)
>       => more maintenance
>   - DPDK PMDs are in fact outdated while kernel drivers are supporting
> latest HW (port management - optics - , latest HW revisions, managing
> firmware updates) more frequently
>   - do not run when not root applications
>   - cannot reuse port management tools from the kernels (ethtool, etc.)
> 
> ixgbe pmd
> =========
>    100% lines of userland
>    must be run as root
>    ~ 40K lines
>    ~ kernel overlaps with
> http://dpdk.org/browse/dpdk/tree/drivers/net/ixgbe/base
>    ~ mostly only
> http://dpdk.org/browse/dpdk/tree/drivers/net/ixgbe/ixgbe_rxtx[_vec].c is
> needed, other files are not needed (6k lines of code)
> 
> mlx4 pmd
> ========
>    10% lines userland, then using kernel's ones
>    could be run as a non root process
>    ~ 5.2K lines of code
>    ~ the rxtx.c file is http://dpdk.org/browse/dpdk/tree/drivers/net/mlx4/
>    BUT using infiniband :( => should be cleand up to use bare minimum like,
>     + /* Allocate protection domain. */
>       http://dpdk.org/browse/dpdk/tree/drivers/net/mlx4/mlx4.c#n4828
> 
>     + Register memory regions -> ibv_reg_mr()
>       http://dpdk.org/browse/dpdk/tree/drivers/net/mlx4/mlx4.c#n792
> 
> Best regards,
>   Vincent

Hi Vincent,

is the approach described for the mlx4 driver not dependent upon the hardware
having support for infiniband verbs, in which case I don't see how it could be
used for other NICs. Has an attempt been made to create a driver for another
NIC family using this approach to prove it is generally applicable?

/Bruce

Re: [dpdk-dev] DPDK User Space: Session onUseability and Ease of Use

[Simon Kågström]

On 2015-10-13 18:36, Mcnamara, John wrote:
> 
> * More sample apps
> 
>   - Some more examples of using secondary processes.

One thing I would appreciate is more basic samples for the more advanced
functionality. For example, the qos_sched example is (in my opimion)
quite hard to follow since it has both an integrated command line and a
config file.

I think it would be better to split the example to multiple smaller
samples, which uses static setups to more clearly illustrate some
particular functionality of the QoS framework.

// Simon