[dpdk-dev] DPDK Community Conference Call - Friday 31st October

[dpdk-dev] DPDK Community Conference Call - Friday 31st October

[O'driscoll, Tim]

We're planning to hold our first community conference call on Friday 31st October. It's impossible to find a time that suits everybody, so we've chosen to do this in the afternoon/evening in Europe, which is the morning in the USA. This does unfortunately limit participation from PRC, Japan and other parts of the world. Here's the time and date in a variety of time zones:

Dublin (Ireland)				Friday, October 31, 2014 at 4:00:00 PM    GMT UTC         
Paris (France)				Friday, October 31, 2014 at 5:00:00 PM    CET UTC+1 hour  
San Francisco (U.S.A. - California)	Friday, October 31, 2014 at 9:00:00 AM    PDT UTC-7 hours 
New York (U.S.A. - New York)		Friday, October 31, 2014 at 12:00:00 Noon EDT UTC-4 hours 
Tel Aviv (Israel)				Friday, October 31, 2014 at 6:00:00 PM    IST UTC+2 hours 
Moscow (Russia)			Friday, October 31, 2014 at 7:00:00 PM    MSK UTC+3 hours


Audio bridge details are:
France:		+33 1588 77298
Germany:	+49 8999 143191
Israel:		+972 2589 6577
Russia: 		+7 495 641 4663
UK:		+44 1793 402663
USA:		+1 916 356 2663

Bridge: 5
Conference ID: 1264677285

If anybody needs an access number for another country, let me know.


Agenda:
Discuss feature list for DPDK 2.0 (Q1 2015).
Suggestions for topics for future calls.


Thanks,
Tim

Re: [dpdk-dev] DPDK Community Conference Call - Friday 31st October

[Michael Marchetti]

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of O'driscoll, Tim
> Sent: Friday, October 24, 2014 5:22 AM
> To: dev@dpdk.org
> Subject: [dpdk-dev] DPDK Community Conference Call - Friday 31st October
> 
> We're planning to hold our first community conference call on Friday 31st
> October. It's impossible to find a time that suits everybody, so we've chosen
> to do this in the afternoon/evening in Europe, which is the morning in the
> USA. This does unfortunately limit participation from PRC, Japan and other
> parts of the world. Here's the time and date in a variety of time zones:
> 
> Dublin (Ireland)				Friday, October 31, 2014 at
> 4:00:00 PM    GMT UTC
> Paris (France)				Friday, October 31, 2014 at 5:00:00
> PM    CET UTC+1 hour
> San Francisco (U.S.A. - California)	Friday, October 31, 2014 at 9:00:00
> AM    PDT UTC-7 hours
> New York (U.S.A. - New York)		Friday, October 31, 2014 at 12:00:00
> Noon EDT UTC-4 hours
> Tel Aviv (Israel)				Friday, October 31, 2014 at 6:00:00
> PM    IST UTC+2 hours
> Moscow (Russia)			Friday, October 31, 2014 at 7:00:00
> PM    MSK UTC+3 hours
> 
> 
> Audio bridge details are:
> France:		+33 1588 77298
> Germany:	+49 8999 143191
> Israel:		+972 2589 6577
> Russia: 		+7 495 641 4663
> UK:		+44 1793 402663
> USA:		+1 916 356 2663
> 
> Bridge: 5
> Conference ID: 1264677285
> 
> If anybody needs an access number for another country, let me know.

Can you provide a number for Canada?  thanks, Mike.


> 
> 
> Agenda:
> Discuss feature list for DPDK 2.0 (Q1 2015).
> Suggestions for topics for future calls.
> 
> 
> Thanks,
> Tim

Re: [dpdk-dev] DPDK Community Conference Call - Friday 31st October

[O'driscoll, Tim]

> From: Michael Marchetti [mailto:mmarchetti@sandvine.com]
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of O'driscoll, Tim
> > Audio bridge details are:
> > France:		+33 1588 77298
> > Germany:	+49 8999 143191
> > Israel:		+972 2589 6577
> > Russia: 		+7 495 641 4663
> > UK:		+44 1793 402663
> > USA:		+1 916 356 2663
> >
> > Bridge: 5
> > Conference ID: 1264677285
> >
> > If anybody needs an access number for another country, let me know.
> 
> Can you provide a number for Canada?  thanks, Mike.

No problem. The USA number above should work, or else you can use +1-888-875-9370.


Tim

Re: [dpdk-dev] DPDK Community Conference Call - Friday 31st October

[O'driscoll, Tim]

This is just a reminder for anybody who's interested that this will be on in 30 minutes, and that we'll be discussing the feature list for the DPDK 2.0 release in March 2015.

Audio bridge details are:
France:		+33 1588 77298
Germany:	+49 8999 143191
Israel:		+972 2589 6577
Russia: 		+7 495 641 4663
UK:		+44 1793 402663
USA/Canada:	+1 916 356 2663 or +1-888-875-9370

Bridge: 5
Conference ID: 1264677285


Tim

> -----Original Message-----
> From: O'driscoll, Tim
> Sent: Friday, October 24, 2014 10:22 AM
> To: dev@dpdk.org
> Subject: DPDK Community Conference Call - Friday 31st October
> 
> We're planning to hold our first community conference call on Friday 31st
> October. It's impossible to find a time that suits everybody, so we've chosen
> to do this in the afternoon/evening in Europe, which is the morning in the
> USA. This does unfortunately limit participation from PRC, Japan and other
> parts of the world. Here's the time and date in a variety of time zones:
> 
> Dublin (Ireland)			Friday, October 31, 2014 at
> 4:00:00 PM    GMT UTC
> Paris (France)				Friday, October 31, 2014 at 5:00:00
> PM    CET UTC+1 hour
> San Francisco (U.S.A. - California)	Friday, October 31, 2014 at 9:00:00
> AM    PDT UTC-7 hours
> New York (U.S.A. - New York)		Friday, October 31, 2014 at 12:00:00
> Noon EDT UTC-4 hours
> Tel Aviv (Israel)				Friday, October 31, 2014 at 6:00:00
> PM    IST UTC+2 hours
> Moscow (Russia)			Friday, October 31, 2014 at 7:00:00
> PM    MSK UTC+3 hours
> 
> 
> Audio bridge details are:
> France:		+33 1588 77298
> Germany:	+49 8999 143191
> Israel:		+972 2589 6577
> Russia: 		+7 495 641 4663
> UK:		+44 1793 402663
> USA:		+1 916 356 2663
> 
> Bridge: 5
> Conference ID: 1264677285
> 
> If anybody needs an access number for another country, let me know.
> 
> 
> Agenda:
> Discuss feature list for DPDK 2.0 (Q1 2015).
> Suggestions for topics for future calls.
> 
> 
> Thanks,
> Tim

Re: [dpdk-dev] DPDK Community Conference Call - Friday 31st October

[O'driscoll, Tim]

Thanks again to those who attended the call earlier. Hopefully people found it useful.

We'll schedule a follow-up call for 2 weeks' time. One thing that we do want to look into is an easy way to allow screen sharing, so that we can use some slides to guide the discussion. Internally within Intel we use MS Lync. We can try that, but it's not always very user-friendly for external participants, and doesn't have a Linux client. Other options would include GoToMeeting or WebEx. If anybody has input on a good tool for this, let me know.

We covered the following features from our 2.0 list today, and will discuss the remainder on the next call. I've called out below who on our side was describing each of the features, and included their email addresses. If anybody has further questions on these, feel free to either ask openly on the mailing list, or else contact the relevant person directly.

Bifurcated Driver (Danny.Zhou@intel.com)
Packet Reordering/Packet Distributor (Bruce.Richardson@intel.com)
New Hardware Support (Walter.E.Gilmore@intel.com)
Fortville features (Heqing.Zhu@intel.com)
Support Multiple Threads per Core (Venky.Venkatesan@intel.com)
Cuckoo Hash (Bruce.Richardson@intel.com, Venky.Venkatesan@intel.com)

The Cuckoo Hash paper that was mentioned is available at: http://www.cs.cmu.edu/~dongz/papers/cuckooswitch.pdf.

Finally, if anybody has suggestions for topics for future calls, please let me know.


Thanks,
Tim

> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of O'driscoll, Tim
> Sent: Friday, October 31, 2014 3:35 PM
> To: 'dev@dpdk.org'
> Subject: Re: [dpdk-dev] DPDK Community Conference Call - Friday 31st
> October
> 
> This is just a reminder for anybody who's interested that this will be on in 30
> minutes, and that we'll be discussing the feature list for the DPDK 2.0 release
> in March 2015.
> 
> Audio bridge details are:
> France:		+33 1588 77298
> Germany:	+49 8999 143191
> Israel:		+972 2589 6577
> Russia: 		+7 495 641 4663
> UK:		+44 1793 402663
> USA/Canada:	+1 916 356 2663 or +1-888-875-9370
> 
> Bridge: 5
> Conference ID: 1264677285
> 
> 
> Tim
> 
> > -----Original Message-----
> > From: O'driscoll, Tim
> > Sent: Friday, October 24, 2014 10:22 AM
> > To: dev@dpdk.org
> > Subject: DPDK Community Conference Call - Friday 31st October
> >
> > We're planning to hold our first community conference call on Friday
> > 31st October. It's impossible to find a time that suits everybody, so
> > we've chosen to do this in the afternoon/evening in Europe, which is
> > the morning in the USA. This does unfortunately limit participation
> > from PRC, Japan and other parts of the world. Here's the time and date in a
> variety of time zones:
> >
> > Dublin (Ireland)			Friday, October 31, 2014 at
> > 4:00:00 PM    GMT UTC
> > Paris (France)				Friday, October 31, 2014 at 5:00:00
> > PM    CET UTC+1 hour
> > San Francisco (U.S.A. - California)	Friday, October 31, 2014 at 9:00:00
> > AM    PDT UTC-7 hours
> > New York (U.S.A. - New York)		Friday, October 31, 2014 at
> 12:00:00
> > Noon EDT UTC-4 hours
> > Tel Aviv (Israel)				Friday, October 31, 2014 at
> 6:00:00
> > PM    IST UTC+2 hours
> > Moscow (Russia)			Friday, October 31, 2014 at 7:00:00
> > PM    MSK UTC+3 hours
> >
> >
> > Audio bridge details are:
> > France:		+33 1588 77298
> > Germany:	+49 8999 143191
> > Israel:		+972 2589 6577
> > Russia: 		+7 495 641 4663
> > UK:		+44 1793 402663
> > USA:		+1 916 356 2663
> >
> > Bridge: 5
> > Conference ID: 1264677285
> >
> > If anybody needs an access number for another country, let me know.
> >
> >
> > Agenda:
> > Discuss feature list for DPDK 2.0 (Q1 2015).
> > Suggestions for topics for future calls.
> >
> >
> > Thanks,
> > Tim

Re: [dpdk-dev] DPDK Community Conference Call - Friday 31st October

[Neil Horman]

On Fri, Oct 31, 2014 at 05:36:59PM +0000, O'driscoll, Tim wrote:
> Thanks again to those who attended the call earlier. Hopefully people found it useful.
> 
> We'll schedule a follow-up call for 2 weeks' time. One thing that we do want to look into is an easy way to allow screen sharing, so that we can use some slides to guide the discussion. Internally within Intel we use MS Lync. We can try that, but it's not always very user-friendly for external participants, and doesn't have a Linux client. Other options would include GoToMeeting or WebEx. If anybody has input on a good tool for this, let me know.
> 
Don't use Lync, its a horrid tool.  As you note, there is no linux client (nor a
mac client that I'm aware of).  Google hangouts offers screen sharing, and is
free for anyone to use.

> We covered the following features from our 2.0 list today, and will discuss the remainder on the next call. I've called out below who on our side was describing each of the features, and included their email addresses. If anybody has further questions on these, feel free to either ask openly on the mailing list, or else contact the relevant person directly.
> 
> Bifurcated Driver (Danny.Zhou@intel.com)
> Packet Reordering/Packet Distributor (Bruce.Richardson@intel.com)
> New Hardware Support (Walter.E.Gilmore@intel.com)
> Fortville features (Heqing.Zhu@intel.com)
> Support Multiple Threads per Core (Venky.Venkatesan@intel.com)
> Cuckoo Hash (Bruce.Richardson@intel.com, Venky.Venkatesan@intel.com)
> 
> The Cuckoo Hash paper that was mentioned is available at: http://www.cs.cmu.edu/~dongz/papers/cuckooswitch.pdf.
> 
> Finally, if anybody has suggestions for topics for future calls, please let me know.
> 
ABI versioning, in support of packaging the DPDK for non-rebuilding users.

Neil

> 
> Thanks,
> Tim
> 
> > -----Original Message-----
> > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of O'driscoll, Tim
> > Sent: Friday, October 31, 2014 3:35 PM
> > To: 'dev@dpdk.org'
> > Subject: Re: [dpdk-dev] DPDK Community Conference Call - Friday 31st
> > October
> > 
> > This is just a reminder for anybody who's interested that this will be on in 30
> > minutes, and that we'll be discussing the feature list for the DPDK 2.0 release
> > in March 2015.
> > 
> > Audio bridge details are:
> > France:		+33 1588 77298
> > Germany:	+49 8999 143191
> > Israel:		+972 2589 6577
> > Russia: 		+7 495 641 4663
> > UK:		+44 1793 402663
> > USA/Canada:	+1 916 356 2663 or +1-888-875-9370
> > 
> > Bridge: 5
> > Conference ID: 1264677285
> > 
> > 
> > Tim
> > 
> > > -----Original Message-----
> > > From: O'driscoll, Tim
> > > Sent: Friday, October 24, 2014 10:22 AM
> > > To: dev@dpdk.org
> > > Subject: DPDK Community Conference Call - Friday 31st October
> > >
> > > We're planning to hold our first community conference call on Friday
> > > 31st October. It's impossible to find a time that suits everybody, so
> > > we've chosen to do this in the afternoon/evening in Europe, which is
> > > the morning in the USA. This does unfortunately limit participation
> > > from PRC, Japan and other parts of the world. Here's the time and date in a
> > variety of time zones:
> > >
> > > Dublin (Ireland)			Friday, October 31, 2014 at
> > > 4:00:00 PM    GMT UTC
> > > Paris (France)				Friday, October 31, 2014 at 5:00:00
> > > PM    CET UTC+1 hour
> > > San Francisco (U.S.A. - California)	Friday, October 31, 2014 at 9:00:00
> > > AM    PDT UTC-7 hours
> > > New York (U.S.A. - New York)		Friday, October 31, 2014 at
> > 12:00:00
> > > Noon EDT UTC-4 hours
> > > Tel Aviv (Israel)				Friday, October 31, 2014 at
> > 6:00:00
> > > PM    IST UTC+2 hours
> > > Moscow (Russia)			Friday, October 31, 2014 at 7:00:00
> > > PM    MSK UTC+3 hours
> > >
> > >
> > > Audio bridge details are:
> > > France:		+33 1588 77298
> > > Germany:	+49 8999 143191
> > > Israel:		+972 2589 6577
> > > Russia: 		+7 495 641 4663
> > > UK:		+44 1793 402663
> > > USA:		+1 916 356 2663
> > >
> > > Bridge: 5
> > > Conference ID: 1264677285
> > >
> > > If anybody needs an access number for another country, let me know.
> > >
> > >
> > > Agenda:
> > > Discuss feature list for DPDK 2.0 (Q1 2015).
> > > Suggestions for topics for future calls.
> > >
> > >
> > > Thanks,
> > > Tim
> 

Re: [dpdk-dev] DPDK Community Conference Call - Friday 31st October

[Vincent JARDIN]

+1 for hangout
Le 1 nov. 2014 13:59, "Neil Horman" <nhorman@tuxdriver.com> a écrit :

> On Fri, Oct 31, 2014 at 05:36:59PM +0000, O'driscoll, Tim wrote:
> > Thanks again to those who attended the call earlier. Hopefully people
> found it useful.
> >
> > We'll schedule a follow-up call for 2 weeks' time. One thing that we do
> want to look into is an easy way to allow screen sharing, so that we can
> use some slides to guide the discussion. Internally within Intel we use MS
> Lync. We can try that, but it's not always very user-friendly for external
> participants, and doesn't have a Linux client. Other options would include
> GoToMeeting or WebEx. If anybody has input on a good tool for this, let me
> know.
> >
> Don't use Lync, its a horrid tool.  As you note, there is no linux client
> (nor a
> mac client that I'm aware of).  Google hangouts offers screen sharing, and
> is
> free for anyone to use.
>
> > We covered the following features from our 2.0 list today, and will
> discuss the remainder on the next call. I've called out below who on our
> side was describing each of the features, and included their email
> addresses. If anybody has further questions on these, feel free to either
> ask openly on the mailing list, or else contact the relevant person
> directly.
> >
> > Bifurcated Driver (Danny.Zhou@intel.com)
> > Packet Reordering/Packet Distributor (Bruce.Richardson@intel.com)
> > New Hardware Support (Walter.E.Gilmore@intel.com)
> > Fortville features (Heqing.Zhu@intel.com)
> > Support Multiple Threads per Core (Venky.Venkatesan@intel.com)
> > Cuckoo Hash (Bruce.Richardson@intel.com, Venky.Venkatesan@intel.com)
> >
> > The Cuckoo Hash paper that was mentioned is available at:
> http://www.cs.cmu.edu/~dongz/papers/cuckooswitch.pdf.
> >
> > Finally, if anybody has suggestions for topics for future calls, please
> let me know.
> >
> ABI versioning, in support of packaging the DPDK for non-rebuilding users.
>
> Neil
>
> >
> > Thanks,
> > Tim
> >
> > > -----Original Message-----
> > > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of O'driscoll, Tim
> > > Sent: Friday, October 31, 2014 3:35 PM
> > > To: 'dev@dpdk.org'
> > > Subject: Re: [dpdk-dev] DPDK Community Conference Call - Friday 31st
> > > October
> > >
> > > This is just a reminder for anybody who's interested that this will be
> on in 30
> > > minutes, and that we'll be discussing the feature list for the DPDK
> 2.0 release
> > > in March 2015.
> > >
> > > Audio bridge details are:
> > > France:             +33 1588 77298
> > > Germany:    +49 8999 143191
> > > Israel:             +972 2589 6577
> > > Russia:             +7 495 641 4663
> > > UK:         +44 1793 402663
> > > USA/Canada: +1 916 356 2663 or +1-888-875-9370
> > >
> > > Bridge: 5
> > > Conference ID: 1264677285
> > >
> > >
> > > Tim
> > >
> > > > -----Original Message-----
> > > > From: O'driscoll, Tim
> > > > Sent: Friday, October 24, 2014 10:22 AM
> > > > To: dev@dpdk.org
> > > > Subject: DPDK Community Conference Call - Friday 31st October
> > > >
> > > > We're planning to hold our first community conference call on Friday
> > > > 31st October. It's impossible to find a time that suits everybody, so
> > > > we've chosen to do this in the afternoon/evening in Europe, which is
> > > > the morning in the USA. This does unfortunately limit participation
> > > > from PRC, Japan and other parts of the world. Here's the time and
> date in a
> > > variety of time zones:
> > > >
> > > > Dublin (Ireland)                  Friday, October 31, 2014 at
> > > > 4:00:00 PM    GMT UTC
> > > > Paris (France)                            Friday, October 31, 2014
> at 5:00:00
> > > > PM    CET UTC+1 hour
> > > > San Francisco (U.S.A. - California)       Friday, October 31, 2014
> at 9:00:00
> > > > AM    PDT UTC-7 hours
> > > > New York (U.S.A. - New York)              Friday, October 31, 2014 at
> > > 12:00:00
> > > > Noon EDT UTC-4 hours
> > > > Tel Aviv (Israel)                         Friday, October 31, 2014 at
> > > 6:00:00
> > > > PM    IST UTC+2 hours
> > > > Moscow (Russia)                   Friday, October 31, 2014 at 7:00:00
> > > > PM    MSK UTC+3 hours
> > > >
> > > >
> > > > Audio bridge details are:
> > > > France:           +33 1588 77298
> > > > Germany:  +49 8999 143191
> > > > Israel:           +972 2589 6577
> > > > Russia:           +7 495 641 4663
> > > > UK:               +44 1793 402663
> > > > USA:              +1 916 356 2663
> > > >
> > > > Bridge: 5
> > > > Conference ID: 1264677285
> > > >
> > > > If anybody needs an access number for another country, let me know.
> > > >
> > > >
> > > > Agenda:
> > > > Discuss feature list for DPDK 2.0 (Q1 2015).
> > > > Suggestions for topics for future calls.
> > > >
> > > >
> > > > Thanks,
> > > > Tim
> >
>

Re: [dpdk-dev] bifurcated driver

[Thomas Monjalon]

Hi Danny,

2014-10-31 17:36, O'driscoll, Tim:
> Bifurcated Driver (Danny.Zhou@intel.com)

Thanks for the presentation of bifurcated driver during the community call.
I asked if you looked at ibverbs and you wanted a link to check.
The kernel module is here:
	http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/drivers/infiniband/core
The userspace library:
	http://git.kernel.org/cgit/libs/infiniband/libibverbs.git

Extract from Kconfig:
"
config INFINIBAND_USER_ACCESS
	tristate "InfiniBand userspace access (verbs and CM)"
	select ANON_INODES
	---help---
	  Userspace InfiniBand access support.  This enables the
	  kernel side of userspace verbs and the userspace
	  communication manager (CM).  This allows userspace processes
	  to set up connections and directly access InfiniBand
	  hardware for fast-path operations.  You will also need
	  libibverbs, libibcm and a hardware driver library from
	  <http://www.openfabrics.org/git/>.
"

It seems to be close to the bifurcated driver needs.
Not sure if it can solve the security issues if there is no dedicated MMU
in the NIC.

I feel we should sum up pros and cons of
	- igb_uio
	- uio_pci_generic
	- VFIO
	- ibverbs
	- bifurcated driver
I suggest to consider these criterias:
	- upstream status
	- usable with kernel netdev
	- usable in a vm
	- usable for ethernet
	- hardware requirements
	- security protection
	- performance

-- 
Thomas

Re: [dpdk-dev] bifurcated driver

[Alex Markuze]

On Wed, Nov 5, 2014 at 3:00 PM, Thomas Monjalon <thomas.monjalon@6wind.com>
wrote:

> Hi Danny,
>
> 2014-10-31 17:36, O'driscoll, Tim:
> > Bifurcated Driver (Danny.Zhou@intel.com)
>
> Thanks for the presentation of bifurcated driver during the community call.
> I asked if you looked at ibverbs and you wanted a link to check.
> The kernel module is here:
>
> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/drivers/infiniband/core
> The userspace library:
>         http://git.kernel.org/cgit/libs/infiniband/libibverbs.git
>
> Extract from Kconfig:
> "
> config INFINIBAND_USER_ACCESS
>         tristate "InfiniBand userspace access (verbs and CM)"
>         select ANON_INODES
>         ---help---
>           Userspace InfiniBand access support.  This enables the
>           kernel side of userspace verbs and the userspace
>           communication manager (CM).  This allows userspace processes
>           to set up connections and directly access InfiniBand
>           hardware for fast-path operations.  You will also need
>           libibverbs, libibcm and a hardware driver library from
>           <http://www.openfabrics.org/git/>.
> "
>
> It seems to be close to the bifurcated driver needs.
> Not sure if it can solve the security issues if there is no dedicated MMU
> in the NIC.
>

Mellanox NIC's and other  RDMA HW (Infiniband/RoCE/iWARP) have MTT units -
memory translation units - a dedicated MMU. These are filled via an
ibv_reg_mr sys calls - this creates a Process VM to physical/iova memory
mapping in the NIC. Thus each process can access only its own memory via
the NIC. This is the way RNICs resolve the security issue I'm not sure how
standard intel nics could support this scheme.

There is already a 6wind PMD for mellanox Nics. I'm assuming this PMD is
verbs based and behaves similar to the bifurcated driver proposed.
http://www.mellanox.com/page/press_release_item?id=979

One, thing that I don't understand (And will be happy if some one could
shed some light on), is how does the NIC supposed do distinguish between
packets that need to go to the kernel driver rings and packets going to
user space rings.

I feel we should sum up pros and cons of
>         - igb_uio
>         - uio_pci_generic
>         - VFIO
>         - ibverbs
>         - bifurcated driver
> I suggest to consider these criterias:
>         - upstream status
>         - usable with kernel netdev
>         - usable in a vm
>         - usable for ethernet
>         - hardware requirements
>         - security protection
>         - performance
>
> Regarding obverts - I'm not sure how its relevant to future DPDK
development , but this is the run down as I know It.
 This is a veteran package called OFED , or its counterpart Mellanox OFED.
   ---- The kernel drivers are upstream
   ---- The PCI dev stays in the kernels care trough out its life span
   ---- SRIOV support exists, paravirt support exists only(AFAIK) as an
Office of the CTO(VMware) project called vRDMA
   ---- Eth/RoCE (RDMA over Converged Ethernet)/IB
   === HW === RDMA capable HW ONLY.
   ---- Security is designed into RDMA HW
   ---- Stellar performance - Favored by HPC.



> --
> Thomas
>

Re: [dpdk-dev] bifurcated driver

[Alex Markuze]

On Wed, Nov 5, 2014 at 5:14 PM, Alex Markuze <alex@weka.io> wrote:

> On Wed, Nov 5, 2014 at 3:00 PM, Thomas Monjalon <thomas.monjalon@6wind.com
> > wrote:
>
>> Hi Danny,
>>
>> 2014-10-31 17:36, O'driscoll, Tim:
>> > Bifurcated Driver (Danny.Zhou@intel.com)
>>
>> Thanks for the presentation of bifurcated driver during the community
>> call.
>> I asked if you looked at ibverbs and you wanted a link to check.
>> The kernel module is here:
>>
>> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/drivers/infiniband/core
>> The userspace library:
>>         http://git.kernel.org/cgit/libs/infiniband/libibverbs.git
>>
>> Extract from Kconfig:
>> "
>> config INFINIBAND_USER_ACCESS
>>         tristate "InfiniBand userspace access (verbs and CM)"
>>         select ANON_INODES
>>         ---help---
>>           Userspace InfiniBand access support.  This enables the
>>           kernel side of userspace verbs and the userspace
>>           communication manager (CM).  This allows userspace processes
>>           to set up connections and directly access InfiniBand
>>           hardware for fast-path operations.  You will also need
>>           libibverbs, libibcm and a hardware driver library from
>>           <http://www.openfabrics.org/git/>.
>> "
>>
>> It seems to be close to the bifurcated driver needs.
>> Not sure if it can solve the security issues if there is no dedicated MMU
>> in the NIC.
>>
>
> Mellanox NIC's and other  RDMA HW (Infiniband/RoCE/iWARP) have MTT units -
> memory translation units - a dedicated MMU. These are filled via an
> ibv_reg_mr sys calls - this creates a Process VM to physical/iova memory
> mapping in the NIC. Thus each process can access only its own memory via
> the NIC. This is the way RNIC*s resolve the security issue I'm not sure how
> standard intel nics could support this scheme.
>
> There is already a 6wind PMD for mellanox Nics. I'm assuming this PMD is
> verbs based and behaves similar to the bifurcated driver proposed.
> http://www.mellanox.com/page/press_release_item?id=979
>
> One, thing that I don't understand (And will be happy if some one could
> shed some light on), is how does the NIC supposed do distinguish between
> packets that need to go to the kernel driver rings and packets going to
> user space rings.
>
> I feel we should sum up pros and cons of
>>         - igb_uio
>>         - uio_pci_generic
>>         - VFIO
>>         - ibverbs
>>         - bifurcated driver
>> I suggest to consider these criterias:
>>         - upstream status
>>         - usable with kernel netdev
>>         - usable in a vm
>>         - usable for ethernet
>>         - hardware requirements
>>         - security protection
>>         - performance
>>
>> Regarding IBVERBS - I'm not sure how its relevant to future DPDK
> development , but this is the run down as I know It.
>  This is a veteran package called OFED , or its counterpart Mellanox OFED.
>    ---- The kernel drivers are upstream
>    ---- The PCI dev stays in the kernels care trough out its life span
>    ---- SRIOV support exists, paravirt support exists only(AFAIK) as an
> Office of the CTO(VMware) project called vRDMA
>    ---- Eth/RoCE (RDMA over Converged Ethernet)/IB
>    === HW === RDMA capable HW ONLY.
>    ---- Security is designed into RDMA HW
>
   ---- Stellar performance - Favored by HPC.
>

*RNIC - RDMA (Remote DMA - iWARP/Infinibad/RoCE)capable NICs.

>
>
>> --
>> Thomas
>>
>
>

Re: [dpdk-dev] bifurcated driver

[Zhou, Danny]

From: Alex Markuze [mailto:alex@weka.io]
Sent: Wednesday, November 05, 2014 11:19 PM
To: Thomas Monjalon
Cc: Zhou, Danny; dev@dpdk.org; Fastabend, John R
Subject: Re: [dpdk-dev] bifurcated driver



On Wed, Nov 5, 2014 at 5:14 PM, Alex Markuze <alex@weka.io<mailto:alex@weka.io>> wrote:
On Wed, Nov 5, 2014 at 3:00 PM, Thomas Monjalon <thomas.monjalon@6wind.com<mailto:thomas.monjalon@6wind.com>> wrote:
Hi Danny,

2014-10-31 17:36, O'driscoll, Tim:
> Bifurcated Driver (Danny.Zhou@intel.com<mailto:Danny.Zhou@intel.com>)

Thanks for the presentation of bifurcated driver during the community call.
I asked if you looked at ibverbs and you wanted a link to check.
The kernel module is here:
        http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/drivers/infiniband/core
The userspace library:
        http://git.kernel.org/cgit/libs/infiniband/libibverbs.git

Extract from Kconfig:
"
config INFINIBAND_USER_ACCESS
        tristate "InfiniBand userspace access (verbs and CM)"
        select ANON_INODES
        ---help---
          Userspace InfiniBand access support.  This enables the
          kernel side of userspace verbs and the userspace
          communication manager (CM).  This allows userspace processes
          to set up connections and directly access InfiniBand
          hardware for fast-path operations.  You will also need
          libibverbs, libibcm and a hardware driver library from
          <http://www.openfabrics.org/git/>.
"

It seems to be close to the bifurcated driver needs.
Not sure if it can solve the security issues if there is no dedicated MMU
in the NIC.

Mellanox NIC's and other  RDMA HW (Infiniband/RoCE/iWARP) have MTT units - memory translation units - a dedicated MMU. These are filled via an ibv_reg_mr sys calls - this creates a Process VM to physical/iova memory mapping in the NIC. Thus each process can access only its own memory via the NIC. This is the way RNIC*s resolve the security issue I'm not sure how standard intel nics could support this scheme.

DZ:  Intel NICs does not provide such a embedded memory translation unit, but Intel chipset supports IOMMU with a generic memory protection mechanism to provide physical/iova memory mapping for DMA transactions on any PCIe device, rather than NIC only.

There is already a 6wind PMD for mellanox Nics. I'm assuming this PMD is verbs based and behaves similar to the bifurcated driver proposed.
http://www.mellanox.com/page/press_release_item?id=979

DZ: is it open sourced for community to use? I guess answer is No. Also, that PMD should have ported majority of Mellanox kernel driver code to DPDK as lots of NIC control related code needed, while the bifurcated driver approach only needs to support minimum Mellanox NIC specific packet rx/tx routines to achieve the DPDK claimed high performance by using all DPDK performance optimization techniques, such as huge page, fixed-size packet buffer, zero-copy, PMD, etc. Kernel driver still remains NIC control, without porting it to DPDK.

One, thing that I don't understand (And will be happy if some one could shed some light on), is how does the NIC supposed do distinguish between packets that need to go to the kernel driver rings and packets going to user space rings.

DZ: it depends on user. User should use standard ethtool (see below examples) to enable flow director and distribute packets to kernel or user space owned rx queue, by specifying 5-tuple as well as destination rxq index. Flow director embedded in NIC does flow classification and distribution, rather than the software approach like DPDK KNI. If you argue SRIOV has similar rx/tx queue pair partition capability, I would say bifurcated driver approach provides much more flexibility than SRIOV, (e.g, variable number of qpairs allocation for user space, L3 5-tuple based flow classification and distribution rather than SRIOV’ L2 classification based on MAC or VLAN)

ethtool -K ethX ntuple on   # enable flow director
ethtool -N ethX flow-type udp4 src-ip 0.0.0.0 action 0   # distribute udp packet wit source IP 0.0.0.0 to rx queue No.0

I feel we should sum up pros and cons of
        - igb_uio
        - uio_pci_generic
        - VFIO
        - ibverbs
        - bifurcated driver
I suggest to consider these criterias:
        - upstream status
        - usable with kernel netdev
        - usable in a vm
        - usable for ethernet
        - hardware requirements
        - security protection
        - performance
Regarding IBVERBS - I'm not sure how its relevant to future DPDK development , but this is the run down as I know It.
 This is a veteran package called OFED , or its counterpart Mellanox OFED.
   ---- The kernel drivers are upstream
   ---- The PCI dev stays in the kernels care trough out its life span
   ---- SRIOV support exists, paravirt support exists only(AFAIK) as an Office of the CTO(VMware) project called vRDMA
   ---- Eth/RoCE (RDMA over Converged Ethernet)/IB
   === HW === RDMA capable HW ONLY.
   ---- Security is designed into RDMA HW
   ---- Stellar performance - Favored by HPC.

*RNIC - RDMA (Remote DMA - iWARP/Infinibad/RoCE)capable NICs.

--
Thomas

Re: [dpdk-dev] bifurcated driver

[Zhou, Danny]

Hi Thomas,

Thanks for sharing the links to ibverbs, I will take a close look at it and compare it to bifurcated driver. My take
after a rough review is that idea is very much similar, but bifurcated driver implementation is generic for any 
Ethernet device based on existing af_packet mechanism, with extension of exchanging the messages between 
user space and kernel space driver.

I have an internal document to summary the pros and cons of below solutions, except for ibvers, but 
will be adding it shortly.

- igb_uio
- uio_pci_generic
- VFIO
- bifurcated driver

Short answers to your questions:
> 	- upstream status
Adding IOMMU based memory protection and generic descriptor description support now, into version 2 
kernel patches.

> 	- usable with kernel netdev
af_packet based, and relevant patchset will be submitted to netdev for sure.

> 	- usable in a vm
No, it does no coexist with SRIOV for number of reasons. but if you pass-through a PF to a VM, it works perfect.

> 	- usable for Ethernet
It could work with all Ethernet NICs, as flow director is available and NIC driver support new net_ops to split off 
queue pairs for user space.

> 	- hardware requirements
No specific hardware requirements. All mainstream NICs have multiple qpairs and flow director support. 

> 	- security protection
Leverage IOMMU to provide memory protection on Intel platform. Other archs provide similar memory protection
mechanism, so we only use arch-agnostic DMA memory allocation APIs in kernel to support memory protection.

> 	- performance
DPDK native performance on user space queues, as long as drop_en is enabled to avoid head-of-line blocking.

-Danny

> -----Original Message-----
> From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> Sent: Wednesday, November 05, 2014 9:01 PM
> To: Zhou, Danny
> Cc: dev@dpdk.org; Fastabend, John R
> Subject: Re: [dpdk-dev] bifurcated driver
> 
> Hi Danny,
> 
> 2014-10-31 17:36, O'driscoll, Tim:
> > Bifurcated Driver (Danny.Zhou@intel.com)
> 
> Thanks for the presentation of bifurcated driver during the community call.
> I asked if you looked at ibverbs and you wanted a link to check.
> The kernel module is here:
> 	http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/drivers/infiniband/core
> The userspace library:
> 	http://git.kernel.org/cgit/libs/infiniband/libibverbs.git
> 
> Extract from Kconfig:
> "
> config INFINIBAND_USER_ACCESS
> 	tristate "InfiniBand userspace access (verbs and CM)"
> 	select ANON_INODES
> 	---help---
> 	  Userspace InfiniBand access support.  This enables the
> 	  kernel side of userspace verbs and the userspace
> 	  communication manager (CM).  This allows userspace processes
> 	  to set up connections and directly access InfiniBand
> 	  hardware for fast-path operations.  You will also need
> 	  libibverbs, libibcm and a hardware driver library from
> 	  <http://www.openfabrics.org/git/>.
> "
> 
> It seems to be close to the bifurcated driver needs.
> Not sure if it can solve the security issues if there is no dedicated MMU
> in the NIC.
> 
> I feel we should sum up pros and cons of
> 	- igb_uio
> 	- uio_pci_generic
> 	- VFIO
> 	- ibverbs
> 	- bifurcated driver
> I suggest to consider these criterias:
> 	- upstream status
> 	- usable with kernel netdev
> 	- usable in a vm
> 	- usable for ethernet
> 	- hardware requirements
> 	- security protection
> 	- performance
> 
> --
> Thomas

Re: [dpdk-dev] bifurcated driver

[Vincent JARDIN]

+Or

On 05/11/2014 23:48, Zhou, Danny wrote:
> Hi Thomas,
>
> Thanks for sharing the links to ibverbs, I will take a close look at it and compare it to bifurcated driver. My take
> after a rough review is that idea is very much similar, but bifurcated driver implementation is generic for any
> Ethernet device based on existing af_packet mechanism, with extension of exchanging the messages between
> user space and kernel space driver.
>
> I have an internal document to summary the pros and cons of below solutions, except for ibvers, but
> will be adding it shortly.
>
> - igb_uio
> - uio_pci_generic
> - VFIO
> - bifurcated driver
>
> Short answers to your questions:
>> 	- upstream status
> Adding IOMMU based memory protection and generic descriptor description support now, into version 2
> kernel patches.
>
>> 	- usable with kernel netdev
> af_packet based, and relevant patchset will be submitted to netdev for sure.
>
>> 	- usable in a vm
> No, it does no coexist with SRIOV for number of reasons. but if you pass-through a PF to a VM, it works perfect.
>
>> 	- usable for Ethernet
> It could work with all Ethernet NICs, as flow director is available and NIC driver support new net_ops to split off
> queue pairs for user space.
>
>> 	- hardware requirements
> No specific hardware requirements. All mainstream NICs have multiple qpairs and flow director support.
>
>> 	- security protection
> Leverage IOMMU to provide memory protection on Intel platform. Other archs provide similar memory protection
> mechanism, so we only use arch-agnostic DMA memory allocation APIs in kernel to support memory protection.
>
>> 	- performance
> DPDK native performance on user space queues, as long as drop_en is enabled to avoid head-of-line blocking.
>
> -Danny
>
>> -----Original Message-----
>> From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
>> Sent: Wednesday, November 05, 2014 9:01 PM
>> To: Zhou, Danny
>> Cc: dev@dpdk.org; Fastabend, John R
>> Subject: Re: [dpdk-dev] bifurcated driver
>>
>> Hi Danny,
>>
>> 2014-10-31 17:36, O'driscoll, Tim:
>>> Bifurcated Driver (Danny.Zhou@intel.com)
>>
>> Thanks for the presentation of bifurcated driver during the community call.
>> I asked if you looked at ibverbs and you wanted a link to check.
>> The kernel module is here:
>> 	http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/drivers/infiniband/core
>> The userspace library:
>> 	http://git.kernel.org/cgit/libs/infiniband/libibverbs.git
>>
>> Extract from Kconfig:
>> "
>> config INFINIBAND_USER_ACCESS
>> 	tristate "InfiniBand userspace access (verbs and CM)"
>> 	select ANON_INODES
>> 	---help---
>> 	  Userspace InfiniBand access support.  This enables the
>> 	  kernel side of userspace verbs and the userspace
>> 	  communication manager (CM).  This allows userspace processes
>> 	  to set up connections and directly access InfiniBand
>> 	  hardware for fast-path operations.  You will also need
>> 	  libibverbs, libibcm and a hardware driver library from
>> 	  <http://www.openfabrics.org/git/>.
>> "
>>
>> It seems to be close to the bifurcated driver needs.
>> Not sure if it can solve the security issues if there is no dedicated MMU
>> in the NIC.
>>
>> I feel we should sum up pros and cons of
>> 	- igb_uio
>> 	- uio_pci_generic
>> 	- VFIO
>> 	- ibverbs
>> 	- bifurcated driver
>> I suggest to consider these criterias:
>> 	- upstream status
>> 	- usable with kernel netdev
>> 	- usable in a vm
>> 	- usable for ethernet
>> 	- hardware requirements
>> 	- security protection
>> 	- performance
>>
>> --
>> Thomas

Re: [dpdk-dev] bifurcated driver

[Zhou, Danny]

I roughly read libibverbs related code and relevant infiniband/rdma documents, and found though 
many concepts in libibverbs looks similar to bifurcated driver, but there are still lots of differences as 
illustrated below based on my understanding: 

1) Queue pair defined in RDMA specification are abstract concept, where the queue pairs term used in 
  bifurcated driver are rx/tx queue pairs in the NIC.
2) Bifurcated PMD in DPDK directly access NIC resources as a slave driver (no NIC control), while libibverbs
  as a user space library rather than driver offloads certain operations to kernel driver and NIC by invoking 
  "verbs" APIs.
3) Libibverbs invokes infiniband specific system calls to allow user/kernel space communication based on 
  "verbs" defined in infiniband/RDMA spec, while bifurcated driver build on top of af_packet module 
  and new socket options to do things like hw queue split-off , map certain pages on I/O space to user space 
  operations, etc.
4) There is a specific embedded MMU unit in Infiniband/RDMA to provides memory protection, while
  bifurcated driver uses IOMMU rather than NIC to provide memory protection.

IMHO, libibverbs and corresponding kernel modules/drivers are specifically designed and implemented for 
direct access to RDMA hardware from userspace, and it highly depends on "verbs" related system calls 
supported by infiniband/rdma mechanism in kernel, rather than netdev mechanism that bifurcated driver 
solution depends on. 

> -----Original Message-----
> From: Vincent JARDIN [mailto:vincent.jardin@6wind.com]
> Sent: Thursday, November 06, 2014 9:31 AM
> To: Zhou, Danny
> Cc: Thomas Monjalon; dev@dpdk.org; Fastabend, John R; Or Gerlitz
> Subject: Re: [dpdk-dev] bifurcated driver
> 
> +Or
> 
> On 05/11/2014 23:48, Zhou, Danny wrote:
> > Hi Thomas,
> >
> > Thanks for sharing the links to ibverbs, I will take a close look at it and compare it to bifurcated driver. My take
> > after a rough review is that idea is very much similar, but bifurcated driver implementation is generic for any
> > Ethernet device based on existing af_packet mechanism, with extension of exchanging the messages between
> > user space and kernel space driver.
> >
> > I have an internal document to summary the pros and cons of below solutions, except for ibvers, but
> > will be adding it shortly.
> >
> > - igb_uio
> > - uio_pci_generic
> > - VFIO
> > - bifurcated driver
> >
> > Short answers to your questions:
> >> 	- upstream status
> > Adding IOMMU based memory protection and generic descriptor description support now, into version 2
> > kernel patches.
> >
> >> 	- usable with kernel netdev
> > af_packet based, and relevant patchset will be submitted to netdev for sure.
> >
> >> 	- usable in a vm
> > No, it does no coexist with SRIOV for number of reasons. but if you pass-through a PF to a VM, it works perfect.
> >
> >> 	- usable for Ethernet
> > It could work with all Ethernet NICs, as flow director is available and NIC driver support new net_ops to split off
> > queue pairs for user space.
> >
> >> 	- hardware requirements
> > No specific hardware requirements. All mainstream NICs have multiple qpairs and flow director support.
> >
> >> 	- security protection
> > Leverage IOMMU to provide memory protection on Intel platform. Other archs provide similar memory protection
> > mechanism, so we only use arch-agnostic DMA memory allocation APIs in kernel to support memory protection.
> >
> >> 	- performance
> > DPDK native performance on user space queues, as long as drop_en is enabled to avoid head-of-line blocking.
> >
> > -Danny
> >
> >> -----Original Message-----
> >> From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> >> Sent: Wednesday, November 05, 2014 9:01 PM
> >> To: Zhou, Danny
> >> Cc: dev@dpdk.org; Fastabend, John R
> >> Subject: Re: [dpdk-dev] bifurcated driver
> >>
> >> Hi Danny,
> >>
> >> 2014-10-31 17:36, O'driscoll, Tim:
> >>> Bifurcated Driver (Danny.Zhou@intel.com)
> >>
> >> Thanks for the presentation of bifurcated driver during the community call.
> >> I asked if you looked at ibverbs and you wanted a link to check.
> >> The kernel module is here:
> >> 	http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/drivers/infiniband/core
> >> The userspace library:
> >> 	http://git.kernel.org/cgit/libs/infiniband/libibverbs.git
> >>
> >> Extract from Kconfig:
> >> "
> >> config INFINIBAND_USER_ACCESS
> >> 	tristate "InfiniBand userspace access (verbs and CM)"
> >> 	select ANON_INODES
> >> 	---help---
> >> 	  Userspace InfiniBand access support.  This enables the
> >> 	  kernel side of userspace verbs and the userspace
> >> 	  communication manager (CM).  This allows userspace processes
> >> 	  to set up connections and directly access InfiniBand
> >> 	  hardware for fast-path operations.  You will also need
> >> 	  libibverbs, libibcm and a hardware driver library from
> >> 	  <http://www.openfabrics.org/git/>.
> >> "
> >>
> >> It seems to be close to the bifurcated driver needs.
> >> Not sure if it can solve the security issues if there is no dedicated MMU
> >> in the NIC.
> >>
> >> I feel we should sum up pros and cons of
> >> 	- igb_uio
> >> 	- uio_pci_generic
> >> 	- VFIO
> >> 	- ibverbs
> >> 	- bifurcated driver
> >> I suggest to consider these criterias:
> >> 	- upstream status
> >> 	- usable with kernel netdev
> >> 	- usable in a vm
> >> 	- usable for ethernet
> >> 	- hardware requirements
> >> 	- security protection
> >> 	- performance
> >>
> >> --
> >> Thomas

Re: [dpdk-dev] bifurcated driver

[Alex Markuze]

Danny sums up the issue perfectly IMHO.
While both verbs and DPDK aim to provide generic user space networking, the
similarities end there.
verbs and RDMA HW are closely coupled and behave differently then standard
eth nics and are not related to netdev mechanisms.

Or, welcome to this discussion.

Those interested can read the IB spec's (+1K pages) available from
openfabrics*.
*https://www.openfabrics.org/index.php




On Thu, Nov 6, 2014 at 6:45 AM, Zhou, Danny <danny.zhou@intel.com> wrote:

> I roughly read libibverbs related code and relevant infiniband/rdma
> documents, and found though
> many concepts in libibverbs looks similar to bifurcated driver, but there
> are still lots of differences as
> illustrated below based on my understanding:
>
> 1) Queue pair defined in RDMA specification are abstract concept, where
> the queue pairs term used in
>   bifurcated driver are rx/tx queue pairs in the NIC.
> 2) Bifurcated PMD in DPDK directly access NIC resources as a slave driver
> (no NIC control), while libibverbs
>   as a user space library rather than driver offloads certain operations
> to kernel driver and NIC by invoking
>   "verbs" APIs.
> 3) Libibverbs invokes infiniband specific system calls to allow
> user/kernel space communication based on
>   "verbs" defined in infiniband/RDMA spec, while bifurcated driver build
> on top of af_packet module
>   and new socket options to do things like hw queue split-off , map
> certain pages on I/O space to user space
>   operations, etc.
> 4) There is a specific embedded MMU unit in Infiniband/RDMA to provides
> memory protection, while
>   bifurcated driver uses IOMMU rather than NIC to provide memory
> protection.
>
> IMHO, libibverbs and corresponding kernel modules/drivers are specifically
> designed and implemented for
> direct access to RDMA hardware from userspace, and it highly depends on
> "verbs" related system calls
> supported by infiniband/rdma mechanism in kernel, rather than netdev
> mechanism that bifurcated driver
> solution depends on.
>
> > -----Original Message-----
> > From: Vincent JARDIN [mailto:vincent.jardin@6wind.com]
> > Sent: Thursday, November 06, 2014 9:31 AM
> > To: Zhou, Danny
> > Cc: Thomas Monjalon; dev@dpdk.org; Fastabend, John R; Or Gerlitz
> > Subject: Re: [dpdk-dev] bifurcated driver
> >
> > +Or
> >
> > On 05/11/2014 23:48, Zhou, Danny wrote:
> > > Hi Thomas,
> > >
> > > Thanks for sharing the links to ibverbs, I will take a close look at
> it and compare it to bifurcated driver. My take
> > > after a rough review is that idea is very much similar, but bifurcated
> driver implementation is generic for any
> > > Ethernet device based on existing af_packet mechanism, with extension
> of exchanging the messages between
> > > user space and kernel space driver.
> > >
> > > I have an internal document to summary the pros and cons of below
> solutions, except for ibvers, but
> > > will be adding it shortly.
> > >
> > > - igb_uio
> > > - uio_pci_generic
> > > - VFIO
> > > - bifurcated driver
> > >
> > > Short answers to your questions:
> > >>    - upstream status
> > > Adding IOMMU based memory protection and generic descriptor
> description support now, into version 2
> > > kernel patches.
> > >
> > >>    - usable with kernel netdev
> > > af_packet based, and relevant patchset will be submitted to netdev for
> sure.
> > >
> > >>    - usable in a vm
> > > No, it does no coexist with SRIOV for number of reasons. but if you
> pass-through a PF to a VM, it works perfect.
> > >
> > >>    - usable for Ethernet
> > > It could work with all Ethernet NICs, as flow director is available
> and NIC driver support new net_ops to split off
> > > queue pairs for user space.
> > >
> > >>    - hardware requirements
> > > No specific hardware requirements. All mainstream NICs have multiple
> qpairs and flow director support.
> > >
> > >>    - security protection
> > > Leverage IOMMU to provide memory protection on Intel platform. Other
> archs provide similar memory protection
> > > mechanism, so we only use arch-agnostic DMA memory allocation APIs in
> kernel to support memory protection.
> > >
> > >>    - performance
> > > DPDK native performance on user space queues, as long as drop_en is
> enabled to avoid head-of-line blocking.
> > >
> > > -Danny
> > >
> > >> -----Original Message-----
> > >> From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
> > >> Sent: Wednesday, November 05, 2014 9:01 PM
> > >> To: Zhou, Danny
> > >> Cc: dev@dpdk.org; Fastabend, John R
> > >> Subject: Re: [dpdk-dev] bifurcated driver
> > >>
> > >> Hi Danny,
> > >>
> > >> 2014-10-31 17:36, O'driscoll, Tim:
> > >>> Bifurcated Driver (Danny.Zhou@intel.com)
> > >>
> > >> Thanks for the presentation of bifurcated driver during the community
> call.
> > >> I asked if you looked at ibverbs and you wanted a link to check.
> > >> The kernel module is here:
> > >>
> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/drivers/infiniband/core
> > >> The userspace library:
> > >>    http://git.kernel.org/cgit/libs/infiniband/libibverbs.git
> > >>
> > >> Extract from Kconfig:
> > >> "
> > >> config INFINIBAND_USER_ACCESS
> > >>    tristate "InfiniBand userspace access (verbs and CM)"
> > >>    select ANON_INODES
> > >>    ---help---
> > >>      Userspace InfiniBand access support.  This enables the
> > >>      kernel side of userspace verbs and the userspace
> > >>      communication manager (CM).  This allows userspace processes
> > >>      to set up connections and directly access InfiniBand
> > >>      hardware for fast-path operations.  You will also need
> > >>      libibverbs, libibcm and a hardware driver library from
> > >>      <http://www.openfabrics.org/git/>.
> > >> "
> > >>
> > >> It seems to be close to the bifurcated driver needs.
> > >> Not sure if it can solve the security issues if there is no dedicated
> MMU
> > >> in the NIC.
> > >>
> > >> I feel we should sum up pros and cons of
> > >>    - igb_uio
> > >>    - uio_pci_generic
> > >>    - VFIO
> > >>    - ibverbs
> > >>    - bifurcated driver
> > >> I suggest to consider these criterias:
> > >>    - upstream status
> > >>    - usable with kernel netdev
> > >>    - usable in a vm
> > >>    - usable for ethernet
> > >>    - hardware requirements
> > >>    - security protection
> > >>    - performance
> > >>
> > >> --
> > >> Thomas
>
>

Re: [dpdk-dev] bifurcated driver

[Nicolas Dichtel]

Also CC netdev, this thread may interest network folks.

Le 06/11/2014 09:13, Alex Markuze a écrit :
> Danny sums up the issue perfectly IMHO.
> While both verbs and DPDK aim to provide generic user space networking, the
> similarities end there.
> verbs and RDMA HW are closely coupled and behave differently then standard
> eth nics and are not related to netdev mechanisms.
>
> Or, welcome to this discussion.
>
> Those interested can read the IB spec's (+1K pages) available from
> openfabrics*.
> *https://www.openfabrics.org/index.php
>
>
>
>
> On Thu, Nov 6, 2014 at 6:45 AM, Zhou, Danny <danny.zhou@intel.com> wrote:
>
>> I roughly read libibverbs related code and relevant infiniband/rdma
>> documents, and found though
>> many concepts in libibverbs looks similar to bifurcated driver, but there
>> are still lots of differences as
>> illustrated below based on my understanding:
>>
>> 1) Queue pair defined in RDMA specification are abstract concept, where
>> the queue pairs term used in
>>    bifurcated driver are rx/tx queue pairs in the NIC.
>> 2) Bifurcated PMD in DPDK directly access NIC resources as a slave driver
>> (no NIC control), while libibverbs
>>    as a user space library rather than driver offloads certain operations
>> to kernel driver and NIC by invoking
>>    "verbs" APIs.
>> 3) Libibverbs invokes infiniband specific system calls to allow
>> user/kernel space communication based on
>>    "verbs" defined in infiniband/RDMA spec, while bifurcated driver build
>> on top of af_packet module
>>    and new socket options to do things like hw queue split-off , map
>> certain pages on I/O space to user space
>>    operations, etc.
>> 4) There is a specific embedded MMU unit in Infiniband/RDMA to provides
>> memory protection, while
>>    bifurcated driver uses IOMMU rather than NIC to provide memory
>> protection.
>>
>> IMHO, libibverbs and corresponding kernel modules/drivers are specifically
>> designed and implemented for
>> direct access to RDMA hardware from userspace, and it highly depends on
>> "verbs" related system calls
>> supported by infiniband/rdma mechanism in kernel, rather than netdev
>> mechanism that bifurcated driver
>> solution depends on.
>>
>>> -----Original Message-----
>>> From: Vincent JARDIN [mailto:vincent.jardin@6wind.com]
>>> Sent: Thursday, November 06, 2014 9:31 AM
>>> To: Zhou, Danny
>>> Cc: Thomas Monjalon; dev@dpdk.org; Fastabend, John R; Or Gerlitz
>>> Subject: Re: [dpdk-dev] bifurcated driver
>>>
>>> +Or
>>>
>>> On 05/11/2014 23:48, Zhou, Danny wrote:
>>>> Hi Thomas,
>>>>
>>>> Thanks for sharing the links to ibverbs, I will take a close look at
>> it and compare it to bifurcated driver. My take
>>>> after a rough review is that idea is very much similar, but bifurcated
>> driver implementation is generic for any
>>>> Ethernet device based on existing af_packet mechanism, with extension
>> of exchanging the messages between
>>>> user space and kernel space driver.
>>>>
>>>> I have an internal document to summary the pros and cons of below
>> solutions, except for ibvers, but
>>>> will be adding it shortly.
>>>>
>>>> - igb_uio
>>>> - uio_pci_generic
>>>> - VFIO
>>>> - bifurcated driver
>>>>
>>>> Short answers to your questions:
>>>>>     - upstream status
>>>> Adding IOMMU based memory protection and generic descriptor
>> description support now, into version 2
>>>> kernel patches.
>>>>
>>>>>     - usable with kernel netdev
>>>> af_packet based, and relevant patchset will be submitted to netdev for
>> sure.
>>>>
>>>>>     - usable in a vm
>>>> No, it does no coexist with SRIOV for number of reasons. but if you
>> pass-through a PF to a VM, it works perfect.
>>>>
>>>>>     - usable for Ethernet
>>>> It could work with all Ethernet NICs, as flow director is available
>> and NIC driver support new net_ops to split off
>>>> queue pairs for user space.
>>>>
>>>>>     - hardware requirements
>>>> No specific hardware requirements. All mainstream NICs have multiple
>> qpairs and flow director support.
>>>>
>>>>>     - security protection
>>>> Leverage IOMMU to provide memory protection on Intel platform. Other
>> archs provide similar memory protection
>>>> mechanism, so we only use arch-agnostic DMA memory allocation APIs in
>> kernel to support memory protection.
>>>>
>>>>>     - performance
>>>> DPDK native performance on user space queues, as long as drop_en is
>> enabled to avoid head-of-line blocking.
>>>>
>>>> -Danny
>>>>
>>>>> -----Original Message-----
>>>>> From: Thomas Monjalon [mailto:thomas.monjalon@6wind.com]
>>>>> Sent: Wednesday, November 05, 2014 9:01 PM
>>>>> To: Zhou, Danny
>>>>> Cc: dev@dpdk.org; Fastabend, John R
>>>>> Subject: Re: [dpdk-dev] bifurcated driver
>>>>>
>>>>> Hi Danny,
>>>>>
>>>>> 2014-10-31 17:36, O'driscoll, Tim:
>>>>>> Bifurcated Driver (Danny.Zhou@intel.com)
>>>>>
>>>>> Thanks for the presentation of bifurcated driver during the community
>> call.
>>>>> I asked if you looked at ibverbs and you wanted a link to check.
>>>>> The kernel module is here:
>>>>>
>> http://git.kernel.org/cgit/linux/kernel/git/torvalds/linux.git/tree/drivers/infiniband/core
>>>>> The userspace library:
>>>>>     http://git.kernel.org/cgit/libs/infiniband/libibverbs.git
>>>>>
>>>>> Extract from Kconfig:
>>>>> "
>>>>> config INFINIBAND_USER_ACCESS
>>>>>     tristate "InfiniBand userspace access (verbs and CM)"
>>>>>     select ANON_INODES
>>>>>     ---help---
>>>>>       Userspace InfiniBand access support.  This enables the
>>>>>       kernel side of userspace verbs and the userspace
>>>>>       communication manager (CM).  This allows userspace processes
>>>>>       to set up connections and directly access InfiniBand
>>>>>       hardware for fast-path operations.  You will also need
>>>>>       libibverbs, libibcm and a hardware driver library from
>>>>>       <http://www.openfabrics.org/git/>.
>>>>> "
>>>>>
>>>>> It seems to be close to the bifurcated driver needs.
>>>>> Not sure if it can solve the security issues if there is no dedicated
>> MMU
>>>>> in the NIC.
>>>>>
>>>>> I feel we should sum up pros and cons of
>>>>>     - igb_uio
>>>>>     - uio_pci_generic
>>>>>     - VFIO
>>>>>     - ibverbs
>>>>>     - bifurcated driver
>>>>> I suggest to consider these criterias:
>>>>>     - upstream status
>>>>>     - usable with kernel netdev
>>>>>     - usable in a vm
>>>>>     - usable for ethernet
>>>>>     - hardware requirements
>>>>>     - security protection
>>>>>     - performance
>>>>>
>>>>> --
>>>>> Thomas
>>
>>

Re: [dpdk-dev] bifurcated driver

[Luke Gorrie]

On 5 November 2014 at 14:00, Thomas Monjalon <thomas.monjalon@6wind.com>
wrote:

> It seems to be close to the bifurcated driver needs.
> Not sure if it can solve the security issues if there is no dedicated MMU
> in the NIC.
>
> I feel we should sum up pros and cons of
>         - igb_uio
>         - uio_pci_generic
>         - VFIO
>         - ibverbs
>         - bifurcated driver
>

I am also curious about the pros and cons of the bifurcated driver compared
with SR-IOV.

What are the practical differences between running a bifurcated driver vs.
running SR-IOV mode where the kernel owns the PF and userspace applications
own the VFs?

Specifically, could I run the ixgbe driver in the kernel (max_vfs=N),
control it via ethtool, and then access the queues via userspace VF
drivers? If so, how would this differ from the bifurcated driver?

Cheers,
-Luke

Re: [dpdk-dev] bifurcated driver

[Zhou, Danny]

Hope summary below could answer your questions.

From: lukego@gmail.com [mailto:lukego@gmail.com] On Behalf Of Luke Gorrie
Sent: Monday, November 24, 2014 7:58 PM
To: Thomas Monjalon
Cc: Zhou, Danny; dev@dpdk.org; Fastabend, John R
Subject: Re: [dpdk-dev] bifurcated driver

On 5 November 2014 at 14:00, Thomas Monjalon <thomas.monjalon@6wind.com<mailto:thomas.monjalon@6wind.com>> wrote:
It seems to be close to the bifurcated driver needs.
Not sure if it can solve the security issues if there is no dedicated MMU
in the NIC.

I feel we should sum up pros and cons of
        - igb_uio
        - uio_pci_generic
        - VFIO
        - ibverbs
        - bifurcated driver

I am also curious about the pros and cons of the bifurcated driver compared with SR-IOV.
DZ: I have a slide to compare all of them except for ibverbs, but system does not allows me to paste a captured picture. I can send it to you in a separated email.

What are the practical differences between running a bifurcated driver vs. running SR-IOV mode where the kernel owns the PF and userspace applications own the VFs?
DZ: SRIOV can be treated as a rx/tx queue partition approach following PCIe spec, but it diffs from bifurcated driver from as illustrated below:

a)      On ixgbe, each VF can only have at maximum two rx/tx qpairs, while bifurcated driver supports allocating multiple rx/tx qpairs per user space request.

b)      Each PF/VF has a dedicated MAC address and/or VLAN ID, and L2 switch inside a NIC distributes packets to PF/VF based on MAC address or VLAN ID. While bifurcated driver

builds on top of in_NIC flow director which allows up-to 8K L3/L4 filters (e.g. 5-tuple to rx queue), rather than fixed L2 filter, to distribute traffics to either kernel or user space.

c)       VF does not have dedicated flow director.

Specifically, could I run the ixgbe driver in the kernel (max_vfs=N), control it via ethtool, and then access the queues via userspace VF drivers? If so, how would this differ from the bifurcated driver?
DZ: Yes you can do this. But L3/L4 filters you setup to PF (VF does not support it) via ethtool to flow director takes no effect if the packet’s DMAC matches a VF’ MAC, as the packet will be distributed by L2 switch to the VF that rx queue belongs to. Ixgbe does not support per-VF flow director that allows you setup filters distributing packet to two rx queue inside a VF.

Cheers,
-Luke

Re: [dpdk-dev] DPDK Community Conference Call - Friday 31st October

[Kevin Wilson]

Hi, Tim,

> We'll schedule a follow-up call for 2 weeks' time
Just a short question - is this still intended to be held ?

Kevin

On Fri, Oct 31, 2014 at 7:36 PM, O'driscoll, Tim
<tim.o'driscoll@intel.com> wrote:
> Thanks again to those who attended the call earlier. Hopefully people found it useful.
>
> We'll schedule a follow-up call for 2 weeks' time. One thing that we do want to look into is an easy way to allow screen sharing, so that we can use some slides to guide the discussion. Internally within Intel we use MS Lync. We can try that, but it's not always very user-friendly for external participants, and doesn't have a Linux client. Other options would include GoToMeeting or WebEx. If anybody has input on a good tool for this, let me know.
>
> We covered the following features from our 2.0 list today, and will discuss the remainder on the next call. I've called out below who on our side was describing each of the features, and included their email addresses. If anybody has further questions on these, feel free to either ask openly on the mailing list, or else contact the relevant person directly.
>
> Bifurcated Driver (Danny.Zhou@intel.com)
> Packet Reordering/Packet Distributor (Bruce.Richardson@intel.com)
> New Hardware Support (Walter.E.Gilmore@intel.com)
> Fortville features (Heqing.Zhu@intel.com)
> Support Multiple Threads per Core (Venky.Venkatesan@intel.com)
> Cuckoo Hash (Bruce.Richardson@intel.com, Venky.Venkatesan@intel.com)
>
> The Cuckoo Hash paper that was mentioned is available at: http://www.cs.cmu.edu/~dongz/papers/cuckooswitch.pdf.
>
> Finally, if anybody has suggestions for topics for future calls, please let me know.
>
>
> Thanks,
> Tim
>
>> -----Original Message-----
>> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of O'driscoll, Tim
>> Sent: Friday, October 31, 2014 3:35 PM
>> To: 'dev@dpdk.org'
>> Subject: Re: [dpdk-dev] DPDK Community Conference Call - Friday 31st
>> October
>>
>> This is just a reminder for anybody who's interested that this will be on in 30
>> minutes, and that we'll be discussing the feature list for the DPDK 2.0 release
>> in March 2015.
>>
>> Audio bridge details are:
>> France:               +33 1588 77298
>> Germany:      +49 8999 143191
>> Israel:               +972 2589 6577
>> Russia:               +7 495 641 4663
>> UK:           +44 1793 402663
>> USA/Canada:   +1 916 356 2663 or +1-888-875-9370
>>
>> Bridge: 5
>> Conference ID: 1264677285
>>
>>
>> Tim
>>
>> > -----Original Message-----
>> > From: O'driscoll, Tim
>> > Sent: Friday, October 24, 2014 10:22 AM
>> > To: dev@dpdk.org
>> > Subject: DPDK Community Conference Call - Friday 31st October
>> >
>> > We're planning to hold our first community conference call on Friday
>> > 31st October. It's impossible to find a time that suits everybody, so
>> > we've chosen to do this in the afternoon/evening in Europe, which is
>> > the morning in the USA. This does unfortunately limit participation
>> > from PRC, Japan and other parts of the world. Here's the time and date in a
>> variety of time zones:
>> >
>> > Dublin (Ireland)                    Friday, October 31, 2014 at
>> > 4:00:00 PM    GMT UTC
>> > Paris (France)                              Friday, October 31, 2014 at 5:00:00
>> > PM    CET UTC+1 hour
>> > San Francisco (U.S.A. - California) Friday, October 31, 2014 at 9:00:00
>> > AM    PDT UTC-7 hours
>> > New York (U.S.A. - New York)                Friday, October 31, 2014 at
>> 12:00:00
>> > Noon EDT UTC-4 hours
>> > Tel Aviv (Israel)                           Friday, October 31, 2014 at
>> 6:00:00
>> > PM    IST UTC+2 hours
>> > Moscow (Russia)                     Friday, October 31, 2014 at 7:00:00
>> > PM    MSK UTC+3 hours
>> >
>> >
>> > Audio bridge details are:
>> > France:             +33 1588 77298
>> > Germany:    +49 8999 143191
>> > Israel:             +972 2589 6577
>> > Russia:             +7 495 641 4663
>> > UK:         +44 1793 402663
>> > USA:                +1 916 356 2663
>> >
>> > Bridge: 5
>> > Conference ID: 1264677285
>> >
>> > If anybody needs an access number for another country, let me know.
>> >
>> >
>> > Agenda:
>> > Discuss feature list for DPDK 2.0 (Q1 2015).
>> > Suggestions for topics for future calls.
>> >
>> >
>> > Thanks,
>> > Tim

Re: [dpdk-dev] DPDK Community Conference Call - Friday 31st October

[O'driscoll, Tim]

Hi Kevin,

> From: Kevin Wilson [mailto:wkevils@gmail.com]
> > We'll schedule a follow-up call for 2 weeks' time
> Just a short question - is this still intended to be held ?

We had our second call earlier this week, on Tuesday. I'll post a recording of it soon.

The next call will be in 2 weeks' time, probably on Tuesday December 2nd. I just need to finalise the time before confirming this. We have had a couple of requests to alternate between a time that's suitable for USA/Europe, and one that's more suitable for Asia. So, the next call will probably be early in the morning in Europe, afternoon in Asia, and the middle of the night in the USA.


Tim

Re: [dpdk-dev] DPDK Community Conference Call - Friday 31st October

[Kevin Wilson]

Hi,
>I'll post a recording of it soon.
Great idea! most welcomed!

Kevin


On Thu, Nov 20, 2014 at 3:13 PM, O'driscoll, Tim
<tim.o'driscoll@intel.com> wrote:
> Hi Kevin,
>
>> From: Kevin Wilson [mailto:wkevils@gmail.com]
>> > We'll schedule a follow-up call for 2 weeks' time
>> Just a short question - is this still intended to be held ?
>
> We had our second call earlier this week, on Tuesday. I'll post a recording of it soon.
>
> The next call will be in 2 weeks' time, probably on Tuesday December 2nd. I just need to finalise the time before confirming this. We have had a couple of requests to alternate between a time that's suitable for USA/Europe, and one that's more suitable for Asia. So, the next call will probably be early in the morning in Europe, afternoon in Asia, and the middle of the night in the USA.
>
>
> Tim

Re: [dpdk-dev] DPDK Community Conference Call - Friday 31st October

[O'driscoll, Tim]

The video is now accessible at: http://youtu.be/AbHQ4YaWY90. Thomas may want to add a link to this from somewhere on the dpdk.org page.


Tim

> From: Kevin Wilson [mailto:wkevils@gmail.com]
> 
> Hi,
> >I'll post a recording of it soon.
> Great idea! most welcomed!
> 
> Kevin
> 
> 
> On Thu, Nov 20, 2014 at 3:13 PM, O'driscoll, Tim <tim.o'driscoll@intel.com>
> wrote:
> > Hi Kevin,
> >
> >> From: Kevin Wilson [mailto:wkevils@gmail.com]
> >> > We'll schedule a follow-up call for 2 weeks' time
> >> Just a short question - is this still intended to be held ?
> >
> > We had our second call earlier this week, on Tuesday. I'll post a recording of
> it soon.
> >
> > The next call will be in 2 weeks' time, probably on Tuesday December 2nd. I
> just need to finalise the time before confirming this. We have had a couple of
> requests to alternate between a time that's suitable for USA/Europe, and
> one that's more suitable for Asia. So, the next call will probably be early in the
> morning in Europe, afternoon in Asia, and the middle of the night in the USA.
> >
> >
> > Tim

Re: [dpdk-dev] DPDK Community Conference Call - Friday 31st October

[Kevin Wilson]

Tim,
Great, thanks! Keep on the good work and recording in the future.

The quality of the Audio is superb!

Kevin

On Fri, Nov 21, 2014 at 1:26 AM, O'driscoll, Tim
<tim.o'driscoll@intel.com> wrote:
> The video is now accessible at: http://youtu.be/AbHQ4YaWY90. Thomas may want to add a link to this from somewhere on the dpdk.org page.
>
>
> Tim
>
>> From: Kevin Wilson [mailto:wkevils@gmail.com]
>>
>> Hi,
>> >I'll post a recording of it soon.
>> Great idea! most welcomed!
>>
>> Kevin
>>
>>
>> On Thu, Nov 20, 2014 at 3:13 PM, O'driscoll, Tim <tim.o'driscoll@intel.com>
>> wrote:
>> > Hi Kevin,
>> >
>> >> From: Kevin Wilson [mailto:wkevils@gmail.com]
>> >> > We'll schedule a follow-up call for 2 weeks' time
>> >> Just a short question - is this still intended to be held ?
>> >
>> > We had our second call earlier this week, on Tuesday. I'll post a recording of
>> it soon.
>> >
>> > The next call will be in 2 weeks' time, probably on Tuesday December 2nd. I
>> just need to finalise the time before confirming this. We have had a couple of
>> requests to alternate between a time that's suitable for USA/Europe, and
>> one that's more suitable for Asia. So, the next call will probably be early in the
>> morning in Europe, afternoon in Asia, and the middle of the night in the USA.
>> >
>> >
>> > Tim