From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga01.intel.com (mga01.intel.com [192.55.52.88]) by dpdk.org (Postfix) with ESMTP id 5FD0B5A35 for ; Fri, 8 May 2015 06:16:51 +0200 (CEST) Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga101.fm.intel.com with ESMTP; 07 May 2015 21:16:50 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.13,387,1427785200"; d="scan'208";a="707091622" Received: from orsmsx109.amr.corp.intel.com ([10.22.240.7]) by fmsmga001.fm.intel.com with ESMTP; 07 May 2015 21:16:50 -0700 Received: from fmsmsx117.amr.corp.intel.com (10.18.116.17) by ORSMSX109.amr.corp.intel.com (10.22.240.7) with Microsoft SMTP Server (TLS) id 14.3.224.2; Thu, 7 May 2015 21:16:49 -0700 Received: from fmsmsx113.amr.corp.intel.com ([169.254.13.26]) by fmsmsx117.amr.corp.intel.com ([169.254.3.51]) with mapi id 14.03.0224.002; Thu, 7 May 2015 21:16:49 -0700 From: "Wiles, Keith" To: Avi Kivity , "O'Driscoll, Tim" Thread-Topic: [dpdk-dev] Beyond DPDK 2.0 Thread-Index: AdB4KSdm1ftzhuzCQX2G2XNq9ZOWFAE5VxLQAv6mNgD//6KBAIAAdwKA//+O+YCAAHnoAIAAVvuA Date: Fri, 8 May 2015 04:16:48 +0000 Message-ID: References: <26FA93C7ED1EAA44AB77D62FBE1D27BA54D1A917@IRSMSX102.ger.corp.intel.com> <26FA93C7ED1EAA44AB77D62FBE1D27BA54D29B55@IRSMSX102.ger.corp.intel.com> <554B85D5.6010808@cloudius-systems.com> <554B8D48.7010900@cloudius-systems.com> In-Reply-To: <554B8D48.7010900@cloudius-systems.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.254.101.227] Content-Type: text/plain; charset="iso-8859-2" Content-ID: Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Cc: "dev@dpdk.org" Subject: Re: [dpdk-dev] Beyond DPDK 2.0 X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches and discussions about DPDK List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 08 May 2015 04:16:52 -0000 On 5/7/15, 9:05 AM, "Avi Kivity" wrote: >On 05/07/2015 06:49 PM, Wiles, Keith wrote: >> >> On 5/7/15, 8:33 AM, "Avi Kivity" wrote: >> >>> On 05/07/2015 06:27 PM, Wiles, Keith wrote: >>>> On 5/7/15, 7:02 AM, "Avi Kivity" wrote: >>>> >>>>> On Wed, Apr 22, 2015 at 6:11 PM, O'Driscoll, Tim >>>>> >>>>> wrote: >>>>> >>>>>> Does anybody have any input or comments on this? >>>>>> >>>>>> >>>>>>> -----Original Message----- >>>>>>> From: O'Driscoll, Tim >>>>>>> Sent: Thursday, April 16, 2015 11:39 AM >>>>>>> To: dev@dpdk.org >>>>>>> Subject: Beyond DPDK 2.0 >>>>>>> >>>>>>> Following the launch of DPDK by Intel as an internal development >>>>>>> project, the launch of dpdk.org by 6WIND in 2013, and the first >>>>>>>DPDK >>>>>> RPM >>>>>>> packages for Fedora in 2014, 6WIND, Red Hat and Intel would like to >>>>>>> prepare for future releases after DPDK 2.0 by starting a discussion >>>>>>> on >>>>>>> its evolution. Anyone is welcome to join this initiative. >>>>>>> >>>>>>> Since then, the project has grown significantly: >>>>>>> - The number of commits and mailing list posts has increased >>>>>>> steadily. >>>>>>> - Support has been added for a wide range of new NICs (Mellanox >>>>>>> support submitted by 6WIND, Cisco VIC, Intel i40e and fm10k etc.). >>>>>>> - DPDK is now supported on multiple architectures (IBM Power >>>>>> support >>>>>>> in DPDK 1.8, Tile support submitted by EZchip but not yet reviewed >>>>>>>or >>>>>>> applied). >>>>>>> >>>>>>> While this is great progress, we need to make sure that the project >>>>>>> is >>>>>>> structured in a way that enables it to continue to grow. To achieve >>>>>>> this, 6WIND, Red Hat and Intel would like to start a discussion >>>>>>>about >>>>>>> the future of the project, so that we can agree and establish >>>>>> processes >>>>>>> that satisfy the needs of the current and future DPDK community. >>>>>>> >>>>>>> We're very interested in hearing the views of everybody in the >>>>>>> community. In addition to debate on the mailing list, we'll also >>>>>>> schedule community calls to discuss this. >>>>>>> >>>>>>> >>>>>>> Project Goals >>>>>>> ------------- >>>>>>> >>>>>>> Some topics to be considered for the DPDK project include: >>>>>>> - Project Charter: The charter of the DPDK project should be >>>>>> clearly >>>>>>> defined, and should explain the limits of DPDK (what it does and >>>>>>>does >>>>>>> not cover). This does not mean that we would be stuck with a >>>>>>>singular >>>>>>> charter for all time, but the direction and intent of the project >>>>>> should >>>>>>> be well understood. >>>>> One problem we've seen with dpdk is that it is a framework, not a >>>>> library: >>>>> it wants to create threads, manage memory, and generally take over. >>>>> This >>>>> is a problem for us, as we are writing a framework (seastar, [1]) and >>>>> need >>>>> to create threads, manage memory, and generally take over ourselves. >>>>> >>>>> Perhaps dpdk can be split into two layers, a library layer that only >>>>> provides mechanisms, and a framework layer that glues together those >>>>> mechanisms and applies a policy, trading in generality for ease of >>>>>use. >>>> The DPDK system is somewhat divided now between the EAL, PMDS and >>>> utility >>>> functions like malloc/rings/=A9 >>>> >>>> The problem I see is the PMDs need a framework to be usable and the >>>>EAL >>>> plus the ethdev layers provide that support today. Setting up and >>>> initializing the DPDK system is pretty clean just call the EAL init >>>> routines along with the pool creates and the basic configs for the >>>> PMDs/hardware. Once the system is inited one can create new threads >>>>and >>>> not requiring anyone to use DPDK launch routines. Maybe I am not >>>> understanding your needs can you explain more? >>> An initialization routine that accepts argc/argv can hardly be called >>> clean. >> You want a config file or structure initialization design? If that is >>the >> case you can contribute that support as another way to initialize DPDK. > >A config file would be even worse. But we are discussing why >dpdk-as-a-framework is detrimental, not new ways for me to contribute. In a way you stated argc/argv was not a clean, I was only suggesting (more I was asking) what you would like to see? The contribute part was just an example of how you or anyone can help make DPDK better. I wanted to understand why argc/argv was not a clan way for your needs. > >>> In seastar, we have our own malloc() (since seastar is sharded we can >>> provide a faster thread-unsafe malloc implementation). We also have >>>our >>> own threading, and since dpdk is an optional component in seastar, dpdk >>> support requires code duplication. >> DPDK replies one the huge page support for allocation to get the >> performance, do you also not require huge page support. > >Sorry, is this a question? Please rephrase. Sorry, auto correct got me and trying to answer quickly before a meeting. DPDK uses huge pages to get the best performance and is tied into the MBUFS and memory allocation. You can turn off huge page support in DPDK, but your performance will drop. The MBUF support is also a very critical performance designed structure and is used by the PMDs along with just about everything else in DPDK. The PMDs or drivers would not be useful without DPDK MBUFS IMO and huge page support. The huge page support can be written by someone, but the allocation routines for MBUFS/Pools is designed to give the best performance for DPDK, which makes PMDs and MBUFS required IMO. Trying to split DPDK up into just library parts you can use independently is going to be hard IMHO. Like any complex system the functions/features are required to be used together to get the best performance. Can you help me understand which parts of DPDK you would like to split out and use? > >> The malloc system >> in DPDK can be used as a replacement for the standard malloc if that >>works >> for your needs. Also after DPDK inits you can use your own malloc and >>any >> other tools you want to use. > >How is memory partitioned between dpdk and my application? If I >underallocate dpdk memory, something bad will happen. If I overallocate >dpdk memory, then I am depriving my application of this memory. A >common pool means I do not overallocate or underallocate, but since dpdk >insists on managing its own pools, I can't do this. I believe you can define 'malloc' as rte_malloc and use only DPDK memory allocation for your application, but it does take some extra steps to get it working. At least that is what I was told today and if you want to investigate doing something like that let me know and I will get more details. I do not know your application and can not say if that type of change would help. Using only one type of memory allocation or heap is the right way to go as you stated. > >> I do not see a lot of duplicate code here >> IMO. I guess if you are installing into a very small memory system then >> yes it could be a problem, but DPDK is was not designed to run in a >>system >> with limited memory. > >I am not talking about duplicate code, but about duplicate >functionality, done slightly differently. I want to use dpdk in the >same way as I use every other library, by calling its initialization >routine and then calling its functions. In this scenario, the library >is passive, only reacting to my calls. The way dpdk works now is >actively, taking over resources, creating thread, and calling my code >instead of the other way round. Sounds like you want something like libc, but DPDK is a system like a user space OS more then it is a collection of functions that are independent like strlen, strcpy, memcpy, printf or ... Some parts of DPDK are independent and can be used as you suggest, but the real performance sections are tied together. I would like to understand what parts of DPDK you need to be standalone or different. Maybe if you give some examples we could possible figure out how to make DPDK more reasonable for your projects. > >> >>> I would like to launch my own threads, pin them where I like, and call >>> PMD drivers to send and receive packets. Practically everything else >>> that dpdk does gets in my way, including mbuf pools. I'd much prefer >>>to >>> allocate mbufs myself. >> You do not need to use the lauching of threads in the EAL and can supply >> your own, right? > >Right. > >> Regards, >> ++Keith >>> >>>>> [1] http://seastar-project.org >