From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from out1-smtp.messagingengine.com (out1-smtp.messagingengine.com [66.111.4.25]) by dpdk.org (Postfix) with ESMTP id 84C9C1B88C for ; Mon, 17 Dec 2018 17:12:09 +0100 (CET) Received: from compute1.internal (compute1.nyi.internal [10.202.2.41]) by mailout.nyi.internal (Postfix) with ESMTP id 13A8F223DC; Mon, 17 Dec 2018 11:12:09 -0500 (EST) Received: from mailfrontend1 ([10.202.2.162]) by compute1.internal (MEProxy); Mon, 17 Dec 2018 11:12:09 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=monjalon.net; h= from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding:content-type; s=mesmtp; bh=4ZjCAzFdBdYgRN/cU9XExr95K05ng2uLvSARiZ/Xko0=; b=RCSoGPpYa6k7 afdpyAx+ERie701xHbw0iAPRG3fV/SSCqrt/+XGYHsYxh5TvzT4S2CtM6sv+ZAGV 6EnOLgkivG6+ZR5uY4Le3ZL185hog1QZncyGcg//PNzVyqa4Ylk7/qi/2Yw4BRMk THlOo8ft8BDaV7t+zOfJTdKK9Zn1myA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d= messagingengine.com; h=cc:content-transfer-encoding:content-type :date:from:in-reply-to:message-id:mime-version:references :subject:to:x-me-proxy:x-me-proxy:x-me-sender:x-me-sender :x-sasl-enc; s=fm1; bh=4ZjCAzFdBdYgRN/cU9XExr95K05ng2uLvSARiZ/Xk o0=; b=Zr6ccC3YfJTEHY3QUDn0jlUDbEt/Qpkil0WOdS9mFZxbgb5BjE1ctR/vK BpHFT2Q2LbPDGTXdNWutIfeG73M8i2KrHD5RZCYXb/mrEf6zFW1vfxFrK9n/rScP g1TuoXIYR12hzgwdcuJRCT9bjpR2w06o67g3A9jx1MRPGM+0+lcLJsBVyUs+0pwm aZMFReWn8cyCDkHGglFFPPpJZuFU4F7zGUaxkd3Hl4lqUEaEmGZrsQYn2ppGMjze 0xXpPiOJPHYFyuZU2wEbe18jMNm5/O1uQEiuGE9bmRpRN5vXea3hkjmjG2BhXzBR YhNHaVR36T1RpED+Z9+cbYkXbj/HA== X-ME-Sender: X-ME-Proxy-Cause: gggruggvucftvghtrhhoucdtuddrgedtkedrudeivddgkeeiucetufdoteggodetrfdotf fvucfrrhhofhhilhgvmecuhfgrshhtofgrihhlpdfquhhtnecuuegrihhlohhuthemucef tddtnecusecvtfgvtghiphhivghnthhsucdlqddutddtmdenucfjughrpefhvffufffkjg hfggfgtgesthhqredttddtjeenucfhrhhomhepvfhhohhmrghsucfoohhnjhgrlhhonhcu oehthhhomhgrshesmhhonhhjrghlohhnrdhnvghtqeenucffohhmrghinhepghhithhhuh gsrdgtohhmnecukfhppeejjedrudefgedrvddtfedrudekgeenucfrrghrrghmpehmrghi lhhfrhhomhepthhhohhmrghssehmohhnjhgrlhhonhdrnhgvthenucevlhhushhtvghruf hiiigvpedt X-ME-Proxy: Received: from xps.localnet (184.203.134.77.rev.sfr.net [77.134.203.184]) by mail.messagingengine.com (Postfix) with ESMTPA id CF35FE459A; Mon, 17 Dec 2018 11:12:07 -0500 (EST) From: Thomas Monjalon To: "Wiles, Keith" Cc: "dev@dpdk.org" Date: Mon, 17 Dec 2018 17:12:06 +0100 Message-ID: <2356197.taPHoSjNZe@xps> In-Reply-To: <57278D80-18C5-4207-BE72-AFC61BA98419@intel.com> References: <20181216174604.91445-1-keith.wiles@intel.com> <2437900.lBdCot2IvA@xps> <57278D80-18C5-4207-BE72-AFC61BA98419@intel.com> MIME-Version: 1.0 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset="UTF-8" Subject: Re: [dpdk-dev] [PATCH v3 1/3] dfs:add FUSE based filesystem for DPDK X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Mon, 17 Dec 2018 16:12:09 -0000 Thanks for your detailed answer Keith. Honestly, I do not have much more to say, except some implementation details that I may comment later. I hope we will read more opinions from the community. 17/12/2018 16:01, Wiles, Keith: > > On Dec 17, 2018, at 5:45 AM, Thomas Monjalon wrot= e: > > 16/12/2018 18:46, Keith Wiles: > >> DFS stands for DPDK Filesystem, which helps expose data > >> and control files in a FUSE based filesystem. The dfs requires > >> libfuse3 and libjansson to be present in the Linux system. > >=20 > > You presented this idea at the DPDK Summit in Dublin, > > and I have not seen any public discussion about the idea. > > Let's try to debate it here. > >=20 > >> DFS provides a simplified API on top of the FUSE APIs to > >> simplify creating and using FUSE. > >>=20 > >> Here is the github repo: https://github.com/akheron/jansson > >> Here is the github repo: https://github.com/libfuse/libfuse > >>=20 > >> Please use you system updater tool yum, apt-get, ... to add > >> support for these two libraries. Also read the dfs documentation > >> in the docs directory for more details. > > [...] > >> +DPDK File System (DFS) > >> +=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D > >> + > >> +This DPDK library provides a pseudo file system much like Linux /proc= or /system > >> +file systems, but this one is a userspace file system. Allowing appli= cations to > >> +create and design file system directories and files using the FUSE 3.2 > >> +https://github.com/libfuse/libfuse code. > >=20 > > My first thought is we are missing the problem statement. > > What are you trying to solve? Which use case? >=20 > The issue is we do not have a clean way for users and developers to extra= ct information from DPDK or the application for monitoring/control. Using A= PIs is fine from the application perspective to get information and set inf= ormation, but the user or admin of a system does not have access to those A= PIs without a developer giving access via a command line or other method. E= ach developer creating an application would have to provide this basic leve= l of information via some method in a cloud or VNF system to allow the user= or orchestration access. Using DFS the developer would not normally have t= o write the access methods/display himself, saving him time to develop his = application. >=20 > A file system is the simplest and easiest method to get host command line= access to that information. Control or initialization can also be added at= this same simple method for DPDK and the application. Currently I do not h= ave any control support in DFS and it would be reasonable to add these cont= rols (in a limited way) to remove DPDK command line options or cmdline cmds= when starting DPDK. Giving some runtime control is better for the applicat= ion in a cloud or NFV environments. >=20 > >=20 > > In DPDK, we have some run-time options accessible from the command line, > > and some public API functions. >=20 > We have cmdline access to these functions and lets face it cmdline is ver= y difficult for most to use :-), but we do not have access from the system = level. The APIs in DFS is much easier and cleaner to allow access to the re= quired information. The application developer also can use these APIs to ex= pose this information without having to give some type of cmdline access. T= he application may not want a cmdline interface (or allowed to give cmdline= access), but does want to get access to the information in the application= and DPDK.=20 >=20 > Having access via the command line or bash shell is much easier to use an= d provides simple access by other languages like Go, Python, Bash, Lua =E2= =80=A6 any language that can read or write into a normal filesystem. Then t= he system level administrator or application developer can write tools in a= ny language he see fit. >=20 > > I think it is agreed that the primary interface must be the API functio= ns. > > The command line options should be a responsibility of the application > > to implement them or not. >=20 > It is not agreed, customers I have spoke with do not agree that DPDK info= rmation must be supplied by the application, it should be the responsibilit= y of DPDK in this case to provide that information in some easy to access m= ethod. If the application side wants to provide information or control then= the developer can also use DFS or not. > >=20 > > I think exposing a filesystem tree interface would be also an applicati= on > > responsibility. Actually, it could be part of a DPDK application, > > or be run as a secondary process application. >=20 > The secondary process method means the user must use the same version of = the application and DPDK to access the primary or we can crash the secondar= y or primary application. It is very easy to use the wrong version of DPDK = secondary application and get invalid information as the primary is differe= nt is some offset in a structure. >=20 > Remember cloud or NFV systems will have a large number of applications an= d versions. Using a secondary process model is also a bit of a security pro= blem and can cause the primary application to crash if the secondary does s= omething wrong. Also the system would have to provide a different secondary= application matching every DPDK/Application on the platform. Having to mat= ch up secondary applications to each and every application is a nightmare f= or the admin or developer. >=20 > > It is probably a good idea to implement it as a ready-to-use library. > > As it can be implemented on top of DPDK API, I think it should live > > outside of the main repository. As any other DPDK applications, it will > > require to be updated when the DPDK APIs are updated or extended. > > In my opinion, we should not mandate DPDK contributors to update DFS > > when adding or updating an API. That's why it should not be part of the > > DPDK package. >=20 > The only changes from DPDK developers is when a common API in DPDK change= s and is used by DFS. It would not be difficult to keep these changes updat= ed as these changes in API do not happen often. Requiring the developer of = DPDK to maintain the changes is already required for other parts of DPDK, c= orrect? >=20 > Maintaining DFS outside of DPDK is an artificial requirement as it does n= ot placed any more burden on DPDK developers IMO. It does place some one ti= me work on the test system for DPDK to test DFS. The testing system would n= eed to build DPDK with DFS and provide some basic level of verification. >=20 > The test system can also use DFS in normal testing to extract DPDK inform= ation without having to use expect scripts against the cmdline system, whic= h are very error prone and the syntax of the command could change. If the t= esting system in DPDK does not do this level of testing I think we should b= e adding that level to verify DPDK. >=20 > > One more argument to keep it outside: for diversity and innovation, > > it is better to not enforce one (new) control interface as the official > > (and suggested) one way. >=20 > The current control method does not work for all cases, it works only for= the developer. The system admin or the developer must remember cmdline syn= taxes, which are difficult to use (Look at testpmd ;-). In some cases using= a command line built into the application may not be reasonable in a norma= l runtime environment. Accessing the filesystem is normal and easy as admin= , developer, user and orchestration have access to the filesystem. >=20 > >=20 > > One more question, > > Is there something exposed in DFS that has not a public DPDK API? > > If so, we should try fill the gaps. >=20 > Yes, only a couple of debug routine ring and tailq walk routines I includ= ed in the patch. I was not focused on adding more APIs to DPDK main code on= ly exposing the information in DPDK using the current APIs. DFS should only= use standard DPDK APIs to expose information to the file system and the ap= plication can write very simple routines to expose any other information he= wants via DFS from his application. >=20 > The current exposed APIs in DFS are just the simple stats and information= I could collect from DPDK today, but I believe we can expose other informa= tion as we define these very quickly.