From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from mga11.intel.com (mga11.intel.com [192.55.52.93]) by dpdk.org (Postfix) with ESMTP id 7CD541B1F0 for ; Fri, 30 Nov 2018 14:13:14 +0100 (CET) X-Amp-Result: SKIPPED(no attachment in message) X-Amp-File-Uploaded: False Received: from fmsmga001.fm.intel.com ([10.253.24.23]) by fmsmga102.fm.intel.com with ESMTP/TLS/DHE-RSA-AES256-GCM-SHA384; 30 Nov 2018 05:13:13 -0800 X-ExtLoop1: 1 X-IronPort-AV: E=Sophos;i="5.56,298,1539673200"; d="scan'208";a="120367644" Received: from fmsmsx105.amr.corp.intel.com ([10.18.124.203]) by fmsmga001.fm.intel.com with ESMTP; 30 Nov 2018 05:13:13 -0800 Received: from fmsmsx153.amr.corp.intel.com (10.18.125.6) by FMSMSX105.amr.corp.intel.com (10.18.124.203) with Microsoft SMTP Server (TLS) id 14.3.408.0; Fri, 30 Nov 2018 05:13:13 -0800 Received: from shsmsx151.ccr.corp.intel.com (10.239.6.50) by FMSMSX153.amr.corp.intel.com (10.18.125.6) with Microsoft SMTP Server (TLS) id 14.3.408.0; Fri, 30 Nov 2018 05:13:12 -0800 Received: from shsmsx103.ccr.corp.intel.com ([169.254.4.59]) by SHSMSX151.ccr.corp.intel.com ([169.254.3.46]) with mapi id 14.03.0415.000; Fri, 30 Nov 2018 21:13:09 +0800 From: "Zhang, Qi Z" To: "Wiles, Keith" , dpdk-dev CC: "Richardson, Bruce" , "Yigit, Ferruh" , Thomas Monjalon , "Ananyev, Konstantin" Thread-Topic: [dpdk-dev] [RFC] DFS (DPDK Filesystem) Thread-Index: AQHUX1g+3LgzLEXi8EiFigRgY1m60qVjfviAgAUEHcA= Date: Fri, 30 Nov 2018 13:13:09 +0000 Message-ID: <039ED4275CED7440929022BC67E70611532F7D53@SHSMSX103.ccr.corp.intel.com> References: <897126DE-FC79-48FF-9B35-3E262EBDF093@intel.com> In-Reply-To: <897126DE-FC79-48FF-9B35-3E262EBDF093@intel.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-titus-metadata-40: eyJDYXRlZ29yeUxhYmVscyI6IiIsIk1ldGFkYXRhIjp7Im5zIjoiaHR0cDpcL1wvd3d3LnRpdHVzLmNvbVwvbnNcL0ludGVsMyIsImlkIjoiODg4ZTUxNDEtZjRhMC00YTlmLWI4NmItOTA2ZWRkOWNjZjRjIiwicHJvcHMiOlt7Im4iOiJDVFBDbGFzc2lmaWNhdGlvbiIsInZhbHMiOlt7InZhbHVlIjoiQ1RQX05UIn1dfV19LCJTdWJqZWN0TGFiZWxzIjpbXSwiVE1DVmVyc2lvbiI6IjE3LjEwLjE4MDQuNDkiLCJUcnVzdGVkTGFiZWxIYXNoIjoidHh3djVLTm8xcmJtb1p4VUdXYVpkV1hEU255SXorM3g4eVwvT1plcFRENUh1YVBDZnF6YmxFMHpsSW1UdHNcL3YxIn0= x-ctpclassification: CTP_NT dlp-product: dlpe-windows dlp-version: 11.0.400.15 dlp-reaction: no-action x-originating-ip: [10.239.127.40] Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 Subject: Re: [dpdk-dev] [RFC] DFS (DPDK Filesystem) X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , X-List-Received-Date: Fri, 30 Nov 2018 13:13:15 -0000 Hi Keith: > -----Original Message----- > From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Wiles, Keith > Sent: Tuesday, November 27, 2018 11:00 PM > To: dpdk-dev > Cc: Richardson, Bruce ; Yigit, Ferruh > ; Thomas Monjalon ; > Ananyev, Konstantin > Subject: Re: [dpdk-dev] [RFC] DFS (DPDK Filesystem) >=20 >=20 >=20 > > On Oct 8, 2018, at 5:42 PM, Wiles, Keith wrote: > > > > Hi Everyone, > > > > I would like to request comments on DFS as I presented at the DPDK summ= it > in Ireland. > > > > As we know DPDK can be difficult to manage at run time and as DPDK > becomes more dynamic we need to address how to configure and monitor > DPDK and DPDK-based applications. A possible solution is to use FUSE file > system support already in the Linux kernel through a DPDK library to expo= se a > directory structure for anyone to be able to monitor and control the app = by > reading/writing files. > > > > Primarily DPDK runs on Linux based systems, where FUSE is well supporte= d. > FUSE was added to FreeBSD 10 which appears to be the same design as Linux= . > Windows does not support FUSE directly from MS, but a open source design > has a FUSE based set of wrapper APIs to help port FUSE based file systems= . > > > > Today the only way to get information out of DPDK is to use log files o= r a > command line within DPDK to display the information. The information is n= ot > easy for an external application to collect or monitor. Using DFS any app= lication > can read and write into this virtual file system (FUSE) and retrieve info= rmation > from DPDK/Application via a raw ASCII text file and/or JSON formatted tex= t. > > > > Data read from DFS can be static or dynamic e.g. version/copyright text > would be static, but port stats would be dynamic and generated when the u= ser > requests the information. Writing new configuration data into DPDK can be= a > simple number or text or a JSON data structure. Writing simple ASCII text= data > is simple, but writing a long JSON or configuration file may some care, i= .e. write > a path to a file in DFS to the location of the configuration and use that= file path > for the configuration. > > > > DPDK command line is getting to be very complex and long, but we can > eliminate or greatly reduce the command line by using DFS to configure DP= DK > at startup along with the application configuration. Having DPDK start up= with > minimum resources and allow updating/creating a file in the DPDK file sys= tem > to configure DPDK would be easier. > > > > The DFS is a FUSE filesystem anchored at /dpdk in the host file system,= then a > directory is created under /dpdk for each DPDK instance or application us= ing > - as a directory name format. Then it is up to DPDK/Applicatio= n to > create the directory structures, permissions and files within the directo= ry > mount point. > > > > Currently I have a directory and files layout for DPDK, that needs more > attention as I was focusing on FUSE then the layout as it can be changed = fairly > easily. I can email the working documentation for DFS if needed please se= nd me > a short email or I can post to the dev list. The docs also try to explain= the design > and APIs used by the developer for creating files/directories. The docs a= re still a > work in progress and more attention needs to be done as we progress. > > > > DFS is just a library and not required to be used plus it does not modi= fy DPDK > APIs or structures except for adding a few new dump routines for features= that > do not have one today. The file system or FUSE backend in DPDK is a free > floating thread at this time, but could be affinitized to a core if requi= red. > > > > The FUSE protocol to the backend code (in DPDK) has a large number of > operations, but we have simplified the interface for developers to 5-6. T= he > interface on top of libfuse3 is very simple and currently provides most o= f the > needed APIs for managing the files and directories. The FUSE file system = with > the added simpler API is dynamic and can be changed at runtime. The > directory/files can be constructed via APIs at startup or anytime after s= tartup. > > > > I would like to put a repo someplace with DPDK/DFS say on GitHub to all= ow > others to play with the design in the short term. I also have a doc file = for DFS if > anyone wants to learn more information and will be in the repo as well. I= f we > agree I can then create a set of patches for DPDK. >=20 > Well the 18.11 release is out the door, this one seemed pretty big and I = lot of > hard work was put into this one, thanks to everyone. >=20 >=20 > It is time to try and get some feedback on this RFC. I know the code in t= he repo > is not finished, but I would like to get some folks to give it a try and = give > feedback. I would like to submit the basic DFS code with some low hanging > items mostly readonly information for the 19.02 release. >=20 > Here is the repo again you can look at and play with. The code is mostly = ready > for upstream but still more work needs to be done. I will try to update t= he repo > to the 18.11 release code based soon. >=20 > https://github.com/pktgen/dfs-dpdk >=20 > Thanks for giving this a review. Some thoughts base on a quick test and review. Overall, I think this is a very handy feature. I really hope it can be merg= ed quickly :) Below are some captures: 1) when I try to test dfs with exist application (say testpmd), I need to r= eplace rte_eal_init with dfs_eal_init,=20 I think a better way is by using args like --dfs which can avoid the code c= hange, but please let me know if I missed the point. 2) Is PMD allowed to use dfs to expose device specific information (private= API access, utilities for diagnose purpose ...) ? if it is=20 I think dfs could provide more standard way for PMD to add content to a s= ub directory , It could be=20 struct dfs_ops { mount(dfs_node parent...); /* create content of in the sub directory */ unmount(dfs_node parent...); /* destroy the content. */ } =20 =20 PMD is responsible for creating a dfs_ops instance and setup the callback= functions during initialization.(probe) An API rte_eth_dev_get_dfs_ctx(port_id) can be used to get the context th= at wrapped dfs_ops and related ethdev instance. (similar for cryptodev and = others ) So during dfs init, after we create folder for a port, we can do addition= al step to create a parent node for PMD then call its's dev_ops->mount, the= n all the device specific information will be popped in that folder 3) For primary-secondary model, dfs should works well, but the missing part= is there is no link between primary and secondary process,=20 it will be nature if we can list all secondary process in a primary folde= r or even better if we can jump to secondary folder by some soft link and s= ame thing in the reverse direction.=20 Thanks Qi >=20 > > > > Regards, > > Keith > > >=20 > Regards, > Keith