DPDK patches and discussions
 help / color / mirror / Atom feed
From: "Wiles, Keith" <keith.wiles@intel.com>
To: dpdk-dev <dev@dpdk.org>
Cc: "Richardson, Bruce" <bruce.richardson@intel.com>,
	"Yigit, Ferruh" <ferruh.yigit@intel.com>,
	Thomas Monjalon <thomas@monjalon.net>,
	"Ananyev, Konstantin" <konstantin.ananyev@intel.com>
Subject: Re: [dpdk-dev] [RFC] DFS (DPDK Filesystem)
Date: Tue, 27 Nov 2018 15:00:07 +0000	[thread overview]
Message-ID: <897126DE-FC79-48FF-9B35-3E262EBDF093@intel.com> (raw)
In-Reply-To: <DD984D0C-526B-4859-BE9F-6D39564D9D8B@intel.com>



> On Oct 8, 2018, at 5:42 PM, Wiles, Keith <keith.wiles@intel.com> wrote:
> 
> Hi Everyone,
> 
> I would like to request comments on DFS as I presented at the DPDK summit in Ireland.
> 
> As we know DPDK can be difficult to manage at run time and as DPDK becomes more dynamic we need to address how to configure and monitor DPDK and DPDK-based applications. A possible solution is to use FUSE file system support already in the Linux kernel through a DPDK library to expose a directory structure for anyone to be able to monitor and control the app by reading/writing files.
> 
> Primarily DPDK runs on Linux based systems, where FUSE is well supported. FUSE was added to FreeBSD 10 which appears to be the same design as Linux. Windows does not support FUSE directly from MS, but a open source design has a FUSE based set of wrapper APIs to help port FUSE based file systems.
> 
> Today the only way to get information out of DPDK is to use log files or a command line within DPDK to display the information. The information is not easy for an external application to collect or monitor. Using DFS any application can read and write into this virtual file system (FUSE) and retrieve information from DPDK/Application via a raw ASCII text file and/or JSON formatted text.
> 
> Data read from DFS can be static or dynamic e.g. version/copyright text would be static, but port stats would be dynamic and generated when the user requests the information. Writing new configuration data into DPDK can be a simple number or text or a JSON data structure. Writing simple ASCII text data is simple, but writing a long JSON or configuration file may some care, i.e. write a path to a file in DFS to the location of the configuration and use that file path for the configuration.
> 
> DPDK command line is getting to be very complex and long, but we can eliminate or greatly reduce the command line by using DFS to configure DPDK at startup along with the application configuration. Having DPDK start up with minimum resources and allow updating/creating a file in the DPDK file system to configure DPDK would be easier.
> 
> The DFS is a FUSE filesystem anchored at /dpdk in the host file system, then a directory is created under /dpdk for each DPDK instance or application using <name>-<PID> as a directory name format. Then it is up to DPDK/Application to create the directory structures, permissions and files within the directory mount point.
> 
> Currently I have a directory and files layout for DPDK, that needs more attention as I was focusing on FUSE then the layout as it can be changed fairly easily. I can email the working documentation for DFS if needed please send me a short email or I can post to the dev list. The docs also try to explain the design and APIs used by the developer for creating files/directories. The docs are still a work in progress and more attention needs to be done as we progress.
> 
> DFS is just a library and not required to be used plus it does not modify DPDK APIs or structures except for adding a few new dump routines for features that do not have one today. The file system or FUSE backend in DPDK is a free floating thread at this time, but could be affinitized to a core if required.
> 
> The FUSE protocol to the backend code (in DPDK) has a large number of operations, but we have simplified the interface for developers to 5-6. The interface on top of libfuse3 is very simple and currently provides most of the needed APIs for managing the files and directories. The FUSE file system with the added simpler API is dynamic and can be changed at runtime. The directory/files can be constructed via APIs at startup or anytime after startup.
> 
> I would like to put a repo someplace with DPDK/DFS say on GitHub to allow others to play with the design in the short term. I also have a doc file for DFS if anyone wants to learn more information and will be in the repo as well. If we agree I can then create a set of patches for DPDK.

Well the 18.11 release is out the door, this one seemed pretty big and I lot of hard work was put into this one, thanks to everyone.


It is time to try and get some feedback on this RFC. I know the code in the repo is not finished, but I would like to get some folks to give it a try and give feedback. I would like to submit the basic DFS code with some low hanging items mostly readonly information for the 19.02 release.

Here is the repo again you can look at and play with. The code is mostly ready for upstream but still more work needs to be done. I will try to update the repo to the 18.11 release code based soon.

https://github.com/pktgen/dfs-dpdk

Thanks for giving this a review.

> 
> Regards,
> Keith
> 

Regards,
Keith

  parent reply	other threads:[~2018-11-27 15:00 UTC|newest]

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-10-08 22:42 Wiles, Keith
2018-10-09 22:52 ` Wiles, Keith
2018-10-16 17:11   ` Wiles, Keith
2018-11-27 15:00 ` Wiles, Keith [this message]
2018-11-30 13:13   ` Zhang, Qi Z
2018-11-30 15:37     ` Wiles, Keith
2018-11-30 16:01     ` Wiles, Keith
2018-12-02 18:40       ` Wiles, Keith

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=897126DE-FC79-48FF-9B35-3E262EBDF093@intel.com \
    --to=keith.wiles@intel.com \
    --cc=bruce.richardson@intel.com \
    --cc=dev@dpdk.org \
    --cc=ferruh.yigit@intel.com \
    --cc=konstantin.ananyev@intel.com \
    --cc=thomas@monjalon.net \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).