RE: seeking community input on adapting DPDK to P4Runtime backend

[Ori Kam <orika@nvidia.com>]

Hi Zhang,

I think we both want the same thing and share the same basic concepts.

PSB, some answers,

Best,
Ori


> -----Original Message-----
> From: Zhang, Qi Z <qi.z.zhang@intel.com>
> Sent: Thursday, May 18, 2023 1:33 PM
> 
> 
> 
> > -----Original Message-----
> > From: Ori Kam <orika@nvidia.com>
> > Sent: Wednesday, May 17, 2023 11:19 PM
> > To: Zhang, Qi Z <qi.z.zhang@intel.com>; dev@dpdk.org
> > Cc: techboard@dpdk.org; Richardson, Bruce
> <bruce.richardson@intel.com>;
> > Burakov, Anatoly <anatoly.burakov@intel.com>; Wiles, Keith
> > <keith.wiles@intel.com>; Liang, Cunming <cunming.liang@intel.com>; Wu,
> > Jingjing <jingjing.wu@intel.com>; Zhang, Helin <helin.zhang@intel.com>;
> > Mcnamara, John <john.mcnamara@intel.com>; Xu, Rosen
> > <rosen.xu@intel.com>
> > Subject: RE: seeking community input on adapting DPDK to P4Runtime
> > backend
> >
> > Hi Zhang,
> >
> > rte_flow is an excellent candidate for implementing P4.
> > We and some internal tests that shows great promise in this regard.
> >
> > I would be very happy to supply any needed information and have
> > discussion on how to continue with this project.
> 
> Thank you Ori! Please check my following comments
> 
> Regards
> Qi
> 
> >
> > Please see inline detailed answers.
> >
> > Best,
> > Ori Kam
> >
> >
> >
> >
> > > -----Original Message-----
> > > From: Zhang, Qi Z <qi.z.zhang@intel.com>
> > > Sent: Monday, May 8, 2023 9:40 AM
> > > Subject: seeking community input on adapting DPDK to P4Runtime
> > backend
> > >
> > > Hi:
> > >
> > > Our team is currently working on developing a DPDK PMD for a P4-
> > > programmed network controller, based on customer feedback to
> integrate
> > > DPDK into the P4Runtime backend .[https://p4.org/p4-
> > > spec/p4runtime/main/P4Runtime-Spec.html]
> > >
> > > (*) However, we are facing challenges in adapting DPDK's rte_flow API
> > > to the P4Runtime API, primarily due to the transition from a
> > > table-based API with fields of arbitrary bits width at arbitrary
> > > offset to a protocol-based API (more detail be described in post-script).
> > >
> > > We are seeking suggestions and best practices from the open-source
> > > community to help us with this integration. Specifically, we are
> > > interested in
> > > learning:
> > >
> > > (*) If anyone has previously attempted to map rte_flow to P4-based
> > devices.
> >
> > We did try successfully.
> >
> > > (*) Thoughts on how to map from table-based matching to protocol-
> based
> > > matching like in rte_flow.
> >
> > Rte_flow is table based (groups), now with the introduction of template
> API
> > rte_flow is even more table based (we added the concept of tables) which
> > are just what
> > p4 requires.
> 
> Yes, the rte_flow template can be used to map a sequence of patterns to a
> P4 table and a sequence of actions to a P4 action. However, Using a fixed
> rte_flow template can be problematic when handling different P4 programs
> in the same driver. To provide more flexibility, the mapping of patterns and
> actions can be externalized into a configuration file or part of the firmware
> can be learned from driver, allowing for customization based on the specific
> requirements of each P4 pipeline. actually we have enabled this approach in
> order to accommodate different P4 programs.
> 
> However, an alternative approach to consider is whether it would be feasible
> to directly expose the P4 table and action names or IDs to the application,
> rather than relying on rte_flow templates. This approach offers several
> potential benefits:
> 
> Integration with P4runtime Backend: By exposing the P4 table and action
> names or IDs directly, DPDK could be easily integrated as a P4runtime
> backend. This eliminates the need for translation from the P4runtime API to
> rte_flow templates in the application, simplifying the integration process.
> 
> Elimination of Manual Mapping: Exposing the P4 table and action names or
> IDs to the application would remove the requirement for the engineering
> team to manually map each pipeline to specific rte_flow templates. This is
> particularly beneficial in cases where hardware vendors provide customers
> with a toolchain to create their own P4 pipelines but do not necessarily own
> the P4 programs. By eliminating the dependency on rte_flow templates, this
> approach reduces complexity in using DPDK as the driver.
> 
> To be more specific, the proposed API for exposing P4 table and action
> names or IDs directly to the application could be as follows:
> 
> /* Get the table info */
> struct rte_p4_table_info tbl_info;
> rte_p4_table_info_get_by_name(port_id, "decap_vxlan_tcp_table",
> &tbl_info);
> 
> /* Create the key */
> struct rte_p4_table_key *key;
> rte_p4_table_key_create(port_id, tbl_info->id, &key);
> 
> /* Set the key fields */
> rte_p4_table_key_field_set_by_name(port_id, key, "wire_port",
> &wire_port, 2);
> rte_p4_table_key_field_set_by_name(port_id, key, "tun_ip_src",
> &tun_ip_src, 4);
> rte_p4_table_key_field_set_by_name(port_id, key, "tun_ip_dst",
> &tun_ip_dst, 4);
> rte_p4_table_key_field_set_by_name(port_id, key, "vni", &vni, 3);
> rte_p4_table_key_field_set_by_name(port_id, key, "ipv4_src", &ipv4_src, 4);
> rte_p4_table_key_field_set_by_name(port_id, key, "ipv4_dst", &ipv4_dst,
> 4);
> rte_p4_table_key_field_set_by_name(port_id, key, "src_port", &src_port,
> 2);
> rte_p4_table_key_field_set_by_name(port_id, key, "dst_port", &dst_port,
> 2);
> 
> /* Get the action spec info */
> struct rte_p4_action_spec_info as_info;
> rte_p4_action_spec_info_get_by_name(port_id, "decap_vxlan_fwd",
> &as_info);
> 
> 
> /* Create the action */
> struct rte_p4_action *action;
> rte_p4_action_create(port_id, as_info->id, &action);
> 
> 
> /* Set the action fields */
> rte_p4_table_action_field_set_by_name(port_id, action, "mod_id",
> &mod_id, 3);
> rte_p4_table_action_field_set_by_name(port_id, action, "port_id",
> &target_port_id, 2);
> 
> /* Add the entry */
> rte_p4_table_entry_add(port_id, tbl_info->id, key, action);
> 
> ...
> 

I think that introduce some API that knows P4 is the way to go,
but I think that this should be a very simple API which calls rte_flow.


> 
> 
> 
> 
> >
> > > (*) Any ideas on how to extend or expand the rte_flow APIs to better
> > > accommodate P4-based or other table-matching based devices.
> > >
> >
> > Lets discuss any issue you have.
> >
> > > Your insights and feedback would be greatly appreciated!
> > >
> > > ======================= Post-Script
> ============================
> > >
> > > More details on the problem below, for anyone interested
> > >
> > > In P4, flow offloading can be implemented using the P4Runtime API,
> > > which provides a standard interface for controlling and configuring
> > > the data plane behavior of network devices. P4Runtime allows network
> > > operators to dynamically add, modify, and remove flow rules in the
> > > hardware forwarding tables of P4-enabled devices.
> > >
> > > The P4Runtime API is a table-based API, it assume the packet process
> > > pipeline was consists of one or more key/action units (tables). In
> > > P4Runtime, each table defines the fields to be matched and the actions
> > > to be taken on incoming packets. During compilation, the P4 compiler
> > > assigns a unique
> > > uint32 ID to each table, action, and field, which is associated with
> > > its corresponding string name. These IDs have no inherent relationship
> > > to any network protocol but instead serve as a means to identify
> > > different components of a P4 program within the P4Runtime API.
> > >
> > This is the concept of tables and groups in rte_flow.
> >
> > > If we choose to use rte_flow as the low-level API for P4Runtime, a
> > > translation layer is needed in the application to map the P4 tables
> > > and actions to the corresponding rte_flow rules. However, this
> > > translation layer can be problematic as it is not easily scalable.
> > > When the P4 pipeline is refined or updated, the translation rules may
> > > also need to be updated, which can result in errors and reduced
> efficiency.
> > >
> > I don't understand why.
> >
> > > On the other hand, a hardware vendor that provides a P4-enabled device
> > > is required to implement an rte_flow interface in their DPDK PMD.
> > > Typically, the
> > > P4 compiler generates hints for the driver on how to map P4 tables to
> > > hardware resources, and how to convert table entry add/modify/delete
> > > actions into low-level hardware configurations. However, because
> > > rte_flow is protocol-based, it poses an additional challenge for
> > > driver developers, who must create another translation layer to
> > > convert rte_flow tokens into P4 object identifiers. This translation
> > > layer must be carefully designed and implemented to ensure optimal
> > > performance and scalability, and to ensure that the driver can efficiently
> > handle the dynamic nature of P4 programs.
> > >
> > Right, but some of the translation can be done in shared code by all PMDs
> > and the translation is static for the compilation so inserting rules can be
> > supper fast with no need for extra work.
> >
> > > To better understand the problem, let's consider the following example
> > > that demonstrates how to use the P4Runtime API to program a rule for
> > > processing a VXLAN packet. The rule matches a VXLAN packet,
> > > decapsulates the tunnel header, and forwards it to a specific port.
> > >
> > > The P4 source code below describes the VXLAN decap table
> > > decap_vxlan_tcp_table, which matches the outer IP address, VNI, inner
> > > IP address, and inner TCP port. For each rule, four action
> > > specifications can be selected. We will focus on one action
> > > specification decap_vxlan_fwd that performs decapsulation and forwards
> > the packet to a specific port.
> > >
> > > table decap_vxlan_tcp_table {
> > >     key = {
> > >         hdrs.ipv4[meta.depth-1].src_ip: exact @name("tun_ip_src");
> > >         hdrs.ipv4[meta.depth-1].dst_ip: exact @name("tun_ip_dst");
> > >         hdrs.vxlan[meta.depth-1].vni  : exact @name("vni");
> > >         hdrs.ipv4[meta.depth].src_ip  : exact @name("ipv4_src");
> > >         hdrs.ipv4[meta.depth].dst_ip  : exact @name("ipv4_dst");
> > >         hdrs.tcp.sport                : exact @name("src_port");
> > >         hdrs.tcp.dport                : exact @name("dst_port");
> > >     }
> > >     actions = {
> > >         @tableonly decap_vxlan_fwd;
> > >         @tableonly decap_vxlan_dnat_fwd;
> > >         @tableonly decap_vxlan_snat_fwd;
> > >         @defaultonly set_exception;
> > >     }
> > > }
> > Translate to rte_flow:
> > template pattern relaxed_mode = 1 pattern =  Ipv4_src / ipv4_dst  / vni /
> > ipv4_src / ipv4_dst / tcp_sport / tcp_dport map structure = {
> > 	tun_ip_src = &pattern[ipv4_src]
> > 	....
> > }
> > > ...
> > >
> > > action decap_vxlan_fwd(PortId_t port_id) {
> > >     meta.mod_action = (bit<11>)VXLAN_DECAP_OUTER_IPV4;
> > >     send_to_port(port_id);
> > > }
> > >
> > Same as above just with action template
> >
> > > Below is an example of the hint that the compiler will generate for
> > > the
> > > decap_vxlan_tcp_table:
> > >
> > > Table ID:      8454144
> > > Name:          decap_vxlan_tcp_table
> > > Field ID       Name                          Match Type     Bit Width
> > > Byte Width     Byte Order
> > > 1              tun_ip_src                    exact          32
> > > 4              network
> > > 2              tun_ip_dst                    exact          32
> > > 4              network
> > > 3              vni                           exact          24
> > > 3              network
> > > 4              ipv4_src                      exact          32
> > > 4              network
> > > 5              ipv4_dst                      exact          32
> > > 4              network
> > > 6              src_port                      exact          16
> > > 2              network
> > > 7              dst_port                      exact          16
> > > 2              network Spec ID        Name
> > > 8519716        decap_vxlan_fwd
> > > 8519718        decap_vxlan_dnat_fwd
> > > 8519720        decap_vxlan_snat_fwd
> > > 8519695        set_exception
> > >
> > > And the hint of action spec "decap_vxlan_fwd" as below:
> > >
> > > Spec ID:       8519716
> > > Name:          decap_vxlan_fwd
> > > Field ID       Name                          Bit Width      Byte Width
> > > Byte Order
> > > 1              port_id                       32             4
> > > host
> > >
> > > Please note that different compilers may assign different IDs.
> > >
> > > Below is an example of how to program a rule using the P4 runtime API
> > > in JSON format. This rule matches fields and directs packets to port 5.
> > >
> > > {
> > >     "type": 1,  //INSERT
> > >     "entity": {
> > >         "table_entry": {
> > >             "table_id": 8454144,
> > >             "match": [
> > >                 { "field_id": 1, "exact": { "value": [10, 0, 0, 1] }
> > > },   // outer src IP =
> > > 10.0.0.1
> > >                 { "field_id": 2, "exact": { "value": [10, 0, 0, 2] }
> > > },  // outer dst IP =
> > > 10.0.0.2
> > >                 { "field_id": 3, "exact": { "value": [0, 0, 10] } },
> > > //  vni = 10,
> > >                 { "field_id": 4, "exact": { "value": [192, 0, 0, 1] }
> > > }, // inner src IP =
> > > 192.0.0.1
> > >                 {"field_id": 5, "exact": { "value": [192, 0, 0, 2] }
> > > }, // inner dst IP =
> > > 192.0.0.2
> > >                 {"field_id": 6, "exact": { "value": [0, 200] } }, //
> > > tcp src port = 200
> > >                 {"field_id": 7, "exact": { "value": [0, 201] } }, //
> > > tcp dst port = 201
> > >             ],
> > >             "action": {
> > >                 "action": {
> > >                     "action_id": 8519716,
> > >                     "params": [
> > >                         { "param_id": 1, "value": [5, 0, 0, 0] }
> > >                     ]
> > >                 }
> > >             },
> > >             ...
> > >         }
> > >     }    ...
> > > }
> > >
> > > Please note that this is only a part of the full command. For more
> > > information, please refer to the p4runtime.proto[2]
> > >
> > > 1. https://p4.org/p4-spec/p4runtime/main/P4Runtime-Spec.html
> > > 2.
> > >
> >
> https://github.com/p4lang/p4runtime/blob/main/proto/p4/v1/p4runtime.p
> > r
> > > oto
> > >
> > > Thank you for your attention to this matter.
> > >
> >
> > I think that we should schedule some meeting to see how much gaps we
> > really have between the rte_flow and
> > P4 and how we can improve the rte_flow to allow the best experience.
> 
> Sound a good idea!
> >
> > > Regards
> > > Qi