* Re: [dpdk-dev] Need your thoughts on DPDK hash table / DPDK lookup/insert API's
2015-09-25 22:26 [dpdk-dev] Need your thoughts on DPDK hash table / DPDK lookup/insert API's Yeddula, Avinash
@ 2015-09-28 20:46 ` Dumitrescu, Cristian
0 siblings, 0 replies; 2+ messages in thread
From: Dumitrescu, Cristian @ 2015-09-28 20:46 UTC (permalink / raw)
To: Yeddula, Avinash, dev
> -----Original Message-----
> From: dev [mailto:dev-bounces@dpdk.org] On Behalf Of Yeddula, Avinash
> Sent: Friday, September 25, 2015 11:27 PM
> To: dev@dpdk.org
> Subject: [dpdk-dev] Need your thoughts on DPDK hash table / DPDK
> lookup/insert API's
>
> Hello All,
>
>
> 1. I've a scenario where I need to walk the entire Hash table to retrieve the
> data. I'm currently using DPDK extensible bucket hash in the rte_table library
> of packet framework.
>
> Since I'll not be storing the keys anywhere else, I don't have a way to walk
> hash table.
>
> I'm planning to write one for my application, but just wanted to check with
> the DPDK community on their thoughts.
>
Please take a look at the examples/ip_pipeline application (from DPDK release 2.1+). You can look at any pipeline (flow classification, firewall, routing, etc), all of them implement the strategy detailed below.
The way we are solving this problem is by creating two copies of the same functional table, one fast copy and one slow copy, which are kept in sync while used for different purposes:
-fast table copy: used by data plane, implemented using the rte_table API, lookup is packet-oriented (works with a packet burst), lookup operation optimized for performance
-slow table copy: used by the control/management plane, not necessarily slow for lookup but optimized for queries, kept in sync with the fast table copy; main point is: queries from management are executed without impacting the data plane performance
To avoid locks, the data plane thread is the only thread that accesses the fast table copy, including lookup and add/delete entries. Besides polling the input packet queues for packet processing, the data plane thread also polls its input message queues (with a much lower frequency) to handle requests for updates on the fast table copy (such as adding new table entries, updating existing entries, deleting entries). The management thread updates the slow copy and sends requests to the data plane thread to update the fast table copy.
This way, complex table queries required by the management plane are implemented without impact to the data plane thread. These queries are usually defined by the application and it is not possible to do them in a generic way, here are some examples:
-hash table used for flow classification: list all flows with certain value/regex for whatever field in the flow key tuple, list all flows ordered based on whatever application dependent criteria (e.g. subscriber name, area code, etc)
-LPM table used for routing: list all routes with a specific output interface in the descending order of the depth of their IP prefix
A few documentation pointers:
http://www.dpdk.org/doc/guides/sample_app_ug/ip_pipeline.html#table-copies
http://www.dpdk.org/doc/guides/prog_guide/packet_framework.html#shared-data-structures
>
>
> 2. I have a scenario where the components whose are not part of the
> pipeline needs to call the DPDK lookup/add apis. Moreover they are
> interested in lookup/insert one entry at a time. With the current approach, I
> know everything works in bursts to achieve more/better performance.
>
>
>
> Currently a lookup api's looks like this.
>
> static int rte_table_array_lookup( void *table, struct rte_mbuf **pkts,
> uint64_t pkts_mask, uint64_t *lookup_hit_mask, void **entries)
>
>
>
> New addition to the existing lookup, I would like it to be something like this
> (not exactly, something similar). With this the outside guy doesn't have to
> construct "rte_mbuf and put the key in the metadata for the DPDK lookup
> API to act "
>
> static int rte_table_array_single_pkt_lookup( void *table, void *key, void
> *entry)
>
>
The packet-oriented lookup API of rte_table is intended for packet processing threads (data plane), you should have a different lookup mechanism for management thread, which should probably use a different table copy, as proposed above.
For example, please table a look at flow classification pipeline from the examples/ip_pipeline application:
-file pipeline_flow_classification_be.c: the back end of the pipeline (the data plane thread) uses the rte_table lookup API (packet oriented)
-file pipeline_flow_classification.c: the front-end of the pipeline (the CLI code executed by the management/master thread) implements a different hash mechanism with linked list buckets built on top of TAILQ (struct app_pipeline_fc::flows), which is kept in sync with the back-end copy through message requests
The back-end copy is exclusively used for packet processing, while the front-end copy is used for queries and list/display operations.
>
> Any thoughts on adding these 2 items officially to dpdk library.
>
>
>
> Thanks
>
> -Avinash
^ permalink raw reply [flat|nested] 2+ messages in thread