[dpdk-dev] [RFC] net/mlx5: move to non-cached mode for flow rules

[Bing Zhao <bingz@mellanox.com>]

In the current PMD implementation, all the flow rules' related information is
stored by default, like the table resources, matchers and actions and so on that
are already translated into the hardware descriptions via the interfaces. When
restarting a device after stopping it, all the flow rules will be retained and
reapplied to the NIC automatically. Handling for each packet will remain the
same after the device restarting. And this is what we are talking about "cached
mode" for flow rules.

There are also two disadvantages of cached mode, listed as below:
  - redundancy information and memory consumption: due to the fact that usually
    the application on top of DPDK will also store such information for
    management purpose.
  - moreover, flows may be not valid or used anymore after a device restarting
   Process. There might even be some conflict with the configuration.
And in the meanwhile, from the description and definition of 'RTE Ethernet
Device API', there is no need for flow rules to be retained during the
rte_eth_dev_stop()/rte_eth_dev_start() process. Caching the flow rules makes no
sense, and cached mode is almost useless.

So we will remove the flow rules cache and move to the non-cached mode. In the
device closing stage, all the software resources for flows created will be freed
and corresponding hardware resources will be released. Then the total cost of
the memory will be reduced and the behavior of mlx5 PMD will comply fully with
the ethdev API expectations.
After a device restarting, all the needed flow rules should be reinserted via
the create routine in the rte_flow lib.