[PATCH 00/10] net/mlx5: improve MAC address and VLAN add latency

[Dariusz Sosnowski <dsosnowski@nvidia.com>]

Whenever a new MAC address is added to the port, mlx5 PMD will:

- Add this address to `dev->data->mac_addrs[]`.
- Destroy all control flow rules.
- Recreate all control flow rules.

Similar logic is also implemented for VLAN filters.

Because of such logic, the latency of adding the new MAC address
(i.e., latency of `rte_eth_dev_mac_addr_add()` function call)
is actually linear to number of MAC addresses already configured.
Since each operation of creating/destroying a control flow rule,
involves an `ioctl()` syscall, on some setups the latency of adding
a single MAC address can reach ~100ms, when port is operating with >= 100 MAC addresses.
The same problem exists for VLAN filters (and even compounded by it).

This patchset aims to resolve these issues,
by reworking how mlx5 PMD handles adding/removing MAC addresses and VLAN filters.
Instead of recreating all control flow rules,
only necessary flow rules will be created/removed on each operation,
thus minimizing number of syscalls triggered.

Summary of patches:

- Patch 1-2 - Extends existing `mlx5_hw_ctrl_flow_type` enum with special variants,
  which will be used for tracking MAC and VLAN control flow rules.
- Patch 3-4 - Refactors HWS code for control flow rule creation to allow
  creation of specific control flow rules with unicast MAC/VLAN match.
  Also functions are added for deletion of specific rules.
- Patch 5-6 - Prepares the control flow rules list, used by HWS flow engine,
  to be used by other flow engine.
  Goal is to reuse the similar logic in Verbs and DV flow engines.
- Patch 7-8 - Adjusts legacy flow engines, so that unicast DMAC/VLAN control flow rules
  are added to the control flow rules list.
  Also exposes functions for creating/destroying specific ones.
- Patch 9-10 - Extends `mlx5_traffic_*` interface with `mlx5_traffic_mac_add/remove` and
  `mlx5_traffic_vlan_add/remove` functions.
  They are used in implementations of DPDK APIs for adding/removing MAC addresses/VLAN filters
  and their goal is to update the set of control flow rules in a minimal number of steps possible,
  without recreating the rules.

As a result of these patches the time to add 128th MAC address,
after 127th was added drops **from ~72 ms to ~197 us** (at least on my setup).

Dariusz Sosnowski (10):
  net/mlx5: track unicast DMAC control flow rules
  net/mlx5: add checking if unicast flow rule exists
  net/mlx5: rework creation of unicast flow rules
  net/mlx5: support destroying unicast flow rules
  net/mlx5: rename control flow rules types
  net/mlx5: shared init of control flow rules
  net/mlx5: add legacy unicast flow rules management
  net/mlx5: add legacy unicast flow rule registration
  net/mlx5: add dynamic unicast flow rule management
  net/mlx5: optimize MAC address and VLAN filter handling

 drivers/net/mlx5/linux/mlx5_os.c      |   3 +
 drivers/net/mlx5/meson.build          |   1 +
 drivers/net/mlx5/mlx5.h               |  62 +++--
 drivers/net/mlx5/mlx5_flow.c          | 149 ++++++++++-
 drivers/net/mlx5/mlx5_flow.h          |  36 +++
 drivers/net/mlx5/mlx5_flow_hw.c       | 349 ++++++++++++++++++++------
 drivers/net/mlx5/mlx5_flow_hw_stubs.c |  68 +++++
 drivers/net/mlx5/mlx5_mac.c           |  41 ++-
 drivers/net/mlx5/mlx5_trigger.c       | 262 ++++++++++++++++++-
 drivers/net/mlx5/mlx5_vlan.c          |   9 +-
 drivers/net/mlx5/windows/mlx5_os.c    |   3 +
 11 files changed, 867 insertions(+), 116 deletions(-)
 create mode 100644 drivers/net/mlx5/mlx5_flow_hw_stubs.c

--
2.39.5