How to ensure packets of a flow will be distributed in order by rte_distributor

[Mohammad Malihi Aqdam <Aqdam@graph-inc.ir>]

[-- Attachment #1: Type: text/plain, Size: 1891 bytes --]

we are working on an application which uses rte_distributor to distribute packets based on 5 tuple extracted from packets. Each flow associates with an object (e.g connection) which will be used by worker logical cores. The packets of the flow must be processed in order and no connection instance must be accessed by two or more workers at the same time.
The underlying data structure as a flow manager is of type std::unordeted_map<unsigned, connection*> (key is the hash value obtained from nic rss offload based on 5 tuple).
To ensure that a connection instance will not be accessed by two worker cores, each connection class has a member named ref_cnt. Before processing the packets of a specific connection, the ref_cnt member value increases and after the completion the member decreases (using rte_atomic16_inc(&connection->ref_cnt) and rte_atomic16_dec(&connection->ref_cnt)). So the value of the ref_cnt before processing of each packet must be zero (i.e the connection is not is use). But according to test results at some iterations the value of ref_cnt is 1 which determines that the related instance has been distributed to another worker core.
According to Packet Distributor Library documentation (https://doc.dpdk.org/guides/prog_guide/packet_distrib_lib.html) No packet ordering guarantees are made about packets which do not share a common packet tag and it seems that the ordering will be preserved for packets of a particular flow (with the same tag) but the test results shows that the packets of a flow may be distributed to two distinct workers.
I want to know, if the distributor decides to distribute the packets of a flow while the older related packets still are under process in a worker, whether the distributor waits until the processing of older packets finishes or it distribute the new packets to another worker.

Thanks for considering my request


[-- Attachment #2: Type: text/html, Size: 2412 bytes --]