From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 60E96A0613 for ; Thu, 26 Sep 2019 11:34:28 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id BB8EA1BF97; Thu, 26 Sep 2019 11:34:26 +0200 (CEST) Received: from EUR01-VE1-obe.outbound.protection.outlook.com (mail-eopbgr140041.outbound.protection.outlook.com [40.107.14.41]) by dpdk.org (Postfix) with ESMTP id 8C4BB1BF97 for ; Thu, 26 Sep 2019 11:34:24 +0200 (CEST) ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=XIvAvAaWPys793cEix54gCuh0PBEuRIBd/oeSPLEAbV6kvBz3kr7BScPOXFcpTZv/oK7s4uyYktI4dExXW7DU4V/rRue4nVVFgXZm2V63lKQMFVeayZ7gWGvQfeP3WEMGQJe4g7hl08PZ4vAyrKUzXzezM4Jscb9Tn3boyPREVqevCIUKNe5IhmXW5I3ai9erJGANWhd3sgBCjVH4GJBHFhh+sGECBHvUuBc6WK4S8Hbr9RCNtbEdF4BCBUOn28A6A7x+fIoBP4j+0Q9V43eqCKh9AQhrHsGT4WBLd3/pcKI3buRpz5MZEs3ibaX0tpxP8sn8jraXUexzfDnoQlbAQ== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AF58WgHCJd+ujukyESDikjilfsFWnyTEey/KyDSZ1YM=; b=Rp3KxMh85ivKjahC37j/KExEOTNDBOFFq153X+7XcqmJrh7olFqzL6XJYnvO2Iskem60uYgWTLvaVCYRC5Xz1OBIGWJeAxSYS9LkVKOZnbtcIelkdHf31qiJT3okbEtPrWOecik5/tv3YPQckylV5wQUdEsPCvqLWgI43jrR7DbYs+OtL1F0M3Y67WVIaWrGazU9XxnNVkm9utBQPsi2lULJwkryH74ek+Dkz6mAzxWgxFPPxPPLezxFWIfAT5QBOKMYcfh7U8fW7dMFpndEegUHO4zZ7dadJtwyZh1/kHjqryb8MwmSWDOg9MM+QvfgRa/NGpgo9SXtX1GBeER8cw== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=mellanox.com; dmarc=pass action=none header.from=mellanox.com; dkim=pass header.d=mellanox.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Mellanox.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=AF58WgHCJd+ujukyESDikjilfsFWnyTEey/KyDSZ1YM=; b=pgfUZYZ/mnzC1m71Pl39craSM4O5BPv9WIXGAcUAXWDqnEa7JkA60tHoKrpT6PrdWXKXoUt8Zz4e/qfflkyMidLl8hhwiysADlrrIwHjWmwVrMR01HqOJ9X7KmsAyXM+EAkGcyAWdWHWeu0KV65AVjxHZaWNvybumEOKIjtynl8= Received: from AM4PR05MB3265.eurprd05.prod.outlook.com (10.171.188.154) by AM4PR05MB3474.eurprd05.prod.outlook.com (10.171.186.159) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2284.20; Thu, 26 Sep 2019 09:34:23 +0000 Received: from AM4PR05MB3265.eurprd05.prod.outlook.com ([fe80::da9:65ba:1323:a39b]) by AM4PR05MB3265.eurprd05.prod.outlook.com ([fe80::da9:65ba:1323:a39b%7]) with mapi id 15.20.2305.017; Thu, 26 Sep 2019 09:34:23 +0000 From: Slava Ovsiienko To: Ori Kam , Matan Azrad , Shahaf Shuler CC: "dev@dpdk.org" , Ori Kam , "jingjing.wu@intel.com" , "stephen@networkplumber.org" Thread-Topic: [PATCH 12/13] net/mlx5: split hairpin flows Thread-Index: AQHVdDPkasjiyXK9IUGONNL0oz4mlqc9sriQ Date: Thu, 26 Sep 2019 09:34:23 +0000 Message-ID: References: <1569479349-36962-1-git-send-email-orika@mellanox.com> <1569479349-36962-13-git-send-email-orika@mellanox.com> In-Reply-To: <1569479349-36962-13-git-send-email-orika@mellanox.com> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: authentication-results: spf=none (sender IP is ) smtp.mailfrom=viacheslavo@mellanox.com; x-originating-ip: [95.67.35.250] x-ms-publictraffictype: Email x-ms-office365-filtering-correlation-id: f3eb923e-d2c9-4835-79c9-08d74264b761 x-ms-office365-filtering-ht: Tenant x-microsoft-antispam: BCL:0; PCL:0; RULEID:(2390118)(7020095)(4652040)(8989299)(5600167)(711020)(4605104)(1401327)(4618075)(4534185)(4627221)(201703031133081)(201702281549075)(8990200)(2017052603328)(7193020); SRVR:AM4PR05MB3474; x-ms-traffictypediagnostic: AM4PR05MB3474:|AM4PR05MB3474: x-ms-exchange-transport-forked: True x-microsoft-antispam-prvs: x-ms-oob-tlc-oobclassifiers: OLM:1265; x-forefront-prvs: 0172F0EF77 x-forefront-antispam-report: SFV:NSPM; SFS:(10009020)(4636009)(136003)(376002)(366004)(396003)(39860400002)(346002)(51234002)(13464003)(189003)(199004)(478600001)(71190400001)(71200400001)(81156014)(81166006)(66066001)(8936002)(30864003)(6636002)(5660300002)(52536014)(86362001)(8676002)(7696005)(99286004)(305945005)(76176011)(7736002)(186003)(229853002)(102836004)(55016002)(6436002)(53546011)(9686003)(14444005)(33656002)(66476007)(4326008)(66556008)(6246003)(2906002)(66446008)(64756008)(6506007)(26005)(54906003)(66946007)(316002)(14454004)(76116006)(110136005)(3846002)(6116002)(256004)(11346002)(25786009)(476003)(446003)(486006)(74316002)(579004); DIR:OUT; SFP:1101; SCL:1; SRVR:AM4PR05MB3474; H:AM4PR05MB3265.eurprd05.prod.outlook.com; FPR:; SPF:None; LANG:en; PTR:InfoNoRecords; MX:1; A:1; received-spf: None (protection.outlook.com: mellanox.com does not designate permitted sender hosts) x-ms-exchange-senderadcheck: 1 x-microsoft-antispam-message-info: +UDRbDw4zwSA/PQoz1ItljyL/LeDDwYCS6oMRqO2qTJvTxYn+2e8Xza4oIfx9ggxjNVDq68WuCF4oOCuuheF9RSV9oS8rv7u654Us8nzTQjyo0zaPffeSwj0BIaX4Gu9Jx7HZu7DBmgwmPq+zp2kEluv65shlZ2g+fzbyzobCt/5AnLyxEqhvoHBlWIrqFNbyv3sJohUVoZc4QzpK21jmhhsaOYATKJOLuUEBb8MC7pRh0c3fwtW6sLneC/OAy/T1lehq9E6r9v0HRMHF6JOLCamHZ1F0BiQh/mfTui6mlazH9bcPt0fyR2f6zLtvOwEv5f8t/VG+Obxw2M8hDRRFDig8rZmMuju173VdzR8gAMfdNKvHShLzo7nhFTnOwLEPOjdStyLQ4zdn2A4ABV8Eve8lKqsi4TdLXjkYDEwVAY= Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-OriginatorOrg: Mellanox.com X-MS-Exchange-CrossTenant-Network-Message-Id: f3eb923e-d2c9-4835-79c9-08d74264b761 X-MS-Exchange-CrossTenant-originalarrivaltime: 26 Sep 2019 09:34:23.5786 (UTC) X-MS-Exchange-CrossTenant-fromentityheader: Hosted X-MS-Exchange-CrossTenant-id: a652971c-7d2e-4d9b-a6a4-d149256f461b X-MS-Exchange-CrossTenant-mailboxtype: HOSTED X-MS-Exchange-CrossTenant-userprincipalname: M5YyqSBB0Ndni4g0cyYikkvp2tJ5wdxw9XzmM5elb+0UlvAWH0Yr9OBRC2pazLV+OryRi0l0RvkXp4YynJfpH4tgaKVIz7tMo97cFHVAKmk= X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM4PR05MB3474 Subject: Re: [dpdk-dev] [PATCH 12/13] net/mlx5: split hairpin flows X-BeenThere: dev@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: DPDK patches and discussions List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: dev-bounces@dpdk.org Sender: "dev" > -----Original Message----- > From: Ori Kam > Sent: Thursday, September 26, 2019 9:29 > To: Matan Azrad ; Shahaf Shuler > ; Slava Ovsiienko > Cc: dev@dpdk.org; Ori Kam ; jingjing.wu@intel.com; > stephen@networkplumber.org > Subject: [PATCH 12/13] net/mlx5: split hairpin flows >=20 > Since the encap action is not supported in RX, we need to split the hairp= in > flow into RX and TX. >=20 > Signed-off-by: Ori Kam Acked-by: Viacheslav Ovsiienko > --- > drivers/net/mlx5/mlx5.c | 10 ++ > drivers/net/mlx5/mlx5.h | 10 ++ > drivers/net/mlx5/mlx5_flow.c | 281 > +++++++++++++++++++++++++++++++++++-- > drivers/net/mlx5/mlx5_flow.h | 14 +- > drivers/net/mlx5/mlx5_flow_dv.c | 10 +- > drivers/net/mlx5/mlx5_flow_verbs.c | 11 +- > drivers/net/mlx5/mlx5_rxq.c | 26 ++++ > drivers/net/mlx5/mlx5_rxtx.h | 2 + > 8 files changed, 334 insertions(+), 30 deletions(-) >=20 > diff --git a/drivers/net/mlx5/mlx5.c b/drivers/net/mlx5/mlx5.c index > 940503d..2837cba 100644 > --- a/drivers/net/mlx5/mlx5.c > +++ b/drivers/net/mlx5/mlx5.c > @@ -528,6 +528,12 @@ struct mlx5_flow_id_pool * > err =3D ENOMEM; > goto error; > } > + sh->flow_id_pool =3D mlx5_flow_id_pool_alloc(); > + if (!sh->flow_id_pool) { > + DRV_LOG(ERR, "can't create flow id pool"); > + err =3D ENOMEM; > + goto error; > + } > #endif /* HAVE_IBV_FLOW_DV_SUPPORT */ > /* > * Once the device is added to the list of memory event @@ -567,6 > +573,8 @@ struct mlx5_flow_id_pool * > claim_zero(mlx5_glue->dealloc_pd(sh->pd)); > if (sh->ctx) > claim_zero(mlx5_glue->close_device(sh->ctx)); > + if (sh->flow_id_pool) > + mlx5_flow_id_pool_release(sh->flow_id_pool); > rte_free(sh); > assert(err > 0); > rte_errno =3D err; > @@ -629,6 +637,8 @@ struct mlx5_flow_id_pool * > claim_zero(mlx5_devx_cmd_destroy(sh->td)); > if (sh->ctx) > claim_zero(mlx5_glue->close_device(sh->ctx)); > + if (sh->flow_id_pool) > + mlx5_flow_id_pool_release(sh->flow_id_pool); > rte_free(sh); > exit: > pthread_mutex_unlock(&mlx5_ibv_list_mutex); > diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index > 5f1a25d..5336554 100644 > --- a/drivers/net/mlx5/mlx5.h > +++ b/drivers/net/mlx5/mlx5.h > @@ -574,6 +574,15 @@ struct mlx5_devx_dbr_page { > uint64_t dbr_bitmap[MLX5_DBR_BITMAP_SIZE]; }; >=20 > +/* ID generation structure. */ > +struct mlx5_flow_id_pool { > + uint32_t *free_arr; /**< Pointer to the a array of free values. */ > + uint32_t base_index; > + /**< The next index that can be used without any free elements. */ > + uint32_t *curr; /**< Pointer to the index to pop. */ > + uint32_t *last; /**< Pointer to the last element in the empty arrray. > +*/ }; > + > /* > * Shared Infiniband device context for Master/Representors > * which belong to same IB device with multiple IB ports. > @@ -632,6 +641,7 @@ struct mlx5_ibv_shared { > struct mlx5dv_devx_cmd_comp *devx_comp; /* DEVX async comp > obj. */ > struct mlx5_devx_obj *tis; /* TIS object. */ > struct mlx5_devx_obj *td; /* Transport domain. */ > + struct mlx5_flow_id_pool *flow_id_pool; /* Flow ID pool. */ > struct mlx5_ibv_shared_port port[]; /* per device port data array. */ > }; >=20 > diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c > index 33ed204..50e1d11 100644 > --- a/drivers/net/mlx5/mlx5_flow.c > +++ b/drivers/net/mlx5/mlx5_flow.c > @@ -606,7 +606,7 @@ uint32_t mlx5_flow_adjust_priority(struct > rte_eth_dev *dev, int32_t priority, { > struct mlx5_priv *priv =3D dev->data->dev_private; > struct rte_flow *flow =3D dev_flow->flow; > - const int mark =3D !!(flow->actions & > + const int mark =3D !!(dev_flow->actions & > (MLX5_FLOW_ACTION_FLAG | > MLX5_FLOW_ACTION_MARK)); > const int tunnel =3D !!(dev_flow->layers & > MLX5_FLOW_LAYER_TUNNEL); > unsigned int i; > @@ -669,7 +669,7 @@ uint32_t mlx5_flow_adjust_priority(struct > rte_eth_dev *dev, int32_t priority, { > struct mlx5_priv *priv =3D dev->data->dev_private; > struct rte_flow *flow =3D dev_flow->flow; > - const int mark =3D !!(flow->actions & > + const int mark =3D !!(dev_flow->actions & > (MLX5_FLOW_ACTION_FLAG | > MLX5_FLOW_ACTION_MARK)); > const int tunnel =3D !!(dev_flow->layers & > MLX5_FLOW_LAYER_TUNNEL); > unsigned int i; > @@ -2419,6 +2419,210 @@ uint32_t mlx5_flow_adjust_priority(struct > rte_eth_dev *dev, int32_t priority, } >=20 > /** > + * Check if the flow should be splited due to hairpin. > + * The reason for the split is that in current HW we can't > + * support encap on Rx, so if a flow have encap we move it > + * to Tx. > + * > + * @param dev > + * Pointer to Ethernet device. > + * @param[in] attr > + * Flow rule attributes. > + * @param[in] actions > + * Associated actions (list terminated by the END action). > + * > + * @return > + * > 0 the number of actions and the flow should be split, > + * 0 when no split required. > + */ > +static int > +flow_check_hairpin_split(struct rte_eth_dev *dev, > + const struct rte_flow_attr *attr, > + const struct rte_flow_action actions[]) { > + int queue_action =3D 0; > + int action_n =3D 0; > + int encap =3D 0; > + const struct rte_flow_action_queue *queue; > + const struct rte_flow_action_rss *rss; > + const struct rte_flow_action_raw_encap *raw_encap; > + > + if (!attr->ingress) > + return 0; > + for (; actions->type !=3D RTE_FLOW_ACTION_TYPE_END; actions++) { > + switch (actions->type) { > + case RTE_FLOW_ACTION_TYPE_QUEUE: > + queue =3D actions->conf; > + if (mlx5_rxq_get_type(dev, queue->index) !=3D > + MLX5_RXQ_TYPE_HAIRPIN) > + return 0; > + queue_action =3D 1; > + action_n++; > + break; > + case RTE_FLOW_ACTION_TYPE_RSS: > + rss =3D actions->conf; > + if (mlx5_rxq_get_type(dev, rss->queue[0]) !=3D > + MLX5_RXQ_TYPE_HAIRPIN) > + return 0; > + queue_action =3D 1; > + action_n++; > + break; > + case RTE_FLOW_ACTION_TYPE_VXLAN_ENCAP: > + case RTE_FLOW_ACTION_TYPE_NVGRE_ENCAP: > + encap =3D 1; > + action_n++; > + break; > + case RTE_FLOW_ACTION_TYPE_RAW_ENCAP: > + raw_encap =3D actions->conf; > + if (raw_encap->size > > + (sizeof(struct rte_flow_item_eth) + > + sizeof(struct rte_flow_item_ipv4))) > + encap =3D 1; > + action_n++; > + break; > + default: > + action_n++; > + break; > + } > + } > + if (encap =3D=3D 1 && queue_action) > + return action_n; > + return 0; > +} > + > +#define MLX5_MAX_SPLIT_ACTIONS 24 > +#define MLX5_MAX_SPLIT_ITEMS 24 > + > +/** > + * Split the hairpin flow. > + * Since HW can't support encap on Rx we move the encap to Tx. > + * If the count action is after the encap then we also > + * move the count action. in this case the count will also measure > + * the outer bytes. > + * > + * @param dev > + * Pointer to Ethernet device. > + * @param[in] actions > + * Associated actions (list terminated by the END action). > + * @param[out] actions_rx > + * Rx flow actions. > + * @param[out] actions_tx > + * Tx flow actions.. > + * @param[out] pattern_tx > + * The pattern items for the Tx flow. > + * @param[out] flow_id > + * The flow ID connected to this flow. > + * > + * @return > + * 0 on success. > + */ > +static int > +flow_hairpin_split(struct rte_eth_dev *dev, > + const struct rte_flow_action actions[], > + struct rte_flow_action actions_rx[], > + struct rte_flow_action actions_tx[], > + struct rte_flow_item pattern_tx[], > + uint32_t *flow_id) > +{ > + struct mlx5_priv *priv =3D dev->data->dev_private; > + const struct rte_flow_action_raw_encap *raw_encap; > + const struct rte_flow_action_raw_decap *raw_decap; > + struct mlx5_rte_flow_action_set_tag *set_tag; > + struct rte_flow_action *tag_action; > + struct mlx5_rte_flow_item_tag *tag_item; > + struct rte_flow_item *item; > + char *addr; > + struct rte_flow_error error; > + int encap =3D 0; > + > + mlx5_flow_id_get(priv->sh->flow_id_pool, flow_id); > + for (; actions->type !=3D RTE_FLOW_ACTION_TYPE_END; actions++) { > + switch (actions->type) { > + case RTE_FLOW_ACTION_TYPE_VXLAN_ENCAP: > + case RTE_FLOW_ACTION_TYPE_NVGRE_ENCAP: > + rte_memcpy(actions_tx, actions, > + sizeof(struct rte_flow_action)); > + actions_tx++; > + break; > + case RTE_FLOW_ACTION_TYPE_COUNT: > + if (encap) { > + rte_memcpy(actions_tx, actions, > + sizeof(struct rte_flow_action)); > + actions_tx++; > + } else { > + rte_memcpy(actions_rx, actions, > + sizeof(struct rte_flow_action)); > + actions_rx++; > + } > + break; > + case RTE_FLOW_ACTION_TYPE_RAW_ENCAP: > + raw_encap =3D actions->conf; > + if (raw_encap->size > > + (sizeof(struct rte_flow_item_eth) + > + sizeof(struct rte_flow_item_ipv4))) { > + memcpy(actions_tx, actions, > + sizeof(struct rte_flow_action)); > + actions_tx++; > + encap =3D 1; > + } else { > + rte_memcpy(actions_rx, actions, > + sizeof(struct rte_flow_action)); > + actions_rx++; > + } > + break; > + case RTE_FLOW_ACTION_TYPE_RAW_DECAP: > + raw_decap =3D actions->conf; > + if (raw_decap->size < > + (sizeof(struct rte_flow_item_eth) + > + sizeof(struct rte_flow_item_ipv4))) { > + memcpy(actions_tx, actions, > + sizeof(struct rte_flow_action)); > + actions_tx++; > + } else { > + rte_memcpy(actions_rx, actions, > + sizeof(struct rte_flow_action)); > + actions_rx++; > + } > + break; > + default: > + rte_memcpy(actions_rx, actions, > + sizeof(struct rte_flow_action)); > + actions_rx++; > + break; > + } > + } > + /* Add set meta action and end action for the Rx flow. */ > + tag_action =3D actions_rx; > + tag_action->type =3D MLX5_RTE_FLOW_ACTION_TYPE_TAG; > + actions_rx++; > + rte_memcpy(actions_rx, actions, sizeof(struct rte_flow_action)); > + actions_rx++; > + set_tag =3D (void *)actions_rx; > + set_tag->id =3D flow_get_reg_id(dev, MLX5_HAIRPIN_RX, 0, &error); > + set_tag->data =3D rte_cpu_to_be_32(*flow_id); > + tag_action->conf =3D set_tag; > + /* Create Tx item list. */ > + rte_memcpy(actions_tx, actions, sizeof(struct rte_flow_action)); > + addr =3D (void *)&pattern_tx[2]; > + item =3D pattern_tx; > + item->type =3D MLX5_RTE_FLOW_ITEM_TYPE_TAG; > + tag_item =3D (void *)addr; > + tag_item->data =3D rte_cpu_to_be_32(*flow_id); > + tag_item->id =3D flow_get_reg_id(dev, MLX5_HAIRPIN_TX, 0, &error); > + item->spec =3D tag_item; > + addr +=3D sizeof(struct mlx5_rte_flow_item_tag); > + tag_item =3D (void *)addr; > + tag_item->data =3D UINT32_MAX; > + tag_item->id =3D UINT16_MAX; > + item->mask =3D tag_item; > + addr +=3D sizeof(struct mlx5_rte_flow_item_tag); > + item->last =3D NULL; > + item++; > + item->type =3D RTE_FLOW_ITEM_TYPE_END; > + return 0; > +} > + > +/** > * Create a flow and add it to @p list. > * > * @param dev > @@ -2446,6 +2650,7 @@ uint32_t mlx5_flow_adjust_priority(struct > rte_eth_dev *dev, int32_t priority, > const struct rte_flow_action actions[], > bool external, struct rte_flow_error *error) { > + struct mlx5_priv *priv =3D dev->data->dev_private; > struct rte_flow *flow =3D NULL; > struct mlx5_flow *dev_flow; > const struct rte_flow_action_rss *rss; @@ -2453,16 +2658,44 @@ > uint32_t mlx5_flow_adjust_priority(struct rte_eth_dev *dev, int32_t prior= ity, > struct rte_flow_expand_rss buf; > uint8_t buffer[2048]; > } expand_buffer; > + union { > + struct rte_flow_action actions[MLX5_MAX_SPLIT_ACTIONS]; > + uint8_t buffer[2048]; > + } actions_rx; > + union { > + struct rte_flow_action actions[MLX5_MAX_SPLIT_ACTIONS]; > + uint8_t buffer[2048]; > + } actions_hairpin_tx; > + union { > + struct rte_flow_item items[MLX5_MAX_SPLIT_ITEMS]; > + uint8_t buffer[2048]; > + } items_tx; > struct rte_flow_expand_rss *buf =3D &expand_buffer.buf; > + const struct rte_flow_action *p_actions_rx =3D actions; > int ret; > uint32_t i; > uint32_t flow_size; > + int hairpin_flow =3D 0; > + uint32_t hairpin_id =3D 0; > + struct rte_flow_attr attr_tx =3D { .priority =3D 0 }; >=20 > - ret =3D flow_drv_validate(dev, attr, items, actions, external, error); > + hairpin_flow =3D flow_check_hairpin_split(dev, attr, actions); > + if (hairpin_flow > 0) { > + if (hairpin_flow > MLX5_MAX_SPLIT_ACTIONS) { > + rte_errno =3D EINVAL; > + return NULL; > + } > + flow_hairpin_split(dev, actions, actions_rx.actions, > + actions_hairpin_tx.actions, items_tx.items, > + &hairpin_id); > + p_actions_rx =3D actions_rx.actions; > + } > + ret =3D flow_drv_validate(dev, attr, items, p_actions_rx, external, > + error); > if (ret < 0) > - return NULL; > + goto error_before_flow; > flow_size =3D sizeof(struct rte_flow); > - rss =3D flow_get_rss_action(actions); > + rss =3D flow_get_rss_action(p_actions_rx); > if (rss) > flow_size +=3D RTE_ALIGN_CEIL(rss->queue_num * > sizeof(uint16_t), > sizeof(void *)); > @@ -2471,11 +2704,13 @@ uint32_t mlx5_flow_adjust_priority(struct > rte_eth_dev *dev, int32_t priority, > flow =3D rte_calloc(__func__, 1, flow_size, 0); > if (!flow) { > rte_errno =3D ENOMEM; > - return NULL; > + goto error_before_flow; > } > flow->drv_type =3D flow_get_drv_type(dev, attr); > flow->ingress =3D attr->ingress; > flow->transfer =3D attr->transfer; > + if (hairpin_id !=3D 0) > + flow->hairpin_flow_id =3D hairpin_id; > assert(flow->drv_type > MLX5_FLOW_TYPE_MIN && > flow->drv_type < MLX5_FLOW_TYPE_MAX); > flow->queue =3D (void *)(flow + 1); > @@ -2496,7 +2731,7 @@ uint32_t mlx5_flow_adjust_priority(struct > rte_eth_dev *dev, int32_t priority, > } > for (i =3D 0; i < buf->entries; ++i) { > dev_flow =3D flow_drv_prepare(flow, attr, buf- > >entry[i].pattern, > - actions, error); > + p_actions_rx, error); > if (!dev_flow) > goto error; > dev_flow->flow =3D flow; > @@ -2504,7 +2739,24 @@ uint32_t mlx5_flow_adjust_priority(struct > rte_eth_dev *dev, int32_t priority, > LIST_INSERT_HEAD(&flow->dev_flows, dev_flow, next); > ret =3D flow_drv_translate(dev, dev_flow, attr, > buf->entry[i].pattern, > - actions, error); > + p_actions_rx, error); > + if (ret < 0) > + goto error; > + } > + /* Create the tx flow. */ > + if (hairpin_flow) { > + attr_tx.group =3D MLX5_HAIRPIN_TX_TABLE; > + attr_tx.ingress =3D 0; > + attr_tx.egress =3D 1; > + dev_flow =3D flow_drv_prepare(flow, &attr_tx, items_tx.items, > + actions_hairpin_tx.actions, error); > + if (!dev_flow) > + goto error; > + dev_flow->flow =3D flow; > + LIST_INSERT_HEAD(&flow->dev_flows, dev_flow, next); > + ret =3D flow_drv_translate(dev, dev_flow, &attr_tx, > + items_tx.items, > + actions_hairpin_tx.actions, error); > if (ret < 0) > goto error; > } > @@ -2516,8 +2768,16 @@ uint32_t mlx5_flow_adjust_priority(struct > rte_eth_dev *dev, int32_t priority, > TAILQ_INSERT_TAIL(list, flow, next); > flow_rxq_flags_set(dev, flow); > return flow; > +error_before_flow: > + if (hairpin_id) > + mlx5_flow_id_release(priv->sh->flow_id_pool, > + hairpin_id); > + return NULL; > error: > ret =3D rte_errno; /* Save rte_errno before cleanup. */ > + if (flow->hairpin_flow_id) > + mlx5_flow_id_release(priv->sh->flow_id_pool, > + flow->hairpin_flow_id); > assert(flow); > flow_drv_destroy(dev, flow); > rte_free(flow); > @@ -2607,12 +2867,17 @@ struct rte_flow * flow_list_destroy(struct > rte_eth_dev *dev, struct mlx5_flows *list, > struct rte_flow *flow) > { > + struct mlx5_priv *priv =3D dev->data->dev_private; > + > /* > * Update RX queue flags only if port is started, otherwise it is > * already clean. > */ > if (dev->data->dev_started) > flow_rxq_flags_trim(dev, flow); > + if (flow->hairpin_flow_id) > + mlx5_flow_id_release(priv->sh->flow_id_pool, > + flow->hairpin_flow_id); > flow_drv_destroy(dev, flow); > TAILQ_REMOVE(list, flow, next); > rte_free(flow->fdir); > diff --git a/drivers/net/mlx5/mlx5_flow.h b/drivers/net/mlx5/mlx5_flow.h > index bb67380..90a289e 100644 > --- a/drivers/net/mlx5/mlx5_flow.h > +++ b/drivers/net/mlx5/mlx5_flow.h > @@ -434,6 +434,8 @@ struct mlx5_flow { > struct rte_flow *flow; /**< Pointer to the main flow. */ > uint64_t layers; > /**< Bit-fields of present layers, see MLX5_FLOW_LAYER_*. */ > + uint64_t actions; > + /**< Bit-fields of detected actions, see MLX5_FLOW_ACTION_*. */ > union { > #ifdef HAVE_IBV_FLOW_DV_SUPPORT > struct mlx5_flow_dv dv; > @@ -455,12 +457,11 @@ struct rte_flow { > uint16_t (*queue)[]; /**< Destination queues to redirect traffic to. */ > LIST_HEAD(dev_flows, mlx5_flow) dev_flows; > /**< Device flows that are part of the flow. */ > - uint64_t actions; > - /**< Bit-fields of detected actions, see MLX5_FLOW_ACTION_*. */ > struct mlx5_fdir *fdir; /**< Pointer to associated FDIR if any. */ > uint8_t ingress; /**< 1 if the flow is ingress. */ > uint32_t group; /**< The group index. */ > uint8_t transfer; /**< 1 if the flow is E-Switch flow. */ > + uint32_t hairpin_flow_id; /**< The flow id used for hairpin. */ > }; >=20 > typedef int (*mlx5_flow_validate_t)(struct rte_eth_dev *dev, @@ -504,15 > +505,6 @@ struct mlx5_flow_driver_ops { #define > MLX5_CNT_CONTAINER_UNUSED(sh, batch, thread) (&(sh)->cmng.ccont \ > [(~((sh)->cmng.mhi[batch] >> (thread)) & 0x1) * 2 + (batch)]) >=20 > -/* ID generation structure. */ > -struct mlx5_flow_id_pool { > - uint32_t *free_arr; /**< Pointer to the a array of free values. */ > - uint32_t base_index; > - /**< The next index that can be used without any free elements. */ > - uint32_t *curr; /**< Pointer to the index to pop. */ > - uint32_t *last; /**< Pointer to the last element in the empty arrray. > */ > -}; > - > /* mlx5_flow.c */ >=20 > struct mlx5_flow_id_pool *mlx5_flow_id_pool_alloc(void); diff --git > a/drivers/net/mlx5/mlx5_flow_dv.c b/drivers/net/mlx5/mlx5_flow_dv.c > index 2b48680..6828bd1 100644 > --- a/drivers/net/mlx5/mlx5_flow_dv.c > +++ b/drivers/net/mlx5/mlx5_flow_dv.c > @@ -5763,7 +5763,7 @@ struct field_modify_info modify_tcp[] =3D { > modify_action_position =3D actions_n++; > } > dev_flow->dv.actions_n =3D actions_n; > - flow->actions =3D action_flags; > + dev_flow->actions =3D action_flags; > for (; items->type !=3D RTE_FLOW_ITEM_TYPE_END; items++) { > int tunnel =3D !!(item_flags & MLX5_FLOW_LAYER_TUNNEL); > int item_type =3D items->type; > @@ -5985,7 +5985,7 @@ struct field_modify_info modify_tcp[] =3D { > LIST_FOREACH(dev_flow, &flow->dev_flows, next) { > dv =3D &dev_flow->dv; > n =3D dv->actions_n; > - if (flow->actions & MLX5_FLOW_ACTION_DROP) { > + if (dev_flow->actions & MLX5_FLOW_ACTION_DROP) { > if (flow->transfer) { > dv->actions[n++] =3D priv->sh- > >esw_drop_action; > } else { > @@ -6000,7 +6000,7 @@ struct field_modify_info modify_tcp[] =3D { > } > dv->actions[n++] =3D dv->hrxq->action; > } > - } else if (flow->actions & > + } else if (dev_flow->actions & > (MLX5_FLOW_ACTION_QUEUE | > MLX5_FLOW_ACTION_RSS)) { > struct mlx5_hrxq *hrxq; >=20 > @@ -6056,7 +6056,7 @@ struct field_modify_info modify_tcp[] =3D { > LIST_FOREACH(dev_flow, &flow->dev_flows, next) { > struct mlx5_flow_dv *dv =3D &dev_flow->dv; > if (dv->hrxq) { > - if (flow->actions & MLX5_FLOW_ACTION_DROP) > + if (dev_flow->actions & > MLX5_FLOW_ACTION_DROP) > mlx5_hrxq_drop_release(dev); > else > mlx5_hrxq_release(dev, dv->hrxq); > @@ -6290,7 +6290,7 @@ struct field_modify_info modify_tcp[] =3D { > dv->flow =3D NULL; > } > if (dv->hrxq) { > - if (flow->actions & MLX5_FLOW_ACTION_DROP) > + if (dev_flow->actions & > MLX5_FLOW_ACTION_DROP) > mlx5_hrxq_drop_release(dev); > else > mlx5_hrxq_release(dev, dv->hrxq); > diff --git a/drivers/net/mlx5/mlx5_flow_verbs.c > b/drivers/net/mlx5/mlx5_flow_verbs.c > index 23110f2..fd27f6c 100644 > --- a/drivers/net/mlx5/mlx5_flow_verbs.c > +++ b/drivers/net/mlx5/mlx5_flow_verbs.c > @@ -191,7 +191,7 @@ > { > #if defined(HAVE_IBV_DEVICE_COUNTERS_SET_V42) || \ > defined(HAVE_IBV_DEVICE_COUNTERS_SET_V45) > - if (flow->actions & MLX5_FLOW_ACTION_COUNT) { > + if (flow->counter->cs) { > struct rte_flow_query_count *qc =3D data; > uint64_t counters[2] =3D {0, 0}; > #if defined(HAVE_IBV_DEVICE_COUNTERS_SET_V42) > @@ -1410,7 +1410,6 @@ > const struct rte_flow_action actions[], > struct rte_flow_error *error) > { > - struct rte_flow *flow =3D dev_flow->flow; > uint64_t item_flags =3D 0; > uint64_t action_flags =3D 0; > uint64_t priority =3D attr->priority; > @@ -1460,7 +1459,7 @@ > "action not supported"); > } > } > - flow->actions =3D action_flags; > + dev_flow->actions =3D action_flags; > for (; items->type !=3D RTE_FLOW_ITEM_TYPE_END; items++) { > int tunnel =3D !!(item_flags & MLX5_FLOW_LAYER_TUNNEL); >=20 > @@ -1592,7 +1591,7 @@ > verbs->flow =3D NULL; > } > if (verbs->hrxq) { > - if (flow->actions & MLX5_FLOW_ACTION_DROP) > + if (dev_flow->actions & > MLX5_FLOW_ACTION_DROP) > mlx5_hrxq_drop_release(dev); > else > mlx5_hrxq_release(dev, verbs->hrxq); @@ - > 1656,7 +1655,7 @@ >=20 > LIST_FOREACH(dev_flow, &flow->dev_flows, next) { > verbs =3D &dev_flow->verbs; > - if (flow->actions & MLX5_FLOW_ACTION_DROP) { > + if (dev_flow->actions & MLX5_FLOW_ACTION_DROP) { > verbs->hrxq =3D mlx5_hrxq_drop_new(dev); > if (!verbs->hrxq) { > rte_flow_error_set > @@ -1717,7 +1716,7 @@ > LIST_FOREACH(dev_flow, &flow->dev_flows, next) { > verbs =3D &dev_flow->verbs; > if (verbs->hrxq) { > - if (flow->actions & MLX5_FLOW_ACTION_DROP) > + if (dev_flow->actions & > MLX5_FLOW_ACTION_DROP) > mlx5_hrxq_drop_release(dev); > else > mlx5_hrxq_release(dev, verbs->hrxq); diff -- > git a/drivers/net/mlx5/mlx5_rxq.c b/drivers/net/mlx5/mlx5_rxq.c index > bf39112..e51a0c6 100644 > --- a/drivers/net/mlx5/mlx5_rxq.c > +++ b/drivers/net/mlx5/mlx5_rxq.c > @@ -2113,6 +2113,32 @@ struct mlx5_rxq_ctrl * } >=20 > /** > + * Get a Rx queue type. > + * > + * @param dev > + * Pointer to Ethernet device. > + * @param idx > + * Rx queue index. > + * > + * @return > + * The Rx queue type. > + */ > +enum mlx5_rxq_type > +mlx5_rxq_get_type(struct rte_eth_dev *dev, uint16_t idx) { > + struct mlx5_priv *priv =3D dev->data->dev_private; > + struct mlx5_rxq_ctrl *rxq_ctrl =3D NULL; > + > + if ((*priv->rxqs)[idx]) { > + rxq_ctrl =3D container_of((*priv->rxqs)[idx], > + struct mlx5_rxq_ctrl, > + rxq); > + return rxq_ctrl->type; > + } > + return MLX5_RXQ_TYPE_UNDEFINED; > +} > + > +/** > * Create an indirection table. > * > * @param dev > diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h > index 8fa22e5..4707b29 100644 > --- a/drivers/net/mlx5/mlx5_rxtx.h > +++ b/drivers/net/mlx5/mlx5_rxtx.h > @@ -166,6 +166,7 @@ enum mlx5_rxq_obj_type { enum mlx5_rxq_type { > MLX5_RXQ_TYPE_STANDARD, /* Standard Rx queue. */ > MLX5_RXQ_TYPE_HAIRPIN, /* Hairpin Rx queue. */ > + MLX5_RXQ_TYPE_UNDEFINED, > }; >=20 > /* Verbs/DevX Rx queue elements. */ > @@ -408,6 +409,7 @@ struct mlx5_hrxq *mlx5_hrxq_get(struct rte_eth_dev > *dev, > const uint16_t *queues, uint32_t queues_n); > int mlx5_hrxq_release(struct rte_eth_dev *dev, struct mlx5_hrxq *hxrq); = int > mlx5_hrxq_verify(struct rte_eth_dev *dev); > +enum mlx5_rxq_type mlx5_rxq_get_type(struct rte_eth_dev *dev, uint16_t > +idx); > struct mlx5_hrxq *mlx5_hrxq_drop_new(struct rte_eth_dev *dev); void > mlx5_hrxq_drop_release(struct rte_eth_dev *dev); uint64_t > mlx5_get_rx_port_offloads(void); > -- > 1.8.3.1