From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: Received: from dpdk.org (dpdk.org [92.243.14.124]) by inbox.dpdk.org (Postfix) with ESMTP id 16AC4A0093 for ; Tue, 19 May 2020 15:11:36 +0200 (CEST) Received: from [92.243.14.124] (localhost [127.0.0.1]) by dpdk.org (Postfix) with ESMTP id 0EAEF1D702; Tue, 19 May 2020 15:11:36 +0200 (CEST) Received: from mail-wr1-f67.google.com (mail-wr1-f67.google.com [209.85.221.67]) by dpdk.org (Postfix) with ESMTP id 8DBAD1D702 for ; Tue, 19 May 2020 15:11:33 +0200 (CEST) Received: by mail-wr1-f67.google.com with SMTP id s8so15845363wrt.9 for ; Tue, 19 May 2020 06:11:33 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=8q4h/cTXkIJYsMVZaP1qUsno0zqutTwOELivbXObRUY=; b=P4F2s3eojPtD7nfa14W/1KkyII6HG5H7mwgcC5tp1lWypUoy1LvoOPFY5RmZAbzDem 0YUkMlJsDcB0bt8a1FZjmDISFD+TSewdzWAqL9vsVbkTzHQ+qhHZqUXIuzGdLVHrPHFG nZEmoBtzZHTw5q2bp1DO65F1G3auhMgUt3g52CA+hzNt+vb2MPeyg/kHS29A62GQy72A 5d5626IV8wgkwUR5KHYiHDundWw1u87XfgRHepIpP1CWxwkCTgLspABWFE32Nt7PF5vE hhldzH/sb9li8S+96lEkMCkRwwijpnXegmFbxVBdgiL347d4aTGKJ45YWHaiZdSHpBFE jJlg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=8q4h/cTXkIJYsMVZaP1qUsno0zqutTwOELivbXObRUY=; b=O1EW2ODYa4tStoSMqEN7+ZM0n7v2nhXRC7X06qq/RjrQDLLxMG4MsZXj/lrFzg1c36 hU4KOSO+dRu34YII3ccoJ9UwT4E2fLjtM5eyXjmrnv3xo53HrjZVGNjWWPIkEuQUlsr+ GJMQi6ePGSXXJxvn7+LfYzU289O6Di4g+mDwjyWq/8/Vue6W0ynPB6qObg5VEW4zR9NU UCshCb2uvqGNYpp3bDjOiZ0jMN5iu6deu3JnCMhBpPtbhWIWLCIx6jWlSnmt0eVEBR4C wErgoUoKx1uXZ6q2SK4ZjHh4X6Cga6zq+01SyR0fW8qu0PRu/ar/pJE0zUj/vVr0bCBB hrZw== X-Gm-Message-State: AOAM532SjTp1UQMDWjePrMuGiCHiB9UIO4tezmV5r7GfBJuTU9xX9pYm 3QUT2o1PE+qfuFRgUPf1WbXBwcOygGYZvTyh X-Google-Smtp-Source: ABdhPJwWrwj65VIgTBEMcfgTxUNGY50EUmxytDz7ieBun6I2mGj3kfpPTonBlZO9u5YSn8Lpn/0ftA== X-Received: by 2002:adf:e50d:: with SMTP id j13mr24667942wrm.383.1589893893161; Tue, 19 May 2020 06:11:33 -0700 (PDT) Received: from localhost ([88.98.246.218]) by smtp.gmail.com with ESMTPSA id t6sm3976105wma.4.2020.05.19.06.11.32 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 19 May 2020 06:11:32 -0700 (PDT) From: luca.boccassi@gmail.com To: Alexander Kozyrev Cc: Viacheslav Ovsiienko , dpdk stable Date: Tue, 19 May 2020 14:04:02 +0100 Message-Id: <20200519130549.112823-107-luca.boccassi@gmail.com> X-Mailer: git-send-email 2.20.1 In-Reply-To: <20200519130549.112823-1-luca.boccassi@gmail.com> References: <20200519125804.104349-1-luca.boccassi@gmail.com> <20200519130549.112823-1-luca.boccassi@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Subject: [dpdk-stable] patch 'net/mlx5: set dynamic flow metadata in Rx queues' has been queued to stable release 19.11.3 X-BeenThere: stable@dpdk.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: patches for DPDK stable branches List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Errors-To: stable-bounces@dpdk.org Sender: "stable" Hi, FYI, your patch has been queued to stable release 19.11.3 Note it hasn't been pushed to http://dpdk.org/browse/dpdk-stable yet. It will be pushed if I get no objections before 05/21/20. So please shout if anyone has objections. Also note that after the patch there's a diff of the upstream commit vs the patch applied to the branch. This will indicate if there was any rebasing needed to apply to the stable branch. If there were code changes for rebasing (ie: not only metadata diffs), please double check that the rebase was correctly done. Thanks. Luca Boccassi --- >From f0a019c95077c363ebf20fb07067e03c8b3f87a1 Mon Sep 17 00:00:00 2001 From: Alexander Kozyrev Date: Fri, 17 Apr 2020 17:14:53 +0000 Subject: [PATCH] net/mlx5: set dynamic flow metadata in Rx queues [ upstream commit 6c55b622a95693c6269ef8add75328813ee08e95 ] Using a global mbuf dynamic field for metadata incurs some performance penalty on a datapath. Store this information in the Rx queue descriptor for a better cache locality. Fixes: a18ac6113331 ("net/mlx5: add metadata support to Rx datapath") Signed-off-by: Alexander Kozyrev Acked-by: Viacheslav Ovsiienko --- drivers/net/mlx5/mlx5.h | 1 + drivers/net/mlx5/mlx5_flow.c | 29 ++++++++++++++ drivers/net/mlx5/mlx5_rxtx.c | 7 ++-- drivers/net/mlx5/mlx5_rxtx.h | 4 +- drivers/net/mlx5/mlx5_rxtx_vec_altivec.h | 27 +++++++------ drivers/net/mlx5/mlx5_rxtx_vec_neon.h | 49 ++++++++++++++---------- drivers/net/mlx5/mlx5_rxtx_vec_sse.h | 49 ++++++++++++++---------- drivers/net/mlx5/mlx5_trigger.c | 2 + lib/librte_ethdev/rte_flow.c | 2 +- lib/librte_ethdev/rte_flow.h | 2 +- 10 files changed, 113 insertions(+), 59 deletions(-) diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h index 73142085f7..bbf6ecda63 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h @@ -975,6 +975,7 @@ struct mlx5_flow_counter *mlx5_counter_alloc(struct rte_eth_dev *dev); void mlx5_counter_free(struct rte_eth_dev *dev, struct mlx5_flow_counter *cnt); int mlx5_counter_query(struct rte_eth_dev *dev, struct mlx5_flow_counter *cnt, bool clear, uint64_t *pkts, uint64_t *bytes); +void mlx5_flow_rxq_dynf_metadata_set(struct rte_eth_dev *dev); /* mlx5_mp.c */ void mlx5_mp_req_start_rxtx(struct rte_eth_dev *dev); diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c index 05e2d4c82a..bb8538e816 100644 --- a/drivers/net/mlx5/mlx5_flow.c +++ b/drivers/net/mlx5/mlx5_flow.c @@ -869,6 +869,35 @@ flow_rxq_flags_clear(struct rte_eth_dev *dev) } } +/** + * Set the Rx queue dynamic metadata (mask and offset) for a flow + * + * @param[in] dev + * Pointer to the Ethernet device structure. + */ +void +mlx5_flow_rxq_dynf_metadata_set(struct rte_eth_dev *dev) +{ + struct mlx5_priv *priv = dev->data->dev_private; + struct mlx5_rxq_data *data; + unsigned int i; + + for (i = 0; i != priv->rxqs_n; ++i) { + if (!(*priv->rxqs)[i]) + continue; + data = (*priv->rxqs)[i]; + if (!rte_flow_dynf_metadata_avail()) { + data->dynf_meta = 0; + data->flow_meta_mask = 0; + data->flow_meta_offset = -1; + } else { + data->dynf_meta = 1; + data->flow_meta_mask = rte_flow_dynf_metadata_mask; + data->flow_meta_offset = rte_flow_dynf_metadata_offs; + } + } +} + /* * return a pointer to the desired action in the list of actions. * diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c index ae89406d65..9d61dd68f7 100644 --- a/drivers/net/mlx5/mlx5_rxtx.c +++ b/drivers/net/mlx5/mlx5_rxtx.c @@ -1249,9 +1249,10 @@ rxq_cq_to_mbuf(struct mlx5_rxq_data *rxq, struct rte_mbuf *pkt, pkt->hash.fdir.hi = mlx5_flow_mark_get(mark); } } - if (rte_flow_dynf_metadata_avail() && cqe->flow_table_metadata) { - pkt->ol_flags |= PKT_RX_DYNF_METADATA; - *RTE_FLOW_DYNF_METADATA(pkt) = cqe->flow_table_metadata; + if (rxq->dynf_meta && cqe->flow_table_metadata) { + pkt->ol_flags |= rxq->flow_meta_mask; + *RTE_MBUF_DYNFIELD(pkt, rxq->flow_meta_offset, uint32_t *) = + cqe->flow_table_metadata; } if (rxq->csum) pkt->ol_flags |= rxq_cq_to_ol_flags(cqe); diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h index aa6fabbd3d..a50f057c1e 100644 --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h @@ -116,7 +116,7 @@ struct mlx5_rxq_data { unsigned int err_state:2; /* enum mlx5_rxq_err_state. */ unsigned int strd_scatter_en:1; /* Scattered packets from a stride. */ unsigned int lro:1; /* Enable LRO. */ - unsigned int :1; /* Remaining bits. */ + unsigned int dynf_meta:1; /* Dynamic metadata is configured. */ volatile uint32_t *rq_db; volatile uint32_t *cq_db; uint16_t port_id; @@ -154,6 +154,8 @@ struct mlx5_rxq_data { /* CQ (UAR) access lock required for 32bit implementations */ #endif uint32_t tunnel; /* Tunnel information. */ + uint64_t flow_meta_mask; + int32_t flow_meta_offset; } __rte_cache_aligned; enum mlx5_rxq_obj_type { diff --git a/drivers/net/mlx5/mlx5_rxtx_vec_altivec.h b/drivers/net/mlx5/mlx5_rxtx_vec_altivec.h index 4df13b6d80..5bc47c714b 100644 --- a/drivers/net/mlx5/mlx5_rxtx_vec_altivec.h +++ b/drivers/net/mlx5/mlx5_rxtx_vec_altivec.h @@ -263,17 +263,22 @@ rxq_cq_decompress_v(struct mlx5_rxq_data *rxq, volatile struct mlx5_cqe *cq, elts[pos + 2]->hash.fdir.hi = flow_tag; elts[pos + 3]->hash.fdir.hi = flow_tag; } - if (rte_flow_dynf_metadata_avail()) { - const uint32_t meta = *RTE_FLOW_DYNF_METADATA(t_pkt); + if (!!rxq->flow_meta_mask) { + int32_t offs = rxq->flow_meta_offset; + const uint32_t meta = + *RTE_MBUF_DYNFIELD(t_pkt, offs, uint32_t *); /* Check if title packet has valid metadata. */ if (meta) { - assert(t_pkt->ol_flags & - PKT_RX_DYNF_METADATA); - *RTE_FLOW_DYNF_METADATA(elts[pos]) = meta; - *RTE_FLOW_DYNF_METADATA(elts[pos + 1]) = meta; - *RTE_FLOW_DYNF_METADATA(elts[pos + 2]) = meta; - *RTE_FLOW_DYNF_METADATA(elts[pos + 3]) = meta; + assert(t_pkt->ol_flags & offs); + *RTE_MBUF_DYNFIELD(elts[pos], offs, + uint32_t *) = meta; + *RTE_MBUF_DYNFIELD(elts[pos + 1], offs, + uint32_t *) = meta; + *RTE_MBUF_DYNFIELD(elts[pos + 2], offs, + uint32_t *) = meta; + *RTE_MBUF_DYNFIELD(elts[pos + 3], offs, + uint32_t *) = meta; } } @@ -1023,9 +1028,9 @@ rxq_burst_v(struct mlx5_rxq_data *rxq, struct rte_mbuf **pkts, uint16_t pkts_n, pkts[pos + 3]->timestamp = rte_be_to_cpu_64(cq[pos + p3].timestamp); } - if (rte_flow_dynf_metadata_avail()) { - uint64_t flag = rte_flow_dynf_metadata_mask; - int offs = rte_flow_dynf_metadata_offs; + if (!!rxq->flow_meta_mask) { + uint64_t flag = rxq->flow_meta_mask; + int32_t offs = rxq->flow_meta_offset; uint32_t metadata; /* This code is subject for futher optimization. */ diff --git a/drivers/net/mlx5/mlx5_rxtx_vec_neon.h b/drivers/net/mlx5/mlx5_rxtx_vec_neon.h index b3a3e028ee..2880f97af1 100644 --- a/drivers/net/mlx5/mlx5_rxtx_vec_neon.h +++ b/drivers/net/mlx5/mlx5_rxtx_vec_neon.h @@ -205,17 +205,22 @@ rxq_cq_decompress_v(struct mlx5_rxq_data *rxq, volatile struct mlx5_cqe *cq, elts[pos + 2]->hash.fdir.hi = flow_tag; elts[pos + 3]->hash.fdir.hi = flow_tag; } - if (rte_flow_dynf_metadata_avail()) { - const uint32_t meta = *RTE_FLOW_DYNF_METADATA(t_pkt); + if (!!rxq->flow_meta_mask) { + int32_t offs = rxq->flow_meta_offset; + const uint32_t meta = + *RTE_MBUF_DYNFIELD(t_pkt, offs, uint32_t *); /* Check if title packet has valid metadata. */ if (meta) { - assert(t_pkt->ol_flags & - PKT_RX_DYNF_METADATA); - *RTE_FLOW_DYNF_METADATA(elts[pos]) = meta; - *RTE_FLOW_DYNF_METADATA(elts[pos + 1]) = meta; - *RTE_FLOW_DYNF_METADATA(elts[pos + 2]) = meta; - *RTE_FLOW_DYNF_METADATA(elts[pos + 3]) = meta; + assert(t_pkt->ol_flags & offs); + *RTE_MBUF_DYNFIELD(elts[pos], offs, + uint32_t *) = meta; + *RTE_MBUF_DYNFIELD(elts[pos + 1], offs, + uint32_t *) = meta; + *RTE_MBUF_DYNFIELD(elts[pos + 2], offs, + uint32_t *) = meta; + *RTE_MBUF_DYNFIELD(elts[pos + 3], offs, + uint32_t *) = meta; } } pos += MLX5_VPMD_DESCS_PER_LOOP; @@ -700,28 +705,30 @@ rxq_burst_v(struct mlx5_rxq_data *rxq, struct rte_mbuf **pkts, uint16_t pkts_n, container_of(p3, struct mlx5_cqe, pkt_info)->timestamp); } - if (rte_flow_dynf_metadata_avail()) { + if (!!rxq->flow_meta_mask) { /* This code is subject for futher optimization. */ - *RTE_FLOW_DYNF_METADATA(elts[pos]) = + int32_t offs = rxq->flow_meta_offset; + + *RTE_MBUF_DYNFIELD(pkts[pos], offs, uint32_t *) = container_of(p0, struct mlx5_cqe, pkt_info)->flow_table_metadata; - *RTE_FLOW_DYNF_METADATA(elts[pos + 1]) = + *RTE_MBUF_DYNFIELD(pkts[pos], offs, uint32_t *) = container_of(p1, struct mlx5_cqe, pkt_info)->flow_table_metadata; - *RTE_FLOW_DYNF_METADATA(elts[pos + 2]) = + *RTE_MBUF_DYNFIELD(pkts[pos], offs, uint32_t *) = container_of(p2, struct mlx5_cqe, pkt_info)->flow_table_metadata; - *RTE_FLOW_DYNF_METADATA(elts[pos + 3]) = + *RTE_MBUF_DYNFIELD(pkts[pos], offs, uint32_t *) = container_of(p3, struct mlx5_cqe, pkt_info)->flow_table_metadata; - if (*RTE_FLOW_DYNF_METADATA(elts[pos])) - elts[pos]->ol_flags |= PKT_RX_DYNF_METADATA; - if (*RTE_FLOW_DYNF_METADATA(elts[pos + 1])) - elts[pos + 1]->ol_flags |= PKT_RX_DYNF_METADATA; - if (*RTE_FLOW_DYNF_METADATA(elts[pos + 2])) - elts[pos + 2]->ol_flags |= PKT_RX_DYNF_METADATA; - if (*RTE_FLOW_DYNF_METADATA(elts[pos + 3])) - elts[pos + 3]->ol_flags |= PKT_RX_DYNF_METADATA; + if (*RTE_MBUF_DYNFIELD(pkts[pos], offs, uint32_t *)) + elts[pos]->ol_flags |= rxq->flow_meta_mask; + if (*RTE_MBUF_DYNFIELD(pkts[pos + 1], offs, uint32_t *)) + elts[pos + 1]->ol_flags |= rxq->flow_meta_mask; + if (*RTE_MBUF_DYNFIELD(pkts[pos + 2], offs, uint32_t *)) + elts[pos + 2]->ol_flags |= rxq->flow_meta_mask; + if (*RTE_MBUF_DYNFIELD(pkts[pos + 3], offs, uint32_t *)) + elts[pos + 3]->ol_flags |= rxq->flow_meta_mask; } #ifdef MLX5_PMD_SOFT_COUNTERS /* Add up received bytes count. */ diff --git a/drivers/net/mlx5/mlx5_rxtx_vec_sse.h b/drivers/net/mlx5/mlx5_rxtx_vec_sse.h index 8aa3f0489c..ac8568ed4b 100644 --- a/drivers/net/mlx5/mlx5_rxtx_vec_sse.h +++ b/drivers/net/mlx5/mlx5_rxtx_vec_sse.h @@ -190,17 +190,22 @@ rxq_cq_decompress_v(struct mlx5_rxq_data *rxq, volatile struct mlx5_cqe *cq, elts[pos + 2]->hash.fdir.hi = flow_tag; elts[pos + 3]->hash.fdir.hi = flow_tag; } - if (rte_flow_dynf_metadata_avail()) { - const uint32_t meta = *RTE_FLOW_DYNF_METADATA(t_pkt); + if (rxq->dynf_meta) { + int32_t offs = rxq->flow_meta_offset; + const uint32_t meta = + *RTE_MBUF_DYNFIELD(t_pkt, offs, uint32_t *); /* Check if title packet has valid metadata. */ if (meta) { - assert(t_pkt->ol_flags & - PKT_RX_DYNF_METADATA); - *RTE_FLOW_DYNF_METADATA(elts[pos]) = meta; - *RTE_FLOW_DYNF_METADATA(elts[pos + 1]) = meta; - *RTE_FLOW_DYNF_METADATA(elts[pos + 2]) = meta; - *RTE_FLOW_DYNF_METADATA(elts[pos + 3]) = meta; + assert(t_pkt->ol_flags & offs); + *RTE_MBUF_DYNFIELD(elts[pos], offs, + uint32_t *) = meta; + *RTE_MBUF_DYNFIELD(elts[pos + 1], offs, + uint32_t *) = meta; + *RTE_MBUF_DYNFIELD(elts[pos + 2], offs, + uint32_t *) = meta; + *RTE_MBUF_DYNFIELD(elts[pos + 3], offs, + uint32_t *) = meta; } } pos += MLX5_VPMD_DESCS_PER_LOOP; @@ -652,24 +657,26 @@ rxq_burst_v(struct mlx5_rxq_data *rxq, struct rte_mbuf **pkts, uint16_t pkts_n, pkts[pos + 3]->timestamp = rte_be_to_cpu_64(cq[pos + p3].timestamp); } - if (rte_flow_dynf_metadata_avail()) { + if (rxq->dynf_meta) { /* This code is subject for futher optimization. */ - *RTE_FLOW_DYNF_METADATA(pkts[pos]) = + int32_t offs = rxq->flow_meta_offset; + + *RTE_MBUF_DYNFIELD(pkts[pos], offs, uint32_t *) = cq[pos].flow_table_metadata; - *RTE_FLOW_DYNF_METADATA(pkts[pos + 1]) = + *RTE_MBUF_DYNFIELD(pkts[pos + 1], offs, uint32_t *) = cq[pos + p1].flow_table_metadata; - *RTE_FLOW_DYNF_METADATA(pkts[pos + 2]) = + *RTE_MBUF_DYNFIELD(pkts[pos + 2], offs, uint32_t *) = cq[pos + p2].flow_table_metadata; - *RTE_FLOW_DYNF_METADATA(pkts[pos + 3]) = + *RTE_MBUF_DYNFIELD(pkts[pos + 3], offs, uint32_t *) = cq[pos + p3].flow_table_metadata; - if (*RTE_FLOW_DYNF_METADATA(pkts[pos])) - pkts[pos]->ol_flags |= PKT_RX_DYNF_METADATA; - if (*RTE_FLOW_DYNF_METADATA(pkts[pos + 1])) - pkts[pos + 1]->ol_flags |= PKT_RX_DYNF_METADATA; - if (*RTE_FLOW_DYNF_METADATA(pkts[pos + 2])) - pkts[pos + 2]->ol_flags |= PKT_RX_DYNF_METADATA; - if (*RTE_FLOW_DYNF_METADATA(pkts[pos + 3])) - pkts[pos + 3]->ol_flags |= PKT_RX_DYNF_METADATA; + if (*RTE_MBUF_DYNFIELD(pkts[pos], offs, uint32_t *)) + pkts[pos]->ol_flags |= rxq->flow_meta_mask; + if (*RTE_MBUF_DYNFIELD(pkts[pos + 1], offs, uint32_t *)) + pkts[pos + 1]->ol_flags |= rxq->flow_meta_mask; + if (*RTE_MBUF_DYNFIELD(pkts[pos + 2], offs, uint32_t *)) + pkts[pos + 2]->ol_flags |= rxq->flow_meta_mask; + if (*RTE_MBUF_DYNFIELD(pkts[pos + 3], offs, uint32_t *)) + pkts[pos + 3]->ol_flags |= rxq->flow_meta_mask; } #ifdef MLX5_PMD_SOFT_COUNTERS /* Add up received bytes count. */ diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c index e3ea2862a5..6fc4190f4e 100644 --- a/drivers/net/mlx5/mlx5_trigger.c +++ b/drivers/net/mlx5/mlx5_trigger.c @@ -314,6 +314,8 @@ mlx5_dev_start(struct rte_eth_dev *dev) dev->data->port_id); goto error; } + /* Set a mask and offset of dynamic metadata flows into Rx queues*/ + mlx5_flow_rxq_dynf_metadata_set(dev); ret = mlx5_flow_start(dev, &priv->flows); if (ret) { DRV_LOG(DEBUG, "port %u failed to set flows", diff --git a/lib/librte_ethdev/rte_flow.c b/lib/librte_ethdev/rte_flow.c index 87a3e8c4c6..391165646a 100644 --- a/lib/librte_ethdev/rte_flow.c +++ b/lib/librte_ethdev/rte_flow.c @@ -19,7 +19,7 @@ #include "rte_flow.h" /* Mbuf dynamic field name for metadata. */ -int rte_flow_dynf_metadata_offs = -1; +int32_t rte_flow_dynf_metadata_offs = -1; /* Mbuf dynamic field flag bit number for metadata. */ uint64_t rte_flow_dynf_metadata_mask; diff --git a/lib/librte_ethdev/rte_flow.h b/lib/librte_ethdev/rte_flow.h index 58b50265d2..693824da8a 100644 --- a/lib/librte_ethdev/rte_flow.h +++ b/lib/librte_ethdev/rte_flow.h @@ -2531,7 +2531,7 @@ struct rte_flow_action_set_meta { }; /* Mbuf dynamic field offset for metadata. */ -extern int rte_flow_dynf_metadata_offs; +extern int32_t rte_flow_dynf_metadata_offs; /* Mbuf dynamic field flag mask for metadata. */ extern uint64_t rte_flow_dynf_metadata_mask; -- 2.20.1 --- Diff of the applied patch vs upstream commit (please double-check if non-empty: --- --- - 2020-05-19 14:04:48.827164928 +0100 +++ 0107-net-mlx5-set-dynamic-flow-metadata-in-Rx-queues.patch 2020-05-19 14:04:44.352650837 +0100 @@ -1,14 +1,15 @@ -From 6c55b622a95693c6269ef8add75328813ee08e95 Mon Sep 17 00:00:00 2001 +From f0a019c95077c363ebf20fb07067e03c8b3f87a1 Mon Sep 17 00:00:00 2001 From: Alexander Kozyrev Date: Fri, 17 Apr 2020 17:14:53 +0000 Subject: [PATCH] net/mlx5: set dynamic flow metadata in Rx queues +[ upstream commit 6c55b622a95693c6269ef8add75328813ee08e95 ] + Using a global mbuf dynamic field for metadata incurs some performance penalty on a datapath. Store this information in the Rx queue descriptor for a better cache locality. Fixes: a18ac6113331 ("net/mlx5: add metadata support to Rx datapath") -Cc: stable@dpdk.org Signed-off-by: Alexander Kozyrev Acked-by: Viacheslav Ovsiienko @@ -26,22 +27,22 @@ 10 files changed, 113 insertions(+), 59 deletions(-) diff --git a/drivers/net/mlx5/mlx5.h b/drivers/net/mlx5/mlx5.h -index 770425c4f0..50349abf34 100644 +index 73142085f7..bbf6ecda63 100644 --- a/drivers/net/mlx5/mlx5.h +++ b/drivers/net/mlx5/mlx5.h -@@ -753,6 +753,7 @@ int mlx5_counter_query(struct rte_eth_dev *dev, uint32_t cnt, +@@ -975,6 +975,7 @@ struct mlx5_flow_counter *mlx5_counter_alloc(struct rte_eth_dev *dev); + void mlx5_counter_free(struct rte_eth_dev *dev, struct mlx5_flow_counter *cnt); + int mlx5_counter_query(struct rte_eth_dev *dev, struct mlx5_flow_counter *cnt, bool clear, uint64_t *pkts, uint64_t *bytes); - int mlx5_flow_dev_dump(struct rte_eth_dev *dev, FILE *file, - struct rte_flow_error *error); +void mlx5_flow_rxq_dynf_metadata_set(struct rte_eth_dev *dev); /* mlx5_mp.c */ - int mlx5_mp_primary_handle(const struct rte_mp_msg *mp_msg, const void *peer); + void mlx5_mp_req_start_rxtx(struct rte_eth_dev *dev); diff --git a/drivers/net/mlx5/mlx5_flow.c b/drivers/net/mlx5/mlx5_flow.c -index 84aa069db4..cba1f23e81 100644 +index 05e2d4c82a..bb8538e816 100644 --- a/drivers/net/mlx5/mlx5_flow.c +++ b/drivers/net/mlx5/mlx5_flow.c -@@ -894,6 +894,35 @@ flow_rxq_flags_clear(struct rte_eth_dev *dev) +@@ -869,6 +869,35 @@ flow_rxq_flags_clear(struct rte_eth_dev *dev) } } @@ -78,10 +79,10 @@ * return a pointer to the desired action in the list of actions. * diff --git a/drivers/net/mlx5/mlx5_rxtx.c b/drivers/net/mlx5/mlx5_rxtx.c -index 3e583d49a6..0010b423df 100644 +index ae89406d65..9d61dd68f7 100644 --- a/drivers/net/mlx5/mlx5_rxtx.c +++ b/drivers/net/mlx5/mlx5_rxtx.c -@@ -1339,9 +1339,10 @@ rxq_cq_to_mbuf(struct mlx5_rxq_data *rxq, struct rte_mbuf *pkt, +@@ -1249,9 +1249,10 @@ rxq_cq_to_mbuf(struct mlx5_rxq_data *rxq, struct rte_mbuf *pkt, pkt->hash.fdir.hi = mlx5_flow_mark_get(mark); } } @@ -96,10 +97,10 @@ if (rxq->csum) pkt->ol_flags |= rxq_cq_to_ol_flags(cqe); diff --git a/drivers/net/mlx5/mlx5_rxtx.h b/drivers/net/mlx5/mlx5_rxtx.h -index 869521841e..48f2b79411 100644 +index aa6fabbd3d..a50f057c1e 100644 --- a/drivers/net/mlx5/mlx5_rxtx.h +++ b/drivers/net/mlx5/mlx5_rxtx.h -@@ -121,7 +121,7 @@ struct mlx5_rxq_data { +@@ -116,7 +116,7 @@ struct mlx5_rxq_data { unsigned int err_state:2; /* enum mlx5_rxq_err_state. */ unsigned int strd_scatter_en:1; /* Scattered packets from a stride. */ unsigned int lro:1; /* Enable LRO. */ @@ -108,7 +109,7 @@ volatile uint32_t *rq_db; volatile uint32_t *cq_db; uint16_t port_id; -@@ -159,6 +159,8 @@ struct mlx5_rxq_data { +@@ -154,6 +154,8 @@ struct mlx5_rxq_data { /* CQ (UAR) access lock required for 32bit implementations */ #endif uint32_t tunnel; /* Tunnel information. */ @@ -118,10 +119,10 @@ enum mlx5_rxq_obj_type { diff --git a/drivers/net/mlx5/mlx5_rxtx_vec_altivec.h b/drivers/net/mlx5/mlx5_rxtx_vec_altivec.h -index 9778b0bbc1..45ff8e65b6 100644 +index 4df13b6d80..5bc47c714b 100644 --- a/drivers/net/mlx5/mlx5_rxtx_vec_altivec.h +++ b/drivers/net/mlx5/mlx5_rxtx_vec_altivec.h -@@ -264,17 +264,22 @@ rxq_cq_decompress_v(struct mlx5_rxq_data *rxq, volatile struct mlx5_cqe *cq, +@@ -263,17 +263,22 @@ rxq_cq_decompress_v(struct mlx5_rxq_data *rxq, volatile struct mlx5_cqe *cq, elts[pos + 2]->hash.fdir.hi = flow_tag; elts[pos + 3]->hash.fdir.hi = flow_tag; } @@ -134,13 +135,13 @@ /* Check if title packet has valid metadata. */ if (meta) { -- MLX5_ASSERT(t_pkt->ol_flags & +- assert(t_pkt->ol_flags & - PKT_RX_DYNF_METADATA); - *RTE_FLOW_DYNF_METADATA(elts[pos]) = meta; - *RTE_FLOW_DYNF_METADATA(elts[pos + 1]) = meta; - *RTE_FLOW_DYNF_METADATA(elts[pos + 2]) = meta; - *RTE_FLOW_DYNF_METADATA(elts[pos + 3]) = meta; -+ MLX5_ASSERT(t_pkt->ol_flags & offs); ++ assert(t_pkt->ol_flags & offs); + *RTE_MBUF_DYNFIELD(elts[pos], offs, + uint32_t *) = meta; + *RTE_MBUF_DYNFIELD(elts[pos + 1], offs, @@ -166,7 +167,7 @@ /* This code is subject for futher optimization. */ diff --git a/drivers/net/mlx5/mlx5_rxtx_vec_neon.h b/drivers/net/mlx5/mlx5_rxtx_vec_neon.h -index 7b6c5db491..d39e72680b 100644 +index b3a3e028ee..2880f97af1 100644 --- a/drivers/net/mlx5/mlx5_rxtx_vec_neon.h +++ b/drivers/net/mlx5/mlx5_rxtx_vec_neon.h @@ -205,17 +205,22 @@ rxq_cq_decompress_v(struct mlx5_rxq_data *rxq, volatile struct mlx5_cqe *cq, @@ -182,13 +183,13 @@ /* Check if title packet has valid metadata. */ if (meta) { -- MLX5_ASSERT(t_pkt->ol_flags & +- assert(t_pkt->ol_flags & - PKT_RX_DYNF_METADATA); - *RTE_FLOW_DYNF_METADATA(elts[pos]) = meta; - *RTE_FLOW_DYNF_METADATA(elts[pos + 1]) = meta; - *RTE_FLOW_DYNF_METADATA(elts[pos + 2]) = meta; - *RTE_FLOW_DYNF_METADATA(elts[pos + 3]) = meta; -+ MLX5_ASSERT(t_pkt->ol_flags & offs); ++ assert(t_pkt->ol_flags & offs); + *RTE_MBUF_DYNFIELD(elts[pos], offs, + uint32_t *) = meta; + *RTE_MBUF_DYNFIELD(elts[pos + 1], offs, @@ -200,7 +201,7 @@ } } pos += MLX5_VPMD_DESCS_PER_LOOP; -@@ -701,28 +706,30 @@ rxq_burst_v(struct mlx5_rxq_data *rxq, struct rte_mbuf **pkts, uint16_t pkts_n, +@@ -700,28 +705,30 @@ rxq_burst_v(struct mlx5_rxq_data *rxq, struct rte_mbuf **pkts, uint16_t pkts_n, container_of(p3, struct mlx5_cqe, pkt_info)->timestamp); } @@ -245,10 +246,10 @@ #ifdef MLX5_PMD_SOFT_COUNTERS /* Add up received bytes count. */ diff --git a/drivers/net/mlx5/mlx5_rxtx_vec_sse.h b/drivers/net/mlx5/mlx5_rxtx_vec_sse.h -index 4b711f0f1f..f110f73b4a 100644 +index 8aa3f0489c..ac8568ed4b 100644 --- a/drivers/net/mlx5/mlx5_rxtx_vec_sse.h +++ b/drivers/net/mlx5/mlx5_rxtx_vec_sse.h -@@ -192,17 +192,22 @@ rxq_cq_decompress_v(struct mlx5_rxq_data *rxq, volatile struct mlx5_cqe *cq, +@@ -190,17 +190,22 @@ rxq_cq_decompress_v(struct mlx5_rxq_data *rxq, volatile struct mlx5_cqe *cq, elts[pos + 2]->hash.fdir.hi = flow_tag; elts[pos + 3]->hash.fdir.hi = flow_tag; } @@ -261,13 +262,13 @@ /* Check if title packet has valid metadata. */ if (meta) { -- MLX5_ASSERT(t_pkt->ol_flags & +- assert(t_pkt->ol_flags & - PKT_RX_DYNF_METADATA); - *RTE_FLOW_DYNF_METADATA(elts[pos]) = meta; - *RTE_FLOW_DYNF_METADATA(elts[pos + 1]) = meta; - *RTE_FLOW_DYNF_METADATA(elts[pos + 2]) = meta; - *RTE_FLOW_DYNF_METADATA(elts[pos + 3]) = meta; -+ MLX5_ASSERT(t_pkt->ol_flags & offs); ++ assert(t_pkt->ol_flags & offs); + *RTE_MBUF_DYNFIELD(elts[pos], offs, + uint32_t *) = meta; + *RTE_MBUF_DYNFIELD(elts[pos + 1], offs, @@ -279,7 +280,7 @@ } } pos += MLX5_VPMD_DESCS_PER_LOOP; -@@ -654,24 +659,26 @@ rxq_burst_v(struct mlx5_rxq_data *rxq, struct rte_mbuf **pkts, uint16_t pkts_n, +@@ -652,24 +657,26 @@ rxq_burst_v(struct mlx5_rxq_data *rxq, struct rte_mbuf **pkts, uint16_t pkts_n, pkts[pos + 3]->timestamp = rte_be_to_cpu_64(cq[pos + p3].timestamp); } @@ -320,20 +321,20 @@ #ifdef MLX5_PMD_SOFT_COUNTERS /* Add up received bytes count. */ diff --git a/drivers/net/mlx5/mlx5_trigger.c b/drivers/net/mlx5/mlx5_trigger.c -index 759670408b..feb9154199 100644 +index e3ea2862a5..6fc4190f4e 100644 --- a/drivers/net/mlx5/mlx5_trigger.c +++ b/drivers/net/mlx5/mlx5_trigger.c -@@ -323,6 +323,8 @@ mlx5_dev_start(struct rte_eth_dev *dev) +@@ -314,6 +314,8 @@ mlx5_dev_start(struct rte_eth_dev *dev) dev->data->port_id); goto error; } + /* Set a mask and offset of dynamic metadata flows into Rx queues*/ + mlx5_flow_rxq_dynf_metadata_set(dev); - /* - * In non-cached mode, it only needs to start the default mreg copy - * action and no flow created by application exists anymore. + ret = mlx5_flow_start(dev, &priv->flows); + if (ret) { + DRV_LOG(DEBUG, "port %u failed to set flows", diff --git a/lib/librte_ethdev/rte_flow.c b/lib/librte_ethdev/rte_flow.c -index 3699edce49..1685be5f73 100644 +index 87a3e8c4c6..391165646a 100644 --- a/lib/librte_ethdev/rte_flow.c +++ b/lib/librte_ethdev/rte_flow.c @@ -19,7 +19,7 @@ @@ -346,10 +347,10 @@ /* Mbuf dynamic field flag bit number for metadata. */ uint64_t rte_flow_dynf_metadata_mask; diff --git a/lib/librte_ethdev/rte_flow.h b/lib/librte_ethdev/rte_flow.h -index fab44f6c0b..132b44edc6 100644 +index 58b50265d2..693824da8a 100644 --- a/lib/librte_ethdev/rte_flow.h +++ b/lib/librte_ethdev/rte_flow.h -@@ -2653,7 +2653,7 @@ struct rte_flow_action_set_dscp { +@@ -2531,7 +2531,7 @@ struct rte_flow_action_set_meta { }; /* Mbuf dynamic field offset for metadata. */